Adaptive battery state of health manipulation method and system based on reinforcement learning

Abstract

This invention discloses an adaptive battery health state manipulation method and system based on reinforcement learning. The method includes: constructing a base model and a residual detection defense system; constructing an attack environment module and a composite state space containing multi-dimensional feedback information; training an agent based on a dual-deep Q-network to construct a DDQN policy model; and generating attack vectors targeting real battery data based on the DDQN policy model. This method achieves long-term and stable manipulation of health state estimation results across time scales, solves the problem of conflict between attack effectiveness and stealth in existing technologies, and significantly improves the accuracy of generating adversarial use cases.

Description

Technical Field

[0001] This invention relates to the field of cloud-based battery management system technology, and in particular to an adaptive battery health status control method and system based on reinforcement learning. Background Technology

[0002] Lithium-ion batteries have become a core energy storage technology for electric vehicles, smart grids, and portable electronic devices due to their significant advantages such as high energy density, long cycle life, and environmental friendliness. In battery management systems, accurately estimating the battery's state of health is crucial for predicting remaining lifespan, optimizing charge / discharge strategies, and ensuring safe system operation.

[0003] In recent years, data-driven approaches have demonstrated exceptional capabilities in handling degradation patterns in complex battery datasets, adapting to diverse operating conditions and achieving high-precision health state estimation. Current mainstream technologies include deep learning models (such as using gated recurrent unit neural networks combined with hidden Markov models to capture nonlinear degradation features), ensemble learning frameworks, and transfer learning methods utilizing limited data. However, these high-performance models typically place high demands on computational resources and data scale, often exceeding the hardware processing capabilities of onboard battery management systems. Therefore, cloud-based collaborative battery management architectures have become a current technological trend. By leveraging cloud computing and storage resources, cloud-based collaborative battery management architectures can run more advanced data-driven algorithms, significantly improving the accuracy of state estimation and anomaly analysis capabilities by uploading massive amounts of battery metrics (such as voltage, current, temperature, and internal resistance) to the cloud platform.

[0004] However, while cloud-based collaborative battery management systems overcome local resource limitations, they also significantly expand the system's cyber-physical attack surface. This architecture tightly couples physical sensing hardware with cloud analytics through continuous, high-frequency data exchange, introducing multiple security vulnerabilities. For example, unencrypted or weakly authenticated in-vehicle networks and diagnostic ports provide attackers with entry points, allowing them to maliciously tamper with voltage, current, and temperature sensor data via man-in-the-middle attacks, misleading state estimates and potentially leading to accelerated battery degradation or thermal runaway. Among various cyberattacks, spoofed data injection attacks, due to their high destructiveness and stealth, have become a serious threat to battery management system security. Attackers maliciously manipulate measurement data to compromise data integrity, thereby misleading cloud algorithms to generate incorrect health state estimates, ultimately jeopardizing battery safety.

[0005] Currently, to assess the robustness of health status estimation models, existing adversarial testing techniques mainly employ random perturbation attacks, swarm intelligence algorithms such as particle swarm optimization, and methods based on stochastic matrix theory. However, these existing techniques lack the ability to provide dynamic feedback and adaptive adjustments under operating conditions, and when facing the residual detection mechanisms of modern battery management systems, they often fail to maintain sufficient stealth while ensuring attack strength. Summary of the Invention

[0006] The main objective of this invention is to provide an adaptive battery health state control method and system based on reinforcement learning, so as to solve the technical problem in the prior art that it is impossible to maintain sufficient stealth while ensuring attack strength.

[0007] To achieve the above objectives, the present invention provides an adaptive battery health state control method based on reinforcement learning, the method comprising:

[0008] S10, Constructing a basic model and residual detection defense system;

[0009] S20, construct the attack environment module and a composite state space containing multi-dimensional feedback information;

[0010] S30, based on a dual-deep Q-network, trains the agent to construct a DDQN policy model;

[0011] S40, Based on the DDQN strategy model, generate an attack vector targeting real battery data.

[0012] Optionally, step S10 includes the following steps:

[0013] S110 constructs various data-driven models, including multilayer perceptron, gated recurrent unit, long short-term memory network, and a coupled model of convolutional neural network and long short-term memory network.

[0014] S120, Construct a voltage and current prediction model based on time-series correlation learning;

[0015] S130, Based on the voltage and current prediction model, using the current voltage and current time series as input, predict the expected observation value at the next moment in real time;

[0016] S140, compare the expected observation with the actual sampled value of the sensor, and use Euclidean distance to quantify the deviation between the expected observation and the sensor into a residual signal.

[0017] Optionally, step S20 includes the following steps:

[0018] S210, Module for building the attack environment;

[0019] S220, which includes a benchmark evaluation module, a DDQN agent training module, and an attack vector generation and execution module;

[0020] S230 defines a composite state space containing multidimensional feedback information;

[0021] S240, Introduce gradient features into the state space;

[0022] S250, using the gradient features as a guiding signal, adopting a gradient-guided attack strategy, adjusting the perturbation intensity in the inverse gradient direction, and outputting a preset target that approximates the attacker.

[0023] Optionally, step S30 includes the following steps:

[0024] S310, to build a basic dual-deep Q network architecture;

[0025] S320 initializes two neural networks, the main network and the target network. The main network selects the optimal action based on the current state for training, and the target network calculates the target Q value to stabilize the training process.

[0026] In each training round of S330, the agent obtains the current state from the environment and randomly selects actions to explore with a preset probability.

[0027] S340, after the action is performed, the agent receives an immediate reward and enters the next state, and stores this set of experience samples of state-action-reward-next state into the experience replay buffer.

[0028] S350, a preset number of samples are randomly selected from the experience replay buffer for learning. The main network parameters are updated by minimizing the mean square error between the predicted Q value and the target Q value. Every preset number of steps, the parameters of the main network are copied to the target network. DDQN strategy model.

[0029] S360, obtain the DDQN strategy model.

[0030] Optionally, after step S30, the following steps are also included:

[0031] S370 constructs a hybrid action space containing four attack strategies;

[0032] S380 constructs a multi-objective coupled composite reward mechanism.

[0033] Optionally, the four attack strategies include one-dimensional attack, multi-dimensional gradient attack, adaptive attack, and random reset attack.

[0034] Optionally, step S40 includes the following steps:

[0035] S410 initializes a zero-based perturbation vector based on the cycle data of each battery to be attacked, and calculates the initial state vector based on the current battery data.

[0036] S420, enter the iterative optimization loop, and in each iteration, the agent selects the action with the largest Q value from the Q network according to the current state, executes the action, and updates the perturbation vector accordingly.

[0037] S430, the updated perturbation vector is superimposed on the original battery data to obtain the attacked data;

[0038] S440, Input the attacked data into the health status estimation model, calculate the estimated health status value after the attack, and compare it with the preset target health status value to obtain the estimation deviation;

[0039] S450, calculate the residual value of the current attack vector, check whether the voltage and current after the disturbance exceed the preset safety range, and update the state vector at the next moment;

[0040] S460: When any stopping condition is met, stop the iterative optimization loop and obtain the currently accumulated perturbation vector.

[0041] Optionally, the initial state vector includes perturbation statistics, estimation bias, residual value, degree of physical constraint violation, and gradient information.

[0042] Optionally, the stopping conditions include: the health status estimation deviation is less than a preset deviation threshold; the residual is less than a preset residual threshold; and the maximum number of iterations is reached.

[0043] Furthermore, to achieve the above objectives, embodiments of this application also provide an adaptive battery health status control system based on reinforcement learning, the device comprising:

[0044] The model and system construction module is used to build the basic model and the residual detection defense system.

[0045] The model and space construction module is used to construct the attack environment module and the composite state space containing multi-dimensional feedback information.

[0046] The training module is used to train the agent based on the dual deep Q network and build the DDQN policy model.

[0047] The attack vector generation module is used to generate attack vectors targeting real battery data based on the DDQN strategy model.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] The adaptive battery health state manipulation method based on reinforcement learning provided in this application modeles adversarial testing as a Markov decision process, enabling the agent to provide closed-loop feedback based on the real-time perceived "health state estimation bias". This mechanism ensures that the test sequence can adaptively adjust the injection intensity at different stages of the battery's entire life cycle (such as the rapid capacity decay period and the plateau period), thereby achieving a long-term and stable manipulation effect on the health state estimation results across time scales.

[0050] This application trains an agent using a dual-deep Q-network algorithm, enabling the agent to autonomously learn to find the optimal perturbation path within a "narrow gap" without touching a preset residual alarm threshold. This end-to-end optimization method solves the problem of conflict between attack effectiveness and stealth in existing technologies. The generated test samples are highly similar to normal data in statistical characteristics, making them extremely stealthy.

[0051] This application innovatively introduces gradient information of health state deviation relative to voltage and current perturbations into the state space. This technique is equivalent to providing an "advancement map" for the agent, enabling it to accurately identify which sensor data points have the greatest impact on the estimation results, thereby achieving the preset attack target with minimal data modification and significantly improving the accuracy of generating adversarial use cases. Attached Figure Description

[0052] Figure 1 A flowchart of the reinforcement learning-based adaptive battery health state control method provided in the embodiments of this application;

[0053] Figure 2 This is a structural block diagram of an adaptive battery health status control system based on reinforcement learning, provided in an embodiment of this application.

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the application. Rather, these embodiments are provided to make the disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.

[0056] To address the aforementioned technical problems, embodiments of this application provide an adaptive battery health state control method based on reinforcement learning, such as... Figure 1 As shown, the method may include the following steps:

[0057] S10, Construct a basic model and residual detection defense system.

[0058] In an exemplary embodiment, step S10 includes the following steps:

[0059] S110 constructs various data-driven models, including multilayer perceptron, gated recurrent unit, long short-term memory network, and a coupled model of convolutional neural network and long short-term memory network.

[0060] S120, Construct a voltage and current prediction model based on time-series correlation learning;

[0061] S130, Based on the voltage and current prediction model, using the current voltage and current time series as input, predict the expected observation value at the next moment in real time;

[0062] S140, compare the expected observation with the actual sampled value of the sensor, and quantize the deviation between the expected observation and the sensor using Euclidean distance as a residual signal, as shown in the following formula:

[0063] r residual =||z pred -z meas ||2

[0064] In the formula, r residual The Euclidean norm of the residual signal; z pred These are the voltage and current values ​​predicted by the LSTM model at the next time step; z meas These are the actual voltage and current values ​​collected by the sensor; ||||2 represents the Euclidean norm (i.e., the square root of the sum of the squares of the vector elements).

[0065] In the above embodiments, all data-driven models employ the same input data construction method: First, feature extraction is performed, extracting five features from each charge-discharge cycle, including constant current charging time, charging current standard deviation, measured voltage, discharge time, and discharge capacity. Second, a sliding window is constructed, using a fixed-length sliding window to build time-series training samples, with the window size set to 10 cycles. Normalization is then performed, globally normalizing all features to ensure consistent data scaling. Finally, the dataset is partitioned: the dataset is divided into training, validation, and test sets in a 7:2:1 ratio.

[0066] In step S120, the voltage and current prediction model based on temporal correlation learning adopts a structure combining a Long Short-Term Memory (LSTM) network with a fully connected layer (FC). The core network utilizes LSTM layers to capture the long-term dependencies and temporal correlations of voltage and current data over time. The output layer outputs the prediction results through a fully connected layer. The construction and training process of this model is described below:

[0067] First, input data is prepared by using the voltage and current time series during normal battery operation as the model's input. Next, time-series correlation learning is performed, where the model learns the dynamic patterns of voltage and current changes over time (i.e., time-series correlation) through training. Then, target prediction is performed, using data from historical time steps to accurately predict the voltage and current values ​​for the next time step. Finally, residuals are calculated. Although this is the model's usage phase, the ultimate purpose of building the model is to calculate residuals. The residual is defined as the Euclidean distance between the model's "predicted value" and the sensor's "actual measured value."

[0068] In short, this is an LSTM-based time series prediction model that learns the battery voltage / current variation patterns under normal operating conditions to predict the values ​​at the next moment, thereby providing a benchmark for subsequent anomaly detection.

[0069] In this embodiment, multiple data-driven models learn the nonlinear mapping relationship between physical quantities such as voltage and current in battery cycle data and the battery health state, forming diversified "attack targets" to verify the generalization performance of the attack algorithm under different architectures. Simultaneously, to simulate common anomaly detection mechanisms in cloud-based battery management systems, this embodiment specifically constructs a voltage and current prediction model based on time-series correlation learning as the core of residual detection. This prediction model is built based on historical battery observation data under normal operating conditions. It captures the evolution patterns and inherent correlations of voltage and current signals over time using deep learning algorithms. Specifically, it learns the evolution patterns and inherent temporal correlations of voltage and current by training a long short-term memory network, thereby accurately predicting the value at the next time step. Finally, the deviation is quantified by calculating the Euclidean distance between the predicted value and the actual measured value. The historical battery observation data includes the voltage and current time series during normal operation, which serve as the model input to reflect the dynamic characteristics of the battery system.

[0070] In practical operation, the voltage and current prediction model uses the current voltage and current time series as input to predict the expected observation values ​​at the next moment in real time. This predicted value is then compared with the actual sampled values ​​from the sensors, and the deviation between the two is quantified into a residual signal using Euclidean distance. This design not only establishes an evaluation benchmark simulating a real defense system but also provides a concealment constraint for the attack vector by incorporating the residual signal into the reward function generated by subsequent reinforcement learning attacks. Ultimately, this two-layer model system ensures that while accurately inducing deviations in the State of Health (SOH) estimation, the attack vector can control voltage and current disturbances within the normal fluctuation range of the prediction model, thereby achieving effective bypassing of conventional residual detection systems and high-fidelity simulation of adversarial environments.

[0071] S20, construct the attack environment module and a composite state space containing multi-dimensional feedback information.

[0072] In an exemplary embodiment, step S20 includes the following steps:

[0073] S210, Module for building the attack environment;

[0074] S220, which includes a benchmark evaluation module, a DDQN (Double Deep Q-Network) agent training module, and an attack vector generation and execution module;

[0075] S230 defines a composite state space containing multidimensional feedback information;

[0076] S240, Introduce gradient features into the state space;

[0077] S250, using the gradient features as a guiding signal, adopting a gradient-guided attack strategy, adjusting the perturbation intensity in the inverse gradient direction, and outputting a preset target that approximates the attacker.

[0078] Specifically, the construction method of each module in this embodiment is as follows:

[0079] Attack Environment Module:

[0080] Construction method: It is built as an interactive interface, which needs to integrate the pre-trained SOH estimation model, residual prediction model, raw battery cycle data and preset target SOH curve.

[0081] Functionality: This module requires logic to receive the attack vector and perturb the original data, then calculate the residual after the attack, estimate the SOH value and evaluate its deviation from the target, while detecting whether the physical constraints of voltage and current are violated.

[0082] Evaluation benchmark module:

[0083] Construction method: Constructed by training multiple data-driven models.

[0084] Specific steps: Build and train various models for SOH estimation (such as MLP, GRU, LSTM, CNN-LSTM), as well as voltage and current prediction models for residual detection, as the basic "physical laws" of the attack environment and defense simulation.

[0085] DDQN Agent Training Module:

[0086] Construction method: The training logic of the agent is constructed based on the DoubleQN algorithm.

[0087] Specific steps: Build a neural network architecture that includes a main network and a target network, introduce an experience replay mechanism, and update the network parameters through repeated interaction (trial and error) with the above-mentioned "attack environment" to learn the optimal attack strategy.

[0088] Attack vector generation and execution module:

[0089] Construction method: Iterative logic for generating attack sequences is implemented based on a pre-trained Agent.

[0090] Specific steps: Construct an iterative loop, allowing the agent to select the optimal action based on the current state, gradually optimizing the perturbation vector until the SOH deviation is less than the threshold or the maximum number of iterations is reached, and finally apply the generated perturbation to the test data.

[0091] In step S230, the composite state space is defined by constructing a state vector containing multidimensional feedback information. This state vector contains the following four types of key information: perturbation statistical characteristics (mean voltage perturbation, mean current perturbation, standard deviation of voltage perturbation, standard deviation of current perturbation); attack effect feedback (the deviation between the estimated SOH value after the attack and the preset target value); stealth and physical constraint feedback (the residual value generated by the current attack vector, used to measure the detected risk and the degree of violation of physical constraints, and record the maximum value of voltage or current exceeding the safe range); and optimization guidance information (the gradient of SOH deviation relative to voltage and current perturbations, used to guide the agent to optimize the attack strategy in the direction of reducing deviation).

[0092] The above embodiments construct a highly integrated attack environment module, serving as the core interface for dynamic interaction between the deep reinforcement learning agent and the underlying battery data stream. Its functions encompass key aspects such as injecting attack vectors into the original voltage and current sequences, maintaining the physical coherence of battery data, and providing real-time feedback on the environment state. In addition to the attack environment module, this modeling framework also constructs an evaluation benchmark module for training the basic model, a DDQN agent training module responsible for policy evolution, and an attack vector generation and execution module for final practical verification. Through close collaboration among these modules, closed-loop management from data perturbation to effect feedback is achieved. To endow the control process with a high degree of adaptability, the scheme defines a composite state space containing multi-dimensional feedback information. This space integrates the statistical characteristics of the attack vector (mean and standard deviation of voltage and current perturbations), the real-time deviation between the SOH estimate after the attack and the preset target trajectory (used to measure attack accuracy), the magnitude of the residual generated in the current step (used to monitor concealment indicators), and the degree to which the perturbation data violates the physical boundaries of battery safe operation, thus achieving comprehensive perception of complex attack environments. The state space incorporates sensitivity information of the estimation bias relative to the input signal, i.e., gradient features. This is specifically achieved by calculating the gradient direction and magnitude of the SOH estimation bias with respect to voltage and current perturbations, enabling the agent to identify which data points are vulnerable to attack. By using this gradient information as a guiding signal, the agent can adopt a gradient-guided attack strategy, precisely adjusting the perturbation intensity in the inverse gradient direction. This allows the agent to induce the model output to approximate the attacker's preset target with minimal modification cost, providing a detailed and scientific decision-making basis for the automated evolution of attack strategies in dynamic adversarial environments.

[0093] S30, based on a dual-deep Q-network, trains the agent to construct a DDQN policy model.

[0094] In an exemplary embodiment, step S30 includes the following steps:

[0095] S310, to build a basic dual-deep Q network architecture;

[0096] S320 initializes two neural networks, the main network and the target network. The main network selects the optimal action based on the current state for training, and the target network calculates the target Q value to stabilize the training process.

[0097] The calculation of the target Q value is broken down into two steps (separation of action selection and value assessment), and the specific calculation formula is as follows:

[0098] Action selection (based on the main network): utilizing the main network Q main Choose the optimal action in the next state s'. :

[0099]

[0100] Target Q-value calculation (based on the target network): using the target network Q... target Calculate the value of this action to obtain the final target Q value TD. target :

[0101]

[0102] Where: r is the immediate reward; γ is the discount factor; s' is the next state;

[0103] In each training round of S330, the agent obtains the current state from the environment and randomly selects actions to explore with a preset probability.

[0104] S340, after the action is performed, the agent receives an immediate reward and enters the next state, and stores this set of experience samples of state-action-reward-next state into the experience replay buffer.

[0105] S350, a preset number of samples are randomly selected from the experience replay buffer for learning. The main network parameters are updated by minimizing the mean square error between the predicted Q value and the target Q value. Every preset number of steps, the parameters of the main network are copied to the target network. DDQN strategy model.

[0106] S360, obtain the DDQN strategy model.

[0107] Specifically, this embodiment uses a dual-deep Q-network architecture as the core training algorithm to drive the agent to autonomously learn the optimal attack strategy. The specific training process is as follows: First, initialize two neural networks, the main network and the target network. The main network is responsible for selecting the optimal action based on the current state, while the target network is used to calculate the target Q value to stabilize the training process. In each round of training, the agent obtains the current state from the environment and randomly selects an action for exploration with a preset probability (0.1), or selects the action with the largest current Q value for exploitation with a higher probability. After executing the action, the agent obtains an immediate reward and enters the next state. This set of "state-action-reward-next state" experience samples is stored in the experience replay buffer (capacity set to 10000). During training, a preset number of samples (32) are randomly selected from the buffer for learning. The parameters of the main network are updated by minimizing the mean square error between the predicted Q value and the target Q value. Every preset number of steps (set to 100 steps), the parameters of the main network are copied to the target network to ensure the stability of the training. After training, the DDQN policy model is obtained.

[0108] In an exemplary embodiment, after step S30, the following step is further included:

[0109] S370 constructs a hybrid action space containing four attack strategies, enabling agents to dynamically select the optimal attack method based on system sensitivity.

[0110] S380 constructs a multi-objective coupled composite reward mechanism.

[0111] Specifically, the four attack strategies include one-dimensional attack, multi-dimensional gradient attack, adaptive attack, and random reset attack.

[0112] The four attack strategies will be described in detail below:

[0113] The first type is a single-dimensional attack, which applies a fixed-amplitude perturbation to only one dimension of voltage or current at a time (voltage perturbation amplitude set to 0.01V, current perturbation amplitude set to 0.02A). This attack method is simple, direct, and easy to implement, suitable for the initial attack phase when the system's sensitivity is low. The second type is a multi-dimensional gradient attack, where the agent simultaneously calculates the gradient of the health state estimation bias relative to the voltage and current perturbations, and then applies perturbations in the opposite direction of the gradients (voltage perturbation strength coefficient is 0.10, current perturbation strength coefficient is 0.05). This attack method can utilize gradient information to quickly reduce the estimation bias, making the estimation result closer to the target value. The third type is adaptive... The attack involves the agent dynamically adjusting the perturbation intensity based on the current gradient magnitude—automatically reducing the perturbation amplitude when the gradient is large (high system sensitivity) and increasing the perturbation amplitude when the gradient is small (low system sensitivity). The reference perturbation coefficient is set to 0.1, and the sensitivity compensation coefficients for voltage and current are set to 0.05 and 0.02, respectively. The perturbation intensity adjustment factor is limited to a safe range (voltage 0.1-1.0, current 0.1-0.5) to ensure that the attack is both effective and not excessive. The fourth type is a random reset attack, where the agent randomly selects 20% of the perturbation positions in the current attack vector and forces them to zero, helping the agent escape local optima and enhancing the diversity of attack strategies.

[0114] Furthermore, this embodiment constructs a multi-target coupled composite reward mechanism to guide the agent to achieve a balance between attack effectiveness, stealth, and physical feasibility. This composite reward mechanism consists of three main parts:

[0115] The base reward r1 aims to minimize the SOH estimation bias.

[0116] Residual penalty term r2: designed to maintain the stealth of the attack.

[0117] Physical violation penalty item r3: designed to ensure that attacks comply with physical constraints.

[0118] The final total reward function is defined as: .

[0119] Specifically, the base reward r1 adopts a double-exponential structure: when the health state estimation deviation is small, a dominant reward is given (inversely proportional to the square of the deviation); when the deviation is large, an enhancement term (set to -0.1) ensures that the agent still receives a certain reward signal. The residual penalty term r2 is used to constrain the stealth of the attack. The residual generated by the current attack is multiplied by a penalty coefficient (set to 10) and then deducted from the reward. The larger the residual, the heavier the penalty, prompting the agent to generate attack vectors with smaller residuals to bypass detection. The physical constraint penalty term r3 is used to ensure the physical feasibility of the attack. When the voltage or current after the disturbance exceeds a preset safety range, a penalty is applied according to the degree of exceedance (penalty coefficient set to 5) to prevent the generation of abnormal data that does not conform to the battery's electrical laws. Through this multi-objective coupled reward design, the agent can automatically learn during training how to maximize the attack effect while satisfying stealth and physical constraints.

[0120] S40, Based on the DDQN strategy model, generate an attack vector targeting real battery data.

[0121] In an exemplary embodiment, step S40 includes the following steps:

[0122] S410 initializes a zero-based perturbation vector based on the cycle data of each battery to be attacked, and calculates the initial state vector based on the current battery data.

[0123] S420, enter the iterative optimization loop, and in each iteration, the agent selects the action with the largest Q value from the Q network according to the current state, and after executing the action, updates the perturbation vector accordingly (applying perturbation to a specific location of voltage or current according to the selected attack type).

[0124] S430, the updated perturbation vector is superimposed on the original battery data to obtain the attacked data;

[0125] S440, Input the attacked data into the health status estimation model, calculate the estimated health status value after the attack, and compare it with the preset target health status value to obtain the estimation deviation;

[0126] S450, calculate the residual value of the current attack vector, check whether the voltage and current after the disturbance exceed the preset safety range (voltage range is 2.0V-3.6V, current range is set according to the battery rated capacity), and update the state vector at the next moment;

[0127] S460: When any stopping condition is met, the iterative optimization loop stops, and the currently accumulated perturbation vector is obtained, which is the final attack vector for the battery loop. During the actual attack execution, this attack vector is injected into the original data all at once, enabling precise control over the estimation of the current loop's health status.

[0128] In an exemplary embodiment, the initial state vector includes perturbation statistics, estimation bias, residual value, degree of physical constraint violation, and gradient information.

[0129] In an exemplary embodiment, the stopping conditions include: the health status estimation deviation is less than a preset deviation threshold (set to 0.5%, indicating that the attack has successfully made the estimated value track the target trajectory); the residual is less than a preset residual threshold (set to 0.1, indicating that the attack has reached the stealth requirement); and the maximum number of iterations is reached (set to 1000 times).

[0130] Throughout the battery's entire lifespan, the iterative generation process described above is repeated for each cycle. Since the agent has been trained using DDQN to adaptively adjust its attack strategy based on real-time feedback, the generated attack vector can dynamically adjust to the nonlinear changes in the battery's aging state—automatically employing smaller perturbations in the early stages of aging (when capacity decay is slow) and appropriately increasing the perturbation strength in the later stages (when capacity decays rapidly), thus achieving stable tracking of the target trajectory throughout the entire lifespan. This adaptive mechanism ensures the long-term effectiveness and stealth of the attack, overcoming the problem that traditional static attack methods cannot cope with the nonlinear aging characteristics of batteries.

[0131] To demonstrate the effectiveness of the present invention, the following experiment was conducted:

[0132] Step 1: Dataset Preparation

[0133] MIT Dataset: Cyclic data from 124 commercial lithium iron phosphate / graphite batteries were used, and 47 of them were selected for experiments.

[0134] Self-test dataset: Cyclic aging tests were conducted using 9 NMC18650 cells to verify the generalization across cell materials.

[0135] Step 2: Training the target (SOH model):

[0136] Four mainstream deep learning SOH estimation models were trained: MLP (Multilayer Perceptron), GRU (Gated Recurrent Unit), LSTM (Long Short-Term Memory Network), and CNN-LSTM (Convolutional-Long Short-Term Memory Coupled Network).

[0137] Step 3: Compare with the benchmark:

[0138] The proposed method is compared with PSO (Particle Swarm Optimization) attacks and random perturbation attacks.

[0139] Step 4: Attack Execution

[0140] Define a predefined SOH decay trajectory (exponential decay curve) for the attacker. Using a trained agent and a comparison method, generate an attack vector (i.e., a small perturbation to the voltage and current data) for the test battery.

[0141] Step 5, Evaluation Indicators:

[0142] The goodness of fit between the post-attack SOH estimate and the target trajectory is evaluated using R² (coefficient of determination), MAE (mean absolute error), and RMSE (root mean square error). The stealth of the attack is assessed by calculating the residual.

[0143] The experimental results will be explained below:

[0144] 1. Attack Effectiveness (High Fit): In all 16 test scenarios of the MIT dataset, the SOH estimation curves after the proposed attack closely match the preset target trajectory, with R² values ​​approaching or exceeding 0.95. In contrast, the mean R² value of the PSO attack is only -1.366, and the mean R² value of the random attack is -6.747, indicating that these two methods cannot effectively manipulate the SOH estimation value. On the self-test dataset (different battery types), the R² value of the proposed method remains stable above 0.95, demonstrating extremely strong generalization ability.

[0145] 2. Extremely low attack error: The proposed method's MAE and RMSE are significantly lower than the comparative methods. On the self-test dataset, the average MAE of the proposed method is only 0.43, while the MAE of PSO and random attacks are generally above 10. The RMSE of the proposed method is between 0.46 and 0.69, while the RMSE of PSO and random attacks are both above 10.

[0146] 3. Stealth (low and stable residuals): The residual amplitude of the attack vectors generated by the proposed method is mostly below 2 (MIT dataset), and remains highly stable throughout the entire attack period. This low and stable residual characteristic makes the attack difficult to detect by residual-based anomaly detection (BDD) mechanisms, achieving a "stealthy" attack.

[0147] Therefore, the experimental results show that the attack vector generated by this scheme can achieve a good fit R² of over 0.95 between the health status estimate and the target trajectory, while the residual remains at a low level, effectively bypassing the residual-based anomaly detection mechanism.

[0148] The reinforcement learning-based adaptive battery health state manipulation method provided in the above embodiments of this application models adversarial testing as a Markov decision process, enabling the agent to perform closed-loop feedback based on the real-time perceived "health state estimation bias". This mechanism ensures that the test sequence can adaptively adjust the injection intensity at different stages of the battery's entire life cycle (such as the rapid capacity decay period and the plateau period), thereby achieving a long-term and stable manipulation effect on the health state estimation results across time scales.

[0149] This application trains an agent using a dual-deep Q-network algorithm, enabling the agent to autonomously learn to find the optimal perturbation path within a "narrow gap" without touching a preset residual alarm threshold. This end-to-end optimization method solves the problem of conflict between attack effectiveness and stealth in existing technologies. The generated test samples are highly similar to normal data in statistical characteristics, making them extremely stealthy.

[0150] This application innovatively introduces gradient information of health state deviation relative to voltage and current perturbations into the state space. This technique is equivalent to providing an "advancement map" for the agent, enabling it to accurately identify which sensor data points have the greatest impact on the estimation results, thereby achieving the preset attack target with minimal data modification and significantly improving the accuracy of generating adversarial use cases.

[0151] Based on the above embodiments, refer to Figure 2 Another embodiment of this application also provides an adaptive battery health status control system based on reinforcement learning. The adaptive battery health status control system 200 based on reinforcement learning may include the following modules:

[0152] Model and system construction module 210 is used to build the basic model and residual detection defense system;

[0153] Model and space construction module 220 is used to construct the attack environment module and a composite state space containing multi-dimensional feedback information;

[0154] Training module 230 is used to train the agent based on a dual deep Q network and build a DDQN policy model;

[0155] The attack vector generation module 240 is used to generate attack vectors targeting real battery data based on the DDQN strategy model.

[0156] In the description of this application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0157] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0161] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

[0163] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

Claims

1. An adaptive battery health state control method based on reinforcement learning, characterized in that, The method includes the following steps: S10, Constructing a basic model and residual detection defense system; S20, construct the attack environment module and a composite state space containing multi-dimensional feedback information; S30, based on a dual-deep Q-network, trains the agent to construct a DDQN policy model; S40, Based on the DDQN strategy model, generate an attack vector targeting real battery data.

2. The adaptive battery health state control method based on reinforcement learning according to claim 1, characterized in that, Step S10 includes the following steps: S110 constructs various data-driven models, including multilayer perceptron, gated recurrent unit, long short-term memory network, and a coupled model of convolutional neural network and long short-term memory network. S120, Construct a voltage and current prediction model based on time-series correlation learning; S130, Based on the voltage and current prediction model, using the current voltage and current time series as input, predict the expected observation value at the next moment in real time; S140, compare the expected observation with the actual sampled value of the sensor, and use Euclidean distance to quantize the deviation between the expected observation and the sensor into a residual signal.

3. The adaptive battery health state control method based on reinforcement learning according to claim 1, characterized in that, Step S20 includes the following steps: S210, Module for building the attack environment; S220, which includes a benchmark evaluation module, a DDQN agent training module, and an attack vector generation and execution module; S230 defines a composite state space containing multidimensional feedback information; S240, Introduce gradient features into the state space; S250, using the gradient features as a guiding signal, adopting a gradient-guided attack strategy, adjusting the perturbation intensity in the inverse gradient direction, and outputting a preset target that approximates the attacker.

4. The adaptive battery health state control method based on reinforcement learning according to claim 2, characterized in that, Step S30 includes the following steps: S310, to build a basic dual-deep Q network architecture; S320 initializes two neural networks, the main network and the target network. The main network selects the optimal action based on the current state for training, and the target network calculates the target Q value to stabilize the training process. In each training round of S330, the agent obtains the current state from the environment and randomly selects actions to explore with a preset probability. S340, after the action is performed, the agent receives an immediate reward and enters the next state, and stores this set of experience samples of state-action-reward-next state into the experience replay buffer. S350, a preset number of samples are randomly selected from the experience replay buffer for learning. The main network parameters are updated by minimizing the mean square error between the predicted Q value and the target Q value. Every preset number of steps, the parameters of the main network are copied to the target network. DDQN strategy model. S360, obtain the DDQN strategy model.

5. The adaptive battery health state control method based on reinforcement learning according to claim 1, characterized in that, Following step S30, the following steps are also included: S370 constructs a hybrid action space containing four attack strategies; S380 constructs a multi-objective coupled composite reward mechanism.

6. The adaptive battery health state control method based on reinforcement learning according to claim 5, characterized in that, The four attack strategies include one-dimensional attack, multi-dimensional gradient attack, adaptive attack, and random reset attack.

7. The adaptive battery health state control method based on reinforcement learning according to claim 1, characterized in that, Step S40 includes the following steps: S410 initializes a zero-based perturbation vector based on the cycle data of each battery to be attacked, and calculates the initial state vector based on the current battery data. S420, enter the iterative optimization loop, and in each iteration, the agent selects the action with the largest Q value from the Q network according to the current state, executes the action, and updates the perturbation vector accordingly. S430, the updated perturbation vector is superimposed on the original battery data to obtain the attacked data; S440, Input the attacked data into the health status estimation model, calculate the estimated health status value after the attack, and compare it with the preset target health status value to obtain the estimation deviation; S450, calculate the residual value of the current attack vector, check whether the voltage and current after the disturbance exceed the preset safety range, and update the state vector at the next moment; S460: When any stopping condition is met, stop the iterative optimization loop and obtain the currently accumulated perturbation vector.

8. The adaptive battery health state control method based on reinforcement learning according to claim 7, characterized in that, The initial state vector includes perturbation statistics, estimation bias, residual value, degree of physical constraint violation, and gradient information.

9. The adaptive battery health state control method based on reinforcement learning according to claim 7, characterized in that, The stopping conditions include: the health status estimation deviation is less than a preset deviation threshold; the residual is less than a preset residual threshold; and the maximum number of iterations is reached.

10. An adaptive battery health status control system based on reinforcement learning, characterized in that, include: The model and system construction module is used to build the basic model and the residual detection defense system. The model and space construction module is used to construct the attack environment module and the composite state space containing multi-dimensional feedback information. The training module is used to train the agent based on the dual deep Q network and build the DDQN policy model. The attack vector generation module is used to generate attack vectors targeting real battery data based on the DDQN strategy model.

Images