A transfer reinforcement learning control method for energy storage battery safety warning

Abstract

The present invention relates to the technical field of adaptive control systems, and specifically to a transfer reinforcement learning control method for energy storage battery safety early warning, comprising: the system initializes and adaptively adjusts the controller by real-time monitoring of the dynamic state parameters of the operating conditions and combining the control knowledge formed by historical operating experience. The controller iteratively optimizes and outputs safety control instructions through a reinforcement learning mechanism based on the current state parameters and a performance evaluation function with safe operation as the goal. Before execution, the safety control instructions are verified by safety logic, and the actual response is used as a feedback signal for closed-loop update and adaptive improvement of control strategies and experience knowledge, thereby improving the predictability and adaptive response capabilities of the entire control system in terms of energy storage battery safety protection.

Description

Technical Field

[0001] The present invention relates to the field of adaptive control technology, and in particular to a transfer reinforcement learning control method for energy storage battery safety early warning. Background Art

[0002] In the field of complex industrial process control, particularly for energy storage batteries with strong time-varying, nonlinear, and multi-operating-condition characteristics, building control systems with parameter self-tuning and model adaptation capabilities is a core technical challenge for improving system robustness and autonomy. This precise and self-optimizing control is crucial for ensuring safe and efficient system operation.

[0003] To effectively control these complex dynamic systems, existing technologies often employ architectures such as model reference adaptive control (MRAC) or self-tuning regulators (STRs). These approaches achieve adaptive control by online identification of system dynamics and adjusting controller parameters accordingly. For example, they adjust the gains of PID controllers by online updating of process model parameters, or in MRAC, adjust the feedforward and feedback control laws to track a reference model.

[0004] However, these traditional adaptive control models still face many challenges in practical applications. On the one hand, control models that rely on classical parameter adjustment methods such as gradient descent and least squares identification are prone to model mismatch when the state changes suddenly or encounters unmodeled dynamics, which in turn affects control performance and may even require manual intervention to reset the identification parameters. On the other hand, when the operating conditions of the industrial process change, these adaptive control models often need to re-collect large amounts of training data or perform long-term online identification to adapt to the new operating conditions, lacking the ability to quickly migrate control strategies across operating conditions. In addition, although some adaptive control models based on reinforcement learning have shown potential for online learning, they generally lack effective representation, storage, and migration mechanisms for historical control knowledge, resulting in low model learning efficiency when faced with new tasks or environmental changes. This makes it difficult to meet the response speed and decision-making accuracy requirements of real-time safety warnings for objects with multi-timescale dynamic characteristics.

[0005] To this end, a transfer reinforcement learning control method for energy storage battery safety early warning is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide a transfer reinforcement learning control method for energy storage battery safety early warning. By acquiring and preprocessing multi-source dynamic state parameters, a current state representation containing abnormal control parameters is constructed. Based on the current state representation, a dynamic state vector is generated by combining time series feature modeling, spatial correlation modeling, and an abnormal feature enhancement mechanism. The control knowledge base is called to perform similarity matching between the current control task and known tasks in the library and determine the transferable historical knowledge content based on a preset matching strategy. The controller is then initialized with the historical knowledge content and outputs safety control instructions. Multi-level safety logic verification is performed, and the control execution results are used as environmental feedback signals for controller strategy optimization and control knowledge base updates. The present invention can achieve adaptive and accurate early warning and control of energy storage battery safety risks.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A transfer reinforcement learning control method for energy storage battery safety early warning includes:

[0009] Monitor the operating state of the target control object, obtain and pre-process dynamic state data, and construct a current state representation for control decision-making, the current state representation including abnormal control parameters;

[0010] Based on the current state representation, a dynamic state vector is generated by combining a time series feature modeling method, a spatial association modeling method, and an abnormal feature enhancement mechanism;

[0011] Calling a control knowledge base containing multi-condition control strategy experience, performing similarity matching between the current control task and known tasks in the control knowledge base, and determining the historical knowledge content to be migrated based on a preset matching strategy;

[0012] Initializing a controller with the historical knowledge content, using the dynamic state vector as input state information, performing strategy iteration and optimization based on a performance evaluation function constructed by combining abnormal control parameters and operating state, and outputting safety control instructions;

[0013] Before executing the safety control instruction, a safety logic check of the multi-level control structure is performed, and the control execution result is used as an environmental feedback signal for controller strategy optimization and for updating the control knowledge base.

[0014] Furthermore, the process of preprocessing the dynamic state data includes: eliminating sudden change outliers in the voltage data based on the Laida criterion; using a long short-term memory network to predict and fill missing points in the temperature data; using a Daubechies wavelet function with a set number of layers to suppress high-frequency noise in the current signal; and dynamically correcting the low-frequency drift in the temperature data through a Kalman filter algorithm to generate cleaned dynamic state parameters; based on the cleaned dynamic state parameters, using an isolation forest algorithm to perform anomaly detection, generate anomaly confidence scores reflecting the degree of abnormality in the operating state, and combine with context labels to mark potential fault types to obtain the abnormal control parameters.

[0015] Furthermore, the process of generating a dynamic state vector includes:

[0016] A one-dimensional convolutional neural network is used to extract short-term fluctuation features from the current state representation, and a long short-term memory network is used to perform time series modeling on the short-term fluctuation features to generate a time feature vector;

[0017] A topological graph is constructed based on the battery pack cell connections of the target control object; node features of the topological graph include voltage and temperature of each cell, and edge weights of the topological graph are set based on impedance and thermal conductivity; and a graph neural network is used to aggregate the electrochemical and thermodynamic interactions between the cells to generate a spatial feature vector.

[0018] For a preselected feature subset derived from the current state representation, a variational autoencoder is used to identify abnormal states, and potential fault feature components corresponding to the abnormal states are amplified through an attention mechanism to generate an abnormality-enhanced feature vector;

[0019] The temporal feature vector, the spatial feature vector, and the anomaly enhancement feature vector are fused through a fully connected layer and a layer normalization technique is applied, and are weighted using anomaly confidence scores to generate the dynamic state vector.

[0020] Furthermore, the process of constructing the control knowledge base includes:

[0021] Constructing a knowledge graph based on a graph database as the control knowledge base, wherein the knowledge graph is used to associate the multi-condition control strategy experience including different battery chemistry systems, aging levels, and operating conditions;

[0022] The structure of the knowledge graph includes: task nodes, which are used to record the different battery chemical systems, aging degrees and operating parameters; model nodes, which are used to store historical pre-trained model parameters; experience nodes, which are used to store historical feature extractor configuration information and state-action-reward sequences; and edges, which are used to represent semantic relationships between nodes.

[0023] Furthermore, the similarity matching process includes:

[0024] Using a maximum mean difference method, the data distribution of the current control task and the data domain corresponding to each known task in the control knowledge base are compared to obtain a first similarity measurement value;

[0025] A second similarity metric is obtained by using a cosine similarity calculation method based on the task description embedding vector;

[0026] Performing weighted summation on the first similarity metric value and the second similarity metric value according to predetermined weights to calculate a comprehensive similarity score;

[0027] According to the comprehensive similarity score, a model configuration with the highest comprehensive similarity score is selected from the control knowledge base as the historical knowledge content to be migrated.

[0028] Furthermore, the controller adopts the DDPG algorithm, and the initialization process of the controller includes: selecting pre-trained network weights from the historical knowledge content, and loading the pre-trained network weights into corresponding parts of the actor network and the critic network of the controller.

[0029] Furthermore, the safety logic verification of the multi-level control structure includes:

[0030] Performing a validity check on whether each parameter value in the security control instruction is within a preset safe operating range;

[0031] Performing redundancy check on the current state representation and the state parameters expected to be achieved after executing the safety control instruction;

[0032] According to the currently assessed risk level and the corresponding operation mode, the execution priority of different types of security control instructions is set to form arbitration logic.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. This invention constructs a high-quality current state representation through real-time monitoring of the target control object's operating status, high-fidelity acquisition of dynamic state parameters from multiple sensors, and precise preprocessing for deep abnormal control parameter extraction. This not only overcomes the problems of traditional methods, which rely on macroscopic, single-parameter judgments for insufficient information dimensionality and the tendency of early, weak fault signatures to be overwhelmed by noise, but also combines time series feature modeling, spatial correlation modeling, and abnormal feature enhancement mechanisms to generate dynamic state vectors. This achieves a comprehensive, accurate, and multi-dimensional dynamic representation of the complex internal state of the energy storage battery, providing a reliable data foundation and state perception capabilities for subsequent intelligent analysis, precise early warning, and adaptive control decisions of the energy storage battery.

[0035] 2. This invention calls upon a control knowledge base that stores experience with multiple operating condition control strategies, performs a precise similarity match between the current control task and known historical tasks in the library, and then determines and migrates relevant historical knowledge content based on a preset matching strategy to initialize the controller. This not only accelerates the controller's learning convergence when faced with new battery types, different aging stages, or unknown operating conditions, but also reduces the need for large amounts of labeled data for target scenarios and training from scratch; it also improves the initial performance of the energy storage battery control strategy and the ultimate optimization level that can be achieved.

[0036] 3. The controller of the present invention continuously iterates and optimizes strategies based on the performance evaluation function constructed based on real-time abnormal control parameters and the current operating status. It can autonomously learn and output the most optimized safety control instructions, thus realizing the transformation of the intelligent safety management mode from passive response to active prevention. At the same time, a safety logic verification link of the multi-level control structure is set up to ensure the reliability of control decisions and the safety of execution. In addition, by using the actual control execution results as environmental feedback signals, it is not only used for the closed-loop continuous optimization of the controller's own strategy, but also for the dynamic update of the control knowledge base, thereby improving the operational safety level of the energy storage system, extending the battery life and optimizing the operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention provides a flow chart of a transfer reinforcement learning control method for energy storage battery safety early warning;

[0038] Figure 2 A schematic flow chart of a process for generating a dynamic state vector is provided for the present invention;

[0039] Figure 3 A flowchart of the process of similarity matching is provided for the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] See also Figures 1 to 3 The present invention provides a transfer reinforcement learning control method for energy storage battery safety warning, and the technical solution is as follows:

[0042] Example 1:

[0043] This embodiment proposes an innovative solution for the intelligent and safe operation and maintenance of energy storage battery systems, and elaborates on its specific implementation process under complex working conditions. It aims to achieve accurate online tracking of the multi-dimensional status of energy storage batteries through the collaborative work of edge computing nodes and cloud platforms, and to provide advanced and accurate warnings for early thermal runaway and other safety risks. At the same time, it has deep self-adaptation and continuous learning capabilities, providing preliminary control decision assistance for the safe operation of energy storage battery systems. Figure 1 As shown, a transfer reinforcement learning control method for energy storage battery safety warning includes:

[0044] Monitor the operating status of the target control object, obtain and preprocess the dynamic state parameters from multi-source sensors, and construct the current state representation including abnormal control parameters.

[0045] In this embodiment, the target control object is a battery pack consisting of 16 NMC lithium-ion battery cells, which operates under complex operating conditions such as high temperatures and high charge and discharge rates. Dynamic state parameters from multiple sources are acquired in real time through the battery management system (BMS) and external sensors (such as temperature sensors, gas sensors, and acoustic sensors). These parameters include, but are not limited to: battery cell voltage, current, and temperature; battery pack total voltage, total current, and charge and discharge rate; SOC (state of charge), SOH (state of health), and cycle count; acoustic signal characteristics reflecting microscopic changes in the battery, and specific gas concentrations.

[0046] Furthermore, to improve data quality, the process of preprocessing the dynamic state data includes:

[0047] Abrupt outliers in voltage data are removed based on the Laida criterion. Specifically, within a set time window, the mean and standard deviation of the voltage data are calculated. If a sampling point deviates from the mean by more than three standard deviations, it is considered an outlier. This value can be marked as invalid or replaced with an adjacent valid value.

[0048] Long Short-Term Memory (LSTM) networks are used to predict and fill missing points in temperature data (e.g., data loss due to temporary sensor failure or communication interruption). An LSTM model is trained based on historical temperature series, leveraging its ability to model time series trends. When missing data is detected, the missing value is predicted using the preceding valid data.

[0049] High-frequency noise in current signals is suppressed using a Daubechies wavelet function with a set number of layers. A suitable Daubechies wavelet basis (such as db4 or db5) and a suitable number of decomposition layers (e.g., 5) are selected for multiscale wavelet decomposition. Soft or hard thresholding is applied to the high-frequency coefficients, and the signal is reconstructed using an inverse wavelet transform to effectively suppress high-frequency noise.

[0050] The Kalman filter algorithm dynamically corrects low-frequency drift in temperature data. By constructing state and observation equations and leveraging the recursive estimation capabilities of the Kalman filter, the true temperature is estimated in real time based on observations affected by drift and noise, thereby correcting baseline offset. After purification, cleaned dynamic state parameters are generated.

[0051] Next, an anomaly detection algorithm is used based on the cleaned dynamic state parameters using the isolation forest algorithm. This algorithm generates an anomaly confidence score for each sample (i.e., a set of state parameters at a specific point in time) that reflects the degree of abnormality in the operating state. A higher score indicates a greater likelihood or severity of the anomaly. The anomaly is semantically annotated based on contextual features (such as voltage threshold violations, gas level violations, and charge / discharge rates), forming a contextual label for the potential fault. For example, a simultaneous abnormality in both voltage and current is labeled "suspected overcharge / overdischarge." The anomaly confidence score, combined with the contextual label indicating the potential fault type, is used to determine the abnormal control parameters.

[0052] These preprocessing methods effectively remove noise, outliers, missing values, and drift from the original dynamic state parameters, resulting in high-quality cleaned dynamic state parameters. Further anomaly detection using the isolation forest algorithm not only provides a quantitative indication of the degree of anomaly but also, combined with contextual information, provides a preliminary assessment of potential fault types, thereby forming clearly defined abnormal control parameters. This provides a solid data foundation for the subsequent precise construction of the dynamic state vector and effective controller decision-making, enhancing the safety and accuracy of the entire energy storage battery safety control.

[0053] Based on the current state representation, a dynamic state vector is generated by combining a time series feature modeling method, a spatial association modeling method, and an abnormal feature enhancement mechanism.

[0054] Further, if Figure 2 As shown in Figure 2, the process of generating a dynamic state vector includes:

[0055] First, the time series feature modeling method involves using a one-dimensional convolutional neural network (1D-CNN) to extract short-term fluctuation features from the current state representation. By setting different convolution kernel sizes and step sizes, the 1D-CNN can effectively capture local features such as rapid changes in battery state parameters over a short period of time, edge features, or specific waveform segments, which are important clues to early battery abnormalities. Then, a long short-term memory network (LSTM) is used to perform time series modeling on these short-term fluctuation features, extracting their dynamic evolution over time and generating a time feature vector.

[0056] Next, the spatial correlation modeling method includes: constructing a topological graph model based on the connection relationship between each battery cell in the target control object. In this model, the nodes represent each battery cell, and its node characteristics include real-time voltage, temperature, SOC and other state parameters; the edge weights are set according to the internal impedance and thermal conductivity between the cells to reflect their electrochemical and thermodynamic coupling relationship. Through the graph neural network (GNN), the topological graph is modeled to aggregate and capture the complex interactions between battery cells caused by factors such as structural differences, uneven heat distribution or inconsistent aging, and generate a spatial feature vector. For example, a graph neural network with two layers and 64 neurons in each layer can output a 128-dimensional spatial feature vector that fully describes the spatial correlation characteristics of the battery pack.

[0057] The abnormal feature enhancement mechanism then involves inputting a preselected subset of features derived from the current state representation (such as specific gas concentrations, acoustic signal features, etc.) into a pre-trained variational autoencoder (VAE) for anomaly detection. Through an encoding-reconstruction mechanism, the VAE has a small reconstruction error under normal conditions, but its error increases significantly under abnormal conditions. Based on the comparison of the reconstruction error with a set threshold (such as 0.15), the abnormal state can be identified, and the representation of the VAE latent space is used as the abnormality enhancement feature. An attention mechanism is further introduced to assign more attention weight to the aforementioned potential abnormal features, thereby amplifying the potential fault feature components corresponding to the abnormal state and forming an abnormality enhancement feature vector.

[0058] Finally, the temporal feature vector, spatial feature vector, and anomaly enhancement feature vector are fused through a fully connected layer and layer normalization. The anomaly confidence score is also introduced as a weighting factor to enhance the response to high-risk conditions. The fused output is the dynamic state vector, which comprehensively and dynamically reflects the current battery's overall health status, aging level, and potential safety risk level.

[0059] Temporal features characterize the dynamic evolution of the state, spatial features reflect the coupling relationship between battery cells, and the anomaly enhancement mechanism increases the system's sensitivity to early, subtle faults. The resulting dynamic state vector has high information density and strong representation capabilities, providing high-quality, low-dimensional, and easily processable state input for subsequent migration controller decisions. This improves early warning accuracy and control reliability, thereby enhancing the safety and precision of the entire energy storage battery safety control system.

[0060] A control knowledge base containing multi-condition control strategy experience is called, and similarity matching is performed between the current control task and the known tasks in the control knowledge base, and the historical knowledge content to be migrated is determined according to a preset matching strategy.

[0061] Furthermore, the process of constructing the control knowledge base includes:

[0062] Constructing a knowledge graph based on a graph database as the control knowledge base, wherein the knowledge graph is used to associate the multi-condition control strategy experience including different battery chemistry systems, aging levels, and operating conditions;

[0063] In this embodiment, a graph database (such as Neo4j) is used as the underlying data storage and management platform. A knowledge graph is constructed based on this database to describe the complex relationships between batteries, operating conditions, model parameters, and control strategies. The multi-condition control strategy experience in the knowledge graph is derived from historical operating data and simulation results of different battery types under various operating conditions, as well as strategies and models obtained through reinforcement learning or other optimization algorithms.

[0064] The structure of the knowledge graph includes:

[0065] Task nodes are used to record different battery chemistries, aging levels, and operating parameters. Each task node records the battery chemistry (e.g., NMC, LFP, NCA), aging level (e.g., new battery, SOH=90%, SOH=80%), and operating parameters (e.g., ambient temperature, charge / discharge rate, SOC range, number of cycles, and duration).

[0066] Model node, used to store historical pre-trained model parameters; this node includes the structural information and weight configuration of the deep neural network, the parameterized expression of the control strategy, or the hyperparameter settings of a specific feature extractor.

[0067] Edges are used to represent semantic relationships between nodes, including similarity connections between tasks, the attribution of models to tasks, and the association between experience and models. In particular, similarity edges between task nodes can be weighted to quantify the proximity of different tasks in terms of data distribution, operating parameters, and other aspects, thereby supporting rapid similarity retrieval and experience transfer.

[0068] By building a control knowledge base, heterogeneous, multi-condition control strategy experience can be uniformly expressed in a structured, searchable format. The graph-based knowledge graph not only clearly expresses the complex relationships between different battery characteristics, operating conditions, model parameters, and historical experience, but also facilitates dynamic updates and on-demand expansion. This provides a solid foundation for effective knowledge transfer and accelerates the controller's learning process, thereby improving the safety and accuracy of the entire energy storage battery safety control.

[0069] Further, if Figure 3 As shown in FIG, when faced with a new current control task, it is necessary to find the most similar task to the target task from multiple known tasks stored in the control knowledge base. The similarity matching process includes:

[0070] Using a Maximum Mean Discrepancy (MMD) method, the data distribution of the current control task and the data domain corresponding to each known task in the control knowledge base are compared to obtain a first similarity measurement value;

[0071] Specifically, a small amount of representative data from the initial stages of the target task is collected. Simultaneously, historical data or statistical features of each source task are extracted from the knowledge base. MMD, a nonparametric statistical method, can be used to determine whether two sets of samples originate from the same distribution. By calculating the MMD value between the target task and each source task data, the degree of similarity at the data level can be quantitatively assessed. Smaller MMD values ​​indicate closer distributions and higher similarity.

[0072] A second similarity metric is obtained by using a cosine similarity calculation method based on the task description embedding vector;

[0073] Specifically, each task can be represented as a high-dimensional task description embedding vector using its metadata (such as battery chemistry, aging, and operating parameter range). This embedding vector can be generated using a pre-trained task encoder model (such as a Transformer-based encoder) or rule-based feature engineering. By calculating the cosine similarity between the embedding vectors of the current control task and known tasks, their similarity at the task semantic level can be assessed. A cosine similarity closer to 1 indicates more similar task descriptions.

[0074] Performing weighted summation on the first similarity metric value and the second similarity metric value according to predetermined weights to calculate a comprehensive similarity score;

[0075] The weight setting can be adjusted according to the importance and reliability of different similarity dimensions in the current application scenario.

[0076] According to the calculated comprehensive similarity scores of the known tasks, a model configuration with the highest comprehensive similarity score is selected from the control knowledge base as the transferable historical knowledge content.

[0077] Among them, the model configuration includes pre-trained model weights, feature extractor structure parameters, control strategy parameterized representation, etc., and has the transferability to be directly used to initialize new task models.

[0078] This multi-dimensional similarity matching mechanism and matching decision-making process enables efficient screening and scientific migration of massive historical experience. This method establishes a systematic similarity assessment system across multiple dimensions, including data distribution and task characteristics, effectively improving the accuracy and adaptability of migration decisions, thereby enhancing the safety and precision of overall energy storage battery safety control.

[0079] The controller is initialized with the historical knowledge content, the dynamic state vector is used as input state information, and strategy iteration and optimization are performed according to a performance evaluation function constructed by combining abnormal control parameters and operating status to output safety control instructions.

[0080] The performance evaluation function is a reward function in the reinforcement learning framework. This reward function is designed to comprehensively evaluate the safety and effectiveness of the control instructions output by the controller under specific conditions, combining abnormal control parameters with the operating state. Specifically, the reward function comprehensively considers the following dimensions:

[0081] Whether commands can be accurately issued at the early stages of a fault; whether control measures, once implemented, effectively mitigate risks or prevent fault escalation; and whether irreversible damage to the battery caused by overly aggressive control is avoided. Furthermore, the weights of the various components of the reward function can be dynamically adjusted based on the risk level reflected by the abnormal control parameters and the current operating status (such as temperature and charge / discharge rate), thereby improving responsiveness and adaptability to actual operating conditions.

[0082] The controller continuously interacts with the environment, receives signals generated by the reward function, and performs policy iteration and optimization to maximize the long-term cumulative expected return. The output safety control instructions can cover multiple dimensions, including but not limited to adjusting charge and discharge current or power limits, adjusting cooling system intensity, performing battery balancing operations, and even triggering protective shutdowns in extreme cases.

[0083] Furthermore, the controller adopts DDPG algorithm;

[0084] The DDPG controller consists of an actor network and a critic network, both of which are implemented based on deep neural networks and have a preset network hierarchy (for example, 2 to 3 hidden layers, each containing 128 or 256 neurons, and activation functions such as ReLU or tanh).

[0085] The controller initialization process involves selecting pre-trained network weights compatible with the current task from historical knowledge and loading them into the corresponding layers of the actor and critic networks. These pre-trained weights typically originate from similar tasks completed in the past (e.g., deep reinforcement learning models trained on similar battery types under similar operating conditions). If a model trained using the DQN algorithm exists in the historical tasks and its structure is compatible with a specific network layer of the current DDPG controller (e.g., a front-layer feature extraction module or an intermediate value assessment layer), the pre-trained weights of that model can be directly extracted for initialization.

[0086] The initialization process specifically involves loading these selected pre-trained network weights into the corresponding parts of the actor network and critic network of the DDPG controller. This can significantly accelerate the learning convergence speed of the new controller on the target task and may help it achieve better performance.

[0087] By defining the controller so that it can effectively handle the complex continuous control problems of energy storage battery systems, and by initializing the controller by selecting and loading pre-trained network weights from historical knowledge content, effective knowledge transfer is achieved. This initialization method gives the controller a more favorable learning starting point, avoiding the inefficiency and instability of learning from scratch. This speeds up the controller's learning and adaptation to new operating conditions or new battery types, reduces its reliance on large amounts of new interaction data, and helps the controller learn more robust and higher-performance safety control strategies.

[0088] Before executing the safety control instruction, a safety logic check of the multi-level control structure is performed, and the control execution result is used as an environmental feedback signal for controller strategy optimization and for updating the control knowledge base.

[0089] Safety control instructions can only be executed after passing safety logic verification. The control execution results generated during execution (for example, the actual current adjustment value, temperature changes, and battery status parameter responses reported by the BMS) are collected and formed into an environmental feedback signal, which the controller uses to adjust the control strategy in subsequent strategy iterations. Furthermore, these control execution results can be used as effective control experience from actual operation and updated in the control knowledge base for future similar tasks, enabling experience transfer and strategy accumulation.

[0090] Furthermore, the safety logic verification of the multi-level control structure includes:

[0091] Perform a validity check on whether the parameter values ​​in the safety control instructions are within the preset safe operating range. Perform a validity check on the parameter values ​​in the safety control instructions to verify whether they are within the preset safe operating range. For example, after the controller outputs an instruction containing control parameters such as charging current, discharge power limit, cooling system level, etc., the system will compare these values ​​one by one to see if they meet the safety thresholds set for the current battery type, operating status (such as SOC, SOH and temperature) and current mode. For example, for a specific model of NMC battery, under the current temperature conditions, the maximum allowable charging current may be limited to no more than 1.5C, and exceeding this value is considered illegal.

[0092] Perform a redundancy check on the current state representation and the state parameters expected to be achieved after executing the safety control instruction. Compare the current state representation with the target state parameters expected to be achieved after the instruction is executed to identify potential unreasonable control. For example, if the current SOC is 98%, but the control instruction still requires a higher current to continue charging, the instruction presents a logical conflict or potential risk. Redundancy checking can be performed based on a simplified battery model or a rule-based prediction engine to identify instruction logic that is inconsistent with the actual system state.

[0093] According to the currently assessed risk level and the corresponding operation mode, the execution priority of different types of security control instructions is set to form arbitration logic.

[0094] In complex scenarios, the controller may output multiple sub-commands, or there may be control requests from other systems (such as the basic protection logic of the BMS and the dispatch commands of the EMS). In this case, arbitration is required based on the risk level (such as "warning level", "danger level" or "emergency level") and the current operating mode (such as fast charging, normal discharge, and fault emergency handling). For example, when an "emergency level risk" such as the initial signs of thermal runaway is identified, high-priority commands such as "emergency shutdown" or "activation of fire suppression mechanism" will be executed first, and other regular operations will be interrupted or delayed; when the risk is low but the temperature continues to rise, the "enhanced cooling" command may be given higher execution priority than "maintaining current power".

[0095] A multi-level verification mechanism adds a critical safety buffer between the controller's output control instructions and their actual effect on the energy storage battery system. Legality verification ensures that control parameters do not exceed physical or strategic boundaries; redundancy verification improves the match between instructions and system status, preventing execution failures or additional risks; and priority arbitration ensures that the most protective control strategies are prioritized at critical moments. This enhances the safety, reliability, and robustness of control instruction execution, avoids secondary risks or safety incidents caused by improper control, and thus improves the stability and reliability of the energy storage battery.

[0096] By real-time monitoring and meticulously preprocessing multi-source dynamic state parameters, this system constructs input information including key abnormal control parameters. Combined with multidimensional modeling techniques, it generates an accurate dynamic state vector, enabling a comprehensive, real-time assessment of the battery's internal state, improving early fault detection and early warning accuracy. Secondly, leveraging a control knowledge base encompassing multi-operating-condition experience, similarity matching and pre-set strategies identify transferable historical knowledge to initialize the controller, enhancing the system's adaptability and learning efficiency for different battery types, aging stages, and variable operating conditions. The controller iterates and optimizes its strategy based on a performance evaluation function constructed based on abnormal control parameters and the current operating state, learning and executing the optimal safety control instructions and enabling intelligent intervention for potential safety risks. Furthermore, multi-level safety logic verification prior to executing safety control instructions ensures operational safety and reliability. Control execution results are used as environmental feedback to continuously optimize the controller strategy and dynamically update the control knowledge base, forming an effective closed-loop learning and evolutionary mechanism that continuously improves the control performance of energy storage batteries.

[0097] Example 2:

[0098] Based on Example 1, this example further illustrates and demonstrates the actual performance in a simulated application environment through specific data testing and experimental results, quantitatively evaluating the accuracy and timeliness of the method in early safety risk identification, the contribution of the transfer learning mechanism to improving the model's adaptability and learning efficiency to different working conditions and battery states, and the effectiveness of the controller in making adaptive safety control decisions under dynamically changing conditions. A transfer reinforcement learning control method for energy storage battery safety early warning includes:

[0099] Monitor the operating status of the target control object, obtain and preprocess dynamic state parameters from multiple source sensors, and construct a current state representation including abnormal control parameters;

[0100] Based on the current state representation, a dynamic state vector is generated by combining a time series feature modeling method, a spatial association modeling method, and an abnormal feature enhancement mechanism;

[0101] Calling a control knowledge base containing multi-condition control strategy experience, performing similarity matching between the current control task and known tasks in the control knowledge base, and determining the historical knowledge content to be migrated based on a preset matching strategy;

[0102] Initializing a controller with the historical knowledge content, using the dynamic state vector as input state information, performing strategy iteration and optimization based on a performance evaluation function constructed by combining abnormal control parameters and operating state, and outputting safety control instructions;

[0103] Before executing the safety control instruction, a safety logic check of the multi-level control structure is performed, and the control execution result is used as an environmental feedback signal for controller strategy optimization and for updating the control knowledge base.

[0104] In order to objectively evaluate the effect of the present invention, two representative baseline methods were selected for comparison:

[0105] Baseline Method 1: Traditional threshold warning method, which relies on setting fixed upper and lower thresholds for key battery operating parameters (such as voltage, current, and temperature). When the monitored parameter values ​​exceed these preset thresholds, the system will trigger an alarm.

[0106] Baseline Method 2: Standard reinforcement learning, which uses a similar controller architecture to the proposed method but does not utilize knowledge transfer. This means that when faced with new battery types, aging states, or untrained operating conditions, learning and policy optimization must be performed from scratch or based on only a small amount of random initialization.

[0107] In order to examine the optimal performance level of the present invention under ideal or relatively familiar environments, various key indicators are calculated, including: warning accuracy (F1 score), which is statistically calculated based on a test data set containing real fault labels, comparing the system warning output with the actual situation; warning lead time is the average time difference between the time when the system issues the warning and the time when the fault develops to the critical failure point in the fault case where the warning is correct; control effectiveness is mainly evaluated in a controlled experimental environment by observing and quantifying the success probability of the output safety control instructions in mitigating simulated early faults; for learning methods, its learning efficiency is measured by the number of training iterations or samples required to achieve the predetermined performance target or model convergence state; and computational latency is actually measured on the target hardware platform, which is the average time required for the system to receive new data and generate a decision output.

[0108] As shown in Table 1, under known operating conditions, this method, through knowledge transfer and dynamic state representation, can effectively capture complex failure modes with superior accuracy compared to conventional reinforcement learning methods. Furthermore, by leveraging an anomaly enhancement mechanism, this method can detect failures earlier, also with superior accuracy compared to other baseline methods. Furthermore, the application of transferred knowledge reduces the number of training steps, thereby improving learning efficiency and, consequently, enhancing the stability and reliability of the energy storage battery. N / A indicates that no such data was available.

[0109] Table 1 Comparison of known working conditions

[0110]

[0111] Table 2 Comparison of new working conditions

[0112]

[0113] To validate the effectiveness of the transfer learning mechanism of our invention, we placed the original NMC battery in a low-temperature environment where it had not yet fully learned during training and monitored its early anomalies. As shown in Table 2, under these new operating conditions, the accuracy of other baseline methods declined due to starting from scratch or not adapting to the control environment. However, our invention, through transfer and online optimization, maintained a high accuracy, thereby improving the stability and reliability of the energy storage battery.

[0114] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A transfer reinforcement learning control method for energy storage battery safety warning, characterized in that: include: Monitor the operating state of the target control object, obtain and pre-process dynamic state data, and construct a current state representation for control decision-making, the current state representation including abnormal control parameters; Based on the current state representation, a dynamic state vector is generated by combining a time series feature modeling method, a spatial association modeling method, and an abnormal feature enhancement mechanism; Calling a control knowledge base containing multi-condition control strategy experience, performing similarity matching between the current control task and known tasks in the control knowledge base, and determining the historical knowledge content to be migrated based on a preset matching strategy; Initializing a controller with the historical knowledge content, using the dynamic state vector as input state information, performing strategy iteration and optimization based on a performance evaluation function constructed by combining abnormal control parameters and operating state, and outputting safety control instructions; Before executing the safety control instruction, a safety logic check of the multi-level control structure is performed, and the control execution result is used as an environmental feedback signal for controller strategy optimization and for updating the control knowledge base.

2. The transfer reinforcement learning control method for energy storage battery safety warning according to claim 1 is characterized in that: The process of preprocessing the dynamic state data includes: eliminating sudden change outliers in the voltage data based on the Laida criterion; using a long short-term memory network to predict and fill missing points in the temperature data; using a Daubechies wavelet function with a set number of layers to suppress high-frequency noise in the current signal; and dynamically correcting low-frequency drift in the temperature data through a Kalman filter algorithm to generate cleaned dynamic state parameters; based on the cleaned dynamic state parameters, using an isolation forest algorithm to perform anomaly detection, generate anomaly confidence scores reflecting the degree of abnormality in the operating state, and combine with context labels to mark potential fault types to obtain the abnormal control parameters.

3. The transfer reinforcement learning control method for energy storage battery safety warning according to claim 1 is characterized in that: The process of generating a dynamic state vector includes: A one-dimensional convolutional neural network is used to extract short-term fluctuation features from the current state representation, and a long short-term memory network is used to perform time series modeling on the short-term fluctuation features to generate a time feature vector; A topological graph is constructed based on the battery pack cell connections of the target control object; node features of the topological graph include voltage and temperature of each cell, and edge weights of the topological graph are set based on impedance and thermal conductivity; and a graph neural network is used to aggregate the electrochemical and thermodynamic interactions between the cells to generate a spatial feature vector. For a preselected feature subset derived from the current state representation, a variational autoencoder is used to identify abnormal states, and potential fault feature components corresponding to the abnormal states are amplified through an attention mechanism to generate an abnormality-enhanced feature vector; The temporal feature vector, the spatial feature vector, and the anomaly enhancement feature vector are fused through a fully connected layer and a layer normalization technique is applied, and are weighted using anomaly confidence scores to generate the dynamic state vector.

4. The transfer reinforcement learning control method for energy storage battery safety warning according to claim 1 is characterized in that: The process of constructing the control knowledge base includes: Constructing a knowledge graph based on a graph database as the control knowledge base, wherein the knowledge graph is used to associate the multi-condition control strategy experience including different battery chemistry systems, aging levels, and operating conditions; The structure of the knowledge graph includes: task nodes, which are used to record the different battery chemical systems, aging degrees and operating parameters; model nodes, which are used to store historical pre-trained model parameters; experience nodes, which are used to store historical feature extractor configuration information and state-action-reward sequences; and edges, which are used to represent semantic relationships between nodes.

5. The transfer reinforcement learning control method for energy storage battery safety warning according to claim 1 is characterized in that: The similarity matching process includes: Using a maximum mean difference method, the data distribution of the current control task and the data domain corresponding to each known task in the control knowledge base are compared to obtain a first similarity measurement value; A second similarity metric is obtained by using a cosine similarity calculation method based on the task description embedding vector; Performing weighted summation on the first similarity metric value and the second similarity metric value according to predetermined weights to calculate a comprehensive similarity score; According to the comprehensive similarity score, a model configuration with the highest comprehensive similarity score is selected from the control knowledge base as the historical knowledge content to be migrated.

6. The transfer reinforcement learning control method for energy storage battery safety warning according to claim 1 is characterized in that: The controller adopts the DDPG algorithm, and the initialization process of the controller includes: selecting pre-trained network weights from the historical knowledge content, and loading the pre-trained network weights into the corresponding parts of the actor network and the critic network of the controller.

7. The transfer reinforcement learning control method for energy storage battery safety warning according to claim 1 is characterized in that: Safety logic verification of multi-level control structures includes: Performing a validity check on whether each parameter value in the security control instruction is within a preset safe operating range; Performing redundancy check on the current state representation and the state parameters expected to be achieved after executing the safety control instruction; According to the currently assessed risk level and the corresponding operation mode, the execution priority of different types of security control instructions is set to form arbitration logic.

Images