Migration reinforcement learning control method for safety early warning of energy storage battery

Abstract

The invention relates to the technical field of self-adaptive control systems, in particular to an energy storage battery safety early warning-oriented migration reinforcement learning control method, which comprises the following steps that: the system performs initialization and self-adaptive adjustment on a controller by monitoring dynamic state parameters of an operation condition in real time and combining control knowledge formed by historical operation experience; and the controller outputs a safety control instruction through iterative optimization of a reinforcement learning mechanism according to the current state parameter and a performance evaluation function with safe operation as a target. The safety control instruction is subjected to safety logic verification before being executed, actual response serves as a feedback signal and is used for closed-loop updating and self-adaptive improvement of control strategies and experience knowledge, and then predictability and response self-adaptive capacity of the whole control system in the aspect of safety protection of the energy storage battery are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of adaptive control, and specifically to a transfer reinforcement learning control method for energy storage battery safety warning. Background Art

[0002] In the field of complex industrial process control, especially for energy storage batteries with strong time-varying, non-linear and multi-condition characteristics, how to construct a control system with parameter self-tuning and model adaptability is the core technical challenge to improve the robustness and autonomy of the system. This precise and self-optimizing control is crucial for ensuring the safe and efficient operation of the system.

[0003] To achieve effective control of such complex dynamic systems, existing technologies mostly adopt architectures such as model reference adaptive control (MRAC) or self-tuning regulator (STR). These methods identify the dynamic characteristics of the system online and adjust the controller parameters accordingly to achieve the purpose of adaptive control. For example, by updating the process model parameters online to adjust the gain of the PID controller, or adjusting the feedforward and feedback control laws in MRAC to track the reference model.

[0004] However, these traditional adaptive control models still face many challenges in practical applications. On the one hand, control models relying on classical parameter adjustment methods such as gradient descent method and least squares identification are prone to model mismatch when the state changes suddenly or encounters unmodeled dynamics, which will affect the control performance and even require manual intervention to reset the identification parameters. On the other hand, when the working conditions of the industrial process change, these adaptive control models often need to re-collect a large amount of training data or perform long-term online identification to adapt to the new working conditions, lacking the ability to quickly transfer control strategies across working conditions. In addition, although some adaptive control models based on reinforcement learning show the potential of online learning, they usually lack effective mechanisms for representing, storing and transferring historical control knowledge, resulting in low model learning efficiency when facing new tasks or environmental changes, and it is difficult to meet the requirements of real-time safety warning for objects with multi-time scale dynamic characteristics in terms of response speed and decision-making accuracy.

[0005] Therefore, a transfer reinforcement learning control method for energy storage battery safety warning is proposed. Summary of the Invention

[0006] The object of the present invention is to provide a transfer reinforcement learning control method for energy storage battery safety early warning. By acquiring and preprocessing multi-source dynamic state parameters, a current state representation including abnormal control parameters is constructed; based on the current state representation, a dynamic state vector is generated by combining time series feature modeling, spatial correlation modeling and abnormal feature enhancement mechanism; a control knowledge base is called, and by performing similarity matching on the current control task and the known tasks in the library and according to a preset matching strategy, the transferable historical knowledge content is determined; then the controller is initialized with the historical knowledge content, a safety control instruction is output; multi-level safety logic verification is performed, and the control execution result is used as an environmental feedback signal for the strategy optimization of the controller and the update of the control knowledge base. The present invention can realize adaptive and accurate early warning and control of the early safety risks of energy storage batteries.

[0007] To achieve the above object, the present invention provides the following technical solutions: A transfer reinforcement learning control method for energy storage battery safety early warning, comprising: Monitoring the operating state of a target control object, acquiring and preprocessing dynamic state data, and constructing a current state representation for control decision-making, the current state representation including abnormal control parameters; Based on the current state representation, combining a time series feature modeling method, a spatial correlation modeling method and an abnormal feature enhancement mechanism to generate a dynamic state vector; Calling a control knowledge base containing multi-condition control strategy experience, performing similarity matching on the current control task and the known tasks in the control knowledge base, and determining the historical knowledge content to be transferred according to a preset matching strategy; Initializing a controller with the historical knowledge content, using the dynamic state vector as input state information, and performing policy iteration and optimization according to a performance evaluation function constructed by combining abnormal control parameters and operating states, and outputting a safety control instruction; Before executing the safety control instruction, performing safety logic verification on a multi-level control structure, and using the control execution result as an environmental feedback signal for the strategy optimization of the controller and at the same time for updating the control knowledge base.

[0008] Further, the process of preprocessing the dynamic state data includes: eliminating mutation outliers in the voltage data based on the Chauvenet's criterion; predicting and filling in the missing points in the temperature data using a long short-term memory network; suppressing high-frequency noise in the current signal using a Daubechies wavelet function with a set number of layers; and dynamically correcting the low-frequency drift in the temperature data through a Kalman filter algorithm to generate the cleaned dynamic state parameters; based on the cleaned dynamic state parameters, performing anomaly detection using an isolation forest algorithm to generate an anomaly confidence score reflecting the degree of abnormal operation state, and combining context labels annotating potential fault types to obtain the anomaly control parameters.

[0009] Further, the process of generating the dynamic state vector includes: Using a one-dimensional convolutional neural network to extract short-term fluctuation features from the current state representation, and combining a long short-term memory network to perform temporal modeling on the short-term fluctuation features to generate a time feature vector; Constructing a topology graph based on the connection of battery pack monomers of the target control object; the node features of the topology graph include the voltage and temperature of each monomer, and the edge weights of the topology graph are set based on impedance and thermal conductivity; aggregating the electrochemical and thermodynamic interactions between monomers through a graph neural network to generate a spatial feature vector; For a preselected feature subset derived from the current state representation, using a variational autoencoder to identify abnormal states, and amplifying the potential fault feature components corresponding to the abnormal states through an attention mechanism to generate an anomaly-enhanced feature vector; Through a fully connected layer and applying layer normalization technology, fusing the time feature vector, the spatial feature vector, and the anomaly-enhanced feature vector, and weighting them using the anomaly confidence score to generate the dynamic state vector.

[0010] Further, the process of constructing the control knowledge base includes: Constructing a knowledge graph based on a graph database as the control knowledge base, which is used to associate the multi-condition control strategy experiences including different battery chemical systems, aging degrees, and operating conditions; The structure of the knowledge graph includes: task nodes for recording the different battery chemical systems, aging degrees, and condition parameters; model nodes for storing historical pre-trained model parameters; experience nodes for storing historical feature extractor configuration information and state-action-reward sequences; and edges for representing the semantic relationships between nodes.

[0011] Further, the process of the similarity matching includes: Using the maximum mean discrepancy method, compare the data distributions of the data domains corresponding to each known task in the control knowledge base with the current control task to obtain a first similarity metric value; Using the cosine similarity calculation method based on the task description embedding vector, obtain a second similarity metric value; Respectively perform weighted summation on the first similarity metric value and the second similarity metric value according to a predetermined weight to calculate a comprehensive similarity score; According to the comprehensive similarity score, select the model configuration with the highest comprehensive similarity score from the control knowledge base as the historical knowledge content to be migrated.

[0012] Furthermore, the controller adopts the DDPG algorithm. The initialization process of the controller includes: selecting pre-trained network weights from the historical knowledge content and loading the pre-trained network weights to the corresponding parts of the actor network and the critic network of the controller.

[0013] Furthermore, the safety logic verification of the multi-level control structure includes: Perform a legality verification on whether each parameter value in the safety control instruction is within the preset safe operation range; Perform a redundancy verification on the current state representation and the state parameters expected to be reached after executing the safety control instruction; According to the currently evaluated risk level and the corresponding operation mode, set the execution priorities of different types of safety control instructions to form an arbitration logic.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. Through the real-time monitoring of the operating state of the target control object, the high-fidelity acquisition of dynamic state parameters of multi-source sensors, and the precise preprocessing of the extraction of deep abnormal control parameters, the present invention constructs a high-quality current state representation. It not only overcomes the problems of insufficient information dimension when the traditional method relies on macroscopic and single parameters for judgment and the early weak fault characteristics are easily submerged by noise, but also combines time series feature modeling, spatial correlation modeling, and abnormal feature enhancement mechanisms to generate dynamic state vectors, realizing a comprehensive, accurate, and multi-dimensional dynamic representation of the complex state inside the energy storage battery, providing a reliable data basis and state perception ability for the subsequent intelligent analysis, precise early warning, and adaptive control decision-making of the energy storage battery.

[0015] 2. The present invention calls a control knowledge base storing multi - operating - condition control strategy experiences, precisely matches the current control task with known historical tasks in the library, and then determines and migrates relevant historical knowledge content according to a preset matching strategy to initialize the controller. This not only speeds up the learning convergence speed of the controller when facing new battery types, different aging stages or unknown operating conditions, reducing the need for a large amount of labeled data for the target scenario and training from scratch; at the same time, it improves the initial performance of the energy storage battery control strategy and the optimization level that can be ultimately achieved.

[0016] 3. The controller of the present invention continuously performs strategy iteration and optimization based on a performance evaluation function constructed from real - time abnormal control parameters and the current operating state, can autonomously learn and output the most optimized safety control instructions, realizing the transformation from a passive response to an active prevention intelligent safety management mode. At the same time, a safety logic verification link with a multi - level control structure is set up to ensure the reliability of control decisions and the safety of execution. In addition, by using the actual control execution result as an environmental feedback signal, it is not only used for the closed - loop continuous optimization of the controller's own strategy, but also for dynamically updating the control knowledge base, thereby improving the operating safety level of the energy storage system, extending the battery life, and optimizing the operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic flowchart of a transfer reinforcement learning control method for energy storage battery safety warning provided by the present invention; Figure 2 is a schematic flowchart of the process of generating a dynamic state vector provided by the present invention; Figure 3 is a schematic flowchart of the process of similarity matching provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0019] Please refer to Figures 1 to 3 , the present invention provides a transfer reinforcement learning control method for energy storage battery safety warning, and the technical solution is as follows: Embodiment 1: This embodiment proposes an innovative solution for the intelligent safety operation and maintenance of energy storage battery systems, and elaborates on its specific implementation process under complex working conditions. It aims to achieve precise online tracking of the multi-dimensional states of energy storage batteries through the collaborative work of edge computing nodes and cloud platforms, and can give early and accurate warnings for early thermal runaway and other safety risks. At the same time, it has the ability of deep adaptability and continuous learning, providing preliminary control decision-making assistance for the safe operation of energy storage battery systems. As Figure 1 shown, a migration reinforcement learning control method for energy storage battery safety warning includes: Monitor the operating state of the target control object, obtain and preprocess the dynamic state parameters from multi-source sensors, and construct the current state representation including abnormal control parameters.

[0020] In this embodiment, the target control object is a battery pack composed of 16 NMC ternary lithium battery monomers, and this battery pack operates under complex working conditions such as high temperature, high-rate charge and discharge. Through the battery management system (BMS) and external sensors (such as temperature sensors, gas sensors, acoustic sensors, etc.), the dynamic state parameters from multi-source sensors are obtained in real time. These parameters include but are not limited to: battery monomer voltage, current, temperature; battery pack total voltage, total current, charge and discharge rate; SOC (state of charge), SOH (state of health), cycle number; acoustic signal characteristics reflecting battery micro-changes, specific gas concentration, etc.

[0021] Furthermore, to improve the data quality, the process of preprocessing the dynamic state data includes: Eliminate the abrupt outliers in the voltage data based on the 3-sigma principle. Specifically, within a set time window, calculate the mean and standard deviation of the voltage data. If a sampling point deviates from the mean by more than 3 times the standard deviation, it is determined as an outlier. This value can be marked as invalid or replaced with adjacent valid values.

[0022] Use a long short-term memory network to predict and fill in the missing points in the temperature data (for example, data loss due to temporary sensor failures or communication interruptions). Train the LSTM model based on the historical temperature sequence, and use its ability to model the time series trend. When data loss is detected, predict the missing value through the previous valid data.

[0023] Suppress the high-frequency noise of the current signal using a Daubechies wavelet function with a set number of layers. Select a suitable Daubechies wavelet basis (such as db4 or db5) and the decomposition layer number (for example, 5 layers), perform multi-scale wavelet decomposition. Perform soft threshold or hard threshold processing on the high-frequency coefficients, and then reconstruct the signal through wavelet inverse transform to effectively suppress the high-frequency noise.

[0024] And the low-frequency drift in the temperature data is dynamically corrected through the Kalman filtering algorithm. By constructing the state equation and the observation equation, and using the recursive estimation ability of the Kalman filter, the true temperature is estimated in real time according to the observed values affected by drift and noise, thereby correcting the baseline offset problem. After purification processing, the cleaned dynamic state parameters are generated.

[0025] Next, based on the cleaned dynamic state parameters, the isolation forest algorithm is used to perform anomaly detection. This algorithm generates an anomaly confidence score reflecting the degree of abnormal operation for each sample (i.e., the set of state parameters at a certain time point). The higher the score, the higher the likelihood or severity of the anomaly. At the same time, semantic annotation of the anomaly is combined with context features (such as whether the voltage exceeds the threshold, whether a specific gas exceeds the standard, the charge-discharge rate, etc.) to form context labels of potential faults. For example: when the voltage and current are abnormal at the same time, it is labeled as "suspected overcharge / overdischarge". The anomaly confidence score, combined with the context label annotating the potential fault type, yields the anomaly control parameter.

[0026] Through these preprocessing methods, it is possible to effectively remove noise, outliers, missing values, and drift in the original dynamic state parameters, and obtain high-quality cleaned dynamic state parameters. Further anomaly detection through the isolation forest algorithm can not only obtain a quantitative indication of the degree of anomaly, but also give a preliminary judgment of the potential fault type in combination with context information, thereby forming a clearly directed anomaly control parameter, providing a solid data foundation for the accurate construction of the subsequent dynamic state vector and the effective decision-making of the controller, and improving the safety and accuracy of the entire energy storage battery safety control.

[0027] Based on the current state representation, a dynamic state vector is generated by combining the time series feature modeling method, the spatial association modeling method, and the anomaly feature enhancement mechanism.

[0028] Furthermore, as Figure 2 shown, the process of generating the dynamic state vector includes: First, the time series feature modeling method includes: using a one-dimensional convolutional neural network (1D-CNN) to extract short-term fluctuation features from the current state representation. By setting convolutional kernels and strides of different sizes, 1D-CNN can effectively capture local features such as the rapid change patterns, edge features, or specific waveform segments of the battery state parameters in a short period of time, which are important clues indicating early anomalies in the battery. Then, a long short-term memory network (LSTM) is used to perform temporal modeling on the short-term fluctuation features to extract their dynamic laws evolving over time, generating a time feature vector.

[0029] Next, the spatial correlation modeling method includes: constructing a topological graph model based on the connection relationship of each battery cell in the target control object. In this model, nodes represent each battery cell, and its node features include state parameters such as real-time voltage, temperature, and SOC; the weight of the edge is set according to the internal impedance and heat conduction coefficient between the monomers to reflect their electrochemical and thermodynamic coupling relationships. Through a graph neural network (GNN), this topological graph is modeled to aggregate and capture the complex interactions between battery cells caused by factors such as structural differences, uneven heat distribution, or inconsistent aging, generating a spatial feature vector. For example, using a graph neural network with two layers and 64 neurons in each layer, a 128-dimensional spatial feature vector can be output to completely describe the spatial correlation characteristics of the battery pack.

[0030] Then, the abnormal feature enhancement mechanism includes: inputting a preselected feature subset (such as specific gas concentration, acoustic signal features, etc.) from the current state representation into a pre-trained variational autoencoder (VAE) for anomaly detection. The VAE has a small reconstruction error in the normal state through an encoding-reconstruction mechanism, while its error significantly increases in the abnormal state. Based on the comparison between the reconstruction error and a set threshold (such as 0.15), the abnormal state can be identified, and at the same time, the representation in the latent space of the VAE is used as an abnormal enhancement feature. Further introducing an attention mechanism, more attention weights are assigned to the above latent abnormal features, thereby amplifying the latent fault feature components corresponding to the abnormal state to form an abnormal enhancement feature vector.

[0031] Finally, through a fully connected layer and applying layer normalization technology, the time feature vector, the spatial feature vector, and the abnormal enhancement feature vector are fused. At the same time, an abnormal confidence score is introduced as a weighting factor to strengthen the response to high-risk states. The fused output is the dynamic state vector, which can comprehensively and dynamically reflect the current comprehensive health state, aging degree, and potential safety risk level of the battery.

[0032] The time feature depicts the dynamic evolution trend of the state, the spatial feature reflects the coupling relationship between battery cells, and the abnormal enhancement mechanism improves the sensitivity of the system to early weak faults. The finally generated dynamic state vector has high information density and strong representation ability, providing a high-quality, low-dimensional, and easy-to-process state input for the decision-making of the subsequent migration controller, improving the warning accuracy and control reliability, and thus enhancing the safety and precision of the entire energy storage battery safety control.

[0033] Call the control knowledge base containing the experience of multi-condition control strategies, perform similarity matching on the current control task and the known tasks in the control knowledge base, and determine the historical knowledge content to be migrated according to the preset matching strategy.

[0034] Furthermore, the process of constructing the control knowledge base includes: Construct a knowledge graph based on a graph database as the control knowledge base, which is used to associate the multi-condition control strategy experiences including different battery chemical systems, aging degrees, and operating conditions. Among them, in this embodiment, a graph database (such as Neo4j) is selected as the underlying data storage and management platform, and a knowledge graph is constructed on this basis to describe the complex association relationships among batteries, operating conditions, model parameters, and control strategies. The multi-condition control strategy experiences in the knowledge graph are derived from the historical operation data, simulation results of different types of batteries under various usage conditions, as well as the strategies and models obtained through reinforcement learning or other optimization algorithms.

[0035] The structure of the knowledge graph includes: Task nodes, which are used to record the different battery chemical systems, aging degrees, and condition parameters. Each task node records the battery chemical system (such as NMC ternary lithium, LFP lithium iron phosphate, NCA, etc.), the aging degree (such as new battery, SOH = 90%, SOH = 80%, etc.), and the condition parameters (such as ambient temperature, charge and discharge rate, SOC range, number of cycles, and duration, etc.).

[0036] Model nodes, which are used to store historical pre-trained model parameters; this node includes the structure information and weight configuration of the deep neural network, the parameterized expression of the control strategy, or the hyperparameter settings of a specific feature extractor, etc.

[0037] Edges, which are used to represent the semantic relationships between nodes, including similarity connections between tasks, the attribution relationship between models and tasks, the association relationship between experiences and models, etc. In particular, the similarity edges between task nodes can be provided with weights to quantify the proximity of different tasks in terms of data distribution, condition parameters, etc., so as to support fast similarity retrieval and experience transfer.

[0038] By constructing the method of the control knowledge base, heterogeneous multi-condition control strategy experiences can be uniformly expressed in a structured and retrievable form. The knowledge graph based on the graph database can not only clearly express the complex associations among different battery characteristics, conditions, model parameters, and historical experiences, but also facilitate dynamic updates and on-demand expansions. This provides a solid foundation for realizing effective knowledge transfer and accelerating the learning process of the controller, thereby improving the safety and accuracy of the entire energy storage battery safety control.

[0039] Furthermore, as Figure 3 shown, when facing a new current control task, it is necessary to find the most similar one to the target task from multiple known tasks stored in the control knowledge base. The process of the similarity matching includes: Using the Maximum Mean Discrepancy (MMD) method, compare the data distributions of the data domains corresponding to each known task in the control knowledge base with the current control task to obtain a first similarity metric value; Specifically, for the target task, collect a small amount of representative data in its initial stage; at the same time, extract the historical data or its statistical features of each source task from the knowledge base. As a non-parametric statistical method, MMD can be used to determine whether two sets of samples are from the same distribution. By calculating the MMD values between the target task and the data of each source task, the similarity degree at the data level can be quantitatively evaluated. The smaller the MMD value, the closer the distributions are and the higher the similarity.

[0040] Using the cosine similarity calculation method based on the task description embedding vector, obtain a second similarity metric value; Specifically, each task can be characterized as a high-dimensional task description embedding vector through its metadata (such as battery chemistry system, aging degree, operating condition parameter range, etc.). This vector can be generated by a pre-trained task encoder model (such as an encoder based on Transformer) or a rule-based feature engineering method. By calculating the cosine similarity between the embedding vectors of the current control task and the known tasks, the similarity at the task semantic level can be evaluated. The closer the cosine similarity is to 1, the more similar the task descriptions are.

[0041] Respectively perform weighted summation on the first similarity metric value and the second similarity metric value according to a predetermined weight to calculate a comprehensive similarity score; Among them, the setting of the weight can be adjusted according to the importance and reliability of different similarity dimensions in the current application scenario.

[0042] According to the calculated comprehensive similarity scores of each known task, select the model configuration with the highest comprehensive similarity score from the control knowledge base as the transferable historical knowledge content.

[0043] Among them, the model configuration includes pre-trained model weights, feature extractor structure parameters, parameterized representations of control strategies, etc., and has the transferability to be directly used to initialize the new task model.

[0044] Through the above multi-dimensional similarity matching mechanism and matching decision-making process, the efficient screening and scientific transfer of a large amount of historical experience are realized. This method establishes a systematic similarity evaluation system at multiple levels such as data distribution and task characteristics, effectively improving the accuracy and adaptability of transfer decisions, thereby enhancing the safety and accuracy of the entire energy storage battery safety control.

[0045] Initialize the controller with the historical knowledge content, use the dynamic state vector as the input state information, and perform policy iteration and optimization according to the performance evaluation function constructed by combining the abnormal control parameter and the operating state, and output a safety control instruction.

[0046] Among them, the performance evaluation function is the reward function in the reinforcement learning framework. This reward function aims to comprehensively evaluate the safety and effectiveness of the control instructions output by the controller in a specific state by combining the abnormal control parameter and the operating state. Specifically, this reward function comprehensively considers the following multiple dimensions: Whether the instruction can be accurately issued in the early stage of the fault; whether the risk can be effectively mitigated or the fault escalation can be prevented after the control measure is executed; whether the irreversible damage to the battery is avoided due to overly aggressive control. In addition, the weights of the components of the reward function can be dynamically adjusted according to the risk level reflected by the abnormal control parameter and the current operating state (such as temperature and charge-discharge rate, etc.), so as to improve the response ability and adaptability to the actual working conditions.

[0047] By continuously interacting with the environment, the controller receives the signals generated by the reward function, conducts policy iteration and optimization, aiming to maximize the long-term cumulative expected return. The output safety control instructions can cover multiple dimensions, including but not limited to: adjusting the charge-discharge current or power limit, regulating the cooling system intensity, performing battery balancing operations, and even triggering protective shutdown in extreme cases, etc.

[0048] Furthermore, the controller adopts the DDPG algorithm; The DDPG controller consists of an actor network and a critic network, both of which are implemented based on a deep neural network and have a preset network hierarchical structure (for example: 2 to 3 hidden layers, each layer contains 128 or 256 neurons, and activation functions such as ReLU or tanh can be used).

[0049] The initialization process of the controller includes: selecting the pre-trained network weights compatible with the current task from the historical knowledge content and loading them into the corresponding layers of the actor network and the critic network respectively. These pre-trained weights usually come from similar tasks completed in history (for example, the deep reinforcement learning model trained by similar types of batteries under approximate working conditions). If there is a model trained based on the DQN algorithm in the historical task and its structure is compatible with a specific network layer of the current DDPG controller (such as the front-layer feature extraction module or the intermediate value evaluation layer), the pre-trained weights of this model can be directly extracted for initialization.

[0050] The initialization process is specifically manifested as loading these selected pre-trained network weights into the corresponding parts of the actor network and the critic network of the DDPG controller, which can significantly accelerate the learning convergence speed of the new controller on the target task and may help it achieve better performance.

[0051] By clarifying the controller to enable it to effectively handle the complex continuous control problems of the energy storage battery system, and initializing the controller by selecting and loading pre-trained network weights from historical knowledge content, effective knowledge transfer is achieved. This initialization method gives the controller a more favorable learning starting point, avoids the inefficiency and instability of learning from scratch, thus accelerating the learning and adaptation speed of the controller in new working conditions or new battery types, reducing the dependence on a large amount of new interaction data, and helping the controller learn more robust and better-performing safety control strategies.

[0052] Before executing the safety control instruction, perform safety logic verification on the multi-level control structure, and use the control execution result as an environmental feedback signal for the strategy optimization of the controller, and at the same time use it to update the control knowledge base.

[0053] Among them, the safety control instruction can only be executed after passing the safety logic verification. The control execution results generated during the execution process (for example, the actual current adjustment value reported by the BMS, the temperature change situation, the battery state parameter response, etc.) will be collected and constitute a part of the environmental feedback signal, which is used to adjust the control strategy by the controller in subsequent strategy iterations. In addition, the control execution result can also be used as effective control experience in actual operation and updated to the control knowledge base for future similar tasks to call, realizing experience transfer and strategy accumulation.

[0054] Furthermore, performing safety logic verification on the multi-level control structure includes: Perform legality verification on whether each parameter value in the safety control instruction is within the preset safe operation range. Make a legality judgment on each parameter value in the safety control instruction to verify whether it is within the preset safe operation range. For example, after the controller outputs an instruction containing control parameters such as charging current, discharge power limit, and cooling system level, the system will compare these values item by item to see if they meet the safety thresholds set for the current battery type, operating state (such as SOC, SOH, and temperature), and the current mode. For example, for a specific model of NMC battery, under the current temperature condition, the maximum allowable charging current may be limited to no more than 1.5C, and exceeding this value is considered illegal.

[0055] Perform redundancy verification on the current state representation and the state parameters expected to be achieved after executing the safety control instruction. Compare the current state representation with the target state parameters expected to be achieved after the instruction execution to identify potential unreasonable controls. For example, if the current SOC is 98% and the control instruction still requires a large current to continue charging, then this instruction has a logical conflict or potential risk. Redundancy verification can be completed based on a simplified battery model or a rule prediction engine, and is used to identify instruction logics that do not match the actual system state.

[0056] According to the currently evaluated risk level and the corresponding operation mode, set the execution priorities of different types of safety control instructions to form an arbitration logic.

[0057] In complex scenarios, the controller may output multiple sub-instructions, or there may be control requests from other systems (such as the basic protection logic of the BMS and the dispatching commands of the EMS). At this time, arbitration is required based on the risk level (such as "warning level", "hazard level" or "emergency level") and the current operation mode (such as fast charging, normal discharging, fault emergency handling). For example, when identified as an "emergency-level risk" such as the initial signs of thermal runaway, high-priority instructions such as "emergency shutdown" or "activate the fire suppression mechanism" will be executed first, and other normal operations will be interrupted or postponed; while when the risk is relatively low but the temperature continues to rise, the "enhanced cooling" instruction may obtain a higher execution priority than "maintain the current power".

[0058] Through a multi-level verification mechanism, a crucial safety buffer layer is added between the control instructions output by the controller and their actual effects on the energy storage battery system. Legality verification ensures that control parameters do not exceed physical or policy boundaries; redundancy verification improves the matching degree between instructions and system states, preventing execution failures or bringing additional risks; priority arbitration ensures that the most protective control strategies can be implemented first at critical moments, enhancing the safety, reliability, and robustness of the execution of control instructions, avoiding secondary risks or safety accidents caused by improper control, and thus improving the stability and credibility of the energy storage battery.

[0059] By real-time monitoring and refined preprocessing of multi-source dynamic state parameters, construct input information containing key abnormal control parameters, and generate accurate dynamic state vectors in combination with multi-dimensional modeling technology, so as to comprehensively and real-time evaluate the internal state of the battery, improving the capture ability of early faults and the accuracy of early warning. Secondly, use a control knowledge base containing multi-condition experience, determine transferable historical knowledge through similarity matching and preset strategies, initialize the controller, and enhance the system's adaptive ability and learning efficiency for different battery types, aging stages, and variable working conditions. The controller performs strategy iteration and optimization based on the performance evaluation function constructed by abnormal control parameters and the current operating state, learns and executes the optimal safety control instructions, and realizes intelligent intervention in potential safety risks. In addition, the multi-level safety logic verification before the execution of safety control instructions ensures the safety and reliability of operations, and uses the control execution results as environmental feedback for the continuous optimization of the controller's strategy and the dynamic update of the control knowledge base, forming an effective closed-loop learning and evolution mechanism, enabling the continuous improvement of the control performance of the energy storage battery.

[0060] Embodiment 2: Based on Embodiment 1, this embodiment further illustrates and demonstrates the actual performance in a simulated application environment through specific data tests and experimental results, quantitatively evaluates the accuracy and timeliness of the method in early safety risk identification, the contribution of the transfer learning mechanism to improving the adaptability and learning efficiency of the model to different working conditions and battery states, and the effectiveness of the controller in making adaptive safety control decisions under dynamically changing conditions. A transfer reinforcement learning control method for energy storage battery safety warning, comprising: Monitor the operating state of the target control object, acquire and preprocess the dynamic state parameters from multi-source sensors, and construct the current state representation including abnormal control parameters; Based on the current state representation, combine the time series feature modeling method, the spatial correlation modeling method, and the abnormal feature enhancement mechanism to generate a dynamic state vector; Call the control knowledge base containing the experience of multi-condition control strategies, perform similarity matching between the current control task and the known tasks in the control knowledge base, and determine the historical knowledge content to be transferred according to the preset matching strategy; Initialize the controller with the historical knowledge content, use the dynamic state vector as the input state information, and perform policy iteration and optimization according to the performance evaluation function constructed by combining abnormal control parameters and operating states, and output safety control instructions; Before executing the safety control instruction, perform safety logic verification of the multi-level control structure, and use the control execution result as the environmental feedback signal for the policy optimization of the controller, and at the same time use it to update the control knowledge base.

[0061] To objectively evaluate the effect of the present invention, two representative baseline methods are selected for comparison: Baseline method 1: Traditional threshold warning method, which relies on setting fixed upper and lower threshold values for the key operating parameters (such as voltage, current, temperature) of the battery. When the monitored parameter value exceeds these preset thresholds, the system will trigger an alarm.

[0062] Baseline method 2: Standard reinforcement learning method, which adopts a controller architecture similar to that of the present invention, but does not use the knowledge transfer mechanism. That is, when facing new battery types, aging states, or working conditions that have not been trained, it needs to start from scratch or only based on a small amount of random initialization for learning and policy optimization.

[0063] To examine the optimal performance level of the present invention in an ideal or relatively familiar environment, key indicators are calculated, including: the warning accuracy rate (F1 score) is statistically calculated based on the comparison between the system warning output and the actual situation on a test data set containing true fault labels; the warning lead time is the average value of the time difference between the moment when the system issues a warning and the time when the fault develops to the critical failure point in the correctly warned fault cases; the control effectiveness is mainly evaluated by observing and quantifying the success probability of the output safety control instructions in mitigating the simulated early faults in a controlled experimental environment; for learning-based methods, their learning efficiency is measured by the number of training iterations or the number of samples required to reach a predetermined performance target or the model convergence state; and the calculation delay is actually measured on the target hardware platform, which is the average time required for the system to generate a decision output from receiving new data.

[0064] As shown in Table 1, under known working conditions, the present invention can effectively capture complex fault patterns through knowledge transfer and dynamic state representation, and its accuracy rate is better than that of general reinforcement learning methods. In addition, with the help of the anomaly enhancement mechanism, the present invention can detect faults earlier, and its accuracy rate is also better than that of other baseline methods. At the same time, the application of knowledge transfer reduces the number of training steps, thereby improving the learning efficiency, and thus enhancing the stability and reliability of the energy storage battery. Where N / A indicates no such data content.

[0065] Table 1 Comparison of effects under known working conditions Table 2 Comparison of effects under new working conditions To verify the effect of the transfer learning mechanism of the present invention, the original NMC battery was placed in a low-temperature environment that was not fully learned during training and its early anomalies were monitored. As shown in Table 2, under the new working conditions, the accuracy rates of other baseline methods decreased due to learning from scratch or not adapting to the control environment, while the present invention still maintained a high accuracy rate through transfer and online optimization, thereby enhancing the stability and reliability of the energy storage battery.

[0066] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A transfer reinforcement learning control method for energy storage battery safety warning, characterized in that include: Monitor the operating state of the target control object, obtain and pre-process dynamic state data, and construct a current state representation for control decision-making, the current state representation including abnormal control parameters; Based on the current state representation, a dynamic state vector is generated by combining a time series feature modeling method, a spatial association modeling method, and an abnormal feature enhancement mechanism; Calling a control knowledge base containing multi-condition control strategy experience, performing similarity matching between the current control task and known tasks in the control knowledge base, and determining the historical knowledge content to be migrated based on a preset matching strategy; Initializing a controller with the historical knowledge content, using the dynamic state vector as input state information, performing strategy iteration and optimization based on a performance evaluation function constructed by combining abnormal control parameters and operating state, and outputting safety control instructions; Before executing the safety control instruction, a safety logic check of the multi-level control structure is performed, and the control execution result is used as an environmental feedback signal for controller strategy optimization and for updating the control knowledge base.

2. The migration reinforcement learning control method for energy storage battery safety warning according to claim 1, wherein The process of preprocessing the dynamic state data includes: eliminating sudden change outliers in the voltage data based on the Laida criterion; using a long short-term memory network to predict and fill missing points in the temperature data; using a Daubechies wavelet function with a set number of layers to suppress high-frequency noise in the current signal; and dynamically correcting low-frequency drift in the temperature data through a Kalman filter algorithm to generate cleaned dynamic state parameters; based on the cleaned dynamic state parameters, using an isolation forest algorithm to perform anomaly detection, generate anomaly confidence scores reflecting the degree of abnormality in the operating state, and combine with context labels to mark potential fault types to obtain the abnormal control parameters.

3. The migration reinforcement learning control method for energy storage battery safety warning according to claim 1, characterized in that The process of generating a dynamic state vector includes: A one-dimensional convolutional neural network is used to extract short-term fluctuation features from the current state representation, and a long short-term memory network is used to perform time series modeling on the short-term fluctuation features to generate a time feature vector; A topological graph is constructed based on the battery pack cell connections of the target control object; node features of the topological graph include voltage and temperature of each cell, and edge weights of the topological graph are set based on impedance and thermal conductivity; and a graph neural network is used to aggregate the electrochemical and thermodynamic interactions between the cells to generate a spatial feature vector. For a preselected feature subset derived from the current state representation, a variational autoencoder is used to identify abnormal states, and potential fault feature components corresponding to the abnormal states are amplified through an attention mechanism to generate an abnormality-enhanced feature vector; The temporal feature vector, the spatial feature vector, and the anomaly enhancement feature vector are fused through a fully connected layer and by applying a layer normalization technique, and are weighted using anomaly confidence scores to generate the dynamic state vector.

4. The migration reinforcement learning control method for energy storage battery safety warning according to claim 1, wherein The process of constructing the control knowledge base includes: Constructing a knowledge graph based on a graph database as the control knowledge base, wherein the knowledge graph is used to associate the multi-condition control strategy experience including different battery chemistry systems, aging levels, and operating conditions; The structure of the knowledge graph includes: task nodes for recording the different battery chemical systems, aging degrees, and operating condition parameters; model nodes for storing historical pre-trained model parameters; experience nodes for storing historical feature extractor configuration information and state-action-reward sequences; and edges for representing the semantic relationships between nodes.

5. The migration reinforcement learning control method for energy storage battery safety warning according to claim 1, characterized in that The process of the similarity matching includes: Using the maximum mean discrepancy method to compare the data distributions of the corresponding data domains of the current control task and each known task in the control knowledge base, obtaining a first similarity metric value; Using the cosine similarity calculation method based on the task description embedding vector to obtain a second similarity metric value; Respectively performing weighted summation on the first similarity metric value and the second similarity metric value according to a predetermined weight to calculate a comprehensive similarity score; According to the comprehensive similarity score, selecting the model configuration with the highest comprehensive similarity score from the control knowledge base as the historical knowledge content to be migrated.

6. The migration reinforcement learning control method for energy storage battery safety warning according to claim 1, characterized in that The controller adopts the DDPG algorithm. The initialization process of the controller includes: selecting pre-trained network weights from the historical knowledge content and loading the pre-trained network weights to the corresponding parts of the actor network and the critic network of the controller.

7. A migration reinforcement learning control method for energy storage battery safety warning according to claim 1, characterized in that Performing safety logic verification for the multi-level control structure includes: Performing legality verification on whether the parameter values in the safety control instruction are within the preset safe operation range; Performing redundancy verification on the current state representation and the state parameters expected to be reached after executing the safety control instruction; According to the currently evaluated risk level and the corresponding operation mode, setting the execution priorities of different types of safety control instructions to form an arbitration logic.

Images