Energy storage operation and maintenance method and device based on reinforcement learning

Abstract

The application relates to the technical field of energy storage operation and maintenance, and discloses an energy storage operation and maintenance method and device based on reinforcement learning. The method comprises the following steps: constructing a reinforcement learning model for energy storage power station operation and maintenance; constructing an operation and maintenance knowledge base, wherein the operation and maintenance knowledge base comprises an experience pool and a policy library; acquiring operation state data of an energy storage power station; performing feature extraction on the operation state data to obtain target state features; performing similarity retrieval in the policy library based on the target state features to obtain a retrieval result; determining a generation mode of an operation and maintenance action based on the retrieval result and a preset state evaluation rule, wherein the generation mode comprises generating an operation and maintenance action based on the reinforcement learning model and generating an operation and maintenance action based on the operation and maintenance knowledge base; and generating an operation and maintenance action of the energy storage power station based on the generation mode.

Description

Technical Field

[0001] This application relates to the field of energy storage operation and maintenance technology, and more specifically, to an energy storage operation and maintenance method and device based on reinforcement learning. Background Technology

[0002] With the rapid development of the new energy industry, Battery Energy Storage Systems (BESS) are increasingly widely used in scenarios such as grid peak shaving and frequency regulation, microgrids, and industrial and commercial energy storage. An energy storage system comprises multiple complex subsystems, including battery clusters, a Battery Management System (BMS), an Energy Management System (EMS), and a Power Conversion System (PCS). During long-term operation, energy storage power stations face multi-dimensional operation and maintenance challenges, such as equipment aging, environmental interference, and underlying communication failures.

[0003] Currently, the operation and maintenance (O&M) of energy storage power stations relies heavily on the experience of field engineers and traditional static knowledge bases (such as electronic product manuals and static fault diagnosis guides). When the BMS or EMS reports complex alarm logs (such as multi-level voltage anomaly linkage, early minor characteristics of thermal runaway, etc.), O&M personnel usually need to manually search for solutions in the knowledge base using "keyword search".

[0004] Among related technologies, the energy storage operation and maintenance knowledge base system mainly has the following technical problems:

[0005] (1) Low knowledge retrieval efficiency and poor matching: Existing static knowledge bases mostly adopt retrieval methods based on TF-IDF or simple vector similarity. Faults in energy storage systems often have complex contextual relationships (such as a sudden increase in internal resistance under a specific SOC range). Traditional retrieval methods cannot understand the dynamic context of the fault occurrence, resulting in low accuracy of recommended troubleshooting solutions.

[0006] (2) Lack of adaptive and continuous evolution capabilities: Traditional expert system rules are hard-coded, and the knowledge system is solidified, making it impossible to automatically evolve with the environment. When battery cells age with the increase of cycle count, or when the power station introduces new operating strategies, the mapping relationship between old fault characteristics and solutions may become invalid. Existing systems cannot automatically correct and evolve their knowledge recommendation strategies based on actual feedback from maintenance personnel.

[0007] (3) Implicit knowledge is difficult to be effectively accumulated: The "implicit experience" accumulated by senior operation and maintenance experts when dealing with non-standard faults (such as non-standard troubleshooting steps derived from a certain abnormal BMS CSV log data) is difficult to be automatically converted into high-priority structured rules in the knowledge base after the work order ends. Knowledge transfer relies on manual summarization, which is inefficient and has a narrow coverage.

[0008] (4) Lack of feedback loop: The behavior of operation and maintenance personnel in adopting or rejecting suggestions is not effectively utilized by the operation and maintenance system, and continuous optimization of energy storage operation and maintenance cannot be achieved.

[0009] There is currently no effective solution to the aforementioned technical problems. Summary of the Invention

[0010] This application provides a reinforcement learning-based energy storage operation and maintenance method and device to at least solve the technical problems of low retrieval accuracy of traditional knowledge bases in complex energy storage fault scenarios, fixed knowledge bases that cannot evolve with the life cycle of energy storage power stations, difficulty in effectively accumulating tacit knowledge, and inability to achieve continuous optimization of energy storage operation and maintenance.

[0011] According to one aspect of the embodiments of this application, a reinforcement learning-based energy storage operation and maintenance method is provided, comprising:

[0012] A reinforcement learning model for the operation and maintenance of an energy storage power station is constructed, wherein the input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions.

[0013] An operations and maintenance knowledge base is constructed, comprising an experience pool and a strategy base. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step. The strategy base stores strategy knowledge, each strategy knowledge being represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the strategy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs.

[0014] Acquire operational status data of energy storage power stations;

[0015] Feature extraction is performed on the operating status data to obtain the target status features;

[0016] Based on the target state features, a similarity search is performed in the strategy library to obtain the search results;

[0017] The generation method of operation and maintenance actions is determined based on the search results and the preset state evaluation rules, wherein the generation method includes generating operation and maintenance actions based on the reinforcement learning model and generating operation and maintenance actions based on the operation and maintenance knowledge base;

[0018] Based on the aforementioned generation method, the operation and maintenance actions of the energy storage power station are generated.

[0019] Optionally, the construction of the reinforcement learning model for the operation and maintenance of the energy storage power station includes:

[0020] A digital twin simulation environment for an energy storage power station is constructed, comprising a state space S, an action space A, a state transition probability P, and a reward function R. The state space S includes multiple parameters such as battery state of charge, charging power, discharging power, cell temperature, battery health status, grid electricity price, grid load demand, and timestamps. The action space includes at least one of start / stop commands, charging / discharging power commands, net power commands, and maintenance suggestions. The reward function R is a quantified function that integrates economic benefits, equipment losses, and safety constraints.

[0021] Construct an agent for the reinforcement learning model, wherein the agent includes a policy network π_θ and a value network Q_θ. Where θ represents the parameters of the policy network to be trained. The parameters of the value network to be trained are: the input of the policy network π_θ is state s, and the output of the policy network π_θ is action a or the probability distribution of action a.

[0022] The agent is trained using deep reinforcement learning to obtain the reinforcement learning model.

[0023] Optionally, the step of performing a similarity search in the strategy library based on the target state features to obtain search results includes:

[0024] Based on the scene feature vector of the target state features, a similarity search is performed in the policy library; wherein, if the similarity between the scene feature vector and the target scene feature vector in the policy library is greater than a preset similarity threshold, the similarity match is determined to be successful; otherwise, the similarity match is determined to be unsuccessful.

[0025] If the search result indicates a successful similarity match, the corresponding action template and the strategy effect evaluation value are retrieved from the strategy library.

[0026] Optionally, the method for determining the generation of operation and maintenance actions based on the search results and preset status assessment rules includes:

[0027] If the search result is a successful similarity match and the strategy effect evaluation value is greater than the preset effect evaluation threshold, it is determined that the operation and maintenance actions of the energy storage power station are generated based on the action template.

[0028] If the search result is a similarity match failure, or if the search result is a similarity match success and the strategy effect evaluation value is less than or equal to the preset effect evaluation threshold, it is determined that the operation and maintenance actions of the energy storage power station are generated based on the strategy network of the reinforcement learning model.

[0029] Optionally, after generating the operation and maintenance actions of the energy storage power station based on the generation method, the method further includes:

[0030] The generated current action a_t is sent to the energy management system of the energy storage power station for execution, and the next moment state s_{t+1} and instant reward r_t of the energy storage power station are monitored;

[0031] The generated decision experience tuple (s_t, a_t, r_t, s_{t+1}) is stored in the experience pool;

[0032] Periodically cluster the decision-making experiences in the experience pool to update or generate policy knowledge in the policy library, and evaluate and update the policy effects of the policy knowledge in the policy library.

[0033] Optionally, the step of evaluating and updating the policy effectiveness of the policy knowledge in the policy base includes:

[0034] For target strategies already used in the strategy library, update the strategy effectiveness evaluation value and the number of times the target strategy is applied based on the actual reward for new experiences derived from the target strategy.

[0035] Optionally, the method further includes:

[0036] The experience pool is sampled to obtain target experience data;

[0037] Based on the target experience data, the policy network π_θ and the value network Q_θ are trained offline. Update the policy network parameters θ of the policy network π_θ and the value network Q_ Value network parameters The policy network π_θ updates its parameters θ by minimizing the temporal difference error, and the value network Q_ The value network parameters are updated by maximizing the expectation of the value function. ;

[0038] Based on the policy network parameters θ and the value network parameters Update the reinforcement learning model;

[0039] The updated reinforcement learning model is tested in the digital twin simulation environment to evaluate the average cumulative reward of the reinforcement learning model.

[0040] The reinforcement learning model is evaluated based on the average cumulative reward.

[0041] Optionally, the method further includes:

[0042] The decision suggestions and / or decision basis for the operation and maintenance actions are visualized; if the operation and maintenance actions are generated based on the operation and maintenance knowledge base, the strategy records matching the operation and maintenance actions and the historical effects of the strategy records are displayed; if the operation and maintenance actions are generated based on the reinforcement learning model, the key state features of the reinforcement learning model's decision are visualized.

[0043] Optionally, the method further includes:

[0044] Based on the preset target operation and maintenance actions, the action sequence is obtained by deducing in the digital twin simulation environment;

[0045] Predicting the impact of the action sequence on the operating indicators of the energy storage power station within a predetermined future timeframe, wherein the operating indicators of the energy storage power station include at least one of battery state of charge, battery life degradation, and power station revenue curve; and / or

[0046] The method further includes:

[0047] The target operation and maintenance decisions confirmed through simulation and deduction based on the preset target operation and maintenance actions are marked as expert experience.

[0048] The expert experience is stored in the operation and maintenance knowledge base, and a corresponding confidence weight is configured for the expert experience.

[0049] According to another aspect of the embodiments of this application, a reinforcement learning-based energy storage operation and maintenance device is also provided, comprising:

[0050] The first construction module is configured to construct a reinforcement learning model for the operation and maintenance of an energy storage power station. The input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions.

[0051] The second construction module is configured to build an operation and maintenance knowledge base, which includes an experience pool and a strategy library. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step. The strategy library stores strategy knowledge, and each piece of strategy knowledge is represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the strategy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs.

[0052] The acquisition module is configured to acquire the operating status data of the energy storage power station;

[0053] The feature extraction module is configured to extract features from the running state data to obtain target state features;

[0054] The similarity retrieval module is configured to perform a similarity retrieval in the strategy library based on the target state features to obtain retrieval results;

[0055] The generation method determination module is configured to determine the generation method of operation and maintenance actions based on the search results and preset state evaluation rules, wherein the generation method includes generating operation and maintenance actions based on the reinforcement learning model and generating operation and maintenance actions based on the operation and maintenance knowledge base;

[0056] The operation and maintenance action generation module is configured to generate operation and maintenance actions for the energy storage power station based on the aforementioned generation method.

[0057] The energy storage operation and maintenance method and apparatus based on reinforcement learning provided in this application embodiment constructs a reinforcement learning model for the operation and maintenance of energy storage power stations. The input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions. An operation and maintenance knowledge base is constructed, comprising an experience pool and a strategy base. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step. The strategy base stores strategy knowledge, each strategy knowledge item represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the strategy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs. The method also acquires energy storage... The system generates operation status data for the power station; extracts features from the operation status data to obtain target status features; performs a similarity search in the strategy library based on the target status features to obtain search results; determines the generation method of operation and maintenance actions based on the search results and preset status evaluation rules, wherein the generation method includes generating operation and maintenance actions based on the reinforcement learning model and generating operation and maintenance actions based on the operation and maintenance knowledge base; generates operation and maintenance actions for the energy storage power station based on the generation method, which can generate corresponding operation and maintenance actions through strategy library matching and reinforcement learning, combined with case-based reasoning (operation and maintenance knowledge base) and reinforcement learning model-driven reasoning, thereby improving the real-time performance, interpretability, and security of energy storage power station operation and maintenance decisions, and ensuring the safe and reliable operation of the energy storage power station; at the same time, the online update mechanism of the knowledge base enables the system to continuously learn from actual operation, transforming "specific experience" into "general strategies", realizing the dynamic evolution of knowledge and continuous optimization of energy storage operation and maintenance. Attached Figure Description

[0058] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0059] Figure 1 This is a structural block diagram of the energy storage operation and maintenance method based on reinforcement learning provided in the embodiments of this application;

[0060] Figure 2 This is a communication architecture diagram of a battery management system and an energy storage converter based on a reinforcement learning-based energy storage operation and maintenance method provided in the embodiments of this application.

[0061] Figure 3 This is a schematic diagram of the battery cluster arrangement according to the reinforcement learning-based energy storage operation and maintenance method provided in the embodiments of this application;

[0062] Figure 4 This is a partial layout diagram of a battery cluster according to the reinforcement learning-based energy storage operation and maintenance method provided in the embodiments of this application;

[0063] Figure 5 This is a flowchart of an energy storage operation and maintenance device based on reinforcement learning, provided according to an embodiment of this application. Detailed Implementation

[0064] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0065] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0066] According to the embodiments of this application, an embodiment of an energy storage operation and maintenance method and device based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0067] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a reinforcement learning-based energy storage operation and maintenance method is shown. Figure 1As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0068] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0069] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the reinforcement learning-based energy storage operation and maintenance method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the reinforcement learning-based energy storage operation and maintenance method of the aforementioned application. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0070] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0071] Operation and maintenance (O&M) of energy storage power stations refers to a series of tasks including operation monitoring, daily maintenance, fault handling, data analysis, and safety management to ensure their safe, stable, and efficient operation. With the accelerated construction of new power systems, energy storage power stations, acting as "city power banks" or "super power banks," play a crucial role in peak shaving and valley filling, frequency and phase regulation, and improving the absorption of new energy sources. Their O&M has become a vital link in ensuring energy security.

[0072] Figure 2 This is a flowchart illustrating the reinforcement learning-based energy storage operation and maintenance method provided in the embodiments of this application, as shown below. Figure 2 As shown, the method includes the following steps:

[0073] Step S201: Construct a reinforcement learning model for the operation and maintenance of the energy storage power station. The input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions.

[0074] Among them, the energy storage operation and maintenance method is applied to the energy storage operation and maintenance system (hereinafter referred to as the system), and the reinforcement learning model is constructed based on the Markov decision process MDP = (S, A, P, R, γ).

[0075] As an optional embodiment, step S201 can be implemented according to the following steps: the construction of the reinforcement learning model for the operation and maintenance of the energy storage power station includes:

[0076] S2011, Construct a digital twin simulation environment for an energy storage power station, wherein the digital twin simulation environment includes a state space S, an action space A, a state transition probability P, and an instantaneous reward function R;

[0077] S2012, Construct the agent for the reinforcement learning model, wherein the agent includes a policy network π_θ and a value network Q_ Where θ represents the parameters of the policy network to be trained. The parameters of the value network to be trained are: the input of the policy network π_θ is state s, and the output of the policy network π_θ is action a or the probability distribution of action a.

[0078] S2013, The agent is trained through deep reinforcement learning to obtain the reinforcement learning model.

[0079] In reinforcement learning (RL), the agent is an intelligent entity that learns optimal policies through interaction with its environment. The agent is the core engine of the reinforcement learning model's adaptive decision-making; it is the entity that makes decisions and executes actions. A digital twin simulation environment (or simply simulation environment) serves as a training and testing platform for reinforcement learning agents, providing a secure and low-cost training ground.

[0080] The state space S includes multiple parameters such as battery state of charge (SOC), charging power (P_char), discharging power (P_dis), cell temperature (T_cell), battery health status (SOH), grid electricity price (Price_t), grid load demand (Load_t, t), and timestamp.

[0081] The state space S is a combination of multidimensional continuous and discrete variables. For example, s_t = (SOC, P_char, P_dis, T_cell, SOH, Price_t, Load_t, t). Here, SOC ∈ [0, 1] represents the battery's state of charge; P_char represents the adjustable charging power; P_dis represents the adjustable discharging power, the range of which is determined by the device's rated values; T_cell represents the cell temperature, with a safe range; SOH ∈ [0, 1] represents the battery's health status; Price_t represents the real-time electricity price; Load_t represents the load demand; and t represents the time of day.

[0082] Action space A includes at least one of start / stop commands, charge / discharge power commands, net power commands, and maintenance recommendations. For example, action space A is: a_t = (P_cmd, Maintenance_Flag), where P_cmd is the net power command (positive for discharging, negative for charging), and its absolute value must be within the device power range; Maintenance_Flag is a binary variable indicating whether a maintenance check is recommended.

[0083] The program defines a state space S, an action space A, a state transition model P, and a reward function R. S includes battery state of charge, charge / discharge power, temperature, health status, grid electricity price, and load demand; A includes charge / discharge power commands, start / stop commands, and maintenance suggestions; R is a quantified function that integrates economic benefits, equipment losses, and safety constraints.

[0084] The reward function R acts as a guiding principle for the reinforcement learning model, directing the agent towards maximizing returns. In this embodiment, the reward function R is an immediate reward function, a quantified function that integrates economic benefits, equipment wear and tear, and safety constraints. Specifically, the reward function R is a weighted sum of multiple energy storage power station operating indicators, reflecting multi-objective optimization:

[0085] r_t=w1×R_revenue+w2×R_health+w3×R_penalty;

[0086] Where R_revenue = Price_t × P_cmd × Δt, R_revenue represents the electricity revenue during the time period Δt (positive for discharging, negative for charging). R_health = -β × |P_cmd| / (k × SOH) represents the penalty for battery degradation, where β is the degradation coefficient and k is a constant related to the battery type; the more drastic the action and the lower the battery health, the greater the penalty. R_penalty is the penalty for violating safety constraints, for example: if SOC < SOC_min or SOC > SOC_max: R_penalty = C_large_negative; if T_cell > T_max: R_penalty = C_large_negative, where C_large_negative is a very large negative value.

[0087] The MDP model constructed in this embodiment accurately quantifies the decision-making problem of energy storage operation and maintenance. The reward function unifies multiple operation and maintenance objectives such as economy, lifespan, and safety into a scalar that can be maximized, guiding the reinforcement learning agent to learn the comprehensive optimal strategy.

[0088] Construct an agent for the reinforcement learning model, wherein the agent includes a policy network π_θ and a value network Q_θ. Where θ represents the parameters of the policy network to be trained. The parameters of the value network to be trained are given, the input of the policy network π_θ is state s, and the output of the policy network π_θ is action a or the probability distribution of action a.

[0089] In step S2012, an agent for the reinforcement learning model is constructed by initializing a deep reinforcement learning model. For example, the agent can be constructed based on Deep Deterministic Policy Gradient (DDPG) or Proximal Policy Optimization (PPO) algorithms. This agent comprises a policy network π_θ and a value network Q_θ. Where θ represents the parameters of the policy network to be trained. The parameters are those of the value network to be trained. The receiving state s of the policy network π_θ is given by the policy network π_θ, which outputs a deterministic action a or a probability distribution of action a.

[0090] After the agent is built, the agent is trained by deep reinforcement learning in step S2013 to obtain a reinforcement learning model. The optimal operation and maintenance decision can be obtained through the reinforcement learning model, and the final operation and maintenance action output can be generated.

[0091] Step S202: Construct an operation and maintenance knowledge base, wherein the operation and maintenance knowledge base includes an experience pool and a strategy base.

[0092] The constructed operation and maintenance knowledge base is a two-layer structure. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step.

[0093] In practice, decision experience tuples can be stored in an scalable database table or memory buffer, and the experience pool supports efficient insertion, sampling, and eviction (such as first-in, first-out).

[0094] The policy library is used to store high-order policy knowledge abstracted from experience. Each policy knowledge is represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the policy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs.

[0095] In practice, clustering algorithms (such as density-based DBSCAN) can be used to summarize the state-action pairs in the experience pool to form representative "policy patterns" and obtain policy knowledge.

[0096] The two-tiered operation and maintenance knowledge base can be used for experience accumulation and reuse. Its structured storage method is different from the simple experience replay buffer and supports more complex relation queries and semantic matching.

[0097] Step S203: Obtain the operating status data of the energy storage power station.

[0098] The energy storage operation and maintenance system can obtain the operating status data of the energy storage power station monitoring system in real time through the data interface, forming the raw status observation data o_t.

[0099] Step S204: Extract features from the running status data to obtain target status features.

[0100] The energy storage operation and maintenance system preprocesses the raw state observation data o_t (such as normalization and filtering) and extracts features, and standardizes the extracted features to generate standard state features (target state features) s_t that can be used by the reinforcement learning model and the operation and maintenance knowledge base.

[0101] Step S205: Based on the target state features, perform a similarity search in the strategy library to obtain the search results.

[0102] The energy storage operation and maintenance system performs a similarity search in the policy library based on the current target state feature s_t, determines whether there is a state feature in the policy library that matches the target state feature, and obtains the search results.

[0103] Step S206: Based on the retrieval results and preset state evaluation rules, determine the generation method of operation and maintenance actions, wherein the generation method includes generating operation and maintenance actions based on the reinforcement learning model and generating operation and maintenance actions based on the operation and maintenance knowledge base.

[0104] After obtaining the search results through similarity retrieval in the strategy library, the energy storage operation and maintenance system makes operation and maintenance decisions based on the search results and preset state assessment rules, and determines the generation method of operation and maintenance actions.

[0105] Step S207: Based on the generation method, generate the operation and maintenance actions of the energy storage power station.

[0106] Once the generation method for operation and maintenance actions is determined, the energy storage operation and maintenance system generates the final operation and maintenance actions based on that method.

[0107] The energy storage operation and maintenance method based on reinforcement learning provided in this application embodiment constructs a reinforcement learning model for the operation and maintenance of energy storage power stations. The input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions. An operation and maintenance knowledge base is constructed, comprising an experience pool and a strategy base. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step. The strategy base stores strategy knowledge, each strategy knowledge item represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the strategy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs. The method also acquires information about the energy storage power station. The system generates operation and maintenance (O&M) actions based on the following methods: 1) Operational status data; 2) Feature extraction of the operational status data to obtain target status features; 3) Similarity retrieval in the policy library based on the target status features to obtain retrieval results; 4) Determining the generation method of O&M actions based on the retrieval results and preset status evaluation rules, wherein the generation method includes generating O&M actions based on the reinforcement learning model and generating O&M actions based on the O&M knowledge base; 5) Generating O&M actions for the energy storage power station based on the generation method, which can generate corresponding O&M actions through policy library matching and reinforcement learning, combined with case-based reasoning (O&M knowledge base) and reinforcement learning model-driven reasoning, thereby improving the real-time performance, interpretability, and security of O&M decisions for the energy storage power station and ensuring the safe and reliable operation of the energy storage power station; 6) The online update mechanism of the knowledge base enables the system to continuously learn from actual operation, transforming "specific experience" into "general strategies," realizing the dynamic evolution of knowledge and continuous optimization of energy storage O&M.

[0108] The reinforcement learning-based energy storage operation and maintenance method provided in this application is applicable to battery energy storage systems (BESS) in scenarios such as grid peak shaving, microgrids, and industrial and commercial energy storage. It can intelligently diagnose complex and variable fault scenarios, recommend strategies, and manage knowledge self-evolution, thereby improving the operational safety and economy of energy storage power stations and extending their service life.

[0109] As an optional embodiment, such as Figure 3 As shown, in step S205, the following steps can be implemented: based on the target state features, a similarity search is performed in the strategy library to obtain search results, including:

[0110] S2051, based on the scene feature vector of the target state features, perform a similarity search in the policy library; wherein, if the similarity between the scene feature vector and the target scene feature vector in the policy library is greater than a preset similarity threshold, the similarity match is determined to be successful; otherwise, the similarity match is determined to be unsuccessful.

[0111] S2052, if the search result is a successful similarity match, obtain the corresponding action template and the strategy effect evaluation value from the strategy library.

[0112] The scene feature vector f_t is a subset or mapping of the state s. The scene feature vector f_t includes key dimensions that affect the decision. For example, in this embodiment, the scene feature vector f_t has 4 dimensions, and its constituent elements are (SOC, Price_t, Load_t, SOH), where SOC ∈ [0, 1], Price_t ∈ [0, 1] (normalized electricity price), Load_t ∈ [0, 1] (normalized load), and SOH ∈ [0, 1]. The other state dimensions do not participate in the feature vector construction to reduce dimensional redundancy and focus on key decision variables.

[0113] In this embodiment, cosine similarity is used for similarity calculation: sim(f_t, f_i) = (f_t·f_i) / (||f_t|| The value range of ||f_i|| is [-1, 1]. Since all the features are positive, the actual range is [0, 1].

[0114] By performing a similarity search in the policy library, the most similar historical policy to the current scenario can be quickly found. Cosine similarity measures the directional consistency of feature vectors and is insensitive to the absolute magnitude of the value, making it more suitable for comparing operating scenarios of different magnitudes but with similar patterns.

[0115] As an optional embodiment, such as Figure 4As shown, in step S206, determining the generation method of operation and maintenance actions based on the search results and preset status assessment rules includes:

[0116] S2061, if the search result is a successful similarity match and the strategy effect evaluation value is greater than the preset effect evaluation threshold, determine that the operation and maintenance actions of the energy storage power station are generated based on the action template;

[0117] S2062, if the search result is a similarity matching failure, or if the search result is a similarity matching success, and the strategy effect evaluation value is less than or equal to the preset effect evaluation threshold, determine that the operation and maintenance actions of the energy storage power station are generated based on the strategy network of the reinforcement learning model.

[0118] If sim(f_t, f_i) > σ (σ is a preset similarity threshold), a successful match is determined, and the corresponding action template a_template_i and the policy effect evaluation Q value are obtained. The system determines the operation and maintenance decision based on the search results and preset state evaluation rules. If the match is successful and Q_i > Q_threshold (effect evaluation threshold), the recommended action a_kb = a_template_i from the knowledge base is directly adopted. If the match fails, or if the match is successful but the effect evaluation is poor, the current reinforcement learning policy network is called to generate the action a_rl = π_θ(s_t). The energy storage operation and maintenance system finally outputs the adopted current action a_t.

[0119] As can be seen from the above, this application implements hybrid decision-making, giving priority to the use of the policy library rather than solely relying on the current RL model to determine the generation method of operation and maintenance actions.

[0120] As an optional embodiment, after generating the operation and maintenance actions of the energy storage power station based on the generation method in step S207, the method further includes:

[0121] S301, the generated current action a_t is sent to the energy management system of the energy storage power station for execution, and the next moment state s_{t+1} and instant reward r_t of the energy storage power station are monitored;

[0122] S302, the generated decision experience tuple (s_t, a_t, r_t, s_{t+1}) is stored in the experience pool;

[0123] S303, periodically cluster the decision-making experience in the experience pool, update or generate the strategy knowledge in the strategy library, and evaluate and update the strategy effect of the strategy knowledge in the strategy library.

[0124] The energy storage operation and maintenance system (O&M system) issues the current action a_t determined after the decision to the energy management system (EMS) of the energy storage power station for execution, and continuously monitors the state s_{t+1} and immediate reward r_t at the next moment. The O&M system stores the generated decision experience tuple (s_t, a_t, r_t, s_{t+1}) in an experience pool. Simultaneously, an asynchronous knowledge extraction process is initiated, updating or generating policy knowledge in the policy library through periodic strategy clustering. Specifically, every N new experiences accumulated, the program performs cluster analysis on the state-action pairs of new and old experiences in the experience pool, updating or generating new policy patterns and storing them in the policy library. The O&M system can also evaluate and update the policy effectiveness of the policy knowledge in the policy library.

[0125] As an optional embodiment, in step S303, the evaluation and updating of the policy effectiveness of the policy knowledge in the policy base includes:

[0126] S3031, for the target strategies that have been used in the strategy library, update the strategy effect evaluation value and the number of times the target strategy is applied based on the actual reward for new experience derived from the target strategy.

[0127] For strategies already used in the strategy library, the energy storage operation and maintenance system updates its Q value (strategy effectiveness evaluation value) and the number of times it is applied, n, based on the actual reward for the new experience derived from it. The Q value can be updated using a moving average, for example: Q_new = (1-α)×Q_old + α×R_new.

[0128] As an optional embodiment, the method further includes:

[0129] S401, Sample the experience pool to obtain target experience data;

[0130] S402, based on the target experience data, train the policy network π_θ and the value network Q_θ offline. Update the policy network parameters θ of the policy network π_θ and the value network Q_ Value network parameters The policy network π_θ updates its parameters θ by minimizing the temporal difference error, and the value network Q_ The value network parameters are updated by maximizing the expectation of the value function. ;

[0131] S403, based on the policy network parameters θ and the value network parameters Update the reinforcement learning model;

[0132] S404, Test the updated reinforcement learning model in the digital twin simulation environment and evaluate the average cumulative reward of the reinforcement learning model;

[0133] S405, Evaluate the reinforcement learning model based on the average cumulative reward.

[0134] The energy storage operation and maintenance system performs batch experience sampling, periodically (e.g., daily at dawn) randomly sampling a batch of target experience data B = {e_j} from the experience pool for training the reinforcement learning model. The system uses the sampled target experience data B to update the network parameters through the objective function of the reinforcement learning algorithm (taking DDPG as an example).

[0135] Value Network Q_ Update by minimizing the timing difference error (TD-error): L( = E_(s,a,r,s')~B [ ( Q_ (s, a) - (r + γ×Q_ '(s', π_θ'(s')) ) )^2 ]where γ is the discount factor, Q_ ' and π_θ' are the target network parameters, which are periodically soft-updated from the current network. The policy network π_θ is updated by maximizing the expectation of the value function: _θ J(θ) ≈ E_(s)~B [ _a Q_ (s, a)|_(a=π_θ(s))× _θ π_θ(s) ].

[0136] Subsequently, the energy storage operation and maintenance system conducts multiple rounds of testing on the updated model in a simulation environment to evaluate its average cumulative reward. Based on the average cumulative reward, the reinforcement learning model is evaluated and deployed. If the performance exceeds that of the current online model by a certain percentage, the new model parameters are deployed as a new policy network π_θ for online decision-making.

[0137] As can be seen from the above, in this embodiment, the continuous optimization of the operation and maintenance decision model (including the operation and maintenance knowledge base and the reinforcement learning model) is achieved through the aforementioned steps S301 to S303 and S401 to S405. Offline training utilizes the massive amount of real-world operational experience accumulated in the knowledge base, avoiding the risks that online exploration may bring, which belongs to the offline reinforcement learning (Offline RL) paradigm. This enables the system to fully utilize historical data, continuously refine better decision-making strategies, and feed them back into the online decision-making process for generating operation and maintenance actions, forming a reinforced closed loop of "decision-experience-learning-optimization".

[0138] This application establishes a closed-loop knowledge evolution system of "perception-decision-feedback-learning-optimization" through the continuous operation of the energy storage operation and maintenance system. Through the continuously evolving operation and maintenance knowledge base and the constantly optimized reinforcement learning decision model, the mean time to repair (MTTR) is gradually reduced and the battery life maintenance rate is gradually improved, thereby enhancing the effectiveness and efficiency of pure energy operation and maintenance.

[0139] As an optional embodiment, the reinforcement learning-based energy storage operation and maintenance method can also be implemented according to the following steps: The method further includes:

[0140] S501, Visualize the decision suggestions and / or decision basis for the operation and maintenance actions;

[0141] If the operation and maintenance action is generated based on the operation and maintenance knowledge base, then the strategy record matching the operation and maintenance action and the historical effect of the strategy record are displayed; if the operation and maintenance action is generated based on the reinforcement learning model, then the key state features of the reinforcement learning model's decision are visualized.

[0142] Operations and maintenance personnel can input a hypothetical or historical operating state through the graphical interface of the energy storage operations and maintenance system. The graphical interface first displays the system's decision suggestions for that state (the aforementioned operations and maintenance actions a_rl or a_kb), and simultaneously provides the decision basis: if the operations and maintenance actions come from the operations and maintenance knowledge base, it displays the matching strategy records and their historical effects; if the operations and maintenance actions come from the RL model, it can use attention mechanisms and other methods to visualize the key state characteristics of the model's decision.

[0143] In this embodiment, the operation and maintenance actions and corresponding decision-making basis can be displayed in real time through human-computer interaction, which facilitates energy storage operation and maintenance.

[0144] As an optional embodiment, the method further includes:

[0145] S601, Based on the preset target operation and maintenance actions, the action sequence is obtained by deducing in the digital twin simulation environment;

[0146] S602, predict the impact of the action sequence on the operating indicators of the energy storage power station within a preset time period in the future, wherein the operating indicators of the energy storage power station include at least one of battery state of charge, battery life degradation and power station revenue curve.

[0147] Operations and maintenance personnel can modify the maintenance actions suggested by the energy storage operation and maintenance system, or specify their own actions, and then conduct multi-step simulations in the simulation environment. The energy storage operation and maintenance system will predict and display the impact of different action sequences on key indicators such as battery SOC, lifespan degradation, and yield curves over a period of time in the future.

[0148] As an optional embodiment, the method further includes:

[0149] S701, the target operation and maintenance decision confirmed by simulation based on the preset target operation and maintenance action is marked as expert experience;

[0150] S702, store the expert experience in the operation and maintenance knowledge base, and configure the corresponding confidence weight for the expert experience.

[0151] Operations and maintenance personnel can manually mark excellent decision-making solutions confirmed in simulations as "expert experience" and store them in the operations and maintenance knowledge base. Since this expert experience is a credible and reliable excellent operations and maintenance decision-making solution, the operations and maintenance system can assign a high initial confidence weight to this expert experience, enabling it to quickly influence subsequent decisions and improve the efficiency of operations and maintenance decision-making.

[0152] As shown above, this application provides an interpretable intervention window for the "black box" RL model through human-computer interaction and simulation, enhancing the trust of operation and maintenance personnel. The simulation function is equivalent to a decision-making sandbox, allowing for "hypothesis analysis." Furthermore, this application achieves the integration of human domain knowledge and AI data-driven knowledge through an expert experience injection mechanism, which is a crucial guarantee for the system to achieve rapid cold start-up and error correction of energy storage power stations.

[0153] By manually injecting expert experience and conducting human-computer interaction simulations, a closed-loop knowledge system of "perception-decision-feedback-learning-evolution" is constructed. This overcomes the problems of low matching accuracy and inability to adaptively evolve in traditional static knowledge bases, significantly reducing fault handling time (MTTR), realizing the automated accumulation and continuous evolution of operation and maintenance experience, and improving the safety, economy, and lifespan of energy storage systems.

[0154] The following detailed description of the reinforcement learning-based energy storage operation and maintenance method provided in this application will be based on specific embodiments.

[0155] Example 1: System Deployment and Initialization

[0156] This energy storage operation and maintenance system is deployed in a 100MW / 200MWh energy storage power station. Real-time data from the BMS, EMS, and PCS are collected to construct a digital twin model. The state space S is defined as a 12-dimensional vector, containing SOC, temperature curve, voltage range, power commands, etc. The action space A is P_cmd ∈ [-50, 50] MW, and Maintenance_Flag ∈ {0,1}.

[0157] The weights in the reward function are set as follows: w1=0.5 (economic efficiency), w2=0.3 (lifetime), and w3=0.2 (safety). The reinforcement learning agent is initialized using the PPO algorithm, and the policy network is a three-layer fully connected neural network (256→128→64). The input is a standardized state vector, and the output is continuous actions and classification labels.

[0158] Knowledge base initialization: Import 2000 historical work orders (text format) and BMS logs, extract key features through NLP processing, and build an initial strategy library of 50 items.

[0159] Example 2: Online Decision-Making Process for Operation and Maintenance Actions

[0160] On [Date] at 14:15, the energy storage operation and maintenance system monitored the status s_t = (SOC=0.92, Price_t=0.85 yuan / kWh, Load_t=85MW, SOH=0.78, T_cell_max=42.1℃).

[0161] (1) Feature extraction: Generate f_t = (0.92, 0.85, 85, 0.78);

[0162] (2) Policy library matching: Calculate the cosine similarity with 50 policies in the policy library. Policy #23 is found to be f_i=(0.90, 0.82, 87, 0.80), sim=0.96>σ=0.9, which is a successful match. Q=0.87>Q_threshold=0.8;

[0163] (3) Decision determination: Adopt the action corresponding to strategy #23 a_kb={P_cmd=-15MW,Maintenance_Flag=1}, that is, it is recommended to discharge 15MW and start maintenance inspection;

[0164] (4) Execution and monitoring: The operation and maintenance actions were sent to EMS. The system was monitored for 30 minutes. The SOC dropped to 0.89, the temperature stabilized, and no safety alarm was triggered. The reward r_t = +120 yuan (profit) - 8.3 (loss) = +111.7 yuan;

[0165] (5) Experience storage: The decision experience tuple (s_t, a_t, r_t, s_{t+1}) is stored in the experience pool;

[0166] (6) Update the operation and maintenance knowledge base: Asynchronous task starts cluster analysis, classifies the experience into strategy #23, and updates Q_new = 0.87×0.9 + 111.7 / 120×0.1 = 0.88, n=127.

[0167] Example 3: Offline Training and Optimization of Reinforcement Learning Models

[0168] Every day at 2:00 AM, the energy storage operation and maintenance system randomly samples 10,000 experience points from the experience pool for offline PPO training. The value network updates with the goal of minimizing TD-error, while the policy network maximizes the expected reward through policy gradients. After 5 rounds of training, the new policy achieves an average reward increase of 18.7% in the simulation environment and is then deployed as an online policy network.

[0169] Example 4: Injection of Expert Experience

[0170] Senior engineer A discovered in a simulation that, under the scenario of "SOC > 0.9 + temperature difference > 3℃", adopting a strategy of "pausing charging + forced balancing for 10 minutes" can significantly delay battery consistency degradation. He manually marked this solution as "expert experience," and the energy storage operation and maintenance system automatically created a new strategy: f = (0.92, 0.75, 80, 0.75), a_template = "pause charging + balancing for 10 minutes", Q = 0.95, n = 1. This strategy was recommended by the system 3 times the following day, achieving a 100% adoption rate.

[0171] In summary, the embodiments of this application achieve intelligent generation and continuous evolution of operation and maintenance knowledge through a dual closed-loop architecture of "online hybrid decision-making + offline knowledge evolution": On the one hand, operation and maintenance decisions are modeled as Markov decision processes (MDPs), and a reward function is constructed by combining real-time operating status (such as SOC, temperature rise rate, etc.) and operation and maintenance feedback (adoption rate, troubleshooting time) to train reinforcement learning agents to dynamically generate optimal strategies; on the other hand, through a two-layer knowledge structure of "experience pool + strategy library", clustering algorithms are used to abstract original experience into searchable high-order strategy patterns, and the automatic evaluation and updating of strategy weights are achieved through the Actor-Critic architecture, enabling the system to automatically extract, verify and evolve knowledge from actual operation and maintenance data, breaking through the inherent limitations of traditional static knowledge bases, and realizing the automated accumulation, adaptive optimization and cross-scenario reuse of operation and maintenance experience.

[0172] In related technologies, reinforcement learning controllers or rule-based expert systems are typically used alone. This application adopts a dual-loop architecture of "online hybrid decision-making (knowledge base matching + RL inference) + offline experience extraction and model training". In the online phase, the structured knowledge base is queried first, ensuring the efficiency and reliability of the decision-making; in the offline phase, the sparse experience generated online is transformed into dense training data and abstracted into policy knowledge, which is then fed back into the operation and maintenance knowledge base and RL model. This closed-loop process of "perception-decision-storage-learning" is the core logic of this application.

[0173] Existing reinforcement learning systems' experience replay buffers only store raw experience tuples. This application, by setting up a two-layer knowledge base structure, separates the management of raw experience (experience pool) and abstract policies (policy base), providing unique data processing and knowledge representation. The "policy patterns" in the policy base are a high-level knowledge representation that inductively summarizes discrete experiences through clustering, resulting in better generalization and retrieval of knowledge. This is a key technical means to achieve experience reuse across scenarios and devices.

[0174] In reinforcement learning training, this application primarily employs offline reinforcement learning (Offline RL). Compared to traditional online reinforcement learning (Online RL), which requires interaction with an online environment, offline reinforcement learning learns directly from the experience pool of the operations and maintenance knowledge base. This effectively meets the extremely high requirements of energy storage operations and maintenance for safety and stability, avoiding suboptimal or dangerous actions caused by exploration in the early stages of training. Simultaneously, utilizing high-quality historical data and expert-injected experience from the operations and maintenance knowledge base significantly improves the quality and diversity of training data, alleviates distribution bias issues, and makes training more efficient and stable.

[0175] Compared with existing technologies, the energy storage operation and maintenance method based on reinforcement learning provided in this application has the following beneficial effects:

[0176] (1) Significantly reduced troubleshooting time (MTTR): It eliminates the traditional reliance on rote memorization or keyword search. The system can directly push the best troubleshooting steps that have been verified in practice based on the underlying data characteristics and alarm context of the BMS, thereby significantly improving the availability of energy storage power stations and reducing economic losses and safety risks caused by long-term downtime.

[0177] (2) Achieve automated inheritance and evolution of operation and maintenance experience: Newly hired operation and maintenance personnel can receive guidance equivalent to that of senior experts. As the power station's operating years increase (battery aging characteristics appear), the system automatically discards inapplicable old rules through a reinforcement learning reward mechanism, discovers and strengthens new rules that are suitable for the current battery state, and realizes that the knowledge base "gets smarter the more it is used".

[0178] (3) Provide data-driven in-depth diagnostic support: By integrating BMS time series data features with text knowledge, the system can identify hidden fault modes that are difficult to detect by traditional manual methods (such as deep consistency problems caused by small pressure differences), thereby playing a synergistic role in early warning and accurate positioning.

[0179] According to an embodiment of this application, a reinforcement learning-based energy storage operation and maintenance device is also provided for implementing the above-described reinforcement learning-based energy storage operation and maintenance method. Figure 5 This is a structural block diagram of an energy storage operation and maintenance device based on reinforcement learning, provided in the embodiments of this application, as follows: Figure 5As shown, the reinforcement learning-based energy storage operation and maintenance device includes:

[0180] The first construction module 10 is configured to construct a reinforcement learning model for the operation and maintenance of an energy storage power station. The input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions.

[0181] The second construction module 20 is configured to build an operation and maintenance knowledge base, which includes an experience pool and a strategy library. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step. The strategy library stores strategy knowledge, and each piece of strategy knowledge is represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the strategy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs.

[0182] The acquisition module 30 is configured to acquire the operating status data of the energy storage power station;

[0183] Feature extraction module 40 is configured to extract features from the running state data to obtain target state features;

[0184] The similarity retrieval module 50 is configured to perform a similarity retrieval in the strategy library based on the target state features to obtain retrieval results;

[0185] The generation method determination module 60 is configured to determine the generation method of operation and maintenance actions based on the search results and preset state evaluation rules, wherein the generation method includes generating operation and maintenance actions based on the reinforcement learning model and generating operation and maintenance actions based on the operation and maintenance knowledge base;

[0186] The operation and maintenance action generation module 70 is configured to generate operation and maintenance actions for the energy storage power station based on the generation method.

[0187] Optionally, the first building module 10 is also configured as follows:

[0188] A digital twin simulation environment for an energy storage power station is constructed, comprising a state space S, an action space A, a state transition probability P, and a reward function R. The state space S includes multiple parameters such as battery state of charge, charging power, discharging power, cell temperature, battery health status, grid electricity price, grid load demand, and timestamps. The action space includes at least one of start / stop commands, charging / discharging power commands, net power commands, and maintenance suggestions. The reward function R is a quantified function that integrates economic benefits, equipment losses, and safety constraints.

[0189] Construct an agent for the reinforcement learning model, wherein the agent includes a policy network π_θ and a value network Q_θ. Where θ represents the parameters of the policy network to be trained. The parameters of the value network to be trained are: the input of the policy network π_θ is state s, and the output of the policy network π_θ is action a or the probability distribution of action a.

[0190] The agent is trained using deep reinforcement learning to obtain the reinforcement learning model.

[0191] Optionally, the similarity retrieval module 50 is also configured as follows:

[0192] Based on the scene feature vector of the target state features, a similarity search is performed in the policy library; wherein, if the similarity between the scene feature vector and the target scene feature vector in the policy library is greater than a preset similarity threshold, the similarity match is determined to be successful; otherwise, the similarity match is determined to be unsuccessful.

[0193] If the search result indicates a successful similarity match, the corresponding action template and the strategy effect evaluation value are retrieved from the strategy library.

[0194] Optionally, the generation method determination module 60 is also configured as follows:

[0195] If the search result is a successful similarity match and the strategy effect evaluation value is greater than the preset effect evaluation threshold, it is determined that the operation and maintenance actions of the energy storage power station are generated based on the action template.

[0196] If the search result is a similarity match failure, or if the search result is a similarity match success and the strategy effect evaluation value is less than or equal to the preset effect evaluation threshold, it is determined that the operation and maintenance actions of the energy storage power station are generated based on the strategy network of the reinforcement learning model.

[0197] Optionally, the reinforcement learning-based energy storage operation and maintenance device also includes an evaluation module, configured to, after generating the operation and maintenance actions of the energy storage power station based on the generation method,

[0198] The generated current action a_t is sent to the energy management system of the energy storage power station for execution, and the next moment state s_{t+1} and instant reward r_t of the energy storage power station are monitored;

[0199] The generated decision experience tuple (s_t, a_t, r_t, s_{t+1}) is stored in the experience pool;

[0200] Periodically cluster the decision-making experiences in the experience pool to update or generate policy knowledge in the policy library, and evaluate and update the policy effects of the policy knowledge in the policy library.

[0201] Optionally, the evaluation module is further configured to:

[0202] For target strategies already used in the strategy library, update the strategy effectiveness evaluation value and the number of times the target strategy is applied based on the actual reward for new experiences derived from the target strategy.

[0203] Optionally, the evaluation module is further configured to:

[0204] The experience pool is sampled to obtain target experience data;

[0205] Based on the target experience data, the policy network π_θ and the value network Q_θ are trained offline. Update the policy network parameters θ of the policy network π_θ and the value network Q_ Value network parameters The policy network π_θ updates its parameters θ by minimizing the temporal difference error, and the value network Q_ The value network parameters are updated by maximizing the expectation of the value function. ;

[0206] The reinforcement learning model is updated based on the policy network parameters θ and the value network parameters Φ;

[0207] The updated reinforcement learning model is tested in the digital twin simulation environment to evaluate the average cumulative reward of the reinforcement learning model.

[0208] The reinforcement learning model is evaluated based on the average cumulative reward.

[0209] Optionally, the reinforcement learning-based energy storage operation and maintenance device also includes a visualization module, configured as follows:

[0210] The decision suggestions and / or decision basis for the operation and maintenance actions are visualized; if the operation and maintenance actions are generated based on the operation and maintenance knowledge base, the strategy records matching the operation and maintenance actions and the historical effects of the strategy records are displayed; if the operation and maintenance actions are generated based on the reinforcement learning model, the key state features of the reinforcement learning model's decision are visualized.

[0211] Optionally, the reinforcement learning-based energy storage operation and maintenance device also includes a deduction module, configured as follows:

[0212] Based on the preset target operation and maintenance actions, the action sequence is obtained by deducing in the digital twin simulation environment;

[0213] Predicting the impact of the action sequence on the operating indicators of the energy storage power station within a predetermined future timeframe, wherein the operating indicators of the energy storage power station include at least one of battery state of charge, battery life degradation, and power station revenue curve; and / or

[0214] The energy storage operation and maintenance device based on reinforcement learning also includes an expert experience determination module, configured as follows:

[0215] The target operation and maintenance decisions confirmed through simulation and deduction based on the preset target operation and maintenance actions are marked as expert experience.

[0216] The expert experience is stored in the operation and maintenance knowledge base, and a corresponding confidence weight is configured for the expert experience.

[0217] It should be noted that the first construction module 10, the second construction module 20, the acquisition module 30, the feature extraction module 40, the similarity retrieval module 50, the generation method determination module 60, and the operation and maintenance action generation module 70 mentioned above correspond to steps S201 to S207 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in the embodiments.

[0218] Embodiments of this application may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0219] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the reinforcement learning-based energy storage operation and maintenance method and device in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned reinforcement learning-based energy storage operation and maintenance method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0220] The processor can access the information and application programs stored in the memory via a transmission device to execute the above-described steps.

[0221] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0222] Embodiments of this application also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the reinforcement learning-based energy storage operation and maintenance method provided in the above embodiments.

[0223] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0224] Embodiments of this application also provide a computer program product, including a computer program. Optionally, in this embodiment, the computer program can implement the above-described method when executed by a processor.

[0225] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0226] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0230] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0231] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for energy storage operation and maintenance based on reinforcement learning, characterized in that, include: A reinforcement learning model for the operation and maintenance of an energy storage power station is constructed, wherein the input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions. An operations and maintenance knowledge base is constructed, comprising an experience pool and a strategy base. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step. The strategy base stores strategy knowledge, each strategy knowledge being represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the strategy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs. Acquire operational status data of energy storage power stations; Feature extraction is performed on the operating status data to obtain the target status features; Based on the target state features, a similarity search is performed in the strategy library to obtain the search results; The generation method of operation and maintenance actions is determined based on the search results and the preset state evaluation rules, wherein the generation method includes generating operation and maintenance actions based on the reinforcement learning model and generating operation and maintenance actions based on the operation and maintenance knowledge base; Based on the aforementioned generation method, the operation and maintenance actions of the energy storage power station are generated.

2. The method of claim 1, wherein, The reinforcement learning model for the operation and maintenance of energy storage power stations includes: A digital twin simulation environment for an energy storage power station is constructed, comprising a state space S, an action space A, a state transition probability P, and a reward function R. The state space S includes multiple parameters such as battery state of charge, charging power, discharging power, cell temperature, battery health status, grid electricity price, grid load demand, and timestamps. The action space includes at least one of start / stop commands, charging / discharging power commands, net power commands, and maintenance suggestions. The reward function R is a quantified function that integrates economic benefits, equipment losses, and safety constraints. constructing an agent of the reinforcement learning model, wherein the agent comprises a policy network π θ and a value network Q wherein θ is a policy network parameter to be trained, is a value network parameter to be trained, an input of the policy network π θ is a state s, and an output of the policy network π θ is an action a or a probability distribution of the action a; The agent is trained using deep reinforcement learning to obtain the reinforcement learning model.

3. The method of claim 1, wherein, The step of performing a similarity search in the strategy library based on the target state features to obtain search results includes: Based on the scene feature vector of the target state features, a similarity search is performed in the policy library; wherein, if the similarity between the scene feature vector and the target scene feature vector in the policy library is greater than a preset similarity threshold, the similarity match is determined to be successful; otherwise, the similarity match is determined to be unsuccessful. If the search result indicates a successful similarity match, the corresponding action template and the strategy effect evaluation value are retrieved from the strategy library.

4. The method of claim 3, wherein, The method for determining the generation of operation and maintenance actions based on the search results and preset status assessment rules includes: If the search result is a successful similarity match and the strategy effect evaluation value is greater than the preset effect evaluation threshold, it is determined that the operation and maintenance actions of the energy storage power station are generated based on the action template. If the search result is a similarity match failure, or if the search result is a similarity match success and the strategy effect evaluation value is less than or equal to the preset effect evaluation threshold, it is determined that the operation and maintenance actions of the energy storage power station are generated based on the strategy network of the reinforcement learning model.

5. The method of claim 1, wherein, After generating the operation and maintenance actions of the energy storage power station based on the aforementioned generation method, the method further includes: The generated current action a_t is sent to the energy management system of the energy storage power station for execution, and the next moment state s_{t+1} and instant reward r_t of the energy storage power station are monitored; The generated decision experience tuple (s_t, a_t, r_t, s_{t+1}) is stored in the experience pool; Periodically cluster the decision-making experiences in the experience pool to update or generate policy knowledge in the policy library, and evaluate and update the policy effects of the policy knowledge in the policy library.

6. The method of claim 5, wherein, The evaluation and updating of the policy effectiveness of the policy knowledge in the policy base includes: For target strategies already used in the strategy library, update the strategy effectiveness evaluation value and the number of times the target strategy is applied based on the actual reward for new experiences derived from the target strategy.

7. The method of claim 2, wherein, The method further includes: The experience pool is sampled to obtain target experience data; training the policy network and the value network offline based on the target experience data , updating policy network parameters of the policy network and value network parameters of the value network wherein the policy network updates the policy network parameters by minimizing a temporal difference error, and the value network updates the value network parameters by maximizing an expectation of a value function ;​​ based on the policy network parameters θ and the value network parameters updating the reinforcement learning model; The updated reinforcement learning model is tested in the digital twin simulation environment to evaluate the average cumulative reward of the reinforcement learning model. The reinforcement learning model is evaluated based on the average cumulative reward.

8. The method of claim 3, wherein, The method further includes: The decision suggestions and / or decision basis for the operation and maintenance actions are visualized; if the operation and maintenance actions are generated based on the operation and maintenance knowledge base, the strategy records matching the operation and maintenance actions and the historical effects of the strategy records are displayed; if the operation and maintenance actions are generated based on the reinforcement learning model, the key state features of the reinforcement learning model's decision are visualized.

9. The method of claim 2, wherein, The method further includes: Based on the preset target operation and maintenance actions, the action sequence is obtained by deducing in the digital twin simulation environment; Predicting the impact of the action sequence on the operating indicators of the energy storage power station within a predetermined future timeframe, wherein the operating indicators of the energy storage power station include at least one of battery state of charge, battery life degradation, and power station revenue curve; and / or The method further includes: The target operation and maintenance decisions confirmed through simulation and deduction based on the preset target operation and maintenance actions are marked as expert experience. The expert experience is stored in the operation and maintenance knowledge base, and a corresponding confidence weight is configured for the expert experience.

10. An energy storage operation and maintenance device based on reinforcement learning, characterized in that, include: The first construction module is configured to construct a reinforcement learning model for the operation and maintenance of an energy storage power station. The input of the reinforcement learning model includes the operating status information of the energy storage power station, and the output of the reinforcement learning model includes operation and maintenance actions. The second construction module is configured to build an operation and maintenance knowledge base, which includes an experience pool and a strategy library. The experience pool stores decision experience tuples (s_t, a_t, r_t, s_{t+1}), where s_t is the current state, a_t is the current action, r_t is the immediate reward, s_{t+1} is the state at the next moment, and t is the time step. The strategy library stores strategy knowledge, and each piece of strategy knowledge is represented as a quintuple (f, a_template, Q, n, E_ids), where f is the scene feature vector, a_template is the action template, Q is the strategy effect evaluation value, n is the number of times it is applied, and E_ids is the set of source experience IDs. The acquisition module is configured to acquire the operating status data of the energy storage power station; The feature extraction module is configured to extract features from the running state data to obtain target state features; The similarity retrieval module is configured to perform a similarity retrieval in the strategy library based on the target state features to obtain retrieval results; The generation method determination module is configured to determine the generation method of operation and maintenance actions based on the search results and preset state evaluation rules, wherein the generation method includes generating operation and maintenance actions based on the reinforcement learning model and generating operation and maintenance actions based on the operation and maintenance knowledge base; The operation and maintenance action generation module is configured to generate operation and maintenance actions for the energy storage power station based on the aforementioned generation method.

Images