A reinforcement learning-based maintenance decision optimization method, device, and system for energy storage systems based on multi-component lifetime collaborative modeling.

Abstract

This invention discloses a method, device, and system for optimizing maintenance decisions in energy storage systems based on reinforcement learning-based multi-component lifetime collaborative modeling. The method constructs a collaborative lifetime decay model for key power devices (including IGBTs and DC bus capacitors) in the energy storage converter, collects operational data in real time, and establishes a multi-state fusion method for characterizing operational health. Furthermore, it utilizes reinforcement learning algorithms to optimize maintenance strategies. By jointly modeling the lifetime recovery effect, cost, failure risk, and system stability of different maintenance behaviors, a multi-objective reward function is constructed to achieve adaptive optimization of maintenance decisions. This invention can automatically determine the timing of overhaul or preventative maintenance based on the equipment's operating status and aging condition, eliminating the need for fixed-cycle manual maintenance, effectively reducing failure rates and maintenance costs. This invention is applicable to intelligent operation and maintenance and state management scenarios for energy storage converters.

Description

Technical Field

[0001] This invention belongs to the field of operation and maintenance control technology of energy storage power stations and their power electronic equipment. Specifically, it relates to a reinforcement learning maintenance decision optimization method, device and system for energy storage systems based on multi-component lifespan collaborative modeling. It is applicable to lifespan management and intelligent maintenance decision-making scenarios for key components including energy storage converters (PCS), DC bus capacitors and power semiconductor devices (such as IGBTs and MOSFETs). Background Technology

[0002] Energy storage systems (ESS) are widely used in scenarios such as grid frequency regulation, peak shaving, renewable energy grid integration, and backup power. Their core component, the power conversion system (PCS), undertakes key functions such as bidirectional energy conversion, power quality control, and system stability regulation. With the continuous expansion of application scale, the operational reliability and maintenance strategies of the PCS have become important factors affecting the total lifecycle cost of energy storage power stations.

[0003] Typical components in PCS that are most prone to aging and failure include: IGBT power modules are susceptible to failure modes such as solder layer fatigue, chip cracks, and gate oxidation due to factors such as switching losses, junction temperature cycling, and changes in parasitic parameters.

[0004] DC bus capacitors: Over time, phenomena such as electrolyte evaporation, increase in equivalent series resistance (ESR), and capacity decay occur. Excessive ripple current will accelerate the degradation.

[0005] These two types of devices not only exhibit significant aging characteristics, but their degradation behaviors are also coupled and mutually influential. For example, an increase in capacitor ESR leads to increased bus ripple, thereby increasing IGBT switching stress and accelerating aging; increased IGBT heat generation also raises capacitor temperature, accelerating its failure. However, current engineering maintenance strategies are mostly based on fixed-cycle maintenance or single-component lifespan thresholds, which are difficult to accurately reflect the true health status of the equipment.

[0006] To improve the lifetime prediction capability of power devices, some patents and technical solutions have proposed lifetime modeling methods for single components, but none of them have solved the problems of "multi-component lifetime collaborative modeling" and "intelligent maintenance strategy optimization".

[0007] Chinese patent document CN120855588A discloses a "reconfigurable modular intelligent battery system and control method," which achieves dynamic adjustment of the battery cell connection topology through a modular design that integrates a switch network and a control unit within the battery module. This solves the problems of low balancing efficiency and poor fault tolerance in traditional battery packs. However, it still relies on traditional static or complex excitation injection, making it difficult to obtain OCV in real time during operation, thus limiting its application effectiveness in dynamic balancing and state assessment.

[0008] Therefore, a new method is needed that can automatically collect relaxation fragments and extract robust features during RBS operation, and use machine learning methods (such as GPR) to predict OCV.

[0009] In summary, existing technologies mainly focus on single-component lifetime prediction, fixed threshold determination, or static maintenance modes, and generally suffer from the following problems: 1. Unable to simultaneously address the coordinated aging issues of critical components such as IGBTs and capacitors; 2. It is impossible to integrate lifespan models with maintenance actions, maintenance costs, and system risks into the optimization framework; 3. Lacks a maintenance strategy learning mechanism, and cannot autonomously adjust maintenance timing based on operational status; 4. Unable to generate dynamic maintenance strategies in real time based on the degree of component degradation; 5. Without the introduction of adaptive optimization techniques such as reinforcement learning, it is impossible to achieve a balance between optimal maintenance benefits and minimum failure risk.

[0010] To address the aforementioned issues, a new approach is needed to simultaneously establish a lifespan collaborative model for IGBTs and capacitors at the system level; to jointly optimize multiple indicators such as operational risk, lifespan recovery, and maintenance costs; to automatically generate the optimal maintenance strategy based on the system state using intelligent algorithms such as reinforcement learning; and to possess real-time performance and scalability, making it suitable for the long-term operation and maintenance of energy storage converters.

[0011] This invention aims to overcome the limitations of existing maintenance technologies and realize a unified system for multi-component collaborative health management and intelligent maintenance decision-making. Summary of the Invention

[0012] This invention aims to overcome the following shortcomings in existing energy storage system maintenance strategies: (1) Lack of modeling for the lifetime synergy between IGBTs and DC bus capacitors; (2) The maintenance strategy cannot be dynamically adjusted based on the real-time operating status; (3) Relying on fixed thresholds or periodic maintenance makes it impossible to balance maintenance costs and failure risks; (4) It is impossible to achieve multi-objective joint optimization based on lifespan, cost, risk and system stability.

[0013] To this end, this invention proposes a reinforcement learning-based maintenance decision optimization method, device, and system for energy storage systems based on multi-component lifetime collaborative modeling. By constructing a device lifetime model, a state representation model, a multi-objective reward model, and a reinforcement learning strategy model, adaptive and intelligent maintenance optimization of the energy storage converter operation process is achieved.

[0014] The technical solution of the present invention is as follows: On the one hand, this invention provides a reinforcement learning-based maintenance decision optimization method for energy storage systems based on multi-component lifetime collaborative modeling, characterized by the following steps: S1: Establish a lifetime coordination model for the coordinated degradation process of at least IGBT modules and DC bus capacitors in the energy storage converter; wherein, the lifetime coordination model includes independent lifetime decay sub-models established for IGBT power modules and DC bus capacitors respectively, and a lifetime coupling relationship sub-model for quantifying the mutually accelerated aging effect caused by thermal and electrical coupling between the two during operation; based on this model, the remaining lifetime value of each component and the corresponding dynamic failure probability at any time period can be calculated; S2: Based on the remaining lifetime value and dynamic fault probability output by the lifetime coordination model, and combined with the real-time collected system operating parameters, a multi-dimensional system state vector is constructed; the system state vector serves as a quantitative representation of the overall health status of the system, and includes at least the current IGBT remaining lifetime value, DC bus capacitor remaining lifetime value, corresponding dynamic fault probability value, and operating time since the last maintenance. S3: Based on the information contained in the system state vector, construct a multi-objective reward function; the design goal of the reward function is to balance maintenance costs and system risks, and its calculation items include at least: maintenance cost penalty items directly related to the maintenance actions performed, expected failure loss penalty items calculated based on the dynamic failure probability value, and positive benefit items generated by the component life recovery brought about by the maintenance actions; S4: Using the system state vector as input, the preset set of discretized maintenance actions as output space, and the multi-objective reward function as the evaluation criterion for the optimization objective, the reinforcement learning algorithm is used for offline or online policy training, and finally a trained reinforcement learning policy model that can map the optimal maintenance action according to the input state is obtained. S5: During the actual operation of the energy storage system, the following closed-loop process is executed periodically or triggered: First, the system state vector at the current moment is generated in real time according to steps S1 and S2; second, the system state vector is input into the reinforcement learning policy model obtained in step S4, and the optimal maintenance decision output by it is obtained and executed; finally, the internal state of the system is updated according to the effect of maintenance execution, and it serves as the starting point of the next decision cycle.

[0015] Furthermore, in step S1, the lifetime coupling relationship sub-model specifically quantifies the following physical processes: the increase in the equivalent series resistance (ESR) of the DC bus capacitor leads to an increase in the DC bus ripple current, which in turn increases the switching losses and junction temperature of the IGBT power module, accelerating its aging; at the same time, the power loss of the IGBT power module causes heat generation, which causes the ambient temperature of the DC bus capacitor to rise, thereby accelerating the drying of its electrolyte and parameter degradation.

[0016] Furthermore, in step S2, the system operating parameters include parameters obtained directly or indirectly from sensors or controllers, and the system state vector is further expanded by at least one of the operating stability indicators, ambient temperature, and load current level in the operating parameters to more comprehensively reflect the system operating conditions.

[0017] Furthermore, in step S3, the multi-objective reward function also introduces a penalty term for the deterioration of the system's operational stability index, so as to guide the reinforcement learning strategy to take into account the stability of system performance when optimizing maintenance decisions.

[0018] Furthermore, in step S4, the reinforcement learning algorithm adopts a deep Q-network (DQN) algorithm based on value function approximation, or a proximal policy optimization (PPO) algorithm based on policy gradient; the preset set of discretized maintenance actions includes at least: no operation, routine inspection, preventive maintenance of IGBT power modules, preventive maintenance of DC bus capacitors, and joint maintenance of both.

[0019] Furthermore, in step S5, updating the internal state of the system based on the effect of maintenance execution specifically includes: if the maintenance action is to replace or repair a component, then according to the type of maintenance action, the remaining lifespan value of the corresponding component in the lifespan coordination model is reset or partially restored to a preset level, and the running time since the last maintenance is reset.

[0020] Second, the present invention also provides an energy storage system maintenance decision-making device for implementing the above method, characterized in that it includes: The lifetime collaborative modeling unit is used to build and maintain the lifetime collaborative model and output the remaining lifetime value and dynamic failure probability of the component. A state characterization unit, connected to the lifetime collaborative modeling unit and the external data acquisition system, is used to generate the system state vector; The policy execution and learning unit includes the reinforcement learning policy model and is configured to execute the policy obtained in step S4 during the deployment phase to complete the decision, and / or execute step S4 using a reinforcement learning algorithm during the training phase to optimize the policy. The maintenance execution interface unit is connected to the strategy execution and learning unit and the energy storage converter control system, and is configured to trigger and coordinate specific maintenance operations based on the decision instructions output by the strategy execution and learning unit.

[0021] Furthermore, the policy execution and learning unit integrates multiple sub-models based on different reinforcement learning algorithm architectures, and includes an adaptive selection module for selecting and activating the most suitable sub-model for decision inference based on the current system state or preset rules.

[0022] Third, the present invention also provides an energy storage system, including at least one energy storage converter, a sampling module for collecting operating data, and a system control module. The feature is that it also includes an energy storage system maintenance decision device as described above. The maintenance decision device is communicatively connected to the system control module, receives operating data and sends maintenance decision instructions to it, thereby upgrading the maintenance mode based on fixed rules to an adaptive intelligent maintenance mode based on real-time health status and multi-objective optimization.

[0023] Fourth, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the above-described method.

[0024] Compared with the prior art, the present invention has the following significant advantages: ① Achieve collaborative modeling of the lifespan of multiple components: For the first time in the field of energy storage system maintenance, the degradation behavior of two key components, IGBT and capacitor, and their mutual influence are considered simultaneously, thereby improving the accuracy of the model.

[0025] ② Adaptive optimization of maintenance strategy: The maintenance timing is automatically adjusted through reinforcement learning algorithm, without relying on the experience of engineers or fixed threshold judgment.

[0026] ③ System-level risk reduction: By introducing a failure probability and risk loss model, the agent can avoid high-risk operation and effectively reduce the failure rate.

[0027] ④ Significantly reduce maintenance costs: By optimizing maintenance timing, unnecessary preventative maintenance is reduced, and overall operation and maintenance costs are expected to decrease by 15%–30%.

[0028] ⑤ Enhance the stability of the energy storage system: Dynamic optimization of maintenance decisions can ensure the long-term stable operation of the PCS, reduce the accumulation of device fatigue, and extend its service life.

[0029] ⑥ It has good scalability: This invention can be applied to energy storage converters of different sizes and different types of power devices, and supports model expansion.

[0030] In summary, this invention effectively solves the problems of insufficient multi-component coordination, maintenance cost optimization, and intelligent strategy generation in existing energy storage system maintenance technologies, and has good engineering value and promotion significance. Attached Figure Description

[0031] To more clearly illustrate the technical solution of the present invention, the accompanying drawings are provided below for further explanation. It should be understood that the drawings are only used to assist in illustrating the principles and implementation steps of the present invention, and are not intended to limit the scope of protection of the present invention.

[0032] Figure 1 This is a schematic diagram of the overall framework of the energy storage system maintenance decision optimization device and system of the present invention.

[0033] Figure 2 This is a schematic diagram of the overall structure of the energy storage system of the present invention.

[0034] This diagram illustrates the basic components of an energy storage power station, including an energy storage converter (PCS), DC bus capacitors, IGBT power modules, sampling circuits, a control module, a temperature monitoring unit, and the maintenance decision-making device proposed in this invention. The maintenance decision-making device is connected to the control module, and its output maintenance strategy can be used to drive overhaul or preventative maintenance operations.

[0035] Figure 3 This is a schematic diagram of the multi-component lifetime coordination model of the present invention.

[0036] The schematic diagram illustrates the lifetime degradation curves of two types of devices: IGBTs and DC bus capacitors, and their coupling relationship: the increase in capacitor ESR leads to an increase in bus ripple, which further increases the IGBT junction temperature; the increased heat generation of the IGBT may in turn increase the capacitor temperature, accelerating its aging. This figure emphasizes the core idea of ​​this invention: collaborative modeling of the lifetime of multiple components.

[0037] Figure 4 This is a schematic diagram of the component failure probability mapping relationship in the lifetime model of this invention.

[0038] This figure illustrates the functional relationship between remaining lifetime and failure probability. By setting a non-linear mapping, the lower the component lifetime, the higher its failure probability, which is used to quantify the risk of failure in maintenance decisions.

[0039] Figure 5 This is a structural diagram of the system state vector construction of the present invention.

[0040] The diagram shows the elements contained in the state vector, including the remaining life of the IGBT, the remaining life of the capacitor, the number of operating days, the last maintenance interval, system stability indicators, and the probability of failure, which are used to illustrate the working principle of the state representation unit.

[0041] Figure 6 This is a schematic diagram of the multi-objective reward function structure of the present invention.

[0042] The diagram illustrates that the reward function consists of multiple parameters, including maintenance cost, failure loss, lifetime recovery benefit, system stability loss, and risk penalty, and is used to quantify the comprehensive effect of different maintenance strategies.

[0043] Figure 7 This is a schematic diagram of the maintenance strategy training process based on DQN of the present invention.

[0044] The figure illustrates the steps of experience replay pool construction, state sampling, Q-value update, policy network and target network synchronization, and ε-greedy action selection, explaining the policy learning mechanism of DQN.

[0045] Figure 8 This is a schematic diagram of the maintenance strategy training process based on PPO of the present invention.

[0046] The diagram illustrates the steps of advantage function estimation, policy pruning, iterative optimization, and policy update, which are used to explain the training characteristics of the PPO algorithm and its role in maintaining policy optimization.

[0047] Figure 9 This is a functional structure block diagram of the maintenance decision-making device of the present invention.

[0048] The block diagram shows that the device of the present invention consists of a lifespan collaborative modeling unit, a state representation unit, a reward evaluation unit, a reinforcement learning decision-making unit, and a maintenance execution unit. The units are interconnected through data paths to jointly complete the generation and execution of maintenance strategies.

[0049] Figure 10 This is a sequence diagram of the maintenance strategy decision-making and execution process of this invention.

[0050] The diagram illustrates the system's operation, where the agent generates maintenance actions based on state inputs, updates the state after maintenance is performed, and then proceeds to the next round of decision-making, thus achieving a closed-loop adjustment process.

[0051] Figure 11 This is a schematic diagram illustrating the selection of maintenance strategies during the operation of an energy storage system according to the present invention.

[0052] The diagram illustrates the maintenance decision-making behavior of the agent under different health states. For example, when the remaining lifespan of the IGBT and capacitor is below a set threshold, the agent tends to perform preventative maintenance, while when the system is running stably, the agent chooses "no maintenance" to reduce unnecessary maintenance costs. Detailed Implementation

[0053] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described herein are for illustrating the principles of the present invention and do not constitute a limitation on the scope of protection of the present invention. Where there is no conflict, the technical features in the following embodiments can be combined with each other.

[0054] A reinforcement learning-based maintenance decision optimization method for energy storage systems based on multi-component lifetime collaborative modeling includes the following steps: S1: Establish a lifetime coordination model for key components of the energy storage converter. The lifetime model describes the dynamic degradation characteristics of IGBTs and DC bus capacitors during operation. The lifetime decay is determined by factors such as operating days, temperature characteristics, load level, and ripple current. By constructing a component lifetime coupling relationship model, the degradation trends of IGBTs and capacitors can influence each other, reflecting the coupled aging mechanism of actual equipment operation. The component failure probability can be obtained by mapping the remaining lifetime and used for subsequent maintenance decision evaluation.

[0055] S2: Construct the system state vector and obtain the operating status in real time. The system state vector includes the remaining lifespan of the IGBTs, the remaining lifespan of the DC bus capacitors, the operating time since the last maintenance, operating stability indicators, component failure probability, ambient temperature, and operating sequence parameters, etc., to comprehensively characterize the current health status of the energy storage converter.

[0056] By collecting and processing the PCS operation data, the state vector can be updated periodically, providing effective input for the reinforcement learning agent.

[0057] S3: Construct a multi-objective reward function to quantify the effectiveness of the maintenance strategy. The reward function considers the following factors: ① Costs of predictive maintenance: including material, labor, and downtime costs; ② Expected losses due to failure risk: determined by component failure probability and failure loss model; ③Life recovery benefits from maintenance: This reflects the value of maintenance in extending the life of equipment; ④ System stability loss: such as performance degradation or operational fluctuations that may occur during long-term operation; ⑤ The potential risks of neglecting maintenance.

[0058] The reward function helps the agent weigh maintenance costs against failure losses among different actions, so that the maintenance strategy converges in the optimal direction.

[0059] S4: Maintenance policy training and updating based on reinforcement learning models. This invention employs deep reinforcement learning algorithms, including but not limited to: ①Deep Q-Networks (DQNs) based on value functions are suitable for maintenance strategy scenarios with low state dimension or small action space; ② The Proximal Policy Optimization (PPO) algorithm based on policy gradient has the characteristics of strong stability and smooth policy updates.

[0060] Reinforcement learning agents continuously interact, collect experiential data, calculate rewards, and update policy networks, enabling maintenance policies to learn optimal behaviors over long-term operation. Policy training includes processes such as experience replay, gradient updates, target network updates, and policy pruning to achieve the optimal selection of maintenance actions.

[0061] S5: Execute real-time maintenance decisions according to the trained strategy. During the operation of the energy storage system, the agent selects maintenance actions based on the current state vector, including but not limited to: no maintenance; routine inspection; IGBT preventive maintenance; capacitor preventive maintenance; and joint maintenance of multiple components.

[0062] After maintenance is performed, the system will use the new health status as input for the next decision, thus achieving closed-loop optimization.

[0063] This strategy can increase the frequency of preventative maintenance when equipment health is continuously declining, and reduce unnecessary maintenance when the equipment is in a stable condition, thereby extending service life and reducing operating costs.

[0064] An energy storage system maintenance decision-making device for implementing the above method includes: (1) Lifetime co-modeling unit, used to generate the lifetime decay model of IGBT and capacitor and their coupling relationship, forming an overall health description.

[0065] (2) The state characterization unit obtains operating data from the sampling circuit, sensors and controller of the energy storage converter, and constructs a state vector according to preset rules.

[0066] (3) Reward evaluation unit, which calculates the immediate reward for the current maintenance behavior based on the reward model, and is used for reinforcement learning strategy training or online decision-making.

[0067] (4) Reinforcement learning decision unit, which includes a policy network trained using algorithms such as DQN and PPO, and outputs maintenance decision actions based on the input state.

[0068] (5) Maintenance execution unit, which interfaces with the energy storage PCS control system, triggers routine inspections, preventive maintenance or multi-component maintenance operations based on decision actions, and provides feedback on maintenance results.

[0069] An energy storage system includes a power converter module, a sampling circuit, a control module, and the aforementioned maintenance decision-making device.

[0070] The maintenance decision-making device and control module work together to realize the real-time updating and execution of the energy storage converter maintenance strategy.

[0071] A computer-readable storage medium having a computer program stored thereon, the program implementing all the steps of the above-described maintenance decision method when executed by a processor.

[0072] This program can be deployed on the controller of an energy storage PCS to enable intelligent maintenance and management of power devices.

[0073] Example 1: Method for constructing a multi-component lifetime collaborative model This embodiment provides a method for modeling the lifetime of IGBTs and DC bus capacitors suitable for the operating environment of energy storage converters. By constructing a lifetime decay model, a temperature dependence model, and a component coupling model, a comprehensive description of the health status of multiple components can be achieved.

[0074] 1.1 IGBT Lifetime Decay Model IGBT lifetime degradation is a complex process influenced by multiple stress factors, primarily junction temperature variations, thermal cycle count, voltage stress, and load current. Its lifetime state... It can be expressed as a percentage and calculated recursively using the following model: 1) Running time decay term: Characterizes natural aging under baseline operating conditions. (1) in, For the remaining lifespan of the IGBT, This represents the attenuation coefficient of IGBT operating time. 2) Temperature stress acceleration: Increased junction temperature will significantly accelerate device aging. (2) in, Caused by temperature stress, It is a function of junction temperature, used to describe the accelerated detrimental effect of high temperature on lifetime. Temperature stress coefficient 3) Current stress term: Load current It affects lifespan by influencing conduction and switching losses.

[0075] (3) in, Lifetime loss caused by current stress The current stress coefficient, For load current, The current load exponent coefficient is greater than 1, characterizing the nonlinear effect of current stress. 4) Overall Lifetime Degradation: The remaining lifetime of the IGBT is calculated by combining the stress factors mentioned above. .

[0076] (4) in, This refers to the lifespan loss caused by operating time.

[0077] 1.2 DC bus capacitor lifetime decay model The lifespan of an electrolytic capacitor is primarily affected by its internal hot spot temperature and ripple current. Degradation manifests as an increase in equivalent series resistance (ESR) and a decrease in capacitance (C). Its lifespan status... The model is as follows: (5) in, This is the capacitance attenuation coefficient. Describe ripple current With internal hot spot temperature The aging rate under the combined effect.

[0078] 1.3 Multi-component lifetime coupling relationship During the operation of the energy storage converter, an increase in capacitor ESR will increase bus ripple and cause the IGBT junction temperature to rise; conversely, the heat generated by the IGBT can also increase the capacitor temperature and accelerate its aging.

[0079] Therefore, the coupling relationship is as follows: (6) (7) in, The IGBT lifespan degradation is caused by the increase in capacitor ESR. The impact of IGBT heat generation on capacitor lifespan. It is the coefficient representing the impact of capacitor ESR on IGBT lifespan. It is a coefficient representing the effect of IGBT temperature on capacitor life. for The estimated value of the equivalent series resistance of the capacitor at any given time. This refers to the junction temperature of the IGBT.

[0080] 1.4 Fault Probability Mapping Model The failure probability is obtained by mapping the remaining lifetime: (8) in, For components The probability of failure, This represents the initial lifetime of the component (usually set to 100). The model indicates that the lower the remaining lifetime, the higher the probability of failure. This probability will be used as part of the state vector and as the basis for calculating the failure risk penalty in the reward function.

[0081] Example 2: System State Vector Construction Method This embodiment describes a method for mapping the operating state of an energy storage system to a reinforcement learning state vector.

[0082] State vector Includes the following: (9) in, For the remaining lifespan of the IGBT, For the remaining lifespan of the capacitor, This represents the probability of component failure. This represents the number of operating days since the last maintenance. As a system stability indicator; These are operating parameters such as temperature or load.

[0083] Example 3: Multi-objective reward function design method reward function R To guide reinforcement learning agents in learning optimal strategies, multiple objectives must be balanced, including maintenance costs, failure risks, lifetime benefits, and system stability. It consists of the following components: 3.1 Maintenance cost penalty: When the agent chooses to perform a maintenance action... In this case, corresponding cost penalties will be incurred: (10) in, To maintain cost penalties, To perform maintenance actions The costs incurred.

[0084] 3.2 Failure Penalty Item: When the agent selects to perform a maintenance action... In this case, corresponding cost penalties will be incurred: (11) in, This is a penalty item for failure losses. It is the average cost of loss due to a failure. This represents the probability of failure.

[0085] 3.3 Lifespan Recovery Benefits: (12) in, For life recovery benefits, These are the weighting coefficients. This refers to the amount of lifespan restored.

[0086] 3.4 System stability penalty items: (13) in, This is a system stability penalty item. These are the weighting coefficients. This is an indicator of system instability.

[0087] 3.5 Total Reward: Combining the above penalty and benefit terms, the total reward function of the optimal strategy can be calculated. .

[0088] (14) Example 4: DQN-based maintenance strategy training method The training process of the Deep Q-Network (DQN) in this embodiment includes the following steps.

[0089] 4.1 Construction of the Experience Replay Pool: Experience Replay Pool This corresponds to a complete interaction process of "state → action → reward → next state". By storing these interactions in a replay pool and randomly sampling samples for training during subsequent training, DQN can learn and optimize decision-making strategies more effectively.

[0090] (15) in, Representing the intelligent agent at time step The system state at that time, The agent at time step The decision-making actions made at that time Represents the time step The immediate feedback received by the intelligent agent after it performs an action. Represents the execution of the current action The next state that the environment enters after that.

[0091] 4.2 Q-Network Update: The goal is to minimize the loss function. This means ensuring that the Q-value predicted by the current Q-network is as close as possible to the target Q-value. By continuously optimizing the Q-network, the agent can learn the optimal policy, thereby selecting the optimal action in each state.

[0092] (16) in, Indicates the current state Next, execute the action. The Q value (i.e., the current predictive value). Indicates the current time Execute action The instant reward received This represents the discount factor, typically in the range [0,1]. This represents the maximum total reward that the intelligent agent can expect to obtain in the next state.

[0093] 4.3 -greedy action selection: The role of the -greedy strategy is to balance "exploration" and "exploitation" during training. In large-scale environments, agents explore the environment more and randomly select actions, which helps avoid getting trapped in local optima. In the short term, the agent relies more on the learned optimal strategy, choosing the action with the highest Q value. By gradually reducing... In the early stages of training, the agent explores more, while in the later stages of training it gradually focuses more on utilizing the learned strategies.

[0094] (17) in, Indicates the current time The action performed The exploration rate represents the balance between exploration and exploitation for the controlling agent. Indicates the state Choose the action with the largest Q value. .

[0095] 4.4 Strategy Update: Once the Q network converges, the resulting strategy can be used for real-time maintenance and optimization of the energy storage system.

[0096] Example 5: Maintenance Strategy Training Method Based on PPO 5.1 Pruning Strategy Update: Pruning is used to avoid over-updating and maintain the stability of the strategy update. Optimizing the loss function ensures that the difference between the new and old strategies is not too large, avoiding oscillations or instability during training.

[0097] (18) in, At time step Below, the current strategy (by...) The ratio of parameterized (parameterized) action probabilities to those of the previous policy. Indicates the current time step Next strategy selection action Compared to the advantage function of the average strategy, Hyperparameters used to limit the scope of policy updates For cutting The value, keep it at Within the range.

[0098] 5.2 Calculation of the dominance function: Dominance function The advantage function is used to measure the merit of the current action; it is the difference between the current reward and the state value function. A positive value of the advantage function indicates that the current action is better than the state. A positive value indicates a better average policy, while a negative value indicates a worse current action. By calculating the advantage function, the agent can be guided to favor actions that are better than the current average policy.

[0099] (19) in, This represents a discount factor used to balance the importance of current rewards and future rewards. Indicates the current state State value function, Indicates the next state The state-value function represents the state value function. Initially, the agent will determine the expected total reward based on the current strategy.

[0100] 5.3 Multi-epoch Update: During PPO training, multiple epochs are typically used to update the policy. Each time, a batch of data is sampled from the experience replay pool, and the policy is updated multiple times. This allows each sample to be utilized multiple times, further improving the convergence and stability of the policy.

[0101] Example 6: Maintenance Execution Process After the maintenance strategy training is completed, perform the following steps during the operation of the energy storage system: 1. When both IGBT and capacitor lifespan are high, the agent chooses "no maintenance" to reduce costs; 2. When the capacitor's lifespan decreases to the risk range, perform "capacitor preventative maintenance"; 3. When both components are at high risk, the agent chooses "joint maintenance"; 4. When system stability indicators are below the threshold, prioritize performing "routine checks"; After maintenance actions are performed, the system state vector is updated in real time, forming a closed-loop maintenance strategy. Based on the trained reinforcement learning maintenance strategy, the maintenance execution during the operation of the energy storage system is described in detail. Maintenance decisions adopt a closed-loop approach, meaning that after each system cycle, the need for maintenance measures is reassessed based on the current state vector.

[0102] The maintenance process is as follows: 1. Status Acquisition: Acquire parameters such as IGBT remaining lifespan, capacitor remaining lifespan, failure probability, and operational stability to form a status vector. ; 2. Action Selection: Input the state into the reinforcement learning policy network to generate maintenance actions. ; 3. Action Execution: Perform corresponding maintenance operations according to different actions, including routine checks, IGBT maintenance, capacitor maintenance, or combined maintenance; 4. Lifespan Update: Update component lifespan based on maintenance effectiveness. : (20) in, This refers to the amount of lifespan restored.

[0103] 5. Status Update: Recalculate the failure probability, stability indicators, and operating parameters for the next cycle; 6. Closed-loop control: The updated state serves as the input for the next moment, enabling rolling decision-making. This closed-loop mechanism ensures that the intelligent agent can dynamically adjust its strategy in real time based on component aging, operational stability, and maintenance costs.

[0104] Example 7: System Implementation This embodiment provides an integration method of the maintenance decision-making method of the present invention into an energy storage converter system, such as... Figure 2 As shown, the maintenance decision-making device can be deployed in the control module or a separate operation and maintenance management unit, and exchanges operational status data with the PCS through an internal communication interface.

[0105] The system architecture includes: 1. Lifetime Co-modeling Unit: Real-time calculation of IGBT and capacitor lifetime degradation and coupling effects; 2. Status characterization unit: Acquires operating data from temperature sensors, current sampling modules, and bus voltage measurement modules; 3. Reward Evaluation Unit: Calculates rewards during the training phase for policy optimization; 4. Reinforcement learning decision-making unit: Based on DQN or PPO policy network, outputting maintenance actions; 5. Maintenance Execution Unit: Interacts with the PCS control panel to perform maintenance actions; 6. Data Recording Unit: Used to record operation and maintenance history, facilitating model retraining. The system communication logic is as follows: PCS → StateUnit → RLAgent → MaintenanceUnit → PCS The above modules work together to enable the real-time execution of the maintenance and optimization strategy of this invention.

[0106] Example 8: Implementation using computer program and storage medium This embodiment provides a computer program implementation method that, when the program runs in a processor, can execute all the steps of the maintenance decision method described in this invention.

[0107] The program includes the following core modules: 1. ModelUpdate(): Updates the lifetime model; 2. StateEncode(): Encodes runtime data into a state vector; 3. SelectAction(): Invokes the trained policy network to generate actions; 4. ExecuteAction(): Communicates with PCS to perform maintenance; 5. UpdateState(): Updates the health status based on maintenance actions; 6. LogData(): Records the running history for subsequent retraining.

[0108] The pseudocode is as follows: Example 9: Parameter Example This embodiment provides a typical parameter range that can be used in this invention: 1. IGBT aging coefficient : 2. Capacitor aging coefficient : 3. Failure losses 50%–200% of the equipment value 4. Preventive maintenance costs 100–500 yuan 5. Reward Function Weights : The above parameters can be adjusted according to different energy storage system designs.

[0109] Example 10: Environmental Simulation Example To verify the effectiveness of this invention, an energy storage system simulation environment can be established to simulate the aging behavior of IGBTs and capacitors and the execution process of maintenance strategies. The simulation results can be used to illustrate the significant advantages of this invention in reducing failure rates and maintenance costs.

[0110] Simulation includes the following elements: 1. Set the initial lifespan of the IGBT and capacitor to 100%; 2. Simulate daily operation according to the load ratio to gradually reduce the lifespan; 3. Perform maintenance procedures at different aging stages to verify the lifespan recovery effect; 4. Comparison between RL strategies and fixed threshold strategies: (twenty one) (twenty two) in, This represents the maintenance cost under the reinforcement learning strategy. This indicates the maintenance cost under a strong chemical strategy. This represents the failure rate under the reinforcement learning strategy. This represents the failure rate when using a reinforcement learning strategy.

[0111] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A reinforcement learning-based maintenance decision optimization method for energy storage systems based on multi-component lifetime collaborative modeling, characterized in that, Includes the following steps: S1: Establish a lifetime coordination model for the coordinated degradation process of at least IGBT modules and DC bus capacitors in the energy storage converter; wherein, the lifetime coordination model includes independent lifetime decay sub-models established for IGBT power modules and DC bus capacitors respectively, and a lifetime coupling relationship sub-model for quantifying the mutually accelerated aging effect caused by thermal and electrical coupling between the two during operation; based on this model, the remaining lifetime value of each component and the corresponding dynamic failure probability at any time period can be calculated; S2: Based on the remaining lifetime value and dynamic fault probability output by the lifetime coordination model, and combined with the real-time collected system operating parameters, a multi-dimensional system state vector is constructed; the system state vector serves as a quantitative representation of the overall health status of the system, and includes at least the current IGBT remaining lifetime value, DC bus capacitor remaining lifetime value, corresponding dynamic fault probability value, and operating time since the last maintenance. S3: Based on the information contained in the system state vector, construct a multi-objective reward function; the design goal of the reward function is to balance maintenance costs and system risks, and its calculation items include at least: maintenance cost penalty items directly related to the maintenance actions performed, expected failure loss penalty items calculated based on the dynamic failure probability value, and positive benefit items generated by the component life recovery brought about by the maintenance actions; S4: Using the system state vector as input, the preset set of discretized maintenance actions as output space, and the multi-objective reward function as the evaluation criterion for the optimization objective, the reinforcement learning algorithm is used for offline or online policy training, and finally a trained reinforcement learning policy model that can map the optimal maintenance action according to the input state is obtained. S5: During the actual operation of the energy storage system, the following closed-loop process is executed periodically or triggered: First, the system state vector at the current moment is generated in real time according to steps S1 and S2; second, the system state vector is input into the reinforcement learning policy model obtained in step S4, and the optimal maintenance decision output by it is obtained and executed; finally, the internal state of the system is updated according to the effect of maintenance execution, and it serves as the starting point of the next decision cycle.

2. The reinforcement learning maintenance decision optimization method for energy storage systems according to claim 1, characterized in that, In step S1, the lifetime coupling relationship sub-model specifically quantifies the following physical processes: the increase in the equivalent series resistance (ESR) of the DC bus capacitor leads to an increase in the DC bus ripple current, which in turn increases the switching losses and junction temperature of the IGBT power module, accelerating its aging; at the same time, the power loss of the IGBT power module causes heat generation, which causes the ambient temperature of the DC bus capacitor to rise, thereby accelerating the drying of its electrolyte and parameter degradation.

3. The reinforcement learning maintenance decision optimization method for energy storage systems according to claim 1, characterized in that, In step S2, the system operating parameters include parameters obtained directly or indirectly from sensors or controllers. The system state vector is further expanded by at least one of the operating stability indicators, ambient temperature, and load current level in the operating parameters to more comprehensively reflect the system operating conditions.

4. The reinforcement learning maintenance decision optimization method for energy storage systems according to claim 1, characterized in that, In step S3, the multi-objective reward function also introduces a penalty term for the deterioration of the system's operational stability index, so as to guide the reinforcement learning strategy to take into account the stability of system performance when optimizing maintenance decisions.

5. The reinforcement learning maintenance decision optimization method for energy storage systems according to claim 1, characterized in that, In step S4, the reinforcement learning algorithm adopts either the Deep Q-Network (DQN) algorithm based on value function approximation or the Proximal Policy Optimization (PPO) algorithm based on policy gradient. The preset set of discrete maintenance actions includes at least: performing no operation, performing routine checks, performing preventive maintenance on the IGBT power module, performing preventive maintenance on the DC bus capacitor, and performing joint maintenance on both.

6. The reinforcement learning maintenance decision optimization method for energy storage systems according to claim 1, characterized in that, In step S5, updating the internal state of the system based on the effect of maintenance execution specifically includes: if the maintenance action is to replace or repair a component, then according to the type of maintenance action, the remaining lifespan value of the corresponding component in the lifespan coordination model is reset or partially restored to a preset level, and the running time since the last maintenance is reset.

7. A maintenance decision-making device for an energy storage system for implementing the method according to any one of claims 1 to 6, characterized in that, include: The lifetime collaborative modeling unit is used to build and maintain the lifetime collaborative model and output the remaining lifetime value and dynamic failure probability of the component. A state characterization unit, connected to the lifetime collaborative modeling unit and the external data acquisition system, is used to generate the system state vector; The policy execution and learning unit includes the reinforcement learning policy model and is configured to execute the policy obtained in step S4 during the deployment phase to complete the decision, and / or execute step S4 using a reinforcement learning algorithm during the training phase to optimize the policy. The maintenance execution interface unit is connected to the strategy execution and learning unit and the energy storage converter control system, and is configured to trigger and coordinate specific maintenance operations based on the decision instructions output by the strategy execution and learning unit.

8. The energy storage system maintenance decision-making device according to claim 8, characterized in that, The policy execution and learning unit integrates multiple sub-models based on different reinforcement learning algorithm architectures, and includes an adaptive selection module for selecting and activating the most suitable sub-model for decision inference based on the current system state or preset rules.

9. An energy storage system, comprising at least one energy storage converter, a sampling module for acquiring operational data, and a system control module, characterized in that, It also includes the energy storage system maintenance decision-making device as described in claim 8 or 9; the maintenance decision-making device is communicatively connected to the system control module, receives operating data and sends maintenance decision instructions to it, thereby upgrading the maintenance mode based on fixed rules to an adaptive intelligent maintenance mode based on real-time health status and multi-objective optimization.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the method of any one of claims 1 to 7.

Images