Multi-component system maintenance decision-making method based on safety reinforcement learning

Abstract

The invention discloses a multi-component system maintenance decision-making method based on safety reinforcement learning. The method comprises the following steps: establishing a two-parameter Weibull distribution failure rate function of a key component according to historical fault data; the influence of factors such as maintenance types, maintenance times and maintenance cost on performance is considered, a multi-mode maintenance strategy is formed, and the influence of maintenance behaviors on component performance improvement is represented by using a service age backspacing factor; a Lagrange multiplier method is introduced, and a security reinforcement learning network model based on an independent near-end strategy optimization algorithm is established; and training the model by taking the minimum maintenance cost as a target under the safety constraint, and completing the maintenance simulation of the multi-component system under the reinforcement learning framework. According to the method, the problems that a complex system objective function is difficult to model and lacks safety constraints can be solved, the overall operation and maintenance cost of the system is reduced, the problems of time and how to maintain key components are effectively solved, and a safe, economical and reliable maintenance scheme is provided for a multi-component system.

Description

Technical Field

[0001] The present invention relates to the field of mechanical system maintenance strategy optimization, and in particular to a multi-component system maintenance decision-making method based on safety reinforcement learning. Background Art

[0002] Maintenance decision-making is the core link of fault prediction and health management, and is crucial to ensuring the safe and reliable operation of mechanical equipment. To reasonably formulate a maintenance strategy for the entire life cycle of equipment, it is necessary to comprehensively consider multiple factors such as maintenance cost, maintenance timing, maintenance method, and equipment operating status. For the maintenance decision-making problem under multi-component system and multi-objective conditions, there are few related studies and no effective solution due to the numerous constraints and complex scenarios involved.

[0003] Traditional maintenance decision-making technology requires the establishment of an objective function and the application of heuristic algorithms for optimization. Typical algorithms include ant colony optimization algorithm, differential evolution algorithm and particle swarm optimization algorithm, most of which are only applicable to decision-making scenarios for a single objective and a single variable. Multi-component systems have complex structures and many components, and the constraints between maintenance costs and reliability need to be considered. When dealing with complex coupling problems of multiple variables and multiple objectives, traditional technologies have difficulties such as large search space dimensions, large amount of calculations and difficulty in modeling objective functions, which limits their transferability. As the number of intelligent agents increases, reinforcement learning is more suitable for multi-objective optimization problems and will not encounter the curse of dimensionality. It is a more promising decision-making optimization technology.

[0004] Mechanical equipment will inevitably fail during its service life, posing major economic and safety risks. In reinforcement learning, the previous solution to avoid safety risks is to add failure penalties to the reward function, but it is difficult to manually design the penalty value. A penalty value that is too low will lead to a lack of necessary maintenance of the equipment and frequent failures. A penalty value that is too high will lead to excessive maintenance and waste of maintenance resources. In risk-sensitive fields such as nuclear power and aerospace, due to the high value and complexity of high-end equipment, operation and maintenance personnel often adopt conservative maintenance strategies, ensuring safety requirements through frequent inspections, which increases maintenance costs. Therefore, when designing a maintenance plan for a multi-component system, further consideration should be given to how to achieve optimal economic benefits while meeting safety constraints. Summary of the invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide a multi-component system maintenance decision-making method based on safety reinforcement learning. The present invention introduces the IPPO-Lagrangian algorithm in the safety reinforcement learning method into maintenance decision-making, realizes the synchronous planning of maintenance strategies for multi-component systems, and further improves the safety and economy in the health management process of multi-component systems.

[0006] In order to achieve the above object, the present invention adopts the following scheme.

[0007] A maintenance decision method for a multi-component system based on safety reinforcement learning, comprising:

[0008] Step 1: Establish the failure rate function of system components based on historical fault data and establish the state space of safety reinforcement learning;

[0009] Step 2: Design a multi-mode maintenance strategy based on maintenance type, maintenance frequency, and maintenance cost, and construct an action space for safety reinforcement learning;

[0010] Step 3: Design the reward function and risk function of safety reinforcement learning, taking the inspection cost and maintenance cost at each maintenance node as the reward value and the unplanned failure of the component as the cost value;

[0011] Step 4: Using the state space, action space, reward function and risk function, a secure reinforcement learning network model based on the IPPO algorithm is constructed;

[0012] Step 5: Train the IPPO algorithm-based security reinforcement learning network model, migrate the trained model to a real industrial system, and formulate an optimal maintenance strategy for a multi-component system while avoiding unplanned downtime.

[0013] Optionally, step 1 includes: fitting the failure rate of a single component using a two-parameter Weibull distribution function, the expression of which is:

[0014]

[0015] In the formula, is the equipment running time, is the probability of component failure at time t, is the shape parameter, is the size parameter;

[0016] The cumulative distribution function of the two-parameter Weibull distribution of a single component is expressed as:

[0017] ;

[0018] For a series structure system consisting of N independent components, its cumulative distribution function is The expression is:

[0019] ;

[0020] The expression of the state function of a multi-component system is:

[0021]

[0022] Where N is the total number of components in a multi-component system, is the state of N components at time t, is the failure rate of component i, is an indicator for evaluating whether component i is operating normally. The failure rate of component i does not exceed the failure threshold When , it indicates normal operation. Set to 1, otherwise it indicates a fault. Set to 0.

[0023] Optionally, in step 2, the multi-mode maintenance strategy includes: failure replacement, preventive replacement and incomplete maintenance.

[0024] Optionally, the IPPO algorithm-based security reinforcement learning network model includes multiple agent strategy networks, which transform the multi-component system state function After being input into the agent strategy network, it is processed by the multi-layer perceptron and outputs the action , indicating that a certain maintenance action is taken on component i at time t, If set to 0, it means no repair, if set to 1, it means incomplete repair, otherwise it means replacement.

[0025] Optionally, after the agent strategy network selects a corresponding action, the environment changes accordingly, and the components In the The expression of the failure rate function within an overhaul cycle is:

[0026]

[0027] In the formula, For passing The service age regression factor after the incomplete maintenance, is the incomplete repair cost, For preventive replacement costs, It is the adjustment parameter of input cost, which is used to adjust the influence of maintenance input cost on service life, satisfying , is the adjustment parameter of the maintenance frequency, satisfying , both of which are derived from the empirical failure data of the equipment, for Time passes The failure rate after the first repair, The time interval for maintenance activities.

[0028] Optionally, in step 3, the inspection cost and maintenance cost at each maintenance node are used as the reward value, and the unplanned failure of the component is used as the cost value.

[0029] Optionally, in step 4, constructing a secure reinforcement learning network model based on the IPPO algorithm includes:

[0030] Step 4.1: Determine the state, action, reward function and risk function according to the characteristics of the real industrial system;

[0031] Step 4.2: Number of Agent Policy Networks and Number of Components Same, input global state observation , after passing through the multi-layer perceptron and softmax function, the probability distribution of discrete actions is obtained, and the maintenance action is output after random sampling;

[0032] Step 4.3: The number of value networks is twice the number of components and consists of multi-layer perceptrons, where The value network uses the reward and state values ​​obtained by the interaction between the agent and the environment to learn the value function. The value network uses the risk value and state value to learn the value function and jointly guide the policy network to update the policy.

[0033] Optionally, in step 5, a safety boundary is set for the search area of ​​the optimal solution, and the safety reinforcement learning network model is trained with the goal of minimizing the maintenance cost.

[0034] A computer-readable storage medium is used to store a computer program, wherein the computer program is configured to implement the method when called by a processor.

[0035] An electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor; wherein the method is implemented when the processor executes the program.

[0036] Compared with the prior art, the present invention has the following beneficial technical effects:

[0037] 1. Synchronous planning of multi-component maintenance strategies. This technology uses multi-agent reinforcement learning technology based on the IPPO algorithm. Each agent formulates a maintenance strategy for the corresponding component, effectively overcoming the dimensional disaster problem encountered by traditional optimization algorithms when dealing with multi-variable and multi-objective optimization, and realizing synchronous planning of multi-component system maintenance solutions.

[0038] 2. Safety constraints. This technology transforms safety reinforcement learning from a full-state constraint problem into an unconstrained problem based on the Lagrange multiplier method. It minimizes the maintenance cost of high-risk equipment while ensuring that no unplanned downtime occurs, solves the problem of difficult penalty item design, and provides an efficient and safe general framework for multi-component system maintenance decisions.

[0039] 3. Maintenance based on equipment operating status. The present invention uses intelligent algorithms and sensing technologies to take dynamic maintenance measures according to the degradation status of components, effectively alleviating the problems of intensive maintenance and high maintenance costs in traditional periodic maintenance technologies, reducing dependence on expert experience and expertise, and promoting the upgrade of maintenance strategies from time-based maintenance to condition-based maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings illustrate exemplary embodiments of the present invention and together with the description serve to explain the principles of the present invention. These drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification.

[0041] Figure 1 It is a flow chart of a multi-component system maintenance decision method based on safety reinforcement learning in an embodiment of the present invention;

[0042] Figure 2 A simplified diagram of the nuclear power water supply system structure in an embodiment of the present invention;

[0043] Figure 3 A system / equipment failure probability curve diagram in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of an event tree in an embodiment of the present invention;

[0045] Figure 5 It is a framework diagram of the IPPO network system in an embodiment of the present invention;

[0046] Figure 6 It is a basic framework diagram of the PPO-Lagragian network in an embodiment of the present invention;

[0047] Figure 7 Schematic diagram of the change of failure probability of a water supply system over time in an embodiment of the present invention;

[0048] Figure 8 This is a comparison chart of the number of maintenance times under different maintenance strategies in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following is combined with Figures 1 to 8 The present invention is further described in detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant contents, rather than to limit the present invention. It is also necessary to explain that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings.

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0051] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as exemplary features that provide various details of some ways in which the technical concept of the present invention can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present invention.

[0052] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0053] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0054] For descriptive purposes, the present invention may use spatially relative terms such as "below", "beneath", "under", "lower", "above", "upper", "above", "higher", and "side (e.g., as in "sidewall")" to describe the relationship of one component to another (other) component as shown in the accompanying drawings. The spatially relative terms are intended to encompass different orientations of the device in use, operation, and / or manufacture in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, components described as "below" or "beneath" other components or features would then be positioned "above" the other components or features. Thus, the exemplary term "below" can encompass both the "above" and "below" orientations. Furthermore, the device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0055] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0056] In one embodiment, the present invention provides a multi-component system maintenance decision method based on safety reinforcement learning, comprising:

[0057] Step 1: Establish the failure rate function of system components based on historical fault data and establish the state space of safety reinforcement learning;

[0058] Step 2: Design a multi-mode maintenance strategy based on maintenance type, maintenance frequency, and maintenance cost, and construct an action space for safety reinforcement learning;

[0059] Step 3: Design the reward function and risk function of safety reinforcement learning, taking the inspection cost and maintenance cost at each maintenance node as the reward value and the unplanned failure of the component as the cost value;

[0060] Step 4: Using the state space, action space, reward function and risk function, a secure reinforcement learning network model based on the IPPO algorithm is constructed;

[0061] Step 5: Train the IPPO algorithm-based security reinforcement learning network model, migrate the trained model to a real industrial system, and formulate an optimal maintenance strategy for a multi-component system while avoiding unplanned downtime.

[0062] Optionally, step 1 includes: fitting the failure rate of a single component using a two-parameter Weibull distribution function, the expression of which is:

[0063]

[0064] In the formula, is the equipment running time, is the probability of component failure at time t, is the shape parameter, is the size parameter;

[0065] The cumulative distribution function of the two-parameter Weibull distribution of a single component is expressed as:

[0066] ;

[0067] For a series structure system consisting of N independent components, its cumulative distribution function is The expression is:

[0068] ;

[0069] The expression of the state function of a multi-component system is:

[0070]

[0071] Where N is the total number of components in a multi-component system, is the state of N components at time t, is the failure rate of component i, is an indicator for evaluating whether component i is operating normally. The failure rate of component i does not exceed the failure threshold When , it indicates normal operation. Set to 1, otherwise it indicates a fault. Set to 0.

[0072] Optionally, in step 2, the multi-mode maintenance strategy includes: failure replacement, preventive replacement and incomplete maintenance.

[0073] Optionally, the IPPO algorithm-based security reinforcement learning network model includes multiple agent strategy networks, which transform the multi-component system state function After being input into the agent strategy network, it is processed by the multi-layer perceptron and outputs the action , indicating that a certain maintenance action is taken on component i at time t, If set to 0, it means no repair, if set to 1, it means incomplete repair, otherwise it means replacement.

[0074] Optionally, after the agent strategy network selects a corresponding action, the environment changes accordingly, and the components In the The expression of the failure rate function within an overhaul cycle is:

[0075]

[0076] In the formula, For passing The service age regression factor after the incomplete maintenance, is the incomplete repair cost, For preventive replacement costs, It is the adjustment parameter of input cost, which is used to adjust the influence of maintenance input cost on service life, satisfying , is the adjustment parameter of the maintenance frequency, satisfying , both of which are derived from the empirical failure data of the equipment, for Time passes The failure rate after the first repair, The time interval for maintenance activities.

[0077] Optionally, in step 3, the inspection cost and maintenance cost at each maintenance node are used as the reward value, and the unplanned failure of the component is used as the cost value.

[0078] Optionally, in step 4, constructing a secure reinforcement learning network model based on the IPPO algorithm includes:

[0079] Step 4.1: Determine the state, action, reward function and risk function according to the characteristics of the real industrial system;

[0080] Step 4.2: Number of Agent Policy Networks and Number of Components Same, input global state observation , after passing through the multi-layer perceptron and softmax function, the probability distribution of discrete actions is obtained, and the maintenance action is output after random sampling;

[0081] Step 4.3: The number of value networks is twice the number of components and consists of multi-layer perceptrons, where The value network uses the reward and state values ​​obtained by the interaction between the agent and the environment to learn the value function. The value network uses the risk value and state value to learn the value function and jointly guide the policy network to update the policy.

[0082] Optionally, in step 5, a safety boundary is set for the search area of ​​the optimal solution, and the safety reinforcement learning network model is trained with the goal of minimizing the maintenance cost.

[0083] In one embodiment, if Figure 1 As shown, the present invention provides a multi-component system maintenance decision method based on safety reinforcement learning, comprising:

[0084] Step 1: According to historical fault data, the failure rate function of key components of the system is established based on the two-parameter Weibull distribution function, and the state space of safety reinforcement learning is established;

[0085] Step 2: According to the maintenance type, maintenance frequency and maintenance cost, a multi-mode maintenance strategy of "failure replacement + preventive replacement + incomplete maintenance" is designed to construct an action space for safety reinforcement learning. The service age regression factor is introduced to characterize the impact of maintenance behavior on component performance improvement.

[0086] Step 3: Design the reward function and risk function of safety reinforcement learning, taking the inspection cost and maintenance cost at each maintenance node as the reward value and the unplanned failure of the component as the cost value;

[0087] Step 4: Using the state space, action space, reward function and risk function, a secure reinforcement learning network model based on the IPPO algorithm is constructed;

[0088] Step 5: Set a safety boundary for the search area of ​​the optimal solution, train the IPPO algorithm-based safety reinforcement learning network model with the goal of minimizing maintenance cost, migrate the trained model to the real industrial system, and formulate the optimal maintenance strategy for the multi-component system while avoiding unplanned downtime.

[0089] This example takes the nuclear power water supply system as the target object and constructs a numerical example. The reasons for choosing the nuclear power water supply system as the target object are:

[0090] 1) Safety is the most basic requirement for the operation of nuclear power water supply system. Failure of key components will lead to system power reduction or even shutdown, resulting in serious consequences. 2) Nuclear power water supply system consists of multiple components, key components are electric pumps, electric valves and pneumatic valves, which can be simplified into a two-way pump form. During normal operation, one electric pump is in normal use and the other is in standby. When the pump fails, the other one is immediately put into operation, such as Figure 2 3) The main maintenance methods for the electric pumps in the nuclear power water supply system include replacement, lubrication and alignment, and the main maintenance method for valves is replacement, which is consistent with the multi-mode maintenance strategy proposed in the present invention.

[0091] Step 1: According to the historical fault data, the failure rate function of the key components of the system is established based on the two-parameter Weibull distribution function to form the state space of safety reinforcement learning.

[0092] Taking the electric pump, a key equipment in the water supply system, as an example, the calculation object is the historical data of the electric pump status in Table 1. In, when For fault data, define ,otherwise , the randomly truncated fault sample data is recorded as: .

[0093] Table 1 Electric pump status historical data

[0094]

[0095] Step 1.1: Calculate the point estimate of the two-parameter Weibull distribution parameter using the maximum likelihood method, including:

[0096] Construct the likelihood function, the expression is as follows:

[0097]

[0098] Taking partial derivatives of the logarithm of the likelihood function, we can obtain the likelihood equation system:

[0099]

[0100] Further simplifying the above formula, the transcendental equation is as follows:

[0101]

[0102] Step 1.2: Use an iterative method to estimate the parameters of the two-parameter Weibull distribution, including:

[0103] 1) Calculate the initial value of the iteration , where the result of moment estimation is used as the initial value of iteration, and the moment estimation expression is:

[0104] ;

[0105] 2) Calculate using iterative relational formula , the expression is:

[0106] ;

[0107] 3) Discrimination Is it true? If it is true, then end, otherwise proceed to the next step;

[0108] 4) and , return to step 2);

[0109] 5) End the iteration and get the corresponding value;

[0110] 6) Calculation using transcendental equations , output and .

[0111] The two-parameter Weibull distribution parameters are calculated using this random truncated sample , . The KS test is used to compare whether there is a significant difference between the cumulative density function of the two-parameter Weibull distribution and the empirical distribution function, and the result is used to evaluate the applicability of the distribution. The smaller the D value of the test result, the smaller the difference with the empirical distribution function, and the more accurate the fitting result. By looking up the table, it can be seen that when the significance level is 0.05, the critical value is 0.272, and the test result of the two-parameter Weibull distribution is 0.229. The fitting result meets the test standard, indicating that the two-parameter Weibull distribution can fit the life distribution of the equipment well.

[0112] Step 1.3, combining the historical operating status data of key components in the water supply system, using the two-parameter Weibull distribution to fit the failure rate function of a single component;

[0113]

[0114] In the formula, is the equipment running time, is the probability of component failure at time t, is the shape parameter, is the size parameter.

[0115] Construct the cumulative distribution function of the two-parameter Weibull distribution of a single component, which is expressed as:

[0116] ;

[0117] For a series structure system consisting of N independent components, its cumulative distribution function is The expression is:

[0118] ;

[0119] The failure modes and corresponding parameters are shown in Table 2. The fitting results are shown in Figure 3 As shown in the figure, the cumulative failure probability of the valve is higher than that of the electric pump. As the service time increases, the failure rate of the valve increases rapidly.

[0120] Table 2 Equipment failure modes and parameters

[0121]

[0122] The expression of the state function of a multi-component system is:

[0123]

[0124] Where N is the total number of components in a multi-component system, is the state of N components at time t, For components The failure rate, An indicator to evaluate whether component i is operating normally. The failure rate does not exceed the failure threshold When , it indicates normal operation. Set to 1, otherwise it indicates a fault. Set to 0.

[0125] Step 2: Based on the maintenance type, maintenance frequency and maintenance cost, a multi-mode maintenance strategy of "failure replacement + preventive replacement + incomplete maintenance" is constructed to form the action space of safety reinforcement learning, and the service age regression factor is used to characterize the impact of maintenance behavior on component performance improvement.

[0126] The action space of safety reinforcement learning is designed. The multi-mode maintenance strategy includes: failure replacement, preventive replacement and incomplete maintenance. The difference between failure replacement and preventive replacement lies in the failure rate level when the equipment is replaced. Failure replacement means replacing the equipment after it fails, which reduces the system power and increases the maintenance cost. Preventive replacement means replacing the equipment when it can still operate normally, which has a lower maintenance cost.

[0127] The system status value After being input into the agent strategy network, it is processed by the multi-layer perceptron and outputs the action , indicating that a certain maintenance action is taken on component i at time t, If set to 0, it means no repair, if set to 1, it means incomplete repair, otherwise it means replacement.

[0128] After the agent selects the corresponding action, the environment changes accordingly. The service life regression factor is used to represent the relationship between the maintenance type, maintenance frequency, maintenance cost and the performance change after the repair. Replacement makes the equipment status "repaired as new", while incomplete maintenance methods such as lubrication and alignment make the component status between no repair and replacement. The higher the incomplete maintenance cost, the more service life regression, and the better the maintenance effect. As the number of maintenance increases, the maintenance effect gradually decreases. Therefore, the component The expression of the failure rate level function in the i-th overhaul cycle is:

[0129]

[0130] In the formula, For passing The service age regression factor after the incomplete maintenance, is the incomplete repair cost, is the preventive replacement cost, a is the adjustment parameter of the input cost, which is used to adjust the impact of the maintenance input cost on the service life, satisfying , is the adjustment parameter of the maintenance frequency, satisfying , both are derived from the equipment's empirical failure data, a is tentatively set to 1, b is 0.005, is the time after the The failure rate after the first repair, The time interval for maintenance activities.

[0131] Step 3: Design the reward and risk functions of safety reinforcement learning, taking the inspection cost and maintenance cost at each maintenance node as the reward value and the unplanned failure of the component as the cost value.

[0132] The relationship between reward value and cost value and maintenance strategy and component status is as follows: Figure 4 As shown in Figure 2, the reward value consists of the detection cost and the maintenance cost. The optimization goal of the reward function is to minimize the maintenance cost. The detection cost of each cycle is , when the incomplete maintenance strategy is adopted, the total maintenance cost is The total repair cost of the replacement parts is .in is the indicative function, when hour, ,otherwise The failure rate of key components is a safety issue that needs to be considered. When it exceeds the upper limit of failure rate D, a larger cost value is obtained. The cost value can be reduced by optimizing the maintenance strategy to ensure that the components operate within a safe range. When a component fails unplanned, the cost value generated is .

[0133] The objective function of the reward function reflects the reduction of maintenance costs and the avoidance of waste of maintenance resources due to excessive maintenance. The objective function of the risk function reflects the safety of equipment operation. The optimization goal is to ensure that the equipment meets safety constraints throughout its life cycle and avoid equipment failure due to insufficient maintenance.

[0134] Step 4: Combined with the Lagrange multiplier method, using states, actions, reward functions, risk functions, policy networks, and value networks, a secure reinforcement learning network model based on the IPPO algorithm is constructed.

[0135] The IPPO algorithm is a completely decentralized method that has good scalability as the number of agents increases. Figure 5 As shown in FIG. 1 , each agent is independently learned by a single agent reinforcement learning PPO algorithm. The present invention uses N single agent reinforcement learning PPO-Lagragian algorithms to form a multi-agent framework. The basic framework of the PPO-Lagragian network is as follows: Figure 6 shown.

[0136] The framework of the safety reinforcement learning model consists of six basic elements: state, action, reward function, risk function, policy network and value network. The objective function of the reward function reflects the reduction of maintenance costs and avoids the waste of maintenance resources caused by excessive maintenance. The objective function of the risk function reflects the safety of equipment operation, so that the equipment meets safety constraints throughout its life cycle and avoids equipment failure caused by insufficient maintenance.

[0137] The agent strategy network is responsible for representing, updating, constraining and evaluating strategies in the PPO algorithm. By optimizing the parameters of the strategy network, the PPO algorithm can effectively formulate maintenance actions for each maintenance node. The number of networks is the same as the number of components N. At the maintenance node t, information is exchanged with the environment and the state value is input. , after being processed by the multi-layer perceptron and the softmax function, the probability distribution of discrete actions is obtained, and a maintenance action is output after random sampling.

[0138] The value network is used to estimate the state value. By optimizing the parameters of the value network, the PPO algorithm can more accurately evaluate the decision of the agent, thereby improving the optimization effect of the policy network. Its number is twice the number of components and is composed of multi-layer perceptrons. Among them, N value networks use reward values ​​and state values ​​to learn the value function, and the other N value networks use generation values ​​and state values ​​to learn the value function, and jointly guide the policy network to update the policy.

[0139] The construction of a secure reinforcement learning network model based on the IPPO algorithm includes:

[0140] Step 4.1: Determine the state, action, reward function and risk function according to the characteristics of the real industrial system;

[0141] Step 4.2: Number of Agent Policy Networks and Number of Components Same, enter the system status value After passing through the multi-layer perceptron and softmax function, the probability distribution of discrete actions is output, and the maintenance action is output after random sampling ;

[0142] Step 4.3: The number of value networks is twice the number of components and consists of multi-layer perceptrons, where The reward value network uses the reward value r and state value s obtained by the interaction between the agent and the environment to learn the value function. The cost-value network learns the value function from the risk value c and the state value s, and jointly guides the policy network to update the policy.

[0143] Step 5: Set a safety boundary for the search area of ​​the optimal solution, train the safety reinforcement learning network model with the goal of minimizing the maintenance cost, migrate the trained model to the real industrial system, and formulate the optimal maintenance strategy for the multi-component system while avoiding unplanned downtime.

[0144] Training a secure reinforcement learning network model based on the IPPO algorithm, including:

[0145] 1) Initialize the hyperparameters according to the characteristics of the real industrial system, determine the multi-component degradation function and use it as the solution environment of the model, and initialize the environment.

[0146] During model training, the agent policy network is used to obtain all components Status value at the moment , output the maintenance plan of the corresponding component, the value network receives the degradation information of each component, and then evaluates the quality of the maintenance plan formulated by the strategy network, providing a training target for the model parameter update;

[0147] 2) Input the state value into the policy network, and after being processed by the multi-layer perceptron and softmax function, output the probability distribution of discrete actions, which is obtained by random sampling Time widget Maintenance actions ;

[0148] 3) Each agent Always take action After that, the environment will change accordingly, and the environment returns the reward value at time t+1 , cost value and the new state value ;

[0149] 4) During the design life, the multi-component system undergoes T major overhauls in total, and the model makes maintenance decisions at each maintenance node at time t. represents the set of state values ​​of all components in the environment at time t, Represents the set of actions taken by all agents at time t, and the five-tuple data set of state value set at time t, state value set at time t+1, action set, reward value set at time t, and cost value set at time t obtained from steps 2) to 4) Store in the experience pool; if the system operation time does not exceed the design life, continue with step 2) to step 4); if it exceeds the design life but the training is not finished, execute step 1); if the number of samples in the experience pool reaches the preset value, execute step 5) to step 8) until the network converges;

[0150] 5) Randomly sample group B data from the experience pool that stores a large number of training samples Train the model.

[0151] 6) The optimization problem that the agent needs to solve in constrained reinforcement learning is expressed as:

[0152]

[0153] In the formula, Update the value for the parameter, and are the current strategy function and the old strategy function respectively. is the advantage function associated with the value network, is the cost constraint function, is the safety threshold, is the advantage function associated with the cost network, is the KL divergence of the two strategy distributions, is the upper bound threshold of the average KL divergence.

[0154] In order to strengthen the safety constraints, the Lagrange multiplier method is applied and the adaptive penalty coefficient is introduced. In order to simplify the optimization process, the truncation operator is used to replace the KL divergence constraint, and the optimization objective expression of the secure reinforcement learning IPPO-Lagragian is finally obtained as follows:

[0155]

[0156] In the formula, , which limits x to Inside, is a hyperparameter representing the cutoff range.

[0157] 7) Use the sampled gradient descent method to optimize the objective function and the policy network parameters To update, the update process is as follows:

[0158]

[0159] In the formula, is the derivative operator, parameter By subtracting the learning rate and gradient Update the product of .

[0160] 8) By minimizing the error, update the parameters of the value network and reward the loss function of the value network and the loss function of the risk value network The expressions are:

[0161]

[0162] In the formula, is the reward value network with parameters , is the risk value network, and its parameters are , is the discount factor.

[0163] 9) Update the Lagrange multiplier using the gradient descent algorithm , the loss function expression is:

[0164] The present invention combines the working characteristics of the nuclear power water supply system, and the equipment parameters of the given water supply system are shown in Table 3, and the maintenance cost is set proportionally.

[0165] Table 3 Water supply system equipment parameters

[0166]

[0167] The inspection cost of the water supply system is 0.01. Since the failure of key components leads to a decrease in unit power, the huge economic losses caused are difficult to assess. Therefore, the failure replacement cost ratio of the electric pump is tentatively set at 2, and the failure replacement cost ratio of the valve is tentatively set at 2. The maintenance span of nuclear power plant equipment is based on the life of the nuclear power plant, which is 40 years. The basic duration of the overhaul cycle is 18 months. Each maintenance lasts about 1 month, so there are 25 maintenance nodes. The valve failure probability threshold is tentatively set at 0.3, the electric pump failure probability threshold is 0.15, and the upper limit of the water supply system failure probability is 0.4.

[0168] Based on the operating status of the water supply system, safety reinforcement learning is used to develop a maintenance plan with optimal cost and safety constraints for each overhaul node. The final maintenance plan corresponding to the optimal solution is shown in Table 4. After executing the maintenance plan, Figure 7 As shown, the system failure probability is always kept at a low level, the system reliability is higher, and the maintenance plan is more reasonable.

[0169] Table 4 Final maintenance plan corresponding to the optimal solution

[0170]

[0171] The maintenance costs and equipment failure times generated by different maintenance strategies are shown in Table 5. Compared with post-maintenance, the proposed technology can effectively avoid unplanned equipment failures, ensure the safe operation of equipment, and significantly reduce maintenance costs. Compared with regular maintenance, by mining equipment status monitoring information, maintenance based on operating status can reduce the number of inspections and unnecessary maintenance, maintenance resources are fully utilized, and the maintenance cost reduction rate reaches 29.3%.

[0172] Table 5 Cost and equipment failure times of different maintenance strategies

[0173]

[0174] The number of maintenance times under different maintenance strategies is as follows Figure 8 As shown, compared with the regular maintenance strategy adopted by nuclear power plants for key components, the proposed technology formulates maintenance plans based on the operating status of the equipment, significantly reducing the number of inspections and repairs. By introducing incomplete maintenance, the number of replacements of electric pumps is reduced, thereby improving the economic effect of water supply system maintenance.

[0175] In one embodiment, the present invention provides a computer-readable storage medium for storing a computer program, wherein the computer program is configured to implement the method when called by a processor.

[0176] In one embodiment, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor; wherein the method is implemented when the processor executes the program.

[0177] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0178] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0179] It should be understood by those skilled in the art that the above embodiments are only for the purpose of clearly illustrating the present invention, and are not intended to limit the scope of the present invention. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present invention.

Claims

1. A multi-component system maintenance decision method based on safety reinforcement learning, characterized in that: include: Step 1: Establish the failure rate function of system components based on historical fault data and establish the state space of safety reinforcement learning; Step 2: Design a multi-mode maintenance strategy based on maintenance type, maintenance frequency, and maintenance cost, and construct an action space for safety reinforcement learning; Step 3: Design the reward function and risk function for secure reinforcement learning; Step 4: Using the state space, action space, reward function and risk function, a secure reinforcement learning network model based on the IPPO algorithm is constructed; Step 5: Train the IPPO algorithm-based security reinforcement learning network model, migrate the trained model to a real industrial system, and formulate an optimal maintenance strategy for a multi-component system while avoiding unplanned downtime.

2. The method according to claim 1, characterized in that Preferably, step 1 includes: using a two-parameter Weibull distribution function to fit the failure rate of a single component, the expression of which is: ; In the formula, is the equipment running time, is the probability of component failure at time t, is the shape parameter, is the size parameter; The cumulative distribution function of the two-parameter Weibull distribution of a single component is expressed as: ; For a series structure system consisting of N independent components, its cumulative distribution function is The expression is: ; The expression of the state function of a multi-component system is: ; Where N is the total number of components in a multi-component system, is the state of N components at time t, is the failure rate of component i, is an indicator for evaluating whether component i is operating normally. The failure rate of component i does not exceed the failure threshold When , it indicates normal operation. Set to 1, otherwise it indicates a fault. Set to 0.

3. The method according to claim 1, characterized in that In step 2, the multi-mode maintenance strategy includes: failure replacement, preventive replacement and incomplete maintenance.

4. The method according to claim 3, characterized in that The IPPO-based secure reinforcement learning network model includes multiple agent strategy networks. After being input into the agent strategy network, it is processed by the multi-layer perceptron and outputs the action , indicating that a certain maintenance action is taken on component i at time t, If set to 0, it means no repair, if set to 1, it means incomplete repair, otherwise it means replacement.

5. The method according to claim 4, characterized in that After the agent strategy network selects the corresponding action, the environment changes accordingly. In the The expression of the failure rate function within an overhaul cycle is: ; In the formula, For passing The service age regression factor after the incomplete maintenance, is the incomplete repair cost, For preventive replacement costs, It is the adjustment parameter of input cost, which is used to adjust the influence of maintenance input cost on service life, satisfying , is the adjustment parameter of the maintenance frequency, satisfying , both of which are derived from the empirical failure data of the equipment, for Time passes The failure rate after the first repair, The time interval for maintenance activities.

6. The method according to claim 1, characterized in that In step 3, the inspection cost and maintenance cost at each maintenance node are used as the reward value, and the unplanned failure of the component is used as the cost value.

7. The method according to claim 1, characterized in that In step 4, the construction of a secure reinforcement learning network model based on the IPPO algorithm includes: Step 4.1: Determine the state, action, reward function and risk function according to the characteristics of the real industrial system; Step 4.2: Number of Agent Policy Networks and Number of Components Same, input global state observation , after passing through the multi-layer perceptron and softmax function, the probability distribution of discrete actions is obtained, and the maintenance action is output after random sampling; Step 4.3: The number of value networks is twice the number of components and consists of multi-layer perceptrons, where The value network uses the reward and state values ​​obtained by the interaction between the agent and the environment to learn the value function. The value network uses the risk value and state value to learn the value function and jointly guide the policy network to update the policy.

8. The method according to claim 1, characterized in that: In step 5, a safety boundary is set for the search area of ​​the optimal solution, and the safety reinforcement learning network model is trained with the goal of minimizing the maintenance cost.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is configured to implement any one of the methods of claims 1-8 when called by a processor.

10. An electronic device, characterized in that: The electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor; wherein the processor implements any one of the methods of claims 1-8 when executing the program.

Images