Nuclear power pump unit maintenance method based on deep reinforcement learning

Abstract

The invention provides a nuclear power pump set maintenance method based on deep reinforcement learning, which comprises the following steps: acquiring a degradation index result according to historical operation data of a nuclear power pump set, establishing a degradation model by using a random process, and constructing a state space S; aiming at different maintenance activity types of the nuclear power pump set, defining a maintenance action space as a perfect maintenance apm and a non-action aN, and constructing an action space A; constructing a cost function C (t) containing shutdown penalty cost and maintenance action cost to obtain a reward function r; simulation of the nuclear power pump unit is completed in the deep reinforcement learning framework, and a maintenance strategy network of the nuclear power pump unit is trained by using a deep reinforcement learning algorithm (PPO) to obtain an optimal maintenance strategy network model after training is completed; and based on the real-time operation data of the nuclear power pump unit, extracting a degradation index result, and inputting the degradation index result into the optimal maintenance strategy network model as input so as to output the optimal maintenance action of the nuclear power pump unit.

Description

Technical Field

[0001] This disclosure pertains to the field of nuclear power plant pump unit maintenance strategy optimization, and in particular to a nuclear power plant pump unit maintenance method based on deep reinforcement learning. Background Technology

[0002] With the continuous increase in the number and scale of nuclear power equipment, its safety has received widespread attention. During equipment operation, nuclear power pump failures occur frequently, seriously threatening personnel safety. Therefore, a reasonable operation and maintenance management plan is urgently needed to provide timely maintenance recommendations for the equipment, thereby guiding its safe and stable operation.

[0003] However, the internal structure, operating environment, and reliability requirements of nuclear power plant pumps make operational status monitoring, fault diagnosis, and maintenance decision-making difficult, leading to significant maintenance delays and increased maintenance risks and costs. Therefore, a solution is needed to address these issues. Research on this method has significant application potential, providing intelligent maintenance decisions for nuclear power plant pump units to effectively reduce maintenance costs and improve equipment safety. Summary of the Invention

[0004] This disclosure addresses the problems existing in the prior art and provides the following technical solutions:

[0005] A method for maintaining nuclear power plant pump units based on deep reinforcement learning includes the following steps:

[0006] Step S100: Obtain degradation index results based on historical operating data of nuclear power pump units, and then further establish a degradation model using stochastic processes based on the degradation index results, and construct the state space S;

[0007] Step S200: For different maintenance activity types of nuclear power plant pump units, define the maintenance action space as perfect maintenance a. pm and no action a N Construct action space A;

[0008] Step S300: Construct a cost function C that includes downtime penalty costs and maintenance operation costs. (t) This is used to obtain the reward function r;

[0009] Step S400: Based on the state space S, action space A and reward function r, complete the simulation of the nuclear power pump unit within the deep reinforcement learning framework, and use the deep reinforcement learning algorithm (PPO) to train the maintenance strategy network of the nuclear power pump unit to obtain the optimal maintenance strategy network model after training.

[0010] Step S500: Based on the real-time operating data of the nuclear power plant pump unit, extract the degradation index results, use them as input, and input them into the optimal maintenance strategy network model to output the optimal maintenance action of the nuclear power plant pump unit.

[0011] In the method described, the degradation model established using a stochastic process in step S100 is represented as follows:

[0012] ,

[0013] Among them, D i (t) represents the degradation state of the i-th pump when its service time is t; X θ (t) represents a random process; θ represents the parameters to be determined in the random process; FT i is the failure time of the i-th pump; w is the failure threshold.

[0014] In the method described, in step S100, the state space S for all N pumps in the entire pump group is represented as:

[0015] ,

[0016] Among them, O i D is a Boolean variable indicating whether the i-th pump is operating normally. i Let be the degradation amount of the i-th pump.

[0017] In the method described, the O i Let O be a Boolean variable indicating whether the i-th pump is operating normally. Its value is determined by the failure threshold w. i The logic for determining the value is as follows:

[0018] ,

[0019] When the pump degradation is less than the failure threshold w, it is considered to be operating normally; otherwise, it is considered to be shut down and failed.

[0020] In the method described, the perfect maintenance of a in step S200 pm This refers to a thorough overhaul of the selected pump, reducing its degradation by a factor of D. i Restored to 0.

[0021] In the method described, the no-action step S200 is described as a N This means that no operation is performed on the selected pump, and its state will evolve naturally according to the degradation model.

[0022] In the method described, step S300 involves constructing a cost function C based on downtime penalty costs and operation maintenance costs. (t) Represented as:

[0023] ,

[0024] in, For a moment Downtime penalty costs; For a moment Maintenance costs.

[0025] In the method described, the formula for calculating the downtime penalty cost is as follows:

[0026] ,

[0027] in, For a moment Downtime penalty costs; For a moment The number of pumps that perform perfect maintenance; C pm The cost of performing a perfect maintenance; Let t be the number of pumps that are in a stopped and failed state at time t.

[0028] In the method described, the formula for calculating the maintenance cost of the action is:

[0029] ,

[0030] in, For a moment Maintenance costs, For a moment The number of pumps that have undergone perfect maintenance.

[0031] In the method described, when training the maintenance strategy network of the nuclear power plant pump unit using a deep reinforcement learning algorithm (PPO) in step S400, a discount rate γ is set to process a series of maintenance behaviors. The overall objective is:

[0032] ,

[0033] Where C is the future long-term discount cost under the current maintenance strategy; t is the deep reinforcement learning time; and γ is the deep reinforcement learning discount coefficient.

[0034] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0035] This disclosure transforms the maintenance scheme in nuclear power plant pump unit operation and maintenance from the original single-pump phased fixed-interval maintenance to a system based on the overall "state-maintenance action-instant value feedback" of the pump unit, seeking suitable state-action strategies. By introducing downtime penalty costs and maintenance cost constraints, cluster maintenance optimization based on deep reinforcement learning can execute maintenance behaviors under state guidance, thus avoiding fixed-interval maintenance. The establishment of the optimal maintenance strategy network model within the deep reinforcement learning framework fully avoids the problem of large space in traditional optimization algorithms due to the complexity of the operation and maintenance environment, and has the potential to solve multi-level and multi-dimensional representation spaces.

[0036] The description provided is merely an overview of the technical solution disclosed herein. In order to make the technical means of this disclosure clearer and more understandable, to the point that those skilled in the art can implement it according to the contents of the specification, and in order to make the described and other objects, features and advantages of this disclosure more obvious and understandable, specific embodiments of this disclosure are illustrated below. Attached Figure Description

[0037] Various other advantages and benefits of this disclosure will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0038] Figure 1 This is a flowchart illustrating the nuclear power plant pump unit maintenance method based on deep reinforcement learning provided in this disclosure;

[0039] Figure 2 A schematic diagram illustrating the specific process of a nuclear power plant pump group maintenance method based on deep reinforcement learning in one embodiment of this disclosure;

[0040] Figure 3 This is a flowchart illustrating the process of performing deep reinforcement learning based on historical operating data of nuclear power plant pump units in one embodiment of this disclosure;

[0041] Figure 4 This is a schematic diagram illustrating the process of obtaining the optimal maintenance action based on actual operating data of the nuclear power plant pump unit in one embodiment of the present disclosure. Detailed Implementation

[0042] The following will be combined with the appendix Figures 1 to 4 The embodiments described herein are provided in detail and are intended to explain, rather than limit, this disclosure. While specific embodiments of this disclosure are shown in the accompanying drawings, it should be understood that this disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0043] It should be noted that certain terms are used in the specification and claims to refer to specific components. Those skilled in the art will understand that different terms may be used to refer to the same component. This specification and claims do not distinguish components based on differences in terminology, but rather on differences in function. The terms "comprising" or "including" used throughout the specification and claims are open-ended and should be interpreted as "comprising but not limited to." The following descriptions of preferred embodiments of this disclosure are for the purpose of implementing the general principles of the specification and are not intended to limit the scope of this disclosure. The scope of protection of this disclosure is determined by the appended claims.

[0044] To facilitate understanding of the embodiments of this disclosure, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments, and the accompanying drawings do not constitute a limitation on the embodiments of this disclosure.

[0045] A method for maintaining nuclear power plant pump units based on deep reinforcement learning, the flowchart of which can be found here. Figure 1 This includes the following steps:

[0046] Step S100: Obtain degradation index results based on historical operating data of nuclear power pump units, and then further establish a degradation model using stochastic processes based on the degradation index results, and construct the state space S;

[0047] Step S200: For different maintenance activity types of nuclear power plant pump units, define the maintenance action space as perfect maintenance a. pm and no action a N Construct action space A;

[0048] Step S300: Construct a cost function C that includes downtime penalty costs and maintenance operation costs. (t) This is used to obtain the reward function r;

[0049] Step S400: Based on the state space S, action space A and reward function r, complete the simulation of the nuclear power pump unit within the deep reinforcement learning framework, and use the deep reinforcement learning algorithm (PPO) to train the maintenance strategy network of the nuclear power pump unit to obtain the optimal maintenance strategy network model after training.

[0050] Step S500: Based on the real-time operating data of the nuclear power plant pump unit, extract the degradation index results, use them as input, and input them into the optimal maintenance strategy network model to output the optimal maintenance action of the nuclear power plant pump unit.

[0051] In one embodiment, the specific process of the nuclear power plant pump unit maintenance method based on deep reinforcement learning is described in [reference needed]. Figure 2 It includes:

[0052] Step S100: Obtain degradation index results based on historical operating data of nuclear power pump units, and then further establish a degradation model using stochastic processes based on the degradation index results, and construct the state space S;

[0053] Step S101: Obtain degradation index results based on historical operating data of the nuclear power plant pump unit;

[0054] In this embodiment, see Figure 3 The PHM detection system is used to acquire historical operating data of nuclear power plant pump units, such as the full life cycle data of vertical vibration acceleration of nuclear power plant pump bearings. Extract key indicators that characterize equipment degradation, such as root mean square error.

[0055] The formula for calculating the root mean square error is as follows:

[0056] ,

[0057] in, For the first The root mean square error of the data within a sampling period; F is the number of data collection points within a sampling period and F=32768.

[0058] Step S102: Based on the degradation index results, establish a degradation model using a stochastic process;

[0059] In this embodiment, based on the degradation index results extracted in step S101, the model parameters are obtained using the maximum likelihood estimation method in a Python environment, and a degradation model is established using a stochastic process. The expression of the degradation model is as follows:

[0060] ,

[0061] Among them, D i (t) represents the degradation state of the i-th pump when its service time is t; X θ (t) represents a random process; θ represents the parameters to be determined in the random process; FT i is the failure time of the i-th pump; w is the failure threshold.

[0062] Step S103: Construct the state space S;

[0063] For all N pumps in the entire cluster, the state space S is represented as:

[0064] ,

[0065] Among them, O i D is a Boolean variable indicating whether the i-th pump is operating normally. i Let be the degradation amount of the i-th pump.

[0066] Further, the O i Let O be a Boolean variable indicating whether the i-th pump is operating normally. Its value is determined by the failure threshold w. i The logic for determining the value is as follows:

[0067] ,

[0068] When the pump degradation is less than the failure threshold w, it is considered to be operating normally; otherwise, it is considered to be shut down and failed.

[0069] Step S200: For different maintenance activity types of nuclear power plant pump units, define the maintenance action space as perfect maintenance a. pm and no action a N Construct action space A;

[0070] In this embodiment, a maintenance action needs to be selected for each pump, and the action space A is defined as:

[0071] ,

[0072] Among them, a pm For perfect maintenance; a N No action is taken.

[0073] Furthermore, the perfect maintenance of a pm and no action a N This requires deep reinforcement learning to select one action as the maintenance behavior. The perfect maintenance of a... pm This refers to performing a thorough overhaul on the selected pump to restore its degradation to zero; no action a N This means that no operation is performed on the selected pump, and its state will evolve naturally according to the degradation model.

[0074] Step S300: Construct a cost function C that includes downtime penalty costs and maintenance operation costs. (t) This is used to obtain the reward function r;

[0075] Step S301: Calculate the downtime penalty cost ;

[0076] The shutdown penalty is used to quantify the power generation loss and safety risks caused by pump failure shutdowns and pump maintenance shutdowns. Its core design element is the penalty coefficient vector λ, which reflects the increasing marginal loss effect caused by the simultaneous failure of different numbers of pumps. .

[0077] Furthermore, the downtime penalty cost The calculation formula is:

[0078] ,

[0079] in, For a moment Downtime penalty costs; For a moment The number of pumps that perform perfect maintenance; C pm The cost of performing a perfect maintenance; Let t be the number of pumps that are in a stopped and failed state at time t.

[0080] Step S302: Calculate maintenance operation costs ;

[0081] The cost of the maintenance operation is calculated using the following formula.

[0082] ,

[0083] in, For a moment Maintenance costs, For a moment The number of pumps that have undergone perfect maintenance.

[0084] Step S303: Construct a cost function C based on downtime penalty cost and operation maintenance cost. (t) ;

[0085] Including downtime penalty costs and operational maintenance costs, the cost function C (t) It can be represented as:

[0086] ,

[0087] in, For a moment Downtime penalty costs; For a moment Maintenance costs.

[0088] Step S304: Based on cost function C (t) The reward function r is obtained;

[0089] In this embodiment, the reward function r is set to the negative of the cost function, that is, the expression for the reward function r is:

[0090] ,

[0091] Among them, C (t) The constructed cost function.

[0092] Step S400: Based on the state space S, action space A and reward function r, complete the simulation of the nuclear power pump unit within the deep reinforcement learning framework, and use the deep reinforcement learning algorithm (PPO) to train the maintenance strategy network of the nuclear power pump unit to obtain the optimal maintenance strategy network model after training.

[0093] In this embodiment, see Figure 3 The deep reinforcement learning framework includes: a simulation environment module based on historical data and stochastic processes, used to simulate pump group degradation, perform maintenance actions, and calculate rewards according to a cost function; and an agent module containing a policy network and a value network. This deep reinforcement learning framework uses the PPO algorithm to iteratively update network parameters in a stable manner using empirical data generated from the agent's interaction with the environment. Finally, the framework outputs a converged optimal maintenance policy network model, thus completing a closed loop from data modeling to intelligent decision-making policy generation.

[0094] Step S401: Complete the simulation of the nuclear power plant pump unit within the deep reinforcement learning framework;

[0095] In this embodiment, a simulation environment for operation and maintenance is constructed within a deep reinforcement learning framework based on the state space S, action space A, and reward function r of the nuclear power pump unit.

[0096] Furthermore, the simulation process of the nuclear power pump unit within the deep reinforcement learning framework includes:

[0097] In the Python environment, for a runtime of T and a degradation of T time steps, Pump i, based on a degradation model Its parameters Determine the increment of degradation over the time period [T, T+1]. The probability density function is then used to obtain random sampling based on the probability density function. Sample value The simulated value of the degradation at time step T+1 is expressed as:

[0098] .

[0099] Step S402: Train the maintenance strategy network of the nuclear power plant pump unit using a deep reinforcement learning algorithm (PPO);

[0100] In this embodiment, under the simulation conditions, the maintenance strategy network of the nuclear power plant pump unit is trained using the PPO algorithm with the goal of minimizing the cost function, based on the reinforcement learning framework consisting of the state space S, action space A, and reward function r.

[0101] Furthermore, when training the maintenance strategy network for the nuclear power plant pump unit using a deep reinforcement learning algorithm (PPO), a discount rate γ is set to process a series of maintenance behaviors. The overall objective is:

[0102] ,

[0103] Where C is the future long-term discount cost under the current maintenance strategy; t is the deep reinforcement learning time; and γ is the deep reinforcement learning discount coefficient.

[0104] In this embodiment, the training process of the PPO algorithm includes the following steps:

[0105] First, set the reinforcement learning hyperparameters, including the learning rates for the policy network and the value network. Discount factor for future rewards Parameters for the clipping range of the strategy update and the entropy regularization coefficient that enhances strong exploration capabilities .

[0106] Subsequently, two deep neural networks were initialized: one was a policy network. Used to output actions based on the current environmental state s The probability distribution; the second is the value network. , used to estimate the expected cumulative return of state s.

[0107] Furthermore, during the training process, see Figure 3 The agent interacts with the target environment: in each training episode, starting from the initial state, it follows the current policy network... The state at each time step t Sampling action After performing this action, receive the next state from the environment. and instant rewards and the experience of the quadruple Stored in the experience buffer.

[0108] Furthermore, based on the collected empirical data, the time difference advantage function for each step is calculated. (t) It is obtained through value network estimation:

[0109] ,

[0110] Furthermore, a PPO-tailored objective function is constructed. This function introduces the probability ratio between the old and new strategies. and apply it Interval pruning is used to limit the magnitude of each policy update, preventing training instability caused by sudden policy changes. The objective function is defined as follows:

[0111]

[0112] Furthermore, using this objective function, the policy network parameters are adjusted via the gradient ascent method. Optimization is performed; simultaneously, the value network parameters are updated by minimizing the mean squared error between the value network's predicted values ​​and actual returns. It is worth noting that, in order to improve sample utilization efficiency, this disclosure allows the reuse of trajectory data collected in the same round for policy updates multiple times, fixing the old policy parameters at the beginning of each update cycle. And after several iterations, synchronize it with the current policy parameters (i.e. ← ).

[0113] Finally, repeat the above interaction-update process until the policy converges.

[0114] In this embodiment, the final policy convergence maintenance policy network model is obtained. Network model that can be used as the optimal maintenance strategy It can be deployed in actual systems to output highly robust and high-reward action commands in real time under given observation conditions.

[0115] Step S500: Based on the real-time operating data of the nuclear power plant pump unit, extract the degradation index results, use them as input, and input them into the optimal maintenance strategy network model to output the optimal maintenance action of the nuclear power plant pump unit.

[0116] To better understand this disclosure, in a more specific embodiment, for a pump group (N=4) comprising four nuclear power pumps, under the simulation conditions, based on actual degradation data and operation and maintenance cost parameters, the optimal maintenance actions for the nuclear power pump group are output. A schematic diagram of the process is shown below. Figure 4 .

[0117] First, for the operation and maintenance process of a pump group with 4 pumps, a degradation model is established using a linear Wiener process with an exponential trend term, and its expression is as follows:

[0118] ,

[0119] Where a, b, and σ are parameters to be determined; W(t) represents standard Brownian motion, and the established degradation model includes the following parameters:

[0120] In this embodiment, the parameters a, b, σ, and w of the established degradation model are set as follows:

[0121] a=0.026; b=0.0651; σ=0.0258; w=8.3.

[0122] Set the cost of a single perfect maintenance C pm The cost is 500,000 yuan. Design a vector of downtime penalty coefficients. These correspond to the penalty weights for 0 to 4 pumps being unavailable. λ4=10 indicates that a complete shutdown would cause significant nuclear safety risks and economic losses.

[0123] In this embodiment, the nuclear power pump group (N=4) uses 600 inspection cycles as the maintenance decision time. In one maintenance decision, based on the optimal maintenance action output by the optimal maintenance strategy network model, pumps 1-4 were perfectly maintained at the times shown in the table.

[0124]

[0125] The above involves a total of 19 maintenance moments, which are defined as belonging to set M. Throughout the entire simulation cycle... In this case, the number of pump unit shutdowns due to failure is 0. Therefore, for... and Because no maintenance was performed, there are And the system did not experience any downtime events, therefore Therefore, .

[0126] for Only one pump was maintained at any given time, therefore The system cost 10,000 yuan and did not experience any system failures or shutdowns. However, maintenance on a single pump caused the corresponding pump to shut down.

[0127] =0.12×50=60,000 yuan,

[0128] Ten thousand yuan,

[0129] The total maintenance cost for this round of maintenance decisions is: Ten thousand yuan.

[0130] Therefore, comparing the costs of fixed-interval maintenance and post-maintenance, the cost of the optimal maintenance action is RMB 10.64 million, which is a cost reduction of approximately 62% compared to the cost of fixed-interval maintenance (RMB 28 million) and approximately 26.8% compared to the cost of post-maintenance (RMB 14.541 million).

[0131] Although the embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this disclosure is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the teachings of this specification and without departing from the scope of protection of the claims of this disclosure, and all of these are within the scope of protection of this disclosure.

Claims

1. A nuclear power plant pump unit maintenance method based on deep reinforcement learning, characterized in that, Includes the following steps: Step S100: Obtain degradation index results based on historical operating data of nuclear power pump units, and then further establish a degradation model using stochastic processes based on the degradation index results, and construct the state space S; Step S200: For different maintenance activity types of nuclear power plant pump units, define the maintenance action space as perfect maintenance a. pm and no action a N Construct action space A; Step S300: Construct a cost function C that includes downtime penalty costs and maintenance operation costs. (t) This is used to obtain the reward function r; Step S400: Based on the state space S, action space A and reward function r, complete the simulation of the nuclear power pump unit within the deep reinforcement learning framework, and use the deep reinforcement learning algorithm (PPO) to train the maintenance strategy network of the nuclear power pump unit to obtain the optimal maintenance strategy network model after training. Step S500: Based on the real-time operating data of the nuclear power plant pump unit, extract the degradation index results, use them as input, and input them into the optimal maintenance strategy network model to output the optimal maintenance action of the nuclear power plant pump unit.

2. The method as described in claim 1, characterized in that, Preferably, in step S100, the degradation model established using a stochastic process is represented as follows: ， Among them, D i (t) represents the degradation state of the i-th pump when its service time is t; X θ (t) represents a random process; θ represents the parameters to be determined in the random process; FT i is the failure time of the i-th pump; w is the failure threshold.

3. The method as described in claim 1, characterized in that, In step S100, for all N pumps in the entire pump group, the state space S is represented as: ， Among them, O i D is a Boolean variable indicating whether the i-th pump is operating normally. i Let be the degradation amount of the i-th pump.

4. The method as described in claim 3, characterized in that, The O i Let O be a Boolean variable indicating whether the i-th pump is operating normally. Its value is determined by the failure threshold w. i The logic for determining the value is as follows: ， Specifically, when the pump degradation is less than the failure threshold w, it is considered to be operating normally; otherwise, it is considered to be shut down and failed.

5. The method as described in claim 1, characterized in that, In step S200, the perfect maintenance a pm This refers to a thorough overhaul of the selected pump, reducing its degradation by a factor of D. i Restored to 0.

6. The method as described in claim 1, characterized in that, In step S200, the no-action a N This means that no operation is performed on the selected pump, and its state will evolve naturally according to the degradation model.

7. The method as described in claim 1, characterized in that, In step S300, a cost function C is constructed based on the downtime penalty cost and the operation maintenance cost. (t) Represented as: ， in, For a moment Downtime penalty costs; For a moment Maintenance costs.

8. The method as described in claim 7, characterized in that, The formula for calculating the downtime penalty cost is as follows: ， in, For a moment Downtime penalty costs; For a moment The number of pumps that perform perfect maintenance; C pm The cost of performing a perfect maintenance; Let t be the number of pumps that are in a stopped and failed state at time t.

9. The method as described in claim 7, characterized in that, The formula for calculating the maintenance cost of the action is as follows: ， in, For a moment Maintenance costs, For a moment The number of pumps that have undergone perfect maintenance.

10. The method as described in claim 1, characterized in that, In step S400, when training the maintenance strategy network of the nuclear power plant pump unit using the deep reinforcement learning algorithm (PPO), a discount rate γ is set to process a series of maintenance behaviors. The overall objective is: ， Where C is the future long-term discount cost under the current maintenance strategy; t is the deep reinforcement learning time; and γ is the deep reinforcement learning discount coefficient.

Images