Mechanical equipment maintenance and spare part decision-making method and system, medium and equipment

Abstract

The invention discloses a mechanical equipment maintenance and spare part decision-making method and system based on reinforcement learning, a medium and equipment, and the method comprises the steps: carrying out the noise reduction of continuous monitoring data of mechanical equipment through employing a principal component analysis method, fitting a degradation curve based on a Wiener process, and determining a failure threshold value; constructing a state space based on discrete states of equipment degradation and inventory, and constructing an action space based on maintenance actions and ordering opportunities; establishing a cost function based on the maintenance cost and the spare part management cost, and calculating an expected reward value; and by adopting a multi-agent DDQN reinforcement learning algorithm and taking the maximum expected reward value as an optimization target, obtaining optimal maintenance and spare part management behaviors in a discrete state of mechanical equipment degradation.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical system maintenance, and in particular to a mechanical equipment maintenance and spare parts decision-making method, system, medium and equipment based on reinforcement learning. Background Art

[0002] Joint decision-making regarding maintenance and spare parts is crucial for ensuring the safe and efficient operation of complex industrial systems. In modern industry, as mechanical equipment becomes increasingly complex, traditional maintenance strategies (corrective maintenance and scheduled maintenance) suffer from issues of insufficient or excessive maintenance, potentially leading to sudden equipment failures, insufficient spare parts supply, and high operational costs. Existing research has largely focused on developing either preventive maintenance planning models or spare parts management models, failing to comprehensively consider the interrelationships among various factors. This has limited the economic viability and applicability of these maintenance and spare parts strategies.

[0003] Traditional maintenance decision-making methods typically rely on constructing an objective function and optimizing it through heuristic algorithms. Common algorithms include ant colony optimization, differential evolution, and particle swarm optimization. However, these methods are primarily applicable to single-objective, single-variable decision-making scenarios. In complex mechanical equipment, due to the complex structures involved, maintenance and spare parts management require an effective trade-off between maintenance costs and system reliability. When faced with multivariable and multi-objective problems, traditional optimization techniques often encounter challenges such as large search space dimensions, high computational complexity, and difficulty in accurately modeling objective functions. In contrast, reinforcement learning can better handle multi-objective optimization problems, especially as the number of intelligent agents increases. It can avoid the curse of dimensionality and demonstrate greater advantages in complex decision-making optimization. Therefore, as a decision-making optimization technology, reinforcement learning has broader application prospects in multi-objective and complex systems.

[0004] Maintenance strategy formulation must consider not only component degradation but also spare part availability. In practice, excessive spare parts inventory leads to wasted inventory costs, while insufficient inventory can hinder the smooth implementation of maintenance strategies. Research on joint optimization problems has mostly employed threshold control strategies, which lack the flexibility to adjust spare parts ordering strategies based on component operating status. Joint optimization of maintenance decisions and spare parts management is a complex coupling problem, making it difficult to model using objective functions optimized by heuristic algorithms.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] The present invention provides a mechanical equipment maintenance and spare parts decision-making method, system, medium and equipment based on reinforcement learning, which introduces the multi-agent DDQN reinforcement learning method algorithm into the joint decision-making of maintenance and spare parts management, realizes the synchronous planning of maintenance planning and spare parts management, and further improves the safety and economy of the mechanical equipment health management process.

[0007] A reinforcement learning-based decision-making method for mechanical equipment maintenance and spare parts includes:

[0008] Step 1: After using principal component analysis to reduce the noise of the continuous monitoring data of mechanical equipment, the degradation curve is fitted based on the Wiener process and the failure threshold is determined;

[0009] Step 2: Construct a state space based on the discrete states of equipment degradation and inventory, and construct an action space based on maintenance actions and ordering opportunities;

[0010] Step 3: Establish a cost function based on maintenance cost and spare parts management cost, and calculate the expected reward value;

[0011] In step 4, a multi-agent DDQN reinforcement learning algorithm is used to maximize the expected reward value and obtain the optimal maintenance and spare parts management behavior under the discrete state of mechanical equipment degradation.

[0012] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, in step 1, the principal component analysis method is used to perform noise reduction processing on the continuous monitoring of sensors such as temperature, pressure and speed that can reflect the changes in equipment performance, and the one-dimensional signal x = [x1, x2, ..., x N ] T Converted to the time-delay matrix X, the expression is:

[0013]

[0014] Where n=N-m+1, m is the sliding window, and N is the total number of sampling points contained in the one-dimensional signal x.

[0015] Perform SVD decomposition on the delay matrix X:

[0016] X=U∑V T

[0017] Where U is the left singular vector matrix; ∑ is the diagonal matrix; V is the right singular vector matrix.

[0018] The largest singular value σ of the first k diagonal matrices is retained, and the others are regarded as noise and set to zero. The expression is:

[0019] ∑=diag(σ1,...,σ k , 0, ..., 0)

[0020] The matrix after denoising is X denoised =U∑V T , restored to a one-dimensional signal, the expression is:

[0021]

[0022] In the formula [X denoised ] j，i-j+1 represents the element in the jth row and the i-j+1th column in the matrix, w i Indicates the number of elements involved in the reconstruction average;

[0023] The Wiener process is used to fit the preprocessed signal, and the degradation increment of the equipment degradation function is expressed as:

[0024] ΔX(t)=X(t+Δt)-X(t)~N(μΔt,σ 2 Δt),

[0025] Where X(t) is the equipment degradation value, ΔX(t) is the degradation increment, N(·) is the normal distribution, μ represents the drift coefficient, σ represents the diffusion coefficient, and t is the time interval t~t+Δt (Δt>0);

[0026] The cumulative distribution function expression of ΔX(t) is:

[0027]

[0028] The failure threshold F of the equipment degradation value is set based on the equipment maintenance log and the expert opinions of equipment maintenance personnel.

[0029] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, in step 2, the state s t Contains equipment degradation value X(t) and spare parts inventory h t , the expression is as follows:

[0030] s t =[X1(t),...,X N (t), h t ],

[0031] Where N is the total number of equipment,

[0032] Action space A contains maintenance tasks and spare parts ordering Two actions. Maintenance methods include no maintenance, failure replacement and preventive replacement. Action set Indicates the time of mechanical equipment replacement, expressed as:

[0033]

[0034] Where D is the maximum number of cycles before the next check, if Indicates in Replace all equipment after a period of Indicates that no maintenance will be arranged in the next D cycles.

[0035] Spare parts order quantity is The maximum quantity ordered at one time shall not exceed the total number of equipment N. Considering that the spare parts transportation takes k cycles, the spare parts delivery time is

[0036] In the aforementioned reinforcement learning-based mechanical equipment maintenance and spare parts decision-making method, in step 3, maintenance decision-making and spare parts management are two independent tasks, and maintenance costs and spare parts management costs are modeled separately. Maintenance costs include inspection costs, replacement costs, and downtime costs. Inspection costs: Before formulating a maintenance plan, the operating status of the mechanical equipment is inspected. The cost of mechanical equipment status inspection c i ; Replacement cost: The replacement cost of mechanical equipment before failure is c rep When the degradation value of the equipment exceeds the failure threshold, that is, the mechanical equipment fails, the cost increases with the difficulty of maintenance, and the replacement cost is c fail ; Downtime loss cost: Faulty mechanical equipment will incur downtime costs if it is not repaired in time, and additional costs will be incurred each cycle c down , in summary, considering sufficient inventory h t ≥N and insufficient inventory h t When <N, the reward value The sub-formula is as follows:

[0037]

[0038] Where, is an indicative function. When the equipment is replaced, the value is 1, otherwise it is 0. The spare parts management cost includes ordering cost and inventory cost. Among them, the spare parts ordering cost is: Where c0 is the fixed cost of each order, c u is the price of a single spare part; inventory cost: the inventory cost of mechanical equipment is c h The inventory cost per unit time for each spare part; Emergency ordering cost: When there is no spare part reserve, additional emergency ordering costs will be incurred. urg , reward value The piecewise formula is as follows:

[0039]

[0040] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, in step 4, the step of constructing a multi-agent DDQN reinforcement learning network algorithm model includes:

[0041] Step 4.1. First, define the agent's state space S, action space A, and reward function R. The state space S contains the equipment degradation value and the current spare parts inventory information, while the action space A contains the maintenance strategy and spare parts ordering policy. The reward function R provides feedback to the agent by evaluating the contribution of each decision to the overall system benefit.

[0042] Step 4.2: Under the multi-agent framework of reinforcement learning, the model is composed of a maintenance agent and a spare parts agent, which optimize system maintenance and inventory management through collaborative decision-making. The input of each agent is the state observation s t After being processed by a four-layer fully connected network and a ReLU activation layer, a probability distribution of discrete actions is output. The Softmax function is used to sample the probability distribution to determine the final action. The output action of the maintenance agent is the timing of maintenance, and the output action of the spare parts agent is the number of spare parts to be ordered. In this process, the two agents process their own states separately and share state information to improve the collaborative effect of decision-making. The action execution interacts with the environment, and the joint action of the agents a = (a act ,a spa ) will act on the environment. If the maintenance agent chooses to repair, the degradation value of the equipment will be reset to 0 at the next time step t+1, and the spare parts in the inventory will be consumed. If the inventory is insufficient, the repair will be delayed and a downtime penalty will be incurred. The ordering action of the spare parts agent will trigger a supply chain response. New spare parts will be put into the inventory after a fixed lead time. During this period, the inventory level will change dynamically and may incur holding costs or stock-out costs.

[0043] In step 4.3, each agent estimates its Q-value function using DDQN. DDQN addresses the overestimation problem in traditional Q-learning, improving decision stability and accuracy. Each agent's Q-value function is updated using an experience replay mechanism, training on a globally shared experience pool to improve sample efficiency.

[0044] Step 4.4: During the training process, the neural network of each agent consists of multiple fully connected layers, and the input of the network is the feature vector s of the current state. t, outputting the Q-value for each action. Training proceeds through multiple iterations, with each iteration updating the Q-value function using batches of data randomly sampled from the experience replay pool. To ensure training convergence, a maximum number of iterations is set at 500, and model performance is evaluated after each iteration. When the observed loss function falls below the predetermined convergence criterion, training is stopped and the model is considered converged. If the model fails to converge during training, an early stopping strategy is implemented. When the loss function no longer decreases significantly over a period of time, training is stopped early and the best model is selected to ensure stable convergence of the neural network within the given number of training cycles.

[0045] like Figure 2 As shown, in the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, the steps of training the reinforcement learning network model based on the multi-agent DDQN algorithm include:

[0046] 1) First, initialize the hyperparameters. Then, determine the degradation function of the mechanical device and use it as the solution environment of the model. Initialize the environment and the agent inputs the state s at each time step t. t , take action a t , receive reward r t , and transfer to the new state s t+1 ;

[0047] 2) The state value is input into the strategy Q value network. After being processed by the four-layer fully connected network, ReLU activation layer and softmax function, the probability distribution of discrete actions is output. The action of agent i at time t is obtained by random sampling.

[0048] 3) Each agent takes corresponding actions at time t After that, the environment will change accordingly, and the environment returns the reward value r at time t+1 t , cost value c t and the new state value s t+1 ;

[0049] 4) Storage experience(s) t , a t , r t , s t+1 ) In the playback buffer, randomly sample a batch of data from the playback buffer to train the model;

[0050] 5) DQN uses a neural network to approximate the Q value function, and the input is the state s t , the output is for each action a t Q value, loss function:

[0051] L(θ)=E[(r t +γmax a′ Q(s t+1 , a′;θ- )-Q(s t , a t ;θ)) 2 ]

[0052] Where θ is the parameter of the neural network, θ - is the parameter of the target network, γ∈[0,1] is the discount rate, the larger the value, the more attention is paid to future rewards.

[0053] Combined with the loss function, the gradient descent method is used to update the model.

[0054] DDQN reduces the overestimation problem of Q value by decoupling action selection and action evaluation, and updates the formula:

[0055] Q(s t , a t )←Q(s t , a t )+α[r t +γQ(s t+1 , arg max a′ Q(s t+1 , a′;θ);θ - )-Q(s t , a t ;θ)]

[0056] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, the mechanical equipment includes an engine.

[0057] A system for implementing the method includes:

[0058] An acquisition module, which fits the continuous monitoring data reflecting the performance of the mechanical equipment into a degradation curve based on the Wiener process and determines the failure threshold of the mechanical equipment;

[0059] A state space generation module that constructs a state space based on the discrete states of equipment degradation and inventory, and an action space based on maintenance actions and ordering opportunities;

[0060] A cost function generation module, which establishes a cost function based on the maintenance cost and the spare parts management cost and calculates the expected reward value;

[0061] A multi-agent DDQN reinforcement learning algorithm model uses the multi-agent DDQN reinforcement learning algorithm to maximize the expected reward value as the optimization goal, and obtains the optimal maintenance and spare parts management behavior under the discrete state of mechanical equipment degradation.

[0062] A computer storage medium includes computer instructions, which, when executed on a computer, cause the computer to execute the method described above.

[0063] An electronic device, comprising:

[0064] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein:

[0065] When the processor executes the program, the method described is implemented.

[0066] Compared with the existing technology, the present invention has the following advantages: The present invention synchronously plans maintenance and spare parts management. Using multi-agent DDQN-agent reinforcement learning technology, two agents make decisions on maintenance timing and spare parts management respectively, effectively overcoming the dimensional disaster problem encountered by traditional optimization algorithms when dealing with multi-variable and multi-objective optimization. Maintenance based on equipment operating status. Adopting a dynamic maintenance and spare parts decision-making method based on the degradation state of the components effectively alleviates the problems of intensive maintenance, high maintenance costs and insufficient spare parts supply in traditional regular maintenance technologies, reduces dependence on expert experience and expertise, and promotes the upgrade of maintenance strategies from time-based maintenance to state-based maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Various other advantages and benefits of the present invention will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are intended only to illustrate preferred embodiments and are not to be construed as limiting the present invention. It should be understood that the drawings described below are merely examples of the present invention, and that those skilled in the art will be able to derive other drawings from these drawings without inventive effort. Throughout the drawings, identical reference numerals are used to denote identical components.

[0068] In the attached figure:

[0069] Figure 1 This is a flowchart of a joint decision-making technology for mechanical equipment maintenance and spare parts based on reinforcement learning of the present invention;

[0070] Figure 2 This is a basic framework diagram of a multi-agent DDQN network in an embodiment of the present invention;

[0071] Figure 3 This is a model convergence curve diagram in an embodiment of the present invention.

[0072] The present invention will be further explained below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0073] Specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although specific embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0074] It should be noted that certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. This specification and claims do not use the difference in nouns as a way to distinguish components, but use the difference in the functions of the components as the criterion for distinction. As mentioned throughout the specification and claims, "including" or "comprising" is an open term, so it should be interpreted as "including but not limited to". The subsequent description of the specification is a preferred embodiment of the present invention, but the description is based on the general principles of the specification and is not intended to limit the scope of the invention. The scope of protection of the present invention shall be as defined in the attached claims.

[0075] To facilitate understanding of the embodiments of the present invention, further explanation will be given below using specific embodiments as examples in conjunction with the accompanying drawings, and the accompanying drawings do not constitute a limitation on the embodiments of the present invention.

[0076] like Figures 1 to 3 As shown in Figure 2, the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning includes the following steps:

[0077] Step 1: Fitting the continuous monitoring data reflecting the performance of the mechanical equipment into a degradation curve based on the Wiener process to determine the failure threshold of the mechanical equipment;

[0078] Step 2: Construct a state space based on the discrete states of equipment degradation and inventory, and construct an action space based on maintenance actions and ordering opportunities;

[0079] Step 3: Establish a cost function based on maintenance cost and spare parts management cost, and calculate the expected reward value;

[0080] In step 4, a multi-agent DDQN reinforcement learning algorithm is used to maximize the expected reward value and obtain the optimal maintenance and spare parts management behavior under the discrete state of mechanical equipment degradation.

[0081] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, in step 1, the principal component analysis method is used to perform noise reduction processing on the continuous monitoring of sensors such as temperature, pressure and speed that can reflect the changes in equipment performance, and the one-dimensional signal x = [x1, x2, ..., x N ] T Converted to the time-delay matrix X, the expression is:

[0082]

[0083] Where n=N-m+1, m is the sliding window, and N is the total number of sampling points contained in the one-dimensional signal x.

[0084] Perform SVD decomposition on matrix X:

[0085] X=U∑V T

[0086] Where U is the left singular vector matrix; ∑ is the diagonal matrix; V is the right singular vector matrix.

[0087] The largest singular value σ of the first k diagonal matrices is retained, and the others are regarded as noise and set to zero. The expression is:

[0088] ∑=diag(σ1,...,σ k , 0, ..., 0)

[0089] The matrix after denoising is X denoised =U∑V T , restored to a one-dimensional signal, the expression is:

[0090]

[0091] In the formula [X denoised ] j，i-j+1 represents the element in the jth row and the i-j+1th column in the matrix, w i Indicates the number of elements involved in the reconstruction average;

[0092] The Wiener process is used to fit the preprocessed signal, and the degradation increment of the equipment degradation function is expressed as:

[0093] ΔX(t)=X(t+Δt)-X(t)~N(μΔt,σ 2 Δt),

[0094] Where X(t) is the equipment degradation value, ΔX(t) is the degradation increment, N(·) is the normal distribution, μ represents the drift coefficient, σ represents the diffusion coefficient, and t is the time interval t~t+Δt (Δt>0);

[0095] The cumulative distribution function expression of ΔX(t) is:

[0096]

[0097] Set the failure threshold Fstatus of the equipment degradation value based on the equipment maintenance log and the expert opinions of the equipment maintenance personnel t Contains equipment degradation value X(t) and spare parts inventory h t , the expression is as follows:

[0098] s t =[X1(t),...,X N (t), h t ],

[0099] Where N is the total number of equipment,

[0100] Action space A contains maintenance tasks and spare parts ordering Two actions. Maintenance methods include no maintenance, failure replacement and preventive replacement. Action set Indicates the time of mechanical equipment replacement, expressed as:

[0101]

[0102] Where D is the maximum number of cycles before the next check, if Indicates in Replace all equipment after a period of Indicates that no maintenance will be arranged in the next D cycles.

[0103] Spare parts order quantity is The maximum quantity ordered at one time shall not exceed the total number of equipment N. Considering that the spare parts transportation takes k cycles, the spare parts delivery time is

[0104] In the aforementioned reinforcement learning-based mechanical equipment maintenance and spare parts decision-making method, in step 3, maintenance decision-making and spare parts management are two independent tasks, and maintenance costs and spare parts management costs are modeled separately. Maintenance costs include inspection costs, replacement costs, and downtime costs. Inspection costs: Before formulating a maintenance plan, the operating status of the mechanical equipment is inspected to determine its RUL distribution. The status inspection cost of the mechanical equipment c i ; Replacement cost: The replacement cost of mechanical equipment before failure is c rep When the degradation value of the equipment exceeds the failure threshold, that is, the mechanical equipment fails, the cost increases with the difficulty of maintenance, and the replacement cost is c fail ; Downtime loss cost: Faulty mechanical equipment will incur downtime costs if it is not repaired in time, and additional costs will be incurred each cycle c down , in summary, considering sufficient inventory h t ≥N and insufficient inventory h t When <N, the reward value The piecewise formula is as follows:

[0105]

[0106] Where, It is an indicator function. When the device is replaced, the value is 1, otherwise it is 0

[0107] Spare parts management costs include ordering costs and inventory costs, among which spare parts ordering costs: ordering costs are Where c0 is the fixed cost of each order, c u is the price of a single spare part; inventory cost: the inventory cost of mechanical equipment is c h The inventory cost per unit time for each spare part; Emergency ordering cost: When there is no spare part reserve, additional emergency ordering costs will be incurred. urg , reward value The piecewise formula is as follows:

[0108]

[0109] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, in step 4, the steps of constructing a multi-agent DDQN reinforcement learning network algorithm model include:

[0110] Step 4.1. First, define the agent's state space S, action space A, and reward function R. The state space S contains the equipment degradation value and the current spare parts inventory information, while the action space A contains the maintenance strategy and spare parts ordering policy. The reward function R provides feedback to the agent by evaluating the contribution of each decision to the overall system benefit.

[0111] Step 4.2: Under the multi-agent framework of reinforcement learning, the model is composed of a maintenance agent and a spare parts agent, which optimize system maintenance and inventory management through collaborative decision-making. The input of each agent is the state observation s t After being processed by a four-layer fully connected network and a ReLU activation layer, a probability distribution of discrete actions is output. The Softmax function is used to sample the probability distribution to determine the final action. The output action of the maintenance agent is the timing of maintenance, and the output action of the spare parts agent is the number of spare parts to be ordered. In this process, the two agents process their own states separately and share state information to improve the collaborative effect of decision-making. The action execution interacts with the environment, and the joint action of the agents a = (a act , a spa ) will act on the environment. If the maintenance agent chooses to repair, the degradation value of the equipment will be reset to 0 at the next time step t+1, and the spare parts in the inventory will be consumed. If the inventory is insufficient, the repair will be delayed and a downtime penalty will be incurred. The ordering action of the spare parts agent will trigger a supply chain response. New spare parts will be put into the inventory after a fixed lead time. During this period, the inventory level will change dynamically and may incur holding costs or stock-out costs.

[0112] Step 4.3: Each agent estimates its Q-value function using DDQN. DDQN addresses the overestimation problem in traditional Q-learning, improving decision stability and accuracy. Each agent's Q-value function is updated using an experience replay mechanism, training on a globally shared experience pool to improve sample efficiency.

[0113] Step 4.4: During training, each agent's neural network consists of multiple fully connected layers. The network's input is the feature vector of the current state, and its output is the Q-value of each action. Training is performed over multiple iterations, with each iteration updating the Q-value function using batches of data randomly sampled from the experience replay pool. To ensure training convergence, a maximum number of iterations is set to 500, and model performance is evaluated after each iteration. When the observed loss function falls below the predetermined convergence criterion, training is stopped and the model is considered to have converged. If the model fails to converge during training, an early stopping strategy is adopted. When the loss function no longer decreases significantly over a period of time, training is stopped early and the best model is selected to ensure that the neural network converges stably within a given number of training cycles.

[0114] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, Figure 2 As shown, the steps for training the reinforcement learning network model based on the multi-agent DDQN algorithm include:

[0115] 1) First, initialize the hyperparameters. Then, determine the degradation function of the mechanical device and use it as the solution environment of the model. Initialize the environment and the agent inputs the state s at each time step t. t , take action a t , receive reward r t , and transfer to the new state s t+1 ;

[0116] 2) The state value is input into the strategy Q value network. After being processed by the four-layer fully connected network, ReLU activation layer and softmax function, the probability distribution of discrete actions is output. The action of agent i at time t is obtained by random sampling.

[0117] 3) Each agent takes corresponding actions at time t After that, the environment will change accordingly, and the environment returns the reward value r at time t+1 t , cost value c t and the new state value s t+1 ;

[0118] 4) Storage experience(s) t , a t , r t , s t+1) In the playback buffer, randomly sample a batch of data from the playback buffer to train the model;

[0119] 5) DQN uses a neural network to approximate the Q value function, and the input is the state s t , the output is for each action a t Q value, loss function:

[0120] L(θ)=E[(r t +γmax a′ Q(s t+1 , a′;θ - )-Q(s t , a t ;θ)) 2 ]

[0121] Where θ is the parameter of the neural network, θ is the parameter of the target network, and γ∈[0,1] is the discount rate. The larger the value, the more attention is paid to future rewards.

[0122] Combined with the loss function, the gradient descent method is used to update the model.

[0123] DDQN reduces the overestimation problem of Q value by decoupling action selection and action evaluation, and updates the formula:

[0124] Q(s t , a t )←Q(s t , a t )+α[r t +γQ(s t+1 ,arg maX a′ Q(s t+1 , a′;θ);θ - )-Q(s t , a t ;θ)].

[0125] In the mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, the mechanical equipment includes an engine.

[0126] A system for implementing the method includes:

[0127] An acquisition module, which fits the continuous monitoring data reflecting the performance of the mechanical equipment into a degradation curve based on the Wiener process and determines the failure threshold of the mechanical equipment;

[0128] A state space generation module that constructs a state space based on the discrete states of equipment degradation and inventory, and an action space based on maintenance actions and ordering opportunities;

[0129] A cost function generation module, which establishes a cost function based on the maintenance cost and the spare parts management cost and calculates the expected reward value;

[0130] A multi-agent DDQN reinforcement learning algorithm model uses the multi-agent DDQN reinforcement learning algorithm to maximize the expected reward value as the optimization goal, and obtains the optimal maintenance and spare parts management behavior under the discrete state of mechanical equipment degradation.

[0131] In one embodiment, Figure 1 As shown, the method includes, first, performing noise reduction processing on historical monitoring data and establishing a degradation model for mechanical equipment based on the Wiener process; second, constructing a state space based on the discrete states of equipment degradation and inventory, and constructing an action space for maintenance actions and ordering mechanisms; then, establishing a cost function based on maintenance costs and spare parts management costs, and calculating the expected reward value; finally, using a multi-agent DDQN reinforcement learning algorithm with the maximum expected reward as the optimization goal to obtain the optimal maintenance and spare parts management behavior under the discrete state of equipment degradation. A numerical example is constructed using a civil aircraft with four aircraft engines as the target object. The method includes the following specific steps:

[0132] Step 1: Based on the Wiener process, the continuous monitoring data reflecting the engine performance is fitted into a degradation curve to determine the equipment failure threshold.

[0133] The engine's performance is continuously monitored by sensors such as temperature, pressure, and speed. Principal component analysis (PCA) is used to reduce the dimensionality and denoise the monitored data. The fault threshold F is calculated by taking the average of the maximum values ​​of all first principal components. The engine degradation curve is then fitted to the first principal component using a Wiener process. Subsequently, μ and σ are estimated using maximum likelihood estimation, as shown in Table 1.

[0134] Table 1 Wiener process parameter evaluation

[0135] parameter μ σ F estimate 0.073 0.183 11.892

[0136] Status t Contains equipment degradation value X(t), and spare parts inventory h t , the maximum inventory is tentatively set to twice the total amount of equipment. The expression is as follows:

[0137] s t =[X1(t),X2(t),X3(t),X4(t),h t ]

[0138] Where s t is the operating status of the four engines at time t.

[0139] Step 2: Construct a state space based on the discrete states of equipment degradation and inventory, and construct an action space based on maintenance actions and ordering opportunities.

[0140] Maintenance timing for maintenance agents The traditional aircraft engine maintenance management model is to dismantle the engines that need maintenance. Then, these engines are sent to the workshop for maintenance and a new or repaired aircraft engine is installed. The repaired engine is used as a spare engine. Therefore, the action set Indicates the time of engine replacement, which can be expressed as:

[0141]

[0142] Where D is the maximum number of flight cycles before the next inspection, if Indicates in Replace the engine after a flight cycle if Indicates that no maintenance is scheduled within the next D flight cycles.

[0143] Spare parts order quantity for the spare parts agent In order to reduce inventory costs, the maximum quantity ordered at one time should not exceed the total number of engines N, that is, At the same time, considering that the spare parts transportation takes k flight cycles, the spare parts delivery time is

[0144] Step 3: Establish a cost function through maintenance cost and spare parts management cost, and calculate the expected reward value.

[0145] In multi-agent reinforcement learning, the setting of the reward function is closely related to the decision-making level of the agent. In order to avoid adverse interference between agents, this paper designs a unique reward function based on the characteristics of each agent.

[0146] For the maintenance agent, we set the reward function under the premise that there are sufficient spare parts Maintenance costs include inspection costs, replacement costs and downtime costs. (1) Inspection costs: Before making a maintenance plan, it is necessary to inspect the engine's operating status. The cost of engine status inspection is i ; (2) Replacement cost: The replacement cost before engine failure is c rep After an engine failure occurs, the cost increases as the difficulty of repair increases. The replacement cost is c fail (3) Downtime loss cost: If the faulty engine is not repaired in time, it will incur downtime cost, and each flight cycle will generate additional cost c down In summary, the reward value The formula is as follows:

[0147]

[0148] Where, Is an indicative function. When the device is replaced, the value is 1, otherwise it is.

[0149] For the spare parts agent, the reward function It is related to the order quantity, total inventory and maintenance plan. Spare parts management cost includes ordering cost and inventory cost. (1) Spare parts ordering cost: The ordering cost is Where c0 is the fixed cost of each order, c u is the price of a single spare part; (2) Inventory cost: The engine inventory cost is c h The inventory cost per unit time for each spare part; (3) Emergency ordering cost: When there is no spare part reserve, additional emergency ordering costs will be incurred. urg In summary, the reward value The piecewise formula is as follows:

[0150]

[0151] Due to limited public data on various aircraft engine maintenance costs, this study aims to develop an effective method applicable to real-world scenarios. To address this challenge, basic maintenance parameters and associated costs were obtained from relevant research. This paper sets cost parameters based on the proportional relationship between various costs. Table 2 lists the maintenance parameters and associated costs for aircraft engines during actual operation and maintenance.

[0152] Table 2 Maintenance parameters and maintenance costs

[0153] parameter Numerical <![CDATA[Detection cost c i > 2 <![CDATA[Preventive replacement cost c rep > 35 <![CDATA[Post-maintenance cost c fail > 165 <![CDATA[Shutdown cost c down > 16 <![CDATA[Ordering fixed cost c0]]> 10 <![CDATA[Unit price of spare part c u > 230 <![CDATA[Inventory cost per unit time c h > 1 <![CDATA[Emergency order cost c e > 285 Spare parts transportation time (k) 3

[0154] Step 4: Use the multi-agent DDQN reinforcement learning algorithm to maximize the expected reward as the optimization goal to obtain the optimal maintenance and spare parts management plan under the discrete state of equipment degradation. The present invention uses the DDQN reinforcement learning algorithm containing two agents to form a multi-agent framework. The basic framework of the multi-agent DDQN network is as follows: Figure 2 shown.

[0155] The multi-agent DDQN reinforcement learning model framework consists of four basic elements: state, action, reward function, and value function. The objective function of the reward function reflects the goal of reducing maintenance and spare parts management costs and avoiding excessive maintenance that leads to waste of maintenance resources.

[0156] Training a reinforcement learning network model based on the multi-agent DDQN algorithm. The specific steps include:

[0157] 1) Initialize the hyperparameters, then determine the engine degradation function and use it as the solution environment for the model, and initialize the environment. The agent observes the state s at each time step t t , take action a t , receive reward r t , and transfer to the new state s t+1 ;

[0158] 2) The state value is input into the strategy Q value network. After being processed by the four-layer fully connected network, ReLU activation layer and softmax function, the probability distribution of discrete actions is output. The action of agent i at time t is obtained by random sampling.

[0159] 3) Each agent takes corresponding actions at time t After that, the environment will change accordingly, and the environment returns the reward value r at time t+1 t , cost value c t and the new state value s t+1 ;

[0160] 4) Storage experience(s) t , a t , r t , s t+1 ) in the replay buffer. Randomly sample a batch of data from the replay buffer to train the model.

[0161] 5) DQN uses a neural network to approximate the Q-value function. The input is the state s t , the output is for each action a t Q value, loss function:

[0162] L(θ)=E[(r t +γmax a′ Q(s t+1 , a′;θ - )-Q(s t , a t ;θ)) 2 ]

[0163] Where θ is the parameter of the neural network, θ - is the parameter of the target network, γ∈[0,1] is the discount rate, and the larger the value, the more importance is attached to future rewards.

[0164] Combined with the loss function, the gradient descent method is used to update the model.

[0165] DDQN reduces the overestimation problem of Q value by decoupling action selection and action evaluation, and updates the formula:

[0166] Q(s t , a t )←Q(s t , a t )+α[r t +γQ(s t+1 , argmax a′ Q(s t+1 , a′;θ);θ - )-Q(st , d ， ;θ)]

[0167] 6) Use the gradient descent algorithm to derive the loss function and update the value function parameters of each agent.

[0168] Except for the output layer, both the maintenance agent and the spare parts agent use a two-layer fully connected neural network, the structure of which is shown in Table 3. For the maintenance agent, L = D; for the spare parts agent, L = 4. The hyperparameters of the maintenance and spare parts agents are the same: learning rate α = 0.0005, and discount rate γ = 0.99.

[0169] Table 3 Network structure

[0170] Network structure feature Input layer 1×5 Fully connected layer 31×64 Activate Layer ReLU Fully connected layer 64×64 Activate Layer ReLU Output layer 64×L

[0171] The MARL agent was trained using DDQN. One training epoch consisted of 1,000 flight cycles, for a total of 500 training epochs. Both agents achieved convergence after 100 iterations. The maximum maintenance interval D for each decision step was determined according to the aircraft maintenance manual and was set to 30 in this study. Figure 3 The convergence curve of the model is shown.

[0172] Based on the engine's operating status and inventory information, a multi-agent DDQN reinforcement learning approach is used to develop a cost-optimal maintenance and spare parts management plan for each overhaul node. A typical example is shown in Table 4.

[0173] Table 4 Final maintenance plan corresponding to the optimal solution

[0174] cycle <![CDATA[s t ]]> <![CDATA[s t+1 ]]> Replacement timing Order quantity Total cost 30 [0，0，0，0] [1.65，2.94，1.78，1.71，0] No maintenance 0 2 60 [1.65，2.94，1.78，1.71，0] [4.23，4.92，3.81，4.21，0] No maintenance 0 4 90 [4.23，4.92，3.81，4.21，0] [6.32，5.92，4.4，4.76，0] No maintenance 0 6 120 [6.32，6.92，6.44，4.76，0] [7.98，6.07，513，5.71，0] No maintenance 0 8 136 [7.98，10.07，8.13，5.71，0] [0，0，0，0，0] 16 4 1080

[0175] The proposed method is compared with traditional post-maintenance and scheduled maintenance. All strategies are tested on a dataset of 10 episodes, each containing 1000 flight cycles. The following five performance indicators are used: (i) Total cost: maintenance cost and spare parts management cost. (ii) Number of unplanned maintenance: This is a direct indicator of maintenance reliability, aiming to avoid unplanned engine downtime. (iii) Total replacement number: The number of planned and unplanned engine replacements. A lower total replacement number means that each engine has a longer working time. (iv) Inventory cost: A lower storage volume of spare parts can reduce inventory costs and alleviate spare parts redundancy. (v) Emergency ordering cost: This is an indicator to measure the effectiveness of spare parts management. When the inventory is insufficient, the maintenance strategy cannot be carried out smoothly, and additional ordering costs will be incurred.

[0176] The performance metrics of the different strategies are shown in Table 5. The multi-agent reinforcement learning approach based on engine degradation outperformed the other strategies, particularly in terms of total cost, number of unplanned replacements, and inventory cost. It significantly reduced the number of overhauls. By introducing incomplete repairs, the number of engine replacements was reduced, improving the economic benefits of maintenance and ensuring reliable engine operation.

[0177] Table 5 Performance indicators of different strategies

[0178] Total cost Number of unplanned replacements Total replacement quantity Inventory costs Post-maintenance 30774.1(0.0％) 25.6(0.0％) 33.6(0.0％) 7966.8(0.0％) Regular maintenance 23389.2(-23.9％) 7.2(-78.6％) 49.2(+46.4％) 7942.8(-0.3％) Ideal repair 9196.4(-70.1％) 0.0(-100.0％) 33.2(-1.2％) 0.0(-100.0％) Proposed method 10115.2(-67.1％) 0.8(-97.6％) 32.8(-2.3％) 130.5(-98.4％)

[0179] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. A mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning, characterized in that: The steps include: Step 1: After using principal component analysis to reduce the noise of the continuous monitoring data of mechanical equipment, the degradation curve is fitted based on the Wiener process and the failure threshold is determined; Step 2: Construct a state space based on the discrete states of equipment degradation and inventory, and construct an action space based on maintenance actions and ordering opportunities; Step 3: Establish a cost function based on maintenance cost and spare parts management cost, and calculate the expected reward value; In step 4, a multi-agent DDQN reinforcement learning algorithm is used to maximize the expected reward value and obtain the optimal maintenance and spare parts management behavior under the discrete state of mechanical equipment degradation.

2. The mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning according to claim 1 is characterized in that: Preferably, in step 1, the principal component analysis method is used to perform noise reduction processing on the temperature, pressure and speed sensor continuous monitoring data of the performance change of the reaction equipment, and the one-dimensional signal x=[x1, x2, ..., x N ] T Converted to the time-delay matrix X, the expression is: Where n = N-m+1, m is the sliding window, and N is the total number of sampling points contained in the one-dimensional signal x; Perform SVD decomposition on the delay matrix X: X=U∑V T Where U is the left singular vector matrix; ∑ is the diagonal matrix; V is the right singular vector matrix; The largest singular value σ of the first k diagonal matrices is retained, and the others are regarded as noise and set to zero. The expression is: ∑=diag(σ1,...,σ k ,0,...,0) The matrix after denoising is X denoised =U∑V T , restored to a one-dimensional signal, the expression is: In the formula [X denoised ] j，i-j+1 represents the element in the jth row and the i-j+1th column in the matrix, w i Indicates the number of elements involved in the reconstruction average; The Wiener process is used to fit the preprocessed signal, and the degradation increment of the equipment degradation function is expressed as: ΔX(t)=X(t+Δt)-X(t)~N(μΔt,σ 2 Δt), Where X(t) is the equipment degradation value, ΔX(t) is the degradation increment, N(·) is the normal distribution, μ represents the drift coefficient, σ represents the diffusion coefficient, and t is the time interval t~t+Δt (Δt>0); The cumulative distribution function expression of ΔX(t) is: The failure threshold F of the equipment degradation value is set based on the equipment maintenance log and the expert opinions of equipment maintenance personnel.

3. The mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning according to claim 1 is characterized in that: In step 2, state s t Contains equipment degradation value X(t) and spare parts inventory h t , the expression is as follows: s t =[X1(t),...,X N (t),h t ], Where N is the total number of equipment, Action space A contains maintenance tasks and spare parts ordering Two actions, maintenance methods include no maintenance, failure replacement and preventive replacement, action set Indicates the time of mechanical equipment replacement, expressed as: Where D is the maximum number of cycles before the next check, if Indicates in Replace all equipment after a period of Indicates that no maintenance will be arranged in the next D cycles. Spare parts order quantity is The maximum quantity ordered at one time shall not exceed the total number of equipment N. Considering that the spare parts transportation takes k cycles, the spare parts delivery time is 4. The mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning according to claim 1 is characterized in that: In step 3, maintenance decision-making and spare parts management are two independent tasks. The maintenance cost and spare parts management cost are modeled separately. The maintenance cost includes detection cost, replacement cost and downtime cost. Among them, detection cost: Before the maintenance plan is formulated, the operating status of the mechanical equipment is detected. The status detection cost of the mechanical equipment is c i ; Replacement cost: The replacement cost of mechanical equipment before failure is c rep When the degradation value of the equipment exceeds the failure threshold, that is, the mechanical equipment fails, the cost increases with the difficulty of maintenance, and the replacement cost is c fail ; Downtime cost: Faulty mechanical equipment that is not repaired in time will incur downtime costs, and additional costs will be incurred each cycle c down Considering sufficient inventory t ≥N and insufficient inventory h t When <N, the reward value The piecewise formula is as follows: Where, is an indicative function. When the device is replaced, the value is 1, otherwise it is 0. Spare parts management costs include ordering costs and inventory costs, where ordering costs are Where c0 is the fixed cost of each order, c u is the price of a single spare part; inventory cost: the inventory cost of mechanical equipment is c h The inventory cost per unit time for each spare part; Emergency ordering cost: When there is no spare part reserve, additional emergency ordering costs will be incurred. urg , reward value The piecewise formula is as follows:

5. The mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning according to claim 1 is characterized in that: In step 4, the steps to build a multi-agent DDQN reinforcement learning network algorithm model include: Step 4.

1. Define the agent's state space S, action space A, and reward function R. The state space S contains the equipment degradation value and the current spare parts inventory information. The action space A contains the maintenance strategy and spare parts ordering strategy. The reward function R provides feedback to the agent by evaluating the contribution of each decision to the overall system benefit. Step 4.2: Under the multi-agent framework of reinforcement learning, the model is composed of a maintenance agent and a spare parts agent. The two agents optimize system maintenance and inventory management through collaborative decision-making. The input of each agent is the state s t After being processed by a four-layer fully connected network and a ReLU activation layer, a probability distribution of discrete actions is output. The Softmax function is used to sample the probability distribution to determine the final action. The output action of the maintenance agent is the timing of maintenance, and the output action of the spare parts agent is the number of spare parts to be ordered. In this process, the two agents process their own states separately and share state information to improve the collaborative effect of decision-making. The action execution interacts with the environment. The joint action of the agents a=(a act , a spa ) will act on the environment. If the maintenance agent chooses to repair, the degradation value of the equipment will be reset to 0 at the next time step t+1, and the spare parts in the inventory will be consumed at the same time. If the inventory is insufficient, the repair will be delayed and a downtime penalty will be incurred. The ordering action of the spare parts agent will trigger a supply chain response. New spare parts will be put into the warehouse after a fixed delivery period. During this period, the inventory level changes dynamically and may incur holding costs or stock-out costs. Step 4.3: Each agent estimates its Q-value function through DDQN. DDQN improves the stability and accuracy of decision-making by solving the overestimation problem in Q-learning. The Q-value function of each agent is updated through an experience replay mechanism, and training is performed in a global shared experience pool to improve sample efficiency. Step 4.

4. During the training process, the neural network of each agent consists of multiple fully connected layers. The input of the network is the feature vector of the current state, and the output is the Q value of each action. The training is carried out through multiple iterations. Each iteration uses batch data randomly sampled from the experience replay pool to update the Q value function. In order to ensure the convergence of training, the maximum number of iterations is set to 500 times, and the model performance is evaluated after each iteration. When the loss function is lower than the predetermined convergence standard, the training is stopped and the model is considered to have converged. If it is found that the model has failed to converge during the training process, the early stopping strategy is adopted. When the loss function no longer decreases significantly within a period of time, the training is stopped in advance and the best model is selected to ensure that the neural network converges stably within a given number of training times.

6. The mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning according to claim 5 is characterized in that: The steps for training a reinforcement learning network model based on the multi-agent DDQN algorithm include: 1) Initialize the hyperparameters, then determine the degradation function of the mechanical device and use it as the solution environment of the model. Initialize the environment and the state s of the agent at each time step t t , take action a t , receive reward r t , and transfer to the new state s t+1 ; 2) The state value is input into the strategy Q value network. After being processed by the four-layer fully connected network, ReLU activation layer and softmax function, the probability distribution of discrete actions is output. The action a of agent i at time t is obtained by random sampling. t i ; 3) Each agent takes action a at time t t i After that, the environment will change accordingly, and the environment returns the reward value r at time t+1 t , cost value c t and the new state value s t+1 ; 4) Storage experience(s) t , a t , r t , s t+1 ) In the playback buffer, randomly sample a batch of data from the playback buffer to train the model; 5) DQN uses a neural network to approximate the Q value function, and the input is the state s t , the output is for each action a t Q value, loss function: L(θ)=E[(r t +γmax a′ Q(s t+1 a′;θ - )-Q(s t ,a t ;i)) 2 ] Where θ is the parameter of the neural network, θ - is the parameter of the target network, γ∈[0,1] is the discount rate, the larger the value, the more attention is paid to future rewards. Combined with the loss function, the gradient descent method is used to update the model. DDQN reduces the overestimation problem of Q value by decoupling action selection and action evaluation, and updates the formula: Q(s t ,a t )←Q(s t ,a t )+α[r t +γQ(s t+1 ,argmax a′ Q(s t+1 ,a′;θ);θ - )-Q(s t ,a t (I)].

7. The mechanical equipment maintenance and spare parts decision-making method based on reinforcement learning according to claim 1 is characterized in that: Mechanical equipment includes engines.

8. A system for implementing the method according to any one of claims 1 to 7, characterized in that: It includes: An acquisition module, which fits the continuous monitoring data reflecting the performance of the mechanical equipment into a degradation curve based on the Wiener process and determines the failure threshold of the mechanical equipment; A state space generation module that constructs a state space based on the discrete states of equipment degradation and inventory, and an action space based on maintenance actions and ordering opportunities; A cost function generation module, which establishes a cost function based on the maintenance cost and the spare parts management cost and calculates the expected reward value; A multi-agent DDQN reinforcement learning algorithm model uses the multi-agent DDQN reinforcement learning algorithm to maximize the expected reward value as the optimization goal, and obtains the optimal maintenance and spare parts management behavior under the discrete state of mechanical equipment degradation.

9. A computer storage medium, characterized in that The storage medium includes computer instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images