Aircraft engine maintenance policy optimization method based on nonparametric reinforcement learning

Abstract

Disclosed in the present invention is an aircraft engine maintenance policy optimization method based on nonparametric reinforcement learning. Firstly, for sparse aircraft engine operation data, constructing an aircraft engine model by means of a Bayesian network and a Gaussian process; then, establishing a policy network and a value network, interacting with the aircraft engine model to form a state / action value group, storing same into a replay buffer, and performing random sampling for use in a training set; updating the policy network and the value network, and updating the training set; and finally, performing aircraft engine maintenance optimization policy decision-making. Provided in the present invention is, for the first time, a Gaussian process-based nonparametric reinforcement learning method for an aircraft engine, which improves the degree of fit between overall training data and a model by means of dynamic data updating while integrating system uncertainty, thereby improving the sampling efficiency of an algorithm. Action selection is performed on the basis of prior maintenance experience data fused with uncertainty, improving the safety and sampling efficiency of a system and thereby solving the problem of predictive maintenance of an aircraft engine system.

Description

An aero-engine maintenance strategy optimization method based on non-parametric reinforcement learning TECHNICAL FIELD

[0001] The present application relates to the technical field of aero-engine operation and maintenance, and particularly relates to an aero-engine maintenance strategy optimization method based on non-parametric reinforcement learning. BACKGROUND

[0002] Aero-engines are one of the important infrastructures for logistics passenger transport, and reasonable maintenance strategies are crucial for maintaining the normal operation of aero-engine equipment, improving reliability, and saving the cost of airlines. However, in the daily aero-engine maintenance work, the engine equipment is complex, the working environment is relatively harsh, the system sensors have high uncertainty, the sub-components have complex degradation mechanisms, and the engine system needs to be designed for safety, resulting in low efficiency of aero-engine system operation and maintenance and serious waste of resources.

[0003] In order to solve the above problems, system uncertainty needs to be included in the strategy formulation category, and the algorithm efficiency needs to be improved, therefore, a new aero-engine operation scheme is urgently needed, which can balance the safety and economy of aero-engine maintenance, so as to realize the intelligent operation and maintenance of aero-engines. In this background, an aero-engine operation strategy formulation method based on non-parametric reinforcement learning emerges as the times require.

[0004] The existing aero-engine maintenance decision scheme mainly includes fault repair maintenance, preventive maintenance, and state-based maintenance. Among them, fault repair maintenance is a maintenance action taken after the equipment or system fails, that is, a non-planned maintenance, which usually occurs when the equipment fails or stops, mainly including detecting faults, diagnosing problems, repairing equipment, and restoring its normal operation state; preventive maintenance is a planned maintenance activity taken before the equipment fails, the goal of which is to prevent equipment failure by regular inspection, maintenance and repair to ensure that the equipment operates efficiently within the scheduled time; state-based maintenance is a maintenance strategy that determines the maintenance timing according to the actual working state and health condition of the equipment, using sensors, monitoring equipment and technology to collect data to predict the health condition of the equipment and determine when to maintain. The existing aero-engine maintenance decision scheme usually relies on static maintenance plans or current system states, which has deficiencies in scenarios requiring fast response and high safety level, and it is difficult to give correct maintenance schemes for decision-making problems in system uncertainty scenarios such as sensor failure.

[0005] By introducing multi-agent reinforcement learning technology, combined with Gaussian process uncertainty, the system uncertainty in the aero-engine maintenance system can be better solved, and combined with prior maintenance experience data, maintenance decisions can be made for complex coupled multi-subsystem engines, and the operation efficiency of the equipment can be improved.

[0006] SUMMARY

[0007] The present application aims at the deficiencies of the prior art, and provides an aero-engine maintenance strategy optimization method based on non-parametric reinforcement learning.

[0008] To achieve the above-mentioned purpose, the following technical solutions are adopted: as a first aspect of the present application, an aero-engine maintenance strategy optimization method based on non-parametric reinforcement learning is provided, comprising the following steps:

[0009] Collecting aero-engine historical operation data, establishing an aero-engine model;

[0010] Establishing a strategy network and a value network, randomly selecting an initial system state s0, and formulating an initial maintenance action a0; based on the initial system state s0 and the initial maintenance action a0, interacting with the aero-engine model to obtain a set of state-action value groups: system state, maintenance action, system feedback cost and next system state; establishing a replay buffer, storing the state-action value group (s t ,a t ,r(s t ),s t+1 ) of each step in the replay buffer; randomly sampling in the replay buffer as a training set; updating the strategy network and the value network; updating the training set; judging whether the system running step length reaches the set system training step length, if yes, stopping training to obtain the trained network, and outputting the aero-engine maintenance optimization strategy, otherwise, continuing to select actions in the strategy network, and repeating the training until the running step length reaches the training step length.

[0011] Further, the parameter values of the system state include the stagnation pressure stagnation temperature stagnation enthalpy pressure ratio i constant pressure specific heat of working medium specific heat ratio of working medium i efficiency factor i ; the maintenance action includes overhaul and minor repair.

[0012] Further, the aero-engine model includes a system state transition model and a system feedback model.

[0013] The system state transition model is fitted to the system state transition probability based on a Markov process by a Bayesian network, wherein the Bayesian network parameter L is represented by a maximum likelihood estimation method; the expectation maximization algorithm is used for Bayesian network parameter optimization; and a system state transition function P(s t+1 | s t ) is obtained.

[0014] The system feedback model is based on the system state s t The running cost r1(s t ) is fitted by a Gaussian process; the sum of the prices of the maintenance actions of each component is taken as the repair cost r2(s t ) according to the maintenance action determined by the policy network under the system state s t ; and the sum of the running cost r1(s t ) and the repair cost r2(s t ) is taken as the system feedback cost r(s t ) under the system state s t .

[0015] Then, the next system state s t+1 is obtained through the system state transition function P(s t+1 | s t ).

[0016] Further, the policy network selects an action based on the system state s t , and obtains a maintenance action execution probability P net (a t ) through a Gaussian process:

[0017] Wherein, A π (s t , a i ) is the advantage function of the i th maintenance action a i predicted by the policy network under the system state s t .

[0018] The maximum probability estimation method is used to select an action, and the system formulated maintenance action a t is output: a t = argmax(P net (a t ))

[0019] The value network inputs the system state s t , and outputs a value evaluation V π (s t ) of the system state s t based on a Gaussian process.

[0020] Further, the action selection selects the maintenance action exploration or maintenance strategy utilization through an epsilon-greedy algorithm to formulate the next maintenance action; when the epsilon-greedy algorithm sampling is greater than epsilon, the maintenance action exploration is performed, and the advantage function is obtained by random sampling in the Gaussian process uncertainty range through the policy network; when the epsilon-greedy algorithm sampling is less than epsilon, the maintenance strategy utilization is performed, and the advantage function is output by the Gaussian process mean value through the policy network.

[0021] Further, Bayesian data fusion is used to fuse prior maintenance experience data and maintenance actions obtained through the policy network to assist decision-making guidance.

[0022] Further, the updating of the policy network and the value network includes the following sub-steps: based on the original sampling point system state Advantage function A π (s t ,a t ) old and the new sampling point system state Advantage function A π (s t ,a t ) new , the Gaussian process fitting data set is expanded:

[0023] The input is: The output is:

[0024] Wherein, A π (s t ,a t ) is the advantage function under the state s t and the action a t , which is calculated as: A π (s t ,a t ) = r(s t ,a t ) + γV π (s t+1 ) - V π (s t )

[0025] In the formula, gamma is the decay coefficient, and r(s t ,a t ) is the system feedback cost function.

[0026] Based on the covariance matrix, a Gaussian process is constructed as the updated policy network:

[0027] Wherein, is the variance of the observation noise, I is the unit matrix; K is the covariance matrix calculated by the Gaussian process kernel function:

[0028] correcting the value evaluation V by a time difference method π (s t ), and batch training, updating the value network.

[0029] Further, the kernel function of the Gaussian process is:

[0030] where x and x' are input vectors, ||x-x'| 2 is the square of the Euclidean distance, l is the length scale parameter, which controls the smoothness of the function, is the signal variance, which controls the overall amplitude of the kernel function.

[0031] Further, the updating of the training set comprises the following sub-steps: obtaining the fitting degree between each training data in the training set and the value network by calculating the negative log-likelihood; re-sampling the replay buffer in batches, obtaining the fitting degree between the re-sampled points and the value network by calculating the negative log-likelihood; comparing, replacing the data with higher fitting degree with the data with lower fitting degree in the training set, so that the model tends to train the data with lower fitting degree, thereby improving the overall reliability of the model.

[0032] As a second aspect, the application also provides an aero-engine maintenance strategy optimization system based on non-parametric reinforcement learning, comprising:

[0033] An aero-engine model construction module is configured to construct a system state transition model and a system feedback model based on a Bayesian network and a Gaussian process according to the collected aero-engine operation data.

[0034] A network initialization module is configured to initialize a policy network and a value network.

[0035] A maintenance action decision module is configured to formulate a maintenance strategy according to the system state of the current aero-engine model, avoid the occurrence of a highly deteriorated state of the aero-engine, and reduce the system feedback cost.

[0036] A replay buffer module is configured to store the state-action-value set of each step.

[0037] A model training module is configured to train the policy network and the value network using the state-action-value set extracted in batches by the replay buffer module.

[0038] A dynamic updating module is configured to dynamically update the training set, and replace the data with higher fitting degree with the data with lower fitting degree.

[0039] The beneficial effects of the present application are that, by constructing an aero-engine model, the engine model with sparse data can be fitted by a Bayesian network and a Gaussian process based on a non-parametric reinforcement learning method, while ensuring similarity with high-precision simulation. The present application can improve the fitting degree of the overall training data and the model through dynamic data updating while fusing system uncertainty, improve the sampling efficiency of the algorithm, and improve the safety and sampling efficiency of the system through the action selection method based on prior maintenance experience data to fuse uncertainty. The effect is better than that of the prior art, and the present application has the advantages of strong universality, simple use, etc. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1 is a schematic diagram of the agent learning process according to the present application;

[0041] Fig. 2 is a schematic diagram of the aero-engine model establishment process according to the present application;

[0042] Fig. 3 is a block diagram of the aero-engine maintenance strategy optimization system based on non-parametric reinforcement learning according to the present application;

[0043] Fig. 4 is a fitting effect diagram of the aero-engine model, wherein (a) is a comparison diagram of the model prediction result and the actual result, and (b) is a global sensitivity analysis diagram of the feedback model for the high-pressure turbine, the low-pressure turbine, the high-pressure compressor and the low-pressure compressor;

[0044] Fig. 5 is a comparison diagram of the training results of the present method and the training results of the traditional decision algorithm, wherein (a) is a diagram of the algorithm training convergence result, and (b) is a five-time result box diagram of the algorithm after training and the actual application of the aero-engine model;

[0045] Fig. 6 is a specific maintenance strategy making scheme for the high-pressure turbine, the low-pressure turbine, the high-pressure compressor and the low-pressure compressor parts of the aero-engine according to the present method. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0047] It should be noted that the features in the following embodiments and implementation manners can be combined with each other without conflict.

[0048] The core technology of this invention is to mathematically model historical aero-engine data with sparse data features, constructing an aero-engine model. Addressing the system uncertainty and sparse data characteristics of aero-engine cases, nonparametric reinforcement learning is employed to solve the model. Nonparametric reinforcement learning utilizes Gaussian processes to quantify system uncertainty and leverages nonparametric features to integrate system uncertainty in aero-engine scenarios into the overall reinforcement learning algorithm flow. This solves the problems of sparse datasets and uncertainty in real industrial scenarios. Furthermore, a Bayesian data fusion algorithm is used to incorporate prior maintenance experience into the maintenance strategy exploration process, improving algorithm sampling efficiency and thus solving the predictive maintenance problem of aero-engine systems. This results in a method for formulating aero-engine system operation and maintenance strategies that incorporates system uncertainty. The main idea can be summarized as follows:

[0049] 1. Constructing an aero-engine model. Based on historical engine operating data, the state transition probabilities of the aero-engine model are modeled using a Bayesian network, and the system feedback cost is fitted using a Gaussian process to establish an aero-engine model that can interact with the reinforcement learning process.

[0050] 2. A nonparametric reinforcement learning method based on Gaussian processes is proposed. The policy and value functions of the reinforcement learning algorithm are constructed based on Gaussian processes. Actions are explored and utilized based on the uncertainty of Gaussian processes. A Bayesian data fusion algorithm is used to integrate prior maintenance experience data with reinforcement learning decision results, incorporating the uncertainty of the engine system into the reinforcement learning process. The deterioration state of the engine subsystem is used as a safety evaluation index, and the system feedback cost is used as an economic evaluation index for the maintenance plan. This method is used to formulate operation and maintenance strategies for the aero-engine system, improving the overall safety and reliability of the system.

[0051] 3. A dynamic update method for model sampling points is adopted. Based on the negative log-likelihood of the training data used in each model and the value network, the fit between the training data and the value function model is calculated. By dynamically updating the method, training points with lower fit to the model are selected for batch training, which improves the fit of the reinforcement learning network to the overall model sampling points and enhances the reliability of the system.

[0052] 4. A system for optimizing aero-engine maintenance strategies based on nonparametric reinforcement learning is proposed.

[0053] The present invention provides an optimization method for aero-engine maintenance strategies based on nonparametric reinforcement learning, as shown in Figure 1, which specifically includes the following steps:

[0054] (1) Establishment of the aero-engine model: An aero-engine model for algorithm verification is established, which consists of two parts: a system state transition model and a system feedback model. The system state transition model is responsible for judging the engine state deterioration process based on the current aero-engine state, while the system feedback model is responsible for calculating the total cost of each engine operation and maintenance cycle, which serves as the system reward. As shown in Figure 2, the following sub-steps are included:

[0055] (1.1) System State Transition Model: Based on the current state of the aero-engine, the deterioration process of the engine state is determined. Due to the high operating cost of aero-engines, there is a lack of actual flight failure data and historical operation and maintenance data, and most historical aero-engine operating data is sparse. In order to fit the transition probability between different states under each system state, the deterioration state s of the engine at time t is first determined based on the sparse engine flight failure data. t (hereinafter referred to as system state s) t The state transition probability P(s) of engine flight failure data is defined using a Bayesian network. t+1 |s t Fit the data.

[0056] The state transition process between different components of the engine is characterized by isentropic relations and calibrated polytropic efficiency: the state parameter value of the i-th component is the stagnant pressure. Stagnation temperature Stagnation enthalpy pressure ratio π i The specific heat of the working fluid under constant pressure is The specific heat ratio of the working fluid is γ i The efficiency factor is η i The state parameter transition equations between different components are as follows:

[0057] The system's state is defined using a degenerate simulation based on a gamma distribution. The degenerate equation is:

[0058] Where Δη represents damage growth, α is the shape parameter, β is the scale parameter, and Γ is the gamma distribution. 100 engine operation segments are generated, each segment consisting of 50 deterioration steps, and the state is discretized into 5 levels, with higher values ​​indicating increasing deterioration.

[0059] During the fitting process of the Bayesian network, the state transition probability of the aero-engine is obtained based on the Markov process using the following formula:

[0060] Where S0 is the state before the transition, and S1 is the state after the transition.

[0061] The Bayesian network parameters L are expressed using the maximum likelihood estimation method:

[0062] Among them, s i The sampled state transition sequence during a single state transition process is denoted by h, which represents the hidden nodes in the network. Furthermore, this example uses the expectation-maximization algorithm and Jensen's inequality to obtain a lower bound on the log-likelihood:

[0063] Where, the q function is s i The probability distribution of , and satisfies: ∑ h q(h∣s i )=1 and 0≤q(h∣s i )≤1.

[0064] (1.2) System Feedback Model: Based on sparse historical maintenance data, the system fits the operation, maintenance, and repair costs of aero-engines caused by the transition from a deteriorating state. Aero-engine systems incur operation, maintenance, and repair costs during operation and transition from one state to another. Due to the sparsity of the system, issues such as missing data may arise. Therefore, a Gaussian process is used to fit the system feedback costs of the aero-engine system. As a non-parametric stochastic process, the Gaussian process performs better than other fitting algorithms in addressing the data missing problem in sparse systems.

[0065] Collect historical operating cost data for each engine component and analyze the aero-engine system status. t With operating cost r1(s) t Using x (i) ,y (i) The kernel function of a Gaussian process is expressed as:

[0066] This function is used to measure the similarity of input data, where x and x′ are input vectors, and ||xx′|| 2 is the square of the Euclidean distance. l is the length scale parameter, which controls the smoothness of the function. This is the signal variance, which controls the overall magnitude of the kernel function. For a given dataset (X,Y), the marginal likelihood of a Gaussian process is:

[0067] Where K is the covariance matrix and n is the number of training data points. The mean μ and variance σ of the operating cost predicted by the Gaussian process are: μ=k(x * ,x) T K -1 y σ=k(x * ,x * )-k(x *,x) T K -1 k(x * ,x)

[0068] Where, x * Let be the data points to be predicted. Based on the Gaussian process prediction mean, the engine operating cost r1(s) can be obtained. t ).

[0069] Based on the maintenance actions determined by the strategy network for each component in this system state, and the corresponding repair prices (according to current factory quotes), the repair prices of each component are summed up to obtain the engine's repair price in system state s. t The maintenance cost r2(s) t ).

[0070] Engine operating cost r1(s) t ) and maintenance cost r2(s t The sum of these values ​​is used as the system feedback cost r(s). t ).

[0071] (1.3) Aero-engine System Integration and Initialization: The mathematical model of the aero-engine system proposed in this example consists of a system state transition model and a system feedback model. During operation, this model accepts the current state of the system and, based on the current state and maintenance actions, determines the next state and system feedback cost. After periodic operation, the aero-engine system's system transition function P(s) obtained in step (1.1) is... t+1 |s t Based on the current cycle state s t and the maintenance actions taken a t Determine the aircraft engine status s for the next cycle t+1 And calculate the state s in this cycle through the system feedback model. t The corresponding engine operating cost r1(s) t ) and maintenance cost r2(s t The operating cost and maintenance cost are integrated as the system feedback cost r(s). t ).

[0072] Following the method described above, the aero-engine model and initial state are obtained. This example uses non-parametric networks such as Bayesian networks and Gaussian processes for modeling, which can achieve better fitting results for the characteristics of sparse datasets.

[0073] (2) Initialize the decision system network and formulate strategies: Establish and initialize the algorithm strategy function Pnet and value function Vnet, and formulate maintenance strategies based on the initial state of the aero-engine model in step (1) using the strategy network and experience-assisted methods. This includes the following sub-steps:

[0074] (2.1) Based on the state transition function P(s) of the aero-engine model system in step (1) t+1 |s t ) and system feedback R(s t To formulate a maintenance strategy for this engine, this example uses a reinforcement learning policy network Pnet to develop the system maintenance strategy. The system state s is input into the policy network Pnet. t The system-defined probability P(a) of performing maintenance actions is obtained. t ), will change the system state s t With the system's advantage function A π (s t ,a t Using x and y to represent, establish a Gaussian process:

[0075] The kernel function for a Gaussian process is:

[0076] This function is used to measure the similarity of input data, where x and x′ are input vectors, and ||xx′|| 2 is the square of the Euclidean distance. l is the length scale parameter, which controls the smoothness of the function. This is the signal variance, which controls the overall magnitude of the kernel function. For a given dataset (X,Y), the marginal likelihood of the Gaussian process is:

[0077] Where K is the covariance matrix and n is the number of training data points. The mean μ and variance σ of the system feedback predicted by the Gaussian process are: μ=k(x * ,x) T K -1 y σ=k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)

[0078] Where, x * These are the data points to be predicted.

[0079] The policy network (Pnet) selects actions based on the system state. Action selection employs either an ε-greedy algorithm or policy exploitation to obtain the advantage function for each maintenance action, thus determining the next maintenance action. ε is a number between 0 and 1. Random sampling is performed on the range of 0-1. If the sampled value is greater than ε, action exploration is used, meaning the policy network randomly samples within the uncertainty range of the Gaussian process to obtain the advantage function; otherwise, policy exploitation is used, where the policy network outputs the advantage function based on the Gaussian process mean.

[0080] Action exploration is derived by random sampling within the uncertainty range σ of the Gaussian process through a policy network: σ = k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)

[0081] Action exploration can explore random actions within a certain range to find better maintenance actions.

[0082] Policy exploitation, on the other hand, utilizes the Gaussian process mean output of the policy network obtained through training, inheriting the results of previous training.

[0083] The execution probability of each maintenance action is calculated based on the dominance function output by the Gaussian process. The execution probability of each maintenance action is:

[0084] A π (s t ,a i That is, state s. t The i-th maintenance action a predicted by the next policy network i The dominant function;

[0085] The maximum probability estimation method is used to select actions: a t =argmax(P net (a t ))

[0086] a t This refers to the specific maintenance actions that the policy network Pnet optimizes based on the probability of executing maintenance actions according to the current system state, and these actions are the output of the policy network Pnet. In the aviation field, these maintenance actions include major and minor repairs.

[0087] After deriving the maintenance actions of the policy network, Bayesian data fusion is used to combine prior maintenance experience data with the policy network. The combination of prior maintenance experience data provides guidance, improves the security of the system, increases sampling efficiency, and accelerates algorithm convergence.

[0088] Among them, P a Let P be the probability of the decision action in the policy network. ae Let w represent the probability of executing actions based on prior maintenance experience data, and w represent the weight of the influence of prior maintenance experience on decision-making.

[0089] (2.2) The reinforcement learning process requires a value network Vnet to evaluate the value of the current state. In a Markov process, the value function Vnet represents the expected sum of the feedback values ​​of the current state s: V π (s)=E π [G t |S t =s]

[0090] Among them, G t This provides system feedback. The system state s is input into the value network Vnet. t The system-defined action value estimate V is obtained. t The system state s t The estimated value of the system's actions, V t Use x (i) ,y (i) The representation is as follows: a Gaussian process is established, with the Gaussian kernel function and marginal likelihood consistent with step (2.1). The output of the Gaussian process is the value assessment Vnet of the system state by the value network Vnet. π (s).

[0091] (2.3) Based on the reinforcement learning policy network Pnet and value network Vnet generated in steps (2.1) and (2.2), the aero-engine model established in step (1) is initialized with random state, the system state s0 is randomly selected and input into the policy network Pnet, and the first maintenance action a0 is formulated.

[0092] (3) Strategy Iteration and Network Training: Based on the initial maintenance action a0 and initial system state s0 formulated in step (2), interact with the aero-engine model established in step (1) to obtain a set of state action value groups; establish a replay buffer to store the state action value groups (s0, s0, s0, s0) of each step. t ,a t ,R(s t ),s t+1 The data is stored in the replay buffer and sampled as a training set; the policy network Pnet and the value network Vnet are updated; and the training set is dynamically updated using negative log-likelihood.

[0093] (3.1) Based on the initial system state s0 and initial maintenance action a0 obtained in step (2.3), interact with the aero-engine system to obtain the next state of the system and the cost of the state transition process. Organize this process into a state-action group (s t ,a t ,R(s t ),s t+1 Since both the Policy Network (Pnet) and the Value Network (Vnet) require batches of data for training, a data replay buffer needs to be established to store each state-action group and extract them in batches for training the Policy Network and Value Network. The state-action group for this step is stored in the replay buffer.

[0094] (3.2) Based on the replay buffer established in step (3.1), batch extract training data, and based on the value network's prediction V of the state under the guidance of policy π. π (s t ), calculate the dominance function A π A π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t )

[0095] Where γ is the attenuation coefficient, r(s) t ,a t ) is the system feedback cost function.

[0096] Using the dominance function, based on the system state at the original sampling point (i.e., the sampling point of the previous period), Advantage function A π (s t ,a t ) old System state at the new sampling point (i.e., the sampling point in this period) Advantage function A π (s t ,a t ) new Expand the Gaussian process to fit the dataset and update the policy network:

[0097] The input to the extended Gaussian process is: The output is:

[0098] A π (s t ,a t ) represents state s tNext, execute action a t The advantage function in this case, and the expanded global covariance matrix:

[0099] A Gaussian process is constructed based on the covariance matrix and used as the updated policy network:

[0100] Where K is the covariance matrix calculated by the Gaussian process kernel function. I is the variance of the observation noise, and I is the identity matrix.

[0101] The value function V is analyzed using the Time Difference-Error (TD-Error) method. π (s t Corrections are performed, batch training is conducted, and Gaussian process fitting is performed:

[0102] Where p(s) t+1 |s t ,a t ) is the state transition function, r(s) t ,a t ) is the system feedback cost function.

[0103] (3.3) Based on the training data extracted in batches in step (3.2), calculate the negative log-likelihood between each training data point and the value network Vnet:

[0104] Where e is the number of training data points extracted in a batch, σ i With μ i Let y be the variance and mean of the value function prediction for the i-th training data, respectively. i The system state value V after correction using the time difference method π (s i The negative log-likelihood represents the degree of fit between each training data point and the value function model. To select more training data from the replay buffer that correlates with the model's fit, this example also resamples the replay buffer (batch random sampling), resulting in a set of resampled points of e. * And calculate e * The negative log-likelihood represents the negative log-likelihood. A larger negative log-likelihood indicates a poorer fit between the sampled points and the model, and the replacement probability P... u The negative log-likelihood of e was calculated with respect to e and e. * The ratio of the sum of negative log-likelihoods is used as the probability of training data replacement, and this probability is used to replace training data points. A smaller probability indicates a tendency to retain the current training data point e, while a larger probability indicates a tendency to replace the current training data point e with a resampled point e. *This causes the model to tend to be trained on data that is less closely related to the model, thereby improving the overall reliability of the model.

[0105] Where e is the set of sampling points used in training, e * This is the set of sampling points for resampling.

[0106] (3.4) Determine whether the system running step size has reached the preset system training step size: If the training step size has been reached, stop training; otherwise, continue to select actions and repeat step (3) until the running step size has reached the training step size.

[0107] Example 1:

[0108] An embodiment of the present invention was implemented on a machine equipped with an Intel(R) Xeon(R) Gold 6330 CPU, an NVIDIA GTX3090 graphics processor, and 128GB of memory. Using the above-described embodiment, the results shown in Figures 4-6 were obtained.

[0109] Aero-engine modeling was performed on the GE90 engine of General Electric, and the state transition matrix and system feedback of the engine were fitted. The fitting results are shown in Figure 4. Figure 4(a) shows the comparison between the model prediction results and the actual results. The vertical axis represents the predicted value, and the horizontal axis represents the actual value. The circled points are the actual data points, and the solid plus points are the predicted points. This method can accurately predict the state transition process and system feedback of the aero-engine. Figure 4(b) shows the global sensitivity analysis of the system feedback model on the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor. Figure 5(a) shows the effect of this method compared with the rule-based maintenance method, the deep value reinforcement learning algorithm, and the actor-critic reinforcement learning algorithm. The bottom curve is the convergence result of this method, and the upper dashed line is the rule-based maintenance algorithm. Nonparametric reinforcement learning can significantly reduce the system feedback cost compared with the rule-based maintenance algorithm. The two convergence curves at the top are the deep value reinforcement learning algorithm and the actor-critic reinforcement learning algorithm, respectively. Even after 1000 training iterations, the convergence results of these two algorithms were still inferior to those of nonparametric reinforcement learning after only 100 training iterations. Furthermore, both algorithms exhibited strong oscillations during convergence, indicating low sampling efficiency and inability to achieve stable training. In contrast, nonparametric reinforcement learning showed relatively stable training, demonstrating higher sampling efficiency. The results show that nonparametric reinforcement learning improves data utilization efficiency by more than 100 times compared to traditional reinforcement learning methods. This method, while incorporating system uncertainty, improves the fit between the overall training data and the model through dynamic data updates, thereby enhancing the algorithm's sampling efficiency. Figure 5(b) shows the box plot of the average performance of this method compared to rule-based maintenance methods, deep value reinforcement learning algorithms, and actor-critic reinforcement learning algorithms after 5 practical applications following training. The leftmost box plot shows the results of this method. The nonparametric reinforcement learning method significantly lowers the upper, lower, and mean of system feedback costs compared to the comparative methods, and the actor-critic model also exhibits decision divergence. As can be seen, this method can relatively stably formulate maintenance strategies for aero-engines. Figure 6 shows the specific maintenance strategies formulated by this method for the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor; circles represent minor repairs, and inverted triangles represent major repairs. The results show that this method can accurately perform mathematical modeling of aero-engines and correctly formulate reasonable maintenance strategies for different components of the aero-engine.

[0110] This embodiment also provides an aero-engine maintenance strategy optimization system based on nonparametric reinforcement learning, which is used to implement the above embodiments. The terms "module," "unit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions.

[0111] This embodiment provides an aero-engine maintenance strategy optimization system based on nonparametric reinforcement learning, as shown in Figure 3, including:

[0112] The aero-engine model building module is used to build system state transition models and system feedback models based on collected aero-engine operating data and Bayesian networks and Gaussian processes.

[0113] The network initialization module is used to initialize the policy network and the value network.

[0114] The maintenance action decision module is used to formulate maintenance strategies based on the current system state of the aero-engine model, to avoid the aero-engine from entering a highly deteriorated state, and to reduce the system feedback cost r(s). t );

[0115] The replay buffer module is used to store the state-action value group for each step, so that it can be extracted in batches to train the policy network and value network.

[0116] The model training module is used to train the policy network and the value network by using the state-action value groups extracted in batches by the replay buffer module.

[0117] The dynamic update module is used to dynamically update the training set, replacing data with a high degree of fit to the value network with data with a low degree of fit.

[0118] It should be noted that the above embodiments are only used to illustrate the advantages and specific steps of the present invention, and do not limit the scope of application of the present invention. Any other forms of simulation made by those skilled in the art based on the present invention are within the protection scope of the present invention.

Claims

1. A method for optimizing aero-engine maintenance strategies based on nonparametric reinforcement learning, characterized in that, Includes the following steps: Collect historical operating data of aero engines and establish aero engine models; Establish a policy network and a value network, randomly select the initial system state s0, and formulate the initial maintenance action a0; Based on the initial system state s0 and initial maintenance action a0, and through interaction with the aero-engine model, a set of state-action value groups is obtained: system state, maintenance action, system feedback cost, and next system state; a replay buffer is established to store the state-action value group (s0) of each step. t ,a t ,r(s t ),s t+1 Store the data in the replay buffer; perform random sampling in the replay buffer as a training set; update the policy network and the value network; Update the training set; Determine whether the system's running step size has reached the set system training step size. If it has, stop training, obtain the trained network, and output the aircraft engine maintenance optimization strategy.

2. The method according to claim 1, characterized in that, The parameter values ​​of the system state include the stagnation pressure corresponding to each component i of the engine. Stagnation temperature Stagnation enthalpy pressure ratio π i Specific heat of working fluid under constant pressure specific heat ratio of working fluid γ i Efficiency factor η i The maintenance actions include major repairs and minor repairs.

3. The method according to claim 1, characterized in that, The aero-engine model includes a system state transition model and a system feedback model; The system state transition model is obtained by fitting the system state transition probabilities obtained based on the Markov process using a Bayesian network. The Bayesian network parameters L are represented by the maximum likelihood estimation method. The expectation-maximization algorithm is used to optimize the Bayesian network parameters, resulting in the system state transition function P(s). t+1 |s t ); The system feedback model is based on the system state s. t The running cost r1(s) is fitted using a Gaussian process. t According to system state s t The maintenance actions determined by the policy network are summed as the maintenance cost r2(s) for each component. t ); the operating cost r1(s) t ) and maintenance costs r2(s t The sum of these states is used as the system state s. t The system feedback cost r(s) t ); Then, through the system state transition function P(s) t+1 |s t ), to obtain the next system state s t+1 .

4. The method according to claim 1, characterized in that, The policy network is based on system state s t Action selection is performed, and the execution probability P of the maintenance action is obtained through a Gaussian process. net (a t ): Among them, A π (s t ,a i ) represents the system state s t The i-th maintenance action a predicted by the next policy network i The dominant function; The maximum probability estimation method is used to select actions, and the maintenance action 'a' defined by the system is output. t : a t =argmax(P net (a t )) The value network input system state s t Based on the output of the Gaussian process, the system state s t Valuation V π (s t ).

5. The method according to claim 4, characterized in that, The action selection is based on an ε-greedy algorithm to choose between maintenance action exploration or maintenance strategy utilization to formulate the next maintenance action. When the ε-greedy algorithm sampling is greater than ε, maintenance action exploration is performed, and the dominant function is obtained by random sampling within the uncertainty range of the Gaussian process through the policy network. When the ε-greedy algorithm sampling is less than ε, maintenance strategy utilization is performed, and the dominant function is output by taking the mean of the Gaussian process through the policy network.

6. The method according to claim 4, characterized in that, Bayesian data fusion is used to combine prior maintenance experience data with maintenance actions obtained through a policy network to assist in decision-making.

7. The method according to claim 1, characterized in that, The updated strategy network and value network include the following sub-steps: based on the system state of the original sampling points... Advantage function A ρ (s t ,a t ) old With the new sampling point system status Advantage function A π (s t ,a t ) new Extend the Gaussian process to fit the dataset: Input is: The output is: Among them, A π (s t ,a t ) represents state s t Next, execute action a t The advantage function under the given condition is calculated as follows: From π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t ) In the formula, γ is the attenuation coefficient, and r(s) t ,a t ) represents the system feedback cost function; A Gaussian process is constructed based on the covariance matrix and used as the updated policy network: in, I is the variance of the observation noise, and K is the identity matrix; K is the covariance matrix calculated using the Gaussian process kernel function. The value assessment V was corrected using the time difference method. π (s t ), and perform batch training to update the value network.

8. The method according to any one of claims 3-7, characterized in that, The kernel function of the Gaussian process is: Where x and x′ are input vectors, ||xx′|| 2 is the square of the Euclidean distance, and l is the length scale parameter that controls the smoothness of the function. It is the signal variance, which controls the overall amplitude of the kernel function.

9. The method according to claim 1, characterized in that, The updated training set includes the following sub-steps: obtaining the degree of fit between each training data point in the training set and the value network by calculating the negative log-likelihood; re-sampling the replay buffer in batches, obtaining the degree of fit between the resampled points and the value network by calculating the negative log-likelihood; comparing and replacing the data with a higher degree of fit in the training set with the data with a lower degree of fit.

10. A system for optimizing aero-engine maintenance strategies based on nonparametric reinforcement learning, characterized in that, include: The aero-engine model building module is used to build system state transition models and system feedback models based on collected aero-engine operating data and Bayesian networks and Gaussian processes. The network initialization module is used to initialize the policy network and the value network. The maintenance action decision module is used to formulate maintenance strategies based on the current system state of the aero-engine model, so as to avoid the aero-engine from entering a highly deteriorated state and reduce system feedback costs. The replay buffer module is used to store the state-action value group for each step; The model training module is used to train the policy network and the value network by using the state-action value groups extracted in batches by the replay buffer module. The dynamic update module is used to dynamically update the training set, replacing data with a high degree of fit to the value network with data with a low degree of fit.

Images