A method for optimizing aircraft engine maintenance strategy based on non-parametric reinforcement learning

Abstract

The present invention discloses an aircraft engine maintenance strategy optimization method based on non-parametric reinforcement learning. First, for sparse aircraft engine operation data, an aircraft engine model is established through Bayesian network and Gaussian process; then, a policy network and a value network are established to interact with the aircraft engine model to form a state action value group, which is stored in the replay buffer and randomly sampled as a training set; the policy network and the value network are updated, and the training set is updated; finally, the aircraft engine maintenance optimization strategy decision is made. The present invention proposes for the first time a non-parametric reinforcement learning method based on Gaussian process for aircraft engines. While integrating the uncertainty of the system, the degree of fit between the overall training data and the model is improved through dynamic data update, and the sampling efficiency of the algorithm is improved; based on the prior maintenance experience data, the uncertainty is integrated to select actions, improve the safety and sampling efficiency of the system, and thus solve the predictive maintenance problem of the aircraft engine system.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft engine operation and maintenance, and in particular to an aircraft engine maintenance strategy optimization method based on non-parametric reinforcement learning. Background Art

[0002] Aircraft engines are a vital piece of infrastructure for logistics and passenger transport. Developing a sound maintenance strategy is crucial for maintaining the proper functioning of aircraft engine equipment, improving reliability, and saving airlines costs. However, routine aircraft engine maintenance work is plagued by complex equipment, relatively harsh operating environments, high levels of uncertainty in system sensors, complex degradation mechanisms of subcomponents, and the need for safety assurance designs. This leads to low efficiency and significant resource waste in aircraft engine system operation and maintenance.

[0003] To address these issues, it's necessary to incorporate system uncertainty into strategy formulation and improve algorithm efficiency. Therefore, a new aircraft engine operation and maintenance solution is urgently needed that balances safety and economic efficiency, enabling intelligent aircraft engine operation and maintenance. Against this backdrop, a method for formulating aircraft engine operation and maintenance strategies based on nonparametric reinforcement learning has emerged.

[0004] Existing aircraft engine maintenance decision-making schemes primarily include: fix-it maintenance, preventive maintenance, and condition-based maintenance. Fix-it maintenance is a maintenance action taken after a device or system fails. It is a type of unplanned maintenance, typically occurring when equipment fails or shuts down. It primarily involves detecting the fault, diagnosing the problem, repairing the equipment, and restoring its normal operating state. Preventive maintenance is a planned maintenance activity performed before equipment failure occurs. Its goal is to prevent equipment failure through regular inspections, maintenance, and repairs, ensuring that the equipment remains operational and efficient within a predetermined timeframe. Condition-based maintenance is a maintenance strategy that determines maintenance timing based on the actual operating status and health of the equipment. It uses sensors, monitoring equipment, and technologies to collect data to predict the equipment's health and determine when maintenance should be performed. Existing aircraft engine maintenance decision-making schemes typically rely on static maintenance plans or current system status. This is insufficient in scenarios requiring rapid response and a high level of safety. Furthermore, it is difficult to provide accurate maintenance plans for decision-making problems in scenarios with systemic uncertainty, such as sensor failure.

[0005] By introducing multi-agent reinforcement learning technology and combining it with Gaussian process uncertainty, we can better address the system uncertainty in aircraft engine maintenance systems. Combined with prior maintenance experience data, we can make maintenance decisions for complex coupled multi-subsystem engines and improve the operating efficiency of the equipment. Summary of the Invention

[0006] The purpose of the present invention is to provide an aircraft engine maintenance strategy optimization method based on non-parametric reinforcement learning to address the deficiencies of the existing technology.

[0007] To achieve the above objectives, the present invention adopts the following technical solutions: As a first aspect of the present invention, a method for optimizing aircraft engine maintenance strategies based on non-parametric reinforcement learning is provided, comprising the following steps:

[0008] Collect historical operating data of aircraft engines and build aircraft engine models;

[0009] Establish a strategy network and a value network, randomly select the initial system state s0, and formulate the initial maintenance action a0; based on the initial system state s0 and the initial maintenance action a0, interact with the aircraft engine model to obtain a set of state action value groups: system state, maintenance action, system feedback cost and next step system state; establish a replay buffer, and store the state action value group (s0) of each step. t ,a t ,r(s t ),s t+1 ) is stored in the replay buffer; random sampling is performed in the replay buffer as a training set; the policy network and the value network are updated; the training set is updated; it is determined whether the system operation step reaches the set system training step, if so, the training is stopped, the trained network is obtained, and the aircraft engine maintenance optimization strategy is output; otherwise, action selection is continued in the policy network, and the training is repeated until the operation step reaches the training step.

[0010] Furthermore, the parameter value of the system state includes the stagnation pressure corresponding to each component i of the engine stagnation temperature stagnation enthalpy Pressure ratio π i , working fluid constant pressure specific heat Working fluid specific heat ratio γ i , efficiency factor η i ; The maintenance actions include major repairs and minor repairs.

[0011] Furthermore, the aircraft engine model includes a system state transition model and a system feedback model;

[0012] The system state transition model is fitted with the system state transition probability obtained based on the Markov process through the Bayesian network, wherein the Bayesian network parameter L is expressed by the maximum likelihood estimation method; the expectation maximization algorithm is used to optimize the Bayesian network parameters; the system state transition function P(s t+1 ∣s t );

[0013] The system feedback model is based on the system state s t Fitting the running cost r1(s t ); According to the system status s t The maintenance action determined by the strategy network is the sum of the prices corresponding to the maintenance actions of each component as the maintenance cost s2(s t ); the operating cost r1(s t ) and maintenance cost r2(s t ) as the system state s t The system feedback cost r(s t );

[0014] Afterwards, through the system state transfer function P(s t+1 ∣s t ), get the next system status s t+1 .

[0015] Furthermore, the policy network is based on the system state s t , perform action selection, and obtain the maintenance action execution probability P through Gaussian process net (a t ):

[0016]

[0017] Among them, A π (s t ,a i ) is the system state s t The i-th maintenance action a predicted by the next policy network i Advantage function of

[0018] Use the maximum probability estimation method to select the action and output the maintenance action a formulated by the system t :

[0019] a t =argmax(P net (a t ))

[0020] The value network input system state s t , based on the Gaussian process output of the system state s t Value Assessment π (s t ).

[0021] Furthermore, the action selection selects maintenance action exploration or maintenance strategy utilization through the ε-greedy algorithm to formulate the next maintenance action; when the ε-greedy algorithm sampling is greater than ε, maintenance action exploration is performed, and the advantage function is obtained by random sampling within the Gaussian process uncertainty range through the policy network; when the ε-greedy algorithm sampling is less than ε, maintenance strategy utilization is performed, and the Gaussian process mean is output through the policy network. Advantage function.

[0022] Furthermore, Bayesian data fusion is used to fuse prior maintenance experience data with maintenance actions obtained through the strategy network to assist in decision-making guidance.

[0023] Furthermore, the updating strategy network and value network include the following sub-steps: based on the original sampling point system state Advantage function A π (s t ,a t ) old System status with new sampling point Advantage function A π (s t ,a t ) new , extended Gaussian process fitting data set:

[0024] The input is: The output is:

[0025] Among them, A π (s t ,a t ) is state s t Next, perform action a t The advantage function in this case is calculated as:

[0026] A π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t )

[0027] Where γ is the attenuation coefficient, r(s t ,a t ) is the system feedback cost function;

[0028] Construct a Gaussian process based on the covariance matrix as the updated policy network:

[0029]

[0030] in, is the variance of the observation noise, I is the identity matrix; K is the covariance matrix calculated by the Gaussian process kernel function:

[0031]

[0032] Correct the value V by time difference method π (s t ), and perform batch training to update the value network.

[0033] Furthermore, the kernel function of the Gaussian process is:

[0034]

[0035] Among them, x and x ′ is the input vector, ∥xx ′ ∥ 2 is the square of the Euclidean distance, l is the length scale parameter, which controls the smoothness of the function, is the signal variance, which controls the overall magnitude of the kernel function.

[0036] Furthermore, the updating of the training set includes the following sub-steps: obtaining the degree of fit between each training data in the training set and the value network by calculating the negative log-likelihood; re-sampling the replay buffer in batches randomly, and obtaining the degree of fit between the resampling points and the value network by calculating the negative log-likelihood; and comparing, replacing the data with a relatively high degree of fit in the training set with data with a relatively low degree of fit, so that the model tends to train data with a low degree of fit with the model, thereby improving the overall reliability of the model.

[0037] As a second aspect, the present invention further provides an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning, comprising:

[0038] The aircraft engine model construction module is used to build a system state transition model and a system feedback model based on the collected aircraft engine operation data and Bayesian network and Gaussian process;

[0039] Network initialization module, used to initialize the strategy network and value network;

[0040] The maintenance action decision module is used to formulate maintenance strategies based on the current system status of the aircraft engine model, avoid the aircraft engine from experiencing a high degree of deterioration, and reduce system feedback costs;

[0041] Replay buffer module, used to store the state-action-value group of each step;

[0042] The model training module is used to train the policy network and value network using the state-action-value groups extracted in batches by the replay buffer module;

[0043] The dynamic update module is used to dynamically update the training set and replace the data with a higher degree of fit to the value network with data with a lower degree of fit.

[0044] The beneficial effect of the present invention is that, by constructing an aero-engine model and using a non-parametric reinforcement learning method based on a Gaussian process, a data-sparse engine model can be fitted using a Bayesian network and a Gaussian process, while ensuring similarity to high-precision simulation. While integrating system uncertainty, the present invention can improve the degree of fit between the overall training data and the model through dynamic data updates, thereby improving the sampling efficiency of the algorithm. The present invention also improves the safety and sampling efficiency of the system by integrating uncertainty through an action selection method based on prior maintenance empirical data. The effect is better than existing methods, and it has the advantages of strong versatility and ease of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the agent learning process of the present invention;

[0046] Figure 2 This is a schematic diagram of the process of establishing an aero-engine model according to the present invention;

[0047] Figure 3 This is a block diagram of the aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning according to the present invention;

[0048] Figure 4 is the fitting effect of the aircraft engine model, where (a) is the comparison between the model prediction result and the actual result, and (b) is the global sensitivity analysis of the feedback model on the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor;

[0049] Figure 5 The following figure compares the training results of this method with those of the traditional decision-making algorithm. (a) shows the convergence results of the algorithm training, and (b) shows a box plot of the five results of the actual application of the algorithm after training and on the aircraft engine model.

[0050] Figure 6 This method formulates specific maintenance strategies for aircraft engine high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor components. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0053] The core technology of the present invention is to mathematically model the historical data of aircraft engines with sparse data characteristics, construct an aircraft engine model, and use non-parametric reinforcement learning to solve the model based on the system uncertainty and sparse data characteristics of the aircraft engine case. Non-parametric reinforcement learning uses the system uncertainty quantification and non-parametric characteristics of the Gaussian process to integrate the system uncertainty in the aircraft engine scenario into the overall process of the reinforcement learning algorithm, and solves the sparse data set and uncertainty problems of real industrial scenarios. It also uses the Bayesian data fusion algorithm to integrate prior maintenance experience into the maintenance strategy exploration process, improves the algorithm sampling efficiency, and thus solves the predictive maintenance problem of the aircraft engine system, and realizes the aircraft engine system operation and maintenance strategy formulation method that integrates system uncertainty. The ideas can be summarized as follows:

[0054] 1. Build an aircraft engine model. Based on historical engine operating data, the state transition probabilities of the aircraft engine model are modeled using a Bayesian network. The system feedback cost is fitted using a Gaussian process to establish an aircraft engine model that can interact with the reinforcement learning process.

[0055] 2. A non-parametric reinforcement learning method based on Gaussian processes is proposed. The strategy and value functions of the reinforcement learning algorithm are constructed based on Gaussian processes. Actions are explored and utilized based on the uncertainty of the Gaussian process. A Bayesian data fusion algorithm is used to fuse prior maintenance experience data with reinforcement learning decision results. The uncertainty of the engine system is incorporated into the reinforcement learning process. The deterioration of the engine subsystem is used as a safety evaluation indicator, and the system feedback cost is used as an economic evaluation indicator for the maintenance plan. This allows the development of an aircraft engine system operation and maintenance strategy, improving the overall safety and reliability of the system.

[0056] 3. Dynamically update the model sampling points. Based on the negative log-likelihood of the training data and the value network used by each model, the degree of fit between the training data and the value function model is calculated. Through dynamic updating, training points with a low degree of fit to the model are prioritized for batch training. This improves the fit of the reinforcement learning network to the overall model sampling points and enhances the reliability of the system.

[0057] 4. Propose an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning.

[0058] The present invention provides an aircraft engine maintenance strategy optimization method based on non-parametric reinforcement learning, such as Figure 1 As shown, the specific steps include:

[0059] (1) Establishment of aircraft engine model: Establish an aircraft engine model for algorithm verification. The aircraft engine model consists of two parts: the system state transition model and the system feedback model. The system state transition model is responsible for judging the engine state deterioration process according to the current aircraft engine state, and the system feedback model is responsible for calculating the total cost of each engine operation and maintenance as a system reward. Figure 2 As shown, it includes the following sub-steps:

[0060] (1.1) System state transition model: Based on the current state of the aircraft engine, the engine state deterioration process is judged. Due to the high operating cost of aircraft engines, there are few actual flight failure data and historical operation and maintenance data of the engine, and most of the historical aircraft engine operation data is sparse data. In order to fit the transition probability between different states under each system state, the deterioration state s of the engine at time t is first calculated based on the sparse engine flight failure data. t (hereinafter referred to as system state s t ) is defined, and the state transition probability P(s t+1 ∣s t ) for fitting.

[0061] The state transfer process between different components of the engine is characterized by an isentropic relationship and a calibrated polynomial efficiency: the state parameter value of the i-th component is the stagnation pressure stagnation temperature stagnation enthalpy Pressure ratio π i , the specific heat of the working fluid at constant pressure is The specific heat ratio of the working fluid is γ i , the efficiency factor is η i The state parameter transfer equation between different components is:

[0062]

[0063] The state definition of the system is simulated by the gamma distribution. The degradation equation is:

[0064]

[0065] Where Δη is the damage growth, α is the shape parameter, β is the scale parameter, and Γ is the gamma distribution. We generated 100 engine operation segments, each consisting of 50 steps of degradation, and discretized the state into five levels, with higher values ​​indicating increasing degradation.

[0066] During the Bayesian network fitting process, the state transition probability of the aircraft engine is obtained based on the Markov process using the following formula:

[0067]

[0068] S0 is the state before the transfer, and S1 is the state after the transfer.

[0069] The Bayesian network parameter L is expressed by the maximum likelihood estimation method:

[0070]

[0071] Among them, s i is a sampling of the state transition sequence during a state transition process, and h represents the hidden nodes of the network. In addition, this example uses the expectation maximization algorithm and Jensen's inequality to obtain the lower bound of the log-likelihood:

[0072]

[0073] Among them, the q function is s i The probability distribution of , and satisfies: ∑ h q(h|s i )=1 and 0≤q(h|s i )≤1.

[0074] (1.2) System Feedback Model: Based on sparse historical maintenance data, the system fits the aircraft engine operation and maintenance costs resulting from deterioration state transitions. Aircraft engine systems incur system operation and maintenance costs during operation and as they degrade or recover from one state to another. Due to the sparsity of the system, problems such as missing data may occur. Therefore, a Gaussian process is used to fit the system feedback costs of aircraft engine systems. As a non-parametric random process, the Gaussian process performs better than other fitting algorithms in dealing with missing data in sparse systems.

[0075] Collect historical operating cost data of each engine component and analyze the status of the aircraft engine system t and operating cost r1(s t )Use x (i) ,y (i) To express it, the kernel function of the Gaussian process is:

[0076]

[0077] This function is used to measure the similarity of input data, where x and x ′ is the input vector, ∥xx ′ ∥ 2 is the square of the Euclidean distance. l is the length scale parameter that controls the smoothness of the function. is the signal variance, which controls the overall amplitude of the kernel function. For a given data set (X, Y), the marginal likelihood of the Gaussian process is:

[0078]

[0079] Where K is the covariance matrix and n is the number of training data points. The mean μ and variance σ of the running cost predicted by the Gaussian process are:

[0080] μ=k(x * ,x) T K -1 y

[0081] σ=k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)

[0082] Among them, x * is the data point to be predicted. Based on the Gaussian process prediction mean, the engine operating cost r1(s t ).

[0083] According to the maintenance actions determined by the strategy network for each component in the system state and the corresponding maintenance price (according to the current factory quotation), the maintenance prices of each component are summed up as the maintenance price of the engine in the system state s t Maintenance cost r2(s t ).

[0084] Engine operating cost r1(s t ) and maintenance cost r2(s t ) is the system feedback cost r(s t ).

[0085] (1.3) Integration and initialization of the aircraft engine system: The mathematical model of the aircraft engine system proposed in this example consists of a system state transition model and a system feedback model. During operation, the model accepts the current state of the system and gives the next state and system feedback cost based on the current state of the system and maintenance actions. After the aircraft engine model runs for a period of time, the system transfer function P(s) obtained in step (1.1) of the aircraft engine system is t+1 ∣s t ) Based on the state s of this cycle t and maintenance actions taken t Determine the aircraft engine status s for the next cycle t+1 And calculate the state s in this cycle through the system feedback model t The corresponding engine operating cost r1(s t ) and maintenance cost r2(s t), integrating the operating cost and maintenance cost as the system feedback cost r(s t ).

[0086] Following the above method, we obtain the aircraft engine model and initialization state. This example uses non-parametric networks such as Bayesian networks and Gaussian processes for modeling, which can effectively fit the characteristics of sparse data sets.

[0087] (2) Initialize the decision system network and formulate a strategy: Establish and initialize the algorithm strategy function Pnet and the value function Vnet, and formulate a maintenance strategy based on the initial state of the aircraft engine model in step (1) through the strategy network and experience assistance. It includes the following sub-steps:

[0088] (2.1) Based on the aero-engine model system state transfer function P(s) in step (1) t+1 ∣s t ) and system feedback R(s t ), for the maintenance strategy of the engine, this example establishes a reinforcement learning strategy network Pnet to formulate a system maintenance strategy. In the strategy network Pnet, the system state s is input. t , and obtain the maintenance action execution probability P(a t ), the system state s t and the system's advantage function A π (s t ,a t ) Use x,y to express and establish a Gaussian process:

[0089] The kernel function of the Gaussian process is:

[0090]

[0091] This function is used to measure the similarity of input data, where x and x ′ is the input vector, ∥xx ′ ∥ 2 is the square of the Euclidean distance. l is the length scale parameter that controls the smoothness of the function. is the signal variance, which controls the overall amplitude of the kernel function. For a given data set (X, Y), the marginal likelihood of the Gaussian process is:

[0092]

[0093] Where K is the covariance matrix and n is the number of training data points. The mean μ and variance σ of the system feedback predicted by the Gaussian process are:

[0094] μ=k(x * ,x) T K -1 y

[0095] σ=k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)

[0096] Among them, x * is the data point to be predicted.

[0097] The policy network Pnet selects actions based on the system state. Action selection uses the ε-greedy algorithm to choose between action exploration and policy exploitation to determine the advantage function for each maintenance action and then determine the next maintenance action. ε is a number between 0 and 1. Random sampling is performed on the 0-1 range. If the sampled point is greater than ε, action exploration is used, where the policy network randomly samples within the uncertainty range of a Gaussian process to obtain the advantage function. Otherwise, policy exploitation is used, where the policy network performs a Gaussian process mean to output the advantage function.

[0098] Action exploration is performed by random sampling in the Gaussian process uncertainty range σ by the policy network:

[0099] σ=k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)

[0100] Action exploration can realize random action exploration within a certain range to find better maintenance actions.

[0101] Strategy utilization uses the strategy network obtained through training to output the Gaussian process mean, inheriting the results of previous training.

[0102] The execution probability of the maintenance action is calculated based on the advantage function of each maintenance action output by the Gaussian process. The execution probability of the maintenance action is:

[0103]

[0104] a π (s t ,a i ) is the state s t The i-th maintenance action a predicted by the next policy network i Advantage function of

[0105] The maximum probability estimation method is used to select the action:

[0106] at =argmax(P net (a t ))

[0107] a t This is the specific maintenance action that the policy network Pnet optimizes based on the maintenance action execution probability formulated for the current system state, which serves as the output of the policy network Pnet. In the aviation field, the maintenance actions include major repairs and minor repairs.

[0108] After deriving the maintenance actions of the policy network, Bayesian data fusion is used to combine the prior maintenance experience data with the policy network, and the prior maintenance experience data is used for guidance to improve the security of the system, improve sampling efficiency, and accelerate algorithm convergence.

[0109]

[0110] Among them, P a is the decision action probability of the policy network, P ae is the action execution probability of the prior maintenance experience data, and w is the influence weight of the prior maintenance experience on the decision.

[0111] (2.2) The reinforcement learning process requires a value network Vnet to evaluate the value of the current state. In a Markov process, the meaning of the value function Vnet is the expected sum of the feedback values ​​of the current state s:

[0112] V π (s)=E π [G t ∣S t =s]

[0113] Among them, G t Feedback for the system. Input the system status s into the value network Vnet t , get the action value estimate V formulated by the system t . Set the system status s t and the estimated value V of the system action t Use x (i) ,y (i) Representation is performed, and a Gaussian process is established. The Gaussian kernel function and marginal likelihood are consistent with step (2.1). The output of the Gaussian process is the value network Vnet's value assessment of the system state V π (s).

[0114] (2.3) Based on the reinforcement learning policy network Pnet and value network Vnet generated in steps (2.1) and (2.2), the aircraft engine model established in step (1) is randomly initialized, the system state s0 is randomly selected, and the state is input into the policy network Pnet to formulate the first maintenance action a0.

[0115] (3) Strategy iteration and network training: Based on the initial maintenance action a0 and the initial system state s0 formulated in step (2), interact with the aircraft engine model established in step (1) to obtain a set of state-action value groups; establish a replay buffer to store the state-action value group (s t ,a t ,R(s t ),s t+1 ) is stored in a replay buffer and sampled as a training set; the policy network Pnet and the value network Vnet are updated; and the training set is dynamically updated through negative log-likelihood.

[0116] (3.1) Based on the initial system state s0 and initial maintenance action a0 obtained in step (2.3), interact with the aircraft engine system to obtain the next state of the system and the cost of the state transition process. This process is organized into a state-action group (s t ,a t ,R(s t ),s t+1 Since both the policy network Pnet and the value network Vnet require batch data for training, a data replay buffer is needed to store each state-action set and extract them in batches for training the policy network and value network. The state-action set for this step is stored in the replay buffer.

[0117] (3.2) Based on the replay buffer established in step (3.1), batch extract training data and predict V based on the state under the guidance of strategy π according to the value network π (s t ), calculate the advantage function A π :

[0118] A π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t )

[0119] Among them, γ is the attenuation coefficient, r(s t ,a t) is the system feedback cost function.

[0120] Through the advantage function, based on the system state of the original sampling point (i.e. the sampling point of the previous cycle) Advantage function A π (s t ,a t ) old The system status at the new sampling point (i.e. the sampling point of this cycle) Advantage function A π (s t ,a t ) new , expand the Gaussian process to fit the data set and update the policy network:

[0121] The expanded Gaussian process input is: The output is:

[0122] A π (s t ,a t ) is state s t Next, perform action a t The advantage function in the case of , the expanded overall covariance matrix:

[0123]

[0124] Construct a Gaussian process based on the covariance matrix as the updated policy network:

[0125]

[0126] Among them, K is the covariance matrix calculated by the Gaussian process kernel function, is the variance of the observation noise, and I is the identity matrix.

[0127] The value function V is calculated by using the time difference method TD-Error. π (s t ) for correction, batch training, and Gaussian process fitting:

[0128]

[0129] Among them, p(s t+1 ∣s t ,a t ) is the state transfer function, r(s t ,a t ) is the system feedback cost function.

[0130] (3.3) Based on the training data extracted in batches in step (3.2), calculate the negative log-likelihood between each training data and the value network Vnet:

[0131]

[0132] Among them, e is the number of training data extracted in batches, σ i With μ i are the variance and mean of the value function prediction of the i-th training data. i is the system state value V after correction using the time difference method π (s ι ), the negative log-likelihood indicates the degree of fit between each training data and the value function model. In order to select more training data that overlaps with the model fit from the replay buffer for training, this example also resamples the replay buffer (batch random sampling), and the set of resampling points is e * , and calculate e * The negative log-likelihood of the sample point is larger, and the smaller the negative log-likelihood is, the worse the fit between the sample point and the model is. u The negative log-likelihood of e is calculated with respect to e and e * The ratio of the sum of the negative log-likelihoods is used as the training data replacement probability to replace the training data point. The smaller the probability, the more likely it is to retain the current training data point e, otherwise it tends to replace the current training data point e with the re-sampled point e * , making the model tend to train data with a lower degree of model fit, thereby improving the overall reliability of the model.

[0133]

[0134] Where e is the set of sampling points used for training, e * is the set of sampling points for re-sampling.

[0135] (3.4) Determine whether the system operation step has reached the pre-set system training step: if it has reached the training step, stop training; otherwise, continue action selection and repeat step (3) until the operation step reaches the training step.

[0136] Example 1:

[0137] The embodiment of the present invention was implemented on a machine equipped with an Intel(R) Xeon(R) Gold 6330 CPU central processing unit, an NVidia GTX3090 graphics processor and 128GB of memory. Figures 4-6 The results shown.

[0138] The aero-engine model is built for the GE90 engine of General Motors, and the state transfer matrix and system feedback of the engine are fitted. The fitting results are as follows: Figure 4 As shown, Figure 4 (a) shows the comparison between the model prediction results and the actual results. The vertical axis is the predicted value and the horizontal axis is the actual value. The circled points are the actual data points, and the solid plus sign points are the predicted points. This method can accurately predict the state transition process and system feedback of the aircraft engine. Figure 4 (b) Shows the global sensitivity analysis of the system feedback model to the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor. Figure 5 (a) shows the performance of this method compared to rule-based maintenance methods, deep value reinforcement learning, and actor-critic reinforcement learning. The bottom curve represents the convergence of this method, while the dashed line above represents the rule-based maintenance algorithm. Nonparametric reinforcement learning significantly reduces system feedback costs compared to rule-based maintenance algorithms. The two top convergence curves are for deep value reinforcement learning and actor-critic reinforcement learning, respectively. Even after 1000 training cycles, the convergence results of these two algorithms are still inferior to those of nonparametric reinforcement learning after 100 training cycles. Both algorithms exhibit strong oscillations during convergence, indicating low sampling efficiency and instability in training. Nonparametric reinforcement learning, on the other hand, is relatively stable during training, demonstrating higher sampling efficiency. The results demonstrate that nonparametric reinforcement learning improves data utilization efficiency by over a hundred times compared to traditional reinforcement learning methods. While incorporating system uncertainty, this method improves the fit between the overall training data and the model through dynamic data updates, thereby enhancing the algorithm's sampling efficiency. Figure 5 (b) shows the box plot of the average effect of this method after 5 actual applications after training compared with the rule-based maintenance method, deep value reinforcement learning algorithm, and actor-critic reinforcement learning algorithm. The box plot result of this method is on the far left. The upper limit, lower limit and mean of the system feedback cost of the non-parametric reinforcement learning method are much lower than those of the comparison method, and the actor-critic model also has decision divergence. It can be seen that this method can relatively stably formulate maintenance strategies for aircraft engines. Figure 6 The method demonstrates specific maintenance strategies for the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor. Circles represent minor repairs, while inverted triangles represent major repairs. The results demonstrate that this method can accurately mathematically model aircraft engines and accurately formulate appropriate maintenance strategies for their various components.

[0139] This embodiment also provides an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning, which is used to implement the above embodiment. The terms "module", "unit", etc. used below can refer to a combination of software and / or hardware that can implement a predetermined function.

[0140] This embodiment provides an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning, such as Figure 3 Shown, including:

[0141] The aircraft engine model construction module is used to build a system state transition model and a system feedback model based on the collected aircraft engine operation data and Bayesian network and Gaussian process;

[0142] Network initialization module, used to initialize the strategy network and value network;

[0143] The maintenance action decision module is used to formulate maintenance strategies based on the current system status of the aircraft engine model to avoid the aircraft engine from being severely deteriorated and reduce the system feedback cost r(s t );

[0144] The replay buffer module is used to store the state-action-value group of each step so that it can be extracted in batches to train the policy network and value network;

[0145] The model training module is used to train the policy network and value network using the state-action-value groups extracted in batches by the replay buffer module;

[0146] The dynamic update module is used to dynamically update the training set and replace the data with a higher degree of fit to the value network with data with a lower degree of fit.

[0147] It should be noted that the above embodiments are only used to illustrate the advantages and specific steps of the present invention, and do not limit the scope of application of the present invention. Other forms of simulations performed by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for optimizing aircraft engine maintenance strategy based on non-parametric reinforcement learning, characterized in that: The following steps are involved: Collect historical operation data of aircraft engines and establish aircraft engine models; Establish a strategy network and a value network, randomly select the initial system state s0, and formulate the initial maintenance action a0; Based on the initial system state s0 and the initial maintenance action a0, interact with the aircraft engine model to obtain a set of state action value groups: system state, maintenance action, system feedback cost and next system state; establish a replay buffer to replay each step of the state action value group (s t , a t ,r(s t ), s t+1 ) into the replay buffer; performing random sampling in the replay buffer as a training set; Update the strategy network and value network; Update the training set; Determine whether the system operation step length reaches the set system training step length. If so, stop training, obtain the trained network, and output the aircraft engine maintenance optimization strategy; Among them, a t Indicates maintenance action; the aircraft engine model includes a system state transfer model and a system feedback model: The system state transfer model is fitted by a Bayesian network to the system state transfer probability obtained based on the Markov process, wherein the Bayesian network parameter L is expressed by the maximum likelihood estimation method; the expectation maximization algorithm is used to optimize the Bayesian network parameters; and the system state transfer function P(s t+1 |s t ); The system feedback model is based on the system state s t The running cost r1(s) is fitted by Gaussian process t );According to the system status s t The maintenance action determined by the strategy network is the sum of the prices corresponding to the maintenance actions of each component as the maintenance cost r2(s t ); the operating cost r1(s t ) and maintenance cost r2(s t ) is taken as the system state s t The system feedback cost r(s t ); Afterwards, through the system state transfer function P(s t+1 |s t ), get the next system status s t+1 .

2. The method according to claim 1, characterized in that The parameter values ​​of the system state include the stagnation pressure corresponding to each component i of the engine Stagnation temperature Stagnation enthalpy Pressure ratio π i , working fluid constant pressure specific heat Working fluid specific heat ratio γ i , efficiency factor η i ; The maintenance actions include major repairs and minor repairs.

3. The method according to claim 1, characterized in that The policy network is based on the system state s t , select actions, and obtain the maintenance action execution probability P through Gaussian process net (a t ): Among them, A π (s t , a i ) is the system state s t The i-th maintenance action a predicted by the next policy network i Advantage function of Use the maximum probability estimation method to select the action and output the maintenance action a formulated by the system t : a t =argmax(P net (a t )) The value network input system state s t , based on the Gaussian process output of the system state s t Value Assessment π (s t ).

4. The method according to claim 3, characterized in that The action selection selects maintenance action exploration or maintenance strategy utilization through the ε-greedy algorithm to formulate the next maintenance action; when the ε-greedy algorithm sampling is greater than ε, the maintenance action exploration is performed, and the advantage function is obtained by randomly sampling within the Gaussian process uncertainty range through the policy network; when the e-greedy algorithm sampling is less than ε, the maintenance strategy utilization is performed, and the Gaussian process mean output advantage function is performed through the policy network.

5. The method according to claim 3, characterized in that: Bayesian data fusion is used to fuse prior maintenance experience data with maintenance actions obtained through the strategy network to assist in decision-making guidance.

6. The method according to claim 3, characterized in that The updating strategy network and value network include the following sub-steps: based on the original sampling point system state Advantage function A π (s t , a t ) old System status with new sampling point Advantage function A π (s t , a t ) new , expand the Gaussian process to fit the data set: The input is: The output is: Among them, A π (s t , a t ) is state s t Next, perform action a t The advantage function in this case is calculated as: From π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t ) Where γ is the attenuation coefficient, r(s t , a t ) is the system feedback cost function; Construct a Gaussian process based on the covariance matrix as the updated policy network: in, is the variance of the observation noise, I is the unit matrix, and K is the covariance matrix calculated by the Gaussian process kernel function: Correct the value V by time difference method π (s t ), and perform batch training to update the value network.

7. The method according to any one of claims 3 to 6, characterized in that: The kernel function of the Gaussian process is: Among them, x and x′ are input vectors, ||xx′|| 2 is the square of the Euclidean distance, l is the length scale parameter, which controls the smoothness of the function, is the signal variance, which controls the overall magnitude of the kernel function.

8. The method according to claim 1, characterized in that The updating of the training set includes the following sub-steps: obtaining the degree of fit between each training data in the training set and the value network by calculating the negative log-likelihood; re-sampling the replay buffer in batches randomly, and obtaining the degree of fit between the resampling points and the value network by calculating the negative log-likelihood; and comparing, replacing the data with a relatively high degree of fit in the training set with data with a relatively low degree of fit.

9. An aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning, characterized in that: include: The aircraft engine model building module is used to build a system state transfer model and a system feedback model based on the collected aircraft engine operation data and the Bayesian network and Gaussian process; The system state transfer model fits the system state transfer probability obtained based on the Markov process through the Bayesian network, wherein the Bayesian network parameters are represented by the maximum likelihood estimation method; the expectation maximization algorithm is used to optimize the Bayesian network parameters; the system state transfer function is obtained; the system feedback model fits the operating cost through the Gaussian process based on the system state; according to the maintenance action determined by the policy network under the system state, the sum of the prices corresponding to the maintenance action of each component is used as the maintenance cost; the sum of the operating cost and the maintenance cost is used as the system feedback cost under the system state; then, the next system state is obtained through the system state transfer function; Network initialization module, used to initialize the strategy network and value network; The maintenance action decision module is used to formulate maintenance strategies according to the system status of the current aircraft engine model, avoid the aircraft engine from being highly deteriorated, and reduce the system feedback cost; Replay buffer module, used to store the state-action-value group of each step; A model training module, used to train the policy network and the value network using the state-action-value groups extracted in batches by the replay buffer module; The dynamic update module is used to dynamically update the training set and replace the data with a higher degree of fit to the value network with data with a lower degree of fit.

Images