Aero-engine maintenance strategy optimization method based on nonparametric reinforcement learning
Through the non-parametric reinforcement learning method, combined with Bayesian network and Gaussian process, an aero engine model is constructed, which solves the problem of inefficient maintenance decision-making in the existing technology, and achieves more efficient and safer maintenance strategy optimization.
Patent Information
- Application Number
- CN202411050722.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Existing aircraft engine maintenance decision-making solutions are difficult to provide effective maintenance strategies when facing complex system uncertainty and sensor failure scenarios, resulting in inefficient operation and maintenance and waste of resources.
Using a method based on non-parametric reinforcement learning, aero engine models are constructed through Bayesian networks and Gaussian processes, combined with prior maintenance empirical data, dynamically update the training set, and optimize maintenance strategies to improve the safety and economics of the system.
It improves the efficiency and accuracy of aircraft engine maintenance decisions, reduces system feedback costs, enhances the sampling efficiency of the algorithm and the fitting degree of model, and realizes effective handling of system uncertainty.
Smart Images

Figure CN119026460A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of aircraft engine operation and maintenance, and in particular to an aircraft engine maintenance strategy optimization method based on non-parametric reinforcement learning. Background Art
[0002] Aircraft engines are one of the important infrastructures for logistics and passenger transportation. Rational maintenance strategies are essential to maintaining the normal operation of aircraft engine equipment, improving reliability, and saving airline costs. However, in daily aircraft engine maintenance work, the engine equipment is complex, the working environment is relatively harsh, the system sensors have a high degree of uncertainty, the subcomponents have complex degradation mechanisms, and the engine system needs to be designed for safety assurance, resulting in low efficiency in aircraft engine system operation and maintenance and serious waste of resources.
[0003] In order to solve the above problems, it is necessary to incorporate system uncertainty into the scope of strategy formulation and improve the efficiency of the algorithm. Therefore, it is urgent to study a new aircraft engine operation and maintenance solution that can take into account the safety and economy of aircraft engine maintenance to achieve intelligent operation and maintenance of aircraft engines. In this context, an aircraft engine operation and maintenance strategy formulation method based on non-parametric reinforcement learning came into being.
[0004] The existing aircraft engine maintenance decision-making schemes mainly include: fault repair maintenance, preventive maintenance, and condition-based maintenance. Among them, fault repair maintenance is a maintenance action taken after a device or system fails, that is, a kind of unplanned maintenance, which usually occurs when the equipment fails or shuts down, and mainly includes detecting faults, diagnosing problems, repairing equipment, and restoring its normal operating state; preventive maintenance is a planned maintenance activity taken before the equipment fails, and the goal is to prevent equipment failures through regular inspections, maintenance and repairs to ensure that the equipment maintains efficient operation within the predetermined time; condition-based maintenance is a maintenance strategy that determines the timing of maintenance based on the actual working status and health status of the equipment, using sensors, monitoring equipment and technology to collect data to predict the health status of the equipment and decide when to perform maintenance. Existing aircraft engine maintenance decision-making schemes usually rely on static maintenance plans or current system status, which are insufficient in scenarios requiring rapid response and high safety levels, and it is difficult to give a correct maintenance plan for decision-making problems in system uncertainty scenarios such as sensor failure.
[0005] By introducing multi-agent reinforcement learning technology and combining it with Gaussian process uncertainty, we can better solve the system uncertainty in the aircraft engine maintenance system. Combined with prior maintenance experience data, we can make maintenance decisions for complex coupled multi-subsystem engines and improve the operating efficiency of the equipment. Summary of the invention
[0006] The purpose of the present invention is to provide an aircraft engine maintenance strategy optimization method based on non-parametric reinforcement learning in view of the deficiencies of the prior art.
[0007] To achieve the above object, the present invention adopts the following technical solution: As a first aspect of the present invention, a method for optimizing aircraft engine maintenance strategy based on non-parametric reinforcement learning is provided, comprising the following steps:
[0008] Collect historical operation data of aircraft engines and establish aircraft engine models;
[0009] Establish a strategy network and a value network, randomly select the initial system state s0, and formulate an initial maintenance action a0; based on the initial system state s0 and the initial maintenance action a0, interact with the aircraft engine model to obtain a set of state action value groups: system state, maintenance action, system feedback cost and next step system state; establish a replay buffer to replay the state action value group (s0) of each step t ,a t ,r(s t ),s t+1 ) is stored in the replay buffer; random sampling is performed in the replay buffer as a training set; the strategy network and the value network are updated; the training set is updated; it is determined whether the system operation step has reached the set system training step, if so, the training is stopped to obtain a trained network, and the aircraft engine maintenance optimization strategy is output, otherwise, action selection is continued in the strategy network, and the training is repeated until the operation step reaches the training step.
[0010] Furthermore, the parameter value of the system state includes the stagnation pressure corresponding to each component i of the engine Stagnation temperature Stagnation enthalpy Pressure ratio π i , working fluid constant pressure specific heat Working fluid specific heat ratio γ i , efficiency factor η i ; The maintenance actions include major repairs and minor repairs.
[0011] Furthermore, the aircraft engine model includes a system state transfer model and a system feedback model;
[0012] The system state transfer model is fitted by a Bayesian network to the system state transfer probability obtained based on the Markov process, wherein the Bayesian network parameter L is expressed by the maximum likelihood estimation method; the expectation maximization algorithm is used to optimize the Bayesian network parameters; and the system state transfer function P(s t+1 ∣s t );
[0013] The system feedback model is based on the system state s t The running cost r1(s) is fitted by Gaussian process t );According to the system status s t The maintenance action determined by the strategy network is the sum of the prices corresponding to the maintenance actions of each component as the maintenance cost s2(s t ); the operating cost r1(s t ) and maintenance cost r2(s t ) is taken as the system state s t The system feedback cost r(s t );
[0014] Afterwards, through the system state transfer function P(s t+1 ∣s t ), get the next system status s t+1 .
[0015] Furthermore, the policy network is based on the system state s t , select actions, and obtain the maintenance action execution probability P through Gaussian process net (a t ):
[0016]
[0017] Among them, A π (s t ,a i ) is the system state s t The i-th maintenance action a predicted by the next policy network i Advantage function of
[0018] Use the maximum probability estimation method to select the action and output the maintenance action a formulated by the system t :
[0019] a t = argmax(P net (a t ))
[0020] The value network input system state s t , based on the Gaussian process output of the system state s t Value Assessment π (s t ).
[0021] Furthermore, the action selection selects maintenance action exploration or maintenance strategy utilization through the ε-greedy algorithm to formulate the next maintenance action; when the ε-greedy algorithm sampling is greater than ε, maintenance action exploration is performed, and the advantage function is obtained by randomly sampling within the Gaussian process uncertainty range through the policy network; when the ε-greedy algorithm sampling is less than ε, maintenance strategy utilization is performed, and the Gaussian process mean output advantage function is performed through the policy network.
[0022] Furthermore, Bayesian data fusion is used to fuse prior maintenance experience data with maintenance actions obtained through the strategy network to assist in decision-making guidance.
[0023] Furthermore, the updating strategy network and value network include the following sub-steps: based on the original sampling point system state Advantage function A π (s t ,a t ) old System status with new sampling point Advantage function A π (s t ,a t ) new , expand the Gaussian process to fit the data set:
[0024] The input is: The output is:
[0025] Among them, A π (s t ,a t ) is state s t Next, perform action a t The advantage function in this case is calculated as:
[0026] A π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t )
[0027] Where γ is the attenuation coefficient, r(s t ,a t ) is the system feedback cost function;
[0028] Construct a Gaussian process based on the covariance matrix as the updated policy network:
[0029]
[0030] in, is the variance of the observation noise, I is the unit matrix, and K is the covariance matrix calculated by the Gaussian process kernel function:
[0031]
[0032] Correct the value V by time difference method π (s t ), and perform batch training to update the value network.
[0033] Furthermore, the kernel function of the Gaussian process is:
[0034]
[0035] Among them, x and x ′ is the input vector, ∥xx ′ ∥ 2 is the square of the Euclidean distance, l is the length scale parameter, which controls the smoothness of the function, is the signal variance, which controls the overall magnitude of the kernel function.
[0036] Furthermore, the updating of the training set includes the following sub-steps: obtaining the degree of fit between each training data in the training set and the value network by calculating the negative log-likelihood; re-sampling the replay buffer in batches randomly, and obtaining the degree of fit between the resampling points and the value network by calculating the negative log-likelihood; and comparing, replacing the data with a relatively high degree of fit in the training set with data with a relatively low degree of fit, so that the model tends to train the data with a low degree of fit with the model, thereby improving the overall reliability of the model.
[0037] As a second aspect, the present invention further provides an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning, comprising:
[0038] The aircraft engine model building module is used to build a system state transfer model and a system feedback model based on the collected aircraft engine operation data and the Bayesian network and Gaussian process;
[0039] Network initialization module, used to initialize the strategy network and value network;
[0040] The maintenance action decision module is used to formulate maintenance strategies according to the system status of the current aircraft engine model, avoid the aircraft engine from being highly deteriorated, and reduce the system feedback cost;
[0041] Replay buffer module, used to store the state-action-value group of each step;
[0042] A model training module, used to train the policy network and the value network using the state-action-value groups extracted in batches by the replay buffer module;
[0043] The dynamic update module is used to dynamically update the training set and replace the data with a higher degree of fit to the value network with data with a lower degree of fit.
[0044] The beneficial effect of the present invention is that, by constructing an aero-engine model and a non-parametric reinforcement learning method based on Gaussian process, the data-sparse engine model can be fitted through the Bayesian network and the Gaussian process, while ensuring the similarity with the high-precision simulation. While integrating the uncertainty of the system, the present invention can improve the degree of fit between the overall training data and the model by means of dynamic data updating, and improve the sampling efficiency of the algorithm. The present invention also improves the safety and sampling efficiency of the system by means of action selection based on prior maintenance experience data to integrate uncertainty. The effect is better than the existing method, and has the advantages of strong versatility and ease of use. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 is a schematic diagram of the intelligent agent learning process of the present invention;
[0046] Figure 2 is a schematic diagram of the process of establishing the aircraft engine model according to the present invention;
[0047] Figure 3 It is a block diagram of the aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning according to the present invention;
[0048] Figure 4 It is the fitting effect of the aircraft engine model, where (a) is the comparison between the model prediction result and the actual result, and (b) is the global sensitivity analysis of the feedback model to the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor;
[0049] Figure 5 The figure is a comparison diagram of the training results of the method and the traditional decision-making algorithm, where (a) is the algorithm training convergence result diagram, and (b) is a box plot of the five results of the actual application of the algorithm after training and the aircraft engine model;
[0050] Figure 6 This method formulates specific maintenance strategies for aircraft engine high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor components. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.
[0053] The core technology of the present invention is to mathematically model the historical data of aircraft engines with sparse data characteristics, construct an aircraft engine model, and use non-parametric reinforcement learning to solve the model for the system uncertainty and sparse data characteristics of aircraft engine cases. Non-parametric reinforcement learning uses the system uncertainty quantification and non-parametric characteristics of Gaussian processes to integrate the system uncertainty in aircraft engine scenarios into the overall process of reinforcement learning algorithms, and solves the sparse data set and uncertainty problems in real industrial scenarios. It also uses the Bayesian data fusion algorithm to integrate prior maintenance experience into the maintenance strategy exploration process, improves the algorithm sampling efficiency, and thus solves the predictive maintenance problem of aircraft engine systems, and realizes an aircraft engine system operation and maintenance strategy formulation method that integrates system uncertainty. The ideas can be summarized as follows:
[0054] 1. Build an aircraft engine model. Based on the historical operation data of the engine, the state transition probability of the aircraft engine model is modeled through the Bayesian network, and the system feedback cost is fitted through the Gaussian process to build an aircraft engine model that can interact with the reinforcement learning process.
[0055] 2. A non-parametric reinforcement learning method based on Gaussian process is proposed. The strategy function and value function of the reinforcement learning algorithm are constructed based on Gaussian process, and the action is explored and utilized based on the uncertainty of Gaussian process. The Bayesian data fusion algorithm is used to fuse the prior maintenance experience data with the reinforcement learning decision results, and the uncertainty of the engine system is integrated into the reinforcement learning process. The deterioration state of the engine subsystem is used as a safety evaluation indicator, and the system feedback cost is used as an economic evaluation indicator of the maintenance plan. The operation and maintenance strategy of the aviation engine system is formulated, which improves the overall safety and reliability of the system.
[0056] 3. Adopt the dynamic update method of model sampling points. Based on the negative log-likelihood of the training data and the value network used by each model, calculate the degree of fit between the training data and the value function model, and use the dynamic update method to give priority to the training points with a lower degree of fit to the model for batch training, thereby improving the degree of fit of the reinforcement learning network to the overall model sampling points and improving the reliability of the system.
[0057] 4. Propose an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning.
[0058] The present invention provides an aircraft engine maintenance strategy optimization method based on non-parametric reinforcement learning, such as Figure 1 As shown, the specific steps include:
[0059] (1) Establishment of aircraft engine model: Establish an aircraft engine model for algorithm verification. The aircraft engine model consists of two parts: a system state transition model and a system feedback model. The system state transition model is responsible for judging the engine state deterioration process based on the current aircraft engine state, and the system feedback model is responsible for calculating the total cost of each engine operation and maintenance as a system reward. Figure 2 As shown, it includes the following sub-steps:
[0060] (1.1) System state transition model: Based on the current state of the aircraft engine, the engine state deterioration process is judged. Due to the high operating cost of aircraft engines, there are few actual flight failure data and historical operation and maintenance data of the engine, and most of the historical aircraft engine operation data are sparse data. In order to fit the transition probability between different states under each system state, the deterioration state s of the engine at time t is first calculated based on the sparse engine flight failure data. t (hereinafter referred to as system state s t ) is defined, and the state transition probability P(s t+1 ∣s t ) for fitting.
[0061] The state transfer process between different components of the engine is characterized by an isentropic relationship and a calibrated polynomial efficiency: the state parameter value of the i-th component is the stagnation pressure Stagnation temperature Stagnation enthalpy Pressure ratio π i , the specific heat of the working fluid at constant pressure is The specific heat ratio of the working fluid is γ i , the efficiency factor is η i The state parameter transfer equation between different components is:
[0062]
[0063] The state definition of the system is simulated by the gamma distribution. The degradation equation is:
[0064]
[0065] Among them, Δη is the damage growth, α is the shape parameter, β is the scale parameter, and Γ is the gamma distribution. 100 engine operation segments are generated, each segment consists of 50 steps of deterioration process, and the state is discretized into 5 levels, with higher values indicating increasing deterioration.
[0066] In the Bayesian network fitting process, the state transition probability of the aircraft engine is obtained based on the Markov process using the following formula:
[0067]
[0068] Where S0 is the state before the transition and S1 is the state after the transition.
[0069] The Bayesian network parameter L is represented by the maximum likelihood estimation method:
[0070]
[0071] where s i is a sample of the state transition sequence in a single state transition process, and h represents the network hidden node. Moreover, in this example, the lower bound of the log-likelihood is obtained by the expectation maximization algorithm using Jensen's inequality:
[0072]
[0073] where the q function is the probability distribution of s i and satisfies: ∑ h q(h|s i ) = 1 and 0 ≤ q(h|s i ) ≤ 1.
[0074] (1.2) System feedback model: Based on sparse historical maintenance data, the operating and maintenance costs of an aero-engine generated by the deterioration state transition of the aero-engine are fitted. The aero-engine system generates system operating and maintenance costs during operation and during the process of deteriorating or repairing from one state to another. And due to the sparsity of the system, problems such as missing data will occur. Therefore, a Gaussian process is used to fit the system feedback cost of the aero-engine system. As a non-parametric stochastic process, the Gaussian process has a better effect in the face of the data missing problem of the sparse system compared with other fitting algorithms.
[0075] Collect the historical operating cost data of each component of the engine, and represent the aero-engine system state s t and the operating cost r1(s t ) using x (i) , y (i) . The kernel function of the Gaussian process is:
[0076]
[0077] This function is used to measure the similarity of the input data, where x and x ′ are input vectors, and ∥x - x ′ ∥ 2 is the square of the Euclidean distance. l is the length scale parameter that controls the smoothness of the function. is the signal variance that controls the overall amplitude of the kernel function. For a given data set (X, Y), the marginal likelihood of the Gaussian process is:
[0078]
[0079] Where K is the covariance matrix and n is the number of training data points. The mean μ and variance σ of the operating cost predicted by the Gaussian process are:
[0080] μ=k(x * ,x) T K -1 y
[0081] σ=k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)
[0082] Among them, x * is the data point to be predicted. Based on the Gaussian process prediction mean, the engine operating cost r1(s t ).
[0083] According to the maintenance actions determined by the strategy network for each component in the system state, and the corresponding maintenance price of the maintenance action (according to the current factory quotation), the maintenance prices of each component are summed up as the maintenance price of the engine in the system state s t The maintenance cost under r2(s t ).
[0084] The operating cost of the engine is r1(s t ) and maintenance cost r2(s t ) is taken as the system feedback cost r(s t ).
[0085] (1.3) Aircraft engine system integration and initialization: The mathematical model of the aircraft engine system proposed in this example consists of a system state transition model and a system feedback model. During operation, the model accepts the current state of the system and gives the next state and system feedback cost based on the current state of the system and maintenance actions. After the aircraft engine model runs for a period of time, the system transfer function P(s) obtained in step (1.1) of the aircraft engine system t+1 ∣s t ) Based on the state s of this cycle t and maintenance actions taken t Determine the aircraft engine status s for the next cycle t+1 , and calculate the state s in this cycle through the system feedback model t The corresponding engine operating cost r1(s t ) and maintenance cost r2(s t), integrating the operating cost and maintenance cost as the system feedback cost r(s t ).
[0086] According to the above method, the aircraft engine model and initialization state are obtained. This example uses non-parametric networks such as Bayesian networks and Gaussian processes for modeling, which can perform better fitting according to the characteristics of its sparse data set.
[0087] (2) Initialize the decision system network and formulate strategies: Establish and initialize the algorithm strategy function Pnet and the value function Vnet, and formulate a maintenance strategy based on the initial state of the aircraft engine model in step (1) through the strategy network and experience. It includes the following sub-steps:
[0088] (2.1) Based on the aero-engine model system state transfer function P(s) in step (1) t+1 ∣s t ) and system feedback R(s t ), for the maintenance strategy formulation of the engine, this example formulates the system maintenance strategy by establishing a reinforcement learning strategy network Pnet. In the strategy network Pnet, the system state s is input t , and obtain the maintenance action execution probability P(a t ), the system state s t and the system's advantage function A π (s t ,a t )Use x and y to represent and establish a Gaussian process:
[0089] The kernel function of the Gaussian process is:
[0090]
[0091] This function is used to measure the similarity of input data, where x and x ′ is the input vector, ∥xx ′ ∥ 2 is the square of the Euclidean distance. l is the length scale parameter that controls the smoothness of the function. is the signal variance, controlling the overall magnitude of the kernel function. For a given data set (X, Y), the marginal likelihood of the Gaussian process is:
[0092]
[0093] Where K is the covariance matrix and n is the number of training data points. The mean μ and variance σ of the system feedback predicted by the Gaussian process are:
[0094] μ=k(x * ,x) T K -1 y
[0095] σ=k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)
[0096] Among them, x * is the data point to be predicted.
[0097] The policy network Pnet selects actions based on the system state. Action selection is to select action exploration or policy utilization according to the ε-greedy algorithm to obtain the advantage function of each maintenance action to formulate the next maintenance action. ε is a number between 0 and 1, and 0-1 is randomly sampled. If the sampling point is greater than ε, action exploration is used, that is, the policy network is randomly sampled in the Gaussian process uncertainty range to obtain the advantage function; otherwise, policy utilization is used, and the Gaussian process mean output advantage function is performed through the policy network.
[0098] Action exploration is performed by random sampling in the Gaussian process uncertainty range σ by the policy network:
[0099] σ=k(x * ,x * )-k(x * ,x) T K -1 k(x * ,x)
[0100] Action exploration can realize random action exploration within a certain range to find better maintenance actions.
[0101] The strategy utilization uses the strategy network obtained through training to perform Gaussian process mean output, inheriting the results of previous training.
[0102] The execution probability of the maintenance action is calculated based on the advantage function of each maintenance action output by the Gaussian process. The execution probability of the maintenance action is:
[0103]
[0104] a π (s t ,a i ) is the state s t The i-th maintenance action a predicted by the next policy network i Advantage function of
[0105] The maximum probability estimation method is used to select actions:
[0106] at = argmax(P net (a t ))
[0107] a t That is, the specific maintenance action after the policy network Pnet optimizes the execution probability of the maintenance action formulated according to the current system state, which is used as the output of the policy network Pnet. The maintenance action in the aviation field includes overhaul and minor repair.
[0108] After deriving the maintenance actions of the policy network, Bayesian data fusion is used to combine the prior maintenance experience data with the policy network, and the prior maintenance experience data is used for guidance to improve the security of the system, improve sampling efficiency, and accelerate algorithm convergence.
[0109]
[0110] Among them, P a is the decision action probability of the policy network, P ae is the action execution probability of the prior maintenance experience data, and w is the influence weight of the prior maintenance experience on the decision.
[0111] (2.2) The reinforcement learning process requires the value network Vnet to evaluate the value of the current state. In a Markov process, the meaning of the value function Vnet is the expected sum of the feedback values of the current state s:
[0112] V π (s) = E π [G t ∣S t =s]
[0113] Among them, G t It is the system feedback. Input the system status s in the value network Vnet t , get the action value estimate V formulated by the system t . Set the system status s t and the action value estimate V of the system action t Using x (i) ,y (i) Represented, a Gaussian process is established, the Gaussian kernel function and marginal likelihood are consistent with step (2.1), and the Gaussian process output is the value network Vnet's value assessment of the system state V π (s).
[0114] (2.3) Based on the reinforcement learning policy network Pnet and value network Vnet generated in steps (2.1) and (2.2), the aircraft engine model established in step (1) is randomly initialized, the system state s0 is randomly selected and input into the policy network Pnet, and the first maintenance action a0 is formulated.
[0115] (3) Strategy iteration and network training: Based on the initial maintenance action a0 and the initial system state s0 formulated in step (2), interact with the aircraft engine model established in step (1) to obtain a set of state-action value groups; establish a replay buffer to store the state-action value group (s0) of each step. t ,a t ,R(s t ),s t+1 ) is stored in a replay buffer and sampled as a training set; the policy network Pnet and the value network Vnet are updated; and the training set is dynamically updated through negative log-likelihood.
[0116] (3.1) Based on the initial system state s0 and initial maintenance action a0 obtained in step (2.3), the aircraft engine system is interacted with to obtain the next state of the system and the cost of the state transition process. This process is organized into a state-action group (s t ,a t ,R(s t ),s t+1 ). Since both the policy network Pnet and the value network Vnet need batch data for training, it is necessary to establish a data replay buffer to store each state-action group and extract them in batches for training the policy network and the value network. The state-action group of this step is stored in the replay buffer.
[0117] (3.2) Based on the replay buffer established in step (3.1), batch extract training data and predict V based on the state under the guidance of strategy π according to the value network π (s t ), calculate the advantage function A π :
[0118] A π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t )
[0119] Among them, γ is the attenuation coefficient, r(s t ,a t) is the system feedback cost function.
[0120] Through the advantage function, based on the system state of the original sampling point (i.e. the sampling point of the previous cycle), Advantage function A π (s t ,a t ) old The system status of the new sampling point (i.e. the sampling point of this cycle) Advantage function A π (s t ,a t ) new , expand the Gaussian process to fit the data set and update the policy network:
[0121] The expanded Gaussian process input is: The output is:
[0122] A π (s t ,a t ) is state s t Next, perform action a t The advantage function in the case of , the expanded overall covariance matrix is:
[0123]
[0124] Construct a Gaussian process based on the covariance matrix as the updated policy network:
[0125]
[0126] Among them, K is the covariance matrix calculated by the Gaussian process kernel function, is the variance of the observation noise and I is the identity matrix.
[0127] The value function V is calculated by using the time difference method TD-Error. π (s t ) for correction, batch training, and Gaussian process fitting:
[0128]
[0129] Among them, p(s t+1 ∣s t ,a t ) is the state transfer function, r(s t ,a t ) is the system feedback cost function.
[0130] (3.3) Based on the training data extracted in batches in step (3.2), calculate the negative log-likelihood between each training data and the value network Vnet:
[0131]
[0132] Among them, e is the number of training data extracted in batches, σ i With μ i are the variance and mean of the value function prediction of the i-th training data. i is the system state value V after correction using the time difference method π (s ι ), the negative log likelihood indicates the degree of fit between each training data and the value function model. In order to select more training data that intersect the model fit degree from the replay buffer for training, this example also resamples the replay buffer (batch random sampling), and the resampling point set is e * , and calculate e * The negative log likelihood of the sampling point is smaller, and the larger the negative log likelihood is, the worse the fit between the sampling point and the model is. u The negative log-likelihood of e is calculated with respect to e and e * The ratio of the sum of the negative log-likelihoods of is used as the training data replacement probability to replace the training data point. The smaller the probability, the more likely it is to retain the current training data point e, otherwise it tends to replace the current training data point e with the re-sampled point e * , so that the model tends to train data with a lower degree of model fit, thereby improving the overall reliability of the model.
[0133]
[0134] Where e is the set of sampling points used for training, e * is the set of sampling points for re-sampling.
[0135] (3.4) Determine whether the system operation step has reached the preset system training step: if it has reached the training step, stop training; otherwise, continue to select actions and repeat step (3) until the operation step has reached the training step.
[0136] Embodiment 1:
[0137] The implementation example of the present invention is implemented on a machine equipped with an Intel(R) Xeon(R) Gold 6330 CPU central processing unit, an NVidia GTX3090 graphics processor and 128GB memory. Figures 4 to 6 Results shown.
[0138] The aircraft engine modeling is carried out for the GE90 model engine of General Motors, and the state transfer matrix and system feedback of the engine are fitted. The fitting results are as follows: Figure 4 As shown, Figure 4 (a) shows the comparison between the model prediction results and the actual results. The vertical axis is the predicted value and the horizontal axis is the actual value. The circle points are the actual data points, and the solid plus points are the predicted points. This method can accurately predict the state transition process and system feedback of the aircraft engine. Figure 4 (b) Shows the global sensitivity analysis of the system feedback model to the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor. Figure 5 (a) shows the effect of this method compared with the rule-based maintenance method, deep value reinforcement learning algorithm, and actor-critic reinforcement learning algorithm. The bottom curve is the convergence result of this method, and the dotted line above is the rule-based maintenance algorithm. Non-parametric reinforcement learning can significantly reduce the system feedback cost compared with the rule-based maintenance algorithm. The two convergence curves above are the deep value reinforcement learning algorithm and the actor-critic reinforcement learning algorithm. The convergence results of these two algorithms after 1000 trainings are still not as good as the results of non-parametric reinforcement learning after 100 trainings, and these two comparison algorithms show strong oscillations during the convergence process, indicating that the sampling efficiency of these two methods is very low and cannot be stably trained, while non-parametric reinforcement learning is relatively stable during the training process, showing a higher sampling efficiency. It can be seen from the results that non-parametric reinforcement learning has improved the data utilization efficiency by more than 100 times compared with traditional reinforcement learning methods. While integrating system uncertainty, this method improves the fit between the overall training data and the model through dynamic data updates, thereby improving the sampling efficiency of the algorithm. Figure 5 (b) shows the average box plot of the actual effect of this method after 5 applications after training compared with the rule-based maintenance method, deep value reinforcement learning algorithm, and actor-critic reinforcement learning algorithm. The box plot result of this method is on the far left. The upper limit, lower limit and mean of the system feedback cost of the non-parametric reinforcement learning method are much lower than those of the comparison method, and the actor-critic model also has decision divergence. It can be seen that this method can formulate maintenance strategies for aircraft engines relatively stably. Figure 6 The specific maintenance strategies formulated by this method for high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor are shown. The circles represent minor repairs, and the inverted triangles represent major repairs. The results show that this method can accurately mathematically model the aircraft engine and correctly formulate reasonable maintenance strategies for different components of the aircraft engine.
[0139] In this embodiment, an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning is also provided, which is used to implement the above embodiment. The terms "module", "unit" and the like used below may be a combination of software and / or hardware that can implement a predetermined function.
[0140] This embodiment provides an aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning, such as Figure 3 As shown, including:
[0141] The aircraft engine model building module is used to build a system state transfer model and a system feedback model based on the collected aircraft engine operation data and the Bayesian network and Gaussian process;
[0142] Network initialization module, used to initialize the strategy network and value network;
[0143] The maintenance action decision module is used to formulate maintenance strategies according to the system status of the current aircraft engine model, avoid the aircraft engine from being highly deteriorated, and reduce the system feedback cost r(s t );
[0144] The replay buffer module is used to store the state-action-value group of each step so that it can be extracted in batches to train the policy network and the value network;
[0145] A model training module, used to train the policy network and the value network using the state-action-value groups extracted in batches by the replay buffer module;
[0146] The dynamic update module is used to dynamically update the training set and replace the data with a higher degree of fit to the value network with data with a lower degree of fit.
[0147] It should be noted that the above embodiments are only used to illustrate the advantages and specific steps of the present invention, and do not limit the scope of application of the present invention. Other forms of simulations performed by those skilled in the art based on the present invention are within the protection scope of the present invention.
Claims
1. A method for optimizing aircraft engine maintenance strategy based on non-parametric reinforcement learning, characterized in that: The following steps are involved: Collect historical operation data of aircraft engines and establish aircraft engine models; Establish a strategy network and a value network, randomly select the initial system state s0, and formulate the initial maintenance action a0; Based on the initial system state s0 and the initial maintenance action a0, interact with the aircraft engine model to obtain a set of state action value groups: system state, maintenance action, system feedback cost and next system state; establish a replay buffer to replay each step of the state action value group (s t ,a t ,r(s t ),s t+1 ) into the replay buffer; performing random sampling in the replay buffer as a training set; updating the strategy network and the value network; Update the training set; Determine whether the system operation step has reached the set system training step. If so, stop training, obtain the trained network, and output the aircraft engine maintenance optimization strategy.
2. The method according to claim 1, characterized in that The parameter values of the system state include the stagnation pressure corresponding to each component i of the engine Stagnation temperature Stagnation enthalpy Pressure ratio π i , working fluid specific heat at constant pressure Working fluid specific heat ratio γ i , efficiency factor η i ; The maintenance actions include major repairs and minor repairs.
3. The method according to claim 1, characterized in that The aircraft engine model includes a system state transfer model and a system feedback model; The system state transfer model is fitted by a Bayesian network to the system state transfer probability obtained based on the Markov process, wherein the Bayesian network parameter L is expressed by the maximum likelihood estimation method; the expectation maximization algorithm is used to optimize the Bayesian network parameters; and the system state transfer function P(s t+1 ∣s t ); The system feedback model is based on the system state s t The running cost r1(s) is fitted by Gaussian process t );According to the system status s t The maintenance action determined by the strategy network is the sum of the prices corresponding to the maintenance actions of each component as the maintenance cost r2(s t ); the operating cost r1(s t ) and maintenance cost r2(s t ) is taken as the system state s t The system feedback cost r(s t ); Afterwards, through the system state transfer function P(s t+1 ∣s t ), get the next system status s t+1 .
4. The method according to claim 1, characterized in that The policy network is based on the system state s t , select actions, and obtain the maintenance action execution probability P through Gaussian process net (a t ): Among them, A π (s t ,a i ) is the system state s t The i-th maintenance action a predicted by the next policy network i Advantage function of Use the maximum probability estimation method to select the action and output the maintenance action a formulated by the system t : a t =argmax(P net (a t )) The value network input system state s t , based on the Gaussian process output of the system state s t Value Assessment π (s t ).
5. The method according to claim 4, characterized in that The action selection selects maintenance action exploration or maintenance strategy utilization through the ε-greedy algorithm to formulate the next maintenance action; when the ε-greedy algorithm sampling is greater than ε, the maintenance action exploration is performed, and the advantage function is obtained by randomly sampling within the Gaussian process uncertainty range through the policy network; when the ε-greedy algorithm sampling is less than ε, the maintenance strategy utilization is performed, and the Gaussian process mean output advantage function is performed through the policy network.
6. The method according to claim 4, characterized in that Bayesian data is used to fuse prior maintenance experience data with maintenance actions obtained through a strategy network to assist in decision-making guidance.
7. The method according to claim 1, characterized in that The updating strategy network and value network include the following sub-steps: based on the original sampling point system state Advantage function A π (s t ,a t ) old System status with new sampling point Advantage function A π (s t ,a t ) new , expand the Gaussian process to fit the data set: The input is: The output is: Among them, A π (s t ,a t ) is state s t Next, perform action a t The advantage function in this case is calculated as: From π (s t ,a t )=r(s t ,a t )+γV π (s t+1 )-V π (s t ) Where γ is the attenuation coefficient, r(s t ,a t ) is the system feedback cost function; Construct a Gaussian process based on the covariance matrix as the updated policy network: in, is the variance of the observation noise, I is the unit matrix, and K is the covariance matrix calculated by the Gaussian process kernel function: Correct the value V by time difference method π (s t ), and perform batch training to update the value network.
8. The method according to any one of claims 3 to 7, characterized in that: The kernel function of the Gaussian process is: Among them, x and x′ are input vectors, ||xx′|| 2 is the square of the Euclidean distance, l is the length scale parameter, which controls the smoothness of the function, is the signal variance, which controls the overall magnitude of the kernel function.
9. The method according to claim 1, characterized in that: The updating of the training set includes the following sub-steps: obtaining the degree of fit between each training data in the training set and the value network by calculating the negative log-likelihood; re-sampling the replay buffer in batches randomly, and obtaining the degree of fit between the resampling points and the value network by calculating the negative log-likelihood; and comparing, replacing the data with a relatively high degree of fit in the training set with data with a relatively low degree of fit.
10. An aircraft engine maintenance strategy optimization system based on non-parametric reinforcement learning, characterized in that: include: The aircraft engine model building module is used to build a system state transfer model and a system feedback model based on the collected aircraft engine operation data and the Bayesian network and Gaussian process; Network initialization module, used to initialize the strategy network and value network; The maintenance action decision module is used to formulate maintenance strategies according to the system status of the current aircraft engine model, avoid the aircraft engine from being highly deteriorated, and reduce the system feedback cost; Replay buffer module, used to store the state-action-value group of each step; A model training module, used to train the policy network and the value network using the state-action-value groups extracted in batches by the replay buffer module; The dynamic update module is used to dynamically update the training set and replace the data with a higher degree of fit to the value network with data with a lower degree of fit.
Citation Information
Patent Citations
Aero-engine rotating part health parameter updating method based on deep reinforcement learning
CN116843023A
Unmanned aerial vehicle autonomous exploration navigation method based on deep reinforcement learning
CN117804457A
