Aero-engine maintenance strategy optimization method based on multiple agents

Abstract

The invention discloses an aero-engine maintenance strategy optimization method based on multiple agents, and the method comprises the following steps: 1, collecting the operation state parameters and environment parameters of key life parts of an aero-engine, carrying out the preprocessing of the collected operation state parameters and environment parameters, and carrying out the data filling, and obtaining a time series data set; step 2, constructing an auto-encoder network model, and training the unsupervised hybrid auto-encoder network model by using the time sequence data set; 3, outputting a health index estimation value of the aero-engine unit through the trained auto-encoder network model, and performing health state grade division on performance degradation of the aero-engine system in the whole life cycle; and 4, constructing and training a multi-agent deep reinforcement learning model, and formulating a maintenance strategy. According to the method, the reliability of the aero-engine maintenance strategy can be improved, the waste of detection resources is reduced, and the method has wider applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft engine maintenance, and specifically to an aircraft engine maintenance strategy optimization method based on multi-agent. Background Art

[0002] Aircraft engines are the core power systems of aircraft, and their failure during flight may lead to serious catastrophic consequences. Therefore, it is necessary to monitor the key parameters of aircraft engines in real time and analyze their operating rules in depth to accurately assess the health status of aircraft engines. With the development of intelligent manufacturing technology, comprehensive data recording of complex equipment provides reliable data support for prognostic health management (PHM) technology, which aims to reduce maintenance costs and improve flight safety by optimizing the operation and management of aircraft engines.

[0003] At present, the maintenance strategy of aircraft engines is shifting from traditional time-based maintenance (TBM) to condition-based maintenance (CBM). This shift has put forward higher requirements for the accurate assessment of the degradation state of aircraft engines. Therefore, integrating the health index (HI) into the CBM strategy, optimizing the maintenance strategy and improving its reliability have become the focus of research.

[0004] HI estimation methods are mainly divided into physical model-based methods and data-driven methods. Physical model-based methods rely on a deep understanding of the device structure and degradation mechanism. Although they can provide a relatively accurate assessment, they have poor application flexibility and are highly dependent on expert knowledge.

[0005] With the rapid development of deep reinforcement learning (DRL) technology, maintenance strategy optimization methods based on DRL have gradually attracted widespread attention. However, existing DRL methods often regard inspection intervals and maintenance operations as the same step, which may waste detection resources. In addition, these methods usually simplify the degradation process of systems or components, assuming that the degradation process of components follows a Wiener process or a Markov process, and fail to fully utilize the potential of data-driven HI in the DRL environment. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-agent based aircraft engine maintenance strategy optimization method. The present invention can improve the reliability of aircraft engine maintenance strategy, reduce the waste of detection resources, and has wider applicability.

[0007] The technical solution of the present invention is a multi-agent-based aircraft engine maintenance strategy optimization method, comprising the following steps:

[0008] Step 1: Collect the operating status parameters and environmental parameters of the key life components of the aircraft engine, pre-process the collected operating status and environmental parameters, and then fill the data to obtain a time series data set;

[0009] Step 2: Build an attention-residual block enhanced autoencoder network model, and use a time series dataset to train the attention-residual block enhanced autoencoder network model;

[0010] Step 3: Output the estimated health index value of the aircraft engine unit through the trained autoencoder network model; based on the estimated health index value, classify the performance degradation of the aircraft engine system throughout its life cycle into health status levels;

[0011] Step 4: Build and train a multi-agent deep reinforcement learning model, and formulate a maintenance strategy based on the health status level and the multi-agent deep reinforcement learning model.

[0012] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, in step one, the key life components include fans, high-pressure compressors, low-pressure compressors, high-pressure turbines and low-pressure turbines; the operating status parameters include fuel flow, fan speed, core physical speed, LPC outlet temperature, HPC outlet temperature, HPT outlet temperature, LPT outlet temperature, branch pipe total pressure, fan inlet total pressure, fan outlet total pressure, LPC outlet total pressure, HPC outlet static pressure, combustion chamber outlet total pressure, LPT outlet total pressure; the environmental parameters include flight altitude, flight Mach number, throttle-transformer angle, and fan inlet temperature.

[0013] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, the preprocessing includes missing value filling, outlier processing, standardization and normalization processing;

[0014] The missing value filling process is to use the mean filling method to process the missing values ​​of the collected working status and environmental parameters, and fill the gaps by taking the average value of the data on both sides of the missing value. The formula is:

[0015]

[0016] In the above formula, x t represents the missing value at time t, x t-1 and x t+1 Respectively represent the measured values ​​before and after the missing value;

[0017] The process of processing the abnormal value is to discard the data that deviates from the normal working state parameter range of the equipment;

[0018] The standardization process adopts the Z-score standardization method, and the formula is:

[0019]

[0020] In the above formula, Z represents the standardized data, x represents the data before standardization, μ and σ represent the mean and standard deviation of the original data respectively;

[0021] The normalization process adopts max-min normalization, and the formula is:

[0022]

[0023] In the above formula, Represents the normalized data value, x i,j The i-th data point of the j-th sensor, and Respectively represent the minimum and maximum values ​​of the j-th sensor;

[0024] The data filling process is to compare the cycle lengths of various aircraft engine units, select the longest cycle as the basis of the sliding window, align the data of the remaining units, and produce a time series data set.

[0025] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, in step 2, the attention-residual block enhanced autoencoder network model consists of an encoding layer, a fully connected layer, a constraint layer and a decoding layer;

[0026] The encoding layer is used to receive multi-channel time series input, including sensor signal parameters X, operating conditions W, and a time step input T; the input time series data set is processed by a residual-attention block composed of a convolution layer and a self-attention mechanism layer, wherein the attention block generates attention weights through convolution operations and weights the input to enhance the attention of the autoencoder network model to important features, and the residual structure adds the input and the output after the convolution operation element by element;

[0027] The fully connected layer flattens the convolutional features and maps them to the potential representation space;

[0028] The constraint layer combines correlation loss and gradient loss to optimize the representation ability of the autoencoder network model by capturing the relationship between input features and time steps. The process is as follows:

[0029]

[0030] L CG =α c ·C l +α g ·G l +L MAE ;

[0031] Among them, C l is the correlation loss, G l is the gradient loss, L CG is the correlation gradient weighted loss, Z i and t i Represent the output and time step of the coding layer at time i, Z and t represent the mean of Z and t, respectively. c and α g is the weight parameter, L MAE Represents the mean square error loss value, N is the total number of samples;

[0032] The decoding layer repeats the potential representation and inputs it into the decoding stage with the environmental data. The decoding stage gradually restores the dimensions of the original input features through a series of reverse convolutions and residual blocks, and finally outputs a sequence with the same dimension as the input features.

[0033] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, the training process of the attention-residual block enhanced autoencoder network model adopts a dynamic learning rate adjustment strategy: the time series data set is input into the autoencoder network model for training, and the initial learning rate is selected and kept unchanged in the first 10 training rounds during the training process; starting from the 11th round, the learning rate decays exponentially, so that the autoencoder network model converges quickly; and in the later stage, the learning rate is gradually reduced to obtain the optimal solution, thereby completing the training of the attention-residual block enhanced autoencoder network model.

[0034] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, in step 3, the performance degradation of the aircraft engine system throughout its life cycle is graded according to the estimated health index value, and the data volume and typical degradation process of the aircraft engine are comprehensively considered when dividing the health status level; based on the degradation rate of the estimated health index value and its corresponding maintenance measures, the health status of the aircraft engine is divided into three categories:

[0035] When the health index is ≥0.8, the health state HS=0, and the degradation process of the aircraft engine system is relatively slow; when 0.8>health index≥0.4, the health state HS=1, and the aircraft engine system shows moderate degradation; when the health index is <0.4, the health state HS=2, and the performance of the aircraft engine system deteriorates sharply, approaching the critical point of failure.

[0036] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, in step 4, the multi-agent deep reinforcement learning model is constructed by using the DDQN deep learning algorithm to reduce the risk of overestimation of the Q value by using two independent agents; the agent includes a person responsible for determining the inspection interval τ tThe detection agent t observes the initial state s t And choose the inspection interval τ t ; In τ t After a flight cycle, the maintenance agent observes the updated state s' t , and perform corresponding maintenance measures.

[0037] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, the network architecture of the detection agent and the maintenance agent both adopt a two-layer fully connected neural network structure: the first layer contains 64 neurons, the second layer contains 128 neurons, and both use the ReLU activation function; the output layer of the detection agent has 10 neurons, representing possible inspection intervals; the output layer of the maintenance agent has 3 neurons, representing different maintenance actions.

[0038] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, the training process of the multi-agent deep reinforcement learning model is:

[0039] First, the inspection action penalty is introduced by the detection agent, and the formula is as follows:

[0040] I p =λ1C p (I{(a ins <σ1)and(HS=0)}+I{(a ins <σ2)and(HS=2)});

[0041] In the above formula, I p is the check action penalty, C p is the action penalty value, HS is the health status, a ins is the discrete action representation of the checking agent; σ1, σ2 represent the lower and upper bound thresholds of the checking operation respectively; I{·} represents the indicator function; λ1 is the penalty coefficient of the checking action;

[0042] Then, the action penalty is maintained by the maintenance agent, and the formula is as follows:

[0043]

[0044] In the above formula, a act is the discrete action representation of the maintenance agent; φ1 and φ2 are the lower and upper thresholds of the health indicator; λ2~λ7 are the penalty coefficients of the maintenance action.

[0045] In the aforementioned multi-agent-based aircraft engine maintenance strategy optimization method, the training process of the multi-agent deep reinforcement learning model uses a priority experience replay buffer to store and select training samples, and updates the network weights more efficiently by giving priority to experiences with larger TD errors. The priority of the experience is determined by the following formula:

[0046]

[0047] In the above formula, δ i represents the TD error of the i-th sample; ξ is a constant greater than 0; α is the control parameter of the priority degree;

[0048] In addition, the confidence interval upper bound algorithm is used to explore the balance between exploration and development in the strategy optimization training process. In each state s and action a, the reward b(s,a) is defined as:

[0049]

[0050] Among them, τ(s,a) represents the number of visits to state s, and τ(s,a) represents the number of actions a is selected in state s; actions with fewer attempts will receive higher rewards; the state selected by the exploration strategy based on the confidence interval upper bound algorithm is

[0051] After the multi-agent deep reinforcement learning model is trained, a corresponding maintenance strategy is formulated according to the health status.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention improves the reliability of maintenance strategies by introducing the health indicators and health status of aircraft engines into a multi-agent deep reinforcement learning model. The present invention achieves the estimation of aircraft engine health indicators without the need for labeled data through an unsupervised hybrid attention-residual block enhanced autoencoder network model. In addition, the present invention effectively avoids unnecessary waste of detection resources by collaboratively optimizing aircraft engine inspection intervals and maintenance strategies. The present invention efficiently utilizes historical operating data resources throughout the life cycle of aircraft engines, providing a reliable method for aircraft engine health status monitoring and maintenance strategy formulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Flowchart developed for aircraft engine health indicator estimation and maintenance strategy;

[0055] Figure 2 The structure diagram of the autoencoder network model enhanced by the attention-residual block;

[0056] Figure 3Figure 2. Health indicator estimation results of the autoencoder network model enhanced with attention-residual blocks on the test set.

[0057] Figure 4 It is a health status result diagram divided according to health indicators;

[0058] Figure 5 This is a model structure diagram of the multi-agent deep reinforcement learning model.

[0059] Figure 6 Cost comparison of the multi-agent deep reinforcement learning model strategy with the single-agent strategy and the ideal preventive maintenance strategy. DETAILED DESCRIPTION

[0060] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0061] Example

[0062] A multi-agent-based aircraft engine maintenance strategy optimization method, the flow chart is as follows Figure 1 As shown, the specific steps are as follows:

[0063] Step 1: Collect the operating status parameters and environmental parameters of the key life components of the aircraft engine, pre-process the collected operating status and environmental parameters, and then fill the data to obtain a time series data set;

[0064] In this embodiment, an aircraft engine is provided with an aircraft engine state database, a deep learning module, a data acquisition module, and a data storage module. Sensors are deployed in the data acquisition module of the aircraft engine to collect the operating state parameters and environmental parameters of the critical life components of the aircraft engine; the data storage module of the aircraft engine is based on the Sql Server database, with the sensor type as a column and the data acquisition time as a row, and the collected operating state parameters and environmental parameters are organized into a table and saved in the aircraft engine state database to ensure effective data storage and interaction; it supports two-way data transmission between users and the cloud to provide training samples for subsequent steps.

[0065] In step one, the key life components include a fan, a high-pressure compressor, a low-pressure compressor, a high-pressure turbine and a low-pressure turbine; the operating status parameters include fuel flow, fan speed, core physical speed, LPC outlet temperature, HPC outlet temperature, HPT outlet temperature, LPT outlet temperature, branch pipe total pressure, fan inlet total pressure, fan outlet total pressure, LPC outlet total pressure, HPC outlet static pressure, combustion chamber outlet total pressure, LPT outlet total pressure; the environmental parameters include flight altitude, flight Mach number, throttle-transformer angle, fan inlet temperature.

[0066] In this embodiment, the preprocessing includes missing value filling, outlier processing, standardization and normalization processing;

[0067] Specifically, the missing value filling process is to use the mean filling method to process the missing values ​​of the collected working status and environmental parameters, and fill the gaps by taking the average value of the data on both sides of the missing value. The formula is:

[0068]

[0069] In the above formula, x t represents the missing value at time t, x t-1 and x t+1 Respectively represent the measured values ​​before and after the missing value;

[0070] Specifically, the process of processing the abnormal value is to discard the data that deviates from the normal working state parameter range of the equipment;

[0071] Specifically, the standardization process adopts the Z-score standardization method, and the formula is:

[0072]

[0073] In the above formula, Z represents the standardized data, x represents the data before standardization, μ and σ represent the mean and standard deviation of the original data respectively;

[0074] Specifically, the normalization process adopts max-min normalization, and the formula is:

[0075]

[0076] In the above formula, Represents the normalized data value, x i,j The i-th data point of the j-th sensor, and Respectively represent the minimum and maximum values ​​of the j-th sensor;

[0077] Specifically, the data filling process is to compare the cycle lengths of each aircraft engine unit and select the longest cycle as the benchmark of the sliding window, where the longest cycle is 1820; finally, the data of the remaining units are aligned through data filling to produce a time series data set, and the time series data set is divided into a training set, a test set, and a validation set.

[0078] Step 2: Build an attention-residual block enhanced autoencoder network model, and use a time series dataset to train the attention-residual block enhanced autoencoder network model;

[0079] In this example, we design the operating condition input layer of the aircraft engine; construct a loss function to represent the degradation state of the aircraft engine; build and train the attention-residual block enhanced autoencoder network (EARAE) model, such as Figure 2 As shown in the figure, the attention-residual block enhanced autoencoder network model consists of an encoding layer, a fully connected layer, a constraint layer, and a decoding layer.

[0080] Specifically, the encoding layer of the autoencoder network model is used to receive multi-channel time series inputs, including sensor signal parameters X, operating conditions W, and a time step input T; then, the input time series data set is processed by a residual-attention block consisting of a convolutional layer and a self-attention mechanism layer, where the attention block generates attention weights through convolution operations and weights the input to enhance the autoencoder network model's attention to important features. The residual structure adds the input and the output after the convolution operation element by element, aiming to alleviate the gradient vanishing problem during training and promote the autoencoder network model to learn deeper features more efficiently. This residual-attention block design helps the model maintain an effective gradient flow when processing complex time series data sets and enhances its ability to capture key features.

[0081] Specifically, the fully connected layer flattens the convolutional features and maps them to the latent representation space.

[0082] Specifically, in order to constrain the characteristics of the potential representation, the autoencoder network model introduces a constraint layer, which combines the correlation loss and the gradient loss to optimize the representation ability of the autoencoder network model by capturing the relationship between the input features and the time step. The process is as follows:

[0083]

[0084] L CG =α c ·C l +α g ·G l +L MAE ;

[0085] Among them, C l is the correlation loss, G l is the gradient loss, L CG is the correlation gradient weighted loss, Z i and t i Represent the output and time step of the coding layer at time i, Z and t represent the mean of Z and t, respectively. c and α g is the weight parameter, L MAE Represents the mean square error loss value, N is the total number of samples;

[0086] Specifically, after the constraint layer, the decoding layer repeats the potential representation and inputs it into the decoding stage with the environmental data. The decoding stage gradually restores the dimensions of the original input features through a series of reverse convolutions and residual blocks, and finally outputs a sequence with the same dimension as the input features.

[0087] The input size and output size parameters of each layer of the autoencoder network model are shown in Table 1;

[0088] Table 1 Input size and output size of each layer of the autoencoder network model

[0089] Network Layer Input Shape Output shape Input layer (InpX) [None,100,14] [None,100,14] Input layer (inpT) [None,1] [None,1] Input layer (inpW) [None,100,4] [None,100,4] Attention-Residual Block 1 (Conv1D+Attention) [None,100,14] [None,100,128] Attention-Residual Block 2 (Conv1D+Attention) [None,100,128] [None,100,64] Attention-Residual Block 3 (Conv1D+Attention) [None,100,64] [None,100,16] Flatten Layer [None,100,16] [None,1600] Fully connected layer (Z) [None,1600] [None,1] Constraint Function Layer [None,1] [None,1] Repeated representation [None,1] [None,100,1] merge [None,100,1],[None,100,4] [None,100,5] Attention-Residual Block 3 (Conv1D+Attention) [None,100,5] [None,100,16] Attention-Residual Block 2 (Conv1D+Attention) [None,100,16] [None,100,64] Attention-Residual Block 1 (Conv1D+Attention) [None,100,64] [None,100,128] Output Layer [None,100,128] [None,100,14]

[0090] Then, the unsupervised hybrid autoencoder network model is trained, and a dynamic learning rate adjustment strategy is adopted in the training process: the time series dataset is input into the autoencoder network model for training, and 0.0001 is selected as the initial learning rate, which is kept unchanged in the first 10 training rounds during the training process; starting from the 11th round, the learning rate decays exponentially. The purpose of this strategy is to maintain a high learning rate in the early stage to help the autoencoder network model converge quickly; and gradually reduce the learning rate in the later stage to avoid the autoencoder network model skipping the optimal solution and ensure that the autoencoder network model is more stable and accurate; finally, the optimal solution is obtained, and the training of the attention-residual block enhanced autoencoder network model is completed.

[0091] The specific parameters for the training of the autoencoder network model are shown in Table 2;

[0092] Table 2 Hyperparameter settings for training the autoencoder network model

[0093] Hyperparameters Settings Sliding Window 1820 Training rounds 50 Batch size 32 Optimizer Adam Initial learning rate 0.0001 The learning rate remains constant for epochs 10 Learning rate reduction 0.1

[0094] Step 3: Output the estimated health index value of the aircraft engine unit through the trained autoencoder network model; based on the estimated health index value, classify the performance degradation of the aircraft engine system throughout its life cycle into health status levels;

[0095] In this embodiment, the estimated health index HI value of each aircraft engine unit in the test set is obtained through the trained autoencoder network model, and the artificially extracted real health index HI value is used as a reference (only for evaluation), and the relevant evaluation indicators are calculated to measure the difference between the estimated value and the real value of the aircraft engine life prediction; the evaluation indicators include mean absolute error MAE, mean absolute percentage error MAPE, root mean square error RMSE, monotonicity index Mon and trend index Tre, and the calculation formulas are as follows:

[0096]

[0097]

[0098] In the formula, h and Represent the real health index HI and the estimated health index HI respectively, and t u denote the estimated value and period value of the health index HI of engine unit u, respectively, and corr(·) is the Spearman correlation coefficient.

[0099] like Figure 3 As shown, the EARAE algorithm of the attention-residual block enhanced autoencoder network (EARAE algorithm is the key model method of the autoencoder network model of the present invention for estimating the health index of aircraft engines, which is responsible for realizing the estimation of the health index of aircraft engines and is the premise for promoting the optimization of the full life cycle maintenance strategy) performed well in the test unit; except for Unit 10 and Unit 13, which had a small deviation at the end of the health index HI curve, the health index HI estimation results of the other 6 units were highly consistent with the true value;

[0100] Table 3 shows the comparison results of the autoencoder network (EARAE) model with other models on the test set, where the bold data represents the best evaluation index, and the underline represents the second best evaluation index; on the DS03 dataset with a single fault mode, the proposed method is significantly better than other methods in terms of error; specifically, the EARAE algorithm of the autoencoder network model has an average reduction of 40.79% and 16.49% in the three error evaluation indicators compared with the SMSAE and Multi-IFFGRU methods, respectively; compared with the AE-NG method, it has an average reduction of 19.71%; compared with the LSTM-AE method, it has an average reduction of 40.81%; although the supervised learning model has a slight advantage in the monotonicity and trend indicators, the gap is small; in addition, considering that the autoencoder network model does not require label input, this further highlights its advantages.

[0101] Table 3 Comparison results

[0102]

[0103] In this embodiment, the performance degradation of the aircraft engine system during its entire life cycle is graded according to the estimated health index HI value, and the data volume and typical degradation process of the aircraft engine are comprehensively considered when dividing the health status level; based on the degradation rate of the estimated health index value and its corresponding maintenance measures, the health status of the aircraft engine is divided into three categories, such as Figure 4 As shown:

[0104] When the health index HI ≥ 0.8, the health state HS = 0, and the degradation process of the aircraft engine system is relatively slow; when 0.8> health index HI ≥ 0.4, the health state HS = 1, and the aircraft engine system shows moderate degradation; when the health index HI < 0.4, the health state HS = 2, and the performance of the aircraft engine system deteriorates sharply, approaching the critical point of failure.

[0105] Step 4: Build and train a multi-agent deep reinforcement learning model, and formulate a maintenance strategy based on the health status level and the multi-agent deep reinforcement learning model.

[0106] In this embodiment, in order to accurately evaluate the long-term benefits of different actions in the process of multi-agent collaboration, the DDQN deep learning algorithm is used to reduce the risk of overestimation of Q value by using two independent agents, thereby improving the stability and accuracy of strategy selection; the structure of the multi-agent deep reinforcement learning model is as follows: Figure 5 As shown;

[0107] The agent includes a person responsible for determining the inspection interval τ t The detection agent t observes the initial state s t And choose the inspection interval τ t ; In τ t After a flight cycle, the maintenance agent observes the updated state s' t , and perform corresponding maintenance measures. The detection agent (Ins-Agent) and the maintenance agent (Act-Agent) share the same reward system, which includes the costs of detection and maintenance.

[0108] Furthermore, the network architecture of both the detection agent (Ins-Agent) and the maintenance agent (Act-Agent) adopts a two-layer fully connected neural network structure: the first layer contains 64 neurons, the second layer contains 128 neurons, and both use the ReLU activation function. The structures of the two networks are the same for the detection agent (Ins-Agent) and the maintenance agent (Act-Agent). The only difference is the number of neurons in the output layer. The output layer of the detection agent (Ins-Agent) has 10 neurons, representing possible inspection intervals, while the output layer of the maintenance agent (Act-Agent) has 3 neurons, representing different maintenance actions (keep, maintain, and replace).

[0109] During the training process, the main goal of the agent is to learn how to maximize the benefits. In order to guide the agent to complete specific tasks, the goals of these tasks need to be clearly reflected in the reward function. To this end, action penalties and costs need to be introduced. The parameters related to maintenance and penalties are shown in the following table:

[0110] Table 4 Maintenance, penalty parameters and costs

[0111] Maintenance and Penalty Parameters Settings <![CDATA[Inspection cost C ins > 2 <![CDATA[Maintenance cost C repair > 35 <![CDATA[Replacement cost C replace > 165 <![CDATA[Action Penalty C p > 10

[0112] First, the inspection action penalty is introduced by the detection agent, and the formula is as follows:

[0113] I p =λ1C p (I{(a ins <σ1)and(HS=0)}+I{(a ins <σ2)and(HS=2)});

[0114] In the above formula, I p is the check action penalty, C p is the action penalty value, HS is the health status, ai ns is the discrete action representation of the checking agent; σ1, σ2 represent the lower and upper thresholds of the checking operation, which are set to 5 and 3 respectively; I{·} represents the indicator function, whose value is 0 or 1; λ1 is the penalty coefficient of the checking action, which is set to 15.

[0115] Then, the action penalty is maintained by the maintenance agent, and the formula is as follows:

[0116]

[0117] In the above formula, a act is the discrete action representation of the maintenance agent; φ1 and φ2 are the lower and upper thresholds of the health index, which are set to 0.2 and 0.8 respectively; λ2~λ7 are the penalty coefficients of the maintenance action, which are 0.2, 15, 3, 20, 10, and 16 respectively.

[0118] The training process of the multi-agent deep reinforcement learning model uses a priority experience replay buffer to store and select training samples, and updates the network weights more efficiently by giving priority to experiences with larger TD errors (TD errors represent the difference between predicted values ​​and true values ​​obtained from actual interactions). The priority of experiences is determined by the following formula:

[0119]

[0120] In the above formula, δ i represents the TD error of the i-th sample; ξ is a constant greater than 0, in order to ensure that no matter what the TD-error value is, the sampling probability is still greater than 0; α is the control parameter of the priority degree, which is set to 0.6.

[0121] In addition, the confidence interval upper bound (Confidence Interval Upper Bound (UCB) algorithm) algorithm is used to explore the balance between exploration and development during the strategy optimization training process. In each state s and action a, the reward b(s,a) is defined as:

[0122]

[0123] Among them, τ(s,a) represents the number of visits to state s, and τ(s,a) represents the number of actions a is selected in state s; actions with fewer attempts will receive higher rewards; the state selected by the exploration strategy based on the confidence interval upper bound (UCB) algorithm is

[0124] During the training process, the hyperparameter settings are shown in Table 5:

[0125] Table 5 Hyperparameter settings

[0126] Hyperparameter Name Settings Learning rate α 0.0001 Batch size 128 Discount factor γ 0.95 Target network update frequency 5 Playback buffer size 10,000 Priority replay buffer control parameters 0.6 Initial Importance Sampling 0.4 Increment per round 0.001 Training rounds 300

[0127] After the multi-agent deep reinforcement learning model training is completed, the maintenance strategy of the agent is analyzed. From the results in Table 6, it can be seen that the decision-making strategies of the detection agent and the maintenance agent will be dynamically adjusted with the changes in the health status of the aircraft engine. When the aircraft engine is in a good health state, the detection agent tends to set a longer inspection interval to reduce the inspection frequency, thereby optimizing and reducing the inspection cost.

[0128] Table 6 Maintenance strategy

[0129]

[0130] Furthermore, different maintenance strategies are compared on the test set, and the advantages and disadvantages of maintenance strategies are characterized by the size of the cost; among them, the single agent only includes maintenance actions, and the ideal preventive regular maintenance strategy is to perform maintenance every 10 rounds and replace the engine when it is about to fail or fail, so as to maximize the life of the engine. Figure 6As shown in the figure, the total costs of the proposed multi-agent maintenance strategy, single-agent maintenance strategy and ideal preventive maintenance strategy are 1883, 5547 and 2015 respectively; compared with the single-agent maintenance strategy, the proposed multi-agent strategy reduces the cost by 66.05%, and reduces the cost by 6.55% compared with the ideal preventive maintenance strategy; since maintenance decisions need to be made in each round, the cost of the single-agent maintenance strategy in all test sets is much higher than that of other strategies, revealing its problem of resource waste; compared with the ideal preventive maintenance strategy, the proposed multi-agent maintenance strategy has lower costs in 4 engine units, and only one engine has a slightly higher cost than the ideal preventive maintenance strategy. The results show that in most cases, the proposed multi-agent maintenance strategy can more effectively balance the maintenance cost and the life of aircraft engines.

[0131] In summary, the present invention can improve the reliability of aircraft engine maintenance strategies, reduce the waste of detection resources, and has wider applicability.

Claims

1. A multi-agent-based aircraft engine maintenance strategy optimization method, characterized in that: The following steps are involved: Step 1: Collect the operating status parameters and environmental parameters of the key life components of the aircraft engine, pre-process the collected operating status and environmental parameters, and then fill the data to obtain a time series data set; Step 2: Build an attention-residual block enhanced autoencoder network model, and use a time series dataset to train the attention-residual block enhanced autoencoder network model; Step 3: Output the estimated health index value of the aircraft engine unit through the trained autoencoder network model; based on the estimated health index value, classify the performance degradation of the aircraft engine system during its entire life cycle into health status levels; Step 4: Build and train a multi-agent deep reinforcement learning model, and formulate a maintenance strategy based on the health status level and the multi-agent deep reinforcement learning model.

2. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 1 is characterized in that: In step one, the key life components include a fan, a high-pressure compressor, a low-pressure compressor, a high-pressure turbine and a low-pressure turbine; the operating status parameters include fuel flow, fan speed, core physical speed, LPC outlet temperature, HPC outlet temperature, HPT outlet temperature, LPT outlet temperature, branch pipe total pressure, fan inlet total pressure, fan outlet total pressure, LPC outlet total pressure, HPC outlet static pressure, combustion chamber outlet total pressure, LPT outlet total pressure; the environmental parameters include flight altitude, flight Mach number, throttle-transformer angle, fan inlet temperature.

3. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 1 is characterized in that: The preprocessing includes missing value filling, outlier processing, standardization and normalization processing; The missing value filling process is to use the mean filling method to process the missing values ​​of the collected working status and environmental parameters, and fill the gaps by taking the average value of the data on both sides of the missing value. The formula is: In the above formula, x t represents the missing value at time t, x t-1 and x t+1 Respectively represent the measured values ​​before and after the missing value; The process of processing the abnormal value is to discard the data that deviates from the normal working state parameter range of the equipment; The standardization process adopts the Z-score standardization method, and the formula is: In the above formula, Z represents the standardized data, x represents the data before standardization, μ and σ represent the mean and standard deviation of the original data respectively; The normalization process adopts max-min normalization, and the formula is: In the above formula, Represents the normalized data value, x i,j The i-th data point of the j-th sensor, and Respectively represent the minimum and maximum values ​​of the j-th sensor; The data filling process is to compare the cycle lengths of various aircraft engine units, select the longest cycle as the basis of the sliding window, align the data of the remaining units, and produce a time series data set.

4. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 1, characterized in that: In step 2, the attention-residual block enhanced autoencoder network model consists of an encoding layer, a fully connected layer, a constraint layer and a decoding layer; The encoding layer is used to receive multi-channel time series input, including sensor signal parameters X, operating conditions W, and a time step input T; the input time series data set is processed by a residual-attention block composed of a convolution layer and a self-attention mechanism layer, wherein the attention block generates attention weights through convolution operations and weights the input to enhance the attention of the autoencoder network model to important features, and the residual structure adds the input and the output after the convolution operation element by element; The fully connected layer flattens the convolutional features and maps them to the potential representation space; The constraint layer combines correlation loss and gradient loss to optimize the representation ability of the autoencoder network model by capturing the relationship between input features and time steps. The process is as follows: L CG =a c ·C l +a g ·G l +L MAE ; Among them, C l is the correlation loss, G l is the gradient loss, L CG is the correlation gradient weighted loss, Z i and t i Represent the output and time step of the coding layer at time i, Z and t represent the mean of Z and t, respectively. c and α g is the weight parameter, L MAE Represents the mean square error loss value, N is the total number of samples; The decoding layer repeats the potential representation and inputs it into the decoding stage with the environmental data. The decoding stage gradually restores the dimensions of the original input features through a series of reverse convolutions and residual blocks, and finally outputs a sequence with the same dimension as the input features.

5. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 4 is characterized in that: The training process of the attention-residual block enhanced autoencoder network model adopts a dynamic learning rate adjustment strategy: the time series data set is input into the autoencoder network model for training, the initial learning rate is selected, and it remains unchanged in the first 10 training rounds during the training process; starting from the 11th round, the learning rate decays exponentially, so that the autoencoder network model converges quickly; and in the later stage, the learning rate is gradually reduced to obtain the optimal solution, thereby completing the training of the attention-residual block enhanced autoencoder network model.

6. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 5 is characterized in that: In step 3, the performance degradation of the aircraft engine system during its life cycle is graded based on the estimated health index value, and the data volume and typical degradation process of the aircraft engine are comprehensively considered when dividing the health status level; Based on the degradation rate of the estimated health index and its corresponding maintenance measures, the health status of aircraft engines is divided into three categories: When the health index is ≥0.8, the health state = 0, and the degradation process of the aircraft engine system is relatively slow; when 0.8>health index≥0.4, the health state = 1, and the aircraft engine system shows moderate degradation; when the health index is <0.4, the health state = 2, and the performance of the aircraft engine system deteriorates sharply and approaches the critical point of failure.

7. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 1, characterized in that: In step 4, the multi-agent deep reinforcement learning model is constructed by using the DDQN deep learning algorithm to reduce the risk of overestimation of the Q value by using two independent agents; the agent includes a person responsible for determining the inspection interval τ t The detection agent t observes the initial state s t And choose the inspection interval τ t ; In τ t After a flight cycle, the maintenance agent observes the updated state s' t , and perform corresponding maintenance measures.

8. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 7 is characterized in that: The network architectures of the detection agent and the maintenance agent both adopt a two-layer fully connected neural network structure: the first layer contains 64 neurons, the second layer contains 128 neurons, and both use the ReLU activation function; the output layer of the detection agent has 10 neurons, representing possible inspection intervals; the output layer of the maintenance agent has 3 neurons, representing different maintenance actions.

9. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 8, characterized in that: The training process of the multi-agent deep reinforcement learning model is: First, the inspection action penalty is introduced by the detection agent, and the formula is as follows: I p =λ1C p (I{(a ins <σ1)and(HS=0)}+I{(a ins <σ2)and(HS=2)}); In the above formula, I p is the check action penalty, C p is the action penalty value, HS is the health status, a ins is the discrete action representation of the checking agent; σ1, σ2 represent the lower and upper bound thresholds of the checking operation respectively; I{·} represents the indicator function; λ1 is the penalty coefficient of the checking action; Then, the action penalty is maintained by the maintenance agent, and the formula is as follows: In the above formula, a act is the discrete action representation of the maintenance agent; φ1 and φ2 are the lower and upper thresholds of the health indicator; λ2~λ7 are the penalty coefficients of the maintenance action.

10. The multi-agent-based aircraft engine maintenance strategy optimization method according to claim 9, characterized in that: The training process of the multi-agent deep reinforcement learning model uses a priority experience replay buffer to store and select training samples, and updates the network weights more efficiently by giving priority to experiences with larger TD errors. The priority of experiences is determined by the following formula: In the above formula, δ i represents the TD error of the i-th sample; ξ is a constant greater than 0; α is the control parameter of the priority degree; In addition, the confidence interval upper bound algorithm is used to explore the balance between exploration and development in the strategy optimization training process. In each state s and action a, the reward b(s,a) is defined as: Among them, τ(s,a) represents the number of visits to state s, and τ(s,a) represents the number of actions a is selected in state s; actions with fewer attempts will receive higher rewards; the state selected by the exploration strategy based on the confidence interval upper bound algorithm is After the multi-agent deep reinforcement learning model is trained, a corresponding maintenance strategy is formulated according to the health status.

Images