Aero-engine maintenance strategy optimization method based on multi-agent

Abstract

The application discloses an aero-engine maintenance strategy optimization method based on multiple agents, which comprises the following steps: step one: collecting the operation state parameters and environmental parameters of the key life components of an aero-engine, pre-processing the collected operation state and environmental parameters, then performing data filling to obtain a time series data set; step two: constructing a self-encoder network model, and training the unsupervised mixed self-encoder network model by using the time series data set; step three: outputting the health index estimation value of the aero-engine unit by the trained self-encoder network model, and dividing the health state grade of the performance degradation of the aero-engine system in the whole life cycle; and step four: constructing and training a multiple-agent deep reinforcement learning model, and formulating a maintenance strategy. The application can improve the reliability of the aero-engine maintenance strategy, reduce the waste of detection resources, and has wider applicability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of aero-engine maintenance, and specifically relates to a multi-agent-based aero-engine maintenance strategy optimization method. BACKGROUND

[0002] An aero-engine is the core power system of an aircraft, and its failure during flight can lead to serious catastrophic consequences. Therefore, it is necessary to monitor the key parameters of the aero-engine in real time and deeply analyze its operation rules to accurately assess the health status of the aero-engine. With the development of intelligent manufacturing technology, comprehensive data recording of complex equipment provides reliable data support for predictive and health management (PHM) technology, which aims to reduce maintenance costs and improve flight safety by optimizing the operation and management of aero-engines.

[0003] Currently, the maintenance strategy of aero-engines is shifting from traditional time-based maintenance (TBM) to condition-based maintenance (CBM), which puts higher requirements on the accurate assessment of the degradation state of aero-engines. Therefore, integrating health index (HI) into the CBM strategy to optimize the maintenance strategy and improve its reliability has become a research focus.

[0004] HI estimation methods mainly include physical model-based methods and data-driven methods. Physical model-based methods rely on in-depth understanding of the structure and degradation mechanism of the equipment, and although they can provide more accurate assessment, they have poor application flexibility and are highly dependent on expert knowledge.

[0005] With the rapid development of deep reinforcement learning (DRL) technology, DRL-based maintenance strategy optimization methods have gradually attracted widespread attention. However, existing DRL methods often treat inspection intervals and maintenance operations as the same step, which may waste detection resources. In addition, these methods usually simplify the degradation process of the system or component, assuming that the degradation process of the component follows a Wiener process or a Markov process, and fail to fully utilize the potential of data-driven HI in the DRL environment. SUMMARY

[0006] The purpose of the present application is to provide a multi-agent-based aero-engine maintenance strategy optimization method. The present application can improve the reliability of the aero-engine maintenance strategy, reduce the waste of detection resources, and has wider applicability.

[0007] The technical solution of the present application is a multi-agent-based aero-engine maintenance strategy optimization method, which includes the following steps:

[0008] Step one: collect the operating state parameters and environmental parameters of the key life components of the aero-engine, preprocess the collected operating state and environmental parameters, then perform data filling to obtain a time series data set;

[0009] Step two: construct an attention-residual block enhanced autoencoder network model, and train the attention-residual block enhanced autoencoder network model using the time series data set;

[0010] Step three: output the health index estimate value of the aero-engine unit through the trained autoencoder network model; based on the health index estimate value, divide the performance degradation of the aero-engine system in the whole life cycle into health state grades;

[0011] Step four: construct and train a multi-agent deep reinforcement learning model, and formulate a maintenance strategy according to the health state grades and the multi-agent deep reinforcement learning model.

[0012] In the foregoing multi-agent based aero-engine maintenance strategy optimization method, in step one, the key life components include a fan, a high-pressure compressor, a low-pressure compressor, a high-pressure turbine, and a low-pressure turbine; the operating state parameters include fuel flow, fan speed, core physical rotation speed, LPC outlet temperature, HPC outlet temperature, HPT outlet temperature, LPT outlet temperature, branch pipeline total pressure, fan inlet total pressure, fan outlet total pressure, LPC outlet total pressure, HPC outlet static pressure, combustion chamber outlet total pressure, and LPT outlet total pressure; and the environmental parameters include flight altitude, flight Mach number, throttle-transformer angle, and fan inlet temperature.

[0013] In the foregoing multi-agent based aero-engine maintenance strategy optimization method, the preprocessing includes missing value filling, outlier processing, standardization, and normalization processing;

[0014] The process of missing value filling is to use the mean filling method to process the missing values of the collected operating state and environmental parameters, and to fill the vacancy by taking the average value of the data on both sides of the missing value, the formula being:

[0015]

[0016] In the above formula, x t represents the missing value at time t, x t-1 and x t+1 represent the measurement values before and after the missing value, respectively;

[0017] The process of outlier processing is to discard the data deviating from the normal operating state parameter range of the equipment;

[0018] The process of standardization adopts the Z-score standardization method, the formula being:

[0019]

[0020] In the above formula, Z represents the standardized data, x represents the data before standardization, and μ and σ represent the mean and standard deviation of the original data, respectively;

[0021] The normalization processing adopts max-min normalization, and the formula is:

[0022]

[0023] In the above formula, represents the normalized data value, x i,j The i-th data point of the j-th sensor, and respectively represent the minimum and maximum values of the j-th sensor;

[0024] The data filling process is to compare the cycle lengths of each aero-engine unit, select the longest cycle as the reference of the sliding window, align the data of the remaining units, and make a time series data set.

[0025] In the foregoing multi-agent-based aero-engine maintenance strategy optimization method, in step two, the attention-residual block enhanced autoencoder network model is composed of an encoding layer, a fully connected layer, a constraint layer, and a decoding layer;

[0026] The encoding layer is used to receive multi-channel time series input, including sensor signal parameters X, operating conditions W, and a time step input T; the input time series data set is processed by a residual-attention block composed of a convolution layer and a self-attention mechanism layer, wherein the attention block generates attention weights through convolution operation and weights the input, enhancing the attention of the autoencoder network model to important features, and the residual structure is obtained by element-wise addition of the input and the output after convolution operation;

[0027] The fully connected layer flattens the convolution features and maps them to the latent representation space;

[0028] The constraint layer combines correlation loss and gradient loss, optimizes the representation ability of the autoencoder network model by capturing the relationship between the input features and the time step, and the process is as follows:

[0029]

[0030] L CG = α c · C l + α g · G l + L MAE ;

[0031] where C l is the correlation loss, G l is the gradient loss, L CG is the correlation gradient weighted loss, Z i and t i denote the output of the encoding layer and the time step at i, Z and t denote the mean of Z and t, respectively, and a c and a g are weight parameters, L MAE denotes the mean square error loss value, and N is the total number of samples.

[0032] The decoding layer repeats the latent representation and inputs it with the environmental data into the decoding stage, which gradually recovers the dimension of the original input features through a series of reverse convolution and residual blocks, and finally outputs a sequence with the same dimension as the input features.

[0033] In the foregoing multi-agent-based aero-engine maintenance strategy optimization method, the training process of the attention-residual block enhanced autoencoder network model adopts a dynamic learning rate adjustment strategy: inputting the time series data set into the autoencoder network model for training, selecting an initial learning rate, and keeping it unchanged for the first 10 training rounds; starting from the 11th round, the learning rate is exponentially decayed to make the autoencoder network model converge quickly; and the learning rate is gradually reduced in the later period to obtain an optimal solution, thereby completing the training of the attention-residual block enhanced autoencoder network model.

[0034] In the foregoing multi-agent-based aero-engine maintenance strategy optimization method, in step three, the performance degradation of the aero-engine system throughout its life cycle is graded according to the estimated health index, and when grading the health state, the data volume and typical degradation process of the aero-engine are considered comprehensively; based on the degradation rate of the estimated health index and the corresponding maintenance measures, the health state of the aero-engine is divided into three categories:

[0035] When the health index is greater than or equal to 0.8, the health state HS = 0, and the degradation process of the aero-engine system is relatively slow; when 0.8 > health index ≥ 0.4, the health state HS = 1, and the aero-engine system shows moderate degradation; when the health index is less than 0.4, the health state HS = 2, and the performance of the aero-engine system deteriorates sharply, approaching the failure critical point.

[0036] In the foregoing multi-agent-based aero-engine maintenance strategy optimization method, in step four, the construction process of the multi-agent deep reinforcement learning model is to use the DDQN deep learning algorithm to reduce the risk of overestimation of Q values by using two independent agents; the agents include an agent responsible for determining the inspection interval t tThe detection agent and the maintenance agent are responsible for selecting the maintenance measure, the detection agent t observes the initial state s t and selects the inspection interval τ t ; after τ t flight cycles, the maintenance agent observes the updated state s' t and performs the corresponding maintenance measure.

[0037] In the foregoing multi-agent-based aero-engine maintenance strategy optimization method, the network architecture of the detection agent and the maintenance agent both adopts a double-layer fully connected neural network structure: the first layer contains 64 neurons, and the second layer contains 128 neurons, and both use a ReLU activation function; the output layer of the detection agent has 10 neurons, representing possible inspection intervals; the output layer of the maintenance agent has 3 neurons, representing different maintenance actions.

[0038] In the foregoing multi-agent-based aero-engine maintenance strategy optimization method, the training process of the multi-agent deep reinforcement learning model is as follows:

[0039] First, the detection agent introduces a check action penalty, and the formula is as follows:

[0040] I p = λ1C p (I{(a ins < σ1) and (HS = 0)} + I{(a ins < σ2) and (HS = 2)});

[0041] In the above formula, I p is the check action penalty, C p is the action penalty value, HS is the health state, a ins is the discrete action representation of the detection agent; σ1 and σ2 represent the lower and upper threshold values of the check operation, respectively; I{·} represents the indicator function; λ1 is the penalty coefficient of the check action;

[0042] Then, the maintenance agent introduces a maintenance action penalty, and the formula is as follows:

[0043]

[0044] In the above formula, a act is the discrete action representation of the maintenance agent; φ1 and φ2 are the lower and upper threshold values of the health index; λ2-λ7 are the penalty coefficients of the maintenance action.

[0045] In the foregoing multi-agent-based aero-engine maintenance strategy optimization method, the training process of the multi-agent deep reinforcement learning model uses a priority experience replay buffer to store and select training samples, and the network weights are updated more effectively by preferentially selecting experiences with larger TD errors, and the priority of the experience is determined by the following formula:

[0046]

[0047] In the above formula, delta i represents the TD error of the i-th sample; xi is a constant greater than 0; and alpha is a control parameter of the priority degree;

[0048] In addition, the upper confidence bound algorithm exploration strategy is used to achieve a balance between exploration and development in the strategy optimization training process, and in each state s and action a, the reward b(s, a) is defined as:

[0049]

[0050] Where tau(s, a) represents the number of visits to state s, and tau(s, a) represents the number of times action a is selected in state s; actions with fewer attempts will receive higher rewards; the state selected by the upper confidence bound algorithm exploration strategy is

[0051] After the training of the multi-agent deep reinforcement learning model is completed, a corresponding maintenance strategy is formulated according to the health state.

[0052] Compared with the prior art, the present application has the following beneficial effects:

[0053] The present application improves the reliability of the maintenance strategy by introducing the health indicators and health states of the aero-engine into the multi-agent deep reinforcement learning model. The present application realizes the estimation of the aero-engine health indicators without label data through the unsupervised hybrid attention-residual block enhanced autoencoder network model. In addition, the present application optimizes the aero-engine detection interval and maintenance strategy in coordination, effectively avoiding unnecessary waste of detection resources. The present application efficiently utilizes the historical operation data resources of the aero-engine throughout its life cycle, providing a reliable method for aero-engine health state monitoring and maintenance strategy formulation. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 Flowchart for aero-engine health indicator estimation and maintenance strategy formulation;

[0055] Figure 2 Structure diagram of the attention-residual block enhanced autoencoder network model;

[0056] Figure 3A health index estimation result graph of an attention-residual block enhanced autoencoder network model on a test set;

[0057] Figure 4 A health state result graph divided according to a health index;

[0058] Figure 5 A model structure diagram of a multi-agent deep reinforcement learning model.

[0059] Figure 6 A cost comparison result diagram of a multi-agent deep reinforcement learning model strategy and a single-agent strategy and an ideal preventive maintenance strategy. DETAILED DESCRIPTION

[0060] The application will be further described below in conjunction with the accompanying drawings and examples, but not as a basis for limiting the application.

[0061] EMBODIMENT

[0062] A multi-agent-based aero-engine maintenance strategy optimization method, a flowchart as shown in Figure 1 The specific steps are as follows:

[0063] Step one: collect the operating state parameters and environmental parameters of the key life components of the aero-engine, pretreat the collected operating state and environmental parameters, then perform data filling to obtain a time series data set;

[0064] In this embodiment, an aero-engine state database, a deep learning module, a data acquisition module and a data storage module are arranged in the aero-engine. Sensors are arranged in the data acquisition module of the aero-engine for collecting the operating state parameters and environmental parameters of the key life components of the aero-engine; the data storage module of the aero-engine is based on a Sql Server database, takes the sensor type as a column and the data acquisition time as a row, arranges the collected operating state parameters and environmental parameters into a table and saves them into the aero-engine state database, to ensure the effective storage and interaction of the data; supports the bidirectional data transmission of the user and the cloud, to provide training samples for the subsequent steps.

[0065] In step one, the key life components include a fan, a high-pressure compressor, a low-pressure compressor, a high-pressure turbine and a low-pressure turbine; the operating state parameters include fuel flow, fan speed, core physical rotation speed, LPC outlet temperature, HPC outlet temperature, HPT outlet temperature, LPT outlet temperature, branch pipeline total pressure, fan inlet total pressure, fan outlet total pressure, LPC outlet total pressure, HPC outlet static pressure, combustion chamber outlet total pressure and LPT outlet total pressure; the environmental parameters include flight altitude, flight Mach number, throttle-transformer angle and fan inlet temperature.

[0066] In this embodiment, the preprocessing includes missing value filling, outlier processing, standardization and normalization processing;

[0067] Specifically, the process of missing value filling is to use the mean filling method to process the missing values of the collected working state and environmental parameters. The missing values are filled by taking the average of the data on both sides of the missing values, and the formula is:

[0068]

[0069] In the above formula, x t represents the missing value at time t, x t-1 and x t+1 represent the measurement values before and after the missing value, respectively.

[0070] Specifically, the process of outlier processing is to discard the data deviating from the parameter range of the normal working state of the device.

[0071] Specifically, the process of standardization adopts the Z-score standardization method, and the formula is:

[0072]

[0073] In the above formula, Z represents the standardized data, x represents the data before standardization, and μ and σ represent the mean and standard deviation of the original data, respectively.

[0074] Specifically, the normalization processing adopts max-min normalization, and the formula is:

[0075]

[0076] In the above formula, represents the normalized data value, x i,j is the i-th data point of the j-th sensor, and represent the minimum and maximum values of the j-th sensor, respectively.

[0077] Specifically, the process of data filling is to compare the cycle lengths of each aero-engine unit, select the longest cycle as the reference of the sliding window, and the longest cycle is 1820. Finally, the data of the remaining units are aligned through data filling, and the time series data set is made. The time series data set is divided into training set, test set and validation set.

[0078] Step two: build an attention-residual block enhanced autoencoder network model, and use the time series data set to train the attention-residual block enhanced autoencoder network model;

[0079] In this example, the operating condition input layer of the aero-engine is designed; the loss function for representing the degradation state of the aero-engine is constructed; the attention-residual block enhanced autoencoder network (EARAE) model is built and trained, as shown in Figure 2 The attention-residual block enhanced autoencoder network model is composed of an encoding layer, a fully connected layer, a constraint layer and a decoding layer.

[0080] Specifically, the encoding layer of the autoencoder network model is used to receive multi-channel time series input, including sensor signal parameters X, operating conditions W and a time step input T; then, the input time series data set is processed by the residual-attention block composed of a convolution layer and a self-attention mechanism layer, wherein the attention block generates attention weights through convolution operation and weights the input, enhances the attention of the autoencoder network model to important features, and the residual structure is to add the input and the output after convolution operation element by element, aiming to alleviate the gradient vanishing problem in the training process, and promote the autoencoder network model to learn deeper features more efficiently. The design of such residual-attention block helps the model to maintain effective gradient flow when processing complex time series data sets, and enhances the ability to capture key features.

[0081] Specifically, the fully connected layer flattens the convolutional features and maps them to the latent representation space.

[0082] Specifically, in order to constrain the characteristics of the latent representation, the autoencoder network model introduces a constraint layer, which combines correlation loss and gradient loss, optimizes the representation ability of the autoencoder network model by capturing the relationship between input features and time steps, as follows:

[0083]

[0084] L CG =α c ·C l +α g ·G l +L MAE ;

[0085] Where C l is the correlation loss, G l is the gradient loss, L CG is the correlation gradient weighted loss, Z i and t i represent the output of the encoding layer at i time and the time step, Z and t represent the mean of Z and t, α c and α g are weight parameters, L MAE represents the mean square error loss value, and N is the total number of samples.

[0086] Specifically, after the constraint layer, the decoding layer repeats the latent representation and inputs the environmental data into the decoding stage, which gradually recovers the dimension of the original input features through a series of reverse convolution and residual blocks, and finally outputs a sequence with the same dimension as the input features.

[0087] The input size and output size parameters of each layer of the autoencoder network model are shown in Table 1.

[0088] Table 1 Input size and output size of each layer of the autoencoder network model

[0089] Network layer Input shape Output shape Input layer (InpX) [None, 100, 14] [None, 100, 14] Input layer (inpT) [None, 1] [None, 1] Input layer (inpW) [None, 100, 4] [None, 100, 4] Attention-residual block 1 (Conv1D + Attention) [None, 100, 14] [None, 100, 128] Attention-residual block 2 (Conv1D + Attention) [None, 100, 128] [None, 100, 64] Attention-residual block 3 (Conv1D + Attention) [None, 100, 64] [None, 100, 16] Flatten layer [None, 100, 16] [None, 1600] Fully connected layer (Z) [None, 1600] [None, 1] Constraint function layer [None, 1] [None, 1] Replicated representation [None, 1] [None, 100, 1] Merge [None, 100, 1], [None, 100, 4] [None, 100, 5] Attention-residual block 3 (Conv1D + Attention) [None, 100, 5] [None, 100, 16] Attention-residual block 2 (Conv1D + Attention) [None, 100, 16] [None, 100, 64] Attention-residual block 1 (Conv1D + Attention) [None, 100, 64] [None, 100, 128] Output layer [None, 100, 128] [None, 100, 14]

[0090] Then, the unsupervised hybrid autoencoder network model is trained, and a dynamic learning rate adjustment strategy is adopted in the training process: the time series data set is input into the autoencoder network model for training, and 0.0001 is selected as the initial learning rate, which remains unchanged in the first 10 training rounds; from the 11th round, the learning rate is exponentially decayed, the purpose of this strategy is to maintain a high learning rate in the early stage to help the autoencoder network model converge quickly; and gradually reduce the learning rate in the later stage to avoid the autoencoder network model skipping the optimal solution and ensure the autoencoder network model more stable and accurate; finally, the optimal solution is obtained, and the attention-residual block enhanced autoencoder network model training is completed.

[0091] The specific parameters of the autoencoder network model training are shown in Table 2.

[0092] Table 2 Hyperparameter settings for training the autoencoder network model

[0093] Hyperparameters Set values Sliding window 1820 Training epochs 50 Batch size 32 Optimizer Adam Initial learning rate 0.0001 Learning rate freeze epochs 10 Learning rate decrease 0.1

[0094] Step three: output the health index estimate value of the aero-engine unit through the trained autoencoder network model; based on the health index estimate value, the performance degradation of the aero-engine system in the whole life cycle is divided into health state grades;

[0095] In this embodiment, the health index HI estimate value of each aero-engine unit in the test set is obtained through the trained autoencoder network model, and the artificially extracted true health index HI value is used as a reference (only for evaluation), and the relevant evaluation indexes are calculated to measure the difference between the estimated value and the true value of the aero-engine life prediction; the evaluation indexes include mean absolute error MAE, mean absolute percentage error MAPE, root mean square error RMSE, monotonicity index Mon and trend index Tre, and the calculation formulas are as follows:

[0096]

[0097]

[0098] where h and denote the real health index HI and the estimated health index HI, respectively, and t u denote the health index HI estimate and the period value of the engine unit u, respectively, and corr(·) is the Spearman correlation coefficient.

[0099] As Figure 3 shown, the EARAE algorithm of the attention-residual block enhanced autoencoder network (EARAE) model (the EARAE algorithm is the key model method of the autoencoder network model of the present application for health index estimation of an aero-engine, responsible for realizing the estimation of the health index of the aero-engine, and is the premise of promoting the optimization of the life cycle maintenance strategy) performs well in the test units; except that Unit 10 and Unit 13 have a small deviation at the end of the health index HI curve, the health index HI estimation results of the other 6 units are highly consistent with the true values;

[0100] Table 3 shows the comparison results of the autoencoder network (EARAE) model and other models on the test set, where the bold data represents the best evaluation index, and the underlined data represents the second best evaluation index; on the DS03 data set of a single failure mode, the proposed method is significantly better than other methods in terms of error; specifically, the EARAE algorithm of the autoencoder network model reduces the three error evaluation indexes by an average of 40.79% and 16.49% compared with the SMSAE and Multi-IFFGRU methods, respectively; compared with the AE-NG method, it is reduced by an average of 19.71%; compared with the LSTM-AE method, it is reduced by an average of 40.81%; although the supervised learning model has a slight advantage in monotonicity and trend indicators, the gap is small; in addition, considering that the autoencoder network model does not require label input, this further highlights its advantages.

[0101] Table 3 comparison results

[0102]

[0103] In this embodiment, according to the obtained health index HI estimate, the performance degradation of the aero-engine system throughout its life cycle is classified, and when classifying the health state level, the data volume and typical degradation process of the aero-engine are comprehensively considered; based on the degradation rate of the health index estimate and the corresponding maintenance measures, the health state of the aero-engine is divided into three categories, as shown in Figure 4

[0104] ​When the health index HI≥0.8, the health state HS=0, the degradation process of the aero-engine system is relatively slow; when 0.8>health index HI≥0.4, the health state HS=1, the aero-engine system shows moderate speed degradation; when the health index HI<0.4, the health state HS=2, the performance of the aero-engine system deteriorates sharply, close to the failure critical point.

[0105] Step four: constructing and training the multi-agent deep reinforcement learning model, and formulating the maintenance strategy according to the health state grade and the multi-agent deep reinforcement learning model.

[0106] In this embodiment, in order to accurately evaluate the long-term benefits of different actions in the cooperation process of multi-agent, the DDQN deep learning algorithm is used, and two independent agents are used to reduce the risk of Q value overestimation, thereby improving the stability and accuracy of policy selection; the structure of the multi-agent deep reinforcement learning model is as shown in Figure 5

[0107] The agent includes a detection agent responsible for determining the inspection interval τ t and a maintenance agent responsible for selecting maintenance measures, the detection agent t observes the initial state s t and selects the inspection interval τ t ; after τ t flight cycles, the maintenance agent observes the updated state s' t , and performs the corresponding maintenance measures. The detection agent (Ins-Agent) and the maintenance agent (Act-Agent) share the same reward system, which includes the cost of detection and maintenance.

[0108] Further, the network architecture of the detection agent (Ins-Agent) and the maintenance agent (Act-Agent) both adopts a double-layer fully connected neural network structure: the first layer contains 64 neurons, and the second layer contains 128 neurons, and both use ReLU activation function. The structure of the detection agent (Ins-Agent) and the maintenance agent (Act-Agent) is the same, and the only difference is the number of neurons in the output layer, the output layer of the detection agent (Ins-Agent) has 10 neurons, representing the possible inspection intervals, and the output layer of the maintenance agent (Act-Agent) has 3 neurons, representing different maintenance actions (keep, maintain and replace).

[0109] During the training process, the main goal of the agent is to learn how to maximize the benefits; in order to guide the agent to complete a specific task, it is necessary to explicitly reflect the goal of these tasks in the reward function; for this purpose, action penalty and cost need to be introduced; among them, the parameters related to maintenance and penalty are as shown in the following table: ​

[0110] Table 4 Maintenance, penalty parameters and fees

[0111] Maintenance, penalty parameters Set values Check cost C ins ]] 2 Maintenance cost C repair ]]> 35 Replacement cost C replace ]] 165 Action penalty C p ]] 10

[0112] Firstly, the checking action penalty is introduced by detecting the agent, and the formula is as follows:

[0113] I p = λ1C p (I{(a ins < σ1) and (HS = 0)} + I{(a ins < σ2) and (HS = 2)}};

[0114] In the above formula, I p is the checking action penalty, C p is the action penalty value, HS is the health state, ai ns is the discrete action representation of the checking agent; σ1 and σ2 represent the lower and upper threshold values of the checking operation, respectively, and are set to 5 and 3; I{·} represents the indicator function, which has a value of 0 or 1; λ1 is the penalty coefficient of the checking action, and is set to 15.

[0115] Then, the maintenance action penalty is introduced by the maintenance agent, and the formula is as follows:

[0116]

[0117] In the above formula, a act is the discrete action representation of the maintenance agent; φ1 and φ2 are the lower and upper threshold values of the health index, respectively, and are set to 0.2 and 0.8; λ2-λ7 are the penalty coefficients of the maintenance action, and are 0.2, 15, 3, 20, 10, and 16, respectively.

[0118] The training process of the multi-agent deep reinforcement learning model uses a priority experience replay buffer to store and select training samples, and more effectively updates the network weights by preferentially selecting experiences with larger TD errors (TD errors represent the difference between predicted values and actual values obtained through interaction), and the priority of the experience is determined by the following formula:

[0119]

[0120] In the above formula, δ i represents the TD error of the i-th sample; ξ is a constant greater than 0, in order to ensure that the sampling probability is still greater than 0 regardless of the value of the TD-error; α is a control parameter of the priority degree, and is set to 0.6.

[0121] In addition, the upper confidence bound (Upper Confidence Bound, UCB) algorithm is used to explore the strategy optimization in the training process to balance the exploration and development. In each state s and action a, the reward b(s, a) is defined as:

[0122]

[0123] Where τ(s, a) represents the number of visits to state s, and τ(s, a) represents the number of times action a is selected in state s. Actions with fewer attempts will receive higher rewards. The state selected by the Upper Confidence Bound (UCB) algorithm-based exploration strategy is

[0124] During the training process, the setting of hyperparameters is shown in Table 5:

[0125] Table 5 Hyperparameter Settings

[0126] Hyperparameter name Set values Learning rate a 0.0001 Batch size 128 Discount factor g 0.95 Target network update frequency 5 Replay buffer size 10,000 Prioritized replay buffer control parameters 0.6 Initial importance sampling 0.4 Increment per epoch 0.001 Training epochs 300

[0127] After the training of the multi-agent deep reinforcement learning model, the maintenance strategy of the agent is analyzed. From the results in Table 6, it can be seen that the decision-making strategy of the detection agent and the maintenance agent will be dynamically adjusted with the change of the health state of the aero-engine. When the aero-engine is in a good health state, the detection agent tends to set a longer inspection interval to reduce the inspection frequency, thereby optimizing and reducing the inspection cost.

[0128] Table 6 Maintenance Strategy

[0129]

[0130] Further, different maintenance strategies are compared on the test set, and the cost is used to represent the pros and cons of the maintenance strategy. Among them, the single-agent only includes maintenance actions, and the ideal preventive periodic maintenance strategy is to maintain every 10 rounds, and to complete replacement when the engine is about to fail or fail, thereby realizing the maximum life of the engine. The cost comparison of different strategies is as follows Figure 6As shown, the total costs of the proposed multi-agent maintenance strategy, single-agent maintenance strategy and ideal preventive periodic maintenance strategy are 1883, 5547 and 2015 respectively; compared with the single-agent maintenance strategy, the proposed multi-agent strategy reduces the cost by 66.05%, and the cost is reduced by 6.55% compared with the ideal preventive periodic maintenance strategy; since maintenance decision needs to be made in each round, the single-agent maintenance strategy produces a cost much higher than other strategies in all test sets, revealing the problem of resource waste; compared with the ideal periodic preventive maintenance strategy, the proposed multi-agent maintenance strategy has a lower cost in 4 engine units, and only one engine has a cost slightly higher than the ideal periodic preventive maintenance strategy. The results show that in most cases, the proposed multi-agent maintenance strategy can more effectively balance the maintenance cost and the life of the aero-engine.

[0131] In conclusion, the present application can improve the reliability of the aero-engine maintenance strategy, reduce the waste of detection resources, and has wider applicability.

Claims

1. A method for optimizing aero-engine maintenance strategies based on multi-agent systems, characterized in that, Includes the following steps: Step 1: Collect the operating status parameters and environmental parameters of key life-cycle components of the aero-engine, preprocess the collected operating status and environmental parameters, and then fill in the data to obtain a time series dataset; Step 2: Construct an attention-residual block augmented autoencoder network model and train it using a time series dataset; Step 3: Output the health index estimate of the aero-engine unit through the trained autoencoder network model; Based on the health index estimate, classify the health status level of the aero-engine system's performance degradation throughout its entire life cycle; Step 4: Build and train a multi-agent deep reinforcement learning model, and formulate a maintenance strategy based on the health status level and the multi-agent deep reinforcement learning model; In step two, the attention-residual block enhanced autoencoder network model consists of an encoding layer, a fully connected layer, a constraint layer, and a decoding layer; The coding layer is used to receive multi-channel time-series inputs, including sensor signal parameters. Operating conditions And a time step input The input time series dataset is processed through a residual-attention block consisting of convolutional layers and self-attention mechanism layers. The attention block generates attention weights through convolutional operations and weights the input to enhance the autoencoder network model's attention to important features. The residual structure adds the input to the output after the convolutional operation element by element. The fully connected layer flattens the convolutional features and maps them to the latent representation space; The constraint layer combines correlation loss and gradient loss to optimize the representational power of the autoencoder network model by capturing the relationship between input features and time steps. The process is as follows: ； ； ； in, It is correlation loss. It is gradient loss. It is a correlation gradient-weighted loss. and They represent The output of the coding layer at time step and the time step size. and They represent and the mean of t, and For weight parameters, This represents the mean squared error loss value. It is the total number of samples; The decoding layer repeats the latent representation and inputs it along with the environmental data into the decoding stage. The decoding stage gradually recovers the dimensionality of the original input features through a series of inverse convolutions and residual blocks, and finally outputs a sequence with the same dimensionality as the input features. The training process of the attention-residual block-enhanced autoencoder network model adopts a dynamic learning rate adjustment strategy: the time series dataset is input into the autoencoder network model for training, an initial learning rate is selected and kept constant for the first 10 training rounds; from the 11th round onwards, the learning rate decays exponentially, enabling the autoencoder network model to converge quickly; and in the later stages, the learning rate is gradually reduced to obtain the optimal solution, thus completing the training of the attention-residual block-enhanced autoencoder network model.

2. The method for optimizing aero-engine maintenance strategies based on multi-agent systems according to claim 1, characterized in that: In step one, the key life-cycle components include the fan, high-pressure compressor, low-pressure compressor, high-pressure turbine, and low-pressure turbine; the operating status parameters include fuel flow rate, fan speed, core physical speed, LPC outlet temperature, HPC outlet temperature, HPT outlet temperature, LPT outlet temperature, branch pipeline total pressure, fan inlet total pressure, fan outlet total pressure, LPC outlet total pressure, HPC outlet static pressure, combustion chamber outlet total pressure, and LPT outlet total pressure; the environmental parameters include flight altitude, flight Mach number, throttle-transformer angle, and fan inlet temperature.

3. The method for optimizing aero-engine maintenance strategies based on multi-agent systems according to claim 1, characterized in that: The preprocessing includes missing value imputation, outlier handling, standardization, and normalization. The missing value imputation process uses the mean imputation method to handle missing values ​​of the collected working status and environmental parameters. The missing value is filled by averaging the data on both sides of it. The formula is as follows: ； In the above formula, x t x represents the missing value at time t. t-1 and x t+1 These represent the measurements before and after the missing value, respectively. The process of handling outliers involves discarding data that deviates from the normal operating parameters of the equipment. The standardization process employs the Z-score standardization method, with the following formula: ； In the above formula, The data is represented by x after standardization, and x represents the data before standardization. and These represent the mean and standard deviation of the original data, respectively. The normalization process uses max-min normalization, and the formula is: ； In the above formula, x represents the normalized data value. i,j No. The first sensor Data points, and Let represent the minimum and maximum values ​​of the j-th sensor, respectively; The data filling process involves comparing the cycle lengths of each aero-engine unit, selecting the longest cycle as the reference for the sliding window, aligning the data of the remaining units, and creating a time-series dataset.

4. The multi-agent-based aero-engine maintenance strategy optimization method according to claim 1, characterized in that: In step three, based on the obtained health index estimate, the performance degradation of the aero-engine system throughout its entire life cycle is classified, and when classifying the health status level, the amount of aero-engine data and typical degradation processes are comprehensively considered. Based on the degradation rate of health index estimates and the corresponding maintenance measures, the health status of aero engines is divided into three categories: When the health index is ≥0.8, the health status is 0, and the degradation process of the aero-engine system is relatively slow; when 0.8 > health index ≥0.4, the health status is 1, and the aero-engine system exhibits moderate degradation; when the health index <0.4, the health status is 2, and the performance of the aero-engine system deteriorates sharply, approaching the failure threshold.

5. The method for optimizing aero-engine maintenance strategies based on multi-agent systems according to claim 1, characterized in that: In step four, the construction process of the multi-agent deep reinforcement learning model utilizes the DDQN deep learning algorithm, employing two independent agents to reduce... The risk of overestimation; the agent includes one responsible for determining the inspection interval. The detection agent and the maintenance agent responsible for selecting repair measures; the detection agent t observes the initial state s. t And select the inspection interval. ;exist After one flight cycle, the maintenance agent observes the updated state s' t And implement the corresponding maintenance measures.

6. The method for optimizing aero-engine maintenance strategies based on multi-agent systems according to claim 5, characterized in that: The network architecture of both the detection agent and the maintenance agent adopts a two-layer fully connected neural network structure: the first layer contains 64 neurons and the second layer contains 128 neurons, both using the ReLU activation function; the output layer of the detection agent has 10 neurons, representing possible inspection intervals; the output layer of the maintenance agent has 3 neurons, representing different maintenance actions.

7. The multi-agent-based aero-engine maintenance strategy optimization method according to claim 6, characterized in that: The training process of the multi-agent deep reinforcement learning model is as follows: First, a penalty for the checking action is introduced by detecting the intelligent agent, as shown in the following formula: ； In the above formula, It is a check for action punishment. It is the action penalty value. For a healthy state, a ins To examine the discrete action representations of an agent; , These represent the lower and upper bound thresholds of the inspection operation, respectively. This represents the indicator function; λ1 is the penalty coefficient for the check action; Then, the maintenance action penalty of the maintenance agent is calculated using the following formula: ； In the above formula, a act To maintain the discrete action representation of the agent; and These are the lower and upper bound thresholds for health indicators; To maintain the penalty coefficient for actions.

8. The method for optimizing aero-engine maintenance strategies based on multi-agent systems according to claim 7, characterized in that: The training process of the multi-agent deep reinforcement learning model uses a priority experience replay buffer to store and select training samples. By prioritizing experiences with larger TD errors, the network weights are updated more effectively. The priority of the experiences is determined by the following formula: ； In the above formula, δ i Indicates the first The TD error of each sample; ξ is a constant greater than 0; α is a control parameter for priority level; Furthermore, the confidence interval upper bound algorithm is used to optimize the training process and achieve a balance between exploration and exploitation, in each state. and actions In the middle, define rewards for: ； in, This indicates the number of times state s has been accessed. This represents the number of times action a is chosen in state s; actions with fewer attempts will receive higher rewards; the state chosen by the exploration strategy based on the upper bound of the confidence interval algorithm is... ; After the multi-agent deep reinforcement learning model is trained, a corresponding maintenance strategy is formulated based on its health status.

Images