Remaining completion time prediction method based on adaptive attention mechanism

Abstract

The invention discloses a residual completion time prediction method based on a self-adaptive attention mechanism. The method comprises the following steps: S1, constructing a residual completion time prediction method comprising a self-adaptive optimization feature self-selection module and a self-adaptive weight distribution attention mechanism time sequence prediction module; s2, performing data processing such as missing value filling and normalization on the data; s3, constructing a feature selection method based on adaptive feature optimization in an adaptive optimization feature self-selection module, performing iterative training on the preprocessed data, continuously eliminating negative features, and selecting a key feature subset; s4, carrying out feature coding on the selected key features; s5, calculating a query weight matrix, a key weight matrix and a value weight matrix in a self-attention mechanism through linear transformation according to the importance of the features; and S6, substituting the key feature subset data into the self-attention mechanism subjected to weight allocation in a self-adaptive weight allocation attention mechanism time sequence prediction module to perform model training.

Description

Technical Field

[0001] The present invention relates to the field of remaining completion time prediction and deep learning technology, and in particular to a remaining completion time prediction method based on an adaptive attention mechanism. Background Art

[0002] Under the background of Industry 4.0, the manufacturing process of the manufacturing industry is becoming increasingly complex, the types of product demand are increasing, and the replacement is getting faster and faster. The richer product variety has driven many companies to switch from traditional inventory production to a production-to-order model. In the production-to-order model, production tasks and production processes will change frequently according to factors such as order demand, raw material supply, or production line failures, which also puts forward higher requirements for dynamically adjusting production plans and continuously optimizing production processes. Therefore, accurate prediction of the remaining completion time of complex manufacturing processes has become a research hotspot. In recent years, deep learning has shown good performance in the field of remaining completion time prediction.

[0003] With the rapid development of industrial digitalization and intelligence, some scholars have applied deep learning to the field of remaining completion time prediction. The article [Yu Jiahui, Li Tieke, Wang Bolin, et al. CNN-SVR-based prediction method for the completion cycle of job shop orders [J]. Mechanical Design, 2023, 40(02): 57-64.] Aiming at the problem of predicting the remaining completion time of job shop orders under multiple products, multiple machines and multiple processes, a combined prediction method (CNN-SVR) based on convolutional neural network (CNN) and support vector regression (SVR) is proposed. This method has good performance in terms of goodness of fit and prediction error, and realizes accurate prediction of the remaining completion time of job shop orders under multiple products, multiple machines and multiple processes. Article [Li Beilei. Research on the prediction method of the remaining completion time of orders for automobile assembly workshops [D]. Hefei University of Technology, 2022.] Aiming at the problem of predicting the remaining completion time of orders in automobile assembly workshops, a method for predicting the remaining completion time of orders based on bidirectional long short-term memory network (Bilateral LSTM) and multi-constant error unit (MCEC) is proposed. This method can effectively complete the accurate prediction of the remaining completion time of automobile assembly workshops, and provides an important reference for workshop scheduling and optimization. In order to further improve the accuracy of the remaining completion time prediction, some scholars have turned their attention to feature selection.

[0004] In terms of feature selection, the article [Zhu Haihua, Wang Jianjie, Li Fei, et al. Prediction of remaining completion time of workshop orders based on feature selection and SSA-LSTM [J / OL]. Journal of Mechanical Engineering, 1-15.] proposed a feature selection method based on Pearson correlation coefficient and L1-L2 regularization to select key data from massive manufacturing data, and optimize hyperparameters through sparrow search algorithm (SSA) to improve the accuracy and efficiency of workshop order remaining completion time (ORCT) prediction; the article [Fang Weiguang, Guo Yu, Huang Shaohua, et al. Research on intelligent control methods of production processes in discrete manufacturing workshops driven by big data [J]. Journal of Mechanical Engineering, 2021, 57(20): 277-291.] proposed a filtering feature selection algorithm based on self-organizing map neural network (SOM) and feature weight fuzzy C-means clustering (FWFCM), which successfully screened out features that are highly correlated with the remaining completion time of orders from a large number of candidate features, providing high-quality input data for subsequent prediction models and improving prediction accuracy. Although the above method selects a feature subset associated with the remaining completion time to a certain extent, in a highly dynamic production environment, the model may need to be updated frequently, which will lead to a decrease in the real-time prediction performance of the model. Summary of the invention

[0005] The purpose of the present invention is to provide a method for predicting the remaining completion time based on an adaptive attention mechanism, which uses an adaptive optimization feature selection method to continuously eliminate negative features with the lowest feature importance and extract high-quality feature subsets from a high-dimensional manufacturing feature data set. At the same time, a time series prediction method based on an adaptive weight allocation attention mechanism is used to assign different weights to key features of different product batches, thereby achieving personalized and accurate prediction of the remaining completion time under complex working conditions.

[0006] The technical solution adopted by the present invention is:

[0007] A method for predicting remaining completion time based on an adaptive attention mechanism comprises the following steps:

[0008] S1: Construct a method framework including an adaptive optimization feature self-selection module and an adaptive weight allocation attention mechanism time series prediction module;

[0009] S2: For missing values ​​of text features in the dataset, fill them with "None", and for missing values ​​of numerical features in the dataset, fill them with the mode of the same numerical feature; after the missing values ​​are filled, the numerical features are normalized;

[0010] S3: A feature selection method based on adaptive feature optimization is constructed in the adaptive optimization feature self-selection module. First, the XGBoost model is used to train the data set to obtain the feature importance of all features in the data set. Secondly, the non-active features with the lowest importance are removed iteratively. After each removal, K-fold cross validation is performed to calculate the score of the current data set. Finally, when the score of the K-fold cross validation is stable, the remaining features are the key feature subsets.

[0011] S4: Encode key features, perform Word2Vec encoding on "activity" features, and for other categorical features, use random initialization vector encoding if there are many feature value categories, or use index-based encoding if there are few feature value categories;

[0012] S5: Normalize the feature importance scores of the key features, and then perform linear transformation to obtain the self-attention mechanism Q, K, and V matrices. During the linear transformation process, set an attenuation factor to control the size of the weight matrix element value;

[0013] S6: The adaptive weight allocation attention mechanism timing prediction module consists of an input layer, an LSTM layer, a self-attention layer, and an output layer. Substitute the key feature data into the adaptive weight allocation attention mechanism timing prediction model for prediction to obtain the remaining completion time.

[0014] Furthermore, in step S3, a feature selection method based on adaptive feature optimization is constructed in the adaptive optimization feature self-selection module to screen out a key feature subset, including the following steps:

[0015] 1-1) Use the XGBoost model to train the data set. When processing the XGBoost objective function, add a regularization term to reduce the variance of the model and prevent the model from overfitting. The objective function of XGBoost is Obj (t) :

[0016]

[0017] In the formula, is the loss function of the model; y i is the true value of the i-th data, is the predicted value of the i-th data, f t is the regularization term of the complexity of the t-th tree, and c is the constant term for the initial prediction value offset of the model. t ) is expressed as follows:

[0018]

[0019] Where γ and λ are penalty coefficients, K is the number of leaves that controls the complexity of each tree, ∥ω∥ 2L2 norm to control the complexity of the model;

[0020] 1-2) The feature importance score is calculated during the decision tree construction process of XGBoost. The gain of a single split point during the decision tree construction process is:

[0021]

[0022] in, and is the weight of the left and right child nodes after splitting, and is the predicted value of the child node, and p is the regularization parameter. With each iteration of the XGBoost model, the gain of each feature as a split node in all decision trees will be accumulated to form the total gain of the feature. The total gain of each feature TotalGain(f) is:

[0023]

[0024] The sum is the accumulation of the gains of all split points involving feature f. After obtaining the total gain of each feature, the importance score of each feature, Importance(f), is:

[0025]

[0026] Here, MaxTotalGain is the maximum value of the total gain of all features, and Importance(f) is the importance score of feature f;

[0027] 1-3) Set a flag variable flag for each feature in the dataset i , flag i When it is 1, it means that the feature is an active feature. The initial value of the marker variable of the active feature is 1, and the initial value of the marker variables of the other features is 0. Before each round of feature selection, the marker variables are initialized according to the above rules;

[0028] 1-4) After obtaining the importance scores of each feature, the non-positive features with the lowest importance scores are continuously removed. After each feature is removed, K-fold cross validation is used to evaluate the scores of the remaining features. The score of K-fold cross validation is Score MAE The calculation formula is:

[0029]

[0030] Among them, K is the number of cross-validation folds, n k is the number of samples in the k-fold test set, y i,k is the i-th observation in the k-fold test set, is the i-th predicted value in the k-fold test set;

[0031] 1-5) If the K-fold cross-validation score increases after removing a feature, it means that the removed feature is a negative feature; if the K-fold cross-validation score decreases after removing a feature, the removal operation is canceled and the marker variable of the feature is set to 1, indicating that the feature is a positive feature in this round of feature selection;

[0032] 1-6) Perform multiple rounds of feature selection on the feature set until there are no negative features in a certain round of feature selection, and the remaining features are the high-quality feature subsets that are most strongly correlated with the remaining completion time.

[0033] Furthermore, in step S5, the normalized feature importance scores are linearly transformed to obtain the query weight matrix, key weight matrix, and value weight matrix of the self-attention mechanism, including the following steps:

[0034] 2-1) Normalize the importance scores of the selected features. The normalization calculation formula is as follows:

[0035]

[0036] Among them, X score(i) For feature X i The corresponding feature importance score, X score(min) is the minimum value in the feature importance score set, X score(max) is the maximum value in the feature importance score set, Yes X score(i) Normalized value;

[0037] 2-2) Perform a linear transformation on the normalized feature importance scores to obtain the self-attention mechanism weight matrix. During the linear transformation process, a decay factor is set to control the size of the weight matrix element value. The linear transformation formula is as follows:

[0038]

[0039] Where W is the weight matrix of the self-attention mechanism obtained by linear transformation, W[i][j] is the element in the i-th row and j-th column of the weight matrix, and df is the attenuation factor;

[0040] 2-3) Obtain the query weight matrix W through the self-attention mechanism weight matrix Q , key weight matrix W K , value weight matrix W V :W Q =W K =W V =W.

[0041] Furthermore, in step S6, the key feature data is substituted into the adaptive weight allocation attention mechanism time series prediction model for prediction, including the following steps:

[0042] 3-1) The prediction model selects LSTM as the base model. The model mainly consists of the input layer, LSTM layer, self-attention layer and output layer. The input layer receives the feature vector as input, and the input data passes through the LSTM layer to obtain the output vector of each time step:

[0043] h t =LSTM(x t ,h t-1 )

[0044] Among them, h t is the LSTM hidden state vector at time step t, x t is the input vector at time step t, h t-1 is the hidden state vector at the previous time step;

[0045] 3-2) The hidden state vector output by the LSTM layer is used as the input of the self-attention layer, and the query vector (Q t ), key vector (K t ), value vector (V t ):

[0046] Q t =h t W Q +b Q

[0047] K t =h t W K +b K

[0048] V t =h t W V +b V

[0049] Among them, b Q 、b K 、b V They are query bias, key bias and value bias respectively;

[0050] 3-3) When the query vector and key vector are known, feature X can be obtained by calculation i Attention score:

[0051]

[0052] in Represents feature X iThe key vector at time step t, T represents the transpose operation;

[0053] 3-4) Attention score Score ti Perform Softmax normalization to obtain feature X i The attention weight att ti for:

[0054]

[0055] 3-5) Weighted fusion of attention weights and value vectors:

[0056]

[0057] Among them, Context t is the context vector at time step t, V ti It is feature X i A vector of values ​​of ;

[0058] 3-6) The hidden state vector h output by the LSTM layer t The context vector Context generated by the self-attention layer t Fusion, fusion vector h t 'for:

[0059] h t '=h t +Context t

[0060] 3-7) h t 'Substitute into the output layer to get the target value, that is, the remaining completion time:

[0061]

[0062] Among them, tanh is the activation function of the output layer, is the predicted remaining completion time.

[0063] The beneficial effects of adopting the above technical solution are:

[0064] The present invention proposes a method for predicting the remaining completion time based on an adaptive attention mechanism. Aiming at the problem of "multiple noises and high redundancy" in the manufacturing data generated in the complex manufacturing process, an adaptive optimization feature self-selection algorithm is proposed. The XGBoost model is used to train the data set to obtain the feature importance of all features in the data set, and the non-active features with the lowest importance are iteratively eliminated. After each elimination, K-fold cross validation is performed to calculate the score of the current data set. When the score of the K-fold cross validation is stable, the remaining features are high-quality feature subsets. Aiming at the problem of strong dynamics of production conditions in complex manufacturing processes, a time series prediction model based on an adaptive weight allocation attention mechanism is proposed. The model introduces an adaptive weight allocation attention mechanism to adaptively adjust the weights assigned to different key features, adapt to the dynamics and uncertainty in the complex manufacturing process, and achieve accurate prediction of the remaining completion time.

[0065] The proposed remaining completion time prediction method is applied to the Helpdesk dataset and the actual production process dataset of a domestic wheel hub machining factory. The experiments show that the mean absolute error (MAE) and root mean square error (RMSE) of the proposed method on the Helpdesk dataset are 2.85 and 4.79 respectively, and the mean absolute error (MAE) and root mean square error (RMSE) on the wheel hub machining dataset are 3.02 and 7.75 respectively. Compared with the existing remaining completion time prediction methods, the mean absolute error (MAE) and root mean square error (RMSE) on the two datasets are reduced. It verifies that the proposed remaining completion time prediction method can adapt to the dynamics and uncertainties in complex manufacturing processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a structural diagram of the remaining completion time prediction method based on the adaptive attention mechanism;

[0067] Figure 2 It is the structure diagram of the adaptive optimization feature self-selection module;

[0068] Figure 3 This is the structure diagram of the time series prediction module based on the adaptive weight allocation attention mechanism;

[0069] Figure 4 This is the prediction result of the comparative experiment on the Helpdesk dataset;

[0070] Figure 5 This is a graph of the prediction results of the comparative experiment on the wheel hub machining data set;

[0071] Figure 6 This is the result of the AATP-NonAFOM ablation experiment;

[0072] Figure 7This is the result of AATP-NonSAM ablation experiment. DETAILED DESCRIPTION

[0073] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0074] This paper takes the prediction of remaining completion time as the background and uses deep learning technology as the main carrier. It proposes a feature self-selection algorithm based on adaptive optimization and a remaining completion time prediction method based on adaptive weight allocation attention mechanism. Its architecture is as follows Figure 1 As shown, the following steps are included:

[0075] S1: Construct a method framework including an adaptive optimization feature self-selection module and an adaptive weight allocation attention mechanism time series prediction module;

[0076] S2: For missing values ​​of text features in the dataset, fill them with "None", and for missing values ​​of numerical features in the dataset, fill them with the mode of the same numerical feature; after the missing values ​​are filled, the numerical features are normalized;

[0077] S3: A feature selection method based on adaptive feature optimization is constructed in the adaptive optimization feature self-selection module. First, the XGBoost model is used to train the data set to obtain the feature importance of all features in the data set. Secondly, the non-active features with the lowest importance are iteratively removed. After each removal, K-fold cross validation is performed to calculate the score of the current data set. Finally, when the score of the K-fold cross validation is stable, the remaining features are the key feature subsets, including the following steps:

[0078] 1-1) Use the XGBoost model to train the data set. When processing the XGBoost objective function, add a regularization term to reduce the variance of the model and prevent the model from overfitting. The objective function of XGBoost is Obj (t) :

[0079]

[0080] In the formula, is the loss function of the model; y i is the true value of the i-th data, is the predicted value of the i-th data, f t is the regularization term of the complexity of the t-th tree, and c is the constant term for the initial prediction value offset of the model. t ) is expressed as follows:

[0081]

[0082] Where γ and λ are penalty coefficients, K is the number of leaves that controls the complexity of each tree, ∥ω∥2 L2 norm to control the complexity of the model;

[0083] 1-2) The feature importance score is calculated during the decision tree construction process of XGBoost. The gain of a single split point during the decision tree construction process is:

[0084]

[0085] in, and is the weight of the left and right child nodes after splitting, and is the predicted value of the child node, and p is the regularization parameter. With each iteration of the XGBoost model, the gain of each feature as a split node in all decision trees is accumulated to form the total gain of the feature. The total gain of each feature TotalGain(f) is:

[0086]

[0087] The sum is the accumulation of the gains of all split points involving feature f. After obtaining the total gain of each feature, the importance score of each feature, Importance(f), is:

[0088]

[0089] Here, MaxTotalGain is the maximum value of the total gain of all features, and Importance(f) is the importance score of feature f;

[0090] 1-3) Set a flag variable flag for each feature in the dataset i , flag i When it is 1, it means that the feature is an active feature. The initial value of the marker variable of the active feature is 1, and the initial value of the marker variables of the other features is 0. Before each round of feature selection, the marker variables are initialized according to the above rules;

[0091] 1-4) After obtaining the importance scores of each feature, the non-positive features with the lowest importance scores are continuously removed. After each feature is removed, K-fold cross validation is used to evaluate the scores of the remaining features. The score of K-fold cross validation is Score MAE The calculation formula is:

[0092]

[0093] Among them, K is the number of cross-validation folds, n k is the number of samples in the k-fold test set, y i,k is the i-th observation in the k-fold test set, is the i-th predicted value in the k-fold test set;

[0094] 1-5) If the K-fold cross-validation score increases after removing a feature, it means that the removed feature is a negative feature; if the K-fold cross-validation score decreases after removing a feature, the removal operation is canceled and the marker variable of the feature is set to 1, indicating that the feature is a positive feature in this round of feature selection;

[0095] 1-6) Perform multiple rounds of feature selection on the feature set until there are no negative features in a certain round of feature selection, and the remaining features are the high-quality feature subsets that are most strongly correlated with the remaining completion time.

[0096] S4: Encode key features, perform Word2Vec encoding on "activity" features, and for other categorical features, use random initialization vector encoding if there are many feature value categories, and use index-based encoding if there are few feature value categories;

[0097] S5: Normalize the feature importance scores of the key features, and then perform linear transformation to obtain the self-attention mechanism query weight matrix, key weight matrix, and value weight matrix. During the linear transformation process, set an attenuation factor to control the size of the weight matrix element value, including the following steps:

[0098] 2-1) Normalize the importance scores of the selected features. The normalization calculation formula is as follows:

[0099]

[0100] Among them, X score(i) For feature X i The corresponding feature importance score, X score(min) is the minimum value in the feature importance score set, X score(max) is the maximum value in the feature importance score set, Yes X score(i) Normalized value;

[0101] 2-2) Perform a linear transformation on the normalized feature importance scores to obtain the self-attention mechanism weight matrix. During the linear transformation process, a decay factor is set to control the size of the weight matrix element value. The linear transformation formula is as follows:

[0102]

[0103] Among them, W is the weight matrix of the self-attention mechanism obtained by linear transformation, W[i][j] is the element in the i-th row and j-th column of the weight matrix, and df is the attenuation factor

[0104] 2-3) Obtain the query weight matrix W through the self-attention mechanism weight matrix Q , key weight matrix W K , value weight matrix W V :W Q =W K =W V =W.

[0105] S6: The adaptive weight allocation attention mechanism timing prediction module consists of an input layer, an LSTM layer, a self-attention layer, and an output layer. Substitute the key feature data into the adaptive weight allocation attention mechanism timing prediction model for prediction to obtain the remaining completion time, including the following steps:

[0106] 3-1) The prediction model selects LSTM as the base model. The model mainly consists of the input layer, LSTM layer, self-attention layer and output layer. The input layer receives the feature vector as input, and the input data passes through the LSTM layer to obtain the output vector of each time step:

[0107] h t =LSTM(x t ,h t-1 )

[0108] Among them, h t is the LSTM hidden state vector at time step t, x t is the input vector at time step t, h t-1 is the hidden state vector at the previous time step;

[0109] 3-2) The hidden state vector output by the LSTM layer is used as the input of the self-attention layer, and the query vector (Q t ), key vector (K t ), value vector (V t ):

[0110] Q t =h t W Q +b Q

[0111] K t =h t W K +b K

[0112] V t =h t W V +b V

[0113] Among them, b Q 、b K 、b VThey are query bias, key bias and value bias respectively;

[0114] 3-3) When the query vector and key vector are known, feature X can be obtained by calculation i Attention score Score ti :

[0115]

[0116] in Represents feature X i The key vector at time step t, T represents the transpose operation;

[0117] 3-4) Attention score Score ti Perform Softmax normalization to obtain feature X i The attention weight att ti for:

[0118]

[0119] 3-5) Weighted fusion of attention weights and value vectors:

[0120]

[0121] Among them, Context t is the context vector at time step t, V ti It is feature X i A vector of values ​​of ;

[0122] 3-6) The hidden state vector h output by the LSTM layer t The context vector Context generated by the self-attention layer t Fusion, fusion vector h t 'for:

[0123] h t '=h t +Context t

[0124] 3-7) h t 'Substitute into the output layer to get the target value, that is, the remaining completion time:

[0125]

[0126] Among them, tanh is the activation function of the output layer, is the predicted remaining completion time.

[0127] Based on the above steps, the present invention effectively solves the problems of "multiple noises and high redundancy" in manufacturing data generated in complex manufacturing processes and strong uncertainty and dynamics in complex working conditions, and proposes a method for predicting the remaining completion time based on an adaptive attention mechanism. First, in view of the problem of "multiple noises and high redundancy" in manufacturing data generated in complex manufacturing processes, an adaptive optimization feature self-selection algorithm is proposed to extract a subset of key features from manufacturing data with strong noise and redundancy; secondly, in view of the problems of strong dynamics and uncertainty in complex manufacturing processes, a time series prediction algorithm based on an adaptive weight allocation attention mechanism is proposed, and an adaptive weight allocation attention mechanism is introduced to dynamically adjust the weights according to the importance of the input features, further improving the accuracy of the remaining completion time prediction under complex working conditions.

[0128] Experimental verification of the remaining completion time prediction method based on the adaptive attention mechanism of the present invention:

[0129] 1. Test environment

[0130] The computer was equipped with Intel(R) Core(TM) i5-9300H CPU@2.40GHz processor, Windows 10, 64-bit operating system, Python 3.6 as the program development environment, and Pytorch framework was used for the experiment.

[0131] 2. Test verification

[0132] Experimental results and analysis on the Helpdesk dataset and the actual production process dataset of a domestic wheel machining factory

[0133] (1) Dataset description

[0134] Dataset 1: Helpdesk dataset. The Helpdesk dataset is a ticket management process of the help desk of an Italian software company. It contains 4580 trajectories, that is, the dataset can be divided into 4580 groups of data. Each group of data is a complete process of ticket processing. Each trajectory contains a different number of events, with a minimum of two events and a maximum of 10 events. The dataset contains a total of 21348 data, each of which is an event in the ticket processing process. The dataset has 15 original features. The Helpdesk dataset is processed, and the remaining time corresponding to each event from the end of the processing process to which it belongs, remaining_time, is extracted from the dataset and used as the target variable. Table 1 shows some event logs of the Helpdesk dataset.

[0135] Table 1 Partial event log of Helpdesk dataset

[0136]

[0137]

[0138] Dataset 2: This experiment uses actual production data from a wheel hub machining factory in China. Wheel hub machining includes die casting, heat treatment, rough turning, spraying, laser engraving and other processes. Different types of wheel hub machining require different processes and production processes. The dataset contains 1538 wheel hub machining order samples, that is, the dataset can be divided into 1538 groups of data, each group of data is a complete process of wheel hub machining and manufacturing, and the dataset contains 20305 data in total, each of which is a processing data record of a process in the wheel hub machining and manufacturing process. Each order sample has 68 manufacturing features, and 17 features are retained as input variables after feature selection. The original data is processed, and the remaining completion time corresponding to each data from the entire order is extracted from the original data as the target variable. The characteristic variable table of the wheel hub manufacturing process is shown in Table 2.

[0139] Table 2 Wheel hub manufacturing process characteristic variables

[0140] Feature Name Feature description Feature Name Feature description content Activity same_mould_info Same mold information year Operation time (year) plan_roughcast_num Planned delivery quantity month Operation time (month) plan_send_samples_dt Planned completion time week Operation time (week) plan_product_place Production workshop day Operation time (day) operation_user_id Operator Number hour Operation time (hours) wheeled_poka_yoke Wheel error proofing executiontime Execution time back_reason Reason for rework totalexecutiontime Total execution time form_work_str Process information remarks Remark

[0141] (2) Implementation details and evaluation indicators

[0142] The model training uses the Adam (adaptive moment estimation) optimizer, which is an optimization algorithm used in deep learning. It combines the ideas of gradient descent momentum and RMSprop algorithm. The learning rate is 0.001, the training rounds are 300, and the batch size is 32. Data processing is performed before input. Missing values ​​of text features are filled with "none", and missing values ​​of numerical features are filled with the mode of the same numerical feature. The model consists of input layer, LSTM layer, self-attention layer and output layer. Input data is feature encoded in the input layer, "activity" features are Word2vec encoded, and for other classification features, if there are more feature value categories, random initialization vector encoding is used. If there are fewer feature value categories, index-based encoding is used, and numerical features are normalized.

[0143] In order to better evaluate the prediction performance of the model, this paper uses the mean absolute error (MAE) and root mean square error (RMSE) as evaluation indicators, which are defined as follows:

[0144]

[0145] Where i = 1, 2, ..., N, N represents the total length of the test set, y i , Represent the true value of the target variable and the model predicted value respectively. MAE is less sensitive to outliers and can better reflect the average deviation between the predicted value and the true value, while RMSE is more sensitive to larger errors and can analyze the degree of deviation in the model prediction in more detail. The smaller the values ​​of MAE and RMSE, the smaller the error between the true value and the predicted value, and the more accurate the model prediction.

[0146] (3) Analysis of comparative experimental results

[0147] In order to verify the effectiveness of the remaining completion time prediction method based on the adaptive attention mechanism, the method is denoted as AATP. The model is compared with the Helpdesk dataset and the actual production process dataset of a domestic wheel hub machining factory. The experimental results are shown in Tables 3 and 4. The brief description of each comparison model is as follows:

[0148] Remaining completion time prediction model based on attention mechanism and sparrow search algorithm (Att-SSA-LSTM): The Att-SSA-LSTM model takes LSTM as the base model, introduces the attention mechanism to redistribute the weights of various manufacturing features, and improves the sparrow search algorithm to optimize the hyperparameters of the model, thereby improving the accuracy of the remaining completion time prediction.

[0149] Bidirectional long short-term memory network prediction model integrating multi-constant Trojan units (BiLSTM_MCEC): The BiLSTM_MCEC model is based on the bidirectional long short-term memory neural network model (BiLSTM), and expands the multi-constant Trojan unit (MCEC) on the basis of the LSTM model, adding the CEC unit to store information at different times, which is conducive to learning feature information of different dimensions.

[0150] Table 3 Comparison of prediction results of Helpdesk dataset comparative experiment

[0151] Model MAE RMSE Att-SSA-LSTM 3.08 5.24 BiLSTM_MCEC 3.24 5.85 AATP 2.85 4.79

[0152] From the prediction results of the comparative experiment of the Helpdesk dataset in Table 3, it can be seen that on the Helpdesk dataset, the AATP model has a 7.4% decrease in MAE and an 8.6% decrease in RMSE compared to the Att-SSA-LSTM model. Compared with the BiLSTM_MCEC model, the AATP model has a 12% decrease in MAE and an 18.1% decrease in RMSE. The number of events contained in each trajectory in the Helpdesk dataset is different, the lengths of different trajectories are also different, and the differences between trajectories are large. Among the three models, AATP has the best effect on the Helpdesk dataset, indicating that the AATP model can better adapt to the differences and dynamics between different trajectories than the other two models, and can make targeted predictions for different key features of different trajectories.

[0153] Table 4 Comparison of prediction results of wheel hub machining data set comparison experiment

[0154]

[0155]

[0156] From the prediction results of the wheel hub machining data set comparison experiment in Table 4, it can be seen that in the actual production data set of the wheel hub machining factory, the AATP model has a 2.2% decrease in MAE and a 1.3% decrease in RMSE compared with the Att-SSA-LSTM model. Compared with the BiLSTM_MCEC model, the MAE of the AATP model has decreased by 2.8% and the RMSE has decreased by 1.4%, indicating that the AATP model can make targeted predictions for different key features of different product batches, and has a high accuracy in predicting the remaining completion time under complex working conditions with strong dynamics and uncertainty.

[0157] Furthermore, in order to explore the contribution of each module in the AATP model to performance, an ablation experiment was conducted, that is, each module was gradually removed from the complete model to observe the changes in model performance to prove the effectiveness of each module of AATP. The experiment was conducted on the Helpdesk dataset, and the experimental results are shown in Table 5, where a brief description of each model is as follows:

[0158] AATP-NonAFOM stands for No Adaptive Feature Optimization-based Feature Selection Module, which only uses the original input data as the input of the prediction model.

[0159] AATP-NonSAM stands for No Adaptive Weight Allocation Attention Mechanism Module, and only the base model LSTM is used for remaining completion time prediction.

[0160] AATP represents the complete model of this method.

[0161] Table 5 Comparison of prediction results of ablation experiments

[0162] Model MAE RMSE AATP-NonAFOM 5.91 9.17 AATP-NonSAM 5.12 7.49 AATP 2.85 4.79

[0163] According to Table 5, for AATP-NonAFOM, after removing the feature selection module based on adaptive feature optimization, both MAE and RMSE increased significantly. This shows that the feature selection module based on adaptive feature optimization is significantly effective and can successfully extract key feature subsets from the original data set with strong noise and redundancy. For AATP-NonSAM, after removing the adaptive weight allocation attention mechanism module, MAE and RMSE increased significantly. This shows that the adaptive weight allocation attention mechanism module is significantly effective and can successfully assign different weights to the key features of different product batches, improving the adaptability of the model to the uncertainty and dynamics of complex manufacturing processes.

[0164] In summary, the AATP prediction method proposed in this paper has better prediction effects on the wheel hub machining data set than other prediction models. The AATP model adaptively optimizes the feature self-selection module, iteratively removes the negative features with the lowest feature importance, and selects a subset of key features from the "noisy and highly redundant" original data set. At the same time, a time series prediction model based on an adaptive weight allocation attention mechanism is constructed to assign different weights to the key features of different product batches, thereby improving the adaptability of the model to the uncertainty and dynamics of complex working conditions, thereby improving the accuracy of the remaining completion time prediction.

[0165] 3. Conclusion

[0166] Aiming at the problems of "multiple noises, high redundancy" and strong dynamics of production status in manufacturing data generated in complex manufacturing processes, this paper proposes a method for predicting remaining completion time based on adaptive attention mechanism. This method consists of two modules. First, the XGBoost model is used to calculate the feature importance, and the negative features with the lowest feature importance are iteratively eliminated to select the key feature subset with the strongest correlation with the remaining completion time prediction; secondly, a time series prediction module based on adaptive weight allocation attention mechanism is established, and the feature importance of key features is linearly transformed to obtain the weight matrix, and different weights are assigned to the key features of different product batches. After experimental verification with the Helpdesk dataset and the actual production dataset of a domestic wheel hub machining factory, it is proved that the remaining completion time prediction method based on adaptive attention mechanism can effectively extract key feature subsets from the original dataset, assign different weights to the key features of different product batches, and improve the accuracy of the remaining completion time prediction of complex manufacturing processes.

[0167] The above examples of the present invention are described in detail, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for predicting remaining completion time based on an adaptive attention mechanism, characterized in that: The steps include: S1: Construct a method framework including an adaptive optimization feature self-selection module and an adaptive weight allocation attention mechanism time series prediction module; S2: For missing values ​​of text features in the data set, fill them with "None", and for missing values ​​of numerical features in the data set, fill them with the mode of the same numerical feature; after the missing values ​​are filled, the numerical features are normalized; S3: Construct a feature selection method based on adaptive feature optimization in the adaptive optimization feature self-selection module. XGBoost is an efficient machine learning algorithm used to solve regression, classification and other machine learning tasks. First, the XGBoost model is used to train the data set to obtain the feature importance of all features in the data set. Secondly, iteratively remove the least important non-positive features. K-fold cross validation is a statistical method used to evaluate and compare the performance of machine learning models. K-fold cross validation is performed after each removal to calculate the score of the current data set. Finally, when the score of the K-fold cross validation is stable, the remaining features are the key feature subsets. S4: Encode key features. Word2Vec is a model for generating word embeddings. It can convert words into numerical vectors. These vectors can capture the semantic relationship between words. Word2Vec encoding is performed on the "activity" feature. For other categorical features, if there are many feature value categories, random initialization vector encoding is used. If there are fewer feature value categories, index-based encoding is used. S5: Normalize the feature importance scores of the key features, and then perform linear transformation to obtain the self-attention mechanism query weight matrix Q, key weight matrix K, and value weight matrix V. During the linear transformation process, set a decay factor to control the size of the weight matrix element value; S6: The adaptive weight allocation attention mechanism timing prediction module consists of an input layer, an LSTM layer, a self-attention layer, and an output layer. LSTM is a special recurrent neural network architecture designed to solve the gradient vanishing and gradient exploding problems encountered by standard RNNs when processing long sequence data. The key feature data is substituted into the adaptive weight allocation attention mechanism timing prediction model for prediction to obtain the remaining completion time.

2. The method for predicting remaining completion time based on an adaptive attention mechanism according to claim 1, characterized in that: In step S3, a feature selection method based on adaptive feature optimization is constructed in the adaptive optimization feature self-selection module to screen out a key feature subset, including the following steps: 1-1) Use the XGBoost model to train the data set. When processing the XGBoost objective function, add a regularization term to reduce the variance of the model and prevent the model from overfitting. The objective function of XGBoost Obj (t) for: In the formula, is the loss function of the model; y i is the true value of the i-th data, is the predicted value of the ith data, f t is the regularization term of the complexity of the t-th tree, c is the constant term for the initial prediction value offset of the model, and the regularization term Ω(f t ) is expressed as follows: Where γ and λ are penalty coefficients, K is the number of leaves that controls the complexity of each tree, ∥ω∥ 2 L2 norm to control the complexity of the model; 1-2) The feature importance score is calculated during the decision tree construction process of XGBoost. The gain of a single split point during the decision tree construction process is: in, and is the weight of the left and right child nodes after splitting, and is the predicted value of the child node, p is the regularization parameter. With each iteration of the XGBoost model, the gain of each feature as a split node in all decision trees will be accumulated to form the total gain of the feature. The total gain of each feature TotalGain(f) is: The sum is the accumulation of the gains of all split points involving feature f. After obtaining the total gain of each feature, the importance score Importance(f) of each feature is: Here, MaxTotalGain is the maximum value of the total gain of all features, and Importance(f) is the importance score of feature f; 1-3) Set a flag variable flag for each feature in the dataset i , flag i When it is 1, it means that the feature is an active feature. The initial value of the marker variable of the active feature is 1, and the initial values ​​of the marker variables of the other features are 0. Before each round of feature selection, the marker variables are initialized according to the above rules; 1-4) After obtaining the importance scores of each feature, the non-positive features with the lowest importance scores are continuously removed. After each feature is removed, K-fold cross validation is used to evaluate the scores of the remaining features. The score of K-fold cross validation is Score MAE The calculation formula is: Among them, K is the number of cross-validation folds, n k is the number of samples in the k-fold test set, y i,k is the i-th observation in the k-fold test set, is the i-th predicted value in the k-fold test set; 1-5) If the K-fold cross-validation score increases after removing a feature, it means that the removed feature is a negative feature; if the K-fold cross-validation score decreases after removing a feature, the removal operation is canceled and the marker variable of the feature is set to 1, indicating that the feature is a positive feature in this round of feature selection; 1-6) Perform multiple rounds of feature selection on the feature set until there are no negative features in a certain round of feature selection, and the remaining features are the high-quality feature subsets that are most strongly correlated with the remaining completion time.

3. The method for predicting remaining completion time based on an adaptive attention mechanism according to claim 1, characterized in that: In step S5, the normalized feature importance scores are linearly transformed to obtain the query weight matrix, key weight matrix, and value weight matrix of the self-attention mechanism, including the following steps: 2-1) Normalize the importance scores of the selected features. The normalization calculation formula is as follows: Among them, X score(i) For feature X i The corresponding feature importance score, X score(min) is the minimum value in the feature importance score set, X score(max) is the maximum value in the feature importance score set, Yes X score(i) Normalized value; 2-2) Perform a linear transformation on the normalized feature importance scores to obtain the self-attention mechanism weight matrix. During the linear transformation process, a decay factor is set to control the size of the weight matrix element value. The linear transformation formula is as follows: Where W is the weight matrix of the self-attention mechanism obtained by linear transformation, W[i][j] is the element in the i-th row and j-th column of the weight matrix, and df is the attenuation factor; 2-3) Obtain the query weight matrix W through the self-attention mechanism weight matrix Q , key weight matrix W K , value weight matrix W V :W Q =W K =W V =W.

4. The method for predicting remaining completion time based on an adaptive attention mechanism according to claim 1, characterized in that: In step S6, the key feature data is substituted into the adaptive weight allocation attention mechanism time series prediction model for prediction, including the following steps: 3-1) The prediction model selects LSTM as the base model. LSTM is a special recursive neural network that effectively captures long-term dependencies in time series by introducing "memory units" and three gating mechanisms (input gate, forget gate, and output gate). The model mainly consists of the input layer, LSTM layer, self-attention layer, and output layer. The input layer receives the feature vector as input, and the input data passes through the LSTM layer to obtain the output vector of each time step: h t =LSTM(x t ,h t-1 ) Among them, h t is the LSTM hidden state vector at time step t, x t is the input vector at time step t, h t-1 is the hidden state vector at the previous time step; 3-2) The hidden state vector output by the LSTM layer is used as the input of the self-attention layer, and the query vector (Q t ), key vector (K t ), value vector (V t ): Q t =h t W Q +b Q K t =h t W K +b K V t =h t W V +b V Among them, b Q , b K , b V They are query bias, key bias and value bias respectively; 3-3) When the query vector and key vector are known, feature X can be obtained by calculation i Attention score Score ti : in Represents feature X i The key vector at time step t, T represents the transpose operation; 3-4) Attention score Score ti Perform Softmax normalization. Softmax normalization is a commonly used method in multi-class classification problems. It converts a vector or a set of unnormalized prediction values ​​into a set of probability distributions. The exponential function exp(x) is a basic function in mathematics, usually called the natural exponential function, where e is the base of the natural logarithm, which is approximately equal to 2.71828. By performing Softmax normalization with the exp(x) function, the feature X can be obtained. i The attention weight att ti for: 3-5) Weighted fusion of attention weights and value vectors: Among them, Context t is the context vector at time step t, V ti It is feature X i A vector of values ​​of ; 3-6) The hidden state vector h output by the LSTM layer t The context vector Context generated by the self-attention layer t Fusion is performed to obtain the fusion vector h t ': h t ’=h t +Context t 3-7) h t 'Substitute into the output layer to get the target value, that is, the remaining completion time: Among them, tanh is the activation function of the output layer, is the predicted remaining completion time.

Images