A remaining completion time prediction method based on an adaptive attention mechanism

Abstract

The application discloses a remaining work time prediction method based on an adaptive attention mechanism, comprising the following steps: S1. constructing a remaining work time prediction method comprising an adaptive optimization feature self-selection module and an adaptive weight distribution attention mechanism time sequence prediction module; S2. performing data processing such as missing value filling and normalization on data; S3. constructing a feature selection method based on adaptive feature optimization in the adaptive optimization feature self-selection module, iteratively training the preprocessed data, continuously eliminating negative features, and selecting a key feature subset; S4. performing feature coding on the selected key features; S5. calculating a query weight matrix, a key weight matrix and a value weight matrix in the self-attention mechanism through linear transformation according to the importance of the features; and S6. inputting the key feature subset data into the self-attention mechanism with weight distribution in the adaptive weight distribution attention mechanism time sequence prediction module to perform model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remaining completion time prediction and deep learning, and particularly relates to a remaining completion time prediction method based on an adaptive attention mechanism. BACKGROUND

[0002] Under the background of Industry 4.0, the manufacturing process of manufacturing industry is becoming increasingly complex, the types of product demand are increasing, and the update and replacement are becoming faster and faster. A more diverse product category drives many enterprises to change from traditional inventory production to an order-based production mode. In the order-based production mode, production tasks and production processes will change frequently according to order demand, raw material supply or production line failure, etc. Higher requirements are put forward for dynamic adjustment of production plan and continuous optimization of production process. Therefore, the accurate prediction of the remaining completion time of the complex manufacturing process has become a research hotspot. In recent years, deep learning has shown good performance in the field of remaining completion time prediction.

[0003] With the rapid development of industrial digitization and intelligentization, some scholars have applied deep learning to the field of remaining completion time prediction. The article [Yu J H, Li T K, Wang B L, et al. Order completion cycle prediction method for job shop based on CNN-SVR[J]. Mechanical Design, 2023, 40(02): 57-64.] aims to predict the remaining completion time of orders in a job shop under the conditions of multiple products, multiple machines and multiple processes. A combination prediction method (CNN-SVR) based on convolutional neural network (CNN) and support vector regression (SVR) is proposed. This method has good performance in terms of goodness of fit and prediction error, and realizes accurate prediction of the remaining completion time of orders in a job shop under the conditions of multiple products, multiple machines and multiple processes. The article [Li B R. Research on order remaining completion time prediction method for automobile assembly workshop[D]. Hefei University of Technology, 2022.] aims to predict the remaining completion time of orders in an automobile assembly workshop. A method for predicting the remaining completion time of orders based on bilateral long short-term memory network (Bilateral LSTM) and multiple constant error units (MCEC) is proposed. This method can effectively complete accurate prediction of the remaining completion time of an automobile assembly workshop, providing an important reference for workshop scheduling and optimization. In order to further improve the accuracy of the prediction of the remaining completion time, some scholars have turned their attention to feature selection.

[0004] Regarding feature selection, the article [Zhu Haihua, Wang Jianjie, Li Fei, et al. Prediction of Remaining Completion Time of Workshop Orders Based on Feature Selection and SSA-LSTM [J / OL]. Journal of Mechanical Engineering, 1-15.] proposes a feature selection method based on Pearson correlation coefficient and L1-L2 regularization to select key data from massive manufacturing data and optimize hyperparameters through the Sparrow Search Algorithm (SSA) to improve the accuracy and efficiency of remaining completion time of workshop orders (ORCT) prediction; the article [Fang Weiguang, Guo Yu, Huang Shaohua, et al. Research on Intelligent Control Method of Production Process in Discrete Manufacturing Workshop Driven by Big Data [J]. Journal of Mechanical Engineering, 2021, 57(20):277-291.] proposes a filtering feature selection algorithm based on self-organizing map neural network (SOM) and feature weight fuzzy C-means clustering (FWFCM), which successfully screens out features highly correlated with remaining completion time of orders from a large number of candidate features, providing high-quality input data for subsequent prediction models and improving prediction accuracy. While the above methods select a subset of features associated with the remaining completion time to some extent, in a dynamic production environment, the model may need to be updated frequently, which will lead to a decrease in the model's real-time prediction performance. Summary of the Invention

[0005] The purpose of this invention is to provide a method for predicting remaining completion time based on an adaptive attention mechanism. This method utilizes an adaptive optimization feature selection method to continuously eliminate negative features with the lowest feature importance, extracting a high-quality feature subset from a high-dimensional manufacturing feature dataset. Simultaneously, it employs a time-series prediction method based on an adaptive weight allocation attention mechanism to assign different weights to key features of different product batches, thereby achieving personalized and accurate prediction of remaining completion time under complex operating conditions.

[0006] The technical solution adopted in this invention is:

[0007] A method for predicting remaining completion time based on an adaptive attention mechanism includes the following steps:

[0008] S1: Construct a method framework that includes an adaptive optimization feature self-selection module and an adaptive weight allocation attention mechanism temporal prediction module;

[0009] S2: For missing values ​​of text features in the dataset, fill them with "none"; for missing values ​​of numerical features in the dataset, fill them with the mode of the same numerical feature; after filling in the missing values, normalize the numerical features.

[0010] S3: In the adaptive optimization feature self-selection module, an adaptive feature optimization-based feature selection method is constructed. First, an XGBoost model is used to train the data set to obtain the feature importance of all features in the data set. Second, the least important non-positive features are iteratively removed. After each removal, K-fold cross-validation is performed to calculate the current data set score. Finally, when the K-fold cross-validation score is stable, the remaining features are the key feature subset.

[0011] S4: The key features are encoded, the positive features are encoded using Word2Vec, and for other classification features, if the feature value category is more, a random initialization vector encoding method is used, and if the feature value category is less, an index-based encoding method is used.

[0012] S5: The feature importance score of the key features is normalized, and then linearly transformed to obtain the self-attention mechanism Q, K, and V matrices. During linear transformation, a decay factor is set to control the size of the weight matrix element value.

[0013] S6: The adaptive weight allocation attention mechanism time series prediction module is composed of an input layer, an LSTM layer, a self-attention layer, and an output layer. The key feature data is input into the adaptive weight allocation attention mechanism time series prediction model for prediction to obtain the remaining completion time.

[0014] Further, in step S3, in the adaptive optimization feature self-selection module, an adaptive feature optimization-based feature selection method is constructed to screen out a key feature subset, including the following steps:

[0015] 1-1) An XGBoost model is used to train the data set. When processing the XGBoost objective function, a regularization term is added to reduce the model variance and prevent model overfitting. The XGBoost objective function is :

[0016]

[0017] In the formula, is the loss function of the model; is the true value of the i-th data, is the predicted value of the i-th data, is the regularization term of the t-th tree complexity, is the initial prediction value offset constant term of the model. The regularization term is expressed as follows:

[0018]

[0019] In the formula, is the penalty coefficient, and M is the number of leaves that control the complexity of each tree. L2 norm for controlling model complexity;

[0020] 1-2) Feature importance score is calculated in the process of building decision tree in XGBoost, gain of single split point in the process of building decision tree is:

[0021]

[0022] where, and are the weights of the left and right child nodes after splitting, is the prediction value of the child node, is the regularization parameter. With each iteration of the XGBoost model, the gain brought by each feature as a split node in all decision trees is accumulated to form the total gain of the feature. The total gain of each feature is:

[0023]

[0024] where, the summation is the accumulation of the gain of all split points involving the feature . After obtaining the total gain of each feature, the importance score of each feature is:

[0025]

[0026] Here, is the maximum value in the total gain of all features, is the importance score of the feature ;

[0027] 1-3) Set a marker variable for each feature in the data set , 1 represents that the feature is a positive feature. The initial value of the marker variable of the positive feature is 1, and the initial value of the marker variable of the remaining features is 0. Before each round of feature selection, the marker variable is initialized according to the above rules;

[0028] 1-4) After obtaining the importance score of each feature, the non-positive feature with the lowest importance score is continuously removed. After removing the feature each time, the score of the remaining features is evaluated by K-fold cross-validation, and the score of K-fold cross-validation The calculation formula is:

[0029]

[0030] where, is the number of folds of cross-validation, is the the number of samples in the fold test set, is the fold test set, the observation value in the fold test set, is the prediction value in the

[0031] 1-5) If the K-fold cross-validation score increases after removing the feature, it means that the removed feature is a negative feature; if the K-fold cross-validation score decreases after removing the feature, the removal operation is cancelled, and the label variable of the feature is set to 1, indicating that the feature is a positive feature in this round of feature selection;

[0032] 1-6) Perform multiple rounds of feature selection on the feature set until there is no negative feature in a round of feature selection, and the remaining features are the most strongly related high-quality feature subset to the remaining completion time.

[0033] Further, in step S5, the normalized feature importance score is linearly transformed to obtain the query weight matrix, the key weight matrix, and the value weight matrix of the self-attention mechanism, including the following steps:

[0034] 2-1) Normalize the importance score of the selected feature, and the normalization calculation formula is as follows:

[0035]

[0036] wherein, is the importance score of the feature corresponding to the feature, is the minimum value in the feature importance score set, is the maximum value in the feature importance score set, is the normalized value;

[0037] 2-2) Linearly transform the normalized feature importance score to obtain the self-attention mechanism weight matrix, and set a decay factor to control the size of the weight matrix element value during linear transformation. The linear transformation formula is as follows:

[0038]

[0039] wherein, is the self-attention mechanism weight matrix obtained by linear transformation, is the element of the weight matrix in the i-th row and the j-th column, is the decay factor;

[0040] 2-3) Obtain the query weight matrix Q by the self-attention mechanism weight matrix , key weight matrix , value weight matrix : = = = .

[0041] Further, in the step S6, the key feature data is substituted into the adaptive weight allocation attention mechanism time series prediction model for prediction, including the following steps:

[0042] 3-1) The prediction model selects LSTM as the base model, and the model mainly includes input layer, LSTM layer, self-attention layer and output layer. The input layer receives the feature vector as the input, and the input data passes through the LSTM layer to obtain the output vector of each time step:

[0043]

[0044] wherein, is the LSTM hidden state vector at time step , is the input vector at time step , and is the hidden state vector of the previous time step;

[0045] 3-2) The hidden state vector output by the LSTM layer is taken as the input of the self-attention layer, and the weight matrix is combined to obtain the query vector (q ), the key vector (k ) and the value vector (v ):

[0046]

[0047]

[0048]

[0049] wherein, , , are the query bias term, the key bias term and the value bias term, respectively;

[0050] 3-3) The attention score of the feature can be calculated by the query vector and the key vector:

[0051]

[0052] wherein represents the key vector of the feature at time step ,​ Indicates the transpose operation;

[0053] 3-4) Attention scores conduct Normalization yields the features. Attention weights for:

[0054]

[0055] 3-5) Weighted fusion of attention weights and value vectors:

[0056]

[0057] in, In time step The context vector, It is a feature The value vector;

[0058] 3-6) The hidden state vector output by the LSTM layer context vectors generated by the self-attention layer Perform fusion, fusion vector for:

[0059]

[0060] 3-7) Substituting this into the output layer yields the target value, which is the remaining completion time:

[0061]

[0062] in, The activation function of the output layer. This represents the predicted remaining completion time.

[0063] The beneficial effects of adopting the above technical solution are as follows:

[0064] This invention proposes a method for predicting remaining completion time based on an adaptive attention mechanism. Addressing the issues of "high noise and high redundancy" in manufacturing data generated during complex manufacturing processes, an adaptive optimization feature self-selection algorithm is proposed. An XGBoost model is used to train the dataset, obtaining the feature importance of all features. The least important non-active features are iteratively removed. After each removal, K-fold cross-validation is performed, and the score of the current dataset is calculated. When the K-fold cross-validation score stabilizes, the remaining features constitute a high-quality feature subset. To address the highly dynamic nature of production conditions in complex manufacturing processes, a time-series prediction model based on an adaptive weight allocation attention mechanism is proposed. This model introduces an adaptive weight allocation attention mechanism to adaptively adjust the weights assigned to different key features, adapting to the dynamics and uncertainties of complex manufacturing processes, and achieving accurate prediction of remaining completion time.

[0065] The proposed remaining completion time prediction method was applied to the Helpdesk dataset and a dataset from the actual production process of a wheel hub machining factory in China. Experiments show that the mean absolute error (MAE) and root mean square error (RMSE) of the proposed method on the Helpdesk dataset are 2.85 and 4.79, respectively, while those on the wheel hub machining dataset are 3.02 and 7.75, respectively. Compared with existing remaining completion time prediction methods, both the MAE and RMSE are reduced on both datasets. This verifies that the proposed remaining completion time prediction method can adapt to the dynamics and uncertainties of complex manufacturing processes. Attached Figure Description

[0066] Figure 1 This is a structural diagram of a remaining completion time prediction method based on an adaptive attention mechanism;

[0067] Figure 2 This is a diagram of the adaptive optimization feature self-selection module structure;

[0068] Figure 3 This is a structural diagram of a time-series prediction module based on an adaptive weight allocation attention mechanism;

[0069] Figure 4 This is a comparison experiment prediction result image on the Helpdesk dataset;

[0070] Figure 5 This is a comparison experiment prediction result graph on the wheel hub machining dataset;

[0071] Figure 6 This is a diagram showing the results of the AATP-NonAFOM ablation experiment;

[0072] Figure 7is a graph of AATP-NonSAM ablation experiment results. DETAILED DESCRIPTION

[0073] The application will be further described in detail below in combination with the accompanying drawings and specific embodiments.

[0074] The application takes the remaining completion time prediction as the background and takes the deep learning technology as the main carrier, proposes a feature self-selection algorithm based on adaptive optimization and a remaining completion time prediction method based on adaptive weight allocation attention mechanism, and the architecture is as shown in Figure 1 The method comprises the following steps:

[0075] S1: constructing a method framework comprising an adaptive optimization feature self-selection module and an adaptive weight allocation attention mechanism time series prediction module;

[0076] S2: for the missing values of the text features in the data set, filling in "none", and for the missing values of the numerical features in the data set, filling in the mode of the same numerical feature; after the missing value filling is completed, the numerical features are normalized;

[0077] S3: constructing a feature selection method based on adaptive feature optimization in the adaptive optimization feature self-selection module, first training the data set by using the XGBoost model to obtain the feature importance of all features in the data set; secondly, iteratively removing the non-positive features with the lowest importance, and performing K-fold cross-validation after each removal to calculate the current data set score; finally, when the K-fold cross-validation score is stable, the remaining features are the key feature subset, as shown in Figure 2 The method comprises the following steps:

[0078] 1-1) training the data set by using the XGBoost model, and adding a regularization term to reduce the variance of the model when processing the XGBoost objective function to prevent the model from overfitting. The objective function of XGBoost is :

[0079]

[0080] In the formula, is the loss function of the model; is the true value of the i th data, is the predicted value of the i th data, is the regularization term of the t th tree complexity, is the initial predicted value offset constant term of the model. The regularization term is expressed as follows:

[0081]

[0082] In the formula, M is the penalty coefficient, M is the number of leaves to control the complexity of each tree, L2 norm to control the complexity of the model;

[0083] 1-2) Calculate the feature importance score in the decision tree construction process of XGBoost, and the gain of a single split point in the decision tree construction process

[0084]

[0085] and is the weight of the left and right child nodes after splitting, is the prediction value of the child node, is the regularization parameter. With each iteration of the XGBoost model, the gain brought by each feature as a split node in all decision trees is accumulated to form the total gain of the feature. The total gain of each feature

[0086]

[0087] where the sum is the gain of all split points involving the feature After obtaining the total gain of each feature, the importance score of each feature

[0088] Here,

[0089] is the maximum value in the total gain of all features, is the importance score of feature

[0090] 1-3) Set a marker variable for each feature in the data set , 1 represents that the feature is a positive feature. The initial value of the marker variable of the positive feature is 1, and the initial value of the marker variable of the remaining features is 0. Before each round of feature selection, initialize the marker variable according to the above rules;

[0091] 1-4) After obtaining the importance score of each feature, continuously remove the non-positive feature with the lowest importance score, and evaluate the score of the remaining features using K-fold cross-validation after removing the feature each time. The score of K-fold cross-validation The calculation formula is:

[0092]

[0093] is the number of folds of cross-validation,​​​​​​ It is the first The number of samples in the test set. It is the first The first test set One observation value, It is the first The first test set One predicted value;

[0094] 1-5) If the K-fold cross-validation score increases after removing a feature, it indicates that the removed feature is a negative feature; if the K-fold cross-validation score decreases after removing a feature, the removal operation is canceled and the label variable of the feature is set to 1, indicating that the feature is a positive feature in this round of feature selection.

[0095] 1-6) Perform multiple rounds of feature selection on the feature set until there are no negative features in a certain round of feature selection. The remaining features are the high-quality feature subset that is most strongly correlated with the remaining completion time.

[0096] S4: Perform feature encoding on key features, Word2Vec encoding on positive features, and for other categorical features, use random initialization vector encoding if there are many feature value categories, and use index-based encoding if there are few feature value categories.

[0097] S5: Normalize the feature importance scores of key features, then perform a linear transformation to obtain the self-attention mechanism query weight matrix, key weight matrix, and value weight matrix. During the linear transformation, a decay factor is set to control the magnitude of the weight matrix element values, such as... Figure 3 As shown, it includes the following steps:

[0098] 2-1) Normalize the importance scores of the selected features. The normalization calculation formula is shown below:

[0099]

[0100] in, Features The corresponding feature importance score, It is the minimum value in the set of feature importance scores. The maximum value in the set of feature importance scores. yes The normalized value;

[0101] 2-2) The normalized feature importance scores are linearly transformed to obtain the self-attention mechanism weight matrix. During the linear transformation, a decay factor is set to control the magnitude of the weight matrix elements. The linear transformation formula is shown below:

[0102]

[0103] wherein, is a self-attention mechanism weight matrix obtained through linear transformation, is an element of the weight matrix in the i-th row and the j-th column, is a decay factor

[0104] 2-3) Obtain a query weight matrix through the self-attention mechanism weight matrix , a key weight matrix , and a value weight matrix : = = = .

[0105] S6: The adaptive weight allocation attention mechanism time series prediction module is composed of an input layer, an LSTM layer, a self-attention layer, and an output layer. The key feature data is substituted into the adaptive weight allocation attention mechanism time series prediction model for prediction to obtain the remaining completion time, including the following steps:

[0106] 3-1) The prediction model selects LSTM as the base model, and the model mainly includes an input layer, an LSTM layer, a self-attention layer, and an output layer. The input layer receives a feature vector as input, and the input data passes through the LSTM layer to obtain an output vector at each time step:

[0107]

[0108] wherein, is an LSTM hidden state vector at time step , is an input vector at time step , is a hidden state vector at the previous time step;

[0109] 3-2) The hidden state vector output by the LSTM layer is taken as the input of the self-attention layer, and the query vector (Q), the key vector (K), and the value vector (V) are calculated in combination with the weight matrix:

[0110]

[0111]

[0112]

[0113] wherein,​​​ 、 、 are query bias term, key bias term and value bias term respectively;

[0114] 3-3) The attention score of feature can be calculated given the query vector and the key vector :

[0115]

[0116] where denotes the key vector of feature at time step , and denotes the transpose operation;

[0117] 3-4) The attention score is normalized to obtain the attention weight of feature :

[0118]

[0119] 3-5) The attention weight is fused with the value vector:

[0120]

[0121] where is the context vector at time step , and is the value vector of feature ;

[0122] 3-6) The hidden state vector output by the LSTM layer is fused with the context vector generated by the self-attention layer, and the fused vector is:

[0123]

[0124] 3-7) The is substituted into the output layer to obtain the target value, i.e. the remaining completion time:

[0125]

[0126] where is the activation function of the output layer, and is the predicted remaining completion time.

[0127] ​Based on the above steps, the application effectively solves the problems of "multi-noise, high redundancy" and strong uncertainty and dynamics in complex manufacturing process, and proposes a remaining completion time prediction method based on adaptive attention mechanism. First, in view of the problems of "multi-noise, high redundancy" in the manufacturing data generated in the complex manufacturing process, an adaptive optimization feature self-selection algorithm is proposed to extract the key feature subset from the manufacturing data with strong noise and redundancy. Secondly, in view of the strong dynamics and uncertainty in the complex manufacturing process, a time series prediction algorithm based on adaptive weight allocation attention mechanism is proposed, which introduces the adaptive weight allocation attention mechanism, dynamically adjusts the weight according to the importance of the input features, and further improves the accuracy of the remaining completion time prediction under complex working conditions.

[0128] Test verification of the remaining completion time prediction method based on adaptive attention mechanism of the application

[0129] 1. Test environment

[0130] Intel(R)Core(TM)i5-9300H CPU@2.40GHz processor, Windows 10, 64-bit operating system computer, program development environment is Python3.6, and Pytorch framework is used for test.

[0131] 2. Test verification

[0132] Experimental results and analysis on Helpdesk data set and actual production process data set of a domestic hub machining factory

[0133] Data set description

[0134] Data set 1: Helpdesk data set. Helpdesk data set is a ticket management process of a help desk of an Italian software company, which contains 4580 trajectories, i.e. the data set can be divided into 4580 groups of data, each group of data is a complete process of ticket processing, each trajectory contains different number of events, the minimum contains two events and the maximum contains 10 events, the data set contains 21348 data, each data is an event in the ticket processing process, and the data set contains 15 original features. The Helpdesk data set is processed, and the remaining time of each event corresponding to the complete end of the processing process is extracted from the data set as the target variable. Table 1 shows part of the event log of the Helpdesk data set.

[0135] Table 1 Part of the event log of the Helpdesk data set

[0136] Case id Event id Complete Timestamp Activity Resource … e1 2012 / 10 / 9 14:50:00 Assign seriousness Peter … e2 2012 / 10 / 9 14:51:00 Take in charge ticket Peter … 1 e3 2012 / 10 / 12 15:02:00 Take in charge ticket Mike … e4 2012 / 10 / 25 11:54:00 Resolve ticket Peter … e5 2012 / 11 / 9 12:54:00 Closed Niki … … … … … … …

[0137] Data set 2: The actual production data of a domestic hub machining factory was used in the experiment. Hub machining includes die casting, heat treatment, rough turning, spraying, laser engraving and other processes. Different types of hub machining require different processes and production processes. This data set has 1538 hub machining order samples, i.e. the data set can be divided into 1538 groups of data. Each group of data is a complete process of hub machining and manufacturing. The data set contains 20305 data, each data is a process data record in the hub machining and manufacturing process. Each order sample has 68 manufacturing features, and after feature selection, 17 features are retained as input variables. The original data is processed, and the remaining time of each data corresponding to the entire order completion is extracted from the original data, which is used as the target variable. The hub manufacturing process feature variable table is shown in Table 2.

[0138] Table 2 Hub manufacturing process feature variable table

[0139] Feature Name Feature Description Feature Name Feature Description content Activity same_mould_info Same mould information year Operation time (year) plan_roughcast_num Plan delivery quantity month Operation time (month) plan_send_samples_dt Plan completion time week Operation time (week) plan_product_place Production workshop day Operation time (day) operation_user_id Operator number hour Operation time (hour) wheeled_poka_yoke Wheeled poka yoke execution time Execution time back_reason Rework reason total execution time Total execution time form_work_str Form work str remarks Remarks

[0140] Implementation details and evaluation indicators

[0141] The model training uses the Adam (adaptive moment estimation) optimizer, which is an optimization algorithm used in deep learning that combines the momentum of gradient descent and the learning rate of the RMSprop algorithm. The learning rate is 0.001, the training round is 300, and the batch size is 32. Data processing is performed before data input. The missing values of text features are filled with "none", and the missing values of numerical features are filled with the mode of the same numerical feature. The model includes input layer, LSTM layer, self-attention layer and output layer. The input data is encoded in the input layer. The positive features are encoded using Word2vec, and other classification features are encoded using random initialization vectors if the feature value category is large, or using index-based encoding if the feature value category is small. The numerical features are normalized.

[0142] To better evaluate the prediction performance of the model, the mean absolute error (MAE) and root mean square error (RMSE) are used as evaluation indicators, which are defined as follows:

[0143]

[0144]

[0145] Comparative experimental result analysis

[0146] To verify the effectiveness of the remaining completion time prediction method based on adaptive attention mechanism, denoted as AATP, comparative experiments were conducted on the Helpdesk dataset and the actual production process dataset of a domestic hub machining factory. The experimental results are shown in Tables 3 and 4, and the brief descriptions of each comparative model are as follows:

[0147] The remaining completion time prediction model based on attention mechanism and sparrow search algorithm (Att-SSA-LSTM): The Att-SSA-LSTM model takes LSTM as the base model, introduces attention mechanism to redistribute the weights of each manufacturing feature, and improves the sparrow search algorithm to optimize the hyperparameters of the model, thereby improving the accuracy of the remaining completion time prediction.

[0148] The bidirectional long short-term memory network prediction model fused with multiple constant Trojan units (BiLSTM_MCEC): The BiLSTM_MCEC model takes the bidirectional long short-term memory neural network model (BiLSTM) as the base model, and extends the multiple constant Trojan units (MCEC) based on the LSTM model, increasing the CEC unit to store information at different times, which is beneficial for learning feature information of different dimensions.

[0149] Table 3 Comparison of prediction results of comparative experiments on Helpdesk dataset

[0150]

[0151] As Figure 4 shown in Table 3, the comparison of prediction results of comparative experiments on Helpdesk dataset shows that, on the Helpdesk dataset, the AATP model has a 7.4% decrease in MAE and an 8.6% decrease in RMSE compared to the Att-SSA-LSTM model. Compared to the BiLSTM_MCEC model, the AATP model has a 12% decrease in MAE and an 18.1% decrease in RMSE. The number of events included in each trajectory in the Helpdesk dataset is not the same, the length of different trajectories is not equal, and the difference between trajectories is large. Among the three models, AATP performs best on the Helpdesk dataset, indicating that the AATP model can better adapt to the differences and dynamics between different trajectories and can make targeted predictions for different key features of different trajectories.

[0152] Table 4 Comparison of prediction results of comparative experiments on hub machining dataset

[0153] Model MAE RMSE Att-SSA-LSTM 3.86 8.92 BiLSTM_MCEC 4.21 9.06 AATP 3.02 7.75

[0154] As Figure 5As shown in Table 4, the comparison of the experimental prediction results of the hub machining data set shows that, on the actual production data set of the hub machining factory, the MAE of the AATP model is reduced by 2.2% and the RMSE is reduced by 1.3% compared with the Att-SSA-LSTM model. Compared with the BiLSTM_MCEC model, the MAE of the AATP model is reduced by 2.8% and the RMSE is reduced by 1.4%, which shows that the AATP model can make targeted prediction on different key features of different product batches, and has high accuracy in predicting the remaining finishing time under complex working conditions with strong dynamics and uncertainty.

[0155] Further, as shown in Figure 6 and Figure 7 To explore the contribution of each module in the AATP model to the performance, an ablation experiment is performed, i.e., each module is removed from the complete model one by one, and the change in model performance is observed to prove the effectiveness of each module of the AATP. The experiment is performed on the Helpdesk data set, and the experimental results are shown in Table 5, wherein a brief description of each model is as follows:

[0156] AATP-NonAFOM represents a feature selection module without adaptive feature optimization, and only uses the original input data as the input of the prediction model.

[0157] AATP-NonSAM represents a self-adaptive weight allocation attention mechanism module, and only uses the base model LSTM to predict the remaining finishing time.

[0158] AATP represents the complete model of the method in this paper.

[0159] Table 5 Comparison of ablation experiment prediction results

[0160] Model MAE RMSE AATP-NonAFOM 5.91 9.17 AATP-NonSAM 5.12 7.49 AATP 2.85 4.79

[0161] As can be seen from Table 5, for AATP-NonAFOM, after removing the feature selection module based on adaptive feature optimization, the MAE and RMSE are significantly increased. It can be shown that the feature selection module based on adaptive feature optimization is significantly effective, and can successfully extract a key feature subset from the original data set with strong noise and redundancy. For AATP-NonSAM, after removing the self-adaptive weight allocation attention mechanism module, the MAE and RMSE are significantly increased. It can be shown that the self-adaptive weight allocation attention mechanism module is significantly effective, and can successfully assign different weights to the key features of different product batches, and improve the adaptability of the model to the uncertainty and dynamics of the complex manufacturing process.

[0162] In summary, the AATP prediction method proposed in this paper has better prediction effect on the hub machining data set than other prediction models. The AATP model selects the key feature subset from the original data set with "much noise, high redundancy" by iteratively removing the negative features with the lowest feature importance through the adaptive optimization feature self-selection module. At the same time, the time series prediction model based on adaptive weight allocation attention mechanism is constructed, which gives different weight allocation to the key features of different product batches, improves the adaptability of the model to complex working conditions uncertainty and dynamics, and thus improves the accuracy of the remaining completion time prediction.

[0163] 3. Conclusion

[0164] In view of the problems of "much noise, high redundancy" and strong dynamic production conditions in complex manufacturing processes, a remaining completion time prediction method based on adaptive attention mechanism is proposed. This method contains two modules. First, the XGBoost model is used to calculate the feature importance, and the negative features with the lowest feature importance are iteratively removed to select the key feature subset with the strongest correlation with the remaining completion time prediction. Second, the time series prediction module based on adaptive weight allocation attention mechanism is established, and the feature importance of the key features is transformed into a weight matrix through linear transformation to give different weight allocation to the key features of different product batches. Through the Helpdesk data set and the actual production data set of a domestic hub machining factory, it is proved that the remaining completion time prediction method based on adaptive attention mechanism can effectively extract the key feature subset from the original data set, give different weight allocation to the key features of different product batches, and improve the accuracy of the remaining completion time prediction in complex manufacturing processes.

[0165] The above examples of the present application are described in detail, but the content is only the preferred embodiment of the present application, and cannot be considered as limiting the scope of the present application. Any equivalent changes and improvements made in the scope of the present application should still belong to the patent coverage of the present application.

Claims

1. A method for remaining completion time prediction based on adaptive attention mechanism, characterized in that, Comprise the following steps: S1: Construct a method framework comprising an adaptive optimization feature self-selection module and an adaptive weight allocation attention mechanism time series prediction module; S2: The data set comprises a Helpdesk data set and an actual production process data set of a domestic hub machining factory, and the Helpdesk data set is processed to extract the remaining time of each event from the end of the corresponding processing flow as the target variable; The original data of the actual production process of the domestic hub machining factory is processed to extract the remaining completion time of each data from the completion of the entire order as the target variable; for the missing values of the text features in the data set, fill in "none", and for the missing values of the numerical features in the data set, fill in the mode of the same numerical feature; after the missing value filling is completed, the numerical features are normalized; S3: In the adaptive optimization feature self-selection module, a feature selection method based on adaptive feature optimization is constructed, first, an XGBoost model is used to train the data set to obtain the feature importance of all features in the data set; second, the least important non-positive features are iteratively removed, and K-fold cross-validation is performed after each removal to calculate the current data set score; if the K-fold cross-validation score increases after removing the features, it means that the removed features are negative features, if the K-fold cross-validation score decreases after removing the features, the removal operation is cancelled, and the feature label variable is set to 1, indicating that the feature is a positive feature in this round of feature selection, and finally when the K-fold cross-validation score is stable, the remaining features are the key feature subset; The key feature subset of the Helpdesk data set is 15 original features; The key feature subset of the actual production process data set of a domestic hub machining factory is 17 manufacturing features for each order sample; S4: Feature encoding is performed on the key features in the key feature subset, Word2Vec encoding is performed on the positive features, and for other classification features, if the feature value category is more, a random initialization vector encoding method is used, and if the feature value category is less, an index-based encoding method is used; S5: The feature importance scores of the key features are normalized, and then linearly transformed to obtain the query weight matrix Q, the key weight matrix K, and the value weight matrix V; during linear transformation, a decay factor is set to control the size of the weight matrix element value; S6: The adaptive weight allocation attention mechanism time series prediction module is composed of an input layer, an LSTM layer, a self-attention layer and an output layer, and the corresponding data of the key features is input into the adaptive weight allocation attention mechanism time series prediction module for prediction to obtain the remaining completion time. 2.The remaining completion time prediction method based on adaptive attention mechanism according to claim 1, characterized in that: In the adaptive optimization feature self-selection module in step S3, a feature selection method based on adaptive feature optimization is constructed to screen out a key feature subset, comprising the following steps: The XGBoost model is used to train the data set. When processing the XGBoost objective function, a regularization term is added to reduce the variance of the model and prevent overfitting of the model. The objective function of XGBoost is: ​ wherein is a loss function of the model; is the true value of the i-th data, is the predicted value of the i-th data, is a regularization term of the complexity of the t-th tree, is a regularization term of the initial predicted value offset constant of the model is expressed as follows: wherein M is the number of leaves controlling the complexity of each tree, is the L2 norm controlling the model complexity. Feature importance scores are computed during the decision tree construction process in XGBoost, the gain of a single split point during the decision tree construction process is: where, and are the weights of the left and right child nodes after splitting, is the prediction value of the child node, is a regularization parameter, with each iteration of the XGBoost model, the gain brought by each feature when it is used as a splitting node in all decision trees is accumulated to form the total gain of the feature, and the total gain of each feature is: where the sum is over all features involved in the split point and the importance score for each feature is: ​ Here, is the maximum value among the total gains of all features, is the importance score of the feature . Setting a label variable for each feature in the data set , 1 represents that the feature is a positive feature, the initial value of the label variable of the positive feature is 1, and the initial values of the label variables of the remaining features are all 0. Before the start of each round of feature selection, the label variables are initialized according to the above rules; After obtaining the importance score of each feature, the non-positive feature with the lowest importance score is continuously removed, and the score of the remaining features is evaluated by K-fold cross-validation after removing the feature each time, and the score of K-fold cross-validation The calculation formula is: in, It is the number of folds in cross-validation. It is the first The number of samples in the test set. It is the first The first test set One observation value, It is the first The first test set One predicted value; The feature set is subjected to multiple rounds of feature selection until there is no negative feature in a certain round of feature selection, and the remaining features are the most relevant high-quality feature subset. 3.The method of claim 1, wherein: In the step S5, the normalized feature importance score is linearly transformed to obtain the query weight matrix, the key weight matrix and the value weight matrix of the self-attention mechanism, including the following steps: 2-1) The importance score of the selected feature is normalized, and the normalization calculation formula is as shown below: wherein, is a feature corresponding feature importance score, is the minimum value in the set of feature importance scores, is the maximum value in the set of feature importance scores, is is a normalized value; 2-2) The normalized feature importance score is linearly transformed to obtain the self-attention mechanism weight matrix, and a decay factor is set to control the size of the weight matrix element value in the linear transformation process, and the linear transformation formula is as shown below: wherein, is a self-attention mechanism weight matrix obtained by linear transformation, is an element of the weight matrix in the i-th row and the j-th column, is a decay factor; 2-3) Obtain the query weight matrix through the self-attention mechanism weight matrix , the key weight matrix , the value weight matrix : = = = .

4. The method of claim 1, wherein the method is based on an adaptive attention mechanism. In the step S6, the key feature data is substituted into the adaptive weight allocation attention mechanism time series prediction module for prediction, including the following steps: 3-1) The prediction module selects LSTM as the base model, the input layer receives the feature vector after feature encoding as the input, and the input data is subjected to the LSTM layer to obtain the output vector of each time step: in, In time step The LSTM hidden state vector, In time step The input vector, It is the hidden state vector of the previous time step; 3-2) The hidden state vector output by the LSTM layer is taken as the input of the self-attention layer, and the query vector is calculated by combining the weight matrix , key vector , value vector : wherein, , , are a query bias term, a key bias term and a value bias term, respectively; 3-3) The feature can be calculated by computing given the query vector and the key vector Attention score : wherein denotes a feature at time step key vector, denotes a transpose operation; 3-4) Attention score Performs Normalization, Softmax normalization by exp(x) function, e is the base of natural logarithm, get the attention weight of feature is:​ 3-5) The attention weight and the value vector are weighted and fused: wherein, is the context vector at time step is the value vector of the feature ;​ 3-6) the hidden state vector output by the LSTM layer the context vector generated by the self-attention layer fusion vector : ； 3-7) will be Substituting the output layer gives the target value, which is the remaining completion time: wherein, is an activation function for the output layer, is the predicted remaining completion time.

Images