Modeling method of attention-enhanced multi-time scale LSTM for starch liquefaction process

Abstract

The invention relates to the field of artificial intelligence and chemical engineering, in particular to a modeling method of attention-enhanced multi-time-scale LSTM (Long Short Term Memory) for a starch liquefaction process. The historical data of the starch liquefaction process is combined with the deep learning technology to realize the establishment of the prediction model. The attention-enhanced multi-time-scale LSTM can make up for the deficiency of a traditional LSTM in extracting local time sequence information, can obtain dynamic features of different scales of data more deeply, and improves the prediction precision of the model. Meanwhile, the method comprises data processing, hyper-parameter optimization of the model and model training, and has the characteristics of convenience in operation and high automation degree. Finally, the method is successfully applied to product quality estimation in the starch liquefaction process, real-time estimation of the product quality is successfully achieved through the model, and the method has important significance in the starch liquefaction process.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and chemical industry, and in particular to a dynamic modeling soft-sensing method for constructing spatiotemporal data features in a chemical process, specifically a modeling method of an attention-enhanced multi-time-scale LSTM for a starch liquefaction process. The method is mainly applicable to real-time estimation of product quality that is difficult to directly measure in a chemical process. Background Art

[0002] The starch liquefaction process is the first stage of the starch fructose preparation process, and the factory needs to monitor and control the DE value of the liquefied solution. The DE value is the reducing sugar equivalent, which indicates the percentage of reducing sugar in the dry matter of the syrup. If the DE value of starch is too low, the slurry is prone to aging and has a high viscosity, which is not conducive to operation, and will bring incompletely liquefied macromolecular dextrins into the saccharification process, directly affecting the saccharification quality; if the DE value is too high, it is not conducive to the formation of a complex structure between the enzyme and the substrate in the saccharification process, affecting the catalytic efficiency. Due to technical conditions and cost limitations, the factory currently still uses regular offline testing of product samples to detect the DE value of the slurry, which has a serious time lag. Therefore, it is of great significance to improve product quality by using data-driven models to estimate the product DE value in real time and change the operation strategy in a timely manner.

[0003] Data-driven modeling is a type of soft sensor modeling. By selecting variables that are easy to measure in the process and establishing a mathematical model to estimate key variables that are difficult to measure. Due to the widespread use of distributed control systems (DCS), enterprises have accumulated a large amount of historical data that can be used for data-driven modeling. This method has the advantages of less demand for expert knowledge, low modeling cost and fast running speed, making it increasingly popular in industry and academia. In recent years, deep learning technology can fully mine various deep features in the data through stacking network layers and some advanced network structure designs, so as to better fit the relationship between various variables in industrial production. Therefore, deep learning has also been widely introduced into industrial processes to establish data-driven models. Hong et al. (Hong, J., Tian, ​​W., 2023. Prediction in Catalytic Cracking Process Based on Swarm Intelligence Algorithm Optimization of LSTM. Processes 11, 1454.) used a long short-term memory network (LSTM) to extract the time relationship in the catalytic cracking process, and optimized the model parameters through the swarm intelligence algorithm (SIA), and finally achieved accurate prediction of the reaction temperature. The attention mechanism can give different weights to different parts of the data, and can automatically learn the relationship between data input and output during the modeling process. The attention mechanism has been widely used in data-driven modeling, which can effectively improve the prediction effect of the model and enhance the interpretability of the model. Han et al. (Han, Y., Fan, C., Xu, M., Geng, Z., Zhong, Y., 2019. Production capacity analysis and energy saving of complex chemical processes using LSTM based on attention mechanism. Applied Thermal Engineering 160, 114072) combined the attention mechanism with LSTM for capacity analysis, improved the prediction accuracy of the model, and calculated the weights of the layers that have an important impact on the prediction through attention.

[0004] At present, there are relatively few studies on the use of data-driven modeling in the field of starch liquefaction. In addition, simply using traditional deep learning models without targeted improvements to the model structure to adapt to the characteristics of the process often makes it difficult for the resulting data-driven models to achieve good prediction results. Summary of the invention

[0005] In view of the above problems, the present invention proposes a modeling method of using an attention-enhanced multi-time-scale LSTM (AMT-LSTM) for the starch liquefaction process. The method analyzes the historical collected data in the starch liquefaction process, uses the improved attention-enhanced multi-time-scale LSTM of the present invention, extracts the time series characteristics of the historical process data, and then achieves accurate estimation of product quality variables, and establishes an accurate data-driven model of the starch liquefaction process. The method has the advantages of strong real-time performance and high accuracy.

[0006] The specific technical solutions of the present invention are as follows:

[0007] A modeling method of a starch liquefaction process using an attention-enhanced multi-time-scale LSTM includes the following steps:

[0008] S1. Collect historical data of starch liquefaction section

[0009] The quality variables (i.e., DE values) of the products in the starch liquefaction process are defined as the predicted output variables of the model. By analyzing the enzymatic starch liquefaction process, auxiliary variables that are highly correlated with the product DE values ​​are identified and selected as input variables, and a deep learning model between input variables and output variables is constructed to achieve real-time prediction of DE values. These auxiliary variables mainly include process variables and operating parameters such as temperature, pressure, and liquid level of equipment such as starch preparation tanks, high-temperature steam ejectors, maintenance pipes, liquefaction reactors, flash tanks, pumps, and heat exchangers, as well as fluid parameters such as temperature and flow rate of starch emulsions and high-temperature steam in process pipelines, and physical parameters such as dry matter content. Subsequently, historical data of the above output and input variables for a period of time will be collected from the historical database of the distributed control system (DCS) and laboratory tests.

[0010] S2. Preprocess the collected historical data to obtain high-quality data sets to facilitate subsequent modeling.

[0011] S2.1 In order to ensure that variable data from different sources are consistent in time, the data from different measurement devices need to be time-aligned so that each variable has the same timestamp and time interval.

[0012] S2.2 detects and removes outliers in the data, and then processes missing values ​​in the data.

[0013] S2.3 performs noise reduction on the data to reduce the impact of signal fluctuations in production; then, the data is standardized to keep different variables at the same scale.

[0014] S2.4 uses a sliding time window to expand the one-dimensional data into two-dimensional data containing time variables to obtain dynamic time features. In addition, all data are divided into training set, validation set and test set.

[0015] S3. Design of attention-enhanced multi-timescale LSTM network model structure

[0016] The S3.1 model obtains the weights of different input variables on the target variable through the self-attention mechanism to enhance the feature extraction of the input, and uses residual connections to obtain the intermediate feature representation enhanced by hybrid attention. Its formulas are shown in (1)-(3).

[0017] u t =tanh(w w x t +b w ) (1)

[0018] α t =softmax(u t ) (2)

[0019] h sa =Σ t α t x t +x t (3)

[0020] Among them, x t is the input data sample, w w and b w is the weight matrix and bias of the linear transformation, tanh is the hyperbolic tangent activation function, u t is a transformation matrix similar to the input matrix; softmax normalizes the data of a certain dimension of the input sequence to obtain the attention weight α t ;h sa Intermediate feature representation enhanced by hybrid attention.

[0021] S3.2 uses multi-scale dilated causal convolutional layers to achieve effective extraction of temporal features in the intermediate feature representation. Specifically, the model sets multiple one-dimensional dilated causal convolutional layers with different dilation factors in parallel, so that the feature extraction capability with diversified receptive fields can be obtained without increasing the number of convolution kernel parameters. After being processed by dilated causal convolutional layers with different receptive fields, the model generates multiple feature maps, which are consistent in time steps but contain different feature dimensions, as shown in formula (4). Subsequently, these feature maps are spliced ​​along the time dimension to form a feature map with multiple time scales, as shown in formula (5).

[0022] h 1,2,...,N =dcc 1,2,...,N (h) (4)

[0023] h mt =h 1 ·h2 ·...·h N (5)

[0024] In which, · represents the tensor connection of the feature map; dcc represents the atrous causal convolutional layer, and the subscript numbers 1, 2, …, N represent different convolutional layers in parallel; h is the intermediate feature representation of the model, the subscripts 1, 2, …, N represent the N intermediate feature representations after convolution of different convolutional layers, and the subscript mt represents the intermediate feature representation of multiple time scales.

[0025] S3.3 passes the feature map with multiple time scales through the long short-term memory network (LSTM) and the multi-layer perceptron (MLP) in sequence. The LSTM network can gradually extract the dynamic features contained in the time series data, thereby effectively capturing the global time series features and compressing the feature dimensions, as shown in formula (6). Subsequently, the output data of the LSTM layer is flattened from two dimensions to one dimension, and linearly transformed and nonlinearly activated by MLP to obtain the final output result, as shown in formula (7).

[0026] h lstm =LSTM(h mt ) (6)

[0027] y = MLP(flatten(h lstm )) (7)

[0028] Among them, LSTM represents the long short-term memory network layer, h lstm It represents the intermediate feature representation through the LSTM layer; MLP represents the multi-layer perceptron layer; flatten represents flattening the feature map; and y is the final output variable.

[0029] S4. Use the Bayesian optimization method to optimize the model's hyperparameters and save computational overhead as much as possible. First, define the value space of the hyperparameters, including variables such as network structure and learning rate. In the initial training phase, a set of hyperparameters in the value space is randomly selected for training. Subsequently, the Bayesian optimization algorithm is applied to identify the best hyperparameter combination. In this process, the hyperparameter combination obtained in each iteration and its corresponding optimal model results are used to construct a probability model to predict the performance of different hyperparameter configurations, so as to select the best hyperparameter combination for subsequent training.

[0030] S5. Training the attention-enhanced multi-time-scale LSTM network model: The Adam optimizer is selected as the optimization algorithm, and the mean absolute error (MAE) is used as the loss function. Multiple iterations of training are performed on the training set, combined with gradient update and parameter optimization strategies to guide the model's learning process. Finally, the prediction effect of the model will be evaluated on the validation set to verify its generalization ability and performance.

[0031] S6. Repeat steps S4-S5 until the attention-enhanced multi-time-scale LSTM network model can obtain good prediction results on the validation set. The generalization ability of the model is further verified on the test set, and the model with good generalization ability is used for real-time prediction of product quality variables.

[0032] Beneficial effects of the present invention: The present invention proposes a modeling method for starch liquefaction process using an attention-enhanced multi-time-scale LSTM. The establishment of a prediction model is achieved by combining historical data of the starch liquefaction process with deep learning technology. The proposed attention-enhanced multi-time-scale LSTM can make up for the shortcomings of traditional LSTM in extracting local time series information, and can more deeply obtain the dynamic characteristics of data at different scales, thereby improving the prediction accuracy of the model. At the same time, the method includes data processing, model hyperparameter optimization and model training, and is easy to operate and highly automated. Finally, the method described in the present invention was successfully applied to the product quality estimation of the starch liquefaction process, and the real-time estimation of product quality was successfully realized using a data-driven model, which is of great significance to the starch liquefaction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the overall flow chart of the present invention.

[0034] Figure 2 This is a simplified process diagram for corn starch liquefaction.

[0035] Figure 3 Network structure diagram of multi-time-scale long short-term memory network for attention enhancement.

[0036] Figure 4 Fitting curves for the MLP, CNN, ALSTM, LSTM, TCN, and AMT-LSTM model effects on the liquefaction process.

[0037] Figure 5 Box plots of the prediction errors of MLP, CNN, ALSTM, LSTM, TCN, and AMT-LSTM models in the liquefaction process. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0039] In this example, a corn starch liquefaction process example is selected. The overall process is as follows: Figure 1 The specific steps are as follows:

[0040] S1. Collect data on the corn starch liquefaction process and laboratory test data, mainly including indicators such as temperature, pressure, flow, liquid level, product quality (DE value), etc. The corn starch liquefaction process is as follows: Figure 2 As shown, the main equipment includes: several heat exchangers (E), starch storage tanks (V), high-temperature steam ejectors (I), high-temperature maintenance tanks (M), transfer pumps (P), flash evaporators (F) and liquefaction reactors (R). The data collection period is from February 1, 2023 to April 30, 2023. Based on the workers' operating experience and data analysis, feature engineering was carried out, and finally 25 process variables were selected to establish a model for the target variable. The specific input features are shown in Table 1.

[0041] Table 1 Input and output variables of starch liquefaction process

[0042]

[0043] S2. Preprocess the input data

[0044] S2.1 In the production process, the timestamps of process variables in the distributed control system (DCS) are rounded to the minute level to ensure that the time record of the data is accurate to the whole minute. At the same time, the time interval between two samplings is set to 5 minutes.

[0045] S2.2 uses the 3sigma outlier detection method. First, the mean and standard deviation of the data are calculated, and then it is determined whether the data is within the acceptable range. If it exceeds the boundary, the point is considered abnormal and interpolation replacement is performed. The specific algorithm is shown in formulas (8)-(10).

[0046]

[0047] x[i]-μ<3σ (10)

[0048] Among them, x is the input data, i represents the i-th data, N represents a total of N data samples; σ is the standard deviation, and μ is the mean.

[0049] In terms of missing value processing, it is divided into two categories: the first category is individual missing values ​​caused by equipment sensor failure or outlier detection, which are filled by linear interpolation; the second category is missing data involving a period of time, which is processed by deleting the data in that period.

[0050] S2.3 In terms of data denoising, the Savitzky-Golay filter is used to reduce noise by performing polynomial smoothing within a time window. The formula is shown in (11). Data normalization is processed in a standardized manner, and formula (12) is used to convert the data into a Gaussian distribution with a mean of 0 and a variance of 1.

[0051]

[0052] Where x is the input data, the subscript smooth is the smoothed input data, the subscript std is the standardized input data; m is half the size of the input data window, b k is the weighting coefficient of the data in the window.

[0053] S2.4 uses a sliding window to expand the one-dimensional data into two-dimensional data with time features. In this case, 20 is used as the input time step and 25 is the number of input features. The matrix form of the data is shown in formula (13).

[0054]

[0055] Among them, x is the input data, and the subscripts represent the time step and the number of features respectively.

[0056] The expanded two-dimensional data matrix is ​​divided into training set, validation set and test set in a ratio of 8:1:1.

[0057] S3. The network structure obtained by improving the LSTM network in this example is as follows Figure 3 shown.

[0058] S3.1 Input the expanded time window two-dimensional data matrix into the site attention layer to calculate the weight parameters of different sites α = [α 1 ,α 2 ,…,α 25 ]. Next, multiply the obtained weight parameters by the input matrix to obtain a new two-dimensional matrix. Finally, add this new matrix to the original input two-dimensional matrix.

[0059] S3.2 uses multiple one-dimensional hole causal convolutional networks to perform convolution operations on the input two-dimensional data. This case contains 25 input data channels, each of which can be expressed as x = [x 1 ,x 2 ,…,x 20 ]. Define the convolution kernel as K = [k 1d ,k 2d ,k 3d ], where d represents the dilation rate, i.e., the interval size of the two convolution kernel parameters. The convolution kernel is used to perform a convolution operation on the data matrix, as shown in formula (14).

[0060]

[0061] Among them, x′ is the input data after convolution, and the subscript t represents the data at the tth time step.

[0062] The data matrices of multiple atrous causal convolutions are tensor-connected along the channels to obtain a two-dimensional data matrix with multi-time scale characteristics.

[0063] S3,3 uses LSTM to perform cyclic feature extraction on the input data. The input dimension of the network is determined by the sum of the number of channels of the data matrix obtained after multiple void causal convolutions. Then, the two-dimensional matrix after LSTM feature extraction is flattened into one-dimensional data, and regression is performed through the MLP layer to finally obtain the prediction result.

[0064] S4. The optimized hyperparameters of this experiment mainly include: parallel expansion causal convolution structure, LSTM network structure, MLP layer structure and model learning rate. The optimal hyperparameter combination for each step is selected by using the tree-based Bayesian optimization algorithm (TPE). The value range of each hyperparameter and the final optimization results are shown in Table 2.

[0065] Table 2 Hyperparameter optimization list

[0066]

[0067] Among them, NL dc is the number of parallel convolutional layers, NH dc is the number of hidden neurons in the convolutional layer, NL lstm is the number of LSTM hidden layers, NH lstm is the number of neurons in each layer of LSTM, NL mlp is the number of linear layers in the fully connected layer, NH mlp is the number of neurons in the fully connected layer, and Lr is the learning rate used in model training.

[0068] S5, model training process, update the model's gradient and weight parameters through back propagation. This example uses the Adam optimizer as the optimizer function and MAE as the loss function to guide the optimization direction of the model gradient and parameters. The determination coefficient (R 2 ) as evaluation indicators to evaluate the fitting effect of the model. MAE and R 2 The formula is shown in formula (18-19).

[0069]

[0070] in, y i and are the true value, predicted output value and output mean of the i-th sample respectively.

[0071] S6. By continuously repeating S4-S5, the accuracy of the model is continuously improved, and finally a data-driven model with an accuracy within the error range is obtained.

[0072] In order to compare and verify the prediction effect of the improved model of the present invention, the present invention compares the model (AMT-LSTM) with the quality prediction models based on MLP, LSTM, convolutional neural network (CNN), temporal convolutional network (TCN), and LSTM with attention (ALSTM), and lists the comparison results of the prediction indicators of different models in Table 3. 2 To compare the fitting effects of different models, the R of the AMT-LSTM model on the test set is 2 The R value of the other models on the test set is 0.9437, which is significantly higher than that of the other models on the test set. 2 .at the same time, Figure 4 The fitting curve comparisons of the predicted values ​​and true values ​​of all models are given, further emphasizing the advantage of AMT-LSTM in prediction accuracy.

[0073] In order to intuitively display the prediction differences of each model, box plots of the prediction differences of six models are drawn, such as Figure 5 As shown, it can be clearly found that the prediction error of the AMT-LSTM model is more concentrated near 0, the box is relatively small, the number of samples outside the box is also smaller, and the value of the maximum error sample is also smaller than that of other models. It can be seen that the model effect of AMT-LSTM is better.

[0074] Table 3 MAE and R of MLP, CNN, LSTM, ALSTM, TCN, and AMT-LSTM on different datasets 2

[0075]

Claims

1. A modeling method of starch liquefaction process using attention-enhanced multi-time-scale LSTM, characterized in that: The steps include: S1. Collect historical data of starch liquefaction section The quality variable of the product in the starch liquefaction process, namely the DE value, is defined as the predicted output variable of the model; by analyzing the enzymatic starch liquefaction process, auxiliary variables related to the product DE value are identified and selected as input variables, and a deep learning model between the input variables and the output variables is constructed to achieve real-time prediction of the DE value; the auxiliary variables include the process variables and operating parameters of the temperature, pressure, and liquid level of the starch configuration tank, high-temperature steam ejector, maintenance pipe, liquefaction reactor, flash tank, pump, and heat exchanger, as well as the fluid parameters of the temperature and flow rate of the starch emulsion and the fluid in the process pipeline and the physical parameters of the dry matter content; subsequently, the historical data of the above output and input variables for a period of time are collected from the historical database of the distributed control system DCS and the laboratory test; S2. Preprocess the collected historical data to obtain high-quality data sets to facilitate subsequent modeling; S2.1 To ensure that variable data from different sources are consistent in time, the data from different measurement devices are time-aligned so that each variable has the same timestamp and time interval; S2.2 detects and removes outliers in the data, and then processes missing values ​​in the data; S2.3 performs noise reduction on the data to reduce the impact of signal fluctuations in production; then, standardize the data to keep different variables at the same scale; S2.4 uses a sliding time window to expand the one-dimensional data into two-dimensional data containing time variables to obtain dynamic time features; and divides all the data into a training set, a validation set, and a test set; S3. Design of attention-enhanced multi-timescale LSTM network model structure The S3.1 model obtains the weights of different input variables on the target variable through the self-attention mechanism to enhance the feature extraction of the input, and uses residual connections to obtain the intermediate feature representation enhanced by hybrid attention. Its formulas are shown in (1)-(3); u t =tanh(w w x t +b w ) (1) α t =softmax(u t ) (2) h sa =∑ t a t x t +x t (3) Among them, x t is the input data sample, w w and b w is the weight matrix and bias of the linear transformation, tanh is the hyperbolic tangent activation function, u t is a transformation matrix similar to the input matrix; softmax normalizes the data of a certain dimension of the input sequence to obtain the attention weight α t ;h sa Intermediate feature representation enhanced for mixed attention; S3.2 uses multi-scale dilated causal convolutional layers to achieve effective extraction of temporal features in intermediate feature representations. Specifically, the model sets multiple one-dimensional dilated causal convolutional layers with different dilation factors in parallel, so that the feature extraction capability with diversified receptive fields can be obtained without increasing the number of convolution kernel parameters. After being processed by dilated causal convolutional layers with different receptive fields, the model generates multiple feature maps, which are consistent in time steps but contain different feature dimensions, as shown in formula (4). Subsequently, these feature maps are concatenated along the time dimension to form a feature map with multiple time scales, as shown in formula (5). h 1,2,...,N =dcc 1,2,...,N (h) (4) h mt =h1·h2·...·h N (5) Where · represents the tensor connection of the feature map; dcc represents the dilated causal convolution layer, and the subscripts 1, 2, ..., N represent different convolutional layers in parallel; h is the intermediate feature representation of the model, the subscripts 1, 2, ..., N represent the N intermediate feature representations after convolution of different convolutional layers, and the subscript mt represents the intermediate feature representation of multiple time scales; S3.3 passes the feature map with multiple time scales through the long short-term memory network LSTM and the multi-layer perceptron MLP in sequence; the LSTM network gradually extracts the dynamic features contained in the time series data, thereby capturing the global time series features and compressing the feature dimensions, as shown in formula (6); then, the output data of the LSTM layer is flattened from two dimensions to one dimension, and linearly transformed and nonlinearly activated by MLP to obtain the final output result, as shown in formula (7); h lstm =LSTM(h mt ) (6) y=MLP(flatten(h lstm )) (7) Among them, LSTM represents the long short-term memory network layer, h lstm represents the intermediate feature representation through the LSTM layer; MLP represents the multi-layer perceptron layer; flatten represents flattening the feature map; y is the final output variable; S4. Use the Bayesian optimization method to optimize the model's hyperparameters. First, define the value space of the hyperparameters, including the network structure and the learning rate. In the initial training phase, randomly select a set of hyperparameters in the value space for training. Then, apply the Bayesian optimization algorithm to identify the best hyperparameter combination. In the optimization process, use the hyperparameter combination obtained in each iteration and its corresponding optimal model results to build a probability model to predict the performance of different hyperparameter configurations, so as to select the best hyperparameter combination for subsequent training. S5. Train the attention-enhanced multi-time-scale LSTM network model: select the Adam optimizer as the optimization algorithm, and use the mean absolute error MAE as the loss function; conduct multiple iterations of training on the training set, combined with the strategy of gradient update and parameter optimization, to guide the learning process of the model; finally, evaluate the prediction effect of the model on the validation set to verify its generalization ability and performance; S6. Repeat steps S4-S5 until the attention-enhanced multi-time-scale LSTM network model obtains the corresponding prediction effect on the validation set; further verify the generalization ability of the model on the test set, and use the model with good generalization ability for real-time prediction of product quality variables.

Images