A dynamic soft measurement modeling method based on enhanced local perception self-attention

Abstract

The application discloses a dynamic soft measurement modeling method based on enhanced local perception self-attention, first proposes a Transformer encoder structure based on a convolution generated self-attention mechanism, uses the self-attention mechanism, can obtain the correlation between any two time points of process variables or quality variables by traversing sequences, so that global perception of the input sequence is realized. In addition, using Query and Key in the convolution generated self-attention mechanism, a new dot product self-attention calculation method is proposed, which can utilize local information to weaken the potential influence of abnormal data at a certain moment, thereby realizing local perception of the input sequence. Finally, a long short-term memory (LSTM) network is used to deeply extract time series information to obtain the final prediction result. The application fully utilizes the process dynamics and nonlinear characteristics of the sequence, and improves the prediction accuracy and robustness of the soft measurement model.

Description

Technical Field

[0001] This invention relates to a soft measurement modeling method, which has significant application value in the fields of industrial measurement and production control. Background Technology

[0002] In modern industrial processes, some variables are difficult or impossible to measure directly, yet these variables are often key factors in process control and optimization. Real-time acquisition of their information is essential for improving production efficiency and product quality. However, due to technological and safety considerations, traditional hardware sensors often cannot meet these needs. The basic idea of ​​soft sensing technology is to select easily measurable variables (auxiliary variables) based on process knowledge and data analysis, and then use a mathematical model (soft sensing model) to infer and estimate variables that are difficult to measure directly (dominant variables). Currently, soft sensing technology has been widely applied in fields such as chemical engineering, metallurgy, power, and the environment.

[0003] Currently, commonly used soft sensor models fall into three categories: mechanistic analysis-based, data-driven, and hybrid methods. Data-driven soft sensor modeling methods can adapt to complex and changing environments, requiring no physical modeling or assumptions about the process; they only need to automatically learn abstract representations using a large amount of process data. Deep learning methods, in particular, can update or optimize model parameters based on real-time data, capturing complex nonlinear relationships between process variables. Industrial processes possess multiple characteristics that are difficult to describe directly, such as high-dimensional data, nonlinear and strongly correlated relationships between measured variables, and time-varying and dynamic processes. Process dynamics and nonlinearity are two major challenges in soft sensor modeling. Process dynamics refer to the changing relationships between process variables over time, leading to model failure or decreased accuracy. Nonlinearity refers to relationships between process variables that do not conform to simple linear functions, resulting in complex or inaccurate models. Increasingly, deep learning models are being introduced into soft sensor modeling and have shown good performance. However, most of these methods are based on static assumptions; such models can only describe the instantaneous relationships between input and output variables, and their effectiveness depends on steady-state assumptions about the process. However, production processes are usually dynamic. Dynamic soft sensor models, based on static models, consider the process state information of the previous few moments and have a certain ability to describe dynamic characteristics. Meanwhile, dynamic models can incorporate time-series information of quality variables during prediction, ensuring prediction accuracy under non-steady-state conditions. Summary of the Invention

[0004] This invention relates to a dynamic soft sensor modeling method based on enhanced local awareness self-attention. This method fully utilizes the process dynamics and nonlinear characteristics of the process sequence, thereby improving the prediction accuracy and robustness of the soft sensor model. The method employs a Transformer encoder structure and utilizes its self-attention mechanism to obtain the correlation between any two time points between process variables or quality variables by traversing the sequence, thus achieving global awareness of the input sequence. The method also uses convolution to generate the Query and Key in the self-attention mechanism, proposing a novel dot-product self-attention calculation method that can utilize local information to mitigate the potential impact of abnormal data at a certain moment, thereby achieving local awareness of the input sequence. Finally, the method uses a Long Short-Term Memory (LSTM) network to deeply extract time series information to obtain the final prediction result.

[0005] This invention mainly comprises four parts: (1) preprocessing industrial process data; (2) constructing a dynamic soft measurement model to enhance local perception self-attention; (3) training the model; and (4) verifying the performance of the model proposed in this invention.

[0006] The following sections will introduce the contents of the four parts mentioned above:

[0007] (1) Determine the input and output auxiliary variables and quality variables. When training the soft sensor model, select process variables that are more relevant to the quality variables in the entire industrial situation as auxiliary variables.

[0008] (2) First, this model uses a sliding window to transform industrial process data into a supervised time series prediction problem: the sliding window slides one time step at a time along the time direction, and features are extracted by combining the sample data within the time window with the location encoding. Then, the features extracted by convolutional neural networks with different kernel sizes are used as queries, keys, and values, respectively, and enter a multi-head self-attention system to calculate attention scores. Finally, the feature vectors obtained by the encoder are fed into a long short-term memory network regressor to deeply mine temporal information, and combined with a fully connected layer to obtain the final prediction results of the quality variables.

[0009] (3) A dynamic soft sensor model based on enhanced local perception self-attention was trained using historical data. The model diagram is shown below. Figure 1 As shown.

[0010] (4) Train the soft sensor model using the training set and validate it using the test set. Input the test set data into the trained model to obtain the predicted values ​​of the key variables and compare them with the true labels. If the error between the true and predicted values ​​is small, the soft sensor model is effective.

[0011] The detailed implementation steps of this invention based on a dynamic soft sensor model that enhances local perception self-attention are as follows:

[0012] Step 1: Preprocessing industrial process data. To better learn process dynamics and conform to process kinetics, a sliding window is used to process the process data samples into a supervised time series prediction form, mining process temporal dynamics information. This means using features from previous sampling times to predict the mass variables at the current time. With a sliding time window setting of fixed size m, assuming there are n sensors, the process variables are represented as follows: Here, the time variable t is relative; the same t in different time series may represent different time points. Since the value of y at time t is unknown, the average value of the mass variable from time t-m+1 to time t is expressed as... To meet the model requirements, Y t This indicates the quality variable that needs to be predicted.

[0013]

[0014] Step 2: Construct a dynamic soft measurement model to enhance local perception self-attention. The detailed network model construction is divided into three steps, and the specific construction process is explained in steps 2.1, 2.2 and 2.3.

[0015] Step 2.1: Constructing the data encoding module. This invention relates to a soft-sensor quality variable prediction method based on a Transformer encoder. This method utilizes the self-attention mechanism of the Transformer encoder to adapt to different input data distributions and capture long-term dependencies in the input sequence. The method represents the input data in the soft sensor as a matrix composed of several numerical vectors, where each row represents data observed by multiple sensors for easily measurable variables at a certain time point. When the sequence is converted to vector form, this method adds these numerical vectors to the positional encoding to preserve temporal order information. The positional encoding can be relative or absolute, used to represent the positional information of each element in the sequence.

[0016] Where pos is the position, i is the dimension, and d is the dimension. model Let the variable dimension be denoted as 'cosine'. For each odd time step, a vector is created using a cosine function, and for each even time step, a vector is created using a sine function. These vectors are then combined with their corresponding vectors, resulting in position-encoded numerical vectors that serve as input to the first encoder block.

[0017] Step 2.2: This invention relates to an enhanced local perception self-attention module for extracting dynamic features in soft measurement processes. This module effectively addresses the problem of variable autocorrelation changing significantly with the production environment and time, as well as the problem of relationships between different time points being affected by outliers. This invention employs a similarity calculation method based on local region change trends, replacing the point-by-point value-based similarity calculation method in the scaling dot product attention layer of the native Transformer encoder. For example... Figure 2 As shown, in (a), the red dot at the intersection of the curve and the red dashed line only focuses on another single time point with an absolute value close to it, while in (b), the red dot at the intersection of the curve and the red dashed line focuses on multiple time points with similar local change trends. Therefore, the present invention can better capture the locality in time series and improve the performance of soft measurement models.

[0018] Step 2.2.1 Specifically, the derivation process of local perception self-attention is as follows:

[0019]

[0020] In the above formula The input is processed into a supervised time series prediction form, where B is the number of time windows, I is the time step, and N is the dimension of the auxiliary variables. and yes The sequence obtained after passing through two convolutional kernels of different sizes, Cov kernel size(1) With Cov kernel size(k) These represent convolutional kernels with kernel sizes of 1 and k, respectively, with stride set to 1 and padding of 0.

[0021] Step 2.2.2 will and Divide into h distinct subvectors and query vector Q. h Key value vector K h Sum vector V h The calculation formula is:

[0022]

[0023] Where h = 1, ..., H represents the number of multiple heads. The learnable parameter matrix of the model is... and Where d k =d v =d model / h. This method generates Q. h K h and V hIt can fully consider local relationships, and thus measure their similarity based on local information, rather than using a single value, which helps to improve prediction accuracy.

[0024] Step 2.2.3 This invention relates to a sequence vector output method based on scaled dot product attention, used to calculate the attention score of each sub-vector and perform normalization and weighting. The method first performs scaled dot product attention on each sub-vector to obtain its attention score. Then, it performs softmax normalization on the attention score of each row to obtain its attention weight. Finally, it multiplies the normalized attention weight by the corresponding V sub-vector and concatenates them to obtain the final sequence vector output. For each sub-vector, the attention score is calculated and scaled dot product attention is performed, and the attention score of each row is normalized using softmax. Finally, the normalized attention weight is multiplied by the corresponding V sub-vector and concatenated to obtain a series of vector outputs. The derivation formula is as follows:

[0025]

[0026]

[0027] Step 2.2.4 This invention relates to a multi-head self-attention module based on feedforward neural network layers and residual connections, used to further transform and optimize the vectors output by the multi-head self-attention layers. The module first performs two linear transformations and one ReLU activation function operation on the vectors output by each multi-head self-attention layer to obtain the output of the feedforward neural network layer. Then, it performs residual connections and layerNorm operations on the input and output of each sub-layer to preserve the original information and improve gradient propagation efficiency. The specific steps are as follows:

[0028] Given the vector output by the multi-head self-attention layer, where:

[0029] For each vector, two linear transformations and one ReLU activation function operation are performed to obtain the output of the feedforward neural network layer, where W1 and W2 are the model's learnable parameter matrices. Finally, after each sub-layer, to better preserve the original information and improve gradient propagation efficiency, the input and output of the sub-layer are added through residual connections, and each vector is scaled and translated using layerNorm to ensure they have the same mean and variance across different dimensions. This invention, by using feedforward neural network layers and residual connections, effectively transforms and optimizes the vectors output by multi-head self-attention layers, improving the performance and efficiency of the multi-head self-attention module.

[0030] The derived formula is as follows:

[0031] FFN(x) = max(0, xW1+b1)W2+b2

[0032] Eout 1 =LayerNorm(x+MHA(x))

[0033] Eout 2 =LayerNorm(Eout) 1 +FeedForward(Eout 1 ))

[0034] Step 3: Establish the regression model. In this invention, the encoder output can be used as the feature vector for the soft sensor quality variable prediction task, and then used in the subsequent prediction model. The Transformer encoder output is the intermediate layer between the input sequence and the output sequence in the soft sensor quality variable prediction task. This patent uses a Long Short-Term Memory (LSTM) network and a fully connected layer as the regressor of the soft sensor model because LSTM can process sequence data and capture dynamic features, while the fully connected layer can perform nonlinear transformations, increasing the model's ability to extract deeper time series information. Assume x t Y is the input vector at time t. t If h is the output vector at time t, then the formula for the LSTM and fully connected layer as the regressor of the soft measurement model can be expressed as: t ,c t =LSTM(x t ,h t-1 ,c t-1 ), Y t =FC(h t ). Here, LSTM stands for Long Short-Term Memory Network, which has one hidden state h. t and a cell state c t These are all vectors that are updated over time. FC stands for fully connected layer, which has a learnable weight matrix and bias vector.

[0035] Step 4: Train the dynamic soft sensor model based on enhanced local perception self-attention. After the soft sensor model is built, the training data processed in Step 1 is input into the soft sensor model, and the backpropagation algorithm (BP) stochastic gradient descent is used to update the weights until the gradient converges. The weights in the network at this point are the optimal weights in the soft sensor model of this invention. The weight update formula is as follows:

[0036]

[0037] (Lr is the learning rate of the neural network). Gradient descent is performed based on the loss function (Loss) to find suitable weight parameters. The loss function is as follows:

[0038]

[0039] Where: Y real Y is the label for the data. pre This is the output value of the model.

[0040] Step 5: Model Validation. After training the entire soft sensor model, save the parameters of each part of the model. Perform forward propagation of the pre-processed test data through the model to obtain the predicted value y. pre Through indicators such as RMSE, MSE, MAE, and R... 2 The performance of this invention is evaluated. The formulas are defined as follows:

[0041]

[0042]

[0043]

[0044] In the formula Y real Y pre These are the label value and the predicted output value, respectively. A smaller RMSE generally indicates better prediction performance.

[0045]

[0046] Coefficient of determination R 2 Reliability of the reaction model, Y mean R represents the mean of the output values ​​of the test data. 2 The larger the value, the better the predictive performance of the soft measurement model. Attached Figure Description

[0047] Figure 1 This is a flowchart of the dynamic soft sensor model based on enhanced local perception self-attention in this invention.

[0048] Figure 2 This is a detailed structural diagram of the enhanced local perception self-attention module in this invention.

[0049] Figure 3 This is a flowchart of the sulfur recovery process in an embodiment of the present invention.

[0050] Figure 4 This is a comparison chart of the soft-sensor model for predicting hydrogen sulfide concentration in sulfur recovery in an embodiment of the present invention with other modeling methods.

[0051] Figure 5 This is a distribution diagram of the prediction results of the soft sensor model for hydrogen sulfide concentration prediction in sulfur recovery in an embodiment of the present invention, compared with the prediction results of other soft sensor models. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and examples.

[0053] The soft-sensor modeling method of this invention is applied to the air supply control of a sulfur recovery device. Based on the combustion of acidic gases, the air supply in the combustion furnace is adjusted to remove pollutants and reduce the environmental pollution caused by exhaust gas emissions. This is based on a simplified flow chart of the sulfur recovery device (e.g., ...). Figure 3 As shown, the combustion reaction process of acidic gas with air and the effect of air supply on the concentrations of SO2 and H2S in the exhaust gas were analyzed. The acidic gas was combusted with air in combustion furnace F101, producing SO2, H2O, CO2, and other products. These products were further converted in water condenser E101, catalytic reactors R101 and R102, and condensers E102 and E103 to form sulfur products. The combustion of the acidic gas was accurately measured based on the concentrations of SO2 and H2S in the exhaust gas, and the air supply to combustion furnace F101 was adjusted according to the measurement results.

[0054] This invention utilizes measured data and precise measurement methods, applied to the collection of measured data from an Italian oil refinery. Experiments demonstrate that this invention can effectively control the air supply in a sulfur recovery unit, improving its efficiency and environmental friendliness. Five relevant auxiliary variables under this production environment were selected, with the hydrogen sulfide concentration as the target for prediction. After selecting the auxiliary variables, the next step was to establish a dynamic soft sensor model based on enhanced local perception self-attention. This model was trained by collecting and preprocessing 10,081 historical data points.

[0055] The first 8000 test data points were input into the trained soft sensor model to obtain predicted values. These predicted values ​​were compared with the labeled values. Evaluation metrics on the test set were MAR = 0.00789, MSE = 0.0143, RMSE = 0.0143, and R... 2 =0.947. This invention was also compared with traditional LSTM, CNN+LSTM, and Transformer methods. The comparison results are as follows: Figure 4 As shown, this method improves predictive ability in model building compared to traditional methods. Figure 5 The distribution of prediction results for this soft sensor model and comparative soft sensor models is shown in the graph, demonstrating that its error performance is superior to other soft sensor models. In summary, the soft sensor model provided by this invention has good application potential in predicting hydrogen sulfide concentration in sulfur recovery devices.

Claims

1. A dynamic soft sensor modeling method based on enhanced local perception self-attention, characterized in that: Includes the following steps: Step 1: Preprocess the industrial process data. With a fixed sliding time window of size m, assume there are n sensors providing the process variables. Here, the time variable t is relative; the same t in different time series may represent different time points. Since the value of y at time t is unknown, the average value of the mass variable between time t-m+1 and time t is expressed as... To meet the needs of the model, This indicates the quality variables that need to be predicted; Step 2: Construct a dynamic soft sensor model to enhance local perception self-attention. The detailed network model construction is divided into three steps, and the specific construction process is explained in steps 2.1, 2.2 and 2.

3. Step 2.1: Construct a data encoding module to represent the input data in soft measurement as a matrix composed of several numerical vectors, where each row represents the data observed by multiple easily measurable variables at a certain time point. When the sequence is converted into a vector type, these numerical vectors are added to the position code to preserve the time sequence information. The position code is relative or absolute and is used to represent the position information of each element in the sequence. , Where pos is the position, i is the dimension, and d is the dimension. model For the variable dimension, a vector is created using a cosine function for each odd time step and a vector is created using a sine function for each even time step. These vectors are then combined with their corresponding vectors. This position-coded numerical vector is then used as the input to the first encoder block. Step 2.2: An enhanced local perception self-attention module is adopted for the extraction of dynamic features in the soft measurement process. A similarity calculation method based on the local region change trend is used instead of the point-by-point value-based similarity calculation method in the scaling dot product attention layer of the native Transformer encoder. This can better capture the locality in the time series and improve the performance of the soft measurement model. Step 2.2.1 The derivation process of local perception self-attention is as follows: ， In the above formula The input is processed into a supervised time series prediction form, where B is the number of time windows, I is the time step, and N is the dimension of the auxiliary variables. and yes The sequence obtained after passing through two convolution kernels of different sizes and These represent convolutional kernels with kernel sizes of 1 and k, respectively, with stride set to 1 and padding to 0; Step 2.2.2 will and Cut into Query vector with different subvectors Key-value vectors Sum value vector The calculation formula is: , In the above formula The number of long positions; The model can learn the following parameter matrix: and ,in This method generates , and It can fully consider local relationships, and thus measure their similarity based on local information, rather than using a single value, which is beneficial to improving prediction accuracy; Step 2.2.3 employs a sequence vector output method based on scaled dot product attention, calculating the attention score of each sub-vector and performing normalization and weighting. First, scaled dot product attention is applied to each sub-vector to obtain its attention score. Then, the attention score of each row is normalized using softmax to obtain its attention weight. Finally, the normalized attention weights are multiplied by the corresponding V sub-vector and concatenated to obtain the final sequence vector output. The derivation formula is as follows: Step 2.2.4 employs a multi-head self-attention module based on feedforward neural network layers and residual connections to further transform and optimize the vectors output by the multi-head self-attention layers. The module first performs two linear transformations and one ReLU activation function operation on the vectors output by each multi-head self-attention layer to obtain the output of the feedforward neural network layer. Then, it performs residual connections and layerNorm operations on the input and output of each sub-layer to preserve the original information and improve gradient propagation efficiency. The specific steps are as follows: Given the output vector of a multi-head self-attention layer, where for each vector, two linear transformations and one ReLU activation function operation are performed to obtain the output of the feedforward neural network layer, where... and The parameter matrix is ​​the model's learnable parameter matrix. Finally, after each sub-layer, in order to better preserve the original information and improve gradient propagation efficiency, the input and output of the sub-layer are added through residual connections, and each vector is scaled and translated using layerNorm. By using feedforward neural network layers and residual connections, the vectors output by the multi-head self-attention layer can be effectively transformed and optimized, improving the performance and efficiency of the multi-head self-attention module. The derivation formula is as follows: Step 3: Establish a regression model; the encoder output serves as the feature vector for the soft sensor quality variable prediction task, used in the subsequent prediction model. The Long Short-Term Memory (LSTM) network and fully connected layers are used as the regressor for the soft sensor model. Assume... It is the input vector at time t. Let be the output vector at time t. Then, the formula for LSTM and fully connected layers as regressors in a soft measurement model is expressed as: , LSTM has a hidden state. and a cell state FC stands for fully connected layer, which has a learnable weight matrix and bias vector; Step 4: Train a dynamic soft sensor model based on enhanced local perception self-attention. After the soft sensor model is built, input the training data processed in Step 1 into the soft sensor model and use the BP algorithm stochastic gradient descent to update the weights until the gradient converges. The weights in the network at this point are the most suitable weights in the soft sensor model, and their weight update formula is as follows: Lr is the learning rate of the neural network. Gradient descent is performed based on the loss function Loss to find suitable weight parameters; the loss function is as follows: in: Labels for the data, The output value of the model; Step 5: Model Validation. After training the entire soft sensor model, save the parameters of each part of the model; then, perform forward propagation of the pre-processed test data through the model to obtain the predicted values. ; through indicators RMSE, MSE, MAE, R 2 Evaluate the performance of the model.

Images