Data filling end-to-end soft measurement modeling method fusing attention mechanism

Abstract

The invention relates to soft measurement modeling in the process industry, and discloses a data filling end-to-end soft measurement modeling method fusing an attention mechanism, which comprises the following steps of: firstly, constructing complex encoder layer-by-layer learning for better capturing dynamic nature in process data; specifically, an attention mechanism is embedded to capture global dependence on long-distance time and feature dimensions, data mapping vectors are expressed, and a long short-term memory network LSTM focuses on short-distance time dependence on the expression to enhance local information expression; the vector representation is efficiently restored to an original feature space through linear decoding; secondly, the soft measurement module can be flexibly replaced according to specific requirements, and finally, in order to make a filling result more accord with downstream soft measurement modeling requirements and keep filling and prediction target consistency, missing filling and soft measurement are directly integrated into an end-to-end framework for joint optimization, and the performance of the model in the industrial process is jointly improved.

Description

Technical Field

[0001] This invention relates to soft measurement modeling in process industries, and more specifically to a data-filling end-to-end soft measurement modeling method that incorporates an attention mechanism. Background Technology

[0002] Soft sensing uses readily measurable, relevant process variables to build effective models for predicting target variables, providing a reference for the safety and reliability of production in process industries. Increasingly complex industrial systems have expanded the available data for soft sensing, with process data containing a wealth of production patterns and dynamic process characteristics. Compared to building mathematical models based on complex physical and chemical process principles, data-driven modeling methods have found wider application in the field of soft sensing.

[0003] In actual industrial processes, factors such as sensor failure, signal interruption, and harsh production environments lead to missing historical operational data, hindering further software testing and modeling. In multivariate process data, simple deletion can result in the loss of a large number of samples, and outlier removal in practical applications may further exacerbate the data gaps. Missing industrial data is common in process variables, and this patent addresses this issue. In recent years, deep learning methods have driven research on missing data imputation, demonstrating superior applicability in high-dimensional data scenarios through powerful nonlinear representations and time-dependent capture.

[0004] However, soft sensing in industrial processes with missing data typically presents two problems: 1) The processes are complex, nonlinear, and highly dynamic; ignoring these characteristics can lead to biases in the missing data modeling results; 2) Missing data modeling is often done independently, ignoring the supervisory information contained in the dynamic labels of the soft sensing, thus limiting targeted missing data modeling for soft sensing. The nonlinear and dynamic characteristics of the process affect both reasonable missing data modeling and the effective utilization of process variables by soft sensing. Therefore, there is a coupling between dynamic modeling for missing data modeling and the effective utilization of dynamic labels in downstream tasks; they are not independent.

[0005] To address this, a data-filling end-to-end soft measurement modeling method incorporating an attention mechanism is proposed for incomplete data soft measurement modeling of industrial processes. Summary of the Invention

[0006] To address the issue of missing data in highly dynamic industrial process data and its impact on the accuracy of soft sensor modeling, this invention proposes an end-to-end soft sensor modeling method that integrates an attention mechanism. First, a fusion of attention and long short-term memory network structures is used to dynamically impute missing process variables, restoring complete and reasonable process data. Then, the imputed data is input into the soft sensor model to learn the complex mapping relationship between process variables and key quality indicators. Finally, an end-to-end joint optimization mechanism improves both the effectiveness of missing data imputation and the accuracy of soft sensor prediction.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] An end-to-end soft measurement modeling method for data imputation incorporating an attention mechanism, the method comprising the following steps:

[0009] (1) Obtain the original sample data.

[0010] (2) Data preprocessing: A subset of the data is extracted using a sliding time window as a new sample. To ensure the consistency of the collected process data, the data is standardized so that the variables can be arranged on a common scale.

[0011] (3) Constructing the imputation model: Construct an encoder that integrates attention mechanism and LSTM to capture the latent representation of incomplete data, linearly decode to recover missing items, randomly mask the observation data as self-supervised label values, and use the distance between the masked observation values ​​and the corresponding position imputation output as a quantitative indicator of imputation error.

[0012] (4) End-to-end framework for joint soft measurement: Integrate the filling model and the soft measurement model into an end-to-end framework for joint optimization to improve modeling performance in industrial processes.

[0013] (5) Model training and performance evaluation: Soft measurement evaluation metrics, using root mean square error (RMSE) and mean absolute error (MAE).

[0014] Furthermore, the process of step (2) is as follows:

[0015] Step 2.1: Divide the sample

[0016] Each sample is obtained by dividing the data into segments with a fixed window size, and contains T time steps and N feature variables, represented as follows: Represented as a single time step vector, where each X has a binary mask M∈{0,1}. T×N This indicates the validity of the observations at the corresponding location.

[0017] Step 2.2: Sample Normalization

[0018] x in each X t First, subtract the mean μ of each variable, and then divide by the standard deviation σ of the training samples to achieve sample standardization. X below is a standardized sequence sample.

[0019] Furthermore, the process of step (3) is as follows:

[0020] Step 3.1: Embedding Processing

[0021] A linear transformation maps the feature dimension N to a D-dimensional spatial vector. To preserve the ability to perceive sequence position information, the embedded matrix X... (0) The calculation formula is as follows:

[0022] X (0) =XW p +b p +E pos (1)

[0023] In the formula These are learnable parameters. It is the positional encoding information of the sequence. It is the input sequence for the subsequent encoder.

[0024] Step 3.2: Construct the filling encoder

[0025] Subsequently, the embedded representation is fed into the encoder, where parallel multi-head attention is used to calculate the correlations between positions to capture global dependencies. The attention module contains three learnable weight matrices, which map the input to T×d dimensions respectively. k The query Q has a dimension of T×d. k The key K and dimension T×d v The value of V. Typically, d k =d v Here, we uniformly set it to d. The specific linear mapping is as follows:

[0026]

[0027] In the formula as well as Where h∈{1,2,…,

[0028] H}, where H is the number of multi-head attention heads. The scaled dot product of Q and K is calculated, and after applying softmax(.), it is multiplied by V to obtain the attention mechanism attn(.,.,.), where softmax(.) normalizes the similarity scores of each query and all key-value pairs to a probability distribution. The concatenated outputs of each attention head are projected onto the multi-head attention result McGee(.,.,.): McGee(Q,K,V)=Concat(head1,…,head) h …,head H W O

[0029]

[0030] In the formula, Concat(.) is tensor concatenation. It is a learnable matrix. The softmax(.) function is as follows, where exp(.) represents the natural exponential function and ||.||1 represents the first norm.

[0031]

[0032] Multi-head attention mechanisms focus on the input sample sequence from different subspaces, decomposing the global information of the sequence into a superposition of multiple micro-perspectives. After obtaining the globally dependent features, they are then compared with X. (0) Add them together and normalize the results to achieve stable training, denoted as . The input is a two-layer LSTM, which further models the temporal dependency features in the sequence, and then batch-normalizes the feature distribution again to provide a more stable input representation for downstream processes.

[0033]

[0034] In the formula It is X (0) After the output is modified from a single encoding block, Norm(.) represents batch normalization, and Dropout(.) represents randomly dropping a small number of neurons to alleviate network overfitting. The specific definition of lstm(.) is as follows:

[0035]

[0036] In the formula, σ(.) represents a fully connected layer with an activation function, and tanh(.) represents the hyperbolic tangent activation function. t p represents the input vector at the current time step. t It's the input gate, q t It's the Gate of Oblivion, g t It is the candidate cell state, r t It's an output gate, s t It's about cell state updates, z t This is output in a hidden state.t-1 and z t-1 This represents the cell state update and hidden layer state from the previous step; both are typically initialized to 0. W op W oq W og W os W represents the weight matrix input to each gate. zp W zq W zg W zs b represents the weight matrix from the hidden state to each gate. op b oq b og b os This represents the bias of the input to each gate, b zp b zq b zg b zs This represents the bias from the hidden state to each gate. LSTM enhances local information by using the vector representation obtained from multi-head attention processing through state propagation.

[0037] The above content describes the implementation of a single encoder block. The residual structure inside the encoder supports hierarchical stacking. The encoder structure consisting of H attention heads and two layers of LSTM residual connections is denoted as MAL(.). The output of the current encoder block is used as the input of the next encoder block, as shown in the following formula:

[0038] X (c) =MAL(X (c-1) (7)

[0039] In the formula, c∈{1,2,…,C}, This represents the output of the c-th encoder block. The entire encoder consists of C blocks stacked sequentially, with its initial input X. (0) That is, the representation after position embedding.

[0040] Step 3.3: Linear Decoding of the Model

[0041] Encoders are typically designed with complex, multi-layered structures to extract deep features from the original sequence, while decoders maintain a simple design to avoid introducing redundant structures that could lead to overfitting. Therefore, this invention uses linear decoding to map the latent representation back to the real space to obtain the reconstruction result. The encoder's output sequence X... (C) After linear decoding, the prediction of the missing data is obtained:

[0042] X' = ​​X (C) W o +b o (8)

[0043] In the formula, and These are learnable parameters. That is, the output sequence after the model is filled.

[0044] Furthermore, the process of step (4) is as follows:

[0045] Step 4.1: Process the filled reconstruction output

[0046] In the end-to-end framework completion model, a self-supervised approach is used to reconstruct and learn from the observations. Valid values ​​with a proportion of α are randomly selected as self-supervised training labels. The corresponding mask is M label ∈{0,1} T×N The remaining valid observations that were not selected as labels This serves as information accessible to the filler module. (In the mask label M) label At the position = 1, the output X' of the imputation model is constructed with the effective value to form the imputation loss:

[0047] Loss1=MSE(X'⊙M label ,X label (9)

[0048] In the formula, ⊙ represents the Hadamard product, MSE(.) represents the root mean square error (MSE) between the two, and Loss1 is the model loss during the missing data imputation stage.

[0049] X' reconstructs the data for the entire sample. To reduce the error compared to the real data, it is processed as follows before being input into the downstream soft measurement task:

[0050] X filled =X'⊙(1-M)+X⊙M (10)

[0051] In the formula X filled Retain valid observation data for X and data for missing X' positions.

[0052] Step 4.2: Soft Measurement Modeling and Prediction

[0053] H = lstm L (X filled (11)

[0054] In the formula, lstm L (.) indicates that there is an L-level lstm(.) connection. Let U represent the hidden states of the last LSTM layer across all time steps, where U is the dimension of the last LSTM hidden layer. The attention mechanism allows the model to give different levels of attention to each time step, adaptively learning the contribution of each time step and achieving soft measurement for single-point prediction via windowing.

[0055]

[0056] In the formula It is the attention weight vector after softmax(.). This is the attention linear transformation learnable matrix, where 1 indicates that the soft measurement target variable has 1 type. These are learnable weights, where 1 represents a single-point prediction. The output of the soft measurement model, y', is used to construct the model loss using the true label y:

[0057] Loss2 = MSE(y,y') (13)

[0058] Step 4.3: End-to-end joint unified optimization objective

[0059] To improve the model and further fill in the gaps by incorporating downstream soft measurement tasks, this invention jointly trains two independent modules to achieve an end-to-end framework from incomplete process variables to the soft measurement target. Combining formulas 9 and 13, the total loss during end-to-end framework training is constructed as follows:

[0060] Loss=λ1Loss1+λ2Loss2 (14)

[0061] In the formula, λ1 and λ2 are used to assign weights to the losses of the two sub-models, and the gradient of the total loss function is backpropagated to guide the update of the model weights.

[0062] Furthermore, the process of step (5) is as follows:

[0063] Step 5.1: Soft Measurement Prediction Assessment

[0064] The root mean square error (RMSE) and mean absolute error (MAE) are used as evaluation indicators for soft measurement predictions, and their formulas are defined as follows:

[0065]

[0066] In the formula y t and y t ' represents the soft measurement label and the predicted value, respectively, and T represents the prediction time step length.

[0067] The design concept of this invention is as follows:

[0068] First, to better capture the dynamics of process data, a complex encoder is constructed for layer-by-layer learning. Specifically, an attention mechanism is embedded to capture global dependencies over long-range time and feature dimensions, mapping the data to a vector representation. A Long Short-Term Memory (LSTM) network focuses on short-range temporal dependencies on this representation to enhance local information representation. Then, linear decoding efficiently restores the vector representation to the original feature space. Second, the soft measurement module can be flexibly replaced according to specific needs; this invention uses an LSTM-related structure as an example. Finally, to ensure that the missing data filling results better match the downstream soft measurement modeling requirements and maintain consistency between the filling and prediction goals, missing data filling and soft measurement are directly integrated into an end-to-end framework for joint optimization, jointly improving the model's performance in industrial processes.

[0069] The beneficial effects of this invention are as follows: (1) Enhance the dynamic feature capture capability of the data missing filling stage: Introducing the attention mechanism and the collaborative structure of LSTM in the encoding stage of the filling module can simultaneously capture long-term dependence and short-term change features in the time dimension, and more accurately reflect the dynamic correlation characteristics of process variables in the cement clinker fermentation process than the traditional static modeling method; and achieve efficient restoration of missing process data through linear decoding. (2) Achieve joint optimization of missing filling and soft measurement: The missing filling module and the soft measurement module are trained together under a unified framework, so that the filling results can actively adapt to the needs of the downstream f-CaO prediction task, achieve target consistency optimization, and improve the accuracy of the final soft measurement target. Compared with the traditional step-by-step method of "filling first and then modeling", this invention can simultaneously optimize the performance of data recovery and target prediction, and significantly reduce error propagation and model offset problems. (3) Possess good versatility and scalability: The end-to-end joint modeling framework of this invention has modular features, and the soft measurement part can be flexibly adjusted according to the specific industrial process scenario to achieve prediction of key quality indicators with missing process data. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the structure of the filling sub-model of the present invention;

[0071] Figure 2 This is a flowchart of the method for joint optimization of filling and soft measurement according to the present invention;

[0072] Figure 3 This is a comparison chart of soft measurement predictions of the method of this invention and existing methods on a test set in a cement case.

[0073] Figure 4 This is a box plot of the squared error of the predicted target using the method of this invention and existing methods in a cement case. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0075] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0076] The present invention will now be further described with reference to the accompanying drawings.

[0077] Reference Figures 1-4 An end-to-end soft measurement modeling method for data imputation incorporating an attention mechanism, the method comprising the following steps:

[0078] (1) Obtaining raw sample data and preprocessing

[0079] Step 1.1: Dataset Acquisition and Dataset Partitioning

[0080] The effectiveness of the proposed model was verified using a case study of the clinker production process at a large cement plant in China. Free calcium oxide (f-CaO) is the unreacted residue from the calcination of cement clinker, and its content directly reflects clinker quality and process stability. Up to 50 types of variable data are available in the cement clinker production chain. After discussions with engineers, the following 15 process variables were selected to construct the soft measurement model: primary outlet gas analysis (carbon dioxide), primary outlet gas analysis (nitrogen dioxide), high-temperature fan frequency conversion, high-temperature fan current, decomposer outlet temperature, flue gas temperature, kiln current, kiln speed, tertiary air temperature, kiln head hood negative pressure, secondary air temperature, kiln head dust collector inlet temperature, kiln tail coal feed rate, and kiln head coal feed rate.

[0081] In the cement clinker production case, the sampling period for process variables differs from that for the target variable f-CaO. The sampling period for process variables is 3 minutes, while f-CaO is collected and recorded every hour using offline chemical analysis. To reconcile this difference, the process variables are divided into non-overlapping time windows spanning 1 hour in this case. The dataset used spans nearly three months, with 2088 f-CaO samples. These samples are divided into training, validation, and test sets in an approximate 3:1:1 ratio. Statistical analysis shows a 37% missing data rate in the process data, with the missing data type being random.

[0082] Step 1.2: Data preprocessing.

[0083] The samples are divided into groups using a fixed-length time window. Each sample contains T time steps and N feature variables, denoted as follows: Represented as a single time step vector, where each X has a binary mask M∈{0,1}. T×N This indicates the validity of the observations at the corresponding locations, with x in each sample X representing the validity of the observations. t First, subtract the mean μ of each variable, then divide by the standard deviation σ of the training samples, where the mean μ and standard deviation σ are uniformly scaled according to the corresponding scale of the training set. X below represents standardized sequence samples.

[0084] (2) Constructing the filling model

[0085] Step 2.1: Random mask selection of self-supervised labels.

[0086] Randomly select effective values ​​with a proportion of α as self-supervised training labels. The corresponding mask is M label ∈{0,1} T×N The remaining valid observations that were not selected as labels This serves as information accessible to the filler module.

[0087] Step 2.2: Filling in the model embedding process

[0088] A linear transformation maps the feature dimension N to a D-dimensional spatial vector. To preserve the ability to perceive sequence position information, the embedded matrix X... (0) The calculation formula is as follows:

[0089] X (0) =XW p +b p +E pos (16)

[0090] In the formula These are learnable parameters. It is the positional encoding information of the sequence. It is the input sequence for the subsequent encoder.

[0091] Step 2.3: Fill in the model encoder to capture complex features.

[0092] The embedded representation is fed into the encoder, where parallel multi-head attention is used to compute the correlations between positions to capture global dependencies. The attention module contains three learnable weight matrices, which map the input to T×d dimensions respectively. k The query Q has a dimension of T×d. k The key K and dimension T×d v The value of V. Typically, d k =d vHere, we uniformly set it to d. The specific linear mapping is as follows:

[0093]

[0094] In the formula as well as Where h∈{1,2,…,

[0095] H}, where H is the number of multi-head attention heads. The scaled dot product of Q and K is calculated, and after applying softmax(.), it is multiplied by V to obtain the attention mechanism attn(.,.,.), where softmax(.) normalizes the similarity scores of each query and all key-value pairs to a probability distribution. The concatenated outputs of each attention head are projected onto the multi-head attention result McGee(.,.,.): McGee(Q,K,V)=Concat(head1,…,head) h …,head H W O

[0096]

[0097] In the formula, Concat(.) is tensor concatenation. It is a learnable matrix. The softmax(.) function is as follows, where exp(.) represents the natural exponential function and ||.||1 represents the first norm.

[0098]

[0099] Multi-head attention mechanisms focus on the input sample sequence from different subspaces, decomposing the global information of the sequence into a superposition of multiple micro-perspectives. After obtaining the globally dependent features, they are then compared with X. (0) Add them together and normalize the results to achieve stable training, denoted as . The input is a two-layer LSTM, which further models the temporal dependency features in the sequence, and then batch-normalizes the feature distribution again to provide a more stable input representation for downstream processes.

[0100]

[0101] In the formula It is X (0) After the output is modified from a single encoding block, Norm(.) represents batch normalization, and Dropout(.) represents randomly dropping a small number of neurons to alleviate network overfitting. The specific definition of lstm(.) is as follows:

[0102]

[0103] In the formula, σ(.) represents a fully connected layer with an activation function, and tanh(.) represents the hyperbolic tangent activation function.t p represents the input vector at the current time step. t It's the input gate, q t It's the Gate of Oblivion, g t It is the candidate cell state, r t It's an output gate, s t It's about cell state updates, z t This is output in a hidden state. t-1 and z t-1 This represents the cell state update and hidden layer state from the previous step; both are typically initialized to 0. W op W oq W og W os W represents the weight matrix input to each gate. zp W zq W zg W zs b represents the weight matrix from the hidden state to each gate. op b oq b og b os This represents the bias of the input to each gate, b zp b zq b zg b zs This represents the bias from the hidden state to each gate. LSTM enhances local information by using the vector representation obtained from multi-head attention processing through state propagation.

[0104] The above content describes the implementation of a single encoder block. The residual structure inside the encoder supports hierarchical stacking. The encoder structure consisting of H attention heads and two layers of LSTM residual connections is denoted as MAL(.). The output of the current encoder block is used as the input of the next encoder block, as shown in the following formula:

[0105] X (c) =MAL(X (c-1) ) (twenty two)

[0106] In the formula, c∈{1,2,…,C}, This represents the output of the c-th encoder block. The entire encoder consists of C blocks stacked sequentially, with its initial input X. (0) That is, the representation after position embedding.

[0107] Step 2.4: Fill in the sub-model with linear decoding.

[0108] Encoders are typically designed with complex, multi-layered structures to extract deep features from the original sequence, while decoders maintain a simple design to avoid introducing redundant structures that could lead to overfitting. Therefore, this invention uses linear decoding to map the latent representation back to the real space to obtain the reconstruction result. The encoder's output sequence X... (C) After linear decoding, the prediction of the missing data is obtained:

[0109] X' = ​​X (C) W o +b o (twenty three)

[0110] In the formula, and These are learnable parameters. That is, the output sequence after the model is filled.

[0111] The loss is constructed by filling in the model's output X' and the effective values ​​at the effective locations of the self-supervised labels, as shown in the following formula:

[0112] Loss1=MSE(X'⊙M label ,X label ) (twenty four)

[0113] In the formula, MSE(.) represents the root mean square error (MSE) between the two, and Loss1 is the model loss during the missing data imputation stage.

[0114] (3) End-to-end framework for training joint soft measurement submodels

[0115] Step 3.1: Process the infill reconstruction data.

[0116] X' reconstructs the data for the entire sample. To reduce the error compared to the real data, it is processed as follows before being input into the downstream soft measurement task:

[0117] X filled =X'⊙(1-M)+X⊙M (25)

[0118] In the formula X filled Retain valid observation data for X and data for missing X' positions.

[0119] Step 3.2: Soft measurement modeling and prediction.

[0120] H = lstm L (X filled (26)

[0121] In the formula, lstm L (.) indicates that there is an L-level lstm(.) connection. Let U represent the hidden states of the last LSTM layer across all time steps, where U is the dimension of the last LSTM hidden layer. The attention mechanism allows the model to give different levels of attention to each time step, adaptively learning the contribution of each time step and achieving soft measurement for single-point prediction via windowing.

[0122]

[0123] In the formula It is the attention weight vector after softmax(.). This is the attention linear transformation learnable matrix, where 1 indicates that the soft measurement target variable has 1 type. These are learnable weights, where 1 represents a single-point prediction. The output of the soft measurement model, y', is used to construct the model loss using the true label y:

[0124] Loss2 = MSE(y,y') (28)

[0125] Step 3.3: End-to-end joint unified optimization objective.

[0126] In order to fill in the gaps by combining the improved model with the downstream soft measurement task, this invention jointly trains two independent modules to realize the end-to-end framework from incomplete process variables to soft measurement targets. Combining formulas (24) and (28), the total loss during end-to-end framework training is constructed as follows:

[0127] Loss = λ1Loss1 + λ2Loss2 (29)

[0128] In the formula, λ1 and λ2 are used to assign weights to the losses of the two sub-models, and the gradient of the total loss function is backpropagated to guide the update of the model weights.

[0129] (4) Model training and performance evaluation

[0130] Step 4.1: Train the model.

[0131] The hyperparameters for model training were set as follows: a fixed time window was used to divide non-overlapping samples, with each sample containing T time steps and N feature variables. The number of time steps was 20, and there were a total of 15 feature variables. The random masking rate for self-supervised labels was set to 10%. The encoder in the imputation sub-model had 3 layers, and the LSTM in the soft measurement sub-model had 3 layers. The learning rate was uniformly set to 0.001, and the learning rate was automatically adjusted using Adam, which usually resulted in faster and more stable model convergence. In early experiments, we tried to introduce different weight configurations, such as 1:2 and 2:1, but the differences were not significant under different missing rate scenarios. Therefore, following the principle of simplicity, we adopted a simple form with equal weights, i.e., λ1 = λ2 = 1.

[0132] Step 4.2: Performance evaluation of the target soft measurement task.

[0133] To verify the performance of the missing data filling framework proposed in this invention, the following four filling methods were selected for comparison, including PCA. [7] BRITS [8] GP-VAE [9] and PFIDM

[10] Based on the characteristics of the methods, the above methods can be classified into three categories: (1) statistical imputation, such as PCA; (2) machine learning-based independent imputation modeling, such as BRITS and GP-VAE, which do not consider the impact of subsequent tasks; and (3) downstream task-oriented imputation modeling method PFIDM, which introduces downstream soft measurement to achieve task-aware imputation optimization.

[0134] The root mean square error (RMSE) and mean absolute error (MAE) are used as evaluation indicators for soft measurement predictions, and their formulas are defined as follows:

[0135]

[0136] In the formula y t and y t ' represents the soft measurement label and the predicted value, respectively, and T represents the prediction time step length.

[0137] The soft measurement performance of the model after imputation under different missing rates is shown in Table 1. The traditional statistical method PCA has the largest error, indicating that simple linear dimensionality reduction imputation is insufficient to capture the dynamic relationships between complex process variables. Secondly, the independent modeling deep learning methods BRITS and GP-VAE show varying degrees of improvement compared to PCA, indirectly demonstrating the effectiveness of deep learning in introducing time-dependent modeling. The proposed method, which incorporates downstream task information (PFIDM), outperforms the independent imputation modeling method in predicting performance. Compared to PFIDM, the proposed method further reduces RMSE by approximately 17.4% and MAE by 19.8%.

[0138] Table 1. Soft Sensor Tests (RMSE and MAE) for Each Model in Cement Clinker Production Case Study

[0139]

[0140] also, Figure 4Box plots of the squared errors of soft measurement predictions after data imputation using different methods are presented in a cement case study. The method of this invention has the smallest box range, and the triangle representing the mean largely coincides with the median, indicating a relatively balanced distribution of squared errors. Although the squared errors of the method of this invention have some outliers, the outlier identification in the box plot is based on the statistical distribution, not the squared errors themselves. Overall, the method of this invention achieves better performance in the cement f-Cao case study.

[0141] This invention proposes an end-to-end soft measurement modeling method for data imputation that integrates an attention mechanism. This method enhances the dynamic capture capability of the imputation stage by fusing an attention mechanism and LSTM to focus on global dynamic modeling and local dynamic enhancement, respectively. Subsequently, it incorporates dynamic labeling information from downstream tasks to jointly optimize the imputation and downstream soft measurement tasks, thereby improving the final prediction performance of the imputation task when the downstream target soft measurement task is clearly defined. The performance of this method is verified in a cement clinker production case study.

[0142] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A data imputation end-to-end soft measurement modeling method incorporating an attention mechanism, characterized in that, The method includes the following steps: Step 1) Obtain raw data samples and preprocess the data. The data is standardized so that the variables are arranged on a common scale. The standardized data is then used to extract a continuous subset of the data as a new sample using a sliding time window. Step 2) Construct the filling model An LSTM encoder with an integrated attention mechanism is constructed to capture the latent representation of incomplete data. Linear decoding is used to recover missing items. Randomly masked partial observation data is used as self-supervised label values. The distance between the masked partial observation values ​​and the corresponding position filled output is used as a quantitative indicator of the filling error. Step 3) End-to-end framework training and evaluation for joint soft measurement Integrating incomplete models and soft measurement models into an end-to-end framework for joint optimization improves modeling performance in industrial processes. The soft measurement evaluation metrics used are root mean square error (RMSE) and mean absolute error (MAE). The quality assessment of the imputation data employs kernel density estimation (KDE) to visualize the degree of distributional differences in the imputed data.

2. The data imputation end-to-end soft measurement modeling method with fused attention mechanism as described in claim 1, characterized in that, The process of step 1) is as follows: Step 1.1: Divide the sample Each sample is obtained by dividing the data into segments with a fixed window size, and contains T time steps and N feature variables, represented as follows: Represented as a single time step vector, where each X has a binary mask M∈{0,1}. T×N Indicates the validity of the observations at the corresponding locations; Step 1.2: Sample Normalization x in each X t First, subtract the mean μ of each variable, and then divide by the standard deviation σ of the training samples to achieve sample standardization.

3. The data imputation end-to-end soft measurement modeling method with fused attention mechanism as described in claim 1, characterized in that, The process of step 2) is as follows: Step 2.1: Embedding Processing A linear transformation maps the feature dimension N to a D-dimensional spatial vector. To preserve the ability to perceive sequence position information, the embedded matrix X... (0) The calculation formula is as follows: X (0) =XW p +b p +E pos (1) In the formula These are learnable parameters. It is the positional encoding information of the sequence. It is the input sequence for the subsequent encoder; Step 2.2: Construct the filling encoder The embedded representation is fed into the encoder, where a parallel multi-head attention module is used to calculate the correlation between positions to capture global dependencies. The multi-head attention module contains three learnable weight matrices, which map the input to a T×d dimension. k The query Q has a dimension of T×d. k The key K and dimension T×d v The value V is linearly mapped as follows: In the formula as well as Where h∈{1,2,…,H}, H is the number of multi-head attention, the scaled dot product of Q and K is calculated and then multiplied by V after applying softmax(.), to obtain the attention mechanism attn(.,.,.), where softmax(.) normalizes the similarity scores of each query and all key values ​​into a probability distribution; Project the concatenated output of each attention head onto the multi-head attention result MBead(.,.,.): In the formula, Concat(.) is tensor concatenation. It is a learnable matrix. The softmax() function is defined as follows: Where exp(.) denotes the natural exponential function, and ||.||1 denotes the first norm; Multi-head attention mechanisms focus on the input sample sequence from different subspaces, decomposing the global information of the sequence into a superposition of multiple micro-perspectives; after obtaining the globally dependent features, they are then combined with X. (0) Add them together and normalize the results to achieve stable training, denoted as . The input is a two-layer LSTM, which further models the temporal dependency features in the sequence, and then batch-normalizes the feature distribution again to provide a more stable input representation for downstream processes. In the formula It is X (0) After the output is modified by a single coding block, Norm(.) represents batch normalization, and Dropout(.) represents randomly dropping a small number of neurons to alleviate network overfitting. The specific definition of lstm(.) is as follows: In the formula, σ(.) represents a fully connected layer with an activation function, tanh(.) represents the hyperbolic tangent activation function, and o t p represents the input vector at the current time step. t It's the input gate, q t It's the Gate of Oblivion, g t It is the candidate cell state, r t It's an output gate, s t It's about cell state renewal, z t It is output in the hidden state, r t-1 and z t-1 W represents the cell state update and hidden layer state from the previous step. op W oq W og W os W represents the weight matrix input to each gate. zp W zq W zg W zs b represents the weight matrix from the hidden state to each gate. op b oq b og b os This represents the bias of the input to each gate, b zp b zq b zg b zs This indicates the offset of the hidden state to each gate; The residual structure inside the encoder supports hierarchical stacking. The encoder structure consisting of H attention heads and two layers of LSTM residual connections is denoted as MAL(.). The output of the current encoder block is used as the input of the next encoder block, as shown in the following formula: X (c) <MAL(X (c-1) ) (7) In the formula, c∈{1,2,…,C}, This represents the output of the c-th encoder block. The entire encoder is composed of C blocks stacked sequentially, with its initial input X. (0) That is, the representation after position embedding; Step 2.3: Linear Decoding of the Model Linear decoding is used to map the latent representation back to the real space to obtain the reconstruction result; the encoder's output sequence X... (C) After linear decoding, the prediction of the missing data is obtained: X'=X (C) W o +b o (8) In the formula, and These are learnable parameters. That is, the output sequence after the model is filled.

4. The data imputation end-to-end soft measurement modeling method with fused attention mechanism as described in claim 1, characterized in that, The process of step 3) is as follows: Step 3.1: Process the filled reconstruction output The end-to-end frame completion model uses a self-supervised approach to reconstruct and learn from the observations, randomly selecting effective values ​​at a proportion of α as self-supervised training labels. The corresponding mask is M label ∈{0,1} T×N The remaining valid observations that were not selected as labels As information accessible to the filling module; in the mask label M label At the position = 1, the output X' of the imputation model is used to construct the imputation loss with the effective value: Loss1=MSE(X'⊙M label ,X label ) (9) In the formula, ⊙ represents the Hadamard product, MSE(.) represents the root mean square error between the two, and Loss1 is the model loss during the missing data imputation stage; X' reconstructs the data for the entire sample. To reduce the error compared to the real data, it is processed as follows before being input into the downstream soft measurement task: X filled =X'⊙(1-M)+X⊙M (10) In the formula X filled Retain valid observation data for X and X' data for missing positions; Step 3.2: Soft Measurement Modeling and Prediction H=lstm L (X filled ) (11) In the formula, lstm L (.) indicates that there are L layers of lstm(.) connections. U represents the hidden states of the last LSTM layer at all time steps, and U is the dimension of the last LSTM hidden layer. The attention mechanism allows the model to give different degrees of attention to each time step. The model adaptively learns the contribution of each time step and achieves soft measurement for window prediction of a single point. In the formula It is the attention weight vector after softmax(.). This is the attention linear transformation learnable matrix, where 1 indicates that the soft measurement target variable has 1 type. These are learnable weights, where 1 indicates a single-point prediction. The output of the soft measurement model is y', and the model loss is constructed using the true label y: Loss2 = MSE(y,y') (13) Step 3.3: End-to-end joint unified optimization objective In order to fill in the gaps by combining the improved model with the downstream soft measurement task, the two independent models are jointly trained to realize the end-to-end framework from incomplete process variables to soft measurement targets. Combining formulas (9) and (13), the total loss during end-to-end framework training is constructed as follows: Loss=λ1Loss1+λ2Loss2 (14) In the formula, λ1 and λ2 are used to assign weights to the losses of the two sub-models, and the gradient of the total loss function is backpropagated to guide the update of the model weights. Step 3.4: Soft Measurement Prediction and Evaluation The root mean square error (RMSE) and mean absolute error (MAE) are used as evaluation indicators for soft measurement predictions, and their formulas are defined as follows: In the formula y t and y t ' represents the soft measurement label and the predicted value, respectively, and T represents the prediction time step length.

Images