Method for predicting key process variables based on indu dk-llm model

Abstract

The application discloses a key process variable prediction method based on an InduDK-LLM model, which separates the modeling of industrial process variables and operation variables, and combines historical time series data and industrial field knowledge. First, a pre-trained large model (pre-trained LLM) is introduced to process the time series data of the field knowledge and the operation variables to generate semantic-rich industrial prior encoding, wherein the former reduces the need for manual prompt engineering. In addition, the process variables are decomposed by a multi-scale method, and a learnable joint label is designed to promote the interaction between the two types of variables. Finally, an interaction module is designed to model the interdependence and potential logical relationship between the two types of variables. The method realizes accurate prediction of the key process variables of the industry, and provides reference and guidance for actual industrial production.

Description

TECHNICAL FIELD

[0001] The application relates to a key process variable prediction method based on an InduDK-LLM model and belongs to the field of industrial key process variable prediction. BACKGROUND

[0002] In industrial processes, accurate prediction of key process variables is crucial for ensuring product quality, process safety, and production efficiency in industrial production. Reliable prediction can often achieve proactive control, timely troubleshooting, and improve the consistency of the final product, thereby becoming the basis for process monitoring and control. Actual industrial time series data usually has the characteristics of high dimensionality, heterogeneity and multivariate. They can be conceptually divided into process variables (such as temperature, concentration) and operation variables (such as solvent addition rate, valve state, operation step). These two types of variables are often closely coupled and subject to time-varying logical relationships. Changes in operation variables often cause transient or long-term changes in process variables, and the strength of these coupling relationships may change with changes in operating conditions. Therefore, extracting interaction features that can describe the dependence relationship between the two types of variables is of great significance for accurate industrial process modeling and prediction.

[0003] Existing deep learning methods, such as convolutional neural networks (CNN), long short-term memory networks (LSTM), and gated recurrent units (GRU), have been widely used in industrial time series analysis and prediction. CNN can effectively extract representative local features in industrial time series by sliding one-dimensional convolution kernels in the time dimension; recurrent neural networks and their variants are excellent at processing sequence data, effectively capturing long-term patterns and periodic rules in time series. However, when faced with long sequence industrial time data, the limited receptive field of CNN gradually averages, smooths or dilutes long-term dependency information after multiple convolution layers; and the long time transmission chain of recurrent neural networks and their variants cannot be parallel computed; these lead to low accuracy and efficiency of the two methods, which cannot meet the requirements of actual industrial production. In order to solve this problem, models based on Transformer have received widespread attention. Transformer uses multi-head attention mechanism to efficiently model long-distance dependencies in sequence data; and through the expansion of parallel computing, it effectively improves the efficiency of the model. However, existing Transformer-based methods often ignore the complex logical relationship between process variables and operation variables in industrial data, which greatly affects the accuracy of model prediction.

[0004] Recent research has seen the revolution of Large Language Models (LLMs) in numerous fields. The latent knowledge acquired through pre-training on large corpora can be transferred to time series analysis. Carefully designed cue words enable the model to capture complex historical dependencies between data points, resulting in excellent performance and generalization capabilities for LLM-based prediction methods. Existing methods for applying LLM to time series prediction can be broadly categorized into feature engineering recoding and fine-tuning LLM. Feature engineering recoding involves extracting, transforming, and selecting useful information from raw data, converting it into a form more easily processed by the model. This approach is suitable for scenarios with limited data, obvious features, and a need for rapid implementation. Through carefully designed features and cue words, it can achieve good results on limited data. Fine-tuning LLM, on the other hand, involves additional training on a pre-trained model using task-specific data to optimize its performance on that task. This approach is suitable for scenarios with large datasets and where the model needs to automatically learn complex patterns. Fine-tuning allows full utilization of the powerful capabilities of LLM to achieve even better performance.

[0005] Current large language model frameworks in the industrial field often only take external textual information (such as news reports, holiday events, etc.) or time series feature embeddings as input, allowing the large language model to perform single-modal processing. While these methods can capture the correlation between external events and time series to some extent, their understanding of the internal mechanisms, structured knowledge, and industrial production processes of industrial systems remains limited. In particular, existing methods have not fully explored and utilized the rich domain knowledge contained in industrial technical documents, equipment manuals, control logic specifications, and other texts, nor have they enabled the powerful reasoning capabilities of large language models to work synergistically at both the technical document understanding and time series modeling levels. Therefore, neither feature engineering-based recoding methods nor fine-tuning LLMs can achieve high-precision predictions for complex industrial processes. Summary of the Invention

[0006] To achieve high-precision prediction of key process variables in industrial processes, this invention proposes a key process variable prediction method based on the InduDK-LLM model, the method comprising:

[0007] Step 1: For a specific industrial process, acquire its technical documentation data and collect historical data of industrial variables to construct an industrial time series dataset. Preprocess the industrial time series dataset to obtain a standard dataset, and divide the standard dataset into training set, validation set and test set according to the proportion.

[0008] Step 2: Divide the industrial variables in the standard dataset obtained in Step 1 into process variables and operational variables;

[0009] Step three, constructing a prediction model based on InduDK-LLM;

[0010] Step four, training the prediction model constructed in step three by using the data set classified by variables in step two;

[0011] Step five, predicting the industrial key process variables by using the prediction model trained in step four.

[0012] Optionally, the prediction model based on InduDK-LLM in step three includes an inverted embedding module, a static wavelet decomposition module, a process joint blocking module, a domain knowledge and operation variable embedding module, an interaction module, and a prediction module;

[0013] The inverted embedding module is used to project the process variables and operation variables obtained by dividing in step two into high-dimensional embedding in the time dimension respectively, to obtain process variable inverted embedding sequences and operation variable inverted embedding sequences.

[0014] The static wavelet decomposition module is used to decompose the process variable inverted embedding sequences to obtain corresponding coefficient sequences; the process joint blocking module is used to divide the coefficient sequences obtained by the static wavelet decomposition module into multiple patches according to channels.

[0015] The domain knowledge and operation variable embedding module is used to convert the obtained technical document data into labeled numerical embedding, and generate industrial domain semantic enhanced industrial priori coding in combination with the operation variable inverted embedding sequences.

[0016] The interaction module is used to process the multiple patches of each component output by the process joint blocking module and the industrial priori coding output by the domain knowledge and operation variable embedding module, so as to establish the connection between the process variables and the operation variables, and output multi-scale process variable components.

[0017] The prediction module includes an inverse static wavelet transform module and a head layer, the inverse static wavelet transform module is used to reconstruct the signal according to the multi-scale process variable components output by the interaction module, and the head layer obtains the prediction result through a Flatten layer and a linear layer.

[0018] Optionally, the static wavelet decomposition module includes padding operation and convolution operation; the input process variable inverted embedding sequence is first subjected to padding operation to ensure the length unchanged, and then subjected to first layer decomposition to obtain low frequency part and high frequency part, which are respectively subjected to low-pass filter and high-pass filter convolution to extract approximation coefficient and detail coefficient , wherein the approximation coefficient The coefficients are transmitted to the next layer to continue decomposition and convolution operation to obtain higher-level low-frequency and high-frequency features until a preset decomposition number S is reached, and finally an output coefficient sequence composed of the highest-level approximation coefficients and detail coefficients of each layer ] is obtained; the hole rate of each layer of convolution operation is expanded layer by layer to expand the receptive field.

[0019] Optionally, the process joint patch module performs channel-independent patch operation on each channel of each coefficient in the coefficient sequence, and adds a learnable joint mark at the end of each channel patch to facilitate subsequent interaction between the process variable and the operation variable.

[0020] Optionally, the domain knowledge and operation variable embedding module includes a domain knowledge branch, an operation variable branch and a pre-trained large model empowerment part, wherein the domain knowledge branch converts the obtained technical document data into token numerical embedding, the operation variable branch uses a multi-head cross-attention mechanism to align the operation variable inverted embedding sequence with the pre-trained LLM discrete word embedding, and the pre-trained large model empowerment is used to parse the domain knowledge and operation variable inverted embedding sequence and output semantic-rich industrial priori encoding.

[0021] Optionally, the interaction module includes a multi-head self-attention layer, a multi-head cross-attention layer, a feedforward network layer and a head layer, each sub-layer is connected by using residual connection and layer normalization, and the interaction module takes the multiple patches of each component output by the process joint patch module and the industrial priori encoding output by the domain knowledge and operation variable embedding module as inputs to establish the relationship between the process variable and the operation variable.

[0022] Optionally, the specific industrial process includes industrial alcohol precipitation, industrial traditional Chinese medicine extraction, industrial concentration, industrial sulfur recovery, industrial hydrocracking and froth flotation process.

[0023] Optionally, the pre-processing in step one includes Z-Score normalization processing.

[0024] Optionally, the domain knowledge branch in the domain knowledge and operation variable embedding module converts the obtained technical document data into token numerical embedding, including:

[0025] The text and symbol flow data in the technical document data are first subjected to the frozen pre-trained LLM tokenizer for tokenization processing, and then the pre-trained LLM embedder is used to convert the token into token numerical embedding.

[0026] The present application has the following advantages:

[0027] The modeling of process variables and operating variables is explicitly separated, and domain knowledge and time series data are integrated to improve prediction accuracy. First, the domain knowledge and time series data of operating variables contained in the industrial technical documents are combined as the input of the pre-trained LLM, which reduces the workload of manual prompting engineering and generates industrial priori coding with semantic enhancement. Second, a learnable joint mark is introduced to establish the logical connection between process variables and operating variables. Finally, an interaction module is designed to model the interdependence and potential logical relationship between the two types of variables. The method realizes accurate prediction of key process variables in industry and provides a reference for actual industrial production. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0029] Figure 1 is a flowchart of the key process variable prediction method based on the InduDK-LLM model provided by the embodiments of the present application;

[0030] Figure 2 is a schematic diagram of the prediction model structure based on InduDK-LLM in the key process variable prediction method based on the InduDK-LLM model provided by the embodiments of the present application;

[0031] Figure 3 is a schematic diagram of the SWT module of the prediction model based on InduDK-LLM in the key process variable prediction method based on the InduDK-LLM model provided by the embodiments of the present application;

[0032] Figure 4 is a schematic diagram of the ISWT module of the prediction module in the prediction model based on InduDK-LLM in the key process variable prediction method based on the InduDK-LLM model provided by the embodiments of the present application;

[0033] Figure 5 is a schematic diagram of the domain knowledge and operating variable embedding module of the prediction model based on InduDK-LLM in the key process variable prediction method based on the InduDK-LLM model provided by the embodiments of the present application;

[0034] Figure 6 is a schematic diagram of the interaction module of the prediction model based on InduDK-LLM in the key process variable prediction method based on the InduDK-LLM model provided by the embodiments of the present application;

[0035] Figure 7 is a schematic diagram of the prediction result of the temperature under the industrial alcohol settling tank by the key process variable prediction method based on the InduDK-LLM model provided by the embodiment of the present application;

[0036] Figure 8 is a schematic diagram of the prediction result of the temperature under the industrial alcohol settling tank by the existing model DLinear;

[0037] Figure 9 is a schematic diagram of the prediction result of the temperature under the industrial alcohol settling tank by the existing model TimeMixer;

[0038] Figure 10 is a schematic diagram of the prediction result of the temperature under the industrial alcohol settling tank by the existing model PatchTST;

[0039] Figure 11 is a schematic diagram of the prediction result of the temperature under the industrial alcohol settling tank by the existing model Time-LLM;

[0040] Figure 12 is a schematic diagram of the prediction result of the temperature under the industrial alcohol settling tank by the existing model TimeCMA;

[0041] Figure 13 is a schematic diagram of the error distribution violin of the prediction result of the temperature under the industrial alcohol settling tank by the key process variable prediction method based on the InduDK-LLM model provided by the embodiment of the present application and the existing model;

[0042] Figure 14 is a comparison diagram of the evaluation results of different prediction models. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0044] Embodiment one

[0045] The embodiment provides a key process variable prediction method based on an InduDK-LLM model, a flowchart of the method is as shown in Figure 1 First, industrial production data is collected to construct a data set, and the data is cleaned and preprocessed, the industrial production data refers to various variable data in an industrial process, collectively referred to as industrial variables; then the industrial variables are divided into process variables and operation variables; subsequently, a prediction model based on InduDK-LLM is constructed, the prediction model is trained and evaluated according to the constructed data set; finally, the trained prediction model is used to predict the industrial key process variables.

[0046] Specifically includes:

[0047] Step one: Collect the historical data of each industrial variable in the specific industrial process to construct the industrial time series dataset, and preprocess it to obtain the standard dataset and divide it into training set, validation set and test set according to the proportion.

[0048] The specific industrial process includes industrial alcohol precipitation, industrial traditional Chinese medicine extraction, industrial concentration, industrial sulfur recovery, industrial hydrocracking, froth flotation process and other processes. The corresponding industrial variables are different for different industrial processes. The preprocessing includes Z-Score normalization processing. Subtract the mean value of each channel and divide by the standard deviation to standardize the measurement values to the same range. The specific expression is:

[0049] ;

[0050] Wherein, and represent the data before and after standardization, and represent the mean and standard deviation of ;

[0051] The proportion of training set, validation set and test set is 7:2:1.

[0052] Step two: Divide the industrial variables in the standard dataset obtained in step one into process variables and operation variables.

[0053] According to the properties of the variables, the variables in the industrial data are divided, and the process variables and operation variables are modeled by different ways, and an interaction module is designed to capture the mutual dependence relationship and potential logical relationship between them. Among them, the process variables include temperature, concentration and other physical or chemical quantities that can be measured or observed. These variables reflect the state of the process. The operation variables include solvent addition rate, valve state, operation steps and other variables directly controlled and adjusted by the operator or control system.

[0054] Step three: Construct the prediction model based on InduDK-LLM.

[0055] As shown in Figure 2 , the prediction model based on InduDK-LLM includes an inverted embedding module, a static wavelet decomposition module (SWT), a process joint blocking module, a domain knowledge and operation variable embedding module, an interaction module and a prediction module.

[0056] The inverted embedding module includes a linear layer that is used to process both process variable sequences and operational variable sequences, specifically, a multivariate time series is treated as a single token and projected along the time dimension into a high-dimensional embedding, generating a temporally aligned representation for each variable. This inverted embedding enables the model to more effectively capture complex cross-variable dynamics, which can be represented as:

[0057] ;

[0058] where, represents the number of process variables or operational variables, and denote the length of the original sequence before embedding and the hidden dimension of the embedding, respectively.

[0059] As shown in Figure 3 , the static wavelet decomposition module (SWT) includes a padding operation and a convolution operation; the input process variable inverted embedding sequence is first padded to ensure a constant length, and then decomposed at the first layer to obtain a low-frequency part and a high-frequency part, which are convolved by a low-pass filter and a high-pass filter , respectively, to extract the approximation coefficient and the detail coefficient . In each layer of decomposition, the network does not perform downsampling like traditional discrete wavelets, but expands the receptive field through an increasing dilation rate (dilation = 1, 2, 4, …) to enable the convolution to capture longer time-domain features while maintaining the time resolution.

[0060] The approximation coefficient obtained by decomposition is passed to the next layer for convolution operation, thereby obtaining higher-level low-frequency and high-frequency features, until the preset decomposition number is reached. The final output is formed by the highest-level approximation coefficient and the detail coefficients of each layer, forming a complete representation of the signal in multiple scales; its expression is:

[0061] ;

[0062] where, h and represent the learnable low-pass filter and high-pass filter, respectively, represents the number of decompositions. The process variable embedding sequence is decomposed into approximation coefficients and The coefficients are detailed coefficients, where approximate coefficients reflect long-term trends, while detailed coefficients capture short-term fluctuations or sudden changes. At the same time, SWT does not downsample the coefficients obtained from the decomposition, and its time shift invariance ensures that the decomposed coefficients are aligned point-to-point with the original time series on the time axis.

[0063] The process joint block module is used for each coefficient sequence generated by SWT. For each coefficient, a channel-independent patch operation is performed on each channel. Transient events that are temporally localized and causally related to the operated variables are amplified in their original local context, making them more detectable for downstream local feature extractors and attention mechanisms. Furthermore, a learnable joint label is added at the end of each channel patch, providing a trainable bridge to facilitate subsequent interactions between process variables and operated variables. Specifically, taking the first... Detail coefficients For example, for each of its channels The output after block division is ,in Indicates the number of channels. This represents the length of each patch. The total number of patches is represented by the following expression: ,in, This represents the sliding step size for the block operation. This indicates taking the integer part of the number; subsequently, a shape of... Learnable joint tags, the patch representation for each channel becomes Through the action of a linear layer, Embedded into higher dimensions ,in To hide the dimension.

[0064] like Figure 5 As shown, the domain knowledge and operational variable embedding module consists of three parts: the domain knowledge branch, the operational variable branch, and the pre-trained large model (pre-trained LLM) empowerment. It utilizes the powerful capabilities of the pre-trained large model (pre-trained LLM) to generate semantically enhanced industrial domain semantically enhanced industrial prior codes, which helps the model understand industrial processes and uncover the inherent logical relationship between the two industrial variables.

[0065] The domain knowledge branch transforms text data from industrial technical documents into labeled numerical embeddings. Specifically, the text and symbolic flow data are first segmented using a frozen pre-trained LLM tokenizer, and then the labeled data is transformed into labeled numerical embeddings using a pre-trained LLM embedder. The branch incorporates dataset summary, industrial equipment description, standard operating procedure, and explanation of related industrial variables into the model.

[0066] The operation variable branch uses multi-head cross attention mechanism to align the continuous value embedding of operation variables with the discrete word embedding of pre-trained LLM; to reduce the amount of calculation, avoid information redundancy, and improve the efficiency and accuracy of the prediction task, the original pre-trained large model word embedding is projected into industrial field related word embedding , the specific expression is:

[0067] ;

[0068] where, and represent the size of and , is the input dimension of LLM, and represent the learnable parameters of linear layer.

[0069] The operation variable embedding is taken as the query, the industrial related word embedding is taken as the key and value, and the multi-head cross attention mechanism is performed between the two to achieve the alignment purpose, and the expression of the first single attention head is:

[0070] ;

[0071] where, , , are the query, key and value of the first single attention head, , , are the learnable weight matrix;

[0072] Then the results of all single attention heads are combined to get:

[0073] ;

[0074] where, represents the number of attention heads;

[0075] Finally, through a linear layer, the is projected into the dimension of LLM, and the expression is:

[0076] ;

[0077] in, and This represents the learnable parameters of the linear layer.

[0078] like Figure 6 As shown, the interaction module includes a multi-head self-attention layer, a multi-head cross-attention layer, a feedforward network layer, and a... Each sublayer is connected using residual connections and layer normalization. This module is used to establish relationships between process variables and operational variables, capturing their complex temporal and causal relationships. It includes a multi-head self-attention mechanism, a multi-head cross-attention mechanism, a feedforward network layer, and a... Each sub-layer is connected using residual connections and layer normalization to stabilize the training process and alleviate the gradient vanishing problem.

[0079] The input to the multi-head self-attention mechanism is the patch embedding result of the process variable. This modeles dependencies between blocks within process variables and enables the introduced learnable joint tags to acquire knowledge related to process variables from other blocks; the output of a single attention head is:

[0080] ;

[0081] Aggregating a single attention head, the output of this sublayer The expression is:

[0082] ;

[0083] in, Indicates the first The output of each attention head, Represents the number of attention heads, , This is the output weight matrix.

[0084] The output after introducing residual connections and layer normalization at this layer is:

[0085] ;

[0086] in, Representative level normalization, This represents a regularization technique to prevent model overfitting;

[0087] The query for multi-head cross-attention mechanism is Learnable joint labeling of process variable embeddings Both the keys and values ​​are the outputs of the pre-trained LLM. This enables global dependency modeling of process and operational variables, aggregating a single attention head, and representing the output of this sublayer. The expression is:

[0088] ;

[0089] in, Indicates the first The output of a cross-attention head, Represents the number of attention heads, , This is the output weight matrix.

[0090] The output after introducing residual connections and layer normalization at this layer is:

[0091] ;

[0092] The feedforward network layer consists of two linear layers. A nonlinear transformation is introduced into these two linear layers through the GeLu activation function, and its output... The expression is:

[0093] ;

[0094] As head The layer takes input and the interactive module takes output. The expression is:

[0095] ;

[0096] in Indicates the first The output results of multiple scale components.

[0097] The prediction module includes inverse static wavelet transform ( ) and a layer;

[0098] like Figure 4 As shown, the Inverse Static Wavelet Decomposition (ISWT) module includes padding and convolution operations; ISWT realizes the process of gradually recovering the original scale signal from multi-scale coefficients, using the highest-level approximation coefficients. and detail coefficient Starting with the first two, fill them separately, and then pass them through the reconstruction filter. and Convolution yields the reconstructed low-frequency and high-frequency components. Adding and averaging these two components gives a new approximation coefficient. The expansion rate is then halved, and this result is compared with the detail coefficient of the next layer. The next convolution fusion is inputted together; thus, the approximation and detail information are continuously fused through layer-by-layer iteration, and finally the reconstructed signal is obtained; all multi-scale component features are expressed through inverse static wavelet transform reconstructed to the original scale representation, wherein , represents the decomposition number; the expression of the inverse transform is:

[0099] ;

[0100] wherein, and represent the learnable reconstruction low-pass filter and the reconstruction high-pass filter, respectively.

[0101] The normalized prediction result is obtained through another layer:

[0102] ;

[0103] wherein, all multi-scale components are stacked; the inverse normalization is used to obtain the final prediction result .

[0104] Step four: use the data set classified by variables in step two to train the prediction model constructed in step three.

[0105] By comparing the model prediction value with the true value, the prediction model parameters and weights are optimized, and when the mean square error loss of the validation set does not decrease for three consecutive iterations, the model stops training.

[0106] Step five: predict the industrial key process variables according to the prediction model obtained in step four.

[0107] The prediction of industrial key process variables is realized using the prediction model based on InduDK-LLM.

[0108] Embodiment two

[0109] This embodiment provides a key process variable prediction method based on InduDK-LLM model, which is applied to industrial alcohol precipitation process as an example: using the preprocessed data set, the prediction model based on InduDK-LLM is used to realize the prediction of industrial key process variables.

[0110] This embodiment uses a total of 6 industrial alcohol precipitation tank data sets to verify the performance of the prediction model based on InduDK-LLM proposed by us, which were recorded in January 2024, sampled once per minute, and a total of about 44480 time steps.

[0111] In order to verify the performance of the prediction model based on InduDK-LLM proposed in the present application, the present application compares it with DLinear, TimeMixer, PatchTST, Time-LLM, TimeCMA in temperature prediction under the industrial alcohol tank; wherein,

[0112] DLinear can refer to Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023, June). Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS) (Vol. 37, No. 9, pp. 11121-11128);

[0113] TimeMixer can refer to Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L.,... & Zhou, J. (2024). TimeMixer: Decomposable multiscale mixing for time series forecasting. arXiv preprint arXiv:2405.14616 ;

[0114] PatchTST can refer to Nie, Y. (2022). A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. arXiv preprint arXiv:2211.14730 ;

[0115] Time-LLM can refer to Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X.,... & Wen, Q. (2023). Time-LLM: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 ;

[0116] TimeCMA can be specifically referred to Liu, C., Xu, Q., Miao, H., Yang, S., Zhang, L., Long, C.,... & Zhao, R. (2025, April). Timecma: Towards llm-empowered multivariate time series forecasting via cross-modality alignment. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 1878-1888). Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 18, pp. 18780-18788).

[0117] The mean absolute error (MAE), mean square error (MSE) and coefficient of determination (R2) are used as evaluation criteria; the smaller the values of MAE and MSE, the better the performance of the model; if the value of R2 is closer to 1, it means that the model has a better fitting effect and a stronger ability to explain the observed values; the specific calculation formulas of the three indexes are as follows:

[0118] ;

[0119] ;

[0120] ;

[0121] wherein, , and represent the true value, the predicted value and the average value of the true value respectively; at the same time, represents the sample number.

[0122] The comparison results are shown in Table 1, wherein the results in bold represent the best. Figure 14

[0123] Overall, the prediction model based on InduDK-LLM proposed in the present application achieves the lowest MAE, the lowest MSE and the highest R2 on each data set. ​​​, which indicates that its performance is always stable and has a significant advantage; for example, in dataset 1, the MAEs of other comparative models are 0.1664, 0.1684, 0.5206, 0.3539 and 0.1544 respectively, while the MAE of InduDK-LLM is 0.1056, which is reduced by 36.54%, 37.29%, 79.72%, 70.16% and 31.61% respectively; in dataset 2, the proposed model reduces the MAE value by 0.2825, 0.1674, 0.2099, 0.4579, 0.0365, and the average increase is ((56.76% + 43.75% + 49.38% + 68.03% + 14.50%) / 5) = 46.48%; at the same time, the proposed model obtains the highest values in datasets 5 and 6, which are 0.9901 and 0.9957 respectively, and from the perspective of , it also performs better. The prediction model based on InduDK-LLM proposed in the present application can effectively extract the hidden nonlinear dynamic relationship and logical relationship in these variables by explicitly dividing complex industrial alcohol precipitation variables, integrating domain knowledge and operation variables, and using the powerful reasoning ability of the pre-trained large language model to enhance the understanding of the industrial process by the model, thereby significantly improving the prediction accuracy.

[0124] Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 、 Figure 11 、 Figure 12 , which are the temperature prediction results of the industrial alcohol precipitation tank of InduDK-LLM, DLinear, TimeMixer, PatchTST, Time-LLM and TimeCMA respectively; as can be seen from the figure, the prediction curve of the prediction model based on InduDK-LLM is highly consistent with the actual temperature curve, and its performance at temperature stable and turning points, as well as temperature peak and trough, is better than that of the other five comparative models; Figure 13 , which are the absolute error distribution violin plots of the prediction model based on InduDK-LLM and the existing models provided by the present application on dataset 6, the prediction model based on InduDK-LLM presents a narrow and compact chord diagram, with a median closer to zero and a quartile range shorter, which indicates that most of the predicted values are close to the true values, and the situation of large error is rare; it is proved that the prediction model based on InduDK-LLM proposed in the present application can realize accurate prediction.

[0125] Some steps in the embodiments of the present application can be realized by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0126] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for predicting key process variables based on InduDK-LLM model, characterized in that, The method comprises: Step one, for a specific industrial process, obtain its technical document data, and collect historical data of industrial variables to construct an industrial time series dataset, preprocess the industrial time series dataset to obtain a standard dataset, and divide the standard dataset into a training set, a validation set and a test set according to a proportion; Step two, divide the industrial variables in the standard dataset obtained in step one into process variables and operation variables; Step three, construct a prediction model based on InduDK-LLM; Step four, train the prediction model constructed in step three using the dataset classified by variables in step two; Step five, use the prediction model trained in step four to predict the key process variables of the industry; The prediction model based on InduDK-LLM in step three comprises an inverted embedding module, a static wavelet decomposition module, a process joint blocking module, a domain knowledge and operation variable embedding module, an interaction module and a prediction module; The inverted embedding module is used to project the process variables and operation variables divided in step two into high-dimensional embeddings in the time dimension respectively to obtain process variable inverted embedding sequences and operation variable inverted embedding sequences; The static wavelet decomposition module is used to decompose the process variable inverted embedding sequences to obtain corresponding coefficient sequences; the process joint blocking module is used to divide the coefficient sequences decomposed by the static wavelet decomposition module into multiple patches according to channels; The domain knowledge and operation variable embedding module is used to convert the obtained technical document data into labeled numerical embeddings, and generate industrial domain semantic enhanced industrial priori encoding combined with the operation variable inverted embedding sequences; The interaction module is used to process the multiple patches of each component output by the process joint blocking module and the industrial priori encoding output by the domain knowledge and operation variable embedding module, thereby establishing a connection between the process variables and the operation variables, and outputting multi-scale process variable components; The prediction module comprises an inverse static wavelet transform module and a head layer, the inverse static wavelet transform module is used to reconstruct the signal according to the multi-scale process variable components output by the interaction module, and the head layer obtains a prediction result through a Flatten layer and a linear layer; The specific industrial process includes industrial alcohol precipitation, industrial traditional Chinese medicine extraction, industrial concentration, industrial sulfur recovery, industrial hydrocracking and froth flotation process.

2. The method of claim 1, wherein, The static wavelet decomposition module comprises a padding operation and a convolution operation; the input process variable inverted embedded sequence is firstly subjected to the padding operation to ensure constant length, and then subjected to first layer decomposition to obtain a low frequency part and a high frequency part , which are subjected to convolution through a low pass filter and a high pass filter , respectively, to extract an approximation coefficient and a detail coefficient, wherein the approximation coefficient is transmitted to the next layer to continue decomposition and convolution operation to obtain higher layer low frequency and high frequency features, until a preset decomposition number S is reached, and finally a coefficient sequence composed of the highest layer approximation coefficient and each layer detail coefficient is output; the hole rate of each layer convolution operation is expanded layer by layer to expand the receptive field.

3. The method of claim 2, wherein, The process joint blocking module performs channel-independent patch operations on each channel of each coefficient in the coefficient sequence, and adds a learnable joint label at the end of each channel patch to facilitate subsequent interaction between the process variables and the operation variables.

4. The method of claim 3, wherein, The domain knowledge and operation variable embedding module comprises a domain knowledge branch, an operation variable branch and a pre-trained large model empowerment, wherein the domain knowledge branch converts the obtained technical document data into token numerical embedding, the operation variable branch inverts the embedding sequence of the operation variable by using a multi-head cross attention mechanism and aligns the inverted embedding sequence with the discrete word embedding of the pre-trained LLM; and the pre-trained large model empowerment is used to analyze the inverted embedding sequence of the domain knowledge and the operation variable and output semantic-rich industrial prior encoding.

5. The method of claim 4, wherein, The interaction module comprises a multi-head self-attention layer, a multi-head cross-attention layer, a feedforward network layer and a head layer, and each sub-layer is connected by using residual connection and layer normalization; and the interaction module takes the multiple patches of each component output by the process joint block module and the industrial prior encoding output by the domain knowledge and operation variable embedding module as inputs to establish the relationship between the process variables and the operation variables.

6. The method of claim 5, wherein, The preprocessing in step one comprises Z-Score normalization processing.

7. The method of claim 6, wherein, The domain knowledge branch in the domain knowledge and operation variable embedding module converts the obtained technical document data into token numerical embedding, which comprises: The text and symbolic process data in the technical document data are first subjected to word segmentation processing by a frozen pre-trained LLM word segmenter, and then the token is converted into token numerical embedding by using a pre-trained LLM embedder.

Images