Industrial time sequence prediction method fusing domain knowledge enhancement

Abstract

The invention relates to an industrial time series prediction method fusing domain knowledge enhancement, and belongs to the field of time series prediction. The method comprises the following steps of: firstly, preprocessing time sequence data, cutting the data into fixed-length Patches according to a time dimension, and converting each Patch into a high-dimensional embedded vector through linear projection; then, cleaning and word segmentation are carried out on the domain knowledge text, the domain knowledge text is aligned with the corresponding Patch according to a timestamp, and text vector representation is generated by utilizing a pre-training language model; next, a prompt template is designed, the prediction task, the input data type and the model instruction form a prompt text, and a prompt embedding vector is generated for the prompt text; and finally, fusing the time sequence embedding, the text embedding and the prompt embedding, inputting the fused time sequence embedding, the fused text embedding and the fused prompt embedding into a generative pre-training Transform, and predicting a future time step by using an autoregression mode. According to the method, the prediction precision of a complex time sequence and the response capability of an emergency can be effectively improved, and the interpretability and the prediction performance of the model are enhanced.

Description

Technical Field

[0001] The invention relates to an industrial time series prediction method integrated with domain knowledge enhancement, and belongs to the field of time series prediction. Background Art

[0002] Time series forecasting involves using historical information in a retrospective sequence to predict future states. Time series analysis has a wide range of applications in many practical scenarios. For example, industrial time series forecasting can provide a deep understanding of the operating status of industrial processes, thereby supporting intelligent predictive control and optimization decisions. Therefore, time series forecasting has very important practical significance.

[0003] In the industrial field, in addition to time series data, domain expert knowledge and production logs are equally important for enhancing data analysis. Domain experts are able to identify and define features and indicators that are critical to specific problems. When data is scarce or of low quality, the expert's prior knowledge can be used to guide the learning direction of the model. Production logs record real-time data such as equipment operating status, maintenance activities, system failures, etc. This information is critical for predicting equipment failures, maintenance needs, and production efficiency. Analyzing production logs can help identify patterns and trends in the production process, such as identifying frequent failure points on the production line.

[0004] Most existing works only use time series data, lacking attention to domain expert knowledge and production logs. Combining domain expert knowledge and production logs can significantly improve the effectiveness of the prediction model and the accuracy of decision-making. Expert insights provide a theoretical basis and direction for the model, while real-time production logs provide data support, enabling the model to better adapt to actual production conditions and rapidly changing actual needs. Summary of the invention

[0005] The purpose of the present invention is to provide an industrial time series prediction method that integrates domain knowledge enhancement, aiming to solve the technical problem of how to effectively integrate knowledge enhancement time series data during time series prediction.

[0006] To achieve the above object, the present invention provides an industrial time series forecasting method enhanced by integrating domain knowledge, the method comprising:

[0007] Step 1: Collect time series data, pre-process and embed Patch to obtain time series embedding;

[0008] Step 2: Acquire dynamic expert knowledge, convert text-type expert knowledge data into vector representations that can be processed by the model, and obtain text embedding vectors;

[0009] Step 3: Design a prompt template, clarify the model's task objectives, input sources, and output requirements, and generate a prompt embedding vector;

[0010] Step 4: Combine the time series embedding, text embedding vector and prompt embedding vector to form a comprehensive input representation;

[0011] Step 5: Use the generative pre-trained Transformer as a prediction model to generate prediction values ​​for future time steps based on the comprehensive input representation.

[0012] The Step 1 is specifically as follows:

[0013] Representing time series data as a two-dimensional matrix Where n is the number of features and L is the time step;

[0014] Processing missing values, standardizing or normalizing the time series data, and performing detrending or deseasonalizing processing;

[0015] The preprocessed time series data is divided into multiple fixed-length patches of length p according to the time dimension, and each patch is embedded to generate a high-dimensional representation.

[0016] Specifically, the generation of Patch is as follows: the time series X is divided into segments Patch of length p according to the time dimension, which can be expressed as: P is the time step length of each Patch; is the number of patches.

[0017] Embed operation: For each PatchP i Apply linear projection to transform it into a high-level feature representation of fixed dimension: E i =W·Flatten(P i )+b, where W is the weight matrix of the linear transformation, Flatten(P i ) is to convert P i Converted into a one-dimensional vector, E i PatchP i is the embedding representation of , and b is the bias vector.

[0018] Merge all patches into E i , expressed as: P time =[E1; E2; ...; E m ]. time The shape is Where m is the number of patches and d is the embedding dimension.

[0019] The Step 2 is specifically as follows:

[0020] Clean the expert knowledge text, remove stop words, punctuation marks and useless characters, and perform word segmentation;

[0021] Align the expert knowledge text with the corresponding time series patch according to the timestamp;

[0022] Input the aligned expert knowledge text into the pre-trained language model to generate the text embedding vector V text .

[0023] The Step 3 is specifically as follows:

[0024] Design a prompt template, including instructions, data types, and prediction tasks;

[0025] Specifically, through structured prompts, the model can be better guided to handle complex data and tasks. The structure of the template is: a template is a sentence or phrase that clearly describes the task goal and has the following structure: [instructions] + [data type] + [prediction task].

[0026] Use the language model to encode the prompt template and generate the prompt embedding vector V prompt .

[0027] Specifically, the instructions set the direction of the model's processing task, ensuring that the model focuses on the target task. For example: Combine the following.

[0028] Data types help the model recognize the characteristics of the input data, such as time series data and expert knowledge.

[0029] Prediction tasks guide the model to generate results that meet the target requirements and avoid ambiguity, for example: predicting future trends.

[0030] The Step 4 is specifically as follows:

[0031] Input template text: Prompt: "Predict future trends based on time series data and expert knowledge." Output embedding vector V prompt .

[0032] Embed the prompt into vector V prompt , text embedding vector V text and time series Patch embedding P time Concatenate or fuse to form a comprehensive input representation X fusion =[V prompt ; V text ;P time ].

[0033] The Step 5 is specifically as follows:

[0034] The comprehensive input represents X fusion Input to the generative pre-trained Transformer;

[0035] Use the decoder's autoregressive mechanism to generate predictions for future time steps

[0036] The beneficial effects of the present invention are as follows: the present invention can effectively combine time series information in numerical form with expert knowledge in text form, guide the model to focus on the specified prediction target, improve the prediction accuracy of complex time series and the ability to respond to emergencies. By integrating multi-source heterogeneous data and prompt information, the interpretability and prediction performance of the model are significantly enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a structural diagram of a time series prediction method enhanced by integrating domain knowledge in this application. DETAILED DESCRIPTION

[0038] In order to better understand the above technical solution, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] Example 1: The following is combined with the specific application scenario of tin smelting to explain the implementation of the industrial time series analysis and prediction method based on time domain and frequency domain fusion feature extraction, which specifically includes the following:

[0040] Step 1: Collect time series data, pre-process and embed Patch to obtain time series embedding.

[0041] Input data: Time series data is obtained from tin smelting equipment, such as furnace temperature (temperature sensor), reaction pressure, smelting time, and key process variables (such as redox potential). The time series data is represented as a two-dimensional matrix Where n represents the number of features and L represents the length of the time series, i.e. the time step.

[0042] Time series preprocessing includes data normalization, noise removal, and missing value processing to improve data quality and model performance.

[0043] Patch generation: Split the time series X into patches of length p according to the time dimension. Each patch captures the local features within a continuous period of time. It is expressed as: P time =[E1; E2; ...; E m ]. time The shape is Where m is the number of patches and d is the embedding dimension.

[0044] The embedding operation is denoted as E i =W·Flatten(P i )+b, each patch is converted into a fixed-dimensional high-level feature representation E through a linear transformation (weight matrix W and bias vector b) and a Flatten operation (converting the matrix into a vector) i .

[0045] Step 2: Acquire dynamic expert knowledge, convert text-type expert knowledge data into a vector representation that can be processed by the model, and obtain the text embedding vector.

[0046] Domain knowledge input: text content such as operation manuals, historical records, and expert experience.

[0047] The text preprocessing steps include:

[0048] Clean text: remove stop words, punctuation marks, useless characters, etc.

[0049] Word segmentation: decompose the text into words or subwords (according to the word segmentation method of the language model used);

[0050] Remove irrelevant parts: If the text is too long or messy, you can filter out sentences or fragments related to a specific topic or a specific time period.

[0051] Align text with time series patches: Align text with the corresponding time series patches according to timestamps. i Covering the time step from t to t+p-1, the text content that occurred during this period can be collected. Different patches obtain their own text data.

[0052] Text embedding generation: Input the text of each time period into the pre-trained language model for embedding generation. Obtain the text embedding vector V for the corresponding time period text If there are multiple texts, they can be merged by averaging or additive fusion. The text vector corresponding to each time patch can be recorded as V text,i , and P i Corresponding.

[0053] Step 3: Design a prompt template, clarify the model’s task objectives, input sources, and output requirements, and generate a prompt embedding vector.

[0054] The prompt template can be divided into the following parts: [instructions] + [data type] + [prediction task].

[0055] The instructions set the direction for the model to process the task, such as "Please combine the following information", "Analyze based on the following data", etc.

[0056] The data type helps the model recognize whether the input data is a time series, textual expert knowledge, or something else.

[0057] Prediction tasks include "predicting future trends", "making classification decisions", "giving the probability distribution of the next time step", etc.

[0058] Step 4: Combine the time series embedding, text embedding vector and prompt embedding vector to form a comprehensive input representation.

[0059] Based on the previous time series Patch embedding P time and text embedding vector V text , as well as the text of the prompt, the model needs to combine the three for processing.

[0060] Encode the prompt text using the same or similar language model as the text to obtain the prompt embedding vector V prompt .

[0061] Embed the prompt into vector V prompt , text embedding vector V text and time series embedding P time Splicing, represented by: X fusion =[V prompt ; V text ;P time ] where “;” indicates concatenation along the batch dimension or sequence dimension.

[0062] Step 5: Use the generative pre-trained Transformer as a prediction model to generate prediction values ​​for future time steps based on the comprehensive input representation.

[0063] With the help of Transformer's self-attention (and possible cross-attention) mechanism, the model is able to jointly model and reason about time series, textual knowledge, and prompt information in the same network, thereby producing prediction results for complex industrial processes that are more accurate and more interpretable, and can include text descriptions or professional advice.

[0064] For X fusion Adding position encoding can help the model distinguish the position order information of different patches.

[0065] The model first performs an input sequence (i.e., X fusion The sequence is then processed by the decoder (with position encoding) and mapped to the latent space inside the model. This sequence is then input into a multi-layer stacked Transformer decoder layer.

[0066] Each layer in the decoder will perform a self-attention mechanism to weight all positions in the sequence (Patch+text+prompt) and explore the associations between them. Since the time series patch is spliced ​​together with the corresponding text and prompt information in the fusion representation, the model's attention mechanism can "cross-type" focus on the features of any position, thereby better combining the time series features with text expert knowledge and prompt information.

[0067] After the self-attention of each Decoder layer, there is a feed-forward network layer to further perform nonlinear mapping and dimensional transformation on the attention-weighted representation. This process is repeated at each layer of the Transformer to enhance the expressive power of the model.

[0068] After self-attention and feedforward networks, the model usually uses residual connections and layer normalization. Residual connections can help gradients propagate better and avoid difficulties in deep network training; layer normalization helps stabilize the training process.

[0069] Since it is a "generative pre-trained Transformer", after processing the fusion input X fusion After that, various forms of output prediction can be performed, depending on the actual task requirements, including sequence generation, classification, or regression.

[0070] Sequence generation requires the generation of new sequences, such as generating text descriptions, generating suggestions for the next process operation, or predicting the value of future time series. The model can use an auto-regressive method to gradually generate the next token (or value patch). In the tin smelting industry scenario, the model can generate process parameter suggestions or fault possibility descriptions for the next moment based on the temperature, pressure, and text descriptions over the past period of time.

[0071] The classification or regression output is not to generate a whole paragraph of text, but a classification label or regression value, such as "whether an abnormality has occurred", "remaining life prediction", etc. At this time, a specific classification head or regression head can be added to the last layer output of the Decoder to map the final hidden state to obtain the required category or value.

[0072] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An industrial time series forecasting method enhanced by integrating expert knowledge, characterized by: Step 1: Collect time series data, pre-process and embed Patch to obtain time series embedding; Step 2: Acquire dynamic expert knowledge, convert text-type expert knowledge data into vector representations that can be processed by the model, and obtain text embedding vectors; Step 3: Design a prompt template, clarify the model's task objectives, input sources, and output requirements, and generate a prompt embedding vector; Step 4: Combine the time series embedding, text embedding vector and prompt embedding vector to form a comprehensive input representation; Step 5: Use the generative pre-trained Transformer as a prediction model to generate prediction values ​​for future time steps based on the comprehensive input representation.

2. The industrial time series prediction method enhanced by integrating expert knowledge according to claim 1 is characterized in that: The Step 1 is specifically as follows: Representing time series data as a two-dimensional matrix Where n is the number of features and L is the time step; Processing missing values, standardizing or normalizing the time series data, and performing detrending or deseasonalizing processing; The preprocessed time series data is divided into multiple fixed-length patches of length p according to the time dimension, and each patch is embedded to generate a high-dimensional representation.

3. The industrial time series forecasting method enhanced by integrating expert knowledge according to claim 1 is characterized in that: The Step 2 is specifically as follows: Clean the expert knowledge text, remove stop words, punctuation marks and useless characters, and perform word segmentation; Align the expert knowledge text with the corresponding time series patch according to the timestamp; Input the aligned expert knowledge text into the pre-trained language model to generate the text embedding vector V text .

4. The industrial time series forecasting method enhanced by integrating expert knowledge according to claim 1 is characterized in that: The Step 3 is specifically as follows: Design a prompt template, including instructions, data types, and prediction tasks; Use the language model to encode the prompt template and generate the prompt embedding vector V prompt .

5. The industrial time series forecasting method enhanced by integrating expert knowledge according to claim 1 is characterized in that: The Step 4 is specifically as follows: Embed the prompt into vector V prompt , text embedding vector V text and time series Patch embedding P time Concatenate or fuse to form a comprehensive input representation X fusion =[V prompt ; V text ;P time ].

6. The industrial time series forecasting method enhanced by integrating expert knowledge according to claim 1 is characterized in that: The Step 5 is specifically as follows: The comprehensive input represents X fusion Input to the generative pre-trained Transformer; Use the decoder's autoregressive mechanism to generate predictions for future time steps

Images