A pre-training method for multivariate time series forecasting tasks in process industries

Abstract

The present invention discloses a pre-training method for multivariate time series prediction tasks in process industries, which relates to the field of time prediction technology, and specifically includes S1: data division and pre-processing; S2: working condition identification and feature extraction; S3: weight matrix calculation; S4: hidden layer representation generation; S5: pre-training model output; S6: pre-training model training; S7: pre-training model output. The method proposed in the present invention is applied to an attention-based graph convolutional neural network model (ASTGCN). Through verification on an industrial data set of a distillation section provided by a chemical plant, it is found that the root mean square error of the pre-trained model that takes working condition identification into consideration is reduced by about 15% compared to the model that is not pre-trained. In addition, compared to the pre-trained model that does not take working condition identification into consideration, its root mean square error is reduced by about 5%.

Description

Technical Field

[0001] The present invention relates to the technical field of time prediction, and in particular to a pre-training method for multivariate time series prediction tasks in process industries. Background Art

[0002] The pre-training method is a method that trains on a large amount of unlabeled data to enable the model to extract abstract features and key information of the data, so that it can perform better in subsequent tasks.

[0003] In the prior art, Patent 1 with Patent Publication No. CN115688871A proposes a multivariate time series pre-training method based on mask reconstruction. This method masks the fragments in the input time series data, and then reconstructs the remaining information using the codec architecture; the goal of the pre-training process is to minimize the mean absolute error between the original sequence and the reconstructed sequence; in this way, the model is forced to learn the periodic information and trends in the input sequence, providing better data representation for subsequent prediction tasks.

[0004] In the prior art, Patent 2 with Patent Publication No. CN111950810A proposes a pre-training method based on self-evolution, which mainly introduces N univariate time series into the pre-training model, and establishes a linear autoregressive model through a differential feature-based construction method; in the model construction process, multi-order differential information is explicitly integrated into the pre-training model, so that it can automatically learn the trend characteristics and correlations of time series data; this enhances the data representation ability, thereby further improving the performance of the prediction model.

[0005] However, the disadvantage of the above-mentioned prior art is that it cannot adapt well to the characteristics of high lag, high nonlinearity, weak periodicity and changes with working conditions of multivariate time series in process industries. The pre-training method based on mask reconstruction proposed in Patent 1 performs poorly when processing data with weak periodicity; because this method relies on the periodic pattern in the sequence during the reconstruction process, the periodicity in process industry data is relatively weak, and it is difficult for the model to effectively capture periodic information. It may cause the model to pay too much attention to noise, reducing the performance of subsequent models. The preprocessing method proposed in Patent 2 is limited to linear relationships. This method is based on a linear autoregressive model, and therefore cannot well capture the highly nonlinear characteristics of variables in processes such as distillation and synthesis in process industries.

[0006] In addition, the existing technologies have not taken into account the high time lag characteristics of process industry data. High time lag means that the influence between variables in the process industry may take a considerable time to be transmitted or manifested, so the changes of some variables may lag behind other variables in time. This lag phenomenon usually originates from the control link of the process industry and the integral nature or transfer process of the physical process. In the prediction task, it is crucial to accurately capture this time lag information. In addition, industrial processes are usually affected by a variety of factors, such as operating conditions, environmental parameters, and external factors, which are collectively referred to as operating conditions. Changes in operating conditions will affect the operation and performance of industrial processes, which may in turn affect the time lag of variables, autocorrelation coefficients, and changes in the dependencies between variables. Pre-training methods that do not consider operating condition identification may not be able to effectively represent industrial data with variable operating conditions, limiting their application in process industries. Summary of the invention

[0007] The purpose of the present invention is to provide a pre-training method for multivariate time series prediction tasks in process industries, so as to solve the problem that the existing pre-training methods proposed in the above background technology have insufficient feature extraction capabilities when processing process industry data with high time lag, weak periodicity and variable operating conditions.

[0008] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme:

[0009] A pre-training method for multivariate time series prediction tasks in process industries, specifically a pre-training method for multivariate time series prediction tasks in process industries integrating working condition identification, is provided. The steps of use are as follows:

[0010] S1: Data division and preprocessing. First, the multivariate time series data of the process industry is preprocessed as necessary and divided into process variable (PV) series, control variable (OP) series and disturbance variable (DV) series according to the variable type. According to the needs of the task, the data set is divided into the historical input sequence X and the corresponding output sequence Y;

[0011] S2: Operating condition identification and feature extraction. The historical input process variable (PV) sequence, control variable (OP) sequence and disturbance variable (DV) sequence are input into the operating condition identification module. In this module, the implicit representation G of the operating condition characteristics is generated through the combination of one-dimensional convolution operation on the time dimension, global self-attention operation on the time dimension, one-dimensional convolution operation on the space dimension, and two-dimensional convolution operation on the space-time dimension;

[0012] S3: Weight matrix calculation. The implicit representation G of the operating condition characteristics is matrix multiplied with the learnable weight matrix W and added to the bias block Bias element-wise to obtain a weight matrix Λ of the same size as the process variable (PV) sequence and the control variable (OP) sequence;

[0013] S4: Hidden layer representation generation. Use the weight matrix Λ to perform element-by-element product (Hadamard product) on the process variable (PV) sequence and the control variable (OP) sequence to obtain the hidden layer representation H of the original data;

[0014] S5: Pre-trained model output. The hidden layer represents H through a one-dimensional convolution operation on the time dimension and a fully connected layer to obtain the predicted output sequence

[0015] S6: Pre-trained model training. By minimizing the predicted output sequence The mean absolute error between the actual output sequence Y and the learnable parameters in the model are trained using the back-propagation algorithm;

[0016] S7: Pre-training model output. The hidden layer representation H obtained in step S4 is used as the output of the pre-training model and as the input of the subsequent multivariate time series prediction model;

[0017] Furthermore, in the step S1, the collected chemical process data is a discrete multivariate time data series. In order to organize the data into input data acceptable to the model, the data needs to be sampled, aligned, and standardized for preprocessing;

[0018] Furthermore, the sampling alignment means that since different sensors record data at different frequencies, in order to ensure that they have the same time interval and sequence length, the time series data needs to be interpolated;

[0019] Furthermore, the interpolation operation is a first-order spline interpolation, which generates new data points by linearly interpolating known data points, thereby filling the missing values ​​between time series, so that the time series of all variables have the same sampling interval and sequence length. After sampling is completed, the time series can be represented as [x1, x2, ..., x t ,...,x T ],in T represents the length of the time series, which is defined as each discrete time point in the sampled data as a time step, x t represents the input sample of the tth time step. The input sample of each time step contains n variables, where n is the number of sensor variables;

[0020] Furthermore, in step S1, PV represents process variable, OP represents control variable, and DV represents disturbance variable. These variables correspond to the operating state, control parameters and external disturbance of the system in the process industry. In the multivariate time series prediction task of the process industry, usually only the process variable is used as the prediction target;

[0021] Furthermore, in step S1, the process industry time series data set is represented as Where T represents the total length of the time series, p represents the number of process variables, c represents the number of control variables, and d represents the number of interference variables. After classification according to variable type, the process variable (PV) sequence is obtained: Control variable (OP) sequence and the disturbance variable (DV) series

[0022] Furthermore, in step S6, the learnable parameters in the model include the convolution kernel of the temporal one-dimensional convolution in step S2, the weight matrix of the temporal self-attention module in step S2, the convolution kernel of the spatial one-dimensional convolution in step S2, the convolution kernel of the spatiotemporal two-dimensional convolution in step S2, the weight matrix W in step S3, the bias block Bias in step S3, the convolution kernel of the temporal one-dimensional convolution in step S5, and the weight of the fully connected layer in step S5.

[0023] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0024] The method proposed in the present invention is applied to the attention-based graph convolutional neural network model (ASTGCN). Through verification on the industrial data set of the distillation section provided by a chemical plant, it is found that the root mean square error of the pre-trained model after considering the working condition identification is reduced by about 15% compared with the model without pre-training. In addition, compared with the pre-trained model without considering the working condition identification, its root mean square error is reduced by about 5%. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] Figure 1 It is a structural schematic diagram of the pre-training method of the present invention;

[0027] Figure 2 It is a schematic diagram of time series sampling alignment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] The present invention relates to a pre-training method for multivariate time series prediction tasks in process industries. The method proposed in this patent is applied to an attention-based graph convolutional neural network model (ASTGCN). It is verified on an industrial data set of a distillation section provided by a chemical plant. The root mean square error of the pre-trained model that considers working condition identification is reduced by about 15% compared to the model that is not pre-trained. In addition, compared to the pre-trained model that does not consider working condition identification, its root mean square error is reduced by about 5%. The details are as follows:

[0030] 1. Dataset preparation:

[0031] The industrial data and equipment parameters used in this experiment come from a methanol distillation unit in a certain factory. The unit adopts a three-tower methanol distillation process, which mainly includes a pre-distillation tower, a pressure tower, and an atmospheric tower. 23 sensor variables are selected for prediction in the selected section. These include 9 PV variables and their corresponding 9 OP variables and 5 DV variables. Preprocess the data:

[0032] (1) Time series sampling alignment: Since different sensors may record data at different frequencies and the data lengths may also be different, in order to perform subsequent analysis and model training, these time series data need to be interpolated to ensure that they have the same time interval and sequence length. A commonly used interpolation method is first-order spline interpolation, which generates new data points by linearly interpolating known data points to fill in the missing values ​​between time series so that the time series of all variables have the same sampling interval and sequence length. After sampling is completed, the time series can be represented as [x1, x2, ..., x t ,...,x T ],in T represents the length of the time series, which is defined as each discrete time point in the sampled data as a time step, x t represents the input sample of the tth time step, and the input sample of each time step contains n variables, where n is the number of sensor variables. Such sampling alignment operation helps to maintain the consistency and comparability of data.

[0033] (2) Data standardization: The numerical ranges of different variable sensors may vary greatly, which may make it difficult to train the model. In order to reduce the impact of different features on model training, data standardization is a common preprocessing method. The goal of data standardization is to convert each feature into a standard distribution with the same scale and range. Among them, a commonly used standardization method is to calculate the Z-score of each feature. The standardization formula is as follows:

[0034]

[0035] Among them, Z is the standardized value, X is the original data, μ is the mean of the feature, and σ is the standard deviation of the feature. By applying this formula, the data can be converted into a standard normal distribution with a mean of 0 and a standard deviation of 1, thereby eliminating the scale differences between different features, which is beneficial to the stability and accuracy of model training.

[0036] (3) Dataset division: In order to verify the scalability of the model, the dataset needs to be divided into training set, validation set and test set. The common division ratio is 70% of the data for training, 20% of the data for validation, and 10% of the data for testing. Through such division, the rationality of the dataset can be ensured and used for model training, tuning and evaluation. In this example, the dataset contains a total of 18,146 sets of time series data. After sampling, according to the division ratio, the sizes of the training set, validation set and test set are 12,702, 3,629 and 1,815 sets respectively. This division method can provide sufficient data samples for model training and verification, and ensure the independence of the test set, so as to accurately evaluate the generalization ability of the model.

[0037] 2. Implementation process:

[0038] When applying the pre-training method of the present invention to solve the problem of multivariate time series prediction task in process industry, the following steps are performed:

[0039] Step S1: Data division and preprocessing: According to the variable type, the data is divided into process variable (PV) sequence, control variable (OP) sequence and disturbance variable (DV) sequence. According to the task requirements, the data set is divided into historical input sequence X and corresponding output sequence Y.

[0040] Step S2: Operating condition identification and feature extraction: The process variable (PV) sequence, the controlled variable (OP) sequence and the disturbance variable (DV) sequence are input into the operating condition identification module. In the operating condition identification module, the implicit representation G of the operating condition features is generated through operations such as one-dimensional convolution, global self-attention, spatial one-dimensional convolution, spatiotemporal two-dimensional convolution and fully connected layer.

[0041] Step S3: Weight matrix calculation: Perform matrix multiplication operation on the implicit representation G of the operating condition characteristics and the learnable weight matrix W, and add it to the bias block Bias element level to obtain the weight matrix Λ, which has the same size as the process variable (PV) sequence and the control variable (OP) sequence.

[0042] Step S4: Hidden layer representation generation: Use the weight matrix Λ to perform element-by-element product (Hadamard product) on the process variable (PV) sequence and the control variable (OP) sequence to obtain the hidden layer representation H of the original data.

[0043] Step S5: Pre-trained model output: pass the hidden layer representation H through a one-dimensional convolution and a fully connected layer to obtain the predicted output sequence

[0044] Step S6: Pre-trained model training: predict output sequence by minimizing The mean absolute error between the output sequence Y and the actual output sequence Y is used to train the learnable parameters in the model using the back-propagation algorithm.

[0045] Step S7: Pre-training model output: The hidden layer representation H obtained in step S4 is used as the output of the pre-training model and as the input of the subsequent multivariate time series prediction model.

[0046] The hidden layer representation H of the pre-trained model in step S7 is input into ASTGCN (attention-based graph convolutional neural network model), and its model parameters and training parameters are as follows:

[0047]

[0048]

[0049] 3. Evaluation indicators:

[0050] Assume that the model outputs the predicted value of the future H-step time series as The true value is The values ​​of the root mean square error RMSE of the multi-step prediction can be calculated by the following formula:

[0051]

[0052] Non-pretrained model: RMSE: 1.4040;

[0053] Pre-trained model without considering working condition identification: RMSE: 1.1500;

[0054] Pre-trained model considering working condition identification: RMSE: 1.1194.

[0055] 4. Summary:

[0056] 1. The experiment was conducted on an industrial data set of the distillation section provided by a chemical plant. Based on the 60-step time history data, the iterative multi-step prediction task of predicting the next 20 steps, the root mean square error of the pre-trained model considering the working condition identification was reduced by about 15% compared with the model without pre-training. In addition, compared with the pre-trained model without considering the working condition identification, the root mean square error was reduced by about 5%;

[0057] 2. The above-mentioned weight matrix Λ can effectively capture the time lag information between variables by capturing the high time lag of time series data, so as to improve the performance of the model in high time lag data; at the same time, the above-mentioned operating condition identification module can adapt to the variable operating condition characteristics of process industry data, and improve the data representation learning ability of the model under different operating conditions by identifying the factors affecting the operating conditions.

[0058] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A pre-training method for multivariate time series prediction tasks in process industries, characterized in that: The following steps are involved: S1: Data division and preprocessing: Perform necessary preprocessing on the multivariate time series data of process industry and divide it into process variable series, control variable series and interference variable series according to variable type; divide the data set into historical input series X and corresponding output series Y according to the needs of the task; S2: Operating condition identification and feature extraction; the process variable sequence, control variable sequence and disturbance variable sequence of historical input are input into the operating condition identification module; in this module, the implicit representation G of the operating condition feature is generated through the combination of one-dimensional convolution operation on the time dimension, global self-attention operation on the time dimension, one-dimensional convolution operation on the space dimension and two-dimensional convolution operation on the space-time dimension; S3: Weight matrix calculation: perform matrix multiplication on the implicit representation G of the operating condition characteristics and the learnable weight matrix W and add them to the bias block Bias element-wise to obtain a weight matrix Λ with the same size as the process variable sequence and the control variable sequence; S4: Hidden layer representation generation: Use the weight matrix Λ to perform element-by-element product of the process variable sequence and the control variable sequence to obtain the hidden layer representation H of the original data; S5: Pre-trained model output; the hidden layer represents H through a one-dimensional convolution operation on the time dimension and a fully connected layer to obtain the predicted output sequence S6: Pre-trained model training; output sequence by minimizing prediction The mean absolute error between the actual output sequence Y and the learnable parameters in the model are trained using the back-propagation algorithm; S7: pre-training model output; the hidden layer representation H obtained in step S4 is used as the output of the pre-training model and as the input of the subsequent multivariate time series prediction model; In step S1, the collected chemical process data is a discrete multivariate time data series. In order to organize the data into input data acceptable to the model, the data needs to be sampled, aligned, and standardized for preprocessing; In step S1, PV represents process variable, OP represents control variable, and DV represents disturbance variable; These variables correspond to the operating status, control parameters and external disturbances of the system in the process industry. In the multivariate time series prediction task in the process industry, process variables are used as prediction targets.

2. The pre-training method for multivariate time series prediction tasks in process industries according to claim 1, characterized in that: The sampling alignment means that since different sensors record data at different frequencies, in order to ensure that they have the same time interval and sequence length, the time series data needs to be interpolated.

3. The pre-training method for multivariate time series prediction tasks in process industries according to claim 2, characterized in that: The interpolation operation is a first-order spline interpolation, which generates new data points by linearly interpolating known data points, thereby filling the missing values ​​between time series, so that the time series of all variables have the same sampling interval and sequence length; after sampling is completed, the time series can be expressed as [x1, x2, ..., x t ,...,x T ],in T represents the length of the time series, which is defined as each discrete time point in the sampled data as a time step, x t Represents the input sample of the tth time step. The input sample of each time step contains n variables, where n is the number of sensor variables.

4. The pre-training method for multivariate time series prediction tasks in process industries according to claim 1, characterized in that: In step S1, the process industry time series data set is represented as Where T represents the total length of the time series, p represents the number of process variables, c represents the number of control variables, and d represents the number of interference variables; After classification according to variable type, the process variable sequence is obtained Control variable sequence and the disturbance variable sequence 5. The pre-training method for multivariate time series prediction tasks in process industries according to claim 1, characterized in that: In step S6, the learnable parameters in the model include the convolution kernel of the temporal one-dimensional convolution in step S2, the weight matrix of the temporal self-attention module in step S2, the convolution kernel of the spatial one-dimensional convolution in step S2, the convolution kernel of the spatiotemporal two-dimensional convolution in step S2, the weight matrix W in step S3, the bias block Bias in step S3, the convolution kernel of the temporal one-dimensional convolution in step S5, and the weight of the fully connected layer in step S5.

Images