Time sequence state data multi-step prediction method, device and equipment

Abstract

The invention provides a time sequence state data multi-step prediction method, device and equipment. A time sequence state data multi-step prediction model comprises a time sequence state data encoder structure and a multi-step prediction decoder structure; the time sequence state data multi-step prediction model is constructed based on a Transform model architecture; the structure of the time sequence state data encoder is based on a Transform encoder model architecture, and a long short-term memory network model is inserted between a multi-head attention sub-layer and a feed-forward network sub-layer of the Transform encoder for construction. The multi-step prediction decoder structure is constructed on the basis of a Transform decoder model architecture, and comprises a plurality of Transform decoders. According to the invention, a long-short-term memory network model is introduced, and the prediction performance is enhanced by combining global modeling and local time sequence dependence capability. A multi-decoder structure is designed to process a multi-step prediction task, and the problem of error accumulation is effectively solved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a method, device and equipment for multi-step prediction of time series state data. Background Art

[0002] Industrial equipment generates a large amount of time series status data during operation, which is crucial for predicting equipment failures and optimizing maintenance strategies. Traditional prediction methods, such as the Autoregressive Integrated Moving Average Model (ARIMA) and machine learning models, have limited effectiveness in processing nonlinear and complex industrial data. Although time series data prediction methods based on recurrent neural networks, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), gated recurrent units (GRU), and convolutional neural networks (CNN), have made some progress, these methods still have the problem of error accumulation in multi-step prediction tasks, and it is difficult to effectively capture multi-dimensional and multi-time scale dependencies. Multi-step prediction requires that the model can predict the status of the equipment in multiple time steps in the future. However, the shortcomings of existing models in feature extraction, dependency modeling, and multi-task processing cannot meet the complex needs of industrial equipment time series status data prediction. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, apparatus and device for multi-step prediction of time series state data, which are used to solve the problems of insufficient feature extraction, difficulty in capturing dependencies and error accumulation in multi-task processing in the existing models in the process of predicting time series state data.

[0004] Based on the above problems, in a first aspect, an embodiment of the present disclosure provides a multi-step prediction method for time series state data, comprising: Using a convolutional neural network model to extract features from the time series state data to obtain first feature data; Dividing the first feature data according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data to obtain a plurality of second feature data; The multi-step prediction model of time series state data includes: a time series state data encoder structure and a multi-step prediction decoder structure; the multi-step prediction model of time series state data is constructed based on the Transformer model architecture; the time series state data encoder structure is constructed based on the Transformer encoder model architecture, and a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder; the multi-step prediction decoder structure is constructed based on the Transformer decoder model architecture, including multiple Transformer decoders; Input the plurality of second feature data into the time series state data multi-step prediction model, and extract features from the second feature data according to the time series state data encoder structure to obtain third feature data; Each of the third feature data is predicted in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and the first prediction data are spliced ​​to obtain multi-step prediction data corresponding to the first quantity.

[0005] In combination with the first aspect, in a possible implementation manner, the extracting features of the time series state data using a convolutional neural network model to obtain first feature data includes: Based on the convolutional neural network model, a multi-channel one-dimensional convolutional neural network model including a preset number of convolution kernels is constructed; According to the multi-channel one-dimensional convolutional neural network model, convolution and pooling are performed on the time series state data respectively, and the output data of each channel is compressed into one-dimensional data for a preset number of times to obtain first feature data; Among them, for the convolution and pooling operations, the convolution kernel length is set to a, the maximum pooling kernel length is set to b, and the number of features included in the timing state data is set to n, and a+b=n+1 is satisfied, where a, b∈{1, 2, ...n}.

[0006] In combination with the first aspect, in a possible implementation manner, the first feature data is segmented according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data to obtain a plurality of second feature data, including: Determining a first step length according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data; The first feature data is segmented according to the first step length to obtain second feature data corresponding to the first quantity.

[0007] In combination with the first aspect, in a possible implementation, the following method is adopted based on the Transformer encoder model architecture, and a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder to construct the temporal state data encoder structure: Based on the Transformer encoder model architecture, setting a first preset number of cascaded Transformer encoder units; For each layer of the Transformer encoder unit, insert a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer to determine the temporal state data encoder unit of the first preset number of cascade layers; Extracting features from the second feature data according to the timing state data encoder structure to obtain third feature data includes: According to the first-layer temporal state data encoder unit, feature extraction is performed on the second feature data to obtain output data of the first-layer feature extraction; For each layer of timing state data encoder units after the first layer, the output data obtained by the timing state data encoder units of the previous layer is used as the input of this layer for feature extraction, and the output data of the feature extraction of this layer is obtained, until the last layer of timing state data encoder units performs feature extraction to obtain the third feature data.

[0008] In combination with the first aspect, in a possible implementation manner, predicting each of the third feature data in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and concatenating the first prediction data to obtain multi-step prediction data corresponding to the first quantity, includes: According to the multi-step prediction decoder structure, each decoder in the multi-step prediction decoder structure is respectively provided with a Transformer decoder unit having a second preset number of cascade layers; Inputting the third feature data into the first layer of the Transformer decoder units corresponding to each decoder in the multi-step prediction decoder structure in time sequence for prediction, and obtaining output data of the first layer prediction; For each layer of Transformer decoder units after the first layer of each decoder, the output data obtained by the Transformer decoder unit of the previous layer is used as the input of the layer for prediction, and the predicted output data corresponding to the layer is obtained, until the Transformer decoder unit of the last layer performs prediction, and the first predicted data corresponding to the decoder is obtained; The first prediction data obtained by each decoder in the multi-step prediction decoder structure are spliced ​​to obtain multi-step prediction data corresponding to the first quantity.

[0009] In combination with the first aspect, in a possible implementation manner, the first feature data is segmented according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data to obtain a plurality of second feature data, including: The first feature data is segmented according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data, and a position code is inserted into each output data to obtain second feature data including the position code.

[0010] In combination with the first aspect, in a possible implementation manner, before extracting features from the time series state data using a convolutional neural network model, the method further includes: The collected data is smoothed and filtered to obtain the time series state data after removing the high-frequency noise.

[0011] In combination with the first aspect, in a possible implementation manner, the multi-step prediction model for time series state data is trained in the following manner: Obtain the data to be trained from the multi-step prediction data; Determine a loss function after a mean square error and a linear weighting as a first loss function, and determine a minimization optimization condition for the first loss function; Training is performed based on each step of prediction data in the data to be trained and the first loss function to obtain a multi-step prediction model for time series state data.

[0012] In a second aspect, a multi-step prediction device for time series state data is provided, comprising: A feature extraction module, used to extract features from the time series state data using a convolutional neural network model to obtain first feature data; A prediction module is used to divide the first feature data according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of time series state data to obtain multiple second feature data; wherein the multi-step prediction model of time series state data includes: a time series state data encoder structure and a multi-step prediction decoder structure; the multi-step prediction model of time series state data is constructed based on the Transformer model architecture; the time series state data encoder structure is based on the Transformer encoder model architecture, and a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder; the multi-step prediction decoder structure is constructed based on the Transformer decoder model architecture, including multiple Transformer decoders; the multiple second feature data are input into the multi-step prediction model of time series state data, and the second feature data is feature extracted according to the time series state data encoder structure to obtain third feature data; each of the third feature data is predicted in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and the first prediction data are spliced ​​to obtain multi-step prediction data corresponding to the first number.

[0013] In a third aspect, a multi-step prediction device for time series state data is provided, comprising: a multi-step prediction device for time series state data as described in the second aspect.

[0014] The beneficial effects of the embodiments of the present disclosure include: The present disclosure provides a method, device and equipment for multi-step prediction of time series state data, including: using a convolutional neural network model to extract features of time series state data to obtain first feature data; dividing the first feature data according to a first number of decoders in a multi-step prediction decoder structure in the multi-step prediction model of time series state data to obtain multiple second feature data; wherein the multi-step prediction model of time series state data includes: a time series state data encoder structure and a multi-step prediction decoder structure; the multi-step prediction model of time series state data is constructed based on a Transformer model architecture; the time series state data encoder structure is based on a Transformer encoder model architecture , insert a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder; the multi-step prediction decoder structure is constructed based on the Transformer decoder model architecture, including multiple Transformer decoders; multiple second feature data are input into the multi-step prediction model of the time series state data, and the second feature data is feature extracted according to the time series state data encoder structure to obtain third feature data; each third feature data is predicted according to the multi-step prediction decoder structure in time sequence to obtain first prediction data, and the first prediction data is spliced ​​to obtain multi-step prediction data corresponding to the first quantity. A multi-step prediction method for time series state data provided by the embodiment of the present disclosure provides rich input information for the Transformer encoder by using a convolutional neural network at the input end to perform convolution pooling operations on the time series state data. A long short-term memory network model is introduced into the Transformer encoder, and the global modeling and local time series dependency capabilities are combined to enhance the prediction performance. A multi-decoder structure is designed to process multi-step prediction tasks. Each decoder is responsible for the prediction of one time step, and the prediction of multiple future steps is regarded as multiple independent but related tasks, which effectively solves the problem of error accumulation in traditional models and improves the efficiency of multi-task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a multi-step prediction method for time series state data provided by an embodiment of the present disclosure; Figure 2 A structural diagram of a multi-step prediction model for time series state data provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of a multi-head attention mechanism provided for an embodiment of the present disclosure; Figure 4 A structural diagram of a multi-step prediction device for time series state data provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The embodiments of the present disclosure provide a method, device and apparatus for multi-step prediction of time series state data. The preferred embodiments of the present disclosure are described below in conjunction with the drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0017] The present disclosure provides a multi-step prediction method for time series state data. Figure 1 As shown, the following steps are included: S101, using a convolutional neural network model to extract features from the time series state data to obtain first feature data; S102, dividing the first feature data according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of the time series state data to obtain a plurality of second feature data; wherein the multi-step prediction model of the time series state data includes: a time series state data encoder structure and a multi-step prediction decoder structure; the multi-step prediction model of the time series state data is constructed based on the Transformer model architecture; the time series state data encoder structure is based on the Transformer encoder model architecture, and a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder; the multi-step prediction decoder structure is constructed based on the Transformer decoder model architecture, and includes a plurality of Transformer decoders; S103, inputting a plurality of second feature data into a time series state data multi-step prediction model, performing feature extraction on the second feature data according to a time series state data encoder structure, and obtaining third feature data; S104, predicting each third feature data in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and concatenating the first prediction data to obtain multi-step prediction data corresponding to the first quantity.

[0018] The disclosed embodiments can be applied to the industrial field or to the multi-step prediction of various time series state data in the non-industrial field to achieve cross-field application. For example, in the industrial field, a large amount of time series state data generated by industrial equipment during operation can refer to a series of data points that record the operation status of the equipment in chronological order, reflecting various parameters of the equipment at different time points, including the temperature, pressure, vibration frequency, rotation speed, current, voltage, etc. of the equipment. These time series state data are crucial for predicting equipment failures and optimizing maintenance strategies. In traditional prediction methods, such as the Autoregressive Integrated Moving Average Model (ARIMA), it is a statistical model widely used in time series analysis and prediction. It is developed on the basis of the Autoregressive Moving Average Model (ARMA), and is mainly used to process non-stationary time series. However, the generation mechanism of the time series of this model is linear, and the effect is limited when processing nonlinear industrial data. When processing complex industrial data, machine learning models also face many limitations. Although time series data prediction methods based on recurrent neural networks, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), gated recurrent units (GRU), and convolutional neural networks (CNN) have made some progress, these methods still face the problem of error accumulation in multi-step prediction tasks, and it is difficult to effectively capture multi-dimensional and multi-time scale dependencies. Multi-step prediction requires that the model can predict the state of the equipment in multiple time steps in the future, and the prediction of different time steps can involve different tasks in the short and long term. This requires the model to have the ability of multi-task learning to improve the accuracy and reliability of the overall prediction. However, the existing technology still has deficiencies in feature extraction, dependency modeling, and multi-task processing, and cannot meet the complex needs of industrial equipment time series state data prediction.

[0019] In the disclosed embodiments, the convolutional neural network is a deep learning model specially designed for processing data with a grid structure, such as time series, image or audio data. It is mainly composed of a convolutional layer, a pooling layer and a fully-connected layer. Multiple feature extractors can be constructed through convolution and pooling operations to effectively extract feature information of the input data and reduce the size of the input matrix. In the data preprocessing stage, a convolutional neural network model can be used to extract features of the time series state data to obtain the first feature data. Features of different scales are extracted through different channels to provide rich input information for the multi-step prediction model of the time series state data. Figure 2 As shown, the multi-step prediction model of time series state data includes: a time series state data encoder structure 201 and a multi-step prediction decoder structure 202. In order to adapt to the format requirements of the multi-step prediction decoder structure 202 for the input data, a sliding time window can be used to segment the first feature data to obtain the second feature data corresponding to the number of decoders in the multi-step prediction decoder structure 202. The multi-step prediction model of time series state data is constructed based on the Transformer model architecture, wherein the time series state data encoder structure 201 is constructed based on the Transformer encoder model architecture, and a long short-term memory network model (LSTM, Long Short-Term Memorynetwork) is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder. The LSTM gated residual network is introduced into the Transformer encoder to better combine the global modeling ability of the Transformer model and the sequence modeling advantage of the LSTM, thereby enhancing the prediction performance. The multi-step prediction decoder structure 202 is constructed based on the Transformer decoder model architecture, including multiple Transformer decoders. The multi-decoder structure is designed to process multi-step prediction tasks.

[0020] Furthermore, a plurality of second feature data are input into the multi-step prediction model of the time series state data, and the second feature data are feature extracted using the time series state data encoder structure 201 to obtain third feature data. The multi-step prediction decoder structure 202 is used to predict each third feature data in time sequence to obtain first prediction data. Each decoder of the multi-step prediction decoder structure 202 is responsible for the prediction of one time step, and the prediction of the future w steps is regarded as w independent but related tasks, thereby effectively solving the problem of error accumulation in the traditional multi-step prediction method and improving the efficiency of multi-task processing. Finally, the first prediction data output by all decoders are spliced ​​together to obtain the prediction result of the future w steps.

[0021] In the embodiment of the present application, in the data preprocessing stage, a convolutional neural network is used to perform convolution and pooling operations on the time series state data. This process can effectively extract the key features in the data and provide rich and high-quality input information for the subsequent multi-step prediction model of the time series state data. Through the preprocessing of the convolutional neural network, not only the characterization ability of the data can be enhanced, but also the ability to capture complex patterns in the data can be greatly improved. For the construction of the multi-step prediction model of the time series state data, a long short-term memory network model is introduced in the Transformer encoder, combining the global modeling ability of the Transformer with the excellent capture ability of the LSTM for local time series dependencies, so that the model can perform a comprehensive and in-depth analysis of the data at different time scales, which can greatly enhance the prediction performance of the model, so that it can process complex time series data more accurately. For multi-step prediction tasks, a multi-decoder structure is designed. Each decoder is independently responsible for the prediction work of one time step, and the future multi-step prediction task is disassembled into multiple relatively independent tasks. It effectively avoids the problem of error accumulation and significantly improves the efficiency and accuracy of multi-task processing. By making efficient and accurate predictions for multiple time steps in the future, it can provide more reliable support for its application in industrial equipment.

[0022] In another embodiment of the present disclosure, in the above step S101, a convolutional neural network model is used to extract features from the time series state data to obtain first feature data, including: Step 1: Based on the convolutional neural network model, a multi-channel one-dimensional convolutional neural network model including a preset number of convolution kernels is constructed; Step 2: According to the multi-channel one-dimensional convolutional neural network model, the time series state data is convolved and pooled respectively, and the output data of each channel is compressed into one-dimensional data for a preset number of times to obtain the first feature data; Among them, for convolution and pooling operations, the convolution kernel length is set to a, the maximum pooling kernel length is set to b, and the number of features included in the time series state data is set to n, and a+b=n+1 is satisfied, where a, b∈{1, 2, ...n}.

[0023] In the disclosed embodiment, a convolutional neural network model (CNN) is used to extract features from time series state data, which can provide rich input information for the multi-step prediction model of time series state data. The CNN network essentially constructs multiple feature extractors through convolution and pooling operations, effectively extracts feature information of input data and reduces the size of the input matrix, and is often used to process time series, image or audio data. For step 1 above, in the adopted CNN model, the convolution kernel size can be between 3 and 5, and the pooling layer can adopt a maximum pooling strategy to retain key fluctuation characteristics when reducing the data dimension. For a given time series state data sequence , L represents the number of observation vectors, and n represents the size of the observation vector, that is, it contains n feature variables. Based on the convolutional neural network model, a multi-channel one-dimensional convolutional neural network model including a preset number of convolution kernels is constructed to perform multi-scale feature extraction on the time series state data. To ensure effective dimensionality reduction and retain key information, for convolution and pooling operations, the convolution kernel length is set to a, the maximum pooling kernel length is set to b, and a+b=n+1 is satisfied, where a, b∈{1, 2,…n}. For the above step 2, the constructed multi-channel one-dimensional convolutional neural network model is used to perform convolution and pooling operations on the time series state data, and the dimension of the state parameter output by each channel is compressed to one dimension. After the above convolution, pooling, and compression cycles are repeated to extract features for a preset number of times, a high-level feature sequence of multivariate state parameters is obtained, that is, the first feature data, recorded as Through the multi-channel one-dimensional convolutional neural network model, convolution and pooling operations on the time series state data can effectively extract the key features in the data.

[0024] In another embodiment of the present disclosure, in the above step S102, the first feature data is segmented according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of the time series state data to obtain a plurality of second feature data, including: Step 1, determining a first step length according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data; Step 2: Segment the first feature data according to the first step to obtain second feature data corresponding to the first quantity.

[0025] In the present disclosure, Figure 2 As shown, in order to adapt to the format requirements of the multi-step prediction decoder structure 202 in the multi-step prediction model of the time series state data for the input data, the first feature data can be segmented using a sliding time window to obtain the second feature data corresponding to the number of decoders in the multi-step prediction decoder structure 202. Exemplarily, for the above step 1, for the high-level feature sequence of the first feature data , the first step length of the sliding window input time can be set to k, and the sliding window output time step length can be set to w, corresponding to w decoders respectively. For the above step 2, the first feature data is segmented according to the first step length k to obtain w second feature data. For example, the second feature data obtained after segmentation, that is, the input subsequence of the time series state data encoder structure 201, can be expressed as , then the output subsequence corresponding to the multi-step prediction decoder structure 202 can be expressed as , where the first decoder output subsequence can be expressed as , the second decoder output subsequence can be expressed as , the w-th decoder output subsequence can be expressed as By segmenting and preprocessing the first characteristic data, an input data basis can be provided for the multi-step prediction model of time series state data.

[0026] In another embodiment of the present disclosure, Figure 2 As shown, the following method is used to construct a temporal state data encoder structure based on the Transformer encoder model architecture by inserting a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder: Based on the Transformer encoder model architecture, setting a Transformer encoder unit with a first preset number of cascade layers; For each layer of Transformer encoder units, insert a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer, and determine the temporal state data encoder units of the first preset number of cascade layers; In the above step S103, the second feature data is feature extracted according to the time series state data encoder structure to obtain the third feature data, including: Step 1: extract the second feature data according to the first-layer temporal state data encoder unit to obtain the output data of the first-layer feature extraction; Step 2: For each layer of timing state data encoder units after the first layer, the output data obtained by the timing state data encoder units of the previous layer is used as the input of this layer for feature extraction, and the output data of the feature extraction of this layer is obtained, until the last layer of timing state data encoder units performs feature extraction to obtain the third feature data.

[0027] In the present disclosure, Figure 2 As shown, an LSTM gated residual network is introduced into the Transformer encoder, and the LSTM layer is inserted between the multi-head attention mechanism and the feedforward network, combining global modeling with local temporal dependency capabilities to enhance prediction performance. In the process of constructing the temporal state data encoder structure 201, exemplarily, the temporal state data encoder structure 201 is composed of L layers of cascaded Transformer encoder units, each layer has the same structure, and parameters are not shared. For each layer of Transformer encoder unit, a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer to construct L layers of cascaded temporal state data encoder units. As shown Figure 3As shown in the figure, the self-attention mechanism is a core component of the Transformer model, which enables the model to capture the dependencies between different positions in the input sequence when processing sequence data. The self-attention mechanism calculates the correlation (or attention weight) of each position with all other positions, and performs a weighted summation on the input. The output can be expressed as:

[0028] in, is the encoder input, , and is the weight parameter, , and They are query, key and value matrices respectively. is the dimension of the key vector, Represents a normalization operation. The multi-head attention mechanism used by the Transformer model is an extension of the self-attention mechanism. By dividing the input sequence into multiple "heads", each head performs self-attention calculations independently. This allows the model to capture different types of dependencies at the same time. Its mathematical description is:

[0029] in, , is the number of longs, is a linear matrix. , and Respectively represent the original query ,key Sum Matrix projection to The attention mechanism is operated on the space. The LSTM layer is inserted between the multi-head attention sublayer and the feedforward network sublayer to form a gated residual network, which is used to process the features of the multi-head attention output. LSTM controls the flow of information through the forget gate, input gate and output gate, enhancing the modeling ability of temporal dependencies. The update equation of the LSTM unit is as follows:

[0030] in, For the Gate of Forgetfulness, is the input gate, is the output gate, is the candidate cell state, is the cell state, is the hidden state, Represents the combined input and the previous hidden state related, is the bias vector. is the timing input, represents the sigmoid activation function, Represents the tanh activation function, and ⊙ represents the multiplication of elements at the same position of two vectors. When using an encoder for feature extraction, the output of each layer of encoder units is used as the input of the next layer. For the above step 1, the first layer of temporal state data encoder units is used to perform feature extraction on the second feature data to obtain the output data of the first layer of feature extraction. For the above step 2, the output data of the first layer of feature extraction is input to the second layer of temporal state data encoder units for feature extraction. For each layer of temporal state data encoder units after the first layer, the output data obtained by the temporal state data encoder units of the previous layer is used as the input of this layer for feature extraction, and the output data of the feature extraction of this layer is obtained, until the last layer of temporal state data encoder units performs feature extraction, and the third feature data is obtained. Exemplarily, the processing process of the encoder unit is as follows:

[0031] in, , represents the multi-head attention mechanism, represents the residual connection and layer normalization operation, represents the LSTM network, represents a feed-forward neural network. is the input matrix of the encoder unit, is the intermediate output of the encoder unit. In one possible implementation, for the first layer of encoder units, ,in, represents the merging of two matrices in the feature dimension, It means adding the elements at the same position of two matrices. represents the position code, is the input matrix of the encoder unit. A gating mechanism is introduced into the Transformer encoder, that is, the LSTM gated residual network is inserted between the multi-head attention mechanism and the feedforward network, so as to better combine the global modeling ability of the Transformer and the sequence modeling advantage of the LSTM.

[0032] In another embodiment of the present disclosure, in the above step S104, each third feature data is predicted in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and the first prediction data are spliced ​​to obtain multi-step prediction data corresponding to the first quantity, including: Step 1: According to the multi-step prediction decoder structure, a Transformer decoder unit with a second preset number of cascade layers is set for each decoder in the multi-step prediction decoder structure; Step 2: Input the third feature data into the first-layer Transformer decoder units corresponding to each decoder in the multi-step prediction decoder structure in time sequence for prediction, and obtain the output data of the first-layer prediction; Step 3: For each layer of Transformer decoder units after the first layer of each decoder, the output data obtained by the Transformer decoder unit of the previous layer is used as the input of the layer for prediction, and the predicted output data corresponding to the layer is obtained, until the last layer of Transformer decoder units is predicted, and the first predicted data corresponding to the decoder is obtained; Step 4: concatenate the first prediction data obtained by each decoder in the multi-step prediction decoder structure to obtain multi-step prediction data corresponding to the first quantity.

[0033] In the present disclosure, Figure 2 As shown, each decoder in the multi-step prediction decoder structure 202 is provided with a multi-layer cascaded Transformer decoder unit, and the output of each layer of the decoder unit is used as the input of the next layer. The decoder completes the multi-step prediction by gradually transferring and interactively extracting features. For the above step 1, exemplarily, the multi-step prediction decoder structure 202 includes w decoders, each decoder includes L layers of cascaded decoder units, and the w decoders correspond to the prediction tasks of the future w steps, and the structure of each layer is the same, and the parameters are not shared. For the above step 2, the w third feature data are respectively input into the first layer of Transformer decoder units corresponding to each decoder in the multi-step prediction decoder structure 202 in time sequence for prediction, and the output data of the first layer prediction is obtained. For the above step 3, the output data of the first layer prediction is input into the second layer Transformer decoder unit for prediction. For each layer of Transformer decoder units after the first layer of each decoder, the output data obtained by the previous layer of Transformer decoder units is used as the input of the layer for prediction, and the predicted output data corresponding to the layer is obtained, until the last layer of Transformer decoder units are predicted, and the first prediction data corresponding to the decoder is obtained. Exemplarily, the first decoder has the following unit processing process:

[0034] in, represents the masked multi-head attention mechanism, represents the cross-head attention mechanism, represents a fully connected layer. is the input matrix of the first decoder, is the intermediate output of the first decoder, is the final output of the first decoder. In one possible implementation, for the first decoder, ,in, is the reference matrix of the first decoder. For the w-th decoder, its unit processing process is as follows:

[0035] in, is the input matrix of the w-th decoder, , are the intermediate outputs of the w-th decoder, is the final output of the w-th decoder. For the w-th decoder, ,in, is the reference matrix of the w-th decoder. For the above step 4, illustratively, all decoder outputs are concatenated together to obtain the prediction results for the next w steps: , .in, is the final output of the tth decoder. By designing a multi-decoder structure, the prediction task is decomposed, which effectively solves the problem of error accumulation in multi-step prediction and improves the efficiency of multi-task processing.

[0036] In another embodiment of the present disclosure, in the above step S102, the first feature data is segmented according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of the time series state data to obtain a plurality of second feature data, including: The first feature data is segmented according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data, and a position code is inserted into each output data to obtain second feature data including the position code.

[0037] In the present disclosure, Figure 2 As shown in the figure, after the first feature is segmented, the position code is inserted into each output data to obtain the second feature data including the position code. Since the Transformer model cannot directly capture the element order information of the input sequence. In order to handle the time series prediction problem, the fixed position encoding (PE, Positional Embedding) method is adopted to embed the encoded position information on the input sequence. The calculation formula of the position code is as follows:

[0038] Among them, pos represents the position of each token in the sequence. , represents the length of each position encoding vector (the dimension of the word embedding vector), represents the dimension index, , sin and cos functions are used to calculate the values ​​of even and odd dimensions in the position encoding vector, respectively. The relative positions of elements in the input sequence are determined by embedding the encoded position information on the input sequence. By adding position encoding, the model can predict the relationship pattern between elements at different positions, thereby making better predictions.

[0039] In another embodiment of the present disclosure, before using the convolutional neural network model to extract features from the time series state data, the method further includes: The collected data is smoothed and filtered to obtain the time series state data after removing the high-frequency noise.

[0040] In the disclosed embodiments, for example, a Savitzky-Golay filter can be used to perform smoothing filtering on the collected data, i.e., the input sequence. The Savitzky-Golay filter is a smoothing filtering technology widely used in signal processing, and its core principle is the least squares fitting based on local polynomial regression. The filter selects a window of fixed length, fits a polynomial model in each window using the least squares method, and replaces the original signal value with the predicted value of the center point of the model. The window size determines the number of data points of the fitted polynomial. A larger window can provide a better smoothing effect, while the order of the polynomial adjusts the complexity of the fitting curve, balancing the relationship between the smoothing effect and the feature fidelity. For example, the time series of the multivariate state parameters of the equipment collected by the database is expressed as ,in represents the number of observation vectors, Represents the size of the observation vector, that is, contains feature variables. When feature data is missing, interpolation is used to complete it. The smoothing process of the Savitzky-Golay filter is: ,

[0041] in, is the value of the smoothed sequence at time i, For the original sequence The value of the moment, is the weight coefficient obtained after least squares fitting polynomial, is the half length of the window, which is the same as the window length By smoothing and filtering the collected data, high-frequency noise can be removed while retaining the peak width, peak position and shape characteristics of the signal as much as possible.

[0042] In another embodiment of the present disclosure, the multi-step prediction model of time series state data is trained in the following manner: Step 1: Obtain the data to be trained from the multi-step prediction data; Step 2: Determine the loss function after the mean square error and linear weighting as the first loss function, and determine the minimization optimization condition of the first loss function; Step 3: Perform training based on the prediction data for each step in the data to be trained and the first loss function to obtain a multi-step prediction model for time series state data.

[0043] In the embodiment of the present disclosure, when training the multi-step prediction model for time series state data, in order to ensure the rationality and effectiveness of the prediction results of the multi-step prediction task, the loss function of the single-step prediction task is constrained when minimizing the multi-step prediction loss function. For the above step 1, the data to be trained can be obtained from the multi-step prediction data, or the preset data can be used as the data to be trained. For the above step 2, for the multi-step prediction task, the mean square error is used as the loss function. The loss function of the prediction task at the tth moment , defined as follows: ,

[0044] in, is the reference matrix of the tth decoder. Considering the different characteristics and objectives of the multi-step prediction task, a linear weighted combination of the first loss function is designed , defined as follows: ,

[0045] in, is the weight coefficient of the prediction task at the tth moment, satisfying , . Weight coefficient It can be adjusted according to the importance and performance of the task. When minimizing the first loss function, the loss function of the single-step prediction task is constrained. That is, the optimization condition of the first loss function satisfies:

[0046] in, is the upper bound of the loss function of the prediction task at the tth moment, , are the lower and upper bounds of the output of the t-th decoder, respectively. For the above step 4, training is performed according to each step of the prediction data in the training data and the first loss function and its constraints to obtain a multi-step prediction model for time series state data. The first loss function can intuitively reflect the degree of difference between the model prediction result and the true value. In the process of model training, it provides direction for parameter adjustment and guides model training. Setting the optimization condition of the first loss function can be used to train each step of the prediction data for multi-step prediction, thereby improving the accuracy of model prediction. During the training process, the multi-step prediction model for time series state data can also be evaluated according to prediction accuracy, generalization ability (stability) and convergence speed. The prediction accuracy of the model is evaluated by comparing the consistency or approximate error between the measured value and the predicted value. The statistical indicators that can be used include correlation coefficient, determination coefficient, mean absolute error, mean square error and root mean square error. The generalization ability of the model can be tested by multiple tests using cross-validation or appropriate allocation of data sets. The convergence speed of the model can be reflected by the overall response time of the algorithm.

[0047] Based on the same disclosed concept, the embodiments of the present disclosure also provide a device and equipment for multi-step prediction of time series state data. Since the principles of the problems solved by these devices and equipment are similar to those of the aforementioned multi-step prediction method for time series state data, the implementation of the device and equipment can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0048] The present disclosure provides a multi-step prediction device for time series state data, such as Figure 4 As shown, including: A feature extraction module 401 is used to extract features from the time series state data using a convolutional neural network model to obtain first feature data; The prediction module 402 is used to divide the first feature data according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of the time series state data to obtain multiple second feature data; wherein the multi-step prediction model of the time series state data includes: a time series state data encoder structure and a multi-step prediction decoder structure; the multi-step prediction model of the time series state data is constructed based on the Transformer model architecture; the time series state data encoder structure is based on the Transformer encoder model architecture, and a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder; the multi-step prediction decoder structure is constructed based on the Transformer decoder model architecture, including multiple Transformer decoders; the multiple second feature data are input into the multi-step prediction model of the time series state data, and the second feature data is feature extracted according to the time series state data encoder structure to obtain third feature data; each of the third feature data is predicted in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and the first prediction data are spliced ​​to obtain multi-step prediction data corresponding to the first number.

[0049] In another embodiment of the present disclosure, the feature extraction module 401 is used to construct a multi-channel one-dimensional convolutional neural network model including a preset number of convolution kernels based on the convolutional neural network model; According to the multi-channel one-dimensional convolutional neural network model, convolution and pooling are performed on the time series state data respectively, and the output data of each channel is compressed into one-dimensional data for a preset number of times to obtain first feature data; Among them, for the convolution and pooling operations, the convolution kernel length is set to a, the maximum pooling kernel length is set to b, and the number of features included in the timing state data is set to n, and a+b=n+1 is satisfied, where a, b∈{1, 2, ...n}.

[0050] In another embodiment of the present disclosure, the prediction module 402 is used to determine a first step length according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data; The first feature data is segmented according to the first step length to obtain second feature data corresponding to the first quantity.

[0051] In another embodiment of the present disclosure, the prediction module 402 is used to construct the temporal state data encoder structure by inserting a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder based on the Transformer encoder model architecture in the following manner: Based on the Transformer encoder model architecture, setting a Transformer encoder unit with a first preset number of cascade layers; For each layer of the Transformer encoder unit, insert a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer to determine the temporal state data encoder unit of the first preset number of cascade layers; The prediction module 402 is used to perform feature extraction on the second feature data according to the first-layer temporal state data encoder unit to obtain output data of the first-layer feature extraction; For each layer of timing state data encoder units after the first layer, the output data obtained by the timing state data encoder units of the previous layer is used as the input of this layer for feature extraction, and the output data of the feature extraction of this layer is obtained, until the last layer of timing state data encoder units performs feature extraction to obtain the third feature data.

[0052] In another embodiment of the present disclosure, the prediction module 402 is used to set a Transformer decoder unit with a second preset number of cascade layers for each decoder in the multi-step prediction decoder structure according to the multi-step prediction decoder structure; Inputting the third feature data into the first layer of the Transformer decoder units corresponding to each decoder in the multi-step prediction decoder structure in time sequence for prediction, and obtaining output data of the first layer prediction; For each layer of Transformer decoder units after the first layer of each decoder, the output data obtained by the Transformer decoder unit of the previous layer is used as the input of the layer for prediction, and the predicted output data corresponding to the layer is obtained, until the Transformer decoder unit of the last layer performs prediction, and the first predicted data corresponding to the decoder is obtained; The first prediction data obtained by each decoder in the multi-step prediction decoder structure are spliced ​​to obtain multi-step prediction data corresponding to the first quantity.

[0053] In another embodiment of the present disclosure, the prediction module 402 is used to divide the first feature data according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of the timing state data, insert a position code for each output data, and obtain a second feature data including the position code.

[0054] In another embodiment of the present disclosure, the feature extraction module 401, before extracting features from the time series state data using the convolutional neural network model, is further used to: The collected data is smoothed and filtered to obtain the time series state data after removing the high-frequency noise.

[0055] In another embodiment of the present disclosure, the prediction module 402 is used to train the multi-step prediction model of the time series state data in the following manner: Obtain the data to be trained from the multi-step prediction data; Determine a loss function after a mean square error and a linear weighting as a first loss function, and determine a minimization optimization condition for the first loss function; Training is performed based on each step of prediction data in the data to be trained and the first loss function to obtain a multi-step prediction model for time series state data.

[0056] An embodiment of the present disclosure provides a time series state data multi-step prediction device, comprising: a time series state data multi-step prediction device as described in any of the above embodiments.

[0057] Through the description of the above implementation methods, those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0058] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present disclosure.

[0059] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the present embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0060] The serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

[0061] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A multi-step prediction method for time series state data, characterized in that: include: Using a convolutional neural network model to extract features from the time series state data to obtain first feature data; Dividing the first feature data according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data to obtain a plurality of second feature data; The multi-step prediction model of time series state data includes: a time series state data encoder structure and a multi-step prediction decoder structure; the multi-step prediction model of time series state data is constructed based on the Transformer model architecture; the time series state data encoder structure is constructed based on the Transformer encoder model architecture, and a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder; the multi-step prediction decoder structure is constructed based on the Transformer decoder model architecture, including multiple Transformer decoders; Input the plurality of second feature data into the time series state data multi-step prediction model, and extract features from the second feature data according to the time series state data encoder structure to obtain third feature data; Each of the third feature data is predicted in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and the first prediction data are spliced ​​to obtain multi-step prediction data corresponding to the first quantity.

2. The method according to claim 1, characterized in that The method of extracting features from the time series state data using a convolutional neural network model to obtain first feature data includes: Based on the convolutional neural network model, a multi-channel one-dimensional convolutional neural network model including a preset number of convolution kernels is constructed; According to the multi-channel one-dimensional convolutional neural network model, convolution and pooling are performed on the time series state data respectively, and the output data of each channel is compressed into one-dimensional data for a preset number of times to obtain first feature data; Among them, for the convolution and pooling operations, the convolution kernel length is set to a, the maximum pooling kernel length is set to b, and the number of features included in the timing state data is set to n, and a+b=n+1 is satisfied, where a, b∈{1, 2, ...n}.

3. The method according to claim 1, characterized in that The first feature data is segmented according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of the time series state data to obtain a plurality of second feature data, including: Determining a first step length according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data; The first feature data is segmented according to the first step length to obtain second feature data corresponding to the first quantity.

4. The method according to claim 1, characterized in that The following method is used to construct the temporal state data encoder structure based on the Transformer encoder model architecture by inserting a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder: Based on the Transformer encoder model architecture, setting a Transformer encoder unit with a first preset number of cascade layers; For each layer of the Transformer encoder unit, insert a long short-term memory network model between the multi-head attention sublayer and the feedforward network sublayer to determine the temporal state data encoder unit of the first preset number of cascade layers; Extracting features from the second feature data according to the timing state data encoder structure to obtain third feature data includes: According to the first-layer temporal state data encoder unit, feature extraction is performed on the second feature data to obtain output data of the first-layer feature extraction; For each layer of timing state data encoder units after the first layer, the output data obtained by the timing state data encoder units of the previous layer is used as the input of this layer for feature extraction, and the output data of the feature extraction of this layer is obtained, until the last layer of timing state data encoder units performs feature extraction to obtain the third feature data.

5. The method according to claim 1, characterized in that The method of predicting each of the third feature data in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and splicing the first prediction data to obtain multi-step prediction data corresponding to the first quantity, includes: According to the multi-step prediction decoder structure, each decoder in the multi-step prediction decoder structure is respectively provided with a Transformer decoder unit having a second preset number of cascade layers; Inputting the third feature data into the first layer of the Transformer decoder units corresponding to each decoder in the multi-step prediction decoder structure in time sequence for prediction, and obtaining output data of the first layer prediction; For each layer of Transformer decoder units after the first layer of each decoder, the output data obtained by the Transformer decoder unit of the previous layer is used as the input of the layer for prediction, and the predicted output data corresponding to the layer is obtained, until the Transformer decoder unit of the last layer performs prediction, and the first predicted data corresponding to the decoder is obtained; The first prediction data obtained by each decoder in the multi-step prediction decoder structure are spliced ​​to obtain multi-step prediction data corresponding to the first quantity.

6. The method according to claim 1, characterized in that The first feature data is segmented according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of the time series state data to obtain a plurality of second feature data, including: The first feature data is segmented according to a first number of decoders in a multi-step prediction decoder structure in a multi-step prediction model of time series state data, and a position code is inserted into each output data to obtain second feature data including the position code.

7. The method according to claim 1, characterized in that Before the feature extraction of the time series state data using the convolutional neural network model, the method further includes: The collected data is smoothed and filtered to obtain the time series state data after removing the high-frequency noise.

8. The method according to claim 1, characterized in that The multi-step prediction model of the time series state data is trained in the following manner: Obtain the data to be trained from the multi-step prediction data; Determine a loss function after a mean square error and a linear weighting as a first loss function, and determine a minimization optimization condition for the first loss function; Training is performed based on each step of prediction data in the data to be trained and the first loss function to obtain a multi-step prediction model for time series state data.

9. A multi-step prediction device for time series state data, characterized in that: include: A feature extraction module, used to extract features from the time series state data using a convolutional neural network model to obtain first feature data; A prediction module is used to divide the first feature data according to the first number of decoders in the multi-step prediction decoder structure in the multi-step prediction model of time series state data to obtain multiple second feature data; wherein the multi-step prediction model of time series state data includes: a time series state data encoder structure and a multi-step prediction decoder structure; the multi-step prediction model of time series state data is constructed based on the Transformer model architecture; the time series state data encoder structure is based on the Transformer encoder model architecture, and a long short-term memory network model is inserted between the multi-head attention sublayer and the feedforward network sublayer of the Transformer encoder; the multi-step prediction decoder structure is constructed based on the Transformer decoder model architecture, including multiple Transformer decoders; the multiple second feature data are input into the multi-step prediction model of time series state data, and the second feature data is feature extracted according to the time series state data encoder structure to obtain third feature data; each of the third feature data is predicted in time sequence according to the multi-step prediction decoder structure to obtain first prediction data, and the first prediction data are spliced ​​to obtain multi-step prediction data corresponding to the first number.

10. A multi-step prediction device for time series state data, characterized in that: It comprises a multi-step prediction device for time series state data as described in claim 9.

Images