Time sequence prediction method and device based on deep neural network, equipment and medium

Abstract

The invention provides a time sequence prediction method and device based on a deep neural network, equipment and a medium, and the method comprises the steps: carrying out the normalization processing of input load data through a data loading module, and converting the representation of the data into vector representation; the generated vector representation is input into an encoder, the encoder processes time sequence data through multi-scale cavity attention operation and distillation operation, and key feature information is extracted; inputting the key feature information into a decoder, and further processing the key feature information by the decoder through feature extraction and compression operation; the features output by the decoder are input into a final output layer, and the final output layer converts the features into predicted values through an activation function. According to the invention, the problems of low time sequence prediction speed and low time sequence prediction data performance caused by the fact that an effective time sequence prediction method cannot be provided in the prior art can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of time series analysis, which is a branch of statistics and machine learning, and particularly relates to a time series prediction method, device, equipment and medium based on a deep neural network. Background Art

[0002] Time series forecasting technology predicts future trends and developments by analyzing past data. It has a wide range of applications, such as photovoltaic forecasting, electricity load forecasting, and meteorological service forecasting. However, due to high computational complexity and noise interference, general time series forecasting technology cannot be quickly and effectively applied to time series forecasting. A common practice for time series forecasting data is to use time series neural networks to process time series data. Traditional time series forecasting algorithms include exponential smoothing, state space models (SSM), and simple neural networks (Neural Network-based Approaches). They are all suitable for processing data with dependencies, but these methods bring many side effects, such as increased noise and computational complexity of the model.

[0003] In response to the above problems, the prior art has proposed a time series prediction method based on deep learning, such as: Ashish Vaswani et al. proposed a method that completely relies on self-attention to calculate the representation of input and output (Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need [J]. arXiv, 2017. DOI: 10.48550 / arXiv.1706.03762.) which can calculate the dependency between any two positions in the entire input sequence in one step, and directly capture the global dependency (without relying on step-by-step propagation). At the same time, the attention mechanism is also used in the encoder to improve the ability to capture long-distance dependencies; Zhou Haoyi et al. proposed a variant of the self-attention mechanism, the informer (Zhou H, Zhang S, Peng J, et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting[J].2020.DOI:10.48550 / arXiv.2012.07436.) combines the self-attention mechanism and the Transformer model, which can efficiently process time series data. At the same time, a unique sparsity design is adopted to reduce the computational complexity of the model, enabling it to handle large-scale data sets. There is also a solution proposed by Vukasin Bozic et al. to explore shallow feedforward neural networks as an alternative to the attention layer in Transformer (Bozic V, Dordevic D, Coppola D, et al. Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers[J].arxivpreprint arxiv:2311.10642,2023.) The key elements of the attention mechanism in Transformer are replaced by shallow feedforward networks, and these networks are trained through knowledge distillation. These algorithms reduce the complexity of the model to a certain extent, but the prediction accuracy will decrease.

[0004] Therefore, a new time series prediction method is urgently needed to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a time series prediction method, device, equipment and medium based on deep neural network, aiming to solve the problem that the prior art cannot provide an effective time series prediction method, resulting in slow time series prediction speed and low time series prediction data performance.

[0006] A time series prediction method based on a deep neural network comprises the following steps:

[0007] (1) normalizing the input load data through a data loading module and converting the representation into a vector representation, wherein the generation of the vector representation includes generating a corresponding position vector for each position;

[0008] (2) The vector representation generated in step (1) is input into the encoder, which processes the time series data through multi-scale hole attention operation and distillation operation to extract key feature information;

[0009] (3) inputting the key feature information extracted by the encoder in step (2) into the decoder, and the decoder further processes the key feature information through feature extraction and compression operations, and the decoder introduces a masked multi-scale hole attention mechanism;

[0010] (4) The features output by the decoder in step (3) are input into the final output layer, and the final output layer converts the features into prediction values ​​through an activation function.

[0011] Furthermore, the data loading module includes: a submodule for normalizing the input load data and generating a vector representation that is convenient for model processing.

[0012] Furthermore, the position vector is generated based on sine and cosine functions and used in conjunction with the vector representation generated in step (1).

[0013] Furthermore, the multi-scale atrous attention operation uses atrous convolution kernels of different scales to extract local and global feature information from the vector representation generated in step (1).

[0014] Furthermore, the distillation operation performs multi-level extraction on the feature information obtained by the multi-scale void attention operation through information distillation technology, removes redundant features and parameters, and compresses long-sequence feature information into short-sequence feature information.

[0015] Furthermore, the decoder further comprises:

[0016] A feature compression module, used for performing multi-level compression on the feature information encoded in step (2);

[0017] The mask module is used to introduce a masked multi-scale hole attention mechanism, which masks future time step information by adding a mask operation to the multi-scale hole attention mechanism.

[0018] Furthermore, the activation function is a linear activation function or a ReLU activation function, which is used to convert the feature information output by the decoder in step (3) into a nonlinear prediction value

[0019] A time series prediction device based on a deep neural network, comprising:

[0020] A data loading module, used for normalizing the input load data and converting its representation into a vector representation, wherein the generation of the vector representation includes generating a corresponding position vector for each position;

[0021] The encoder receives the vector representation and processes the time series data through multi-scale hole attention and distillation operations to extract key feature information.

[0022] A decoder, used for further processing key feature information through feature extraction and compression operations, wherein the decoder introduces a masked multi-scale hole attention mechanism;

[0023] The final output layer is used to convert the output of the decoder into a predicted value through an activation function.

[0024] A time series prediction device based on a deep neural network includes a processor and a memory, wherein program instructions are stored in the memory, and when the program instructions are executed, the processor executes the method described.

[0025] A computer-readable storage medium stores program instructions, and the program instructions implement the method when executed by a processor.

[0026] The present invention converts the input data representation into a vector representation through an input module, and then generates a position vector for each position of the input data. The encoder deeply mines and encodes the key feature information in the data. The decoder further extracts and compresses features on the basis of the encoder, and introduces a masked multi-scale void attention mechanism to ensure that the model only relies on historical and current information during the prediction process, effectively avoiding illegal leakage of future information. The final output layer, as the end of the model, is responsible for converting the output result of the decoder into the final prediction value through an appropriate activation function. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of a time series prediction method based on a deep neural network provided by the present invention.

[0028] Figure 2A schematic diagram of the structure of a time series prediction device based on a deep neural network provided by the present invention.

[0029] Figure 3 A schematic diagram of the structure of a time series prediction device based on a deep neural network provided by the present invention. DETAILED DESCRIPTION

[0030] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in combination with specific implementation methods. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0031] like Figure 1 As shown, a flowchart of a time series prediction method based on a deep neural network provided by an embodiment of the present invention includes the following steps:

[0032] Step S101, the data loading module must first clarify the data source, such as the photovoltaic power station monitoring system or power grid data, and store it securely locally. Subsequently, through careful data inspection, the accuracy of the data scale, type and timestamp is confirmed, the data format is unified, and problems such as noise, missing values ​​and outliers are identified and solved. In the data cleaning stage, the missing values ​​are flexibly processed by filling or deleting methods according to the data characteristics to ensure the consistency of the time series data in the time dimension. Then, in order to meet the model training requirements, the data needs to be standardized or normalized, and new features (such as daily and hourly power generation) are constructed based on the original data and time features (such as year, month, hour) are extracted, and the representation is converted into a vector representation, and the generation of the vector representation includes generating a corresponding position vector for each position. Finally, the data is carefully divided into training set, validation set and test set, and saved in CSV format suitable for model input, laying a solid foundation for subsequent analysis.

[0033] In step S102, the encoder receives the vector representation output by the data loading module, which contains multi-dimensional features and is sampled at fixed time intervals. First, feature embedding is performed through a one-dimensional convolutional layer to increase the feature dimension to capture richer information. Subsequently, position encoding is added to compensate for the sensitivity of the Transformer model to the sequence order. The encoder uses the self-attention mechanism to efficiently process long sequences, compresses the sequence through distillation operations while retaining key information, and further processes the features with the aid of a position-by-position feedforward neural network. In order to stabilize training and prevent overfitting, normalization and Dropout operations are introduced between layers. The encoder consists of multiple stacked self-attention blocks and feedforward network blocks, and the stacking structure is flexibly adjusted according to the data time scale to deeply mine data features and extract key feature information.

[0034] In step S103, the decoder first receives the key feature information processed by the encoder, and further processes the key feature information through feature extraction and compression operations. The decoder introduces a masked multi-scale hole attention mechanism.

[0035] Step S104, the final prediction result is obtained by the final output layer. This layer is composed of a series of fully connected layers, which is responsible for converting the output of the decoder into prediction values ​​of specific dimensions, which may represent specific prediction results at a certain point in the future. The fully connected layer realizes the conversion of input features and the generation of prediction values ​​through the use of linear transformation and activation function. The dimensional configuration of the output layer is a key link in the model design, which directly determines the diversity and applicability of the model prediction results.

[0036] like Figure 2 As shown, a schematic diagram of the structure of a time series prediction device based on a deep neural network provided by the present invention includes:

[0037] The data loading module is used to convert the input data representation into a vector representation to capture the relationship between the data. Then a position vector is generated for each position of the input data. The position encoding is generated using the sine (sin) and cosine (cos) functions. The sine function is used on even dimensions, and the cosine function is used on odd dimensions. For each different position (pos), a unique position encoding is generated. This alternating method helps the model learn the relative relationship between different positions and ensures that the model can distinguish data at different positions in the input sequence. The formula for its position encoding is pos represents the position of the current data in the input sequence, i represents the dimension index, starting from 0. d is the dimension of the embedding vector;

[0038] The encoder is used to receive the vector representation and process the time series data through multi-scale hole attention operations and distillation operations to extract key feature information.

[0039] The encoder receives the preprocessed time series data X = [x1, x2, ..., x T ], where x T ∈R D The input data is embedded in the matrix W e ∈R D×d Mapping to a high-dimensional space E(X) = [E(x1), E(x2), ..., E(x T )]=XW e , where E(x T )∈R d, d is the embedding dimension, and a one-dimensional convolutional layer Z = Conv1D(E(X)) is applied to the embedded data to increase the feature dimension to capture richer information while retaining the local information of the time series, further improving the representation ability and prediction accuracy of the model. Subsequently, multi-scale hole attention is performed to divide the channels of the feature map into n different heads, and sliding window expansion attention is performed on different heads using different hole rates. The outputs of all heads are connected together, and then feature aggregation is performed through a linear layer. It is beneficial to aggregate semantic information at various scales within the receptive field and effectively reduce the redundancy of the self-attention mechanism without complex operations and additional computational costs. The distillation operation of the encoder reduces the computational complexity by reducing the length of the input sequence while retaining important information. This operation is completed through multiple steps such as segmenting the input sequence, embedding, self-attention calculation, and information concentration. The input time series data is segmented into multiple fixed-length segments. Each segment is mapped to a high-dimensional space through an embedding layer to obtain an embedded representation. The self-attention mechanism is applied to capture the dependencies between different segments. The formula is Q, K, V are query, key, and value matrices, and d is the embedding dimension. The attention score matrix C = max_pooling(A) is condensed using the maximum pooling technique. After information concentration, the model can process data faster and improve overall efficiency. It is further processed by a position-by-position feedforward neural network. To stabilize training and prevent overfitting, normalization and dropout operations are introduced between layers. The encoder consists of multiple stacked attention blocks and feedforward network blocks. The stacking structure is flexibly adjusted according to the data time scale to deeply mine data features.

[0040] The decoder is used to further process key feature information through feature extraction and compression operations, and the decoder introduces a masked multi-scale hole attention mechanism.

[0041] The decoder first receives the feature representation rich in key information processed by the encoder, and then dynamically adds position encoding to maintain the sequence order. In the decoding process, it mainly includes masked multi-scale dilated convolution attention and multi-head attention mechanism. Masked multi-scale dilated convolution attention adopts multi-scale dilated convolution attention mechanism to capture different scale information in the input sequence through different dilation rates. Its formula is Y i =Conv1D(x(X);k,d i ), where k is the convolution kernel size, d i is the void ratio of different scales. Then the features of different scales are fused together so that the model has both local and global information. The formula is Y = Concat (Y1, Y2, ..., Y n). The hollow attention mechanism can expand the receptive field and capture longer dependencies without increasing the computational complexity. In the attention calculation process, the mask matrix is ​​applied to limit the calculation range of the attention weight to ensure that the model can only pay attention to the information before the current position to prevent the leakage of future information. The formula is: The mask M is used to block future information, d e is the embedding dimension, and ⊙ is the element-by-element multiplication. The final output representation is generated based on the calculated attention weights and the embedded representation of the input sequence. Multiple attention heads are calculated in parallel to capture different features in the input sequence. The formula is: Where h is the number of attention heads. Then the results of multiple heads are concatenated and linearly transformed, layer normalization and residual connection are applied, and nonlinear transformation is performed through a feedforward neural network. This enhances the model's modeling ability for the input sequence and improves the model's expressiveness and generalization capabilities.

[0042] The final output layer is used to convert the output of the decoder into a prediction value through an activation function. This layer consists of a series of fully connected layers, which are responsible for converting the output of the decoder into a prediction value of a specific dimension. These prediction values ​​may represent the specific prediction results at a certain point in the future. The fully connected layer realizes the conversion of input features and the generation of prediction values ​​through the use of linear transformation and activation function. The dimensional configuration of the output layer is a key link in model design, which directly determines the diversity and applicability of the model prediction results.

[0043] like Figure 3 As shown, the present invention provides a time series prediction device based on a deep neural network. For the sake of ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0044] The time series prediction device 3 based on a deep neural network in the embodiment of the present invention includes a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, the steps in the above-mentioned time series prediction method based on a deep neural network are implemented, for example Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of the units in the above-mentioned device embodiments are realized.

[0045] On the other hand, the present invention provides a computer-readable storage medium, which stores program instructions. When the program instructions are executed by a processor, they are the steps in the above-mentioned embodiment of the time series prediction method based on a deep neural network.

[0046] The present invention provides a time series prediction method, device, equipment and medium based on a deep neural network, which not only improves the accuracy of the prediction, but also effectively reduces the computational complexity. Through the analysis of real-time data and prediction results, the optimal solution can be formulated more scientifically. This method overcomes the problems of inaccurate prediction, high computational complexity, and inability to predict long sequences in traditional methods, making it have more efficient prediction performance. This highly flexible and adaptable feature makes this method have great potential and broad prospects in application fields such as the time series prediction industry.

[0047] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of embodiments of software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0048] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0049] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A time series prediction method based on deep neural network, characterized in that: The following steps are involved: (1) normalizing the input load data through a data loading module and converting the representation into a vector representation, wherein the generation of the vector representation includes generating a corresponding position vector for each position; (2) The vector representation generated in step (1) is input into the encoder, which processes the time series data through multi-scale hole attention operation and distillation operation to extract key feature information; (3) inputting the key feature information extracted by the encoder in step (2) into the decoder, and the decoder further processes the key feature information through feature extraction and compression operations, and the decoder introduces a masked multi-scale hole attention mechanism; (4) The features output by the decoder in step (3) are input into the final output layer, and the final output layer converts the features into prediction values ​​through an activation function.

2. The method according to claim 1, characterized in that The data loading module includes: a submodule for normalizing the input load data and generating a vector representation that is convenient for model processing.

3. The method according to claim 1, characterized in that The generation of the position vector is based on sine and cosine functions and is used in conjunction with the vector representation generated in step (1).

4. The method according to claim 1, characterized in that The multi-scale atrous attention operation uses atrous convolution kernels of different scales to extract local and global feature information from the vector representation generated in step (1).

5. The method according to claim 1, characterized in that The distillation operation performs multi-level extraction on the feature information obtained by the multi-scale void attention operation through information distillation technology, removes redundant features and parameters, and compresses long-sequence feature information into short-sequence feature information.

6. The method according to claim 1, characterized in that The decoder further comprises: A feature compression module, used for performing multi-level compression on the feature information encoded in step (2); The mask module is used to introduce a masked multi-scale hole attention mechanism, which masks future time step information by adding a mask operation to the multi-scale hole attention mechanism.

7. The method according to claim 1, characterized in that The activation function is a linear activation function or a ReLU activation function, which is used to convert the feature information output by the decoder in step (3) into a nonlinear prediction value.

8. A time series prediction device based on a deep neural network, characterized in that: include: A data loading module, used for normalizing the input load data and converting its representation into a vector representation, wherein the generation of the vector representation includes generating a corresponding position vector for each position; The encoder receives the vector representation and processes the time series data through multi-scale hole attention and distillation operations to extract key feature information. A decoder, used for further processing key feature information through feature extraction and compression operations, wherein the decoder introduces a masked multi-scale hole attention mechanism; The final output layer is used to convert the output of the decoder into a predicted value through an activation function.

9. A time series prediction device based on a deep neural network, characterized in that: The method comprises a processor and a memory, wherein program instructions are stored in the memory, and when the program instructions are executed, the processor executes any one of the methods of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images