Time sequence modeling method and device

Abstract

The invention provides a time sequence modeling method and device, and the method comprises the steps: obtaining the original time sequence data of a to-be-predicted task, and carrying out the wavelet transformation of the original time sequence data, and obtaining a multi-scale component sequence; obtaining an embedded vector set based on the multi-scale component sequence; calculating a clustering label of each embedded vector in the embedded vector set to generate an attention mask matrix; jointly inputting the embedded vector set and the attention mask matrix into a Transform encoder to obtain an initial prediction result; and according to an error between the initial prediction result and a target output label of the to-be-predicted task, iteratively updating trainable parameters of a Transformer encoder and a micro K-Means clustering algorithm until the error meets a preset convergence condition, and obtaining a time sequence prediction model for the to-be-predicted task. In an industrial-grade-oriented long-term time sequence prediction task, the prediction precision and reliability of time sequence prediction can be significantly improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a time series modeling method and device. BACKGROUND

[0002] Long-term time series prediction, as an important basic capability of industrial intelligence, is widely used in meteorological forecasting, energy management, financial analysis, traffic planning, equipment maintenance and other industrial scenarios. Through modeling and prediction of historical data, enterprises can optimize resource allocation, make dynamic decisions and reduce operational risks. With the continuous improvement of business complexity and data scale, industrial long-time series data presents characteristics such as high-dimensional heterogeneity, periodic trend fluctuation, frequent local disturbance and serious noise pollution, which makes the existing prediction methods face obvious challenges in engineering practice.

[0003] In the prior art, the traditional logistic regression model is based on linear assumption, which is difficult to effectively depict the nonlinear periodic law and complex cross-time dependence relationship commonly existing in time series, and is sensitive to high-frequency noise, which easily leads to insufficient generalization ability of the prediction result. Although the recurrent neural network model such as LSTM (Long Short-Term Memory Network) has certain advantages in modeling local time dependence, it is limited by the inherent gradient decay problem of the recurrent structure, and its modeling ability for long-term time series is limited, and the training efficiency is also difficult to meet the requirements of large-scale industrial scenarios. In addition, the above methods generally lack explicit modeling mechanism for global structure, lack quantitative control means for uncertainty of prediction results, and lack the ability to effectively decouple multi-scale structure, which is difficult to meet the comprehensive requirements of accuracy, stability and reliability in complex industrial prediction tasks.

[0004] In recent years, models based on self-attention mechanism represented by Transformer have attracted widespread attention in time series prediction tasks due to their strong long-range dependency modeling capability. The typical improved model Informer improves the accuracy of long sequence prediction to some extent by introducing a probabilistic sparse self-attention mechanism while reducing computational complexity. However, this type of method still has several technical bottlenecks: first, in the process of attention sparsification, key periodic nodes and trend turning points may be missed, affecting the overall prediction accuracy; second, it usually directly models the original time series data without an effective noise reduction and frequency decomposition mechanism on the input side, making it difficult to distinguish long-term trends from high-frequency disturbances in a timely manner; third, there is a lack of systematic structural design in capturing local disturbances, abnormal changes, and controlling the confidence and risk of prediction results, which limits the application of the model in high-risk industrial scenarios.

[0005] In summary, the existing technology often fails to simultaneously consider the fine depiction of long-term trends and local disturbances and the effective expression of the reliability of prediction results when facing high-noise, multi-scale, and long-dependent industrial-level time series, making it difficult to continuously output stable and reliable prediction results that can support key business decisions, and the prediction accuracy is poor. SUMMARY

[0006] Embodiments of the present application provide a time series modeling method and device to solve the technical problem that the existing technology often fails to simultaneously consider the fine depiction of long-term trends and local disturbances and the effective expression of the reliability of prediction results when facing high-noise, multi-scale, and long-dependent industrial-level time series, making it difficult to continuously output stable and reliable prediction results that can support key business decisions, and the prediction accuracy is poor.

[0007] To solve the above technical problems, the present application is implemented as follows:

[0008] In a first aspect, embodiments of the present application provide a time series modeling method, which comprises:

[0009] obtaining original time series data of a to-be-predicted task, and performing wavelet transform processing on the original time series data to obtain a multi-scale component sequence;

[0010] performing reversible normalization, sequence slicing, and vector embedding processing on the multi-scale component sequence in sequence to obtain an embedded vector set;

[0011] calculating the clustering label of each embedded vector in the embedded vector set using a differentiable K-Means clustering algorithm, and generating an attention mask matrix according to the clustering label;

[0012] inputting the embedding vector set and the attention mask matrix into a Transformer encoder to obtain an initial prediction result output by the Transformer encoder;

[0013] updating, according to an error between the initial prediction result and a target output label of the task to be predicted, a trainable parameter of the Transformer encoder and the differentiable K-Means clustering algorithm until the error meets a preset convergence condition, to obtain a time series prediction model for the task to be predicted.

[0014] Optionally, original time series data of a task to be predicted is obtained, and wavelet transform processing is performed on the original time series data to obtain a multi-scale component sequence, including:

[0015] convolution and down-sampling processing are performed on the original time series data by using a low-pass filter to extract an approximation coefficient component reflecting a long-term trend;

[0016] convolution and down-sampling processing are performed on the original time series data by using a high-pass filter to extract a detail coefficient component reflecting high-frequency details;

[0017] The approximation coefficient component and the detail coefficient component are spliced to obtain the multi-scale component sequence.

[0018] Optionally, reversible normalization, sequence slicing and vector embedding processing are sequentially performed on the multi-scale component sequence to obtain an embedding vector set, including:

[0019] The multi-scale component sequence is obtained, and distribution alignment processing is performed on the multi-scale component sequence by using a reversible instance normalization (RevIN) algorithm to obtain a normalized sequence;

[0020] The normalized sequence is slidingly intercepted based on a preset time window length and a preset step to obtain a sliced sequence;

[0021] Each slice in the sliced sequence is mapped to a high-dimensional hidden space vector by using a linear projection layer, and position encoding is superimposed on the high-dimensional hidden space vector to obtain the embedding vector set.

[0022] Optionally, a clustering label of each embedding vector in the embedding vector set is calculated by using a differentiable K-Means clustering algorithm, and an attention mask matrix is generated according to the clustering label, including:

[0023] The Euclidean distance between each embedding vector in the embedding vector set and a preset learnable clustering center is calculated, each vector is assigned to a cluster with the closest Euclidean distance, and a clustering label of each embedding vector is obtained.

[0024] generating an attention mask matrix based on the clustering labels;

[0025] wherein the learnable clustering centers are dynamically adjusted as trainable parameters of the differentiable K-Means clustering algorithm based on backpropagation of a clustering loss constructed by the Euclidean distance during a training process of the time series prediction model.

[0026] Optionally, the embedding vector set and the attention mask matrix are jointly input into a Transformer encoder to obtain an initial prediction result output by the Transformer encoder, including:

[0027] projecting the embedding vector set into a query matrix, a key matrix, and a value matrix respectively;

[0028] performing a masking operation on a dot product result of the query matrix and the key matrix by using the attention mask matrix to obtain a masked attention weight;

[0029] performing weighted summation on the value matrix by using the masked attention weight, and outputting the initial prediction result after processing a result of the weighted summation by a feedforward network.

[0030] Optionally, after the time series prediction model for the to-be-predicted task is obtained by iteratively updating the Transformer encoder and the trainable parameters of the differentiable K-Means clustering algorithm until an error between the initial prediction result and a target output label of the to-be-predicted task meets a preset convergence condition, the method further includes:

[0031] inputting the original time series data into a time series convolutional network to perform residual modeling on non-stationary high-frequency disturbances contained in the original time series data to obtain a residual prediction result, and obtaining a fused prediction value according to the residual prediction result and the initial prediction result;

[0032] based on a Bayesian inference principle, performing multiple forward propagations on the original time series data by using the time series prediction model to obtain multiple sampling prediction samples for the fused prediction value, and performing statistical analysis on the sampling prediction samples to obtain a time series prediction result containing uncertainty estimation.

[0033] Optionally, the original time series data is input into a time series convolutional network to perform residual modeling on non-stationary high-frequency disturbances contained in the original time series data to obtain a residual prediction result, and a fused prediction value is obtained according to the residual prediction result and the initial prediction result, including:

[0034] obtain the original time series data, and input the original time series data into the time series convolution network composed of a stack of a causal convolution layer and a dilated convolution layer;

[0035] capture local mutation features in the original time series data by using an enlarged receptive field of the dilated convolution layer, and output the residual prediction result for high-frequency disturbance based on the local mutation features;

[0036] superimpose the residual prediction result and the initial prediction result to obtain a superimposed prediction value;

[0037] restore the superimposed prediction value by using inverse transformation parameters of the RevIN algorithm to obtain the fusion prediction value.

[0038] Optionally, based on the Bayesian inference principle, the time series prediction model is used for multiple forward propagations for the original time series data to obtain multiple sampling prediction samples for the fusion prediction value, and the sampling prediction samples are statistically analyzed to obtain a time series prediction result containing uncertainty estimation, including:

[0039] introducing a Monte Carlo dropout (MC Dropout) layer before an output layer of the time series prediction model;

[0040] keeping the Monte Carlo dropout layer in an active state in an inference stage of the time series prediction model, driving the time series prediction model to perform K times of forward propagation for the original time series data to obtain K independent sampling prediction samples, wherein K is a positive integer;

[0041] calculating an arithmetic mean of the K sampling prediction samples as a prediction point estimate;

[0042] calculating a statistical variance of the K sampling prediction samples as a confidence index for measuring uncertainty of the time series prediction result;

[0043] combining the prediction point estimate and the confidence index to output the time series prediction result containing uncertainty estimation.

[0044] In a second aspect, an embodiment of the present application provides a time series modeling device, and the device comprises:

[0045] an acquisition module configured to acquire original time series data of a to-be-predicted task, and perform wavelet transform processing on the original time series data to obtain a multi-scale component sequence;

[0046] an execution module configured to sequentially perform reversible normalization, sequence slicing and vector embedding processing on the multi-scale component sequence to obtain an embedded vector set;

[0047] a cluster label of each embedding vector in the set of embedding vectors is calculated by using a differentiable K-Means clustering algorithm, and an attention mask matrix is generated according to the cluster label;

[0048] the set of embedding vectors and the attention mask matrix are jointly input into a Transformer encoder to obtain an initial prediction result output by the Transformer encoder;

[0049] trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm are iteratively updated according to an error between the initial prediction result and a target output label of the to-be-predicted task until the error satisfies a preset convergence condition, and a time series prediction model for the to-be-predicted task is obtained.

[0050] In a third aspect, an embodiment of the present application provides a network device, comprising a processor, a memory, and a program stored in the memory and executable in the processor, and when the program is executed by the processor, the steps of the time series modeling method in the first aspect are implemented.

[0051] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the time series modeling method in the first aspect are implemented.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, and when the computer instructions are executed by a processor, the steps of the time series modeling method in the first aspect are implemented.

[0053] In the embodiment of the present application, by obtaining the original time series data of the to-be-predicted task and performing wavelet transform processing on the original time series data, a multi-scale component sequence can be obtained, so that subsequent modeling can distinguish features of different scales in the frequency domain; further, the multi-scale component sequence is sequentially subjected to reversible normalization, sequence slicing and vector embedding processing, so that the input can be standardized into a set of embedding vectors with stable distribution, uniform granularity and carrying time position information, providing a more regular and easy-to-learn representation space for the deep model; on this basis, the cluster labels of each embedding vector in the embedding vector set are calculated using the differentiable K-Means clustering algorithm, and an attention mask matrix is generated according to the cluster labels, so that subsequent self-attention calculation is only carried out between time segments of the same type or similar structure, thereby suppressing the attention allocation between irrelevant noise segments; then the embedding vector set and the attention mask matrix are jointly input into the Transformer encoder to obtain the initial prediction result output by the Transformer encoder, and the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm are iteratively updated according to the error between the initial prediction result and the target output label of the to-be-predicted task until the error meets the preset convergence condition, thereby obtaining a time series prediction model that converges stably for a specific task.

[0054] Therefore, through the cooperation of the above technical features, the trend and disturbance can be better separated, the key segments with similar structures can be focused on and irrelevant attention can be suppressed in the industrial-level time series scene with high noise, multi-scale and long dependence, thereby improving the accuracy and reliability of long-term time series prediction. BRIEF DESCRIPTION OF DRAWINGS

[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is merely meant to teach a person of ordinary skill a way to make and use the application. It is not meant to put limitations on the scope of the application. Moreover, the same reference numerals are used throughout the several drawings to designate the same or similar parts. In the drawings:

[0056] Figure 1 A flowchart of a time series modeling method provided by an embodiment of the present application;

[0057] Figure 2 A structural block diagram of a time series prediction model provided by an embodiment of the present application;

[0058] Figure 3 A structural block diagram of a time series modeling device provided by an embodiment of the present application;

[0059] Figure 4 A structural block diagram of a network device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0061] The technical terms on which the time series modeling method shown in the embodiments of the present application relies are introduced.

[0062] 1、LTSF (Long-Term Series Forecasting, long-term time series forecasting), refers to the task of predicting the continuous numerical changes of multiple times in the future based on historical observation data. Unlike short-term time series forecasting, long-term time series forecasting needs to capture complex time series dependencies (such as periodicity, trend and sudden change) in the data, while dealing with challenges such as noise interference, external variable influence and prediction error accumulation.

[0063] 2、Transformer: a neural network architecture based on multi-head self-attention (Multi-Head Self-Attention) mechanism. Its core is to parallelize the processing of sequence data and dynamically capture global context dependencies using self-attention mechanism. The model is stacked by an encoder (Encoder) and a decoder (Decoder), each layer contains a multi-head self-attention module and a feedforward neural network, while introducing position encoding to preserve sequence order information. Transformer is widely used in natural language processing, computer vision, time series analysis and other fields due to its efficient training and long-distance dependency modeling capabilities.

[0064] 3、K-Means (K-Means Clustering Algorithm, K-means clustering algorithm): K-Means is a commonly used unsupervised clustering method that divides data into K clusters through iterative optimization, so that each data point is assigned to the cluster corresponding to the nearest cluster center, thereby minimizing the variance within the cluster.

[0065] 4、Pytorch: Pytorch is an open-source deep learning framework designed to flexibly build and train neural network models. Its core features include dynamic graph computation, GPU (Graphics Processing Unit, graphics processing unit) accelerated tensor computation, automatic differentiation, etc., and supports modular model definition and distributed training optimization.

[0066] 5、Wavelet Transform (WT) is a signal processing technique that can convert time series signals from time domain to time-frequency domain, thus analyzing the frequency components of the signal and their changes over time simultaneously. Unlike traditional Fourier Transform, WT has multi-resolution characteristics, which can capture local features at different scales, making it suitable for the analysis of non-stationary signals. Discrete Wavelet Transform is commonly used for signal denoising, trend extraction and feature decomposition, and is widely used in finance, biomedical engineering and fault diagnosis.

[0067] 6、Temporal Convolutional Network (TCN) is a deep learning architecture specifically designed for modeling time series data. It is based on Convolutional Neural Network (CNN), using Causal Convolution and Dilated Convolution to ensure the model maintains the time order when processing sequences and effectively captures long-distance dependencies. TCN structure has good parallelism and stability, and has advantages in training efficiency and gradient propagation compared with recurrent neural networks (such as LSTM), and is widely used in speech recognition, natural language processing and time series prediction tasks.

[0068] 7、Bayesian Neural Network (BNN) is a probabilistic modeling method that introduces Bayesian inference into traditional neural networks. Unlike ordinary neural networks, which treat parameters as fixed values, BNN treats parameters as probability distributions, thus modeling the uncertainty of model predictions. This method not only improves the robustness of the model, but also provides confidence estimates for prediction results, making it suitable for high-risk decision-making scenarios such as autonomous driving, medical diagnosis and financial forecasting. Since the posterior distribution is usually difficult to solve directly, BNN often uses variational inference or Monte Carlo Dropout and other approximation methods for training.

[0069] Figure 1 A time series modeling method is shown according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0070] Step S101, obtaining the original time series data of the to-be-predicted task, and performing wavelet transform processing on the original time series data to obtain a multi-scale component sequence;

[0071] Step S102, sequentially performing reversible normalization, sequence slicing and vector embedding processing on the multi-scale component sequence to obtain an embedded vector set;

[0072] Step S103, calculating the clustering label of each embedding vector in the embedding vector set by using the differentiable K-Means clustering algorithm, and generating an attention mask matrix according to the clustering label;

[0073] Step S104, inputting the embedding vector set and the attention mask matrix into the Transformer encoder together to obtain an initial prediction result output by the Transformer encoder;

[0074] Step S105, iteratively updating the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm according to the error between the initial prediction result and the target output label of the task to be predicted until the error meets a preset convergence condition, and obtaining a time series prediction model for the task to be predicted.

[0075] It should be noted that, first, the original time series data of the task to be predicted is obtained, and the original time series data is processed by wavelet transform to decompose the original time series into a multi-scale component sequence representing long-term trends and high-frequency disturbances at different scales, and the complex time series structure is decomposed in the frequency domain on the input side. Then, the multi-scale component sequence is sequentially subjected to reversible normalization, sequence slicing and vector embedding processing to obtain a set of embedding vectors of a slicing granularity. On this basis, the clustering label of each embedding vector in the embedding vector set is calculated by using the differentiable K-Means clustering algorithm, the slices with similar dynamic patterns are classified into the same cluster through the Euclidean distance relationship between the learnable clustering centers, and the attention mask matrix that allows attention calculation only between slices in the same cluster is generated according to the clustering label.

[0076] Subsequently, the embedding vector set and the attention mask matrix are input into the Transformer encoder together, and the multi-head self-attention calculation and the feedforward network transformation are performed at the slicing granularity by using the masked self-attention mechanism, and the initial prediction result representing the multi-scale trend and the local dynamic is output. Finally, the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm are iteratively updated according to the error between the initial prediction result and the target output label of the task to be predicted until the error meets a preset convergence condition, and a time series prediction model for the task to be predicted is obtained.

[0077] Therefore, through the synergistic effect of the above technical features, a structure-aware time series prediction model is constructed through the cooperation of multi-scale decomposition, sequence slicing, differentiable K-Means clustering and masked self-attention, thereby significantly improving the modeling capability and prediction stability of complex time series structures in high-noise and long-dependent industrial scenarios.

[0078] In a possible implementation, the original time series data of the task to be predicted is acquired, and wavelet transform processing is performed on the original time series data to obtain a multi-scale component sequence, including: performing convolution and down-sampling processing on the original time series data by using a low-pass filter to extract an approximation coefficient component reflecting a long-term trend; performing convolution and down-sampling processing on the original time series data by using a high-pass filter to extract a detail coefficient component reflecting high-frequency details; and splicing the approximation coefficient component and the detail coefficient component to obtain the multi-scale component sequence.

[0079] It should be noted that by explicitly introducing the low-pass filter and the high-pass filter, the original time series data can be decomposed: the approximation coefficient component reflecting the long-term trend is extracted by performing convolution and down-sampling processing on the original time series data by using the low-pass filter, and is used to represent the low-frequency component of the time series; at the same time, the detail coefficient component reflecting the high-frequency details is extracted by performing convolution and down-sampling processing on the original time series data by using the high-pass filter, and is used to represent the local disturbance and the high-frequency change. Then, the approximation coefficient component and the detail coefficient component are spliced to make the two types of components different in frequency domain semantics jointly serve as the multi-scale component sequence input to the subsequent module in the channel dimension. Through such a design, the long-term trend information and the short-term high-frequency disturbance information can be explicitly distinguished at the input stage, providing a structural clear feature basis for subsequent reversible normalization, sequence slicing, and multi-scale modeling based on the Transformer encoder, thereby improving the comprehensive modeling effect of the long-term trend and the high-frequency fluctuation in the complex industrial time series.

[0080] In a possible implementation, reversible normalization, sequence slicing, and vector embedding processing are sequentially performed on the multi-scale component sequence to obtain an embedded vector set, including: acquiring the multi-scale component sequence, performing distribution alignment processing on the multi-scale component sequence by using a reversible instance normalization (RevIN) algorithm to obtain a normalized sequence; based on a preset time window length and a preset step, the normalized sequence is slidingly intercepted to obtain a sliced sequence; each slice in the sliced sequence is mapped to a high-dimensional hidden space vector by using a linear projection layer, and position encoding is superimposed on the high-dimensional hidden space vector to obtain the embedded vector set.

[0081] It should be noted that firstly, a multi-scale component sequence is acquired, a RevIN algorithm is used for distribution alignment processing on the multi-scale component sequence, reversible normalization is performed on the time statistics of each instance, the mean and variance differences between different time periods and different samples are reduced, a normalized sequence is obtained, and the adverse effects of data distribution drift on model training are suppressed. Subsequently, based on a preset time window length and a preset step, the normalized sequence is slidingly intercepted, the long sequence is divided into a series of slice sequences, the model observes the local structure at the slice granularity; then, each slice in the slice sequence is mapped to a high-dimensional hidden space vector by using a linear projection layer, and position encoding is superimposed on the high-dimensional hidden space vector, so that the slice carries numerical features and time sequence information at the same time, thereby obtaining a set of embedded vectors.

[0082] Through the above processing, the complex multi-scale component sequence can be converted into a set of high-dimensional embedded vectors with distribution alignment, slicing and position encoding, providing a unified, stable and easy-to-learn representation for subsequent differentiable K-Means clustering and mask self-attention calculation in the Transformer encoder, thereby improving the robustness and expression ability of long time series modeling.

[0083] In a possible implementation, the cluster label of each embedded vector in the embedded vector set is calculated by using a differentiable K-Means clustering algorithm, and an attention mask matrix is generated according to the cluster label, including: calculating the Euclidean distance between each embedded vector in the embedded vector set and a preset learnable cluster center, assigning each vector to the cluster with the closest Euclidean distance, and obtaining the cluster label of each embedded vector; generating an attention mask matrix based on the cluster label; wherein the learnable cluster center is a trainable parameter of the differentiable K-Means clustering algorithm, and is dynamically adjusted based on the back propagation of the cluster loss constructed by the Euclidean distance in the training process of the time series prediction model.

[0084] It should be noted that firstly, the Euclidean distance between each embedded vector in the embedded vector set and a preset learnable cluster center is calculated, each embedded vector is assigned to the cluster with the closest Euclidean distance by comparing the Euclidean distance between each embedded vector and each cluster center, thereby obtaining the cluster label of each embedded vector, and forming a similarity-based time slice clustering structure in the embedded space. Subsequently, an attention mask matrix is generated based on the cluster label, the visible relationship between the embedded vectors in the same cluster and the shielding relationship between the embedded vectors in different clusters are marked, so that the attention mask matrix can directly act on the attention score in the Transformer encoder, and the calculation range of the self-attention mechanism is limited.

[0085] Further, the learnable clustering centers are trained as trainable parameters of the differentiable K-Means clustering algorithm, and are dynamically adjusted based on the back propagation of the clustering loss constructed by the Euclidean distance during the training process of the time series prediction model, so that the clustering results evolve together with the embedding distribution and the task target.

[0086] Through this design, the differentiable K-Means clustering algorithm and the Transformer encoder are jointly optimized in an end-to-end manner under the same loss target, which not only aggregates time slices with similar structures through clustering, but also directly constrains the clustering structure to the attention calculation through the attention mask matrix, thereby strengthening the focus on similar dynamic patterns and suppressing the interference of dissimilar noise.

[0087] In a possible implementation, the embedding vector set and the attention mask matrix are jointly input into the Transformer encoder to obtain an initial prediction result output by the Transformer encoder, including: projecting the embedding vector set into a query matrix, a key matrix and a value matrix respectively; performing a masking operation on the dot product result of the query matrix and the key matrix by using the attention mask matrix to obtain a masked attention weight; performing weighted summation on the value matrix by using the masked attention weight, and outputting the initial prediction result after processing the result of the weighted summation by a feedforward network.

[0088] It should be noted that the embedding vector set is first projected linearly into a query matrix, a key matrix and a value matrix, i.e., the same set of embedding vectors is mapped to a query space, a key space and a value space suitable for attention calculation through three linear transformations, laying a foundation for attention weight calculation. Subsequently, the dot product result of the query matrix and the key matrix is masked by using the attention mask matrix, and the attention score corresponding to the elements marked as shielded positions in the mask matrix is suppressed or invalidated, to obtain a masked attention weight that only propagates information between slices that satisfy the clustering relationship, so that the self-attention mechanism mainly focuses on time slices with similar structures. Then, the value matrix is weighted and summed by using the masked attention weight, the value vectors corresponding to each slice are linearly combined according to the masked attention weight, and the weighted result is output after being processed by a feedforward network. In this way, the Transformer encoder performs clustering-aware self-attention calculation at the slice granularity under the guidance of the attention mask matrix generated by the differentiable K-Means clustering, which not only reduces the disturbance caused by irrelevant relationships, but also retains the sensitivity to key time series structures, thereby improving the accuracy and stability of the initial prediction result.

[0089] In a possible implementation, after the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm are iteratively updated until the error between the initial prediction result and the target output label of the to-be-predicted task meets the preset convergence condition, and the time series prediction model for the to-be-predicted task is obtained, the method further includes: inputting the original time series data into the time series convolution network, performing residual modeling on the non-stationary high-frequency disturbance contained in the original time series data to obtain a residual prediction result, and obtaining a fusion prediction value according to the residual prediction result and the initial prediction result; based on the Bayesian inference principle, performing multiple forward propagations on the original time series data by using the time series prediction model to obtain multiple sampling prediction samples for the fusion prediction value, and performing statistical analysis on the sampling prediction samples to obtain a time series prediction result containing uncertainty estimation.

[0090] It should be noted that after the time series prediction model for the to-be-predicted task is obtained, the original time series data is first input into the time series convolution network, the non-stationary high-frequency disturbance contained in the original time series data is modeled to obtain a residual prediction result, that is, the time series convolution network composed of causal convolution and dilated convolution is specially learned for local mutation and short-term fluctuation and other high-frequency components. Then, the fusion prediction value is obtained according to the residual prediction result and the initial prediction result, that is, the initial prediction result obtained based on the Transformer encoder and the residual prediction result output by the time series convolution network are fused, so that the final prediction value takes into account both long-term trends and local disturbances. Next, based on the Bayesian inference principle, multiple forward propagations are performed on the original time series data by using the time series prediction model, a Bayesian approximation is formed by introducing Monte Carlo dropout (MC Dropout), multiple random sampling inferences are performed on the same input, multiple sampling prediction samples for the fusion prediction value are obtained, and statistical analysis is performed on the sampling prediction samples to obtain a time series prediction result containing uncertainty estimation.

[0091] Through the multi-stage process, the residual modeling of the time series convolution network is combined with the uncertainty evaluation based on the Bayesian inference principle, which not only improves the accuracy of the prediction point estimation, but also provides a prediction result with confidence explanation for industrial-level scenarios.

[0092] In a possible implementation, the original time series data is input into a time series convolution network, residual modeling is performed on the non-stationary high-frequency disturbance contained in the original time series data, a residual prediction result is obtained, and a fusion prediction value is obtained according to the residual prediction result and an initial prediction result, comprising: obtaining original time series data, and inputting the original time series data into a time series convolution network composed of a causal convolution layer and a dilated convolution layer; capturing local mutation features in the original time series data by using the enlarged receptive field of the dilated convolution layer, and outputting a residual prediction result for high-frequency disturbance based on the local mutation features; superimposing the residual prediction result and the initial prediction result to obtain a superimposed prediction value; and using inverse transformation parameters of a RevIN algorithm to reverse the superimposed prediction value to obtain a fusion prediction value.

[0093] It should be noted that, first, the original time series data is obtained, and the original time series data is input into a time series convolution network composed of a causal convolution layer and a dilated convolution layer, the output is ensured to only depend on the current and historical time steps by the causal convolution, the time causal order requirement is met, the local mutation features in the original time series data are captured by the enlarged receptive field of the dilated convolution layer, and the residual prediction result for the high-frequency disturbance is output based on the local mutation features. Subsequently, the residual prediction result and the initial prediction result are superimposed to obtain a superimposed prediction value, the main trend prediction of the Transformer encoder and the residual prediction of the time series convolution network are fused to form a comprehensive prediction result containing long-term trend and short-term disturbance correction at the same time; on this basis, inverse transformation parameters of a reversible instance normalization algorithm are used to reverse the superimposed prediction value, and the superimposed prediction value in the normalization domain is restored to a fusion prediction value with the same physical dimension and numerical range as the original time series data. Through the combination of the above residual modeling and inverse normalization restoration, the application enhances the response capability to non-stationary high-frequency disturbance while maintaining the stability of training, and outputs a fusion prediction result with physical interpretability.

[0094] In a possible implementation, based on the Bayesian inference principle, the time series prediction model is used for multiple forward propagations on the original time series data, a plurality of sampling prediction samples for the fusion prediction value are obtained, and statistical analysis is performed on the sampling prediction samples to obtain the time series prediction result containing uncertainty estimation, including: introducing a Monte Carlo dropout MC Dropout layer before an output layer of the time series prediction model; keeping the Monte Carlo dropout layer in an activated state in an inference stage of the time series prediction model, driving the time series prediction model to perform K times of forward propagation on the original time series data, to obtain K independent sampling prediction samples, where K is a positive integer; calculating an arithmetic mean of the K sampling prediction samples as a prediction point estimate; calculating a statistical variance of the K sampling prediction samples as a confidence index for measuring uncertainty of the time series prediction result; and combining the prediction point estimate and the confidence index to output a time series prediction result containing uncertainty estimation.

[0095] It should be noted that the Monte Carlo dropout MC Dropout layer is introduced before the output layer of the time series prediction model, and the dropout mechanism is embedded in the network structure close to the output end; the Monte Carlo dropout layer is kept in the activated state in the inference stage of the time series prediction model, driving the time series prediction model to perform K times of forward propagation on the original time series data, so that different sub-network configurations are formed due to random dropout in each forward propagation, and K independent sampling prediction samples are obtained. Subsequently, an arithmetic mean of the K sampling prediction samples is calculated as a prediction point estimate, and a statistical variance of the K sampling prediction samples is calculated as a confidence index for measuring uncertainty of the time series prediction result. The prediction point estimate and the confidence index are combined to output a time series prediction result containing uncertainty estimation. Through this Monte Carlo dropout MC Dropout-based approximate Bayesian inference method, the application provides a quantitative uncertainty evaluation for the prediction result without significantly increasing the structural complexity, which is beneficial to risk control and result credibility judgment in industrial applications.

[0096] In summary, in the embodiments of the present application, by acquiring the original time series data of the to-be-predicted task and performing wavelet transform processing on the original time series data, a multi-scale component sequence can be obtained, so that subsequent modeling can distinguish features of different scales in the frequency domain; further performing reversible normalization, sequence slicing and vector embedding processing on the multi-scale component sequence in turn can make the input be standardized into a set of embedding vectors with stable distribution, uniform granularity and carrying time position information, providing a more regular and easy-to-learn representation space for the deep model; on this basis, the cluster labels of each embedding vector in the embedding vector set are calculated using the differentiable K-Means clustering algorithm, and an attention mask matrix is generated according to the cluster labels, which can make the subsequent self-attention calculation only expand between similar or structurally similar time segments, thereby suppressing the attention allocation between irrelevant noise segments; then the embedding vector set and the attention mask matrix are input into the Transformer encoder together to obtain the initial prediction result output by the Transformer encoder, and according to the error between the initial prediction result and the target output label of the to-be-predicted task, the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm are iteratively updated until the error meets the preset convergence condition, thereby obtaining a time series prediction model that converges stably for a specific task.

[0097] Therefore, through the cooperation of the above technical features, the trend and disturbance can be better separated, the key segments with similar structures can be focused on and irrelevant attention can be suppressed in the high-noise, multi-scale and long-dependent industrial-level time series scene, thereby improving the accuracy and reliability of long-term time series prediction.

[0098] Based on the above overall introduction, the time series modeling method shown in the embodiments of the present application will be introduced from the perspective of specific application scenarios and formulas, which can be specifically divided into the following steps:

[0099] Step 1: Data collection and processing

[0100] The long time series prediction method of the embodiments of the present application has strong data source adaptation capability, and can be widely applied to various time series scenarios such as log data, signaling data, meteorological data, network traffic data, device sensor data, and industrial monitoring data, covering multiple field requirements such as communication, energy, meteorology, intelligent manufacturing, and finance. Through API (Application Programming Interface) interfaces such as RESTful API (Representational State Transfer Application Programming Interface), database connections such as MySQL (My Structured Query Language) / PostgreSQL (PostgreSQL Database), message queues such as Kafka / RabbitMQ, or file imports such as CSV (Comma-Separated Values File) / Parquet (Parquet columnar storage file), etc. Diversified access methods can flexibly interface time series data generated by different systems, only need to include two core elements of time stamp and observation index value, do not need to rely on specific collection tools or data protocols, and significantly improve the universality and cross-platform adaptability of the method.

[0101] In the preprocessing link, for the time granularity difference of different data sources (such as millisecond level signaling data, month level meteorological statistical data), the time resolution is unified to one of seconds, minutes, hours, days, weeks, months and years through downsampling (mean / sum aggregation) or upsampling (linear / spline interpolation), and repeated records are simultaneously removed, abnormal values are repaired, and the time series is ensured to be continuous without gaps. For example, the second level server access log can be aggregated to 5 minute granularity to adapt to business analysis requirements, and the daily meteorological data can be interpolated to generate hour level sequence to improve short term prediction accuracy, and finally form a structured multi-dimensional time series data set where Q is the time step, and M is the number of indicators or variables detected by the system.

[0102] Step two: wavelet transform assisted time series feature preprocessing method

[0103] In the time series modeling method proposed in the embodiments of the present application, in order to enhance the perception ability of the model to long-term trends and multi-scale periodic structures, wavelet transform is introduced as a key step in the preprocessing stage before the Transformer network structure. This step extracts different frequency components (i.e. high-frequency detail information and low-frequency trend information) by performing multi-scale decomposition on the original time series signal, thereby effectively alleviating the problems such as high-frequency noise interference and long-term dependence modeling difficulty faced by the Transformer model in original time domain modeling.

[0104] Let the original input time series be Wavelet transform decomposes X into a number of levels of approximation coefficients and detail coefficients by introducing multi-scale convolution and downsampling operations, as follows:

[0105] ;

[0106] Among them, Aj represents the low-frequency approximation coefficient of the jth layer, reflecting the long-term trend and dominant periodic structure of the sequence;

[0107] Dj represents the high-frequency detail coefficient of the jth layer, reflecting the local disturbance and high-frequency variation characteristics of the sequence;

[0108] J is the set wavelet decomposition level, which satisfies It is determined by the actual sequence length and the modeling accuracy requirement.

[0109] The basic idea of wavelet transform is to use scaling and translation operations at multiple scales to construct a set of orthogonal basis functions, thereby realizing the sparse representation of the signal. Take the scaling function (scaling function) and the mother wavelet function (mother wavelet) As the basis, the function expression of the jth layer and the kth translation is as follows:

[0110]

[0111] The decomposition expression of the signal is:

[0112]

[0113] Among them, is the Jth layer approximation coefficient; is the Jth layer detail coefficient.

[0114] This transformation is essentially equivalent to the process of convolving the original sequence X with low-pass filter h[n] and high-pass filter g[n] respectively and downsampling:

[0115]

[0116] ;

[0117] After wavelet decomposition, we will get time series components at multiple scales , which have heterogeneous frequency domain semantic information and can be regarded as a set of multi-channel embedded inputs:

[0118] Channel splicing method: all components are uniformly mapped to fixed dimensions and spliced as multi-channel input of the Transformer encoder; Channel parallel modeling method: different frequency components are fed into Transformer branches with the same structure but independent parameters, and their output representations are finally fused; Position encoding redesign: design the time index position encoding for each wavelet component to retain the original time context information.

[0119] In this way, the Transformer encoder no longer directly faces the original sequence containing high-frequency disturbances and trends, but can model information based on the explicit multi-scale components after decomposition, thereby improving the modeling accuracy of long-term trends and dominant cycle structures; suppressing the attention weight disturbance caused by high-frequency noise; supporting hierarchical modeling and attention mechanism optimization based on frequency domain structure.

[0120] Compared with the traditional direct modeling method of the original sequence, the wavelet transform auxiliary strategy has the following key advantages: enhanced long-term modeling capability: approximate components explicitly express long-term stable patterns, making it easy to capture global trends; enhanced model robustness: detail components are decoupled from the original sequence, effectively reducing the interference of high-frequency noise on the attention mechanism; provide structural decomposition basis: provide more discriminative frequency domain input structure for subsequent modules (such as multi-head attention, clustering mask mechanism);

[0121] Adapt to multi-stage modeling system: as a preprocessing module, this step can be flexibly combined with subsequent Transformer encoder, TCN residual modeling, Bayesian correction module to meet the diverse needs in complex industrial prediction scenarios.

[0122] Step three: Micro-sequence slicing clustering and Transformer encoder mask self-attention calculation

[0123] For example Figure 2As shown, the embodiments of the present application first slice the input time series, then convert it to a high-dimensional embedding vector through the embedding layer, and then use the differentiable K-Means clustering method to obtain the mask matrix for multi-head self-attention calculation. The model mainly uses the encoder architecture of the Transformer, the input data format is a single variable time series, and the output is the prediction value of the future T steps of the sequence. Finally, the prediction values of each variable at the same time step are spliced, and the final prediction result is obtained.

[0124] Reversible instance normalization:

[0125] One of the main challenges in time series prediction is the data distribution shift problem (DSP), that is, the distribution of data, such as the mean and variance of data, will change over time. In the time series prediction task, the training set and the test set are often divided by time, which will naturally introduce the problem of inconsistent distribution between the training set and the test set. In addition, different input sequences will also have the problem of inconsistent data distribution. To solve the above problems, the embodiments of the present application introduce a reversible instance normalization technology. Specifically, RevIN includes two parts of "normalization" and "inverse normalization". First, normalize the data before it enters the model, and the calculation formula is as follows:

[0126] wherein, is a learnable parameter.

[0127] ;

[0128] De-normalization is usually located in the last layer of the model to process the model output, and it uses the same parameter value as normalization, and the calculation formula is as follows:

[0129] .

[0130] Slice operation:

[0131] Before converting the data into an embedding vector, slice the time series data. For each input model of a single variable time series with a length of L , let the slice length be P and the stride be S, then the slice operation generates a slice sequence , wherein N is the number of slices, which can be calculated from L, P and S:

[0132]

[0133] In the subsequent modeling process, the model converts each slice into an embedding vector as a whole, compared with embedding each data point individually, the number of embedding vectors input to the model can be reduced from L to approximately L / S, which means the memory usage and computational complexity of attention weights are reduced by a scaling factor of S2. Therefore, under the constraints of training time and GPU memory, the slicing operation can enable the model to see longer historical sequences, thereby significantly improving the prediction performance.

[0134] Embedding layer:

[0135] After obtaining the time series slices, a learnable linear projection matrix is used to map the slices to a hidden space with dimension D, while adding a learnable position encoding matrix to capture the time order information of the slices. The embedding process of the slices is as follows:

[0136]

[0137] where represents the input dataset for clustering and self-attention calculation, containing N embedding vectors with dimension D.

[0138] Differentiable k-means clustering:

[0139] K-means clustering is a classic unsupervised learning algorithm that divides data into K clusters, maximizing the similarity of data points within the same cluster and maximizing the difference between clusters. It optimizes the clustering result by minimizing the sum of squared distances from sample points to their cluster centers. K-means is an iterative optimization process, usually including the following steps:

[0140] (1) Initialization: randomly select K data points as initial cluster centers ;

[0141] (2) Cluster assignment: assign each sample point to the nearest cluster center, i.e.

[0142]

[0143] Get the cluster label ;

[0144] (3) For each cluster k, recalculate its centroid:

[0145] ;

[0146] (4) Convergence judgment: if the cluster assignment does not change or the objective function converges, stop; otherwise, repeat steps (2) and (3).

[0147] Unlike the common k-means clustering algorithm, the clustering center vectors are learnable parameters that are updated simultaneously with model training. The specific process of clustering is as follows: for each embedding vector, calculate its distance with all clustering center vectors, and assign it to the nearest clustering center, and then update the center of each cluster according to the mean of all samples in the current cluster, until convergence to stop iteration.

[0148] When getting the clustering label c[i] of each embedding vector , the attention mask matrix M can be generated accordingly. For any two embedding vector pairs (i, j), the element value of the i-th row and the j-th column of the mask matrix is:

[0149]

[0150] The mask matrix is used to shield the attention calculation between embedding vectors of different classes in the subsequent process.

[0151] Encoder:

[0152] The model uses a common Transformer encoder structure, and the core innovation point is to use the clustering mask matrix for multi-head self-attention calculation. For embedding vectors, each head converts them into Q, K and V matrices:

[0153]

[0154] Then use the mask matrix obtained by clustering to calculate the attention weight:

[0155]

[0156] where,

[0157] Finally, the results of multiple heads are spliced and projected to the original dimension:

[0158]

[0159] After the multi-head attention output, it is sequentially passed through a residual connection, a normalization layer and a feedforward network module:

[0160]

[0161] wherein the feedforward module is usually a two-layer nonlinear fully connected network:

[0162]

[0163] By stacking several encoder layers (such as 2-4 layers), the time-slice representation matrix with clustering perception ability is finally obtained:​​​​

[0164]

[0165] Linear flattening layer:

[0166] On the basis of the encoder output, to generate the final prediction result, the sequence vector needs to be restored to the prediction time series. First, the encoder output is flattened into a vector form and mapped to the target prediction sequence of length T through a linear transformation:

[0167] ;

[0168] where, T is usually the target prediction step number.

[0169] Since the input sequence is normalized by RevIN during preprocessing, to ensure the physical interpretability of the prediction output, the prediction result needs to be inverse normalized.

[0170] Step four: residual modeling of time series convolutional neural network

[0171] Considering that the Transformer has problems such as response lag and unstable modeling when dealing with local short-term fluctuations, a residual modeling auxiliary module based on a time series convolutional network is introduced. This module is used to specifically capture high-frequency disturbances and short-term non-stationary components in the original time series that are difficult to model by the Transformer backbone, so as to form a collaborative modeling structure with the backbone model, significantly improving the overall prediction accuracy and robustness. This step aims to model the residual components in the input time series , that is,

[0172] ;

[0173] where, represents the trend part modeled by the Transformer encoder. The residual sequence mainly contains high-frequency fluctuations, local anomalies, and short-term non-stationary components, and the task of the TCN module is to model them, predicting the disturbance correction term .

[0174] The TCN module uses multi-layer causal convolution (Causal Convolution) and dilated convolution (Dilated Convolution) to maintain temporal consistency while expanding the receptive field. Let the input of the l-th layer be then,

[0175] ;

[0176] where, is the dilation factor of the first layer; is the convolution kernel size, is the activation function (e.g., ReLU). The initial input .

[0177] The final prediction result is:

[0178] .

[0179] This module serves as an independent residual modeling path and can be combined with the Transformer backbone in the following two integration strategies.

[0180] Parallel modeling structure: the input sequence is sent to the Transformer and TCN modules respectively. The Transformer learns the global trend, and the TCN learns the local disturbance. The outputs of the two are fused into the final prediction:

[0181] ;

[0182] Serial correction structure: the Transformer first outputs the preliminary prediction , then calculates the residual , and the TCN module predicts and corrects to obtain the final result:

[0183]

[0184] This module decouples the modeling of different frequency band feature information through structure, so that the model has stronger sensitivity and robustness when facing complex and variable sequences (such as business traffic, financial fluctuations, and industrial load).

[0185] Step five, uncertainty estimation and correction based on Bayesian neural network

[0186] To improve the stability, reliability, and explainability of the time series prediction system in actual deployment, the embodiments of the present application introduce an uncertainty estimation and correction module based on Bayesian neural network. This module, as a plug-in component at the output end of the system, independently undertakes the task of uncertainty modeling of the prediction result and can perform prediction correction in high uncertainty scenarios, thereby forming a closed-loop structure of "main prediction - uncertainty assessment - risk perception feedback".

[0187] The biggest difference between Bayesian neural network and traditional neural network is that the network parameters are regarded as a probability distribution, and the parameter uncertainty in the prediction process is modeled. Let the model parameters be , the input be x, and the output be y. The prediction distribution is:

[0188]

[0189] Due to the posterior distribution Generally cannot be solved analytically, this module uses an approximate inference technique, Monte Carlo Dropout method to realize Bayesian neural network modeling. The method enables Dropout in the prediction stage and performs K forward reasoning to obtain the prediction mean and prediction variance:

[0190] ;

[0191]

[0192] Wherein is the prediction result after the kth Dropout activation. The prediction variance can be used as a confidence score to provide a basis for downstream applications such as risk control, automatic alarm, and result screening.

[0193] This module is deployed in the main model backend and can be considered as a plug-in component to perform "confidence check" and "dynamic correction" on the prediction results. When the prediction uncertainty exceeds the set threshold , the following operations can be triggered: degradation using a conservative prediction model (such as a trend line); triggering an exception report; starting a prediction corrector for correction. The BNN module has low construction cost and does not need to change the main model during training. It only needs to enable Dropout in the prediction stage and perform multiple sampling reasoning, which has good industrial integration and deployment convenience.

[0194] Therefore, the embodiments of the present application propose a multi-stage integrated time series modeling system for industrial-level prediction tasks. Wavelet transform is introduced for multi-scale decomposition modeling. By applying discrete wavelet transform to the original time series data, it is decomposed into low-frequency trend components and high-frequency disturbance components, which are modeled respectively, effectively enhancing the model's perception of long-term structure and isolating noise, and improving the accuracy of trend capture.

[0195] A micro-sequence slice clustering mechanism and a mask self-attention structure are designed. A slice strategy that integrates a differentiable K-Means clustering method is proposed to divide the time series into sub-sections with consistent structure, calculate mask attention within the class, improve the attention score of key information, and reduce attention to invalid noise, and realize automatic alignment of focus.

[0196] Only the Transformer encoder structure is used and modeling is performed at the slice granularity, reducing the computational complexity. Compared with the traditional point granularity attention mechanism, the slice-level attention calculation greatly reduces the computational amount. By only retaining the encoder module of the Transformer for modeling, long sequence data can be efficiently used under resource-limited conditions.

[0197] The TCN module is introduced to model residual disturbances. A parallel time convolution path is constructed to model and correct the residuals of the main model prediction results, and non-stationary disturbance signals are specifically modeled to improve the model's response to mutations and jumps and enhance overall robustness.

[0198] Bayesian neural networks are integrated for uncertainty estimation and correction. A Bayesian neural network structure (such as the MC Dropout path) is introduced at the prediction output end to output prediction confidence intervals and variance values. The system can trigger a dynamic correction mechanism based on uncertainty or output a confidence reminder to improve controllability and safety in critical tasks.

[0199] The system has end-to-end training capabilities, with modules decoupled in function and coordinated in training, supporting efficient deployment of multiple types of prediction tasks in industrial production environments. Compared to traditional logistic regression, LSTM, and standard Transformer methods, the system significantly improves long-term trend modeling ability, local disturbance response ability, uncertainty identification and control ability, and other aspects.

[0200] Furthermore, compared with the prior art, the multi-stage integrated time series modeling system for industrial-level prediction tasks proposed in the present application has significant technical advantages in system structure design, modeling precision improvement, computational efficiency optimization, and prediction confidence enhancement. The following details the technical advantages of the present application:

[0201] 1. Significantly enhance long-term trend and cycle structure modeling ability:

[0202] Existing methods usually model directly in the time domain, making it difficult to distinguish between trend and disturbance signals, resulting in poor long-term dependence modeling stability, especially when there are slow trend changes or weak cycle structures. The present application introduces a discrete wavelet transform module as a preprocessing step to decompose the original time series into multiple scales in the frequency domain, extracting low-frequency (long-term trend) and high-frequency (local disturbance) sub-band information. This effectively decouples trends and high-frequency noise, enhances the Transformer backbone's ability to model cycles and trends, and improves prediction accuracy and generalization ability, especially for multi-cycle or weak-cycle industrial time series modeling tasks.

[0203] 2. Improve key feature recognition ability and strengthen model's perception of mutation dynamics:

[0204] Traditional Transformer model adopts global attention mechanism, which treats all inputs equally and easily amplifies the influence of non-key information. Although the sparse method such as Informer reduces the complexity, it may miss important signals such as mutation points and abnormal segments. The application proposes a sequence slicing clustering + mask self-attention mechanism, which performs K-Means clustering on the embedding representation of sequence slices, constructs intra-class masks, and only allows intra-class vector attention calculation. The attention mechanism pays more attention to the same dynamic structure, suppresses the influence of noise across clusters, improves the response ability of the model to key changes such as trend turning points and mutation peaks, and effectively improves the overall prediction accuracy and stability.

[0205] 3. End-to-end collaborative optimization of clustering mechanism and Transformer:

[0206] In existing methods, the clustering operation often exists as an independent preprocessing module, and its clustering parameters are fixed and cannot be dynamically adjusted with model optimization, resulting in a disconnection between the attention mechanism and the structure division, affecting the final prediction quality. The application designs a differentiable clustering structure, which embeds the K-Means clustering process into the self-attention path, and realizes the joint optimization of the clustering center through the error back propagation mechanism. The consistency of the clustering and self-attention module parameters is realized, the dynamic adaptability and semantic consistency of the structure division are improved, and the expression ability and learning robustness of the model are enhanced.

[0207] 4. Integrate TCN module for residual disturbance modeling to improve local fitting ability:

[0208] The Transformer backbone structure mainly models globally, and the response to non-stationary characteristics such as instantaneous mutations and local disturbances is insufficient, which easily causes local prediction distortion. The application introduces a time convolution network module to independently model the residual part of the backbone prediction, and uses expandable convolution to capture local structures. The response ability of the system to peak jumps and short-term anomalies is enhanced, the local detail modeling ability of the backbone structure is supplemented, and the fitting quality and robustness of non-stationary time series are improved.

[0209] 5. Reduce modeling calculation complexity and improve long sequence processing ability:

[0210] The traditional Transformer model calculates the attention graph at the point granularity, with a calculation complexity of O(T2), which brings memory bottlenecks and operation delays when processing ultra-long sequences, making it difficult to adapt to resource-limited deployment environments. The application uses a slicing granularity attention mechanism instead of point-level calculation, combined with a mask mechanism to compress the attention space while preserving the global modeling path, and uses a lightweight structure containing only an encoder to further reduce the calculation demand. This significantly reduces the occupation of computing resources, enabling the model to process longer historical sequences under limited training time and GPU memory conditions, enhancing deployment flexibility and scalability.

[0211] 6、Provide uncertainty output and dynamic correction mechanism, enhance system prediction reliability:

[0212] The mainstream model output is a single point prediction, which lacks expression of prediction confidence, cannot evaluate prediction instability in high-risk situations, and does not have feedback and correction ability for low-confidence samples. This application introduces Bayesian neural network, outputs multiple samples through MC Dropout and other methods, constructs prediction distribution, and calculates variance or confidence interval accordingly. The prediction result with uncertainty evaluation can be output, and the correction mechanism or auxiliary manual intervention is triggered in the confidence insufficient scene, which improves the safety and decision-making ability of the system in industrial key tasks.

[0213] Through integrated innovation in multiple key links such as frequency domain modeling, structure focusing, residual compensation, resource compression and risk control, the application overcomes the technical bottlenecks of existing methods in long-term trend modeling, local disturbance identification, calculation scalability and prediction reliability. The system is particularly suitable for industrial-level scenarios with high precision and strong robustness requirements, such as power load prediction, equipment early warning maintenance, financial quantitative prediction, etc., and has significant engineering practical value and industrial landing potential.

[0214] Figure 3 A time series modeling device according to the application is shown, as shown in Figure 3 The device 30 includes:

[0215] The acquisition module 301 is configured to acquire original time series data of a to-be-predicted task, and perform wavelet transform processing on the original time series data to obtain a multi-scale component sequence.

[0216] The execution module 302 is configured to sequentially perform reversible normalization, sequence slicing and vector embedding processing on the multi-scale component sequence to obtain an embedded vector set.

[0217] The micro K-Means clustering algorithm is used to calculate the clustering label of each embedded vector in the embedded vector set, and the attention mask matrix is generated according to the clustering label.

[0218] The embedded vector set and the attention mask matrix are input into the Transformer encoder to obtain an initial prediction result output by the Transformer encoder.

[0219] According to the error between the initial prediction result and the target output label of the to-be-predicted task, the trainable parameters of the Transformer encoder and the micro K-Means clustering algorithm are iteratively updated until the error meets the preset convergence condition, and a time series prediction model for the to-be-predicted task is obtained.

[0220] Therefore, by cooperation of the above technical features, in the industrial-grade time series scene with high noise, multi-scale and long dependence, the trend and disturbance can be better separated, the key fragments with similar structures can be focused and irrelevant attention can be inhibited, so that the accuracy and reliability of long-term time series prediction are improved.

[0221] The embodiment of the present application provides a network device 40, as shown in the figure, the network device 40 comprises a processor 401, a memory 402 and a program stored in the memory 402 and executable on the processor 401, when the program is executed by the processor 401, the steps of the method for time series modeling shown in the above embodiment are implemented. Figure 4

[0222] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by the processor, the steps of the method for time series modeling shown in the above embodiment are implemented, and the same technical effects can be achieved, to avoid repetition, which will not be repeated here. The computer readable storage medium is, for example, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk.

[0223] The embodiment of the present application further provides a computer program product, comprising computer instructions, when the computer instructions are executed by the processor, the steps of the method for time series modeling shown in the above embodiment are implemented, and the same technical effects can be achieved, to avoid repetition, which will not be repeated here.

[0224] It should be noted that in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the existence of another identical element in the process, method, article or device comprising the element.

[0225] ​Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the contribution to the prior art can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), including a number of instructions to make a terminal (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0226] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, but not limited, those skilled in the art can make many forms without departing from the purpose of the present application and the scope of the claims under the inspiration of the present application, all belong to the protection of the present application.

Claims

1. A time series modeling method, characterized in that, The method includes: Obtain the original time series data of the task to be predicted, and perform wavelet transform processing on the original time series data to obtain a multi-scale component sequence; The multi-scale component sequence is sequentially subjected to reversible normalization, sequence slicing, and vector embedding to obtain a set of embedded vectors. The clustering label of each embedding vector in the set of embedding vectors is calculated using the differentiable K-Means clustering algorithm, and an attention mask matrix is ​​generated based on the clustering label; The embedding vector set and the attention mask matrix are input together into the Transformer encoder to obtain the initial prediction result output by the Transformer encoder; Based on the error between the initial prediction result and the target output label of the task to be predicted, the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm are iteratively updated until the error meets the preset convergence condition, thereby obtaining a time-series prediction model for the task to be predicted.

2. The method according to claim 1, characterized in that, Obtain the original time series data of the task to be predicted, and perform wavelet transform processing on the original time series data to obtain a multi-scale component sequence, including: The original time series data is convolved and downsampled using a low-pass filter to extract approximate coefficient components that reflect long-term trends. The original time series data is convolved and downsampled using a high-pass filter to extract detail coefficients that reflect high-frequency details. The approximation coefficient components are concatenated with the detail coefficient components to obtain the multi-scale component sequence.

3. The method according to claim 1, characterized in that, The clustering label for each embedding vector in the set of embedding vectors is calculated using the differentiable K-Means clustering algorithm. An attention mask matrix is ​​then generated based on these clustering labels, including: Calculate the Euclidean distance between each embedded vector in the set of embedded vectors and the preset learnable cluster centers, assign each vector to the cluster with the closest Euclidean distance, and obtain the cluster label of each embedded vector; An attention mask matrix is ​​generated based on the clustering labels; The learnable cluster centers serve as trainable parameters of the differentiable K-Means clustering algorithm and are dynamically adjusted during the training of the time-series prediction model based on the backpropagation of the clustering loss constructed from the Euclidean distance.

4. The method according to claim 1, characterized in that, The embedding vector set and the attention mask matrix are input together into the Transformer encoder to obtain the initial prediction result output by the Transformer encoder, including: The set of embedded vectors is projected into a query matrix, a key matrix, and a value matrix, respectively. The dot product of the query matrix and the key matrix is ​​masked using the attention mask matrix to obtain the masked attention weights; The value matrix is ​​weighted and summed using the mask attention weights, and the weighted sum is processed by a feedforward network to output the initial prediction result.

5. The method according to any one of claims 1-4, characterized in that, After iteratively updating the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm based on the error between the initial prediction result and the target output label of the task to be predicted, until the error satisfies a preset convergence condition, and obtaining a temporal prediction model for the task to be predicted, the method further includes: The original time series data is input into a temporal convolutional network to perform residual modeling on the non-stationary high-frequency disturbances contained in the original time series data, thereby obtaining residual prediction results. Based on the residual prediction results and the initial prediction results, a fused prediction value is obtained. Based on the Bayesian inference principle, the time series prediction model is used to perform multiple forward propagations on the original time series data to obtain multiple sampled prediction samples for the fused prediction value. Statistical analysis is then performed on the sampled prediction samples to obtain a time series prediction result that includes uncertainty estimation.

6. The method according to claim 5, characterized in that, The original time series data is input into a temporal convolutional network to perform residual modeling on the non-stationary high-frequency disturbances contained in the original time series data, obtaining residual prediction results. A fused prediction value is then obtained based on the residual prediction results and the initial prediction results, including: The original time series data is obtained and input into the temporal convolutional network composed of stacked causal convolutional layers and dilated convolutional layers. The expanded receptive field of the dilated convolutional layer is used to capture local mutation features in the original time series data, and the residual prediction results for high-frequency perturbations are output based on the local mutation features. The residual prediction result is superimposed with the initial prediction result to obtain the superimposed prediction value; The superimposed prediction values ​​are reverse-engineered using the inverse transform parameters of the RevIN algorithm to obtain the fused prediction values.

7. A timing modeling apparatus, characterized in that, The device includes: The acquisition module is used to acquire the original time series data of the task to be predicted, and to perform wavelet transform processing on the original time series data to obtain a multi-scale component sequence. The execution module is used to sequentially perform reversible normalization, sequence slicing, and vector embedding processing on the multi-scale component sequence to obtain an embedded vector set; The clustering label of each embedding vector in the set of embedding vectors is calculated using the differentiable K-Means clustering algorithm, and an attention mask matrix is ​​generated based on the clustering label; The embedding vector set and the attention mask matrix are input together into the Transformer encoder to obtain the initial prediction result output by the Transformer encoder; Based on the error between the initial prediction result and the target output label of the task to be predicted, the trainable parameters of the Transformer encoder and the differentiable K-Means clustering algorithm are iteratively updated until the error meets the preset convergence condition, thereby obtaining a time-series prediction model for the task to be predicted.

8. A network device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of a timing modeling method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a timing modeling method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of a timing modeling method as described in any one of claims 1 to 6.

Images