Hierarchical time sequence prediction method and system based on tree structure coding
By adopting tree structure encoding and hierarchical encoding methods in hierarchical time series prediction, the problem of failing to fully utilize tree hierarchical structure information in the prior art is solved, and higher prediction accuracy and model performance are achieved.
Patent Information
- Application Number
- CN202411472398.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2025-02-18
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing hierarchical time series prediction methods fail to make full use of tree hierarchical information, resulting in limited prediction accuracy.
Using a tree structure encoding method, through hierarchical encoding and tree structure mask, the tree structure information behind the hierarchical time series is fully utilized to perform information fusion and prediction.
Effectively utilizing tree structure information improves the accuracy of time series prediction and improves the model's understanding and prediction ability of hierarchical time series.
Smart Images
Figure CN119475051A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hierarchical time series prediction, and in particular to a hierarchical time series prediction method based on tree structure coding. Background Art
[0002] In the field of time series forecasting, traditional research methods mainly focus on the forecasting of a single time series, while ignoring the fact that in many practical application scenarios, time series data often has the characteristics of a hierarchical structure. Hierarchical time series forecasting can help companies and organizations better understand and predict future trends, thereby making more informed decisions. For example, sales data in the retail supply chain field can be constructed into a hierarchical structure based on factors such as regions, stores, and product categories, and sales volumes of goods at different levels can be predicted, thereby optimizing inventory allocation and logistics operations, and improving the overall efficiency of the supply chain and customer satisfaction; tourism traffic data can be constructed into a hierarchical structure based on geographical levels, thereby adjusting and optimizing tourism services in advance and formulating more targeted marketing strategies.
[0003] These hierarchical time series data not only contain the information of a single time series, but also contain the hierarchical relationships in the hierarchical structure, which have an important impact on the accuracy of the prediction. On the one hand, time series data at different levels are complementary. High-level time series are aggregated from low-level time series data, so they present more macro and overall trends and laws, while low-level time series contain more detailed information, reflecting local changes and short-term fluctuations. On the other hand, hierarchical information can explain the possible mutual influence and transmission mechanism between different levels. If the correlation information between different levels can be fully utilized, it will help the model to more accurately capture the structural characteristics and internal relationships of the data.
[0004] Although the importance of hierarchical time series prediction is becoming increasingly prominent, existing research still faces challenges in processing such data. On the one hand, although general time series prediction models such as NLinear and PatchTST have achieved advanced results, they often fail to effectively capture and utilize information in the hierarchy. Most existing research on hierarchical time series prediction focuses on using hierarchical relationships to constrain the prediction results of the model to meet strict hierarchical consistency. For example, Buttom-Up and Top-Down use hierarchical structures to adjust the final prediction results, but do not fully utilize the value of hierarchical information in prediction. On the other hand, although the Transformer model performs well in processing sequence data, its application on tree-structured data has not been fully studied. Summary of the invention
[0005] In order to solve the problem of insufficient utilization of tree-structured hierarchical information in existing hierarchical time series prediction methods, the present invention proposes a hierarchical time series prediction method based on tree-structured coding. By means of hierarchical coding and tree-structured masking, the tree-structured information behind the hierarchical time series is fully utilized, thereby achieving accurate prediction of the hierarchical time series.
[0006] In order to solve the above technical problems, the present invention provides the following technical solution: a hierarchical time series prediction method based on tree structure coding, comprising the following steps:
[0007] Step 1: Hierarchical time series preprocessing: obtain a time series with a hierarchical structure, standardize the original time series, construct a tree structure based on the hierarchical structure information of the time series, and establish the corresponding relationship between the time series and the nodes in the tree;
[0008] Step 2: Time series encoding: Use the time series encoder to encode each time series independently as the self-representation vector of the corresponding node in the tree;
[0009] Step 3: Obtain the final representation vector, and use hierarchical encoding to add a learnable hierarchical encoding vector shared by the same layer of time series to the representation vector of each time series in the tree hierarchy;
[0010] Step 4: Obtain key information. The final representation vector is passed through the Transformer encoder module with a tree structure mask for effective information fusion. The tree structure mask is used in the multi-head attention layer to help the model focus on key information.
[0011] Step 5: Obtaining the hierarchical time series prediction results, and obtaining the final prediction results of each node through the time series prediction module.
[0012] In a preferred embodiment, since the numerical values of the hierarchical time series may differ greatly, it is necessary to calculate the mean and standard deviation of each time series obtained, and use the mean and standard deviation to standardize the time series to obtain the i-th time series after standardization. It can be expressed as:
[0013]
[0014] Among them, μ i is the mean, σ i is the standard deviation.
[0015] In a preferred embodiment, in step one, a tree structure is constructed according to the hierarchical information of the time series, each time series corresponds to each node in the tree structure, and the hierarchical relationship of the time series corresponds to the parent-child relationship in the tree structure. The nodes in the tree structure are numbered in the order of hierarchical traversal, and each node is numbered sequentially from top to bottom and from left to right.
[0016] In a preferred embodiment, after each time series is standardized in step 2, a time series encoder is used to extract independent features from each time series. The time series encoder is a learnable linear transformation Embedding(·) used to convert the original time series data into a more expressive embedding vector. where d h is the dimension of the vector.
[0017] In a preferred implementation, in step 3, layer(i) is used to identify the layer where the i-th time series is located. Represents the shared learnable hierarchical code of the level where the i-th node is located. Add the embedding vector corresponding to each time series and the corresponding shared hierarchical code to obtain the final representation vector corresponding to each time series. The final representation vector formula is:
[0018]
[0019] In a preferred embodiment, the input of the self-attention module in the Transformer encoder of the lth layer in step 4 is:
[0020]
[0021] in represents the hidden layer vector of the i-th time series after the l-1-th layer Transformer encoder;
[0022] For each attention head in the multi-head self-attention mechanism, the query matrix needs to be calculated Key Matrix Sum Matrix The calculation formula is:
[0023] Q=H (l-1) W Q
[0024] K=H (l-1) W K
[0025] V=H (l-1) W V
[0026] in are three learnable linear transformation matrices, d k represents the dimension of query vector and key vector, d v Represents the dimension of the value vector;
[0027] Each query vector is dot-multiplied with each key vector and scaled according to the dimension of the key vector to obtain the self-attention score matrix The calculation formula is:
[0028]
[0029] In a preferred embodiment, in step 4, a tree structure mask matrix uniquely determined by the tree hierarchy itself needs to be constructed to guide the model to perform information fusion in a targeted manner. The tree structure mask matrix M∈{0,-∞} n×n , contains only two elements, 0 and -∞. For each element M in the matrix M ij The calculation formula is:
[0030]
[0031] Directly add the self-attention score matrix A to the tree structure mask M to obtain the masked attention score matrix Then use the softmax function for normalization, and use it as the weight to perform weighted summation on the value vector to obtain the new hidden layer state The calculation formula is:
[0032]
[0033] The hidden layer state and the input H of the current layer (l-1) After the residual link is regularized, it is passed through a feedforward neural network with residual links and regularized to obtain the output H of the current layer of the Transformer encoder (l) .
[0034] In a preferred embodiment, in step 5, after comprehensively modeling the time series in the entire hierarchical structure and fusing the information between the layers using the N-layer Transformer encoder, the hidden layer state of each time series generated by the N-th layer Transformer encoder is converted into Input into the time series prediction module to get the future prediction value The time series prediction module uses a learnable linear transformation Projection(·), where h is the length of the prediction window and the calculation formula is:
[0035]
[0036] Use the original mean and standard deviation of the time series in the preprocessing to perform inverse normalization to ensure that the prediction results are consistent with the distribution of the original data and obtain the final prediction results. The calculation formula is:
[0037]
[0038] A hierarchical time series prediction system based on tree structure coding, comprising:
[0039] Data preprocessing and hierarchical encoding module, which is used to effectively represent the original time series information and fully integrate the hierarchical position information of the nodes;
[0040] Transformer encoder with tree structure mask, which is used to effectively fuse the tree structure information and the feature association between nodes;
[0041] The time series prediction module is used to effectively predict the future change trend of the sequence.
[0042] In a preferred embodiment, it also includes a training module, which is used to perform end-to-end training on the hierarchical information encoding, Transformer encoder, and time series predictor according to the hierarchical time series with a tree hierarchy.
[0043] Compared with the prior art, the beneficial effects achieved by the present invention are:
[0044] Based on the tree structure information behind the hierarchical time series, the present invention uses hierarchical coding to enable each node to judge the relative level of other nodes to itself when calculating the attention score, and uses the tree structure mask to ensure that each node in the tree structure only interacts with its direct ancestor and descendant nodes, so that the tree structure information in the hierarchical time series can be effectively utilized, so that the Transformer encoder can effectively model when fusing information, further improving the accuracy of time series prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0046] Figure 1 It is a schematic diagram of a hierarchical time series prediction method based on tree structure coding according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0047] The following specific embodiments illustrate the implementation of the present invention, and the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Please refer to Figure 1 As shown, the present invention provides a hierarchical time series prediction method based on tree structure coding, comprising the following steps:
[0049] Step 1: hierarchical time series preprocessing, obtaining a time series with a hierarchical structure, constructing a tree structure based on the hierarchical structure information of the time series, and establishing the corresponding relationship between the time series and the nodes in the tree;
[0050] Step 2: Time series encoding: Use the time series encoder to encode each time series independently as the self-representation vector of the corresponding node in the tree;
[0051] Step 3: Obtain the final representation vector, and use hierarchical encoding to add a learnable hierarchical encoding vector shared by the same layer of time series to the representation vector of each time series in the tree hierarchy;
[0052] Step 4: Obtain key information. The final representation vector is passed through the Transformer encoder module with a tree structure mask for effective information fusion. The tree structure mask is used in the multi-head attention layer to help the model focus on key information.
[0053] Step 5: Obtaining the hierarchical time series prediction results, and obtaining the final prediction results of each node through the time series prediction module.
[0054] In a preferred embodiment, the input of the model is a batch of time series with a tree-like hierarchical structure, which can be expressed as X = {x 1 ,x 2 ,…,x n}, that is, n hierarchical time series, where represents the i-th sequence, w is the input window length of the time series. Since the numerical values between the hierarchical time series may vary greatly, it is necessary to calculate the mean and standard deviation of each time series obtained, and use the mean and standard deviation to standardize the time series to obtain the i-th time series after standardization. It can be expressed as:
[0055]
[0056] Among them, μ i is the mean, σ i is the standard deviation.
[0057] In a preferred embodiment, in step one, a tree structure is constructed according to the hierarchical information of the time series, each time series corresponds to each node in the tree structure, and the hierarchical relationship of the time series corresponds to the parent-child relationship in the tree structure, and the nodes in the tree structure are numbered in the order of hierarchical traversal, and each node is numbered sequentially from top to bottom and from left to right.
[0058] In a preferred embodiment, after each time series is standardized in step 2, a time series encoder is used to extract independent features from each time series. The time series encoder is a learnable linear transformation Embedding(·) used to convert the original time series data into a more expressive embedding vector. where d h is the dimension of the vector.
[0059] In a preferred implementation, in step 3, layer(i) is used to identify the layer where the i-th time series is located. Represents the shared learnable hierarchical code of the level where the i-th node is located. Add the embedding vector corresponding to each time series and the corresponding shared hierarchical code to obtain the final representation vector corresponding to each time series. The final representation vector formula is:
[0060]
[0061] In a preferred embodiment, the input of the self-attention module in the Transformer encoder of the lth layer in step 4 is:
[0062]
[0063] in represents the hidden layer vector of the i-th time series after the l-1-th layer Transformer encoder;
[0064] For each attention head in the multi-head self-attention mechanism, the query matrix needs to be calculated Key Matrix Sum Matrix The calculation formula is:
[0065] Q=H (l-1) W Q
[0066] K=H (l-1) W K
[0067] V=H (l-1) W V
[0068] in are three learnable linear transformation matrices, d k represents the dimension of query vector and key vector, d v Represents the dimension of the value vector;
[0069] Each query vector is dot-multiplied with each key vector and scaled according to the dimension of the key vector to obtain the self-attention score matrix The calculation formula is:
[0070]
[0071] In a preferred embodiment, in step 4, a tree structure mask matrix uniquely determined by the tree hierarchy itself needs to be constructed to guide the model to perform information fusion in a targeted manner. The tree structure mask matrix M∈{0,-∞} n×n , which contains only two elements, 0 and -∞, for each element M in the matrix M ij The calculation formula is:
[0072]
[0073] Directly add the self-attention score matrix A to the tree structure mask M to obtain the masked attention score matrix Then use the softmax function for normalization, and use it as the weight to perform weighted summation on the value vector to obtain the new hidden layer state The calculation formula is:
[0074]
[0075] The hidden layer state and the input H of the current layer (l-1) After the residual link is regularized, it is passed through a feedforward neural network with residual links and regularized to obtain the output H of the current layer of the Transformer encoder (l) .
[0076] In a preferred embodiment, in step 5, after comprehensively modeling the time series in the entire hierarchical structure and fusing the information between the layers using the N-layer Transformer encoder, the hidden layer state of each time series generated by the N-th layer Transformer encoder is converted into Input into the time series prediction module to get the future prediction value The time series prediction module uses a learnable linear transformation Projection(·), where h is the length of the prediction window and the calculation formula is:
[0077]
[0078] The original mean and standard deviation of the time series in the preprocessing are used for inverse normalization to ensure that the prediction results are consistent with the distribution of the original data. The final prediction results are obtained. The calculation formula is:
[0079]
[0080] A hierarchical time series prediction system based on tree structure coding, comprising:
[0081] Data preprocessing and hierarchical encoding module, which is used to effectively represent the original time series information and fully integrate the hierarchical position information of the nodes;
[0082] Transformer encoder with tree structure mask, which is used to effectively fuse the tree structure information and the feature association between nodes;
[0083] The time series prediction module is used to effectively predict the future change trend of the sequence.
[0084] In a preferred embodiment, it also includes a training module, which is used to perform end-to-end training on the hierarchical information encoding, Transformer encoder, and time series predictor according to the hierarchical time series with a tree hierarchy.
[0085] Example:
[0086] This experiment uses a public dataset with a tree-like hierarchical structure, the M5 dataset. The dataset contains the daily sales of more than 30,000 items provided by a company from January 2011 to June 2016, and can construct five hierarchical structures such as states, stores, product categories, departments, and specific products. In this experiment, a 5-layer tree is constructed from the original data. The leaf node data is summed according to the tree structure to obtain the time series data of each non-leaf node. There are a total of 114 tree nodes, corresponding to 114 time series. From top to bottom, the number of nodes in each layer of the tree is 1, 3, 10, 30, and 70 respectively.
[0087] (2) Comparative test
[0088] In order to fully verify the effectiveness of the model of the present invention, this experiment selected two of the most representative advanced time series prediction models and combined them with some cutting-edge technologies in the field of hierarchical time series prediction for experimental comparison. The baseline models compared with the model of the present invention include:
[0089] NLinear: A state-of-the-art time series forecasting model. After normalizing the input time series, a single linear layer is used to forecast the time series, surpassing many Transformer-based time series forecasting models.
[0090] PatchTST: A current advanced time series prediction model. It uses a channel-independent strategy to split the input time series into several segments, then uses the Transformer encoder to model them, and finally uses a linear transformation to achieve time series prediction.
[0091] Bottom-Up: A hierarchical time series forecasting method that only forecasts the bottom-level time series separately, and then uses a bottom-up cumulative summation method to obtain the forecast results of the higher-level time series.
[0092] Top-Down: A hierarchical time series prediction method. Only the top-level time series is predicted separately, and then the prediction results of the top-level time series are distributed downward layer by layer according to a certain ratio to obtain the prediction results of other time series.
[0093] MinT: A hierarchical time series forecasting method. The results of preliminary forecasts are calibrated using the optimal combination method.
[0094] ERM: A hierarchical time series forecasting method. Similar to MinT, but the hierarchical time series forecasting problem is converted into an empirical risk minimization problem for solution.
[0095] The evaluation indicators used in this experiment are mean absolute percentage error (MAPE) and weighted mean absolute percentage error (w-MAPE). The calculation formulas of these two evaluation indicators are as follows:
[0096]
[0097] The test results are shown in Table 1:
[0098]
[0099]
[0100] Table 1
[0101] As can be seen from Table 1, the performance of the present invention in hierarchical time series prediction is better than that of other compared methods. Among them, Bottom-Up and Top-Down only predict the bottom or top time series, and finally use the hierarchy to accumulate and sum upward or distribute downward. Both MinT and ERM methods calibrate the prediction results of the time series of the entire hierarchy based on the preliminary prediction results of each time series. Although this step utilizes hierarchical information to a certain extent, these methods make separate predictions for each time series when generating preliminary prediction results, and fail to make full use of hierarchical information, resulting in their prediction performance being inferior to the method used in the present invention. This result verifies the importance of hierarchical structure information in improving the accuracy of hierarchical time series prediction.
[0102] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A hierarchical time series prediction method based on tree structure coding, characterized in that: The following steps are involved: Step 1: Hierarchical time series preprocessing: obtain a time series with a hierarchical structure, standardize the original time series, construct a tree structure based on the hierarchical structure information of the time series, and establish the corresponding relationship between the time series and the nodes in the tree; Step 2: Time series encoding: Use the time series encoder to encode each time series independently as the self-representation vector of the corresponding node in the tree; Step 3: Obtain the final representation vector, and use hierarchical encoding to add a learnable hierarchical encoding vector shared by the same layer of time series to the representation vector of each time series in the tree hierarchy; Step 4: Obtain key information. The final representation vector is passed through the Transformer encoder module with a tree structure mask for effective information fusion. The tree structure mask is used in the multi-head attention layer to help the model focus on key information. Step 5: Obtaining the hierarchical time series prediction results, and obtaining the final prediction results of each node through the time series prediction module.
2. The hierarchical time series prediction method based on tree structure coding according to claim 1, characterized in that: In the step 1, the mean and standard deviation of each time series are calculated for the acquired time series, and the time series is standardized using the mean and standard deviation to obtain the i-th time series after the standardized processing. It can be expressed as: Among them, μ i is the mean, σ i is the standard deviation.
3. The hierarchical time series prediction method based on tree structure coding according to claim 2, characterized in that: In the step 1, a tree structure is constructed according to the hierarchical information of the time series, each time series corresponds to each node in the tree structure, and the hierarchical relationship of the time series corresponds to the parent-child relationship in the tree structure. The nodes in the tree structure are numbered in the order of hierarchical traversal, and the nodes are numbered sequentially from top to bottom and from left to right.
4. The hierarchical time series prediction method based on tree structure coding according to claim 3 is characterized in that: In step 2, after each time series is standardized, a time series encoder is used to extract independent features from each time series. The time series encoder is a learnable linear transformation Embedding(·) that is used to convert the original time series data into a more expressive embedding vector. where d h is the dimension of the vector.
5. The hierarchical time series prediction method based on tree structure coding according to claim 4, characterized in that: In step 3, layer(i) is used to identify the layer where the i-th time series is located. Represents the shared learnable hierarchical code of the level where the i-th node is located. Add the embedding vector corresponding to each time series and the corresponding shared hierarchical code to obtain the final representation vector corresponding to each time series. The final representation vector formula is:
6. The hierarchical time series prediction method based on tree structure coding according to claim 5, characterized in that: The input of the self-attention module in the Transformer encoder of the lth layer in step 4 is: in represents the hidden layer vector of the i-th time series after the l-1-th layer Transformer encoder; For each attention head in the multi-head self-attention mechanism, the query matrix needs to be calculated Key Matrix Sum Matrix The calculation formula is: Q=H (l-1) W Q K=H (l-1) W K V=H (l-1) W V in are three learnable linear transformation matrices, d k represents the dimension of query vector and key vector, d v Represents the dimension of the value vector; Each query vector is dot-multiplied with each key vector and scaled according to the dimension of the key vector to obtain the self-attention score matrix The calculation formula is:
7. The hierarchical time series prediction method based on tree structure coding according to claim 6, characterized in that: In step 4, a tree structure mask matrix uniquely determined by the tree hierarchy itself needs to be constructed to guide the model to perform information fusion in a targeted manner. The tree structure mask matrix M∈{0,-∞} n×n , contains only two elements, 0 and -∞. For each element M in the matrix M ij The calculation formula is: Directly add the self-attention score matrix A to the tree structure mask M to obtain the masked attention score matrix Then use the softmax function for normalization, and use it as the weight to perform weighted summation on the value vector to obtain the new hidden layer state The calculation formula is: The hidden layer state and the input H of the current layer (l-1) After the residual link is regularized, it is passed through a feedforward neural network with residual links and regularized to obtain the output H of the current layer of the Transformer encoder (l) .
8. The hierarchical time series prediction method based on tree structure coding according to claim 7, characterized in that: In step 5, after the N-layer Transformer encoder is used to comprehensively model the time series in the entire hierarchical structure and fuse the information between the layers, the hidden layer state of each time series generated by the N-th layer Transformer encoder is converted into Input into the time series prediction module to get the future prediction value The time series prediction module uses a learnable linear transformation Projection(·), where h is the length of the prediction window and the calculation formula is: Use the original mean and standard deviation of the time series in the preprocessing to perform inverse normalization to ensure that the prediction results are consistent with the distribution of the original data and obtain the final prediction results. The calculation formula is:
9. A hierarchical time series prediction system based on tree structure coding, characterized in that: A hierarchical time series prediction method based on tree structure coding according to any one of claims 1 to 8, comprising: Data preprocessing and hierarchical encoding module, which is used to effectively represent the original time series information and fully integrate the hierarchical position information of the nodes; Transformer encoder with tree structure mask, which is used to effectively fuse the tree structure information and the feature association between nodes; The time series prediction module is used to effectively predict the future change trend of the sequence.
10. A hierarchical time series prediction system based on tree structure coding according to claim 9, characterized in that: A training module is also included, which is used to perform end-to-end training on the hierarchical information encoding, the Transformer encoder, and the time series predictor according to the hierarchical time series with a tree hierarchy.