Feature extraction model training method, device and equipment, and time sequence data prediction method, device and equipment

Abstract

The invention relates to the technical field of data processing, in particular to a feature extraction model training method, device and equipment and a time sequence data prediction method and device. The first encoder performs first-time first feature extraction on the mask time sequence data based on different feature extraction strategies to obtain a first feature matrix corresponding to each time, and the second encoder performs first-time second feature extraction on the original data based on different feature extraction strategies to obtain a second feature matrix corresponding to each time; according to the method, features of different levels of data are obtained through multiple different feature extraction, features displayed on different time scales are obtained, a prediction feature matrix of each time of feature extraction is determined based on a decoder, and a target loss value is determined according to the deviation between the prediction feature matrix of each time of feature extraction and a second feature matrix. And parameter adjustment is carried out to obtain a feature extraction model which is more accurate in feature extraction of the time sequence data, so that more accurate prediction can be carried out based on the feature extraction model.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to feature extraction model training, time series data prediction methods, devices and equipment. Background Art

[0002] In the natural gas industry, accurate demand forecasting is essential for energy distribution, supply chain management, and price stability. However, natural gas consumption data has obvious multi-scale characteristics, which means that it exhibits different characteristics at different time scales, such as intraday fluctuations, seasonal trends, and long-term changes. The complexity of these multi-scale characteristics makes it difficult for traditional time series feature extraction methods to accurately capture all relevant temporal dynamic characteristics, resulting in insufficient accuracy in subsequent predictions based on the extracted features.

[0003] Therefore, how to improve the accuracy of feature extraction of time series data has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method, device and equipment for feature extraction model training and time series data prediction, so as to solve the problem in the prior art that features cannot be accurately extracted for time series data.

[0005] In a first aspect, the present application provides a method for training a feature extraction model, wherein the initial feature extraction model includes a first encoder, a second encoder, and a decoder, and the method includes:

[0006] Perform masking processing on the acquired sample time series data to obtain masked time series data after masking and original data of the masked part;

[0007] Inputting the masked time series data into the first encoder, the first encoder performs a pre-configured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different;

[0008] Inputting the original data into the second encoder, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different;

[0009] Input each of the first feature matrices into the decoder, and the decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on a cross self-attention mechanism;

[0010] According to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, a target loss value is determined, and the parameters of the initial feature extraction model are adjusted based on the target loss value, and the first encoder after the parameter adjustment is determined as the feature extraction model.

[0011] Furthermore, the first encoder performs a preconfigured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, including:

[0012] The first encoder divides the masked time series data into a target set number of data blocks, extracts a first feature matrix corresponding to each data block, and updates a second number of times the current first feature is extracted;

[0013] If the second number is less than the first number, the first feature matrices corresponding to every two adjacent data blocks are merged into one data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number.

[0014] Furthermore, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, including:

[0015] The second encoder determines the data length of a data block in the target set number of data blocks divided by the first encoder, divides the original data into a plurality of data blocks according to the data length, extracts a second feature matrix corresponding to each data block, and updates a third matrix of the current second feature extraction;

[0016] If the third number is less than the first number, the second characteristic matrices corresponding to each two adjacent data blocks are merged into the same data block, and the step of extracting the second characteristic matrix corresponding to each data block is continued until the third number is equal to the first number.

[0017] Furthermore, the first encoder divides the masked time series data into a target set number of data blocks, including:

[0018] The data before the masked part in the masked time series data is divided into a first set number of data blocks, and the data after the masked part is divided into a second set number of data blocks, wherein the first set number is an even number.

[0019] Further, determining the target loss value according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, includes:

[0020] For each feature extraction, a sub-loss value is determined according to the deviation between the first feature matrix and the second feature matrix corresponding to the feature extraction;

[0021] The sum of the sub-loss values ​​corresponding to each feature extraction is determined as the target loss value.

[0022] In a second aspect, the present application also provides a time series data prediction method, wherein the prediction model includes a feature extraction layer, a feature fusion layer, a linear layer and an output layer, wherein the feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model described above;

[0023] The method comprises:

[0024] Inputting the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction;

[0025] Input each of the first feature matrices into a feature fusion layer in sequence, the feature fusion layer sequentially concatenates each of the first feature matrices according to features to obtain a third feature matrix, and determines a fourth feature matrix corresponding to the third feature matrix based on an attention mechanism;

[0026] Inputting the fourth feature matrix into the linear layer for full connection processing to obtain a prediction feature matrix;

[0027] The prediction feature matrix is ​​input into the output layer for format conversion to obtain a prediction result.

[0028] Furthermore, after determining the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism and before inputting the fourth feature matrix into the linear layer for full connection processing, the method further includes:

[0029] Based on the splicing strategy when splicing to obtain the third feature matrix, split the fourth feature matrix to obtain a fifth feature matrix corresponding to each feature extraction;

[0030] Inputting the fifth feature matrix corresponding to each feature extraction into the linear layer for full connection processing, respectively, to obtain the prediction feature matrix corresponding to each feature extraction;

[0031] The step of inputting the prediction feature matrix into the output layer for format conversion to obtain a prediction result includes:

[0032] The prediction feature matrix corresponding to each feature extraction is input into the output layer, and the output layer performs fusion processing on the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result.

[0033] In a third aspect, the present application further provides a training device for a feature extraction model, wherein the initial feature extraction model includes a first encoder, a second encoder and a decoder, and the device includes:

[0034] The mask module is used to perform mask processing on the acquired sample time series data to obtain the masked time series data after masking and the original data of the masked part;

[0035] A first feature extraction module is used to input the masked time series data into the first encoder, the first encoder performs a pre-configured first number of first feature extractions, and obtains a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different; input the original data into the second encoder, the second encoder performs the first number of second feature extractions, and obtains a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different;

[0036] A first prediction module, configured to input each of the first feature matrices into the decoder, wherein the decoder predicts a predicted feature matrix corresponding to each first feature extraction of the masked part based on a cross self-attention mechanism;

[0037] A training module is used to determine a target loss value based on the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, and adjust the parameters of the initial feature extraction model based on the target loss value, and determine the first encoder after the parameter adjustment as the feature extraction model.

[0038] Furthermore, the first feature extraction module is specifically used for the first encoder to divide the mask time series data into a target set number of data blocks, extract the first feature matrix corresponding to each data block, and update the second number of current first feature extractions; if the second number is less than the first number, the first feature matrices corresponding to each two adjacent data blocks are merged into one data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number.

[0039] Furthermore, the first feature extraction module is specifically used for the second encoder to determine the data length of the data blocks in the target set number of data blocks divided by the first encoder, and divide the original data into multiple data blocks according to the data length, extract the second feature matrix corresponding to each data block, and update the third number of the current second feature extraction; if the third number is less than the first number, the second feature matrices corresponding to each adjacent two data blocks are merged into the same data block, and the step of extracting the second feature matrix corresponding to each data block is continued until the third number is equal to the first number.

[0040] Furthermore, the mask module is specifically used to divide the data before the masked part in the masked time series data into a first set number of data blocks, and divide the data after the masked part into a second set number of data blocks, wherein the first set number is an even number.

[0041] Furthermore, the training module is specifically used to determine the sub-loss value for each feature extraction according to the deviation between the first feature matrix and the second feature matrix corresponding to the feature extraction; and determine the sum of the sub-loss values ​​corresponding to each feature extraction as the target loss value.

[0042] In a fourth aspect, the present application further provides a time series data prediction device, wherein the prediction model includes a feature extraction layer, a feature fusion layer, a linear layer and an output layer, wherein the feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model described above;

[0043] The device comprises:

[0044] A second feature extraction module is used to input the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction;

[0045] The second prediction module is used to input each of the first feature matrices into the feature fusion layer in sequence, the feature fusion layer splices each of the first feature matrices in sequence according to the features to obtain a third feature matrix, and determines the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism; inputs the fourth feature matrix into the linear layer for full connection processing to obtain a predicted feature matrix; inputs the predicted feature matrix into the output layer for format conversion to obtain a prediction result.

[0046] Furthermore, the second prediction module is also used to split the fourth feature matrix based on the splicing strategy when splicing the third feature matrix to obtain a fifth feature matrix corresponding to each feature extraction; input the fifth feature matrix corresponding to each feature extraction into the linear layer for full connection processing to obtain a prediction feature matrix corresponding to each feature extraction; input the prediction feature matrix corresponding to each feature extraction into the output layer, the output layer fuses the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result.

[0047] In a fifth aspect, the present application also provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of the training method of the feature extraction model as described above, or the steps of the time series data prediction method when executing a computer program stored in the memory.

[0048] In a sixth aspect, the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the training method of the feature extraction model as described above, or the steps of the time series data prediction method.

[0049] In the embodiment of the present application, when the initial feature extraction model is trained, the acquired sample time series data is masked to obtain masked time series data after masking and the original data of the masked portion, the masked time series data is input into the first encoder, the first encoder performs a pre-configured first number of first feature extractions based on different feature extraction strategies, and obtains a first feature matrix corresponding to each first feature extraction, and at the same time the original data is input into the second encoder, the second encoder performs a first number of second feature extractions based on different feature extraction strategies, and obtains a second feature matrix corresponding to each second feature extraction, through multiple feature extractions with different feature extraction strategies, features of different levels of the masked time series data and the original data are extracted, The features displayed on different time scales are obtained, and each first feature matrix is ​​input into the decoder. The decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on the cross self-attention mechanism, and then determines the target loss value according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the second feature extraction of that time, and then adjusts the parameters of the initial feature extraction model based on the target loss value, and determines the first encoder after the parameter adjustment as the feature extraction model. That is to say, the second feature matrix extracted by the second encoder each time is used to guide the feature extraction of the first encoder, so as to obtain a more accurate feature extraction model for time series data feature extraction, so as to facilitate more accurate prediction based on the feature extraction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A schematic diagram of a multi-scale phenomenon existing in time series data provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of training a feature extraction model provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of data block division provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a model architecture of an initial feature extraction model provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of a time series data prediction process provided in an embodiment of the present application;

[0056] Figure 6 Experimental results on an open source dataset provided by the embodiments of the present application;

[0057] Figure 7 The experimental results on a natural gas demand prediction data set provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the structure of a training device for a feature extraction model provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of the structure of a time series data prediction device provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0062] Figure 1A schematic diagram of a multi-scale phenomenon existing in time series data provided in an embodiment of the present application, such as Figure 1 As shown in the figure, gas consumption (Comsumption) has been changing over time (Time Steps). Whether from a larger scale range (Coarse Scale) or a finer scale range (Finer Scale), gas consumption seems to have periodic changes, but the change pattern in different time scales is different. Therefore, in the natural gas industry, accurate demand forecasting is crucial for energy distribution, supply chain management and price stability. Therefore, the current forecasting of time series data in the natural gas industry is facing severe problems to be solved, including the following aspects:

[0063] 1. Multi-scale information capture: How to design a model to effectively capture and represent the characteristics of natural gas demand data at different time scales.

[0064] 2. Long-term dependency modeling: How to handle and model dependencies in long-term time series data, which is crucial for predicting future long-term trends.

[0065] 3. Cross-domain generalization capability: How the trained model maintains the accuracy and robustness of feature extraction and prediction in different geographical locations and different consumption scenarios.

[0066] 4. Feasibility in practical applications: The proposed models and methods need to be feasible in practical industrial applications, able to process large-scale data and provide real-time or near real-time predictions.

[0067] Contrastive learning is a self-supervised learning method that learns effective representations of data by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. In the fields of computer vision and natural language processing, contrastive learning has been proven to be a powerful unsupervised feature learning technique. In time series analysis, the goal of contrastive learning is to learn representations that can capture the intrinsic structure and patterns of time series while maintaining the temporal characteristics of time series data. By constructing positive and negative sample pairs, the model is trained to distinguish the similarities and differences between different time series. This method can help the model learn richer and more robust time series features, thereby improving the performance of downstream tasks such as classification, clustering, and prediction. However, contrastive learning has some disadvantages in time series analysis. First, time series data has temporal dependence and multi-scale characteristics, which traditional contrastive learning algorithms often ignore, resulting in the inability to fully capture the intrinsic structure of time series. Second, contrastive learning usually requires a large number of negative samples to enhance the discriminative ability of the model, which may result in excessive computational costs in practical applications, especially on large-scale datasets. In addition, the performance of contrastive learning depends heavily on the selection of contrast pairs between samples, and in time series data, how to design effective contrast pairs is a challenge, because inappropriate contrast pairs may mislead model learning. Finally, contrastive learning may encounter data sparsity problems when processing time series data, especially in long sequence prediction, where it is difficult to find enough negative samples to provide a stable gradient signal.

[0068] Mask modeling is another self-supervised learning paradigm that learns data representations by masking parts of the input data and training the model to predict these masked parts. This approach has achieved great success in the field of natural language processing, especially in language models such as BERT (Bidirectional Encoder Representations from Transformers). In time series analysis, mask modeling can force the model to learn the deep features and intrinsic dependencies of time series data by randomly masking certain parts of the time series data and then reconstructing the masked parts using the unmasked parts. This method can help the model capture the local features and long-term dependencies of time series data, and has potential application value for tasks such as time series prediction, classification, and clustering. In this way, the model can automatically learn useful feature representations from a large amount of unlabeled time series data without external annotations. However, there are still some challenges in time series data analysis. First, the continuity and contextual dependency of time series data require mask modeling methods to take into account the global information of the sequence, while existing methods often only focus on local information, resulting in the inability to effectively capture long-term dependencies. Second, mask modeling may encounter data imbalance problems when dealing with time series data, because the information at some time points may be more important than at other points, and a simple random masking strategy may ignore this imbalance. In addition, mask modeling may encounter future leakage problems in time series prediction, that is, when predicting the masked part, the model may inadvertently use future information, which will affect the prediction accuracy of the model. Finally, existing mask modeling methods may encounter difficulties when dealing with multivariate time series data because they usually do not take into account the correlation and interaction between different variables.

[0069] Based on the above problems, the embodiments of the present application provide a feature extraction model training, time series data prediction method, device and equipment. The initial feature extraction model in the training method includes a first encoder, a second encoder and a decoder. During the training process, the acquired sample time series data is masked to obtain masked time series data after masking, and the original data of the masked part; the masked time series data is input into the first encoder, and the first encoder performs a pre-configured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different; the original data is input into the second encoder, and the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different; each first feature matrix is ​​input into the decoder, and the decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on the cross self-attention mechanism; according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, the target loss value is determined, and the parameters of the initial feature extraction model are adjusted based on the target loss value, and the first encoder after the parameter adjustment is determined as the feature extraction model.

[0070] The feature extraction model training method provided in this application can accurately extract the features of the time series data of natural gas demand and can capture multi-scale features. The method for predicting time series data based on the feature extraction model obtained by the training method can provide more effective decision support in the natural gas industry. The following describes the training method of the feature extraction model and the process of the time series data prediction method in combination with various embodiments.

[0071] Figure 2 A flow chart of training a feature extraction model provided in an embodiment of the present application, such as Figure 2 As shown, the process may include the following steps:

[0072] S201: performing mask processing on the acquired sample time series data to obtain masked time series data after masking and original data of the masked portion.

[0073] The training method of the feature extraction model provided in the present application is applied to an electronic device, which may be a server, a PC, etc.

[0074] In the natural gas demand forecasting scenario, time series data usually have multi-scale characteristics, which means that they show different characteristics and trends at different time scales. For example, natural gas consumption may show daily fluctuations at the hourly level and seasonal changes at the monthly or annual level. Traditional time series forecasting methods often ignore this multi-scale characteristic when extracting features, resulting in the inability of feature extraction to capture information at different time scales, resulting in insufficient forecast accuracy.

[0075] In order to accurately predict natural gas demand, it is necessary to obtain a feature extraction model that can accurately extract features from time series data. The feature extraction model can accurately capture the features of different time scales in the time series data, so as to facilitate subsequent post-processing based on the features. In an embodiment of the present application, the initial feature extraction model can be trained to obtain a feature extraction model, which includes a first encoder, a second encoder and a decoder.

[0076] In order to obtain a high-accuracy feature extraction model, in the embodiment of the present application, a sample set is pre-configured, and the sample set includes a certain number of sample time series data, and the initial feature extraction model can be subsequently subjected to feature extraction based on the sample time series data. The sample time series data can be natural gas consumption data within a certain length of time, or natural gas purchase data of users within a certain length of time, or other types of time series data, and those skilled in the art can configure it as needed.

[0077] After obtaining the sample time series data, the sample time series data can be masked to obtain the masked time series data and the original data of the masked part. The masking process can be performed outside the initial feature extraction model or inside the initial feature extraction model, for example, at the input layer of the initial feature extraction model. When masking the sample time series data, the masking can be performed randomly, and the data length of the masked part can be pre-set or random. The position of the masking process on the sample time series data can be pre-set or random.

[0078] Exemplarily, the sample time series is ABCDEFGH. After masking, the masked time series data obtained may be AB[mask]FGH. Then, the original data of the masked part is CDE.

[0079] S202: Inputting the masked time series data into the first encoder, the first encoder performs a pre-configured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different.

[0080] After obtaining the masked time series data and the original data, the masked time series data can be input into the first encoder, and the first encoder performs the first feature extraction on the masked time series data. In order to capture the features at different time levels in the masked time series data, in an embodiment of the present application, different feature extraction strategies can be pre-configured, and the first encoder performs multiple feature extractions based on the configured different feature extraction strategies to capture the features of different time scales in the masked time series. For example, the first feature extraction of the first number is performed to obtain the first feature matrix corresponding to each first feature extraction, and the first number can be any integer, such as 3, 4, etc. Among them, the first feature matrix can be a row of multiple columns, or multiple rows and multiple columns. When the first feature matrix is ​​a row of multiple columns, then the first feature matrix can be understood as a feature vector. Exemplarily, in the first first feature extraction process, feature extraction method 1 can be used to extract features from the masked time series data, and in the second first feature extraction process, feature extraction method 2 can be used to extract features from the masked time series data, wherein the angles emphasized by feature extraction method 1 and feature extraction method 2 in the feature extraction process are different. Of course, in the first process of extracting the first feature, the masked time series data can be used for feature extraction to obtain the first feature matrix. In the second process of extracting the first feature, the first feature matrix obtained by the first feature extraction can be subjected to feature extraction to obtain the first feature matrix of the current round. In the third process of extracting the first feature, the first feature matrix obtained by the second feature extraction can be subjected to feature extraction to obtain the first feature matrix of the current round, and so on. In the embodiment of the present application, those skilled in the art can configure the feature extraction strategy when performing feature extraction as needed.

[0081] S203: Inputting the original data into the second encoder, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different.

[0082] In an embodiment of the present application, the original data can be input into a second encoder, and the second encoder performs a first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy for each second feature extraction is different. The methods for the first feature extraction and the second feature extraction of the corresponding times can be the same or different, as long as the dimensions of the feature matrix extracted by the final feature extraction can be the same.

[0083] S204: Input each of the first feature matrices into the decoder, and the decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on the cross self-attention mechanism.

[0084] In order to train the feature extraction capability of the first encoder, in an embodiment of the present application, a decoder can be used to predict the masked part according to each first feature matrix. Specifically, after obtaining the first feature matrix corresponding to each first feature extraction, each first feature matrix can be input into the decoder, and the decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on the cross-self-attention mechanism. How the decoder predicts based on the received feature matrix using the cross-self-attention mechanism is a prior art, and this process will not be repeated in the embodiment of the present application.

[0085] S205: Determine a target loss value based on the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, and adjust the parameters of the initial feature extraction model based on the target loss value, and determine the first encoder after the parameter adjustment as the feature extraction model.

[0086] After obtaining the predicted feature matrix corresponding to each first feature extraction, the target loss value can be determined for each feature extraction based on the deviation between the predicted feature matrix corresponding to the first feature extraction and the second feature matrix corresponding to the second feature extraction, and the parameters of the initial feature extraction model can be adjusted based on the target loss value. After a large amount of training, when the preset convergence conditions are met, it means that the feature extraction model training is completed, and the first encoder with adjusted parameters can be determined as the feature extraction model. Specifically, when determining the target loss value, the corresponding sub-loss value can be determined based on the deviation between the predicted feature matrix and the second feature matrix each time, and the average value of the sub-loss values ​​corresponding to each feature extraction can be determined as the target loss value.

[0087] Because in the embodiment of the present application, when the initial feature extraction model is trained, the acquired sample time series data is masked to obtain masked time series data after masking and the original data of the masked part, the masked time series data is input into the first encoder, and the first encoder performs a pre-configured first number of first feature extractions based on different feature extraction strategies to obtain a first feature matrix corresponding to each first feature extraction, and at the same time, the original data is input into the second encoder, and the second encoder performs a first number of second feature extractions based on different feature extraction strategies to obtain a second feature matrix corresponding to each second feature extraction, and through multiple feature extractions with different feature extraction strategies, features of different levels of the masked time series data and the original data are extracted. In this way, features displayed on different time scales are obtained, and each first feature matrix is ​​input into the decoder. The decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on the cross self-attention mechanism, and then determines the target loss value according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction. The parameters of the initial feature extraction model are adjusted based on the target loss value, and the first encoder with adjusted parameters is determined as the feature extraction model. In other words, the second feature matrix extracted by the second encoder each time is used to guide the feature extraction of the first encoder, so as to obtain a more accurate feature extraction model for time series data feature extraction, so as to facilitate more accurate prediction based on the feature extraction model.

[0088] In order to further improve the accuracy of feature extraction, based on the above embodiment, in the embodiment of the present application, the first encoder performs a preconfigured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, including:

[0089] The first encoder divides the masked time series data into a target set number of data blocks, extracts a first feature matrix corresponding to each data block, and updates a second number of times the current first feature is extracted;

[0090] If the second number is less than the first number, the first feature matrices corresponding to every two adjacent data blocks are merged into one data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number.

[0091] In order to further improve the accuracy of feature extraction, in an embodiment of the present application, time information on different time scales can be captured so that the model can learn short-term fluctuations and long-term trends at the same time. In an embodiment of the present application, the first encoder can divide the received mask time series data into a target set number of data blocks during feature extraction, and extract the first feature matrix corresponding to each data block respectively. Since the first encoder needs to perform feature extraction for the first number, after the features corresponding to each data block are extracted, the second number of the current first feature extraction can be updated. If the second number is less than the first number, it means that the first encoder has not reached the set number of feature extractions, and the first feature matrices corresponding to each adjacent two data blocks can be merged, and the merged feature matrix is ​​determined as a data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number. When the first feature matrix is ​​merged, the two first feature matrices can be spliced ​​front and back, and the spliced ​​first feature matrix is ​​mapped to a matrix of set dimension. For example, if the first feature matrix obtained is 32*128, then after merging the first feature matrices corresponding to the two data blocks, the dimension is 32*(128+128). The merged first feature matrix is ​​further subjected to dimensionality reduction processing. For example, the feature matrix after dimensionality reduction processing can be 32*128.

[0092] Specifically, assume that the first number is 3. Figure 3 A schematic diagram of data block division provided in an embodiment of the present application is as follows: Figure 3 As shown, after receiving the mask time series data, the first encoder can divide the visible part of the mask time series data into 8 data blocks according to a certain data length, and perform the first feature extraction for each data block. At this time, it is the first first feature extraction, that is, the second number is 1 at this time. Since the second number is smaller than the first number, it is necessary to merge the first feature matrices corresponding to the two adjacent data blocks into one data block, that is, merge the first feature matrices corresponding to the original 1st and 2nd data blocks into one data block, merge the first feature matrices corresponding to the original 3rd and 4th data blocks into one data block, and so on. After the first feature matrices corresponding to each data block are merged, the second first feature extraction is performed for each data block, and the second number is 2 at this time. Since the second number is smaller than the first number, it is necessary to merge the first feature matrices corresponding to the two adjacent data blocks into one data block, and so on.

[0093] In order to further improve the accuracy of feature extraction, on the basis of the above embodiments, in an embodiment of the present application, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, including:

[0094] The second encoder determines the data length of a data block in the target set number of data blocks divided by the first encoder, divides the original data into a plurality of data blocks according to the data length, extracts a second feature matrix corresponding to each data block, and updates a third matrix of the current second feature extraction;

[0095] If the third number is less than the first number, the second characteristic matrices corresponding to each two adjacent data blocks are merged into the same data block, and the step of extracting the second characteristic matrix corresponding to each data block is continued until the third number is equal to the first number.

[0096] Since it is necessary to guide the first encoder based on the second characteristic matrix extracted by the second encoder, the second encoder can determine the data length of the data block in the target set number of data blocks divided by the first encoder when determining the second characteristic matrix, and divide the original data into multiple data blocks based on the data length. In other words, it is ensured that the length of each data block is the same as the length of the data block initially processed by the first encoder.

[0097] After obtaining each data block, the second feature matrix corresponding to each data block can be extracted respectively. Since the second encoder also needs to perform the second feature extraction of the first number, after the features corresponding to each data block are extracted, the third number of the current second feature extraction can be updated. If the third number is less than the first number, it means that the second encoder has not reached the set number of feature extractions, and the second feature matrices corresponding to each two adjacent data blocks can be merged into one data block, and the steps of extracting the second feature matrix corresponding to each data block are continued until the third number is equal to the first number.

[0098] In the embodiment of the present application, adjacent fine-grained data blocks are merged into coarse-grained data blocks through a data block merging strategy.

[0099] In order to further improve the accuracy of feature extraction, based on the above embodiments, in an embodiment of the present application, the first encoder divides the mask time series data into a target set number of data blocks, including:

[0100] The data before the masked part in the masked time series data is divided into a first set number of data blocks, and the data after the masked part is divided into a second set number of data blocks, wherein the first set number is an even number.

[0101] By the above Figure 3It can be seen that between the data blocks initially divided, the 4th data block and the 5th data block are the masked parts, which also shows that the data corresponding to the 4th data block and the 5th data block are discontinuous. If the feature matrices corresponding to the two data blocks with discontinuous data content are merged during the data merging process, the accuracy of feature extraction will be seriously affected. Therefore, in order to further improve the accuracy of feature extraction, the first encoder should ensure that the contents recorded in the two merged data blocks are continuous when merging the data blocks when performing data division. That is to say, the first encoder divides the data before the masked part in the masked time series data into a first set number of data blocks, and divides the data after the masked part into a second set number of data blocks, wherein the first set number is an even number, for example, the first set number is any even number such as 2, 4, 6, 8, etc. The sum of the first set number and the second set number is the target set number described in the above embodiments.

[0102] In order to further improve the accuracy of feature extraction, on the basis of the above embodiments, in the embodiment of the present application, the target loss value is determined according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, including:

[0103] For each feature extraction, a sub-loss value is determined according to the deviation between the first feature matrix and the second feature matrix corresponding to the feature extraction;

[0104] The sum of the sub-loss values ​​corresponding to each feature extraction is determined as the target loss value.

[0105] In the embodiment of the present application, for each feature extraction, a sub-loss value may be determined according to a deviation between a first feature matrix and a second feature matrix corresponding to the feature extraction.

[0106] After obtaining each sub-loss value, the sum of the sub-loss values ​​corresponding to each feature extraction can be determined as the target loss value.

[0107] For example, the corresponding sub-loss value of the first feature extraction is A1, the corresponding sub-loss value of the second feature extraction is A2, and the corresponding sub-loss value of the third feature extraction is A3. Then, the final target loss value is A=A1+A2+A3, where A represents the target loss value.

[0108] The following describes the model training process in conjunction with a specific embodiment. Figure 4 A schematic diagram of a model architecture of an initial feature extraction model provided in an embodiment of the present application, such as Figure 4As shown, the initial feature extraction model includes a first encoder, a second encoder and a decoder, wherein the first encoder can be understood as a student encoder, and the second encoder can be understood as a teacher encoder. The student encoder is used to process visible time series fragments, i.e., masked time series data, to extract multi-scale features. The teacher encoder uses stop gradient to process the original data of the masked part to provide multi-stage feature-level guidance signals for model training. In other words, the teacher encoder provides multi-scale feature-level supervision signals to help the student encoder learn robust multi-scale features. The decoder uses a cross self-attention mechanism to reconstruct the masked part, that is, to predict the potential representation of the masked part based on the features of the student encoder.

[0109] Specifically, Figure 4 The broken line on the far left (left and right in the figure) side represents the sample time series data, and the sample time series data is masked to obtain the masked time series data and the original data of the masked part.

[0110] The masked time series data is input into the student encoder, and the student encoder processes the visible part of the masked time series data (patching & visble), for example, the data before the masked part of the masked time series data is divided into 4 data blocks, and the data after the masked part of the masked time series data is divided into 4 data blocks. Each small rectangle in the figure represents a data block, and the blank area between two groups of continuous data blocks represents the masked part. After the masked time series data is divided into multiple data blocks, the data corresponding to each data block is mapped in the potential space (Projection & Pos Embedding), that is, the data corresponding to each data block is encoded and mapped to the preset vector space to obtain the input feature matrix, and the input feature matrix is ​​input into Transformer Blocks for first feature extraction to obtain the first feature matrix corresponding to each data block. Transformer Blocks performs feature extraction based on the self-attention mechanism. After obtaining the first feature matrix corresponding to each data block, the first feature matrices corresponding to each adjacent two data blocks are merged (Merge). After the data is merged, the 4 data blocks before the masked part are merged into 2 data blocks, and the data after the masked part is divided into 4 data blocks and merged into 2 data blocks. The input feature matrix corresponding to each data block is input into Transformer Blocks for first feature extraction to obtain the first feature matrix corresponding to each data block. Exemplarily, when merging data blocks, the first feature matrices corresponding to the two data blocks can be spliced, for example, two 1*128 feature matrices are spliced ​​front and back to obtain a 1*256 feature matrix, and then the feature matrix is ​​subjected to dimensionality reduction processing to obtain the first feature matrix corresponding to the merged data block. After that, the first feature matrices corresponding to the data blocks are merged according to the above ideas, and this application will not go into details.

[0111] The original data is input to the teacher encoder, and the teacher encoder performs the second feature extraction the same number of times as the student encoder, and obtains the second feature matrix corresponding to each second feature extraction. The process of feature extraction by the teacher encoder is similar to that by the student encoder, and this embodiment of the present application will not be repeated.

[0112] After obtaining the first feature matrix and the second feature matrix output by the student encoder and the teacher encoder respectively for each feature extraction, a prediction feature matrix can be obtained based on the encoder. The feature extraction includes the first feature extraction and the second feature extraction.

[0113] For each feature extraction, the encoder uses the first feature matrix corresponding to this time as the K (key) matrix and V (value) matrix of the attention mechanism, and the blank matrix corresponding to the masked part can be used as the Q (query) matrix of the attention mechanism. The encoder then performs attention mechanism processing based on the Q, K, and V matrices, and obtains the predicted feature matrix through projection processing. Finally, the sub-loss value is determined based on the deviation between the predicted feature matrix corresponding to this feature extraction and the second feature matrix corresponding to the teacher encoder.

[0114] After obtaining the sub-loss value corresponding to each feature extraction, the sum of all sub-loss values ​​is determined as the target loss value, and the parameters of the initial feature extraction model are adjusted based on the target loss value. In an embodiment of the present application, a hierarchical self-distillation (HSD) loss is introduced to minimize the difference in feature levels and provide a learning signal of multi-scale information for the student encoder. The internal structure of the encoder (Decoder) is shown on the right side of the figure. This part is a prior art and will not be described in detail in the embodiment of the present application.

[0115] Time series forecasting is a hot topic that has long been of concern to industry and academia. It involves extracting patterns and trends from historical data to predict future behaviors or events. With the development of deep learning technology, time series forecasting methods have shifted from traditional statistical methods such as autoregressive integrated moving average model (ARIMA), seasonal decomposition, and exponential smoothing to deep learning-based methods. In particular, deep learning models such as recurrent neural networks (RNN), long short-term memory networks (LSTM), and convolutional neural networks (CNN) have achieved remarkable forecasting performance in multiple fields. In recent years, with the introduction of attention mechanisms and Transformer architectures, the accuracy of time series forecasting has been further improved. These models are able to capture long-range dependencies and complex patterns in time series data, and thus perform well in applications such as energy consumption, stock price fluctuations, and weather forecasting. In order to accurately predict time series data, based on the same inventive concept, an embodiment of the present application provides a time series data prediction method, which performs prediction based on a prediction model. The prediction model includes a feature extraction layer, a feature fusion layer, a linear layer and an output layer. The feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model described in the above embodiments. Figure 5 A flowchart of a time series data prediction process provided by an embodiment of the present application is as follows: Figure 5 As shown, the process may include the following steps:

[0116] S501: Input the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction.

[0117] The time series data prediction method provided in the present application is applied to an electronic device, which may be a server, a PC, a mobile terminal, etc.

[0118] In order to predict the data trend within a set time after the time series data, in an embodiment of the present application, a pre-trained prediction model can be used for prediction. In an embodiment of the present application, the received time series data to be processed can be input into the feature extraction layer to obtain the first feature matrix corresponding to each first feature extraction. Exemplarily, the time series data to be processed can be the gas consumption data for half a month before the current moment, and it is necessary to predict the gas consumption data for 24 hours after the current moment based on the time series data to be processed. The feature extraction layer is deployed with a feature extraction model, which can be understood as a first encoder that meets the convergence condition.

[0119] S502: Input each of the first feature matrices into a feature fusion layer in sequence, and the feature fusion layer sequentially concatenates each of the first feature matrices according to features to obtain a third feature matrix, and determines a fourth feature matrix corresponding to the third feature matrix based on an attention mechanism.

[0120] After obtaining the first feature matrix corresponding to each first feature extraction output by the feature extraction layer, each first feature matrix can be sequentially input into the feature fusion layer, and the feature fusion layer splices each first feature matrix sequentially according to the features to obtain a third feature matrix, and determines the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism. When the feature fusion layer splices each first feature matrix, a technician in this field can configure it as needed. For example, the first feature matrix obtained by each first feature extraction is sequentially spliced ​​to obtain a third feature matrix. Exemplarily, the first first feature extraction obtains a first feature matrix of 64*128 dimensions, the second first feature extraction obtains a first feature matrix of 32*128 dimensions, and the third first feature extraction obtains a first feature matrix of 16*128 dimensions, then the spliced ​​third feature matrix can be (64+32+16)*128 dimensions.

[0121] S503: Input the fourth feature matrix into the linear layer for full connection processing to obtain a prediction feature matrix.

[0122] After the fourth feature matrix is ​​obtained, the fourth feature matrix can be input into the linear layer for full connection processing to obtain a prediction feature matrix.

[0123] S504: Input the prediction feature matrix into the output layer for format conversion to obtain a prediction result.

[0124] After obtaining the prediction matrix, the prediction feature matrix can be input into the output layer for format conversion to obtain a prediction result that is easy to understand.

[0125] In an embodiment of the present application, a pre-trained feature extraction model is deployed in the feature extraction layer of the prediction model. Since the feature extraction model has the ability to capture features of different time levels during training, prediction based on the feature extraction model can capture the features of each time level of the time series data to be processed during the prediction process, so as to facilitate more accurate prediction.

[0126] It should be noted that the prediction model for time series data prediction in the embodiment of the present application can be obtained by fine-tuning training in advance. The process of using the prediction model to predict time series data can be summarized as follows: retain the pre-trained student encoder as a feature extractor and output multi-scale features at different levels. Input these features into the cross-scale attention module to establish associations between features of different scales. Input features of different scales into the linear layer and output predicted values. Aggregate the predicted values ​​of different scales to produce the final prediction result.

[0127] In order to further improve the accuracy of prediction, on the basis of the above embodiments, in the embodiment of the present application, after determining the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism, before inputting the fourth feature matrix into the linear layer for full connection processing, the method further includes:

[0128] Based on the splicing strategy when splicing to obtain the third feature matrix, split the fourth feature matrix to obtain a fifth feature matrix corresponding to each feature extraction;

[0129] Inputting the fifth feature matrix corresponding to each feature extraction into the linear layer for full connection processing, respectively, to obtain the prediction feature matrix corresponding to each feature extraction;

[0130] The step of inputting the prediction feature matrix into the output layer for format conversion to obtain a prediction result includes:

[0131] The prediction feature matrix corresponding to each feature extraction is input into the output layer, and the output layer performs fusion processing on the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result.

[0132] Since the feature extraction strategy of each first feature extraction is different, and the first feature matrix obtained by each first feature extraction represents the features corresponding to the entire time series data to be processed. Therefore, in order to further improve the accuracy of the prediction, after determining the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism, before inputting the fourth feature matrix into the linear layer for full connection processing, the fourth feature matrix can also be split based on the splicing strategy when splicing the third feature matrix to obtain the fifth feature matrix corresponding to each feature extraction.

[0133] Specifically, the first first feature extraction obtained a first feature matrix of 64*128 dimensions, the second first feature extraction obtained a first feature matrix of 32*128 dimensions, and the third first feature extraction obtained a first feature matrix of 16*128 dimensions. Then the spliced ​​third feature matrix can be (64+32+16)*128 dimensions. The attention mechanism generally does not change the dimension of the feature matrix. The fourth feature matrix obtained after processing the third feature matrix based on the attention mechanism is also (64+32+16)*128 dimensions. Then based on the above splicing strategy, the fourth feature matrix is ​​split, and the first 64*128-dimensional submatrix of the fourth feature matrix can be determined as the fifth feature matrix corresponding to the first feature extraction, the second 16*128-dimensional submatrix of the fourth feature matrix can be determined as the fifth feature matrix corresponding to the third feature extraction, and the remaining submatrix can be determined as the fifth feature matrix corresponding to the second feature extraction.

[0134] After obtaining the fifth feature matrix corresponding to each feature extraction, each fifth feature matrix can be input into the linear layer for full connection processing to obtain the prediction feature matrix corresponding to each feature extraction. In the embodiment of the present application, one linear layer can be included, or multiple linear layers can be included. For example, a first number of linear layers can be included, and any linear layer is used to process the feature matrix of the corresponding number of times.

[0135] After obtaining the prediction feature matrix corresponding to each feature extraction, the prediction feature matrix corresponding to each feature extraction can be input into the output layer, and the output layer performs fusion processing on the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result. When the output layer performs fusion processing, the sum of all prediction feature matrices can be determined as the fused feature matrix.

[0136] The Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation (HiMTM) method provided in this application aims to solve the problem of natural gas demand forecasting in long-term forecasting. Based on the introduction above, in order to overcome this challenge, HiMTM enhances the model's ability to extract multi-scale features through the following key technical components:

[0137] 1. Hierarchical multi-scale transformer (HMT): Capturing temporal information at different time scales through transformers at different levels, enabling the model to learn both short-term fluctuations and long-term trends.

[0138] 2. Decoupled encoder-decoder (DED): The encoder focuses on feature extraction, while the decoder focuses on pre-training pre-tasks, such as reconstructing masked time series parts.

[0139] 3. Hierarchical self-distillation (HSD): In the pre-training stage, the self-distillation mechanism provides multi-stage feature-level supervision signals to guide the model to learn multi-scale features at different levels.

[0140] 4. Cross-scale attention fine-tuning (CSA-FT): In the fine-tuning stage, by establishing associations between features of different scales, the model’s adaptability and prediction accuracy for downstream tasks are further improved.

[0141] Through these technical means, HiMTM can effectively capture and model the multi-scale characteristics of natural gas demand data, thereby improving the accuracy of long-term forecasts. This application has verified the advantages of HiMTM over existing self-supervised learning and end-to-end learning methods through extensive experiments on multiple mainstream data sets, especially in cross-domain prediction, where HiMTM has shown significant performance improvements. Figure 6 The experimental results on an open source dataset provided by the embodiment of this application are as follows: Figure 6As shown, the HiMTM provided in this application is compared with other models, such as PatchTST*, SimMTM, Ti-MAE, TSR, LaST, TF-C, CoST, TS2Vec and other models. From the results of predicting time series for the next 96, 192, 336, and 720 hours, the mean square error (MSE) and mean absolute error (MAE) indicators of the HiMTM provided in this application are relatively ideal.

[0142] In addition, HiMTM has also shown good results in actual natural gas demand forecasting applications, proving its potential and value in industrial applications. Figure 7 The experimental results on a natural gas demand prediction data set provided in the embodiment of the present application are as follows: Figure 7 As shown, the HiMTM provided in this application is compared with PatchTST*. From the results of predicting the time series for the next 7, 15, 30, and 60 hours, the MSE and MAE indicators of the HiMTM provided in this application are also the most ideal.

[0143] Figure 8 A structural diagram of a training device for a feature extraction model provided in an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0144] The masking module 801 is used to perform masking processing on the acquired sample time series data to obtain masked time series data after masking and the original data of the masked part;

[0145] The first feature extraction module 802 is used to input the masked time series data into the first encoder, and the first encoder performs a pre-configured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different; input the original data into the second encoder, and the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different;

[0146] A first prediction module 803, configured to input each of the first feature matrices into the decoder, and the decoder predicts a predicted feature matrix corresponding to each first feature extraction of the masked part based on a cross self-attention mechanism;

[0147] The training module 804 is used to determine the target loss value according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, and adjust the parameters of the initial feature extraction model based on the target loss value, and determine the first encoder after the parameter adjustment as the feature extraction model.

[0148] In a possible implementation, the first feature extraction module 802 is specifically used for the first encoder to divide the mask time series data into a target set number of data blocks, extract the first feature matrix corresponding to each data block, and update the second number of times the current first feature is extracted; if the second number is less than the first number, the first feature matrices corresponding to each two adjacent data blocks are merged into one data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number.

[0149] In a possible implementation, the first feature extraction module 802 is specifically used for the second encoder to determine the data length of a data block in the target set number of data blocks divided by the first encoder, and divide the original data into multiple data blocks according to the data length, extract the second feature matrix corresponding to each data block, and update the third number of the current second feature extraction; if the third number is less than the first number, the second feature matrices corresponding to each adjacent two data blocks are merged into the same data block, and the step of extracting the second feature matrix corresponding to each data block is continued until the third number is equal to the first number.

[0150] In a possible implementation, the mask module 801 is specifically used to divide the data before the masked part in the masked time series data into a first set number of data blocks, and divide the data after the masked part into a second set number of data blocks, wherein the first set number is an even number.

[0151] In a possible implementation, the training module 804 is specifically used to determine a sub-loss value for each feature extraction according to a deviation between a first feature matrix and a second feature matrix corresponding to the feature extraction; and determine the sum of the sub-loss values ​​corresponding to each feature extraction as the target loss value.

[0152] Fig. 9 A schematic diagram of the structure of a time series data prediction device provided in an embodiment of the present application, wherein the prediction model comprises a feature extraction layer, a feature fusion layer, a linear layer and an output layer, wherein the feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model described above, such as Fig. 9 As shown, the device comprises:

[0153] The second feature extraction module 901 is used to input the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction;

[0154] The second prediction module 902 is used to input each of the first feature matrices in sequence into the feature fusion layer, the feature fusion layer sequentially splices each of the first feature matrices according to the features to obtain a third feature matrix, and determines the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism; inputs the fourth feature matrix into the linear layer for full connection processing to obtain a predicted feature matrix; inputs the predicted feature matrix into the output layer for format conversion to obtain a prediction result.

[0155] In a possible implementation, the second prediction module 902 is further used to split the fourth feature matrix based on the splicing strategy when splicing the third feature matrix to obtain a fifth feature matrix corresponding to each feature extraction; input the fifth feature matrix corresponding to each feature extraction into the linear layer for full connection processing to obtain a prediction feature matrix corresponding to each feature extraction; input the prediction feature matrix corresponding to each feature extraction into the output layer, the output layer fuses the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result.

[0156] Based on the above embodiments, the present application also provides an electronic device, Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, it includes: a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004;

[0157] The memory 1003 stores a computer program. When the program is executed by the processor 1001, the processor 1001 performs the following steps:

[0158] Perform masking processing on the acquired sample time series data to obtain masked time series data after masking and original data of the masked part;

[0159] Inputting the masked time series data into the first encoder, the first encoder performs a pre-configured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different;

[0160] Inputting the original data into the second encoder, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different;

[0161] Input each of the first feature matrices into the decoder, and the decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on a cross self-attention mechanism;

[0162] According to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, a target loss value is determined, and the parameters of the initial feature extraction model are adjusted based on the target loss value, and the first encoder after the parameter adjustment is determined as the feature extraction model.

[0163] In a possible implementation, the first encoder performs a preconfigured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, including:

[0164] The first encoder divides the masked time series data into a target set number of data blocks, extracts a first feature matrix corresponding to each data block, and updates a second number of times the current first feature is extracted;

[0165] If the second number is less than the first number, the first feature matrices corresponding to every two adjacent data blocks are merged into one data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number.

[0166] In a possible implementation manner, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, including:

[0167] The second encoder determines the data length of a data block in the target set number of data blocks divided by the first encoder, divides the original data into a plurality of data blocks according to the data length, extracts a second feature matrix corresponding to each data block, and updates a third matrix of the current second feature extraction;

[0168] If the third number is less than the first number, the second characteristic matrices corresponding to each two adjacent data blocks are merged into the same data block, and the step of extracting the second characteristic matrix corresponding to each data block is continued until the third number is equal to the first number.

[0169] In a possible implementation manner, the first encoder divides the masked time series data into a target set number of data blocks, including:

[0170] The data before the masked part in the masked time series data is divided into a first set number of data blocks, and the data after the masked part is divided into a second set number of data blocks, wherein the first set number is an even number.

[0171] In a possible implementation, determining the target loss value according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction includes:

[0172] For each feature extraction, a sub-loss value is determined according to the deviation between the first feature matrix and the second feature matrix corresponding to the feature extraction;

[0173] The sum of the sub-loss values ​​corresponding to each feature extraction is determined as the target loss value.

[0174] In a possible implementation, the prediction model includes a feature extraction layer, a feature fusion layer, a linear layer, and an output layer, the feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model described above; the processor 1001 further performs the following steps:

[0175] Inputting the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction;

[0176] Input each of the first feature matrices into a feature fusion layer in sequence, the feature fusion layer sequentially concatenates each of the first feature matrices according to features to obtain a third feature matrix, and determines a fourth feature matrix corresponding to the third feature matrix based on an attention mechanism;

[0177] Inputting the fourth feature matrix into the linear layer for full connection processing to obtain a prediction feature matrix;

[0178] The prediction feature matrix is ​​input into the output layer for format conversion to obtain a prediction result.

[0179] In a possible implementation, after determining the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism and before inputting the fourth feature matrix into the linear layer for full connection processing, the method further includes:

[0180] Based on the splicing strategy when splicing to obtain the third feature matrix, split the fourth feature matrix to obtain a fifth feature matrix corresponding to each feature extraction;

[0181] Inputting the fifth feature matrix corresponding to each feature extraction into the linear layer for full connection processing, respectively, to obtain the prediction feature matrix corresponding to each feature extraction;

[0182] The step of inputting the prediction feature matrix into the output layer for format conversion to obtain a prediction result includes:

[0183] The prediction feature matrix corresponding to each feature extraction is input into the output layer, and the output layer performs fusion processing on the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result.

[0184] Since the principle of solving the problem by the above electronic device is similar to the training method of the feature extraction model or the time series data prediction method, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0185] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1002 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0186] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0187] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program executable by a processor is stored, so that the processor implements the following steps when executing:

[0188] Perform masking processing on the acquired sample time series data to obtain masked time series data after masking and original data of the masked part;

[0189] Inputting the masked time series data into the first encoder, the first encoder performs a pre-configured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different;

[0190] Inputting the original data into the second encoder, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different;

[0191] Input each of the first feature matrices into the decoder, and the decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on a cross self-attention mechanism;

[0192] According to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, a target loss value is determined, and the parameters of the initial feature extraction model are adjusted based on the target loss value, and the first encoder after the parameter adjustment is determined as the feature extraction model.

[0193] In a possible implementation, the first encoder performs a preconfigured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, including:

[0194] The first encoder divides the masked time series data into a target set number of data blocks, extracts a first feature matrix corresponding to each data block, and updates a second number of times the current first feature is extracted;

[0195] If the second number is less than the first number, the first feature matrices corresponding to every two adjacent data blocks are merged into one data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number.

[0196] In a possible implementation manner, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, including:

[0197] The second encoder determines the data length of a data block in the target set number of data blocks divided by the first encoder, divides the original data into a plurality of data blocks according to the data length, extracts a second feature matrix corresponding to each data block, and updates a third matrix of the current second feature extraction;

[0198] If the third number is less than the first number, the second characteristic matrices corresponding to each two adjacent data blocks are merged into the same data block, and the step of extracting the second characteristic matrix corresponding to each data block is continued until the third number is equal to the first number.

[0199] In a possible implementation manner, the first encoder divides the masked time series data into a target set number of data blocks, including:

[0200] The data before the masked part in the masked time series data is divided into a first set number of data blocks, and the data after the masked part is divided into a second set number of data blocks, wherein the first set number is an even number.

[0201] In a possible implementation, determining the target loss value according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction includes:

[0202] For each feature extraction, a sub-loss value is determined according to the deviation between the first feature matrix and the second feature matrix corresponding to the feature extraction;

[0203] The sum of the sub-loss values ​​corresponding to each feature extraction is determined as the target loss value.

[0204] In a possible implementation, the prediction model includes a feature extraction layer, a feature fusion layer, a linear layer, and an output layer, wherein the feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model described above; when executed by the processor, the following steps are also implemented:

[0205] Inputting the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction;

[0206] Input each of the first feature matrices into a feature fusion layer in sequence, the feature fusion layer sequentially concatenates each of the first feature matrices according to features to obtain a third feature matrix, and determines a fourth feature matrix corresponding to the third feature matrix based on an attention mechanism;

[0207] Inputting the fourth feature matrix into the linear layer for full connection processing to obtain a prediction feature matrix;

[0208] The prediction feature matrix is ​​input into the output layer for format conversion to obtain a prediction result.

[0209] In a possible implementation, after determining the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism and before inputting the fourth feature matrix into the linear layer for full connection processing, the method further includes:

[0210] Based on the splicing strategy when splicing to obtain the third feature matrix, split the fourth feature matrix to obtain a fifth feature matrix corresponding to each feature extraction;

[0211] Inputting the fifth feature matrix corresponding to each feature extraction into the linear layer for full connection processing, respectively, to obtain the prediction feature matrix corresponding to each feature extraction;

[0212] The step of inputting the prediction feature matrix into the output layer for format conversion to obtain a prediction result includes:

[0213] The prediction feature matrix corresponding to each feature extraction is input into the output layer, and the output layer performs fusion processing on the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result.

[0214] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0215] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0216] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0217] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0218] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for training a feature extraction model, characterized in that: The initial feature extraction model includes a first encoder, a second encoder and a decoder, and the method includes: Perform masking processing on the acquired sample time series data to obtain masked time series data after masking and original data of the masked part; Inputting the masked time series data into the first encoder, the first encoder performs a pre-configured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different; Inputting the original data into the second encoder, the second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different; Input each of the first feature matrices into the decoder, and the decoder predicts the predicted feature matrix corresponding to each first feature extraction of the masked part based on a cross self-attention mechanism; According to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, a target loss value is determined, and the parameters of the initial feature extraction model are adjusted based on the target loss value, and the first encoder after the parameter adjustment is determined as the feature extraction model.

2. The method according to claim 1, characterized in that The first encoder performs a preconfigured first number of first feature extractions to obtain a first feature matrix corresponding to each first feature extraction, including: The first encoder divides the masked time series data into a target set number of data blocks, extracts a first feature matrix corresponding to each data block, and updates a second number of times the current first feature is extracted; If the second number is less than the first number, the first feature matrices corresponding to every two adjacent data blocks are merged into one data block, and the step of extracting the first feature matrix corresponding to each data block is continued until the second number is equal to the first number.

3. The method according to claim 2, characterized in that The second encoder performs the first number of second feature extractions to obtain a second feature matrix corresponding to each second feature extraction, including: The second encoder determines the data length of a data block in the target set number of data blocks divided by the first encoder, divides the original data into a plurality of data blocks according to the data length, extracts a second feature matrix corresponding to each data block, and updates a third matrix of the current second feature extraction; If the third number is less than the first number, the second characteristic matrices corresponding to each two adjacent data blocks are merged into the same data block, and the step of extracting the second characteristic matrix corresponding to each data block is continued until the third number is equal to the first number.

4. The method according to claim 2, characterized in that: The first encoder divides the masked time series data into a target set number of data blocks, including: The data before the masked part in the masked time series data is divided into a first set number of data blocks, and the data after the masked part is divided into a second set number of data blocks, wherein the first set number is an even number.

5. The method according to claim 1, characterized in that: The determining the target loss value according to the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, comprises: For each feature extraction, a sub-loss value is determined according to the deviation between the first feature matrix and the second feature matrix corresponding to the feature extraction; The sum of the sub-loss values ​​corresponding to each feature extraction is determined as the target loss value.

6. A time series data prediction method, characterized in that: The prediction model includes a feature extraction layer, a feature fusion layer, a linear layer and an output layer, wherein the feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model according to any one of claims 1 to 5; The method comprises: Inputting the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction; Input each of the first feature matrices into a feature fusion layer in sequence, the feature fusion layer sequentially concatenates each of the first feature matrices according to features to obtain a third feature matrix, and determines a fourth feature matrix corresponding to the third feature matrix based on an attention mechanism; Inputting the fourth feature matrix into the linear layer for full connection processing to obtain a prediction feature matrix; The prediction feature matrix is ​​input into the output layer for format conversion to obtain a prediction result.

7. The method according to claim 6, characterized in that After determining the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism, and before inputting the fourth feature matrix into the linear layer for full connection processing, the method further includes: Based on the splicing strategy when splicing to obtain the third feature matrix, split the fourth feature matrix to obtain a fifth feature matrix corresponding to each feature extraction; Inputting the fifth feature matrix corresponding to each feature extraction into the linear layer for full connection processing, respectively, to obtain the prediction feature matrix corresponding to each feature extraction; The step of inputting the prediction feature matrix into the output layer for format conversion to obtain a prediction result includes: The prediction feature matrix corresponding to each feature extraction is input into the output layer, and the output layer performs fusion processing on the received prediction feature matrix to obtain a fused feature matrix, and performs format conversion to obtain a prediction result.

8. A training device for a feature extraction model, characterized in that: The initial feature extraction model includes a first encoder, a second encoder and a decoder, and the device includes: The mask module is used to perform mask processing on the acquired sample time series data to obtain the masked time series data after masking and the original data of the masked part; A first feature extraction module is used to input the masked time series data into the first encoder, the first encoder performs a pre-configured first number of first feature extractions, and obtains a first feature matrix corresponding to each first feature extraction, wherein the feature extraction strategy of each first feature extraction is different; input the original data into the second encoder, the second encoder performs the first number of second feature extractions, and obtains a second feature matrix corresponding to each second feature extraction, wherein the feature extraction strategy of each second feature extraction is different; A first prediction module, configured to input each of the first feature matrices into the decoder, wherein the decoder predicts a predicted feature matrix corresponding to each first feature extraction of the masked part based on a cross self-attention mechanism; A training module is used to determine a target loss value based on the deviation between the predicted feature matrix corresponding to each first feature extraction and the second feature matrix corresponding to the corresponding second feature extraction, and adjust the parameters of the initial feature extraction model based on the target loss value, and determine the first encoder after the parameter adjustment as the feature extraction model.

9. A time series data prediction device, characterized in that: The prediction model includes a feature extraction layer, a feature fusion layer, a linear layer and an output layer, wherein the feature extraction layer is deployed with a feature extraction model, and the feature extraction model is trained based on the training method of the feature extraction model according to any one of claims 1 to 5; The device comprises: A second feature extraction module is used to input the time series data to be processed into the feature extraction layer to obtain a first feature matrix corresponding to each first feature extraction; The second prediction module is used to input each of the first feature matrices into the feature fusion layer in sequence, the feature fusion layer splices each of the first feature matrices in sequence according to the features to obtain a third feature matrix, and determines the fourth feature matrix corresponding to the third feature matrix based on the attention mechanism; inputs the fourth feature matrix into the linear layer for full connection processing to obtain a predicted feature matrix; inputs the predicted feature matrix into the output layer for format conversion to obtain a prediction result.

10. An electronic device, characterized in that: The electronic device includes a processor, which is used to implement the steps of the feature extraction model training method as described in any one of claims 1-5, or the steps of the time series data prediction method as described in any one of claims 6-7 when executing the computer program stored in the memory.

Images