A pre-training model training method, a time series data identification method and device

Abstract

The present disclosure provides a pre-training model training method and a time series data identification method and device, relates to the technical field of artificial intelligence, in particular to the technical field of data processing and machine learning. The specific implementation scheme is: adding noise to the original time series data to obtain sample time series data. Then the trend item and the seasonal item of the sample time series data are extracted through the pre-training model, wherein the trend item represents the change trend of the sample time series data, and the seasonal item represents the change period of the sample time series data. Then the pre-training model is used to predict the time series data before the noise is added based on the trend item and the seasonal item to obtain predicted time series data. Then the pre-training model is trained based on the original time series data and the predicted time series data. Thus, the accuracy of identifying the time series data is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of data processing and machine learning technology. Background Technology

[0002] Time series data is a common data format closely related to production and daily life. For example, in the integrated energy management industry, energy consumption data at each moment within a certain period is treated as a time series data point. Summary of the Invention

[0003] This disclosure provides a training method for a pre-trained model, a method for identifying time-series data, and an apparatus.

[0004] A first aspect of this disclosure provides a method for training a pre-trained model, comprising:

[0005] Noise is added to the original time series data to obtain sample time series data;

[0006] The trend term and seasonal term of the sample time series data are extracted by a pre-trained model. The trend term represents the changing trend of the sample time series data, and the seasonal term represents the changing period of the sample time series data.

[0007] The pre-trained model predicts the time series data before noise is added, based on the trend term and the seasonal term, to obtain the predicted time series data.

[0008] The pre-trained model is trained based on the original time series data and the predicted time series data.

[0009] A second aspect of this disclosure provides a method for identifying time-series data, including:

[0010] Obtain the time series data to be identified;

[0011] The trend term and seasonal term of the time series data to be identified are extracted by the identification model. The trend term represents the changing trend of the time series data to be identified, and the seasonal term represents the changing period of the time series data to be identified. The identification model is obtained based on a pre-trained model trained by the method described in any one of the first aspects.

[0012] The identification model determines the identification result of the time series data to be identified based on the trend item and the seasonal item.

[0013] A third aspect of this disclosure provides a training apparatus for a pre-trained model, comprising:

[0014] Add a module to add noise to the original time series data to obtain sample time series data;

[0015] The extraction module is used to extract the trend term and seasonal term of the sample time series data through a pre-trained model. The trend term represents the changing trend of the sample time series data, and the seasonal term represents the changing period of the sample time series data.

[0016] The prediction module is used to predict the time series data before noise is added based on the trend term and the seasonal term using the pre-trained model, thereby obtaining the predicted time series data.

[0017] The training module is used to train the pre-trained model based on the original time-series data and the predicted time-series data.

[0018] A fourth aspect of this disclosure provides a time-series data identification device, comprising:

[0019] The acquisition module is used to acquire the time series data to be identified;

[0020] An extraction module is used to extract the trend item and seasonal item of the time series data to be identified through an identification model. The trend item represents the changing trend of the time series data to be identified, and the seasonal item represents the changing period of the time series data to be identified. The identification model is obtained based on a pre-trained model trained by the method described in any one of the first aspects.

[0021] The determination module is used to determine the recognition result of the time series data to be identified based on the trend item and the seasonal item through the recognition model.

[0022] A fifth aspect of this disclosure provides an electronic device, including:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in either the first or second aspect.

[0026] A sixth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any one of the first or second aspects.

[0027] A seventh aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the method according to any one of the first or second aspects.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0029] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0030] Figure 1 This is a flowchart of a training method for a pre-trained model provided in an embodiment of this disclosure;

[0031] Figure 2 This is a flowchart of a method for extracting trend items and seasonal items provided in an embodiment of this disclosure;

[0032] Figure 3 This is an exemplary schematic diagram of two trend seasonal extraction modules connected in series, provided in an embodiment of this disclosure;

[0033] Figure 4 This is an exemplary schematic diagram of the training process of a pre-trained model provided in an embodiment of this disclosure;

[0034] Figure 5 This is a flowchart of a time-series data identification method provided in an embodiment of this disclosure;

[0035] Figure 6 This is a schematic diagram of the structure of a training device for a pre-trained model provided in an embodiment of this disclosure;

[0036] Figure 7 This is a schematic diagram of the structure of a time-series data identification device provided in an embodiment of this disclosure;

[0037] Figure 8 This is a block diagram of an electronic device used to implement the training method of the pre-trained model or the identification method of time series data in the embodiments of this disclosure. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] The time-series data generated during business operation can reflect the operational status of the business. Therefore, it is possible to detect whether the business is abnormal, classify the operational status of the business, and predict future time-series data based on the time-series data already generated by the business.

[0040] The aforementioned tasks of detecting, classifying, and predicting time-series data can all be implemented using models. If models are used, the time-series data needs to be labeled beforehand, and the model needs to be trained using the labeled data. However, because traditional industries are still in the digitalization stage, the foundation of intelligent time-series data is poor, and labeled data is lacking, resulting in low generalization ability and low processing accuracy of the trained models.

[0041] Currently, in the field of Natural Language Processing (NLP), it is possible to train pre-built pre-trained models using unlabeled natural language text, and then perform specific tasks such as text sentiment analysis, text translation, or information retrieval based on the pre-trained models.

[0042] Among them, mainstream NLP pre-trained models are divided into bidirectional language models and unidirectional language models. Bidirectional language models are suitable for natural language understanding; for example, a bidirectional language model is a transducer-based bidirectional encoder representation from transformers (BERT) model. Unidirectional language models are suitable for natural language generation; for example, a unidirectional language model is a pre-trained model (Gererate Pre-Training Model, GPT).

[0043] Because of the differences in data types between natural language text and time-series data, neither of the two types of models in the NLP field is suitable for processing time-series data.

[0044] For example, natural language text and time-series data have different information densities; that is, natural language has richer semantic information, while time-series data has stronger local relationships. Adjacent positions in time-series data are often highly correlated, resulting in significant information redundancy. Applying the approach of pre-trained NLP models to time-series data—that is, using pre-trained NLP models to process noisy time-series data, learn its semantics, and reconstruct noise-free time-series data—is problematic. Because of the strong local correlations and significant information redundancy in time-series data, models can easily reconstruct the original time-series data by simply copying or interpolating it, hindering the model's ability to learn the semantics of time-series data.

[0045] Moreover, natural language text is discrete, while time-series data is continuous.

[0046] Therefore, pre-trained models of natural language cannot learn the semantics of time-series data well and cannot process time-series data accurately.

[0047] In order to leverage the value of unlabeled time-series data, this disclosure presents an embodiment that uses unlabeled time-series data to train a pre-trained model based on the idea of ​​a pre-trained model. This allows the pre-trained model to learn the inherent relationships within the time-series data, so that the recognition model subsequently built based on the pre-trained model can more accurately detect, classify, or predict the time-series data.

[0048] This disclosure provides a method for training a pre-trained model, which can be applied to electronic devices, such as servers, computers, or mobile phones, which are devices with data processing capabilities. Figure 1 As shown, the method includes the following steps:

[0049] S101. Add noise to the original time series data to obtain sample time series data.

[0050] Optionally, the aforementioned electronic device can monitor the operational data generated at each moment during the operation of the designated device, thereby using the operational data acquired over a period of time as raw time-series data. Alternatively, the electronic device can receive data from multiple consecutive moments input by the user as raw time-series data. Alternatively, raw time-series data can be acquired through other methods, which are not specifically limited in this embodiment.

[0051] Understandably, the sample time series data obtained by adding noise to the original time series data will differ from the original time series data. For example, some data in the original time series data can be masked (masked), or some data in the original time series data can be replaced, thereby adding noise to the original time series data.

[0052] The intensity of noise can also be selected according to needs. For example, when the accuracy requirement of the pre-trained model is high, a higher intensity can be used, so that most of the data in the original time series data is noise-added. This reduces the need for the pre-trained model to use noise-free data near the noisy data to predict the data before noise was added. In other words, it reduces the need for the pre-trained model to predict the data before noise was added through simple interpolation, thereby improving the ability of the pre-trained model to recognize the semantics of time series data.

[0053] S102. Extract the trend and seasonal terms of the sample time series data through a pre-trained model.

[0054] The trend term represents the changing trend of the sample time series data, such as an upward trend or a downward trend.

[0055] The seasonality term indicates the period of change in the sample time series data, such as a period of one day or a period of one week.

[0056] S103. Using a pre-trained model based on trend and seasonal terms, predict the time series data before adding noise to obtain the predicted time series data.

[0057] Understandably, time series data in practical applications generally consists of trend terms, seasonal terms, and noise. Pre-trained models can extract trend terms and seasonal terms from sample time series data, and can predict time series data before adding noise based on trend terms and seasonal terms. Therefore, pre-trained models are equivalent to denoising input time series data, so they can also be called denoising models.

[0058] S104. Based on the original time series data and the predicted time series data, train the pre-trained model.

[0059] The smaller the error between the original time series data and the predicted time series data, the higher the prediction accuracy of the pre-trained model; conversely, the larger the error between the original time series data and the predicted time series data, the lower the prediction accuracy of the pre-trained model. This rule can be used to train the pre-trained model based on this error.

[0060] For example, the network parameters of the pre-trained model can be adjusted based on this error until the error meets a preset condition, at which point training can be stopped.

[0061] In this embodiment, the pre-trained model can extract trend and seasonal terms from noisy time-series data to predict the original time-series data before adding noise. Based on the predicted time-series data and the original time-series data, the pre-trained model is trained. That is, this embodiment can train a pre-trained model based on unlabeled time-series data, avoiding the problem of limited data volume in labeled time-series data, thus improving the generalization ability and accuracy of the pre-trained model.

[0062] Furthermore, in practical applications, time-series data typically consists of trend terms, seasonal terms, and noise. Because the embodiments of this disclosure enable the pre-trained model to extract trend and seasonal terms more accurately during model training, it improves the pre-trained model's ability to learn the semantics of time-series data and increases its extraction accuracy. This results in higher recognition accuracy for the subsequent recognition model built based on the pre-trained model.

[0063] The training method for the pre-trained model provided in this disclosure is described in detail below:

[0064] The method of adding noise to the original time series data in S101 above can be implemented as follows: mask a portion of the original time series data to obtain sample time series data.

[0065] For example, the original time series data is: [1,2,3,4,5,6,7], a total of 7 data points, each corresponding to a different time point, and the time points corresponding to each data point are consecutive. After masking a portion of the original time series data, the sample time series data is obtained as: [1,0,3,0,0,0,7], where the masked positions are represented by "0".

[0066] By masking a portion of the sample time series data, the sample time series data contains both noise and retains some of the original data, enabling the pre-trained model to construct complete time series data from the partial original data during the training process.

[0067] In some embodiments of this disclosure, masking a portion of the original time-series data can be implemented as follows: for each segment of data included in the original time-series data, determine whether the segment needs to be masked based on a preset mask rate. If yes, then the segment is masked; otherwise, the segment is not masked, i.e., the segment remains unchanged. Each segment contains the same amount of data.

[0068] Understandably, the original time series data has temporal continuity, therefore each segment of the original time series data also has temporal continuity. Furthermore, the segments of the original time series data do not overlap; the union of these segments constitutes the original time series data. For example, in the original time series data, every two data points can be grouped into a segment, or every three data points can be grouped into a segment.

[0069] The masking rate represents the probability of masking each data segment, and can be set according to actual needs. For example, when high prediction accuracy is required for the pre-trained model, or when the time-series data to be identified in practical applications contains a lot of noise, a higher masking rate can be set. For example, setting the masking rate to 85% means that each data segment has an 85% probability of being masked.

[0070] For example, the original time series data is: [1,2,3,4,5,6,7,8,9,10]. Each pair of data is a segment. After masking the segment of data according to the preset mask rate, the sample time series data can be: [0,0,3,4,0,0,7,8,0,0].

[0071] Optionally, each data segment can be determined sequentially according to its arrangement; or, to improve efficiency, each data segment can be determined separately in parallel.

[0072] Using the above method, the embodiments of this disclosure can determine whether each segment of data in the original time series data needs to be masked, thus preserving some continuous data while adding noise to the original time series data. This allows subsequent predictions to be made based on the preserved continuous data, enabling the pre-trained model to learn the temporal sequence between the data in the time series data.

[0073] In this embodiment of the disclosure, before extracting the trend and seasonal terms of the sample time series data through the pre-trained model in S102, the electronic device may further: perform a linear transformation on the sample time series data through the pre-trained model to obtain transformed data.

[0074] Accordingly, the above method of extracting trend and seasonal terms in S102 can be implemented as follows: extracting trend and seasonal terms of transformed data through a pre-trained model.

[0075] The sample time series data includes data from multiple time points and is one-dimensional data. It can be converted into two-dimensional data through a linear transformation module.

[0076] The pre-trained model may include a linear transformation module, which performs a linear transformation on the sample time-series data to obtain transformed data. The linear transformation module can perform a linear transformation on each data point in the sample time-series data, thus obtaining multiple sub-data points. In other words, using linear transformation, a single data point can be represented by multiple sub-data points. For example, if the linear transformation process represents each data point using four sub-data points, assuming the sample time-series data size is N×1, the data size after linear transformation will be N×4, where N is the total number of data points in the sample time-series data.

[0077] Using multiple sub-data to represent a single data point enables multi-dimensional representation of the data, facilitating pre-trained models to learn the intrinsic relationships between data points in the time series data and thus making more accurate predictions.

[0078] In this embodiment of the disclosure, the pre-trained model can extract trend and seasonal terms once for sample time series data, or it can extract trend and seasonal terms multiple times consecutively for sample time series data.

[0079] When the pre-trained model extracts trend and seasonal terms multiple times consecutively from the sample time-series data, see [reference needed]. Figure 2 The method for extracting the trend and seasonal terms of sample time-series data through a pre-trained model in S102 above includes the following steps:

[0080] S201. Perform causal convolution on the sample time series data using a pre-trained model, and obtain the trend term based on the causal convolution result.

[0081] In this embodiment of the disclosure, since the pre-trained model can perform a linear transformation on the sample time series data before extracting the trend term and seasonal term, the pre-trained model can perform causal convolution on the transformed data obtained after linear transformation of the sample time series data, and then obtain the trend term based on the causal convolution result.

[0082] S202. Determine the first difference between the sample time series data and the obtained trend term through the pre-trained model, and perform a fast Fourier transform on the first difference to obtain frequency domain data. Perform a linear transformation on the obtained frequency domain data, and perform an inverse fast Fourier transform on the linear transformation result to obtain the seasonal term.

[0083] Because the pre-trained model can perform a linear transformation on the sample time-series data before extracting the trend and seasonal terms, resulting in transformed data, the first difference is the difference between the transformed data and the extracted trend term. Since the transformed data has the same data size as the trend term, they can be subtracted digit by digit to obtain the first difference between them.

[0084] In this embodiment, it is assumed that the data size of the first difference is N×D, where N is the total amount of data included in the sample time series data, and D is the number of sub-data points corresponding to each data point in the sample time series data after linear transformation. After the first difference undergoes a Fast Fourier Transform, the resulting frequency domain data size is (T / 2+1)×D, which is equivalent to having (T / 2+1) frequencies. A linear transformation is then performed on the data of each frequency to obtain the linear transformation result. Subsequently, an inverse Fast Fourier Transform is performed on the linear transformation result to convert the frequency domain data to the real number domain, thus obtaining the seasonal term.

[0085] S203. Determine the second difference between the first difference and the obtained seasonal item through the pre-trained model, and perform causal convolution on the second difference. Return to the step of obtaining the trend item based on the causal convolution result, until a preset number of trend items and seasonal items are extracted.

[0086] The pre-trained model may include multiple trend and seasonality extraction modules connected in series, each of which includes a trend extraction module and a seasonality extraction module connected in series. The trend and seasonality extraction modules can extract trend items from the input data, and the seasonality extraction modules can extract seasonality items from the input data. In this embodiment, each trend extraction module extracts trend items in the same way, and each seasonality extraction module extracts seasonality items in the same way.

[0087] The trend extraction module can be called the Trend Feature Disentangler (TFD), and the seasonal extraction module can be called the Seasonal Feature Disentangler (SFD).

[0088] For example, Figure 3 This consists of two cascaded trend and seasonality extraction modules. The first module includes both a trend extraction module 1 and a seasonality extraction module 1, while the second module includes both a trend extraction module 2 and a seasonality extraction module 2. Trend extraction module 1 extracts trend term 1 from the transformed data obtained after linear transformation of the sample time-series data. The first difference between the transformed data and trend term 1 is input into seasonality extraction module 1. Seasonality extraction module 1 extracts seasonality term 1 from the first difference and inputs the second difference between the first difference and seasonality term 1 into trend extraction module 2. Trend extraction module 2 extracts trend term 2 from the second difference and inputs the third difference between the second difference and trend term 2 into seasonality extraction module 2. Seasonality extraction module 2 extracts seasonality term 2 from the third difference. If other trend and seasonality extraction modules are connected after the second module, the fourth difference between the third difference and seasonality term 2 can be input into the next module, and so on.

[0089] Using the above method, this embodiment of the present disclosure can extract trend terms from sample time-series data using a pre-trained model. After stripping the trend terms from the sample time-series data, the seasonal extraction module is used to extract seasonal terms. Then, the seasonal terms are stripped again and input into the next trend extraction module, and so on, until a preset number of trend and seasonal terms are extracted. This achieves multiple extractions of trend and seasonal terms from the time-series data, improving the comprehensiveness of trend and seasonal term extraction. Furthermore, stripping the extracted trend and seasonal terms from the data before extraction reduces the impact of the first extracted trend and seasonal terms on subsequent extraction processes, improving the accuracy of trend and seasonal term extraction.

[0090] In this embodiment of the disclosure, the method of performing causal convolution on sample time-series data using a pre-trained model and obtaining a trend term based on the causal convolution result in S201 may include the following steps:

[0091] Step 1: Apply various causal convolutions to the sample time-series data using a pre-trained model. Each causal convolution has a different kernel size.

[0092] As can be seen from the above scheme, trend items can be extracted through the trend extraction module in the trend seasonal extraction module included in the pre-trained model. Therefore, each trend extraction module includes multiple causal convolutional layers, and the kernel size of each causal convolutional layer included in the same trend extraction module is different. The kernel size of each causal convolutional layer can be preset.

[0093] Furthermore, since the sample time series data can be linearly transformed to obtain transformed data before extracting the trend and seasonal terms, each causal convolutional layer of the first trend extraction module can specifically perform causal convolution on the transformed data.

[0094] Step 2: Determine the weight of each causal convolution result through the pre-trained model, and determine the weighted sum of each causal convolution result. Use the weighted sum as the extracted trend term.

[0095] Each trend extraction module includes multiple causal convolutional layers, and each trend extraction module also includes an integration layer. The integration layer can process each causal convolution result with a normalized exponential function (softmax) to obtain a value in the range [0,1], which is then used as the weight of the causal convolution result. Finally, the weighted sum of all causal convolution results is calculated as the trend term.

[0096] It should be noted that steps one and two illustrate the process of the first trend extraction module extracting trend items in the pre-trained model. The other trend extraction modules extract trend items in the same way as the first trend extraction module, and will not be described again here.

[0097] Since different business operations generate time-series data, the appropriate kernel size for causal convolution may vary. By using multiple causal convolutions with different kernel sizes to perform causal convolution on the sample time-series data and then weighting the results of each causal convolution, the problem of mismatch between a single causal convolution layer and the sample time-series data can be reduced, thereby improving the accuracy of trend extraction.

[0098] When the pre-trained model extracts the trend and seasonal terms from the sample time-series data, the method of extracting the trend and seasonal terms from the sample time-series data using the pre-trained model in S102 above includes the following steps:

[0099] Step 1: Perform causal convolution on the sample time series data using a pre-trained model, and obtain the trend term based on the causal convolution result.

[0100] The specific implementation method of step 1 is the same as that of S201 above, and can be referred to the above description, which will not be repeated here.

[0101] Step 2: Determine the first difference between the sample time series data and the obtained trend term through the pre-trained model, and perform a fast Fourier transform on the first difference to obtain frequency domain data. Perform a linear transformation on the obtained frequency domain data, and perform an inverse fast Fourier transform on the linear transformation result to obtain the seasonal term.

[0102] The specific implementation method of step 2 is the same as that of S202 above, and can be referred to the above description, which will not be repeated here.

[0103] Using the above method, embodiments of this disclosure can extract trend terms from sample time-series data using a pre-trained model, and after stripping the trend terms from the sample time-series data, continue to extract seasonal terms. This achieves the extraction of trend terms and seasonal terms from time-series data using a pre-trained model. Furthermore, stripping the trend terms from the sample time-series data after extraction reduces the impact on subsequent extraction of seasonal terms, improving the accuracy of trend and seasonal term extraction.

[0104] In this embodiment of the disclosure, the method of predicting the time series data before adding noise by using a pre-trained model based on trend terms and seasonal terms in S103 can be implemented as follows: the sum of trend terms and seasonal terms extracted from the sample time series data is determined by the pre-trained model, that is, the sum of each trend term and each seasonal term extracted by the pre-trained model is calculated, and then a linear transformation is performed on the sum, and the result of the linear transformation is used as the predicted time series data.

[0105] Since the data dimensions of each trend term and each seasonal term are the same, the trend term and each seasonal term can be added digit by digit to obtain the sum. Meanwhile, because the sample time series data underwent a linear transformation before the trend term and seasonal term extraction, the transformed data has a different dimension than the sample time series data. Therefore, a linear transformation can be performed on this sum. This linear transformation process is the reverse of the linear transformation process before the trend term and seasonal term extraction, thus obtaining predicted time series data with the same dimension as the sample time series data.

[0106] By summing the extracted trend and seasonal terms using the above method, noise in the sample time series data is effectively removed. Then, a linear transformation is performed on the sum to make the size of the predicted time series data the same as the size of the original time series data, thereby improving the accuracy of the reconstructed predicted time series data.

[0107] The method for training the pre-trained model in S104 above can be implemented as follows: Calculate the loss value between the original time-series data and the predicted time-series data using the mean squared error or other preset loss function. Determine whether the pre-trained model has converged. If it has not converged, use the loss value to adjust the network parameters of each module included in the pre-trained model, and return to the step of adding noise to the original time-series data to proceed with the next round of training.

[0108] If convergence is achieved, the pre-trained model is considered to have completed training.

[0109] Optionally, the pre-trained model can be determined to have converged if the loss value calculated in the current training round is less than a preset threshold; otherwise, it can be determined that the pre-trained model has not converged. Alternatively, the pre-trained model can be determined to have converged if the difference between the loss values ​​calculated in the current round and the previous round is less than a preset threshold; otherwise, it can be determined that the pre-trained model has not converged. Alternatively, the pre-trained model can be determined to have converged if the error between the loss values ​​calculated in the most recent preset number of training rounds is less than a preset threshold; otherwise, it can be determined that the pre-trained model has not converged. Alternatively, other methods can be used to determine whether the pre-trained model has converged, and this embodiment of the disclosure does not specifically limit this method.

[0110] Since a smaller error between the predicted time-series data output by the pre-trained model and the original time-series data indicates a stronger semantic recognition ability of the pre-trained model, it also indicates a stronger ability to reconstruct the time-series data, i.e., better noise reduction. Therefore, in this embodiment of the present disclosure, when training the pre-trained model, the network parameters are adjusted based on the loss value between the original time-series data and the predicted time-series data, so that the output result of the pre-trained model becomes closer and closer to the original time-series data during the training process.

[0111] As can be seen, the embodiments of this disclosure fully utilize the value of a large amount of unlabeled time series data, enabling the pre-trained model to learn the inherent patterns of the unlabeled time series data, thereby improving the accuracy of various downstream time series tasks based on the pre-trained model.

[0112] Traditional end-to-end models for processing time-series data, such as anomaly detection, classification, and prediction models, all require training on labeled time-series data. However, the amount of labeled time-series data is limited, resulting in poor generalization ability for models trained on only a small amount of labeled data. Furthermore, unpredictable noise is common in time-series data, and these end-to-end models are easily affected by such noise, leading to poor generalization and low accuracy in actual task processing.

[0113] The pre-trained model in this embodiment is trained using unlabeled time-series data, avoiding the problem of insufficient labeled time-series data. Furthermore, the pre-trained model can learn the semantics of time-series data from a large amount of unlabeled data and, by extracting trend and seasonal terms, mitigate the impact of noise in the time-series data to some extent, thus improving the model's generalization ability. This allows the pre-trained model's ability to extract trend and seasonal terms—the trend and seasonal extraction module—to be combined with a simple task module to obtain the recognition model. Because the pre-trained model learns the semantics of time-series data during training, the trend and seasonal extraction module achieves high accuracy in extracting trend and seasonal terms, thereby improving the accuracy of the subsequently constructed recognition model in tasks such as anomaly detection or classification of time-series data.

[0114] In addition, after obtaining the recognition model, a small amount of labeled time-series data from the business can be used to fine-tune the recognition model, thereby further improving the recognition accuracy of the recognition model in specific business applications.

[0115] See Figure 4 The following describes the prediction process of the pre-trained model in the training method of the pre-trained model provided in this embodiment, in conjunction with practical application scenarios:

[0116] Figure 4 S0~S n Each solid line corresponds to a segment of data, representing multiple consecutive time points. The data corresponding to S1, S3, and S4 are masked, i.e., set to 0. The masked time-series data is then input into the linear transformation module of the pre-trained model. This module performs a linear projection of flattened patches on the time-series data to obtain the transformed data. Figure 3 Transformed data using t0, t * , t2…t n This indicates that t corresponds one-to-one with S, for example, t0 is obtained after performing a linear transformation on S0, and t * This is obtained by performing a linear transformation on the masked S1, and so on.

[0117] Next, the transformed data is input into N concatenated trend and seasonal extraction modules. Figure 3 The right side of the image shows a magnified view of a single trend and seasonality extraction module. For the first trend and seasonality extraction module, causal convolution is performed on the transformed data through each causal convolution layer included in the trend extraction module to obtain the causal convolution result. Each causal convolution result is denoted as F(x), where x represents the sequence number of the causal convolution result. Figure 3 The diagram shows three causal convolutional layers, with kernel sizes of 2... 0 2 1 and 2 L L is a preset value. Then, through the integration layer included in the trend extraction module, the weight of each causal convolution result is determined using the Softmax function and denoted as L(x). The weighted sum of each causal convolution result is determined and used as the trend term extracted by the trend extraction module. The first difference between the transformed data and the obtained trend term is used as the input data of the seasonal extraction module in the first trend seasonal extraction module and input into the seasonal extraction module.

[0118] The seasonal extraction module performs a Fast Fourier Transform (FFT) on the first difference to obtain frequency domain data. A linear transformation is then performed on the obtained frequency domain data, and an Inverse Fast Fourier Transform (iFFT) is performed on the linear transformation result to obtain the seasonal term. The second difference between the first difference and the obtained seasonal term is used as the input data for the trend extraction module in the next trend seasonal extraction module, until the last trend seasonal extraction module extracts the trend term and seasonal term.

[0119] The trend and seasonal terms extracted by the N trend and seasonal extraction modules are respectively input into the prediction head module of the pre-trained network. The prediction head module determines the sum of the trend and seasonal terms extracted by each trend and seasonal extraction module, and performs a linear transformation on the sum. The result of the linear transformation is used as the predicted time series data. Figure 4 The solid curve at the top represents the predicted time series data.

[0120] Based on the same inventive concept, this disclosure also provides a method for identifying time-series data. This method can be applied to electronic devices, such as mobile phones, desktop computers, or servers, which are devices with data processing capabilities. The electronic device used for the time-series data identification method can be the same as or different from the electronic device used for the training method of the aforementioned pre-trained model. Figure 5 As shown, the method for identifying time-series data includes the following steps:

[0121] S501. Obtain the time series data to be identified.

[0122] For details on the specific implementation of S501, please refer to the relevant description of S101 above, which will not be repeated here.

[0123] S502. Extract the trend and seasonal items of the time series data to be identified through the identification model.

[0124] Among them, the trend term represents the changing trend of the time series data to be identified, and the seasonal term represents the changing cycle of the time series data to be identified.

[0125] The recognition model is obtained based on a pre-trained model trained using the methods described above. This recognition model may include a trend and seasonality extraction module, which is derived from the trend and seasonality extraction module in the pre-trained model. For example, the trend and seasonality extraction module in the pre-trained model can be directly used as the trend extraction module in the recognition model. Alternatively, the trend and seasonality extraction module in the pre-trained model can be connected to the recognition module as the initial recognition model, and then fine-tuned using labeled time-series data to obtain the final recognition model.

[0126] S503. Based on the trend and seasonality factors, determine the identification results of the time series data to be identified by the identification model.

[0127] In this embodiment of the disclosure, the identification model may include an identification module for determining the identification result of the time series data to be identified based on trend items and seasonal items.

[0128] The identification results for the time-series data to be identified can be: classification results, detection results, or prediction results, etc. For example, if the time-series data to be identified is operational data generated by the operation of a device, then the classification result can be the category to which the device's operating status belongs, the detection result can be whether the device has any abnormalities, or the prediction result can be the operational data at various times within a future period.

[0129] Optionally, the specific processing procedure of the identification module can be determined according to the type of identification result required, and this embodiment does not specifically limit this. For example, when the required identification result is a detection result, the processing procedure of the identification module can be to perform softmax on the sum of the trend items and seasonal items extracted from the time series data to be identified, and use the softmax result as the detection result.

[0130] In this embodiment, since the pre-trained network is trained on unlabeled time-series data, it avoids the problem of insufficient labeled time-series data, thereby improving the generalization ability and prediction accuracy of the pre-trained model. This enhances the pre-trained model's ability to learn the semantics of time-series data and improves the accuracy of its trend and seasonal data extraction. Consequently, the recognition model built based on the pre-trained model can more accurately extract the trend and seasonal data of the time-series data to be recognized, thus improving the accuracy of the recognition results for the time-series data.

[0131] In this embodiment, the process of the identification model extracting trend and seasonal terms, i.e., the processing procedure and specific structure of the trend and seasonal extraction module included in the identification model, can be referred to the relevant description in the above-mentioned pre-trained model training method, and will not be repeated here. Furthermore, the identification model can also perform a linear transformation on the time-series data to be identified before extracting trend and seasonal terms; that is, the identification model can also include a transformation module. The processing procedure and specific structure of the transformation module can be referred to the relevant description in the above-mentioned pre-trained model training method, and will not be repeated here.

[0132] The collection, storage, use, processing, transmission, provision, and disclosure of time-series data involved in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0133] Based on the same inventive concept, and corresponding to the above method embodiments, this disclosure provides a training apparatus for a pre-trained model, such as... Figure 6 As shown, the device includes: an addition module 601, an extraction module 602, a prediction module 603, and a training module 604;

[0134] Add module 601 to add noise to the original time series data to obtain sample time series data;

[0135] The extraction module 602 is used to extract the trend and seasonal terms of the sample time series data through a pre-trained model. The trend term represents the changing trend of the sample time series data, and the seasonal term represents the changing period of the sample time series data.

[0136] Prediction module 603 is used to predict time series data before noise is added based on trend and seasonal terms using a pre-trained model, thus obtaining predicted time series data.

[0137] Training module 604 is used to train a pre-trained model based on the original time series data and the predicted time series data.

[0138] In some embodiments of this disclosure, the addition module 601 is specifically used for:

[0139] A portion of the original time series data is masked to obtain sample time series data.

[0140] In some embodiments of this disclosure, the added module is specifically used for:

[0141] For each segment of the original time-series data, determine whether to mask that segment based on the preset mask rate; each segment contains the same amount of data.

[0142] If so, then mask that data segment.

[0143] In some embodiments of this disclosure, the extraction module 602 is specifically used for:

[0144] Causal convolution is performed on the sample time series data using a pre-trained model, and the trend term is obtained based on the causal convolution result;

[0145] The first difference between the sample time series data and the obtained trend term is determined by the pre-trained model, and the first difference is subjected to fast Fourier transform to obtain frequency domain data. The obtained frequency domain data is subjected to linear transformation, and the result of the linear transformation is subjected to inverse fast Fourier transform to obtain the seasonal term.

[0146] The second difference between the first difference and the obtained seasonal item is determined by the pre-trained model, and causal convolution is performed on the second difference. The step of obtaining the trend item based on the causal convolution result is returned until a preset number of trend items and seasonal items are extracted.

[0147] In some embodiments of this disclosure, the extraction module 602 is specifically used for:

[0148] Various causal convolutions were performed on the sample time-series data using a pre-trained model; the kernel size for each causal convolution was different.

[0149] The weights of each causal convolution result are determined by a pre-trained model, and the weighted sum of each causal convolution result is determined. The weighted sum is then used as the extracted trend term.

[0150] In some embodiments of this disclosure, the extraction module 602 is specifically used for:

[0151] Causal convolution is performed on the sample time series data using a pre-trained model, and the trend term is obtained based on the causal convolution result;

[0152] The first difference between the sample time series data and the obtained trend term is determined by the pre-trained model, and the first difference is subjected to fast Fourier transform to obtain frequency domain data. The obtained frequency domain data is then subjected to linear transformation, and the result of the linear transformation is subjected to inverse fast Fourier transform to obtain the seasonal term.

[0153] In some embodiments of this disclosure, the prediction module 603 is specifically used for:

[0154] The sum of trend and seasonal terms extracted from the sample time series data is determined by a pre-trained model;

[0155] The sum is linearly transformed using a pre-trained model, and the result of the linear transformation is used as the predicted time series data.

[0156] Based on the same inventive concept, and corresponding to the above method embodiments, this disclosure provides a time-series data identification device, such as... Figure 7 As shown, the device includes: an acquisition module 701, an extraction module 702, and a determination module 703;

[0157] The acquisition module 701 is used to acquire the time series data to be identified;

[0158] The extraction module 702 is used to extract the trend and seasonal terms of the time series data to be identified through the recognition model. The trend term represents the changing trend of the time series data to be identified, and the seasonal term represents the changing period of the time series data to be identified. The recognition model is obtained based on the pre-trained model trained by the above-mentioned pre-trained model training method.

[0159] The determination module 703 is used to determine the recognition result of the time series data to be identified based on the trend item and the seasonal item through the recognition model.

[0160] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0161] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0162] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0163] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0164] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as methods for training pre-trained models and methods for identifying time-series data. For example, in some embodiments, the methods for training pre-trained models and methods for identifying time-series data can be implemented as computer software programs, which are tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the methods for training pre-trained models and methods for identifying time-series data described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured in any other suitable manner (e.g., by means of firmware) to perform a training method for a pre-trained model or a method for identifying time-series data.

[0165] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0169] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0170] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0171] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for identifying time-series data, applied to electronic devices, comprising: Acquire the time-series data to be identified, which includes: the operational data generated by the specified device at each moment within a certain period of time; The trend and seasonal terms of the time series data to be identified are extracted by the identification model. The trend term represents the changing trend of the time series data to be identified, and the seasonal term represents the changing period of the time series data to be identified. The identification model determines the identification result of the time series data to be identified based on the trend item and the seasonal item. The identification result includes: classification result, detection result or prediction result. The classification result includes the category to which the device's operating status belongs. The detection result includes whether the device has any abnormalities. The prediction result includes the device's operating data at each time point in the future. The recognition model is obtained based on the following steps: In parallel, for each segment of the original time-series data, it is determined whether the segment needs to be masked according to the preset mask rate. If so, the segment is masked to obtain sample time-series data. Each segment contains the same amount of data. The original time-series data includes: the running data generated by the specified device at each moment within a certain period of time. The sample time series data is subjected to various causal convolutions through a pre-trained model, and a trend term is obtained based on the causal convolution results. The kernel size of each causal convolution is different. The first difference between the sample time series data and the obtained trend term is determined by the pre-trained model, and the first difference is subjected to fast Fourier transform to obtain frequency domain data. The obtained frequency domain data is subjected to linear transformation, and the result of linear transformation is subjected to inverse fast Fourier transform to obtain the seasonal term. The second difference between the first difference and the obtained seasonal item is determined by the pre-trained model, and causal convolution is performed on the second difference. The step of obtaining the trend item based on the causal convolution result is returned until a preset number of trend items and seasonal items are extracted. The pre-trained model predicts the time series data before noise is added, based on the trend term and the seasonal term, to obtain the predicted time series data. Based on the original time-series data and the predicted time-series data, the pre-trained model is trained, and the pre-trained model is used to determine the recognition result of the time-series data.

2. The method according to claim 1, wherein, The trend term obtained based on the causal convolution result includes: The weights of each causal convolution result are determined by the pre-trained model, and the weighted sum of each causal convolution result is determined, which is then used as the extracted trend term.

3. The method according to claim 1 or 2, wherein, The step of using the pre-trained model to predict time-series data before adding noise, based on the trend term and the seasonal term, to obtain predicted time-series data includes: The sum of trend and seasonal terms extracted from the sample time series data is determined by the pre-trained model. The sum is linearly transformed using the pre-trained model, and the result of the linear transformation is used as the predicted time series data.

4. A time-series data identification device, applied to electronic equipment, comprising: The acquisition module is used to acquire the time-series data to be identified, which includes: the operating data generated by the specified device at each moment within a certain period of time; The extraction module is used to extract the trend item and seasonal item of the time series data to be identified through the recognition model. The trend item represents the changing trend of the time series data to be identified, and the seasonal item represents the changing period of the time series data to be identified. The determination module is used to determine the recognition result of the time series data to be identified based on the trend item and the seasonal item using the recognition model; An addition module is used to determine, in parallel, whether to mask each segment of data included in the original time series data according to a preset mask rate. If so, the segment of data is masked to obtain sample time series data. Each segment of data includes the same amount of data. The original time series data includes: the running data generated by the specified device at each moment within a certain period of time. The extraction module is further configured to perform various causal convolutions on the sample time-series data using a pre-trained model, and obtain trend terms based on the causal convolution results, wherein the kernel size of each causal convolution is different; determine a first difference between the sample time-series data and the obtained trend terms using the pre-trained model, perform a fast Fourier transform on the first difference to obtain frequency domain data, perform a linear transform on the obtained frequency domain data, and perform an inverse fast Fourier transform on the linear transform result to obtain seasonal terms; determine a second difference between the first difference and the obtained seasonal terms using the pre-trained model, perform a causal convolution on the second difference, and return to the step of obtaining trend terms based on the causal convolution results, until a preset number of trend terms and seasonal terms are extracted; The prediction module is used to predict the time series data before noise is added based on the trend term and the seasonal term using the pre-trained model, thereby obtaining the predicted time series data. The training module is used to train the pre-trained model based on the original time-series data and the predicted time-series data. The pre-trained model is used to determine the recognition result of the time-series data. The recognition result includes: classification result, detection result, or prediction result. The classification result includes the category to which the device's operating status belongs. The detection result includes whether the device has any abnormalities. The prediction result includes the device's operating data at various times in the future.

5. The apparatus according to claim 4, wherein, The extraction module is specifically used for: The weights of each causal convolution result are determined by the pre-trained model, and the weighted sum of each causal convolution result is determined, which is then used as the extracted trend term.

6. The apparatus according to claim 4 or 5, wherein, The prediction module is specifically used for: The sum of trend and seasonal terms extracted from the sample time series data is determined by the pre-trained model. The sum is linearly transformed using the pre-trained model, and the result of the linear transformation is used as the predicted time series data.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images