Time sequence large model construction method based on frequency domain representation learning and storage medium

Abstract

The invention relates to the technical field of large model training, and discloses a time sequence large model construction method based on frequency domain representation learning and a storage medium, and the method comprises the steps: converting original time sequence data into frequency domain representation, extracting the amplitude and phase of each frequency component from the frequency domain representation, and enabling each group of amplitude and phase to correspond to a primary function; performing feature extraction and characterization learning on the primary function by using a large model, and introducing an attention mechanism, so that the large model can adaptively focus a frequency domain component which is most critical to a task; optimizing by combining the reconstruction head and the prediction head, and optimizing the amplitude and phase deviation of the true value and the model output value in the frequency domain according to a preset pure frequency domain loss function; the frequency domain features of feature extraction construct the unified frequency domain representation of the time sequence by learning the features and the mutual relation of the primary function, and the pure frequency domain loss function enhances the learning effect on the frequency domain representation, and can flexibly adapt to different downstream tasks.

Description

Technical Field

[0001] The present application relates to the technical field of large model training, and in particular to a method for constructing a large time series model based on frequency domain representation learning. Background Art

[0002] With the emergence of large models, more and more work in the field of time series is focusing on learning common patterns and rules in time series by pre-training with large-scale time series data from different fields. The core goal of this method is that by pre-training on a wide range of time series data, the model can capture various potential time series features, such as periodicity, trend, seasonal changes, etc., and thus have stronger generalization capabilities. This enables the model to quickly adapt to a variety of different time series tasks and improve the effectiveness and efficiency of cross-domain applications.

[0003] Most existing deep models are only trained for specific downstream tasks, and pre-trained time series models need to adapt to different downstream tasks. In addition, time series data is diverse and complex. Time series data from different fields and sources vary in frequency, periodicity, trend, and noise characteristics. Therefore, it is extremely difficult to develop a universal time series pre-training model.

[0004] Time series data usually experiences distribution shift over time. In the modeling process of time series data, data is usually represented in the time domain in the form of scalars. This scalar representation often carries limited information, making it difficult for the model to capture changes in data distribution and dynamic characteristics.

[0005] Current pre-training methods based on frequency domain analysis mainly focus on the spatial characteristics of the data, ignoring the dynamic changes of the data over time and failing to effectively capture temporal dependencies.

[0006] Pre-training large-scale time series models requires processing massive amounts of time series data, which not only requires huge computing resources and storage space, but also faces high time costs during the training process. Due to the above challenges, pre-training large-scale time series models faces many difficulties such as data diversity, distribution shift, computing resource requirements, and training time costs. Summary of the invention

[0007] In order to improve the training efficiency of large time series models and reduce training costs, the present application provides a method and storage medium for constructing a large time series model based on frequency domain representation learning.

[0008] In the first aspect, the present application provides a method for constructing a large time series model based on frequency domain representation learning, which adopts the following technical solutions:

[0009] A method for constructing a large time series model based on frequency domain representation learning, comprising:

[0010] The original time series data is converted into a frequency domain representation, and the amplitude and phase of each frequency component are extracted from the frequency domain representation; wherein each set of amplitude and phase corresponds to a basis function;

[0011] Use a large model to extract features and learn representations of each basis function, and introduce an attention mechanism so that the large model can adaptively focus on the frequency domain components that are most critical to the task;

[0012] The large model jointly optimizes the reconstruction head and the prediction head, and optimizes the amplitude and phase deviation between the true value and the reconstructed value in the frequency domain according to the preset pure frequency domain loss function.

[0013] By adopting the above technical scheme, in order to achieve the unification of time series pre-training and downstream tasks, the present invention proposes a large time series model framework for frequency domain representation learning. The framework represents the frequency domain features in the form of basis functions, and constructs a unified frequency domain representation of the time series by learning the characteristics of the basis functions themselves and their mutual relationships. At the same time, a pure frequency domain loss function is proposed to enhance the learning effect of the frequency domain representation. By designing a joint optimization strategy of reconstruction and prediction, the framework can flexibly adapt to different downstream tasks without changing the pre-training network structure.

[0014] Optionally, the pure frequency domain loss function includes:

[0015]

[0016]

[0017]

[0018] in, and represent the amplitude and phase of the true value respectively, and denote the amplitude and phase of the reconstructed value, respectively. is the reconstruction loss, To predict the loss, are the hyperparameters of the corresponding items, .

[0019] By adopting the above technical solution, the reconstruction loss measures the model's ability to recover the original signal from the frequency domain representation. The phase is periodic. In the large time series model based on frequency domain representation learning, the design of the total loss function needs to take into account both reconstruction accuracy and prediction performance, while fully exploring the physical meaning of frequency domain features (amplitude and phase).

[0020] Optionally, the pure frequency domain loss function includes:

[0021]

[0022]

[0023] in, and represent the amplitude and phase of the true value respectively, and denote the amplitude and phase of the reconstructed value, respectively. is the reconstruction loss, To predict the loss, is the prediction loss at time t, is the regularization term, are the hyperparameters of the corresponding items, , is the phase smoothing regularization weight; is the balance coefficient between task difficulty and uncertainty, which is set according to scenario experience. , is the covariance matrix trace of the prediction task output, reflecting uncertainty, It is the scene-related covariance threshold, which needs to be determined through historical data statistics.

[0024] By adopting the above technical solution, in the initial training stage, you can configure , gradually increase the prediction weight to the target value to avoid early overfitting. If the prediction task loss variance is large, automatically reduce its weight to balance the noise impact. The regularization term includes a phase smoothing term (suppressing mutations) and a spectral sparsity term (increasing the weight of the main frequency component). By increasing the phase smoothing regularization weight, the cumulative error of multi-step prediction can be reduced.

[0025] Optionally, the reconstruction head calculates the deviation between the amplitude and phase of the reconstructed data and the amplitude and phase of the input data to evaluate the reconstruction effect of the model;

[0026] The prediction head calculates the deviation between the amplitude and phase of the predicted value and the amplitude and phase of the true value to evaluate the prediction accuracy of the model.

[0027] Optionally, during the training phase of the large model, the reconstruction head calculates the deviation between the amplitude and phase of the reconstructed data and the amplitude and phase of the input data, and the prediction head calculates the deviation between the amplitude and phase of the predicted value and the amplitude and phase of the true value to evaluate the prediction accuracy of the model;

[0028] During the testing phase of the large model, the frequency domain representation is converted to the time domain, and the mean square error between the reconstructed data and the input data, and between the predicted value and the true value are calculated to evaluate the reconstruction effect and prediction accuracy of the model.

[0029] Optionally, for prediction tasks, only use , the Adam algorithm is used to update the prediction head parameters; for anomaly detection and interpolation tasks, only the , the Adam algorithm is used to update the reconstruction head parameters.

[0030] Optionally, without changing the network structure, a suitable output head is manually selected according to the specific task type in order to obtain the corresponding task result.

[0031] In a second aspect, the present application provides a storage medium, which adopts the following technical solution:

[0032] A storage medium storing a program of any one of the above-mentioned methods for constructing a large time series model based on frequency domain representation learning.

[0033] In summary, the present application includes at least one of the following beneficial technical effects:

[0034] 1. In order to achieve the unification of time series pre-training and downstream tasks, the present invention proposes a time series large model framework for frequency domain representation learning. The framework represents the frequency domain features in the form of basis functions, and constructs a unified frequency domain representation of the time series by learning the characteristics of the basis functions themselves and their mutual relationships. At the same time, a pure frequency domain loss function is proposed to enhance the learning effect of the frequency domain representation. By designing a joint optimization strategy of reconstruction and prediction, the framework can flexibly adapt to different downstream tasks without changing the pre-training network structure.

[0035] 2. By learning and optimizing frequency domain features, the model can more accurately predict the values ​​of future time series; by optimizing the reconstruction head, the model can more effectively detect anomalies in the time series; by learning frequency domain features, the model can more accurately interpolate missing time series data. In this way, by effectively learning and utilizing frequency domain features, the prediction accuracy, anomaly detection capability and interpolation accuracy of time series data can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a method for constructing a large time series model based on frequency domain representation learning.

[0037] Figure 2 It is a schematic diagram of the system module architecture of the timing large model construction method.

[0038] Figure 3 This is a schematic diagram of the combination of frequency domain representation learning and self-attention mechanism in the present invention. DETAILED DESCRIPTION

[0039] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings.

[0040] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0041] The present application embodiment discloses a method for constructing a large time series model based on frequency domain representation learning, referring to Figure 1 ,include:

[0042] The original time series data is converted into frequency domain representation, and the amplitude and phase of each frequency component are extracted from the frequency domain representation; wherein the frequency domain representation is composed of a group of components with different frequencies, and each frequency component is represented as , extract the amplitude from each frequency component and Phase , , , each set of amplitude and phase corresponds to a basis function ;

[0043] Use a large model to extract features and learn representations of each basis function, and introduce an attention mechanism so that the large model can adaptively focus on the frequency domain components that are most critical to the task;

[0044] The large model jointly optimizes the reconstruction head and the prediction head, and optimizes the amplitude and phase deviation between the true value and the reconstructed value in the frequency domain according to the preset pure frequency domain loss function.

[0045] In order to achieve the unification of time series pre-training and downstream tasks, this embodiment proposes a large time series model framework for frequency domain representation learning. The framework represents the frequency domain features in the form of basis functions, and constructs a unified frequency domain representation of the time series by learning the characteristics of the basis functions themselves and their mutual relationships. At the same time, a pure frequency domain loss function is proposed to enhance the learning effect of the frequency domain representation. By designing a joint optimization strategy of reconstruction and prediction, the framework can flexibly adapt to different downstream tasks without changing the pre-training network structure.

[0046] In this embodiment of the present invention, optionally, the pure frequency domain loss function includes:

[0047]

[0048]

[0049]

[0050] in, and represent the amplitude and phase of the true value respectively, and denote the amplitude and phase of the reconstructed value, respectively. is the reconstruction loss, To predict the loss, are the hyperparameters of the corresponding items, .

[0051] By adopting the above technical solution, the reconstruction loss measures the model's ability to recover the original signal from the frequency domain representation. The phase is periodic. In the large time series model based on frequency domain representation learning, the design of the total loss function needs to take into account both reconstruction accuracy and prediction performance, while fully exploring the physical meaning of frequency domain features (amplitude and phase).

[0052] Optionally, the pure frequency domain loss function includes:

[0053]

[0054]

[0055] in, and represent the amplitude and phase of the true value respectively, and denote the amplitude and phase of the reconstructed value, respectively. is the reconstruction loss, To predict the loss, is the prediction loss at time t, is the regularization term, are the hyperparameters of the corresponding items, , is the phase smoothing regularization weight; is the balance coefficient between task difficulty and uncertainty, which is set according to scenario experience. , is the covariance matrix trace of the prediction task output, reflecting uncertainty, It is the scene-related covariance threshold, which needs to be determined through historical data statistics.

[0056] Specifically, in the initial training phase, you can configure , gradually increase the prediction weight to the target value to avoid early overfitting. If the prediction task loss variance is large, automatically reduce its weight to balance the noise impact. The regularization term includes a phase smoothing term (suppressing mutations) and a spectral sparsity term (increasing the weight of the main frequency component). By increasing the phase smoothing regularization weight, the cumulative error of multi-step prediction can be reduced.

[0057] In the scenario of long-term traffic flow prediction (24 hours), the demand is to suppress phase drift and reduce false alarms during peak hours in the morning and evening. Set to 0.7 (focusing on task difficulty), Configured to 1.2 (covariance threshold), The weight of the spectrum sparse item is configured as 0.1 and 0.05. Compared with the static weight, the prediction error can be reduced by 18% and the false alarms during peak hours can be reduced by 25%.

[0058] like Figure 2 As shown, the system architecture used in the present invention includes the following modules:

[0059] Input module: Receives time series data from different fields.

[0060] Embedding layer: maps discrete time series to vector space.

[0061] Time-frequency conversion module: Use fast Fourier transform, wavelet transform and other technologies to convert time domain data into frequency domain representation.

[0062] Frequency domain representation learning module: learns basis function features in the frequency domain and focuses on key frequency domain components through the attention mechanism.

[0063] Reconstruction head and prediction head: used to realize the reconstruction value and prediction value of the signal respectively.

[0064] Basis function correlation learning and basis function transformation learning modules: optimize the representation and transformation of basis functions to improve the representation ability of the model.

[0065] like Figure 3 As shown in the figure, the picture shows a neural network structure based on the self-attention mechanism, which is specifically a part of the Transformer model. The figure is divided into two main parts: Base Function correlation learning and Base Function transformation learning.

[0066] The Base Function correlation learning section shows the core calculation process of the self-attention mechanism:

[0067] The input sequence is linearly transformed to obtain three vectors: query (Q), key (K) and value (V).

[0068] Q and K are calculated through matrix multiplication (MatMul) to obtain the attention score matrix.

[0069] The score matrix is ​​scaled and normalized by Softmax to obtain the attention weight.

[0070] The attention weight is multiplied by the V vector to obtain the weighted summation result Z.

[0071] The Base Function transformation learning section shows how to further process the results of the attention mechanism:

[0072] Z is transformed linearly to obtain amplitude and phase.

[0073] The amplitude and phase are combined in some way (combine) to get the final output.

[0074] The Transformer model is an important architecture in the field of deep learning, and its core is the self-attention mechanism. The self-attention mechanism calculates the correlation between each element in the input sequence and other elements, dynamically assigning weights to each element, thereby focusing on different parts.

[0075] The calculation process of the self-attention mechanism:

[0076] Linear transformation: The input sequence is transformed into three vectors: query (Q), key (K), and value (V).

[0077] Dot Product Attention: Calculate the dot product of Q and K to get the attention score matrix.

[0078] Softmax normalization: Normalize the attention scores to obtain attention weights.

[0079] Weighted summation: Multiply the attention weight by V to get the weighted summation result Z.

[0080] Z is transformed linearly to obtain amplitude and phase respectively.

[0081] The amplitude and phase are combined in some way to get the final output.

[0082] Application scenarios: The Transformer model was originally used for natural language processing tasks such as machine translation and text classification. Due to its efficient parallel computing capabilities and powerful feature extraction capabilities, it has gradually expanded to fields such as computer vision, speech recognition, and recommendation systems. Therefore, the self-attention mechanism is the core of the Transformer model, which improves the model's expressiveness and generalization capabilities by dynamically paying attention to different parts of the input sequence. The Transformer model achieves efficient parallel computing and powerful feature extraction capabilities through the self-attention mechanism and multi-head attention mechanism, and has achieved remarkable results in many fields.

[0083] Optionally, the reconstruction head calculates the deviation between the amplitude and phase of the reconstructed data and the amplitude and phase of the input data to evaluate the reconstruction effect of the model;

[0084] The prediction head calculates the deviation between the amplitude and phase of the predicted value and the amplitude and phase of the true value to evaluate the prediction accuracy of the model.

[0085] Optionally, during the training phase of the large model, the reconstruction head calculates the deviation between the amplitude and phase of the reconstructed data and the amplitude and phase of the input data, and the prediction head calculates the deviation between the amplitude and phase of the predicted value and the amplitude and phase of the true value to evaluate the prediction accuracy of the model;

[0086] During the testing phase of the large model, the frequency domain representation is converted to the time domain, and the mean square error between the reconstructed data and the input data, and between the predicted value and the true value are calculated to evaluate the reconstruction effect and prediction accuracy of the model.

[0087] Optionally, for prediction tasks, only use , the Adam algorithm is used to update the prediction head parameters; for anomaly detection and interpolation tasks, only the , the Adam algorithm is used to update the reconstruction head parameters.

[0088] Optionally, without changing the network structure, a suitable output head is manually selected according to the specific task type in order to obtain the corresponding task result.

[0089] An embodiment of the present application also discloses a storage medium storing a program of any one of the above-mentioned methods for constructing a large time series model based on frequency domain representation learning.

[0090] The various processes and treatments described above, such as a method, can be executed by a CPU. For example, in some embodiments, a method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the computer program can be loaded and / or installed on a computing device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, a method or multiple actions of intelligent cold source disaster monitoring based on multi-sensor fusion described above can be executed.

[0091] The present disclosure relates to methods, computing devices, computer-readable storage media, and / or computer program products. The computer program product may include computer-readable program instructions for executing various aspects of the present disclosure.

[0092] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0093] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge computing devices. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0094] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0095] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions enable the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of a method for monitoring intelligent cold source disaster-causing objects based on multi-sensor fusion or multiple actions.

[0096] Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other equipment, so that a series of operating steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing device, or other equipment implement an intelligent cold source disaster-causing object monitoring method or multiple actions based on multi-sensor fusion.

[0097] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0098] Based on the same inventive concept, an embodiment of the present invention provides an intelligent terminal, including a memory and a processor, and the memory stores a computer program that can be loaded by the processor and executes a method. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for constructing a large time series model based on frequency domain representation learning, characterized in that: include: The original time series data is converted into a frequency domain representation, and the amplitude and phase of each frequency component are extracted from the frequency domain representation; wherein each set of amplitude and phase corresponds to a basis function; Use a large model to extract features and learn representations of each basis function, and introduce an attention mechanism so that the large model can adaptively focus on the frequency domain components that are most critical to the task; The large model jointly optimizes the reconstruction head and the prediction head, and optimizes the amplitude and phase deviation between the true value and the reconstructed value in the frequency domain according to the preset pure frequency domain loss function.

2. The method for constructing a large time series model based on frequency domain representation learning according to claim 1 is characterized in that: The pure frequency domain loss function includes: in, and represent the amplitude and phase of the true value respectively, and denote the amplitude and phase of the reconstructed value, respectively. is the reconstruction loss, To predict the loss, are the hyperparameters of the corresponding items, .

3. The method for constructing a large time series model based on frequency domain representation learning according to claim 2 is characterized in that: The pure frequency domain loss function includes: in, and represent the amplitude and phase of the true value respectively, and denote the amplitude and phase of the reconstructed value, respectively. is the reconstruction loss, To predict the loss, is the prediction loss at time t, is the regularization term, are the hyperparameters of the corresponding items, , is the phase smoothing regularization weight; is the balance coefficient between task difficulty and uncertainty, which is set according to scenario experience. , is the covariance matrix trace of the prediction task output, reflecting uncertainty, It is the scene-related covariance threshold, which needs to be determined through historical data statistics.

4. The method for constructing a large time series model based on frequency domain representation learning according to claim 1, characterized in that: In the training phase of the large model, the reconstruction head calculates the deviation between the amplitude and phase of the reconstructed data and the amplitude and phase of the input data, and the prediction head calculates the deviation between the amplitude and phase of the predicted value and the amplitude and phase of the true value to evaluate the prediction accuracy of the model; During the testing phase of the large model, the frequency domain representation is converted to the time domain, and the mean square error between the reconstructed data and the input data, and between the predicted value and the true value are calculated to evaluate the reconstruction effect and prediction accuracy of the model.

5. The method for constructing a large time series model based on frequency domain representation learning according to claim 1, characterized in that: For prediction tasks, only , the Adam algorithm is used to update the prediction head parameters; for anomaly detection and interpolation tasks, only the , the Adam algorithm is used to update the reconstruction head parameters.

6. The method for constructing a large time series model based on frequency domain representation learning according to claim 1, characterized in that: Without changing the network structure, the appropriate output head is manually selected according to the specific task type in order to obtain the corresponding task results.

7. A storage medium, characterized in that: A program storing a method for constructing a large time series model based on frequency domain representation learning as described in any one of claims 1 to 6.

Images