Time-series forecasting system, device, and method

Abstract

The purpose of an embodiment of the present disclosure is to provide a system comprising at least one processor and a memory storing one or more instructions, wherein the at least one processor generates prediction data by processing an input sequence segmented along a time axis by using a neural network, by executing the one or more instructions, and the neural network includes: a first neural network that applies dilated attention to the input sequence segmented along the time axis; and a second neural network that applies random partition attention to data arranged along a feature axis.

Description

Time series forecasting system, device and method

[0001] The present disclosure relates to a time-series forecasting system, device, and method, and more particularly, to a system, device, and method for predicting the future using data sequentially recorded over time using a neural network.

[0002] Time-series data refers to data recorded sequentially over time. The problem of analyzing observed time-series data to predict the future is called time-series forecasting. Recently, research has primarily focused on predicting the future using multivariate time-series data. Since these variables can range from hundreds to millions, algorithms capable of efficiently processing and learning these variables are becoming increasingly important. The present disclosure provides an efficient and accurate multivariate time-series forecasting system, device, and method.

[0003] Prior art literature

[0004] (Patent Document 1) U.S. Patent Publication No. 10,956,819 (May 23, 2021)

[0005] The present disclosure provides an efficient and highly accurate multivariate time series forecasting system, device, and method.

[0006] One embodiment of the present disclosure provides a time-series forecasting system, device, and method using a neural network.

[0007] One embodiment of the present disclosure provides a system including at least one processor and a memory containing one or more instructions. The at least one processor may generate prediction data by executing the one or more instructions stored in the memory, thereby processing an input sequence segmented along a time axis using a neural network. In addition, the neural network may include at least one of a first neural network that applies extended attention to the input sequence segmented along the time axis; and a second neural network that applies randomly distributed attention to data arranged along a feature axis. The at least one processor may determine temporal relationship information of input data using the first neural network, determine characteristic relationship information of the input data using the second neural network, and generate the prediction data based on the temporal relationship information and the characteristic relationship information. In one embodiment, at least one of the first neural network and the second neural network may include a sparse attention module.

[0008] In one embodiment, the system may further include a segmentation module that segments input data along a time axis to generate one or more segments.

[0009] In one embodiment, the first neural network may include at least one of a first MultiHead Self-Attention (MHSA) module that extracts features between segments within a predetermined period based on rearrangement of the input sequence segmented along the time axis for each feature; and a second MHSA module that extracts features between periods of segments that are periodically spaced apart by the predetermined period.

[0010] In one embodiment, the predetermined period is such that the number of segments is N. SWhen, It could be.

[0011] In one embodiment, the system further includes a random partitioning module that randomly distributes features of input data, and the second neural network may include a third MHSA module that extracts dependencies between features based on rearrangement of the features of the input data according to an arrangement determined by the random partitioning module.

[0012] One embodiment of the present disclosure provides a method for generating predictive data, performed by at least one processor, comprising: generating predictive data by processing a time-axis segmented input sequence using a neural network; wherein the neural network may include a first neural network that applies extended attention to the time-axis segmented input sequence; and a second neural network that applies random partition attention to data in which output data of the first neural network is arranged along a feature axis.

[0013] In one embodiment, the step of generating the prediction data may include: determining temporal relationship information of the input data using the first neural network; determining characteristic relationship information of the input data using the second neural network; and generating the prediction data based on the temporal relationship information and the characteristic relationship information.

[0014] In one embodiment, at least one of the first neural network and the second neural network may include a sparse attention module.

[0015] In one embodiment, the method comprises:

[0016] It may include a step of dividing the input data along a time axis to generate one or more segments.

[0017] In one embodiment, the first neural network may include a first MultiHead Self-Attention (MHSA) module that extracts features between segments within a predetermined period based on rearrangement of the input sequence segmented along the time axis for each feature; and a second MHSA module that extracts features between periods of segments that are periodically spaced apart by the predetermined period.

[0018] In one embodiment, the predetermined period is such that the number of segments is N. S When, It could be.

[0019] In one embodiment, the method further comprises a step of randomly distributing features of input data, and the second neural network may include a third MHSA module that extracts dependencies between features based on rearrangement of the features of the input data according to an arrangement determined by the random division module.

[0020] One embodiment of the present disclosure includes a program stored on a recording medium to cause a computer to execute a method according to one embodiment of the present disclosure.

[0021] One embodiment of the present disclosure includes a computer-readable recording medium having recorded thereon a program for executing a method according to one embodiment of the present disclosure on a computer.

[0022] One embodiment of the present disclosure includes a computer-readable recording medium having recorded thereon a database used in one embodiment of the present disclosure.

[0023] According to one embodiment of the present disclosure, efficient and highly accurate prediction data can be determined using multivariate time series input data.

[0024] FIG. 1a is a diagram illustrating a time series prediction method of a transformer model in which one observation becomes one token according to one embodiment of the present disclosure.

[0025] FIG. 1b is a diagram illustrating a time series prediction method of a segment-based transformer model according to one embodiment of the present disclosure.

[0026] FIG. 2 is a schematic diagram of an ESSformer (Efficient Segment-based Sparse Transformer) block according to one embodiment of the present disclosure.

[0027] FIG. 3 is a flowchart illustrating a method for generating prediction data according to one embodiment of the present disclosure.

[0028] FIG. 4 is a table showing the performance of an ESSformer (Efficient Segment-based Sparse Transformer) block according to one embodiment of the present disclosure.

[0029] FIG. 5 is a block diagram of a prediction data generation device according to one embodiment of the present disclosure.

[0030] To clarify the technical idea of ​​the present disclosure, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, if a detailed description of a related known function or component is determined to unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. Components having substantially the same functional configuration among the drawings are given the same reference numbers and symbols as possible even if they are shown in different drawings. For convenience of explanation, devices and methods are described together when necessary. Each operation of the present disclosure does not necessarily have to be performed in the described order and may be performed in parallel, selectively, or individually.

[0031] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0032] Throughout this disclosure, singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms such as "comprise" or "have" should be understood to indicate the presence of a feature, number, step, operation, component, part, or combination thereof, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. In other words, when it is said throughout this disclosure that a part "comprises" a certain component, unless specifically stated otherwise, this does not mean that other components may be included, but rather that other components may be excluded.

[0033] Expressions such as "at least one" modify the entire list of elements, not individual elements of the list. For example, "at least one of A, B, and C" and "at least one of A, B, or C" refer to only A, only B, only C, both A and B, both B and C, both A and C, all of A, B, and C, or any combination thereof.

[0034] In addition, terms such as “...unit”, “...module”, etc. described in the present disclosure mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0035] Throughout this disclosure, when a part is said to be "connected" to another part, this includes not only cases where the parts are "directly connected," but also cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise specifically stated.

[0036] The expression “configured to” as used throughout this disclosure can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system, together with other devices or components, is “capable of.” For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0037] Throughout this disclosure, the expression [N: M] refers to a set of integers from N to M, including N and excluding M. That is, [N: M] can mean {N, N+1, ..., M-1}.

[0038] Time series forecasting is a fundamental machine learning task that aims to predict future events based on past observations. These forecasting problems often require long-term predictions and may involve multiple variables. For example, stock price forecasting may require multiple market value predictions over a long time horizon. In this multivariate long-term time-series forecasting (M-LTSF) problem, it is crucial to capture both the long-term temporal dependencies between past and future events and the interdependencies between features of different variables.

[0039] In recent years, many deep neural architectures, such as linear models, state-space models, and recurrent neural networks (RNNs), have been developed for the M-LTSF problem. One such model is the Transformer model, a neural network that learns context and meaning by tracking relationships within sequential data, such as words in a sentence. Transformer models have demonstrated remarkable performance in various domains, such as language and image processing. Due to their ability to capture long-term relationships, Transformer models are also being studied in the field of multivariate long-term time series prediction. For example, a Transformer model, in which each observation is represented as a token, as shown in Figure 1a, has been used in time series prediction. Recent research has proposed a segment-based Transformer model, in which each token is represented as a group of continuous observations rather than a single observation, as shown in Figure 1b. However, in the case of self-attention of a segment-based transformer model, one segment is made into one token, and the prediction performance improves as the segment is subdivided, but in this case, the number of tokens increases significantly, which significantly increases the computational cost of attention. In addition, in the case of inter-feature attention that finds correlations between features, as shown in Fig. 1b, prediction can be performed quite inefficiently when the number of features is very large. In order to solve these problems, one embodiment of the present disclosure provides a time series prediction method that maintains performance even with less subdivision and that maintains performance even in inter-feature attention with a large number of features. The transformer model provided by one embodiment of the present disclosure may be referred to as ESSformer (Efficient Segment-based Sparse Transformer).

[0040] FIG. 2 is a schematic diagram of an ESSformer (Efficient Segment-based Sparse Transformer) block according to one embodiment of the present disclosure.

[0041] Referring to Figure 2, Dimension-Segment-Wise (DSW) embedding can be performed to process past time-series information. In DSW embedding, each dimension's series is segmented into segments, which are then embedded into feature vectors. The output of DSW embedding can be a 2D vector matrix with time and dimension as its two axes. To efficiently capture cross-temporal and cross-dimensional dependencies between these vector matrices, a two-stage attention layer can be utilized.

[0042] In one embodiment, the ESSformer block (100) may include sparse attention modules customized for a segment-based transformer. In one embodiment, the ESSformer may include a Dilated Attention (DilA) module (110) that learns interactions between periodically distant segments to efficiently capture temporal dependencies, and a Random-Partition Attention (R-PartA) module (120) that captures inter-feature dependencies. The DilA module (110) may be an attention module in the temporal dimension, and the R-PartA module (120) may be an attention module in the feature dimension. That is, the DilA module (110) may be a model that efficiently learns temporal dependencies, and the R-PartA module (120) may be a model that efficiently learns inter-feature dependencies.

[0043] Hereinafter, the ESSformer block (100) will be described in more detail using a formula. In one embodiment, the DilA module (110) can be designed by configuring extended attention with a stride P and configuring block-diagonal attention with a block size P based on the appearance of a periodic pattern in the self-attention matrix of a segment-based transformer. Through this, the number of segments N is input. S Given the temporal attention layer, the computational cost is at can be reduced to

[0044] In one embodiment, the R-PartA module (120) captures dependencies between various features using the same size S G It can be designed by randomly dividing the features into groups and masking the attention matrix between different groups. Through this design, when the feature size is D, the attention computation cost is O(D 2 ) in O(DS G ) can be reduced. In one embodiment, the inherent stochasticity in the random partitioning of the R-PartA module (120) can enable efficient and effective learning. In addition, in one embodiment, the limitation that the entire inter-feature relationship cannot be captured from masked attention can be resolved by using a test-time ensemble technique in the inference step.

[0045] In one embodiment, the D variable time series observation x at time t t Is can be expressed as , where x t,d refers to the actual value observation of the dth feature at time t. The goal of time series forecasting is to use previous observations Future observations based on can be predicted, where T is the length of the past time step, means the length of the future time step. One embodiment of the present disclosure is a multivariate long-term time series forecasting D>1 and ≫ We aim to provide an efficient time series prediction method for the case of 1.

[0046] In one embodiment, multivariate time series observations is of the same length as N S It can be divided into segments. That is, the b-th segment of the d-th feature can be expressed as in the following [Mathematical Formula 1].

[0047] [Mathematical Formula 1]

[0048]

[0049] In one embodiment, observations are inserted into the latent space through a linear layer, and a learnable temporal encoding and feature-specific location encoding With this addition, the input can be expressed as in the following [Mathematical Formula 2].

[0050] [Equation 2]

[0051]

[0052] Initial expression H (0) Given this input, a segment-based transformer encoder with L layers produces a final representation H (L) , and this output value H (L) can be passed through the decoder to predict future observations.

[0053] In one embodiment, a linear-based decoder is used, is a future observation by a single linear layer can be mapped to.

[0054] Below, we will describe an ESSformer according to an embodiment of the present disclosure based on the above expressions. In one embodiment, the input segment expression is H (0)Given this, each layer of ESSformer can be expressed as in [Mathematical Formula 3] and [Mathematical Formula 4] below.

[0055] [Equation 3]

[0056]

[0057] [Equation 4]

[0058]

[0059] Below, the DilA module (110) is described. In one embodiment, the input segments To capture temporal relations from, the DilA module (110) processes the input through two attention modules (112, 114), each of which can discover distinct temporal relations. In one embodiment, the attention modules (112, 114) can be Multi-head Self Attention (MHSA) modules. For intra-temporal relations, the block-diagonal attention module (112) with block size P can blend features between segments within the same temporal period. Additionally, for inter-temporal relations, the extended attention module (114) with stride P can periodically share representations between distant segments for longer-range contextualization.

[0060] Here, Q, K, V are query, key, and value, respectively, and we assume that MHSA(Q, K, V) is represented by a vanilla MHSA layer. Given a set of numbers C as indices, selecting all indices contained in C (e.g., ) can be meant. At this time, the step-by-step procedure of the DilA module (110) can be expressed as in the following [Mathematical Formula 5] and [Mathematical Formula 6].

[0061] [Equation 5]

[0062]

[0063] [Equation 6]

[0064]

[0065] Here, [j :: P] represents a set of indices starting from j with stride P. That is, [j :: P] := {j, j+P, j+2P, ...} may be satisfied. In one embodiment, the block-diagonal attention module can capture within-period relationships through [Equation 5], and between-period relationships can be considered through [Equation 6].

[0066] If the DilA module (110) is not used, N is used through self-attention. S To encode the segments of the dog, requires computational cost, which can become unwieldy when considering time series data with large T. If we increase the duration of each segment, N S Although it can reduce the segment granularity in transformer-based generative modeling, the inference quality also deteriorates as the segment granularity decreases. Accordingly, considering that time series prediction is similar to generating future observations conditioned on past signals, an efficient architecture with a second-order asymptotic cost in terms of the number of segments is required. To address this limitation, the DilA module (110) according to one embodiment effectively imposes a block diagonal and stride sparse attention mask, thereby reducing the computational cost without significantly sacrificing the expressiveness of self-attention.

[0067] A periodically extended sparsity structure according to one embodiment is proposed inspired by the graphs in Fig. 3, which depict the attention score matrix of various transformer models after training on M-LTSF. In one embodiment, the periodic Because of this, the time and memory complexity at can be reduced to P *Periodically sparse attention using may be sufficient to maintain the downstream functionality of full attention.

[0068] Below, the R-PartA module (120) is described. The segment-based transformer for M-LRSF can improve downstream performance by individually tokenizing each feature and modeling the interaction between features in addition to temporal contextualization. However, this is O(D) with full attention. 2 ) requires a computational cost, and thus it may be difficult to handle a large number (D) of features. In one embodiment, to reduce the cost for D, the R-PartA module (120) first divides the D features into N G Separate group of dogs can be randomly divided into S. Here, all the separation groups are S G have the same size, where And, In one embodiment, a single segment can be sampled before each forward step and used across all layers of the transformer model. The R-PartA module (120) can then blend representations between features within the same group using block-diagonal attention, according to Equation (6).

[0069] [Equation 6]

[0070]

[0071] This task only considers interactions within groups, so the computational cost is O(D). 2 ) in O(DS G ) can be reduced. However, if the prediction procedure is executed once at the inference stage, only partial feature information within each group can be considered. To address the limitation of not utilizing the entire information, the test-time ensemble method randomly splits N E Run the prediction procedure by randomly dividing the time, NE The prediction outputs can be ensembled (e.g., averaged). The ensemble procedure can be performed according to [Algorithm 1] below.

[0072] [Algorithm 1]

[0073]

[0074] According to one embodiment of the present disclosure, not only can computational costs be reduced through the R-PartA module (120), but prediction performance can also be improved.

[0075] In the description referring to FIG. 2, an example is given of using both the DilA module (110) and the R-PartA module (120) in the ESSformer block (100), but it is of course possible to use only one of the DilA module (110) and the R-PartA module (120).

[0076] FIG. 3 is a flowchart illustrating a method for generating prediction data according to one embodiment of the present disclosure.

[0077] Referring to FIG. 3, in operation 310, the method for generating predictive data may include an operation of segmenting input data along a time axis to generate one or more segments. In one embodiment, the input data may include input time series data (210) in the example of FIG. 2. The input time series data (210) may be multivariate time series data. The input data may be segmented along a time axis to generate an input sequence or input segment (220). Accordingly, a system for generating predictive data may include a segmentation module (240) for segmenting the input data along the time axis to generate one or more segments. An ESSformer block (100) according to one embodiment of the present disclosure may include a neural network for generating predictive data (230) using the input sequence (220). The segmentation module (240) may be located outside the ESSformer block (100) as shown in FIG. 2, or may be located within the ESSformer block (100).

[0078] In operation 330, the method for generating predictive data may include an operation of randomly distributing features of input data. Although operation 330 is illustrated as being located between operations 310 and 350 in FIG. 3, this is merely an example, and operation 330 may be performed in any order before operation 370. For example, operation 330 may be performed before operation 310, or between operations 350 and 370.

[0079] In one embodiment, the system may include a random partitioning module (250) that randomly distributes features of input data. In one embodiment, partitioning information (260) of features partitioned by the random partitioning module (250) may be utilized when using a second neural network that extracts dependencies between features.

[0080] In one embodiment, the random segmentation module (250) may be located outside the ESSformer block (100) as shown in FIG. 2, or may be located within the ESSformer block (100). For example, when the random segmentation module (250) and the segmentation module (240) are located outside the ESSformer block (100), the ESSformer block (100) may receive segmentation information (260) of features and a segmented input sequence (220) as inputs and use them to generate prediction data (230).

[0081] In operation 350, the method for generating predictive data may include an operation of determining temporal relationship information of input data using a first neural network. The first neural network may include a neural network that applies extended attention to an input sequence segmented along a time axis.

[0082] In one embodiment, at least one processor performing the predictive data generation method may rearrange the segmented input sequence based on a predetermined period and perform MultiHead Self-Attention (MHSA) on the rearranged data. Here, the predetermined period is It could be.

[0083] For example, let's explain the case where there are 6 segments as in Fig. 2, and let's explain by assigning identification numbers in order from the beginning as segment #0, segment #1, ..., segment #5. In this case, the predetermined cycle is It can be. Accordingly, at least one processor can rearrange the input sequence by breaking it up at each cycle. Accordingly, the first rearranged data (270) can be rearranged into {segment #0, segment #1}, {segment #2, segment #3}, {segment #4, segment #5}. The first MHSA (112) can be applied to the first rearranged data to extract the time-axis dependency. However, in this case, since it is difficult to identify the dependency between segments that are far apart, at least one processor can rearrange the input sequence by grouping segments that are separated by a cycle. Accordingly, the second rearranged data (280) can be rearranged into {segment #0, segment #2, segment #4}, {segment #1, segment #3, segment #5}. At least one processor can identify the dependency between segments that are separated by a cycle by using the second MHSA (114) on the second rearranged data (280). That is, the first neural network may include a first MHSA (MultiHead Self-Attention) module (112) that extracts features between segments within the same time period based on rearrangement of the input sequence segmented along the time axis for each feature, and a second MHSA module (114) that extracts features between the periods of periodically spaced segments.

[0084] In one embodiment, temporal relationship information of input data can be determined by a first neural network. According to one embodiment, prediction performance can be maintained while extracting only temporal dependencies between segments within a cycle and temporal dependencies between segments spaced apart by a cycle, rather than extracting all temporal dependencies between all segments. That is, according to one embodiment, by extracting dependencies between consecutive segments and dependencies between segments spaced apart by a certain cycle, computational complexity can be reduced while maintaining prediction performance.

[0085] In operation 370, the method for generating predictive data may include an operation of determining characteristic relationship information of input data using a second neural network. In one embodiment, the second neural network may include a neural network that applies random partition attention to data arranged along a feature axis. The second neural network may utilize data arranged along a feature axis, such as output data of the first neural network, or may utilize data arranged along a feature axis, such as segmented input sequences.

[0086] In one embodiment, the second neural network may include a third MHSA module (290) that extracts dependencies between features based on rearrangement of features of the input data according to the partition information (260) determined by the random partition module (250). For example, if there are four total features, namely feature #1, feature #2, feature #3, and feature #4 from the top, and the partition information (260) is {feature #4, feature #2}, {feature #3, feature #1} as illustrated in FIG. 2, at least one processor may rearrange data aligned along the feature axis into {feature #4, feature #2}, {feature #3, feature #1} to generate third rearranged data. In addition, at least one processor may apply MHSA to the third rearranged data and rearrange it again based on the partition information (260) to restore the order of the features. Through this, at least one processor may determine characteristic relationship information of the input data using the second neural network.

[0087] In operation 390, the method for generating predictive data may include generating predictive data based on temporal relationship information and characteristic relationship information. In one embodiment, the predictive data may be generated based on temporal relationship information determined using a first neural network and characteristic relationship information determined using a second neural network. That is, at least one processor may generate predictive data by processing an input sequence segmented along a time axis using neural networks.

[0088] In one embodiment, at least one of the first neural network and the second neural network may include a sparse attention module.

[0089] FIG. 4 is a table showing the performance of an ESSformer (Efficient Segment-based Sparse Transformer) block according to one embodiment of the present disclosure.

[0090] Referring to FIG. 4, it can be demonstrated that the ESSformer method achieves the most efficient computational complexity among various segment-based transformers. For example, FIG. 4 shows that ESSformer achieves the highest performance in 27 out of 28 tasks of M-LTSF. It also achieves second place in one remaining task. According to one embodiment of the present disclosure, the ESSformer method can not only reduce computational complexity but also improve prediction performance.

[0091] FIG. 5 is a block diagram of a prediction data generation device according to one embodiment of the present disclosure.

[0092] Referring to FIG. 5, the prediction data generation device (500) (the device may also be referred to as a server or a system) may include a transceiver (510), a memory (520), a data storage (530), and a processor (550). However, not all of the components illustrated in FIG. 5 are essential components of the prediction data generation device (500). The prediction data generation device (500) may be implemented with more components than the components illustrated in FIG. 5, or may be implemented with fewer components than the components illustrated in FIG. 5. In addition, the transceiver (510), the memory (520), and the processor (550) may be implemented in the form of a single chip.

[0093] In one embodiment, the transceiver (510) can communicate with a terminal or other electronic device connected wired or wirelessly to the prediction data generation device (500).

[0094] Various types of data, such as programs and files, such as applications, can be installed and stored in the memory (520). The processor (550) can access and use data stored in the memory (520), or store new data in the memory (520). In addition, the memory (520) can store one or more instructions. The processor (550) can execute one or more instructions stored in the memory. The memory (520) can store information within the system. For example, the memory (520) can be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit.

[0095] The artificial intelligence-related functions according to the present disclosure are operated via a processor and memory. The processor may be comprised of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0096] In one embodiment, the data storage (530) may provide large storage for the predictive data generation device (500). For example, the data storage (530) may be a computer-readable medium. Alternatively, the data storage (530) may include a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large storage device. The data storage (530) may include a trained neural network model (540).

[0097] The processor (550) controls the overall operation of the prediction data generation device (500) and may include at least one processor, such as a CPU or GPU. The processor (550) may control other components included in the prediction data generation device (500) to perform operations for operating the prediction data generation device (500). For example, the processor (550) may generate prediction data by processing an input sequence segmented along a time axis using a neural network by executing one or more commands.

[0098] An embodiment of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically contains computer-readable instructions, data structures, or program modules, and includes any information delivery media.

[0099] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0100] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In a system including at least one processor and one or more instructions and a memory, The at least one processor generates prediction data by processing an input sequence segmented along a time axis using a neural network by executing the one or more instructions stored in the memory, The above neural network, A first neural network that applies extended attention to an input sequence segmented along the time axis; and It includes a second neural network that applies random distribution (partition) attention to data arranged along the feature axis, At least one processor, Using the first neural network, determine the temporal relationship information of the input data, Using the second neural network, determine characteristic relationship information of input data, A system for generating the prediction data based on the temporal relationship information and the characteristic relationship information.

2. In the first paragraph, at least one of the first neural network and the second neural network, A system comprising a sparse attention module.

3. In paragraph 1, the system, A system further comprising a segmentation module for dividing input data along a time axis to generate one or more segments.

4. In the first paragraph, the first neural network, A first MHSA (MultiHead Self-Attention) module that extracts features between segments within a predetermined period based on rearrangement of the input sequence segmented along the time axis for each feature; and A system comprising a second MHSA module for extracting features between periods of segments that are periodically spaced apart by the above-determined period.

5. In paragraph 4, the predetermined period is: The number of segments is N S When, People, system.

6. In paragraph 1, the system, It further includes a random partitioning module that randomly distributes the features of the input data, The above second neural network, A system comprising a third MHSA module that extracts dependencies between features based on rearrangement of features of the input data according to an arrangement determined by the random partition module.

7. A method for generating prediction data performed by at least one processor, A step of generating prediction data by processing an input sequence segmented along a time axis using a neural network, The above neural network, A first neural network that applies extended attention to an input sequence segmented along the time axis; and A second neural network that applies random partition attention to data arranged along a feature axis based on the output data of the first neural network, The step of generating the above prediction data is: A step of determining temporal relationship information of input data using the first neural network; A step of determining characteristic relationship information of input data using the second neural network; and A method comprising a step of generating the prediction data based on the temporal relationship information and the characteristic relationship information.

8. In the 7th paragraph, at least one of the first neural network and the second neural network, A method comprising a sparse attention module.

9. In paragraph 7, the method, A method comprising the step of generating one or more segments by dividing input data along a time axis.

10. In the 7th paragraph, the first neural network, A first MHSA (MultiHead Self-Attention) module that extracts features between segments within a predetermined period based on rearrangement of the input sequence segmented along the time axis for each feature; and A method comprising a second MHSA module for extracting features between periods of segments that are periodically spaced apart by the above-determined period.

11. In paragraph 10, the predetermined period is: The number of segments is N S When, In, method.

12. In the 7th paragraph, the method, Further comprising a step of randomly distributing the features of the input data, The above second neural network, A method comprising a third MHSA module that extracts dependencies between features based on rearrangement of features of the input data according to an arrangement determined by the random partition module.

13. A program stored on a computer-readable recording medium that causes a computer to execute any one of the methods of Articles 7 to 12.

Images