Information processing apparatus, information processing method, and information processing program

Abstract

To analyze time series data using a base model.SOLUTION: The information processing apparatus arranges and inputs, to a model that receives input of a plurality of arranged values, a time direction series extracted from input data that is a time series including a plurality of variables. The information processing apparatus executes calculation of the model based on the input.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Time series data appears in various fields, such as stock prices and exchange rates in the financial sector, sensor data in industry, and product demand volume. Analysis of time series data, i.e., prediction, anomaly detection, and interpolation, has a large impact on industry.

[0003] Some analytical methods based on statistical models or machine learning models have a scheme to create one model for one data set. Such methods require training and model design for each domain, which incurs computational and time costs. For example, a model trained using data acquired at a factory becomes a model specific to that factory and is difficult to reuse.

[0004] In recent years, techniques for pre-training large-scale models with many parameters, such as Transformers, using large amounts of data, have been attracting attention. These techniques have produced remarkable results, particularly in the fields of image processing and natural language processing. Such pre-trained models are called foundational models. Foundational models can solve tasks in a variety of domains without additional training (zero-shot) or with a small number of training steps (few-shot). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-18550 Summary of the Invention [Problem to be solved by the invention]

[0006] However, conventional techniques have the problem that it is difficult to analyze time-series data using a foundation model.

[0007] For example, time series data from different domains may have different dimensionality (number of variables). Many conventional foundation models support a fixed number of dimensionality, and may not be able to handle time series data with different dimensionality. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the objective, an information processing device is characterized by having an input control unit that inputs a series of variable directions extracted from input data that is a time series including a plurality of variables into a model that accepts input of a plurality of arranged values, and a calculation unit that performs calculations of the model based on the input by the input control unit. [Effects of the Invention]

[0009] According to the present invention, it is possible to analyze time series data using a base model. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of time-series data. [Figure 2] FIG. 2 is a diagram illustrating the order between the series of time-series data. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the information processing apparatus according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining a method for inputting data. [Figure 5] FIG. 5 is a flowchart showing the flow of the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a computer that executes a program. [Figure 7] FIG. 7 is a diagram illustrating a conventional data input method. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of an information processing device, an information processing method, and an information processing program according to the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0012] First, we will explain the problems that arise when analyzing time-series data using a conventional platform model. For example, the time-series data may be collected in a factory. For example, the time-series data collected in a factory is data that associates time with temperature, pressure, flow rate, product volume, yield, etc. acquired by sensors.

[0013] Time series data includes multiple series. Each series corresponds to a variable. For example, time series data includes series corresponding to variables such as temperature, pressure, and flow rate. Time series data is sometimes simply called a time series.

[0014] As shown in Figure 1, the number of time series data, i.e., the number of dimensions, may differ depending on the domain. Figure 1 is a diagram showing an example of time series data. In Figure 1, the number of dimensions of the time series data in domain A is three, while the number of dimensions of the time series data in domain B is two.

[0015] For example, the difference between factories that collect time-series data corresponds to the difference between domains. For example, one factory may have 10 sensors, while another factory may have 100. Note that the number of sensors is assumed to be the number of dimensions of the time-series data.

[0016] However, many existing machine learning models have fixed input and output dimensionality, making it difficult to create a foundation model that can handle time-series data with different dimensionality for each domain using such a machine learning model.

[0017] As an alternative, it is possible to fix the number of input dimensions of the model and then increase or decrease the number of dimensions of the actually input time series data.

[0018] For example, when time series data with fewer dimensions than the fixed number of input dimensions is input, the missing series are filled in with values ​​such as 0 or 1, or padded with data from other series. However, the filled series may have some meaning, which may reduce the accuracy of the model.

[0019] Furthermore, for example, if time series data with more dimensions than the fixed number of input dimensions is input, selection is performed on the series, and only the selected appropriate series are actually input to the model. However, in this case, the disadvantages are that the amount of calculation increases and the series selection requires manual work. Furthermore, if the series selection method is inappropriate, the accuracy of the model may decrease.

[0020] Another alternative is to use models that support variable input length, such as Recurrent Neural Networks (RNNs) and Transformers, which are often used in the field of natural language processing.

[0021] In time series data, semantic information is invariant to the order of the series. In other words, as shown in Figure 2, it is desirable that the analysis results using the model do not change even if the series of time series data is swapped. However, the order in the time direction must be maintained. In the example of Figure 2, the series x 2 and series x 3 The order of has been swapped.

[0022] For example, when time-series data is stored in a CSV file, the order of the columns has no chronological meaning and simply indicates the order in which they happened to be stored. For example, if the columns are arranged in the order of temperature, pressure, and flow rate, the meaning of the data is the same as if the columns were arranged in the order of temperature, flow rate, and pressure.

[0023] Models that support variable-length input, such as the aforementioned RNN and Transformer, preserve the order of the input sequence. In other words, the order of the sequence becomes significant and affects the analysis results. For this reason, conventional models that support variable-length input are not suitable as a base model for analyzing time series data.

[0024] In the embodiment, a model capable of solving the problems described above is proposed. For example, the model in the embodiment is based on a model that supports variable length input, such as Transformer, and does not attach any significance to the order of input time series data.

[0025] [First embodiment] [Configuration of the first embodiment] The configuration of the information processing device according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the information processing device according to the first embodiment. As shown in Fig. 3, the information processing device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0026] The communication unit 11 performs data communication with other devices via a network. For example, the communication unit 11 is a network interface card (NIC).

[0027] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. Note that the storage unit 12 may also be a data-rewritable semiconductor memory such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM).

[0028] The storage unit 12 stores an OS (Operating System) and various programs executed by the information processing device 10. The storage unit 12 stores model information 121.

[0029] The model information 121 is information such as parameters for constructing a model, etc. For example, the model information 121 is the weights and biases of each layer of a neural network.

[0030] The control unit 13 controls the entire information processing device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0031] The control unit 13 also has an internal memory for storing programs defining various processing procedures and control data, and executes each process using the internal memory. The control unit 13 also functions as various processing units when various programs are run. For example, the control unit 13 has an input control unit 131, a calculation unit 132, a provision unit 133, and an update unit 134.

[0032] The input control unit 131 inputs time-series data to the model constructed from the model information 121. The model constructed from the model information 121 is a base model based on Transformer.

[0033] The calculation unit 132 executes calculations of the model, i.e., the calculation unit 132 obtains the output of the model to which the time series data is input.

[0034] The providing unit 133 provides information based on the calculation results of the model. For example, the providing unit 133 provides an estimated value of a series at a future time based on the collected time series data. The estimated value may be a predicted value, an anomaly degree, or the like.

[0035] The update unit 134 updates the model parameters (model information 121) so that the calculation results of the model are optimized.

[0036] For example, when inference using a model is performed, the providing unit 133 operates, and when model learning is performed, the updating unit 134 operates.

[0037] As shown in Figure 7, the conventional Transformer computes a multivariate time series {x t 1 ,x t 2 ,…,x t K} and executes the calculation. Conventional Transformers learn the relationship between time series and variables by repeating this input and calculation for each time (t+1, t+2, ...). Figure 7 explains the conventional data input method.

[0038] As described above, in conventional Transformers, the number of input dimensions in the time direction is variable, but the number of sequences K is fixed, which poses the problem described above.

[0039] On the other hand, the input control unit 131 first extracts time series data with a fixed window width W in the time direction. The width W may be set in advance by the user. The larger the width W, the more past information is included in the extracted time series data.

[0040] As shown in FIG. 4, the input control unit 131 extracts the sequence of variables {x t-W 1 ,x t-W+1 1 ,…,x t-W+2 1} is input into the model. Figure 4 is a diagram explaining how to input data.

[0041] The calculation unit 132, which will be described later, performs model calculations on the input sequence. The model here includes the Embedding and Transformer in FIG. 4. The input control unit 131 performs the same input on K sequences. That is, the input control unit 131 calculates the sequence {x t-W2 ,x t-W+1 2 ,…,x t-W+2 2} into the model, and then in the next iteration, the series {x t-W 3 ,x t-W+1 3 ,…,x t-W+2 3} into the model.

[0042] In this way, the input control unit 131 inputs variable-length data to the model in the variable direction, not in the time step (time) direction, which allows the input control unit 131 to input time-series data of any number of sequences to the model.

[0043] Here, as shown in FIG. 7, a conventional Transformer has a function called Positional Encoding for maintaining order relationships. On the other hand, as shown in FIG. 4, the model of this embodiment has a layer (Embedding) that embeds data in the time direction of a sequence instead of Positional Encoding. This Embedding layer makes it possible to learn relationships between variables without depending on the order of the sequences input to the model, while extracting the temporal dependencies of the sequences. For example, this Embedding layer can be implemented using Feed Forward Networks or one-dimensional Convolutional Networks. For example, the Embedding layer in FIG. 4 converts a vector of length W corresponding to each variable into a vector of a predetermined length. In other words, the Embedding layer provides the embedded vector with information indicating the order relationship of each element across time, but does not provide information indicating the order relationship between variables (sequences). This allows the model (especially the Transformer portion after the Embedding layer) to learn relationships between variables without depending on the order of the input sequences.

[0044] [Processing of the first embodiment] FIG. 5 is a flowchart showing the flow of the information processing apparatus according to the first embodiment. As shown in FIG. 5, first, the input control unit 131 acquires a time series (times 0 to T) with K variables (K≧2) (step S101). The number of variables is the number of series in the time series data.

[0045] The input control unit 131 sets j to 1 (step S102). j is the index of a fixed window with width W cut out from the acquired time series data. Also, the maximum value of j is J.

[0046] The input control unit 131 cuts out the time series data included in the j-th fixed window from the time series data acquired in step S101 (step S103).

[0047] Next, the input control unit 131 sets k to 1 (step S104). k is the index of the series of variables input to the model. Also, the maximum value of k is K.

[0048] The input control unit 131 inputs the data (series) at each time of the k-th variable into the Transformer constructed from the model information 121 (step S105). However, in the Transformer here, Positional Encoding is not executed, and embedding for the series time direction is executed.

[0049] If k<K (step S106, Yes), the input control unit 131 increments k by 1 (step S107), returns to step S105, and repeats the process. Note that k<K means that there are remaining series of variables that have not been input.

[0050] If k≮K (step S106, No), the calculation unit 132 executes the calculation of the Transformer (step S108).

[0051] Here, when there is unextracted data, that is, when j < J (step S109, Yes), the input control unit 131 increases j by 1 (step S110), returns to step S103, and repeats the process.

[0052] Here, when there is no unextracted data, that is, when j < J is not satisfied (step S109, No), the providing unit 133 provides information regarding estimation based on the calculation result of the Transformer (step S111). For example, the providing unit 133 provides the value of the variable at a future time output by the Transformer. Note that how the Transformer outputs the calculation result can be controlled by the learning method (for example, the way of giving correct data, the design of tasks such as prediction and anomaly detection).

[0053] Note that during learning, instead of step S111, the updating unit 134 updates the parameters of the Transformer, that is, the model information 121, so that the calculation result of the Transformer is optimized.

[0054] Supplement on step S105 and related processes. First, the time series as the input data is data in which the values of each of a plurality of variables are associated with each of a plurality of times, as shown in FIGS. 2 and 4 for example. The time series data may be expressed as a matrix with the variable direction as rows and the time direction as columns.

[0055] As shown in FIG. 7, in the prior art, the series in the variable direction extracted from the time series data are arranged and input to the Transformer. The series in the variable direction is {x t 1 ,x t 2 ,…}, {x t+1 1 ,x t+1 2 ,…}, etc.

[0056] 5, the input control unit 131 arranges the time-direction sequences extracted from the time-series data and inputs them to the Transformer. k ,x1 k ,…}, {x0 k+1 ,x1 k+1 ,…} etc.

[0057] The model of this embodiment can be said to be a model that removes the Positional Encoding function from a Transformer that accepts input of multiple values ​​of any length arranged in a sequence. Therefore, with the Transformer of this embodiment, it is expected that the order of the inputs will not affect the calculation results.

[0058] In this way, by using the model of this embodiment as a base model, it becomes possible to analyze time series data using the base model.

[0059] The information processing device 10 can train the model of this embodiment using time-series data sampled from a collection of time-series datasets collected from various domains. The model allows variable-length input and is not affected by the order of input variables, so training can be performed on datasets across domains with different numbers of series and different orders of series. Because the trained model is trained using a variety of data, even when an unknown dataset is given, it can make estimations without the need for retraining (zero-shot). Alternatively, the trained model can adapt to an unknown dataset after a few training rounds (few-shot), enabling more accurate estimation.

[0060] [System configuration, etc.] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0061] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0062] [program] In one embodiment, the information processing device 10 can be implemented by installing a learning program that executes the above-described learning process as package software or online software on a desired computer. For example, by executing the above-described learning program on an information processing device, the information processing device can function as the information processing device 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include tablet terminals, smartphones, mobile phones, PHS (Personal Handyphone System) and other mobile communication terminals, as well as slate terminals such as PDAs (Personal Digital Assistants).

[0063] Furthermore, the information processing device 10 can also be implemented as a server that provides services related to the learning process to a client, with the terminal device used by the user being the client.

[0064] 6 is a diagram showing an example of a computer that executes a learning program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0065] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0066] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the information processing device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the information processing device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0067] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0068] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0069] 10. Information processing equipment 11 Communications Department 12 Storage section 13 Control Unit 121 Model Information 131 Input control section 132 Calculation section 133 Provision Department 134 Update Department

Claims

1. an input control unit that inputs, into a model that receives an input of a plurality of arranged values, a time series extracted from input data that is a time series including a plurality of variables; a calculation unit that executes calculations of the model based on inputs from the input control unit; An information processing device comprising:

2. The input control unit extracts a sequence of values ​​corresponding to each of a plurality of times for each of a plurality of variables from the input data in which values ​​of each of the plurality of variables are associated with each of a plurality of times, and inputs the extracted sequence into the model in an order.

2. The information processing apparatus according to claim 1, wherein:

3. The input control unit is a model that receives an input of a plurality of values ​​of an arbitrary length arranged, and inputs the sequence arranged into a model having an Embedding layer that performs embedding in the time direction and a Transformer.

2. The information processing apparatus according to claim 1, wherein:

4. An information processing method executed by an information processing device, an input control step of arranging and inputting a time series extracted from input data that is a time series including a plurality of variables into a model that receives an input of an arranged plurality of values; a calculation step of executing a calculation of the model based on the input from the input control step; An information processing method comprising:

5. A model that accepts input of multiple values ​​arranged in a row is input with a time series extracted from input data that is a time series containing multiple variables, and Performing calculations on said model based on inputs An information processing program that causes a computer to execute a process.

Images