Information processing apparatus, information processing method, and information processing program

Abstract

To analyze time series data using a base model.SOLUTION: The information processing apparatus calculates first information indicating a feature of an endogenous variable using the first model 212, calculates second information indicating a relationship between the endogenous variable and a plurality of exogenous variables from the first information and the plurality of exogenous variables using the second model 222, and calculates a predicted value of the endogenous variable from the first information and the second information using the third model 230. The information processing apparatus updates the parameter of the second model 222 so as to reduce the influence of the order of inputting the plurality of exogenous variables to the second model 222 on the second information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Time series data appear in various fields, such as stock prices and exchange rates in the financial sector, sensor data in industry, and product demand volume. Predicting time series data has a large impact on industry.

[0003] Some prediction methods based on statistical models or machine learning models have a scheme to create one model for one data set. Such methods require training and model design for each domain, which incurs computational and time costs. For example, a model trained using data acquired at a factory becomes a model specific to that factory and is difficult to reuse.

[0004] In recent years, techniques for pre-training large-scale models with many parameters, such as Transformers, using large amounts of data, have been attracting attention. These techniques have produced remarkable results, particularly in the fields of image processing and natural language processing. Such pre-trained models are called foundational models. Foundational models can solve tasks in a variety of domains without additional training (zero-shot) or with a small number of training steps (few-shot). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-18550 Summary of the Invention [Problem to be solved by the invention]

[0006] However, conventional techniques have the problem that it is difficult to analyze time-series data using a foundation model.

[0007] For example, time series data from different domains may have different dimensionality (number of variables). Many conventional foundation models support a fixed number of dimensionality, and may not be able to handle time series data with different dimensionality. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, an information processing device is characterized by having: a calculation unit that calculates first information representing characteristics of an endogenous variable using a first model; calculates second information representing a relationship between the endogenous variable and a plurality of exogenous variables from the first information and a plurality of exogenous variables using a second model that supports variable-length input; and calculates a predicted value of the endogenous variable from the first information and the second information using a third model; and an update unit that updates parameters of the second model so as to reduce the effect on the second information of the order in which the plurality of exogenous variables are input to the second model. [Effects of the Invention]

[0009] According to the present invention, it is possible to analyze time series data using a base model. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of time-series data. [Figure 2] FIG. 2 is a diagram illustrating the order between the series of time-series data. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the information processing device according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining a method for inputting data. [Figure 5] FIG. 5 is a diagram illustrating the configuration of the model. [Figure 6]FIG. 6 is a flowchart showing the flow of the information processing device according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a computer that executes a program. [Figure 8] FIG. 8 is a diagram illustrating a conventional data input method. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of an information processing device, an information processing method, and an information processing program according to the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0012] First, we will explain the problems that arise when analyzing time-series data using a conventional platform model. For example, the time-series data may be collected in a factory. For example, the time-series data collected in a factory is data that associates time with temperature, pressure, flow rate, product volume, yield, etc. acquired by sensors.

[0013] Time series data includes multiple series. Each series corresponds to a variable. For example, time series data includes series corresponding to variables such as temperature, pressure, and flow rate. Time series data is sometimes simply called a time series.

[0014] As shown in Figure 1, the number of series, i.e., the number of dimensions, of time series data may differ depending on the domain. Figure 1 is a diagram showing an example of time series data. In Figure 1, the number of dimensions of the time series data in domain A is three, while the number of dimensions of the time series data in domain B is two.

[0015] For example, the difference between factories that collect time-series data corresponds to the difference between domains. For example, one factory may have 10 sensors, while another factory may have 100. Note that the number of sensors is assumed to be the number of dimensions of the time-series data.

[0016] However, many existing machine learning models have fixed input and output dimensionality, making it difficult to create a foundation model that can handle time-series data with different dimensionality for each domain using such a machine learning model.

[0017] As an alternative, it is possible to fix the number of input dimensions of the model and then increase or decrease the number of dimensions of the actually input time series data.

[0018] For example, when time series data with fewer dimensions than the fixed number of input dimensions is input, the missing series are filled in with values ​​such as 0 or 1, or padded with data from other series. However, the filled series may have some meaning, which may reduce the accuracy of the model.

[0019] Furthermore, for example, if time series data with more dimensions than the fixed number of input dimensions is input, selection is performed on the series, and only the selected appropriate series are actually input to the model. However, in this case, the disadvantages are that the amount of calculation increases and the series selection requires manual work. Furthermore, if the series selection method is inappropriate, the accuracy of the model may decrease.

[0020] Another alternative is to use models that support variable input length, such as Recurrent Neural Networks (RNNs) and Transformers, which are often used in the field of natural language processing.

[0021] In time series data, semantic information is invariant to the order of the series. In other words, as shown in Figure 2, it is desirable that the analysis results using the model do not change even if the series of time series data is swapped. However, the order in the time direction must be maintained. In the example of Figure 2, the series x 2 and series x 3 The order of has been swapped.

[0022] For example, when time-series data is stored in a CSV file, the order of the columns has no chronological meaning and simply indicates the order in which they happened to be stored. For example, if the columns are arranged in the order of temperature, pressure, and flow rate, the meaning of the data is the same as if the columns were arranged in the order of temperature, flow rate, and pressure.

[0023] Models that support variable-length input, such as the aforementioned RNN and Transformer, preserve the order of the input sequence. In other words, the order of the sequence becomes significant and affects the analysis results. For this reason, conventional models that support variable-length input are not suitable as a foundation model for analyzing time series data.

[0024] In the embodiment, a model capable of solving the problems described above is proposed. For example, the model in the embodiment is based on a model that supports variable length input, such as Transformer, and does not attach any significance to the order of input time series data.

[0025] [First embodiment] [Configuration of the first embodiment] The configuration of the information processing device according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the information processing device according to the first embodiment. As shown in Fig. 3, the information processing device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0026] The communication unit 11 performs data communication with other devices via a network. For example, the communication unit 11 is a network interface card (NIC).

[0027] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. Note that the storage unit 12 may also be a data-rewritable semiconductor memory such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM).

[0028] The storage unit 12 stores an OS (Operating System) and various programs executed by the information processing device 10. The storage unit 12 stores model information 121.

[0029] The model information 121 is information such as parameters for constructing a model, etc. For example, the model information 121 is the weights and biases of each layer of a neural network.

[0030] The control unit 13 controls the entire information processing device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0031] The control unit 13 also has an internal memory for storing programs defining various processing procedures and control data, and executes each process using the internal memory. The control unit 13 also functions as various processing units when various programs are run. For example, the control unit 13 has an input control unit 131, a calculation unit 132, a provision unit 133, and an update unit 134.

[0032] The input control unit 131 inputs time-series data to the model constructed from the model information 121. The model constructed from the model information 121 is a base model based on Transformer.

[0033] The calculation unit 132 executes calculations of the model, i.e., the calculation unit 132 obtains the output of the model to which the time series data is input.

[0034] The providing unit 133 provides information based on the calculation results of the model. For example, the providing unit 133 provides predicted values ​​of a series at future times based on collected time series data.

[0035] The update unit 134 updates the model parameters (model information 121) so that the calculation results of the model are optimized.

[0036] For example, when inference using a model is performed, the providing unit 133 operates, and when model learning is performed, the updating unit 134 operates.

[0037] As shown in Figure 8, conventional models such as RNN and Transformer generate a multivariate time series {x t 1 ,x t 2 ,…,x t K} and executes the calculation. Conventional Transformers learn the relationship between time series and variables by repeating this input and calculation for each time (t+1, t+2, ...). Figure 8 explains the conventional data input method.

[0038] As described above, in conventional Transformers, the number of input dimensions in the time direction is variable, but the number of sequences K is fixed, which poses the problem described above.

[0039] As shown in FIG. 4, the input control unit 131 receives a sequence of variables {x t-W 1 ,x t-W+1 1 ,…,x t-W+2 1} can be input to the model. Fig. 4 is a diagram for explaining a data input method. Note that the data input by the input control unit 131 at one time may be data extracted in a fixed window in the time direction.

[0040] In this way, the input control unit 131 inputs variable-length data to the model in the variable direction, not in the time step (time) direction, which allows the input control unit 131 to input time-series data of any number of sequences to the model.

[0041] Furthermore, the model of this embodiment is trained so as not to depend on the order of input sequences. The configuration of the model of this embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram for explaining the configuration of the model.

[0042] As shown in Fig. 5, the model of this embodiment is composed of multiple models. Furthermore, endogenous variables and exogenous variables are input to the model. The endogenous variables and exogenous variables may be rephrased as response variables and explanatory variables, respectively.

[0043] For example, an endogenous variable is a variable to be predicted. That is, a model can predict future endogenous variables from past endogenous and exogenous variables. For example, an exogenous variable is the temperature, pressure, flow rate, etc. measured by sensors installed in a factory. Another example of an endogenous variable is the amount of production in a factory.

[0044] The embedding layer 211, the embedding layer 221, the first model 212, the second model 222, and the third model 230 shown in FIG. 5 may all be neural networks.

[0045] The Embedding layer 211 and the Embedding layer 221 embed the endogenous variables and the exogenous variables, respectively. The Embedding layer 211 embeds the endogenous variables y into a vector z endo Also, Embedding221 transforms the exogenous variables x1, x2, and x3 into a sequence of vectors of a specified length z exo The number of exogenous variables shown in Figure 4 is an example. The number of exogenous variables may be two or more.

[0046] The first model, 212, is endo Based on f(z endo ) is output.endo ) represents the characteristics of the past endogenous variables. f(·) is a function representing the first model 212.

[0047] The second model, 222, is z exo and z exo Based on g(z endo ,z exo ) is output. endo ,z exo ) represents the relationship between endogenous variables and exogenous variables. g(·) is a function representing the second model 222. The second model 222 is a model that supports variable length input, such as an RNN or a Transformer.

[0048] The third model 230 is f(z endo ) and g(z exo ) based on h(f(z endo ),g(z endo ,z exo )) is output. endo ),g(z endo ,z exo )) is the predicted value of the endogenous variable ^y (^ just above y).

[0049] In this way, the calculation unit 132 calculates the first information (f(z endo )) is calculated, and second information (g(z endo ,z exo )) and then use the third model to calculate the predicted value of the endogenous variable (^y) from the first information and the second information.

[0050] During learning, the update unit 134 updates the parameters of each model, i.e., the model information 121, using an index such as the mean square error so that the predicted value ^y approaches the true value (teaching data). The update unit 134 updates the parameters of not only the third model 230 but also the first model 212 and the second model 222.

[0051] where z exocontains z1, z2, and z3, which are the embeddings of variables x1, x2, and x3, respectively. Therefore, g(z endo ,z exo ) is g(z endo ,z1,z2,z3).

[0052] The second model 222 is required to be independent of the input order of the variables x1, x2, and x3. Therefore, the information processing device 10 performs learning using a contrastive learning framework.

[0053] First, the information processing device 10 generates a negative example pair {y, x} (where x={x1, x2, x3}) in addition to the positive example pair {y, x} shown in FIG. other} x other is to convert the elements x1, x2, and x3 of x into another data, x1 other , x2 other , x3 other x1 other , x2 other , x3 other may be randomly generated or may be obtained from other time series data.

[0054] In contrastive learning, semantically similar data are placed close to each other in space (latent space or feature space), and semantically different data are placed far from each other in space.

[0055] Therefore, the information processing device 10 performs learning so that the outputs of the second model 222 when the order of elements x1, x2, and x3 in the positive example pair {y, x} is interchanged become close to each other in space. That is, the meaning of the outputs of the second model 222 does not change even if the order of x1, x2, and x3 is interchanged, so they are reduced to latent (feature) spaces close to each other. Note that the order of x1, x2, and x3 is the same as the order of z1, z2, and z3.

[0056] For example, the update unit 134 updates g(z endo ,z1,z2,z3), g(z endo ,z1,z3,z2), g(zendo ,z2,z1,z3), g(z endo ,z2,z3,z1), g(z endo ,z3,z1,z2), g(z endo The parameters of the second model 222 are updated so that the degree of deviation (for example, Euclidean distance) of two or more of the vectors z1, z2, z3, z4, z5, z6, z7, z8, z9, z10, z11, z12, z13, z14, z15, z16, z17, z18, z19, z20, z21, z22, z23, z24, z25, z26, z27, z28, z29, z30, z31, z21, z22, z23, z24, z25, z26, z27, z28, z29, z31, z32, z22, z13, z22, z22, z14, z22, z22, z23, z24, z25, z26, z27, z28, z29, z32,

[0057] Furthermore, the information processing device 10 outputs the second model 222 for the positive example pair {y, x} and the negative example pair {y, x other}, and the output of the second model 222 are trained so that they are far from each other in space.

[0058] For example, the update unit 134 updates g(z endo ,z1,z2,z3) and g(z endo ,z1 other ,z2 other ,z3 other The parameters of the second model 222 are updated so that the degree of deviation (for example, Euclidean distance) from the first model 222 increases, or the degree of similarity (for example, cosine similarity) decreases.

[0059] L1 is an error index that represents the error from the predicted value of the endogenous variable. L2 is the similarity between the output of the second model 222 when the order of each element in the positive example pair is changed. L3 is the similarity between the output of the second model 222 for the positive example pair {y,x} and the negative example pair {y,x other} and the output of the second model 222. For example, the update unit 134 updates the parameters of the second model 222 so as to minimize the loss function L=αL1-βL2+γL3, where α, β, and γ are coefficients determined in advance.

[0060] As described above, the update unit 134 can use a contrastive loss function or its variants that are generally used in contrastive learning. Furthermore, by adopting the loss function L3 based on negative example pairs, the model can obtain representations that are more robust to input order than when learning using only positive example pairs.

[0061] In this way, the updating unit 134 updates the parameters of the second model 222 so as to reduce the influence on the second information of the order in which the multiple exogenous variables are input to the second model 222. For example, the updating unit 134 updates the parameters of the second model 222 so that the multiple pieces of second information obtained by changing the order in which the multiple exogenous variables are input to the second model 222 become closer to each other (for example, so that L2 becomes larger).

[0062] The calculation unit 132 uses the second model 222 to calculate a negative exogenous variable (x other ) and the first information, calculates fourth information that indicates the relationship between the endogenous variables and the negative endogenous variables. The update unit 134 updates the parameters of the second model 222 so that the plurality of second information and the fourth information become distant from each other (for example, so that L3 becomes smaller).

[0063] [Processing of the first embodiment] Fig. 6 is a flowchart showing the flow of the information processing device according to the first embodiment. As shown in Fig. 6, first, the input control unit 131 acquires time series of one or more endogenous variables and time series of K (K≧2) exogenous variables (step S101). The number of variables is the number of series in the time series data.

[0064] The input control unit 131 inputs the time series of the endogenous variables to the first model 212. Then, the calculation unit 132 executes the calculation of the first model 212 (step S102).

[0065] Next, the input control unit 131 sets k to 1 (step S103). k is an index of the series of exogenous variables input to the model. The maximum value of k is K.

[0066] The input control unit 131 inputs the data (series) of each time of the k-th exogenous variable into the second model 222 (step S104).

[0067] If k < K (step S105, Yes), the input control unit 131 increases k by 1 (step S106), returns to step S104, and repeats the process. Note that k < K means that there is a series of variables that have not been input yet.

[0068] If k ≥ K (step S105, No), the calculation unit 132 executes the calculation of the second model 222 (step S107).

[0069] Then, the calculation unit 132 inputs the calculation results of the first model 212 and the second model 222 into the third model 230 and executes the calculation (step S108).

[0070] Here, the providing unit 133 provides information regarding prediction based on the calculation result of the third model 230 (step S109). For example, the providing unit 133 provides the value of the variable at a future time output by the third model 230. Note that what kind of calculation result the third model 230 outputs can be controlled by the way of learning (for example, the way of giving correct data).

[0071] Note that during learning, instead of step S109, the updating unit 134 updates the parameters of the model, that is, the model information 121.

[0072] Supplement regarding step S104 and related processes. First, the time series which is the input data is data in which the values of each of a plurality of variables are associated with each of a plurality of times as shown in FIGS. 2 and 4 for example. The time series data may be expressed as a matrix with the variable direction as rows and the time direction as columns.

[0073] As shown in Figure 8, in the prior art, variable-direction sequences extracted from time-series data are arranged and input to the model. The variable-direction sequences are t 1 ,x t 2 ,…}, {x t+1 1 ,x t+1 2 ,…} etc.

[0074] 6, the input control unit 131 arranges the time-direction series extracted from the time-series data and inputs them to the embedding layer 221. The time-direction series is x k ={x0 k ,x1 k ,…}, x k+1 ={x0 k+1 ,x1 k+1 , ...} etc. Note that the x input to the embedding layer 221 k , x k+1 , … and z exo z in k , z k+1 , ... is assumed to be in the same order. Therefore, even when the embedding layer 221 is present, the input control unit 131 can be said to indirectly input the exogenous variables to the second model 222.

[0075] In this way, by using the model of this embodiment as a base model, it becomes possible to analyze time series data using the base model.

[0076] The information processing device 10 can train the model of this embodiment using time-series data sampled from a collection of time-series datasets collected from various domains. The model allows variable-length input and is not affected by the order of input variables, so training can be performed on datasets across domains with different numbers of series and different orders of series. Because the trained model is trained using a variety of data, even when an unknown dataset is given, it can make predictions without the need for retraining (zero-shot). Alternatively, the trained model can adapt to an unknown dataset after a few training rounds (few-shot), enabling more accurate predictions.

[0077] [System configuration, etc.] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0078] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0079] [program] In one embodiment, the information processing device 10 can be implemented by installing a learning program that executes the above-described learning process as package software or online software on a desired computer. For example, by executing the above-described learning program on an information processing device, the information processing device can function as the information processing device 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include tablet terminals, smartphones, mobile phones, PHS (Personal Handyphone System) and other mobile communication terminals, as well as slate terminals such as PDAs (Personal Digital Assistants).

[0080] Furthermore, the information processing device 10 can also be implemented as a server that provides services related to the learning process to a client, with the terminal device used by the user being the client.

[0081] 7 is a diagram showing an example of a computer that executes a learning program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0082] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0083] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the information processing device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the information processing device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0084] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0085] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0086] 10. Information processing equipment 11 Communications Department 12 Storage section 13 Control Unit 121 Model Information 131 Input control section 132 Calculation section 133 Provision Department 134 Update Department 211, 221 Embedding layer 212 First Model 222 Second Model 230 Third Model

Claims

1. a calculation unit that calculates first information representing characteristics of an endogenous variable using a first model, calculates second information representing relationships between the endogenous variable and a plurality of exogenous variables from the first information and a plurality of exogenous variables using a second model corresponding to variable length input, and calculates a predicted value of the endogenous variable from the first information and the second information using a third model; an updating unit that updates parameters of the second model so that an influence of an input order of the plurality of exogenous variables to the second model on the second information is reduced; An information processing device comprising:

2. The update unit updates parameters of the second model so that the plurality of pieces of second information obtained by changing the order in which the plurality of exogenous variables are input to the second model become closer to each other.

2. The information processing apparatus according to claim 1, wherein:

3. the calculation unit calculates fourth information representing a relationship between the endogenous variables and the negative extraneous variables from the first information and a negative extraneous variable different from the plurality of exogenous variables, using the second model; The update unit updates parameters of the second model so that the plurality of pieces of second information and the fourth information become farther apart.

2. The information processing apparatus according to claim 1, wherein:

4. An information processing method executed by an information processing device, a calculation step of calculating first information representing characteristics of an endogenous variable using a first model, calculating second information representing a relationship between the endogenous variable and a plurality of exogenous variables from the first information and a plurality of exogenous variables using a second model corresponding to variable length input, and calculating a predicted value of the endogenous variable from the first information and the second information using a third model; an updating step of updating parameters of the second model so that the order in which the plurality of exogenous variables are input into the second model has a small effect on the second information; An information processing method comprising:

5. calculating first information representing characteristics of an endogenous variable using a first model; calculating second information representing a relationship between the endogenous variable and a plurality of exogenous variables from the first information and a plurality of exogenous variables using a second model corresponding to variable length input; and calculating a predicted value of the endogenous variable from the first information and the second information using a third model; The parameters of the second model are updated so that the order in which the plurality of exogenous variables are input into the second model has a small effect on the second information. An information processing program that causes a computer to execute a process.

Images