Information processing device, method for performing time series estimation processing using a neural network, and computer program

Abstract

This technology provides a method for performing time series estimation while taking multiple delays into account. [Solution] The information processing device of the present disclosure includes a memory for storing a neural network having an input layer, an intermediate layer, and an output layer, and a processor for performing time series estimation processing using the neural network. When N is an integer of 1 or more and j is an ordinal number from 1 to N, the input layer has a configuration in which (N+1) delay input layers are arranged in parallel, including a zero-delay input layer that inputs zero-delay data proportional to the time series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, a method for performing time series estimation processing using a neural network, and a computer program. [Background technology]

[0002] One type of time-series estimation problem involves estimating the internal state of a system that cannot be directly observed, such as estimating the temperature distribution of a motor-driven device solely from the motor's current value, or estimating the remaining battery level solely from power consumption. In recent years, neural networks have been used to solve such time-series estimation problems.

[0003] The greatest feature of neural networks lies in their high degree of universality. According to the so-called universal approximation theorem, it is known that a neural network with two or more layers, where the number of parameters per layer is sufficient, can represent any continuous function. In other words, in time series estimation problems, ideally, any system can be represented by using a neural network, no matter how complex the system or framework being studied may be.

[0004] While there are various forms of neural networks, the type commonly used in time series estimation problems is the recurrent neural network (RNN). Recurrent neural networks use the network's output values ​​as inputs for the next step, thereby representing the characteristic of time series changes in general, where past states are incorporated into future states, leading to state changes. Patent Document 1 discloses a method for solving time series estimation problems using RNNs. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-103141

Summary of the Invention

Problems to be Solved by the Invention

[0006] Conventional neural networks such as RNNs can represent a uniform information delay, but cannot represent multiple delays, so there are cases where sufficient high estimation accuracy cannot be obtained. Therefore, a technique capable of performing time series estimation considering multiple delays is desired.

Means for Solving the Problems

[0007] According to the first aspect of the present disclosure, an information processing apparatus is provided. This information processing apparatus includes a memory that stores a neural network having an input layer, an intermediate layer, and an output layer, and a processor that executes a time series estimation process using the neural network. When N is an integer of 1 or more and j is an ordinal number from 1 to N, the input layer includes a zero-delay input layer that inputs zero-delay data proportional to the time series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer. It has a configuration in which (N + 1) delay input layers are arranged in parallel.

[0008] According to the second aspect of the present disclosure, a method for executing a time series estimation process using a neural network having an input layer, an intermediate layer, and an output layer is provided. This method includes a step of inputting time series input data to the input layer and obtaining a time series estimation result output from the output layer. When N is an integer of 1 or more and j is an ordinal number from 1 to N, the input layer includes a zero-delay input layer that inputs zero-delay data proportional to the time series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer. It has a configuration in which (N + 1) delay input layers are arranged in parallel.

[0009] A third embodiment of this disclosure provides a computer program that causes a processor to perform time series estimation processing using a neural network having an input layer, an intermediate layer, and an output layer. In this computer program, when N is an integer of 1 or more and j is an ordinal number from 1 to N, the input layer has a configuration in which (N+1) delay input layers are arranged in parallel, including a zero-delay input layer that inputs zero-delay data proportional to the time series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram of the information processing device in the embodiment. [Figure 2] An explanatory diagram showing the temperature circuit that will be used for time series estimation. [Figure 3] A timing chart showing the estimation results in the comparative example. [Figure 4] A diagram illustrating the method for calculating j-weighted delayed data. [Figure 5] A graph showing a specific example of j-delay data. [Figure 6] A diagram illustrating the configuration of a neural network that takes a single time-series data point as input. [Figure 7] A diagram illustrating the configuration of a neural network that takes multiple time-series input data as input. [Figure 8] A timing chart showing the estimation results in the embodiment. [Modes for carrying out the invention]

[0011] Figure 1 is a block diagram showing the functions of the information processing device 100 in an embodiment. The information processing device 100 includes a processor 110, a memory 120, an interface circuit 130, and an input device 140 and a display device 150 connected to the interface circuit 130. For example, the processor 110 not only has the function of executing the processing detailed below, but also has the function of displaying the data obtained by the processing and the data generated in the process of the processing on the display device 150. The information processing device 100 can be implemented by a computer such as a personal computer.

[0012] The processor 110 implements the function of the estimation unit 112, which performs time series estimation processing using the neural network 200. The function of the estimation unit 112 is realized by the processor 110 executing a computer program stored in the memory 120. However, the functions of each of these parts may also be realized by hardware circuits. The term "processor" as used herein includes such hardware circuits. The memory 120 stores the neural network 200 and the time series input data TID.

[0013] Figure 2 is an explanatory diagram showing the temperature circuit 400 that is the subject of time series estimation in this embodiment. This temperature circuit 400 has four nodes N0, N1, N2, and N3. Heat quantity Qa is input to the first node N0 from a heat source, and this heat propagates to each node N0 to N3, so that the temperature of each node N0 to N3 becomes T0 to T3. There is a thermal resistance Rjj between adjacent nodes, and the way heat is transferred is not uniform. The subscripts j,j in the thermal resistance Rjj are numbers that indicate two adjacent nodes. Node N3 is the outside environment. The temperature of the outside environment T3=Tc is, for example, 300[K].

[0014] Of the various values ​​related to the temperature circuit 400, only the heat quantity Qa from the heat source and the ambient temperature Tc are observable; the thermal resistance Rjj cannot be determined. Time series estimation is a process that estimates the temperatures T0 to T3 of each node N0 to N3 from the observed heat quantity Qa.

[0015] Figure 3 is a timing chart showing the estimation results in the comparative example. The vertical axis of the five graphs represents heat quantity Qa [J] and temperature T0~T3 [K], and the horizontal axis represents time [sec]. The solid line shows the values ​​obtained from the simulation of temperature circuit 400, and the dashed line shows the estimated values ​​obtained using a neural network called a reservoir computer (RC). The reservoir computer in the comparative example was configured to take heat quantity Qa as input and output temperature T0~T3.

[0016] The reservoir computer completed pre-training using training data before obtaining the estimation results shown in Figure 3. The training data was prepared as follows: First, each thermal resistance Rjj was fixed to an arbitrary value, and the temperatures T0 to T3 of each node were calculated by simulation for randomly generated heat Qa, and these values ​​Qa,T0 to T3 were used as training data. However, in the training data, T3=Tc=300K was assumed. The calculation was performed at a rate of one sample per second, and the length of one data set was approximately 700 seconds. Forty patterns of such data were prepared and trained, and the time series estimation was performed using evaluation data from a different pattern than the training data, which is shown in Figure 3.

[0017] As seen in Figure 3, the difference between the simulation results and the estimated values ​​increases in the order of temperatures T0, T1, and T2, and the estimation accuracy worsens as the distance from the heat source increases. Since temperature changes are the result of heat propagation, it is natural that temperature changes occur with a delay as you move away from the heat source. From the perspective of estimating the temperature of each node based on the heat quantity Qa input from the heat source, this means that nodes farther away from the heat source must use more past information, i.e., more delayed information, to estimate the temperature compared to nodes closer to the heat source. In recurrent neural networks such as RNNs and RCs, increasing the latency of the information is equivalent to increasing the recursiveness of the network. Therefore, the poor estimation accuracy of nodes farther away from the heat source means that the recursiveness of the network used for verification was not sufficiently high.

[0018] To improve the accuracy of temperature estimation for nodes far from the heat source, one might consider increasing the recursiveness of the network. However, increasing the recursiveness of the network leads to a problem where the accuracy of temperature estimation for nodes close to the heat source deteriorates. In other words, the degree of information delay required for temperature estimation depends on the distance from the heat source. Therefore, existing neural networks that only represent a uniform information delay cannot solve this problem.

[0019] Therefore, in this embodiment, a neural network with temporal multiplexing is constructed by configuring the input layer to input a parallel sequence of multiple delay data, which is obtained by delaying time-series input data multiple times, into the hidden layer. A "multiple delay data sequence" is a sequence of delay data that has been delayed multiple times, such as single delay data obtained by delaying the time-series input data, double delay data obtained by delaying the single delay data further, triple delay data obtained by delaying the double delay data further, and so on. These sequences are arranged in parallel and input into the hidden layer. The time-series input data is also input into the hidden layer as zero-delay data.

[0020] In this way, by using a multiple-delay data sequence in which time-series input data is delayed multiple times, multiple time information can be reproduced, and furthermore, by parallelizing these, the multiple time information can be replaced with a spatial representation.

[0021] In this embodiment, a set of time-series input data X is represented as follows:

number

[0022] When j is an integer representing the number of delays, the j-delayed data X (j) This can be expressed as follows: [Number]

[0023] Each element x on the right side of equation (q2) t (j) is the data value at time t of the j-fold delayed data X (j) and is calculated according to equations (q3) to (q5). The x on the right side of equation (q4) t is the data value at time t of the time-series input data X. c0 is a constant, for example, c0 = 1. That is, the 0-fold delayed data X (0) is calculated to be proportional to the time-series input data X. Equation (q5) represents the weighting coefficient k j of equation (q3), and α is a fixed delay degree. k0 in equation (q5) is a constant, for example, k0 = 1. According to equation (q5), the weighting coefficient k (j) for the j-fold delayed data X j is proportional to the value α to the power of j of the delay degree α j . It is preferable to set the delay degree α such that 0 < α < 1. Also, the data value x0 t (j) at time t = 0, which is the initial value of x (j) is preferably set to 0.

[0024] According to the above equation (q5), the weighting coefficient k j is calculated by multiplying by a constant delay degree α every time the delay advances by one. In this way, as the delay advances, the weighting coefficient k j decreases, so the influence of past data can be reduced. However, a constant value independent of j may be used as the weighting coefficient k j , or the weighting coefficient k j may be increased as the delay advances. However, if the weighting coefficient k j is set to decrease as the delay advances, it is preferable in that the time-series estimation process can be executed with higher precision. In any case, the weighting coefficient k j is preferably such that 0 < k j < 1 is satisfied.

[0025] According to equation (q3), the j-fold delayed data X(j) The data value x at each time step t. t (j) This is the j-delayed data X (j) The data value x at the previous time (t-1) t-1 (j) weight coefficient k j The first component multiplied by and (j-1) heavily delayed data X (j-1) The data value x at time t. t (j-1) (1-k j It is calculated as the sum of the second component obtained by multiplying by ) and .

[0026] Figure 4 is an explanatory diagram of the method for calculating j-weighted delayed data. Here, for j=0,1,2, the data values ​​x at times t=0, t=1, and t=2 are shown. t (j) This is shown. The data value x1 at time t=1 in single-delay data. (1) This is the data value x0 at the previous time t=0 in single-delay data. (1) The first component obtained by multiplying by the weight coefficient k1, and the data value x1 at time t=1 in the zero-delay data. (0) It is calculated as the sum of the second component obtained by multiplying by (1-k1) and .

[0027] Figure 5 shows the j-delay data X. (j) This graph shows a concrete example. Here, the zero-delay data X (0) As such, time-series input data that changes in a rectangular wave pattern is used. In this case, single-delay data X (1) This is zero-delay data X (0) This corresponds to data with a first-order lag added. Also, j-delay data X (j) This shows that the order of the delay increases as the number of delays j increases. It also shows that the influence of past data decreases as the delay progresses.

[0028] Figure 6 is an explanatory diagram showing the configuration of the neural network 200 in the embodiment. This neural network 200 takes one time-series input data X as input and has an input layer 210, an intermediate layer 220, and an output layer 230.

[0029] The input layer 210 receives the time-series input data TID(=X) and processes the multiple delayed data sequence X (0) ,X (1) ...is configured to be input to the intermediate layer 220. Multiple delayed data sequence X (0) ,X (1) ...is the j-weighted delayed data X given by equations (q2) to (q5) above. (j) These are arranged in parallel.

[0030] The input layer 210 generates zero-delay data X proportional to the time-series input data X. (0) The zero-delay input layer 210_0 inputs the data to the intermediate layer 220, and the zero-delay data X (0) Single-delayed data X, which is the same data as X, but delayed once. (1) A single-delay input layer 210_1 inputs the single-delay data X to the intermediate layer 220, and the single-delay data X (1) Double-delayed data X, which is the same data but delayed by another step. (2) It includes a double-delay input layer 210_2 that inputs to the intermediate layer 220. Generally, when N is an integer of 1 or more, the input layer 210 has a configuration in which (N+1) delay input layers 210_0 to 210_N are arranged in parallel.

[0031] The value at each time t in the single-delay input layer 210_1 is calculated by adding the result of multiplying the value at the previous time (t-1), which is stored in memory, by a weight coefficient k1, and the result of multiplying the value at time t provided by the zero-delay input layer 210_0 by (1-k1). The sum is then stored in memory, and the value is passed to the intermediate layer 220. This process corresponds to the operation performed according to equation (q3) described above. The same applies to the other delay input layers from the double-delay input layer 210_2 onward.

[0032] As shown in Figure 6, when only one set of time-series input data X is input to the neural network 200, each of the delayed input layers 210_0 to 210_N consists of one node.

[0033] The constant c0, the delay α, and the constant k0 used in equations (q2) to (q5) above are all fixed values ​​and are set as hyperparameters.

[0034] The hidden layer 220 can be any configuration of hidden layer. For example, the hidden layer 220 may have the structure of a non-recurrent hidden layer in a typical hierarchical neural network, or it may have the structure of a recurrent hidden layer used in recurrent neural networks or reservoir computers.

[0035] The output layer 230 is configured to output one or more outputs y1, y2…. When applied to the time series estimation problem of the temperature circuit 400 shown in Figure 2, the outputs y1, y2… are the temperatures T0 to T3 of each node. The time series input data X provided to the input layer 210 is the time change of the heat quantity Qa.

[0036] Figure 7 is an explanatory diagram showing the configuration of a neural network 300 that takes multiple time-series input data as input. The input layer 310, hidden layer 320, and output layer 330 correspond to the input layer 210, hidden layer 220, and output layer 230 of the neural network 200 shown in Figure 6, respectively. The input layer 310 is configured to include N+1 delayed input layers 310_0 to 310_N.

[0037] In the example in Figure 7, there are m time-series input data X1~X m The following is input to neural network 300. m is an integer greater than or equal to 2. When i is an ordinal number from 1 to m, the i-th time-series input data X i This can be expressed as follows:

number

[0038] i-th time-series input data X i j heavy delay data X i (j) This can be expressed as follows:

number

[0039] As shown in Figure 7, the neural network 300 receives m time-series input data X1~X m When the input is such that each of the delayed input layers 310_0 to 310_N is configured to contain m nodes.

[0040] Figure 8 is a timing chart showing the estimation results in the embodiment. The solid line shows the values ​​obtained from the simulation of the temperature circuit 400, and the dashed line shows the estimated values ​​estimated by the estimation calculation unit 112 using the neural network 200 shown in Figure 6. Compared with the estimation results of the comparative example shown in Figure 3, it can be seen that in this embodiment, the temperatures T0 to T3 of all nodes are accurately estimated, and the estimation accuracy has improved.

[0041] As described above, in this embodiment, multiple delayed data sequences obtained by delaying the time-series input data TID multiple times are parallelized and input from the input layer 210 to the hidden layer 220, so that time-series estimation processing can be performed using a neural network 200 that has temporal multiplexing of information.

[0042] Other embodiments: This disclosure is not limited to the embodiments described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms (aspects). The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure, or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate.

[0043] (1) According to a first embodiment of the present disclosure, an information processing device is provided. This information processing device includes a memory for storing a neural network having an input layer, an intermediate layer and an output layer, and a processor for performing time series estimation processing using the neural network. When N is an integer of 1 or more and j is an ordinal number from 1 to N, the input layer has a configuration in which (N+1) delay input layers are arranged in parallel, including a zero-delay input layer that inputs zero-delay data proportional to the time series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer. According to this information processing device, multiple delayed data sequences, obtained by delaying time-series input data multiple times, are parallelized and input from the input layer to the hidden layer. Therefore, time-series estimation processing can be performed using a neural network that has temporal multiplexing of information.

[0044] (2) In the above information processing device, k j When the weight coefficient is greater than 0 and less than 1, the data value at each time t in the j-weighted delayed data is the data value at the previous time (t-1) in the j-weighted delayed data multiplied by the weight coefficient k j The first component obtained by multiplying by (1-k) and the data value at time t in the (j-1) heavily delayed data. j It may also be considered as the sum of the second component obtained by multiplying by ) and . This information processing device allows for the easy calculation of j-delay data.

[0045] (3) In the above information processing device, when α is set to a delay degree greater than 0 and less than 1, the weight coefficient k in the j-weighted delay data j This is the value of the delay degree α raised to the power of j, α j It may also be considered to be proportional to [the given value]. According to this information processing device, the greater the delay, the less influence past data can be had.

[0046] (4) A second embodiment of the present disclosure provides a method for performing time series estimation using a neural network having an input layer, an intermediate layer, and an output layer. This method comprises the steps of inputting time series input data into the input layer and obtaining a time series estimation result output from the output layer. When N is an integer of 1 or more and j is an ordinal number from 1 to N, the input layer has a configuration in which (N+1) delay input layers are arranged in parallel, including a zero-delay input layer that inputs zero-delay data proportional to the time series input data into the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data into the intermediate layer. This method involves parallelizing multiple delayed data sequences, obtained by delaying time-series input data multiple times, and inputting these parallelized sequences from the input layer to the hidden layer. This allows for time-series estimation processing using a neural network that possesses temporal multiplexing of information.

[0047] (5) According to a third embodiment of the present disclosure, a computer program is provided that causes a processor to perform time series estimation processing using a neural network having an input layer, an intermediate layer and an output layer. When N is an integer of 1 or more and j is an ordinal number from 1 to N, the input layer has a configuration in which (N+1) delay input layers are arranged in parallel, including a zero-delay input layer that inputs zero-delay data proportional to the time series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer. According to this computer program, multiple delayed data sequences, each obtained by delaying time-series input data multiple times, are parallelized and input from the input layer to the hidden layer. This allows for time-series estimation processing using a neural network that possesses temporal multiplexing of information.

[0048] This disclosure can also be implemented in various forms other than those described above. For example, it can be implemented in the form of an information processing device, a computer program to implement its functions, or a non-transit storage medium on which the computer program is recorded. [Explanation of Symbols]

[0049] 100...Information processing unit, 110...Processor, 112...Estimation calculation unit, 120...Memory, 130...Interface circuit, 140...Input device, 150...Display device, 200...Neural network, 210...Input layer, 210_0...0-delay input layer, 210_1...1-delay input layer, 210_2...Double-delay input layer, 220...Hidden layer, 230...Output layer, 300...Neural network, 310...Input layer, 310_0...0-delay input layer, 310_1...1-delay input layer, 310_2...Double-delay input layer, 320...Hidden layer, 330...Output layer, 400...Temperature circuit

Claims

1. An information processing device, A memory that stores a neural network having an input layer, a hidden layer, and an output layer, A processor that performs time series estimation processing using the aforementioned neural network, Equipped with, Let N be an integer greater than or equal to 1, and let j be an ordinal number from 1 to N. The information processing apparatus has an input layer having a configuration in which (N+1) delay input layers are arranged in parallel, including a zero-delay input layer that inputs zero-delay data proportional to time-series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer.

2. An information processing apparatus according to claim 1, k ｊ When we set the weighting coefficient to be greater than 0 and less than 1, The data value at each time t in the aforementioned j-weighted delay data is obtained by multiplying the data value at the previous time (t-1) in the j-weighted delay data by the weight coefficient k. ｊ The first component obtained by multiplying by (1-k) and the data value at time t in the (j-1) heavily delayed data. ｊ An information processing device whose second component is obtained by multiplying by ) and is the sum of .

3. An information processing apparatus according to claim 2, When α is defined as a delay degree greater than 0 and less than 1, The weighting coefficient k in the aforementioned j-weighted delay data ｊ This is the value of the delay degree α raised to the power of j, α ｊ An information processing device that is proportional to [a certain value].

4. A method for performing time series estimation using a neural network having an input layer, a hidden layer, and an output layer, The process includes inputting time-series input data into the input layer and obtaining time-series estimation results output from the output layer, Let N be an integer greater than or equal to 1, and let j be an ordinal number from 1 to N. The method comprises an input layer having a configuration in which (N+1) delay input layers are arranged in parallel, including a zero-delay input layer that inputs zero-delay data proportional to the time-series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data obtained by sequentially delaying the zero-delay data to the intermediate layer.

5. A computer program that causes a processor to perform time series estimation processing using a neural network having an input layer, a hidden layer, and an output layer, Let N be an integer greater than or equal to 1, and let j be an ordinal number from 1 to N. A computer program having an input layer configured such that (N+1) delay input layers are arranged in parallel, each including a zero-delay input layer that inputs zero-delay data proportional to time-series input data to the intermediate layer, and a j-delay input layer that inputs j-delay data, obtained by sequentially delaying the zero-delay data, to the intermediate layer.

Images