Learning device, estimation device, their methods, and programs

Abstract

Provided are a training device and the like that train a gated RNN so as to perform information processing utilizing the correlation between data at separate times in long series data without increasing the amount of memory and time required for calculation processing as much as prior art. The training device includes a gate calculation unit that obtains a gate zt,S at time t using the data xt,n,S at time t and the state vector ht-1,S at time t included in series data for training. The gate calculation unit uses an activation function γ that double-exponentially converges to 0 or 1 with respect to the input when obtaining the gate zt,S.

Description

[Technical Field]

[0001] The present invention relates to a learning device that trains a gated recurrent neural network (hereinafter also referred to as "RNN") used in sequence data analysis, an estimation device that performs estimation using the trained gated recurrent neural network, and methods and programs for these. [Background technology]

[0002] Sequence data analysis is a fundamental technology applied in many technical fields, such as time series prediction, speech recognition, and natural language processing. In particular, sequence data analysis using RNN, a type of machine learning model, is an important technology because it can perform accurate inference even on high-dimensional, complex sequence data without cumbersome data preprocessing procedures.

[0003] Among RNNs, gated RNNs, which incorporate a computational structure called a gate mechanism, are widely used in practical applications. Specific examples of gated RNNs include Long Short Term Memory (hereinafter referred to as "LSTM"; see Non-Patent Document 1, etc.) and Gated Recurrent Unit (hereinafter referred to as "GRU"; see Non-Patent Document 2, etc.).

[0004] The computational structure of the gate mechanism is explained below. First, RNNs generally have a state vector inside that represents the features of sequential data, and are structured to calculate and update a new state vector from the input data at a certain point in time and the state vector at that time. In gated RNNs, this update formula is defined as a component-by-component convex combination of the state vector at that point in time and another vector (hereafter also referred to as the update term) obtained by appropriate calculation using the input data at that point in time. The coefficient vector in this convex combination, which takes a value between 0 and 1, is called a gate. The state vector at a certain point in time t is defined as h t-1 The data x at the input time t t Using the appropriate calculation, we can obtain another vector -If h is the gate and z is the gate, the update formula for the state vector is

number

number

[0005] When the gate z takes a value close to 0, the state vector hardly changes due to the state update, and when the gate z takes a value close to 1, the state vector changes to a different vector. - h. Because the gates are calculated based on the state vector and input data at each point in time, this computational structure allows gated RNNs to dynamically learn to retain or replace information. In other words, gated RNNs can dynamically learn the appropriate rate of change of the state vector. This is important for RNNs to learn long-term dependencies in sequential data. In other words, to utilize information from data at a certain point in time in processing data at a later, distant point in time, the gates take values ​​close to 0 at the intervening points in time, preventing the corresponding part of the state vector from changing significantly, allowing the RNN to learn long-term dependencies. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] S. Hochreiter, J. Shcmidhuber, "Long Short-Term Memory", Neural Computation, 9(8), 1735-1780, 1997. [Non-patent document 2] K. Cho, et al. "Learning Representations using RNN Encoder-decoder for Statistical Machine Translation", arXiv preprint arXiv:1406.1078, 2014. Summary of the Invention [Problem to be solved by the invention]

[0007] However, there is a problem in that it is difficult to train RNN gates to take values ​​very close to 0 or 1. This is due to the following factors.

[0008] RNN training is performed using a gradient-based method. In other words, the gradient of a preset error function is calculated for the RNN parameters, and the parameters are repeatedly updated based on this. During this training, as the gate value approaches 0 or 1, the gradient for the parameters inside the gate approaches 0. As a result, the parameters cannot be updated sufficiently to further bring the gate closer to 0 or 1. Due to this issue, gated RNNs cannot retain information about input data over more than several thousand time periods, resulting in a decrease in accuracy when processing data that is correlated with data from distant time periods.

[0009] Existing techniques for training gates to take values ​​close to 0 or 1 include adding an independent gate function to force the gate's value to approach 0 or 1 (see Reference 1).

[0010] (Reference 1) A. Gu, et al. "Improving the Gating Mechanism of Recurrent Neural Networks", ICML pp.3800-3809, 2020. However, this conventional technique has the drawback of increasing the number of parameters and operations required for gate calculation, thereby increasing the amount of memory and time required for calculation processing.

[0011] The present invention aims to provide a learning device that trains a gated RNN to perform information processing that utilizes correlations between data at distant times in long series of data, without increasing the amount of memory and time required for calculation processing to a degree comparable to conventional techniques, an estimation device that uses the trained gated RNN, and methods and programs for these. [Means for solving the problem]

[0012] In order to solve the above problem, according to one aspect of the present invention, a learning device learns a gated recurrent neural network. The learning device learns data x at time t included in learning sequence data. t,n,S and the state vector h at time t t-1,S and gate z at time t t,S The gate calculation unit calculates the gate z t,S When calculating the activation function γ, we use an activation function γ that converges to 0 or 1 doubly exponentially with respect to the input. [Effects of the Invention]

[0013] According to the present invention, it is possible to perform information processing that utilizes the correlation of data at distant times in long sequence data, without increasing the amount of memory and time required for calculation processing as much as in the prior art. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 10 is a diagram showing the effect of the first embodiment. [Figure 2] FIG. 10 is a diagram showing the effect of the first embodiment. [Figure 3] FIG. 1 is a diagram showing an example of the configuration of an estimation system according to a first embodiment. [Figure 4] FIG. 1 is a functional block diagram of a learning device according to a first embodiment. [Figure 5] FIG. 4 is a diagram showing an example of a processing flow of the learning device according to the first embodiment. [Figure 6] FIG. 1 is a functional block diagram of an estimation device according to a first embodiment. [Figure 7]FIG. 3 is a diagram showing an example of a processing flow of the estimation device according to the first embodiment. [Figure 8] 10A and 10B are diagrams showing learning times of a learning device according to a conventional technique and a learning device according to a first embodiment. [Figure 9] FIG. 1 is a diagram showing an example of the configuration of a computer to which the present technique is applied. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described. In the drawings used in the following description, components having the same functions and steps performing the same processes will be denoted by the same reference numerals, and duplicated explanations will be omitted. In the following description, the symbols " - " etc. should normally be written directly above the character immediately following it, but due to limitations in text notation, they are written immediately before the character in question. In formulas, these symbols are written in their original positions. Furthermore, unless otherwise specified, processing performed on each element of a vector or matrix is ​​assumed to apply to all elements of that vector or matrix.

[0016] <Key Points of the First Embodiment> In this embodiment, an auxiliary function is applied before the application of the activation function used in the gate calculation, and the calculation structure of the gate mechanism is modified so that the gate takes a value very close to 0 or 1. Based on the modified calculation structure, a gated RNN is trained, and the same calculation structure of the gate mechanism is also used during estimation.

[0017] By training the gates of a gated RNN to output values ​​that are very close to 0 or 1, the trained gated RNN can be used to perform highly accurate inference on sequence data that has correlation between inputs at different times.

[0018] In this embodiment, when updating parameters during learning, the gradient of the parameters related to the gate is prevented from approaching extremely close to 0 even when the gate takes a value close to 0 or 1. This configuration makes it possible to learn to reduce the time change of the state vector of the gated RNN, and it is possible to appropriately process input correlations and perform inference with high accuracy.

[0019] Before describing this embodiment, the computational structure of gates in gated RNNs such as LSTM and GRU will be described.

[0020] <Gated RNN> For simplicity, we will explain the gate in GRU. Let t be the index representing time, x be the input data at time t, and R be the set of all real numbers. Let h be the state vector of GRU at time t. t ∈R n , the state update equation is h t =F GRU (h t-1 ,x t ) and then,

number

number

number

number

number

number

[0021] The training of a GRU, or more generally an RNN, is carried out as follows: First, a training dataset consisting of a set of input sequence data and desired output values, and a corresponding loss function L are prepared. The state vector h obtained by processing the training data input with the GRU is t The gradient of the loss function L for each parameter of the GRU is calculated for the output calculated using a linear transformation or other method and the loss function L relating to the desired output value. The obtained gradient value is used to update the parameters using an optimization method such as a gradient method. Learning is performed by repeating the above steps.

[0022] Here, the parameters related to the update gate, for example, parameter b z The gradient of the loss function L with respect to is calculated as follows:

number

[0023] <Gated RNN of this embodiment> Next, we will explain this embodiment. In this embodiment, the gates are trained to take values ​​very close to 0 or 1, and the trained gated RNN can perform accurate inference on sequence data that has correlation with data at distant times.

[0024] In the gated RNN of this embodiment, the activation function used in the gate calculation is replaced with a function that has a large gradient even when the gate takes a value close to 0 or 1. This realizes a gated RNN that makes it easier for the gate to learn values ​​close to 0 or 1.

[0025] The reason why it is difficult to train a gate to take values ​​close enough to 0 or 1 is that the closer the gate value is to 0 or 1, the more the gradient with respect to the parameters associated with the gate

number

[0026] FIG. 4 is a functional block diagram of the learning device 100 according to the first embodiment, and FIG. 5 shows the processing flow thereof.

[0027] The learning device 100 includes a preprocessing unit 110 , an update term calculation unit 120 , a gate calculation unit 130 , a state update calculation unit 140 , an output calculation unit 150 , and a parameter update unit 160 .

[0028] The learning device 100 uses N pieces of sequence data X S =(X 1,S ,X 2,S ,…,X N,S ) and the corresponding N desired output values ​​Y S =(Y 1,S ,Y 2,S ,…,Y N,S ) training dataset (X S ,Y S ) as input, train a gated RNN, and output the trained gated RNN parameters P. Here, N is the total number of sequence data included in the training dataset, n = 1, 2, ..., N, and Y n,S is the nth series data X n,S The subscript S is a symbol indicating that the data is for training.

[0029] FIG. 6 is a functional block diagram of an estimation device 200 according to the first embodiment, and FIG. 7 shows the processing flow thereof.

[0030] The estimation device 200 includes a preprocessing unit 210 , an update term calculation unit 220 , a gate calculation unit 230 , a state update calculation unit 240 , and an output calculation unit 250 .

[0031] Prior to the estimation process, the estimation device 200 receives the trained gated RNN parameters P. The estimation device 200 receives the sequence data X to be estimated as input, performs estimation on the sequence data X, and outputs the estimation result Y.

[0032] The learning device 100 and the estimation device 200 are special devices configured by loading a special program into a publicly known or dedicated computer having, for example, a central processing unit (CPU), a main memory (RAM), etc. The learning device 100 and the estimation device 200 execute each process under the control of, for example, the central processing unit. Data input to the learning device 100 and the estimation device 200 and data obtained in each process are stored in, for example, the main memory, and the data stored in the main memory is read by the central processing unit as needed and used for other processes. At least a portion of each processing unit in the learning device 100 and the estimation device 200 may be configured using hardware such as an integrated circuit. Each storage unit included in the learning device 100 and the estimation device 200 can be configured using, for example, a main memory such as a random access memory (RAM), or middleware such as a relational database or a key-value store. However, each storage unit does not necessarily need to be provided inside the learning device 100 and the estimation device 200, but may be configured as an auxiliary storage device made up of a semiconductor memory element such as a hard disk, optical disk, or flash memory, and may be configured to be provided outside the learning device 100 and the estimation device 200.

[0033] <Learning device 100> First, each unit of the learning device 100 will be described.

[0034] <Pre-processing unit 110> The preprocessing unit 110 processes N pieces of sequence data X S =(X 1,S ,X 2,S ,…,X N,S ) is input and converted into a form such as a vector that can be processed by the gated RNN (S110), and the converted sequence data X' S =(X' 1,S ,X' 2,S ,…,X' N,S ) is output. n,S =(x 1,n,S ,x2,n,S ,…,x T_n,n,S ,) where t=1,2,...,T_n, and T_n is the nth sequence data X' n,S is the total number of data contained in x t,n,S is the t-th data included in the n-th series data. The output data x t,n,S are sequentially input to the gated RNN. Note that the update term calculation unit 120, gate calculation unit 130, state update calculation unit 140, and output calculation unit 150 correspond to the gated RNN.

[0035] A gated RNN generates a state vector h at each time point t. t-1 and the input data x t,n,S Two processes are performed independently on the data x t,n,S This is a calculation process for extracting information on the gate z, and is called the calculation of the update term. The other is the calculation process of the gate z. Normally, this calculation applies the activation function σ after a linear transformation, but in this embodiment, an auxiliary function is applied during this process. This will be explained in detail in the section <Gate calculation unit 130> below.

[0036] <Update item calculation unit 120> The update term calculation unit 120 calculates the data x t,n,S and the state vector h t-1,S and parameter U r ,U h ∈R n×n , W r ,W h ∈R n×d , B r ,B h ∈R n Using the following equation, the update term - h t,S is calculated (S120) and output.

number

[0037] <Gate calculation unit 130> The gate calculation unit 130 calculates the data x t,n,S and the state vector h t-1,S and parameter U z ∈R n×n , W z ∈R n×d , B z ∈R n and an activation function γ that converges to 0 or 1 doubly exponentially with respect to the input, we calculate the gate z by the following equation. t,S is calculated (S130) and output.

number

[0038] For example, the gate calculation unit 130 includes a linear transformation calculation unit 131 , an auxiliary function calculation unit 133 , and an activation function calculation unit 135 .

[0039] The linear transformation calculation unit 131 calculates the data x t,n,S and the state vector h t-1,S and parameter U z ∈R n×n , Wz ∈R n×d , B z ∈R n Using this, the linear transformation of the following equation is calculated (S131), and the transformation result J t Output.

number

[0040] The auxiliary function calculation unit 133 calculates the conversion result J t,S is used as input, and the auxiliary function α(J t,S ) is calculated (S133), and the calculation result K t,S =α(J t,S ) is output. As the auxiliary function α(), for example, a hyperbolic sine function sinh() can be used. The auxiliary function α is a function that satisfies the following property (i). (i) For large positive or negative x, α(x)>>x Furthermore, the auxiliary function α may satisfy the following properties (ii) and (iii) so that its behavior does not deviate significantly from that of conventional activation functions. (ii) α'(0)=0 (the derivative at the origin is 0) (iii) α(-x)=-α(x) (α is an odd function)

[0041] The activation function calculation unit 135 calculates the value K of the auxiliary function. t,S is used as input, and the activation function β(K t,S ) (S135), and the calculation result is passed to gate z t,S Therefore, z t,S =β(K t,S ) As the activation function β(), for example, a sigmoid function σ() can be used.

[0042] <Status Update Calculation Unit 140> The state update calculation unit 140 calculates the update term - h t,Sand Gate Z t,S is input, and the gate z t,S Using the following equation, the update term - h t,S and the state vector h t-1,S and calculate the convex combination of the state vector h at time t t,S is calculated (S140) and output.

number

[0043] The above processes S120 to S140 are performed on T_n pieces of data x 1,n,S ,…,x T_n,n,S This is repeated (S141).

[0044] The state update calculation unit 140 calculates the data x 1,n,S ,…,x T_n-1,n,S The state vector h 1,S ,…,h T_n-1,S to the update term calculation unit 120 and the gate calculation unit 130, and the data x T_n,n,S The state vector h corresponding to T_n,S is output to the output calculation unit 150. <Output calculation unit 150> The output calculation unit 150 calculates the state vector h T_n,S is input, and the state vector h T_n,S (S150) calculates an estimate of the desired output using n,S Output.

[0045] <Parameter update unit 160> The parameter update unit 160 calculates the desired output value Y n,S and the calculation result Y' n,Sand are used as inputs, and for the loss function L of these values, each parameter U z ,U r ,U h ∈R n×n , W z ,W r ,W h ∈R n×d , B z ,B r ,B h ∈R n The parameter update unit 160 uses the calculated gradient value to update the parameters by an optimization method such as a gradient method (S160), and updates the updated parameters P′=(U z ,U r ,U h ,W z ,W r ,W h ,B z ,B r ,B h ) is output to each part.

[0046] The parameter update unit 160 controls each unit to repeat the above-mentioned processes S120 to S160 until a predetermined condition is met, and when the predetermined condition is met (YES in S161), outputs the parameters at that time as the trained gated RNN parameters P. The predetermined condition is a condition for determining whether the learning of the gated RNN has converged, and may be, for example, that the number of repetitions exceeds a predetermined number, or that the difference between the parameter P before and after the update is equal to or less than a predetermined value.

[0047] In addition, the training dataset (X S ,Y S ) may be used for training, or a portion of it may be used for training. The same training data may be used only once or repeatedly.

[0048] <Estimation device 200> Next, the estimation device 200 will be described with reference to FIGS. 6 and 7. The estimation device 200 does not include a parameter update unit 160 and uses a training data set (X S ,Y S), it receives sequence data X to be estimated and outputs an estimation result Y. The processing of each unit will be described below. However, the processing method of each unit is the same as that of each unit in learning device 100.

[0049] <Preprocessing unit 210> The pre-processing unit 210 receives the sequence data X to be estimated, converts it into a form such as a vector that can be processed by the gated RNN (S210), and outputs the converted sequence data X'. The same conversion process as the pre-processing unit 110 can be performed. X'=(x1, x2, ..., x T ,) where t=1,2,…,T, T is the total number of data included in the sequence data X', and x t is the t-th data included in the sequence data X'. The output data x t are sequentially input to the gated RNN.

[0050] <Update item calculation unit 220> The update term calculation unit 220 calculates the data x t and the state vector h t-1 and are input, and the parameter U included in the trained parameter P is r ,U h ∈R n×n , W r ,W h ∈R n×d , B r ,B h ∈R n Using the following equation, the update term - h t is calculated (S220) and output.

number

[0051] <Gate calculation unit 230> The gate calculation unit 230 calculates the data x t and the state vector h t-1 and are input, and the gate z is t is calculated (S230) and output.

number

[0052] For example, the gate calculation unit 230 includes a linear transformation calculation unit 231 , an auxiliary function calculation unit 233 , and an activation function calculation unit 235 .

[0053] The linear transformation calculation unit 231 calculates the data x t and the state vector h t-1 and are input, and the parameter U included in the trained parameter P is z ∈R n×n , W z ∈R n×d , B z ∈R n Using this, the linear transformation of the following equation is calculated (S231), and the transformation result J t Output.

number

[0054] The auxiliary function calculation unit 233 calculates the conversion result J t is used as input, and the auxiliary function α(J t ) is calculated (S233), and the calculation result K t =α(J t ) is output.

[0055] The activation function calculation unit 235 calculates the value of the auxiliary function K t is used as input, and the activation function β(K t ) (S235), and the calculation result is passed to gate z t Therefore, z t =β(K t )

[0056] <Status Update Calculation Unit 240> The state update calculation unit 240 calculates the update term - h t and Gate Z t is input, and the gate z t Using the following equation, the update term - h t and the state vector h t-1 and calculate the convex combination of the state vector h at time t t is calculated (S240) and output.

number

[0057] The state update calculation unit 240 calculates the data x1, . . . , x T-1 The state vectors h1,…,h T-1 to the update term calculation unit 220 and the gate calculation unit 230, and the data x T The state vector h corresponding to T is output to the output calculation unit 250.

[0058] <Output calculation unit 250> The output calculation unit 250 calculates the state vector h T is input, and the state vector h T The estimated value of the desired output is calculated using (S250), and the calculation result Y is output.

[0059] <Effects> With the above configuration, the estimation device can reflect the correlation between input data at different times in the estimation results, enabling accurate inference. In doing so, it is not necessary to increase the memory capacity and time required for calculation processing as much as in conventional techniques.

[0060] <Modification> In this embodiment, the update gate z in the GRU has been described as an example, but since this embodiment applies an auxiliary function prior to applying the activation function σ(x), it can be applied to any gated RNN using a sigmoid function.

[0061] <Experimental Results> For reference, Fig. 8 shows the training times actually measured for a gated RNN using a gate with only a sigmoid function that does not use an auxiliary function, the gate proposed in Reference 1, and the gate of the present invention. While the method in Reference 1 significantly increases the calculation time, the increase in calculation time due to the addition of the calculation of the auxiliary function of the gate in this embodiment is minimal, and can be said to be more desirable. <Other variations> The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices that execute the processes or as needed. Other modifications are possible within the scope of the present invention.

[0062] <Programs and recording media> The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 9, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0063] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0064] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0065] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the received program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (such as data that is not a direct instruction to the computer but has properties that define computer processing).

[0066] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware. [Explanation of symbols]

[0067] 100 Learning Device 110 Pretreatment section 120 Update term calculation section 130 Gate Calculation Unit 131 Linear transformation calculation unit 133 Auxiliary Function Calculation Unit 135 Activation Function Calculation Unit 140 State update calculation unit 150 Output calculation unit 160 Parameter Update Unit 200 Estimation device 210 Pretreatment section 220 Update term calculation section 230 Gate Calculation Unit 231 Linear Transformation Calculation Unit 233 Auxiliary Function Calculation Unit 235 Activation Function Calculation Unit 240 State update calculation unit 250 Output Calculation Unit 2000 Computer 2010 Control Unit 2020 Records Department 2030 Input section 2040 Output Unit 2050 Display section

Claims

1. A learning device for learning a gated recurrent neural network, comprising: Data x at time t included in the learning sequence data t,n,S and the state vector h at time t t-1,S and gate z at time t t,S a gate calculation unit for calculating The gate calculation unit calculates the gate z t,S When calculating, we use an activation function γ that converges to 0 or 1 doubly exponentially with respect to the input. Learning device.

2. 2. The learning device of claim 1, The gate calculation unit The data x t,n,S is a d-dimensional vector, and the state vector h t-1,S is an n-dimensional vector, and the parameter U z ∈R n×n , W z ∈R n×d , B z ∈R n a linear transformation calculation unit that calculates a linear transformation using an auxiliary function calculation unit that applies the result J of the linear transformation to an auxiliary function α and performs calculation; an activation function calculation unit that calculates an activation function β by providing a calculation result of the auxiliary function calculation unit to the activation function β; The auxiliary function α is a hyperbolic sine function, the activation function β is a sigmoid function, and the activation function γ is a composite function β(α(J)) of the auxiliary function α and the activation function β. Learning device.

3. An estimation device using a gated recurrent neural network trained by the learning device of claim 1, Data x at time t included in the series data to be estimated t and the state vector h at time t t-1 and gate z at time t t a gate calculation unit for calculating The gate calculation unit calculates the gate z t When calculating the input, an activation function that converges to 0 or 1 doubly exponentially is used. Estimation device.

4. A method for training a gated recurrent neural network, comprising: The computer calculates the data x at time t included in the learning sequence data. t,n,S and the state vector h at time t t-1,S and gate z at time t t,S a gate calculation step for calculating The gate calculation step is t,S When calculating, we use an activation function γ that converges to 0 or 1 doubly exponentially with respect to the input. How to learn.

5. An estimation method using a gated recurrent neural network trained by the training method of claim 4, The computer calculates data x at time t included in the sequence data to be estimated. t and the state vector h at time t t-1 and gate z at time t t a gate calculation step for calculating The gate calculation step is t When calculating the input, an activation function that converges to 0 or 1 doubly exponentially is used. Estimation method.

6. A program for causing a computer to function as the learning device according to claim 1.

7. A program for causing a computer to function as the estimation device according to claim 3.

Images