Analysis device and program

Abstract

The analysis device according to one aspect includes an input unit configured to input learning data for a neural network model f which has a parameter θ formed of a plurality of elements each takes a value in an entire space A on a compact space Z and a parameter optimization unit configured to use the learning data to learn the parameter θ that minimizes a function Lreg containing a prescribed loss function L.

Description

[Technical field]

[0001] The present disclosure relates to an analysis device and a program. [Background technology]

[0002] Neural networks are widely applied to, for example, regression problems, classification problems, data generation, data distribution estimation, etc. In recent years, research on continuity of neural networks has also been conducted (for example, Non-Patent Document 1 and Non-Patent Document 2, etc.), and they are applied to various applications and theoretical analyses. In Neural ODE (Ordinary Differential Equations) described in Non-Patent Document 1, the transformation from input to output is regarded as a continuous dynamical system, and a solution method for continuous differential equations is used. This method can be said to be continuity in the layer direction (horizontal direction). In addition, in the integral representation theory described in Non-Patent Document 2, the transformation in each layer is made continuous and expressed using an integral operator, and theoretical analysis of neural networks is performed using the theory of harmonic analysis. This method can be said to be continuity in the unit direction (vertical direction). Such continuity makes it possible to use operations on continuous functions such as differentiation and integration, and it can be said that performance has been improved and theoretical analysis has been refined.

[0003] On the other hand, C * Tamaki and Hilbert C * Research is being conducted on the application of mathematical concepts such as -modules to data analysis (for example, Non-Patent Document 3), and basic properties necessary for data analysis have been shown. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Chen, RTQ, Rubanova, Y., Bettencourt, J., and Duvenaud, DK Neural ordinary differential equations. In Proceedings of the Advances in Neural Information Processing Systems 31 (NeurIPS), 2018. [Non-Patent Document 2] Sonoda, S. and Murata, N. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233-268, 2017. [Non-Patent Document 3] Hashimoto, Y., Ishikawa, I., Ikeda, M., Komura, F., Katsura, T., and Kawahara, Y. Reproducing kernel Hilbert C*-module and kernel mean embeddings. Journal of Machine Learning Research, 22(267):1-56, 2021. Summary of the Invention [Problem to be solved by the invention]

[0005] However, existing research on continuity of neural networks has focused on one model. On the other hand, in recent years, methods such as ensemble learning and meta-learning, which can simultaneously learn multiple models or learn features common to multiple models, have been attracting attention. For this reason, it is considered necessary to have a method for efficiently learning related to multiple models by targeting multiple models.

[0006] The present disclosure has been made in consideration of the above points, and provides a technology for efficiently performing learning related to a plurality of models. [Means for solving the problem]

[0007] The analysis device according to one aspect of the present disclosure includes an input unit configured to input training data for a neural network model f having a parameter θ composed of a plurality of elements that take values ​​in a space A of all continuous functions in a compact space Z, and an input unit configured to input training data for the neural network model f, and a function L including a predetermined loss function L using the training data. reg and a parameter optimization unit configured to learn the parameter θ that minimizes Effect of the Invention

[0008] Techniques are provided for efficiently performing learning related to multiple models. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an analysis device according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of an analysis device according to the present embodiment. [Diagram 3] 10 is a flowchart illustrating an example of a parameter optimization process according to the present embodiment. [Figure 4] FIG. 13 is a diagram showing an example of a sample. [Diagram 5] FIG. 1 is a diagram showing an example of an estimation result of a density function (part 1). [Figure 6] FIG. 2 is a diagram showing an example of the estimation result of a density function (part 2). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An embodiment of the present invention will be described below. In the following, an analysis device 10 that can efficiently perform learning related to a plurality of models by continuously connecting the plurality of models will be described. In this case, in order to continuously express the plurality of models, * Tamaki and Hilbert C * -Use mathematical concepts such as modules.

[0011] According to the analysis device 10 of this embodiment, it is possible to efficiently perform learning related to multiple models (i.e., learning while interacting between multiple models), and as a result, it becomes possible to efficiently analyze continuous data.

[0012] <Theoretical structure> The theoretical configuration of this embodiment will be described.

[0013] <Model and parameter settings> Let A be the space of all continuous functions on a compact space Z. For any commuting C * Note that the ring is isomorphic to such a space A. Consider a neural network model with K hidden layers, whose parameters (weights) take values ​​in A.

[0014] Let N0, ,N represent the dimensions of each layer of the neural network model. K+1 Here, N0 is the dimension of the input layer, N K+1 is the dimension of the output layer.

[0015] For i=1, ,K+1, N i-1 ×N i A matrix whose elements have values ​​in A is called W i In addition, A N_i From A N_i Conversion to σ i However, "N_i" is "N i " σ i corresponds to the activation function, and by subjecting it to a nonlinear transformation, nonlinearity occurs in the transformation of the entire neural network.

[0016] At this time, A N_0 Input a vector of A N_(K+1) Consider a neural network that outputs a vector as shown in the following equation (1). Here, "N_0" is "N0" and "N_(K+1)" is "N K+1 " respectively.

[0017]

number

[0018]

number

[0019] σ iAfter fixing and setting a loss function L that depends on the parameter θ, the optimal f is found by minimizing the loss function L when training data is given. The loss function L is a mapping that maps the parameter θ to an element A. Considering a parameter (function value) that has a value in A is equivalent to continuously connecting multiple models. The training data includes at least samples that represent the input to the neural network f. In addition, when finding the optimal f by supervised learning, the training data also includes a teacher for the sample (i.e., a correct example of the output of the neural network f when that sample is input). Note that the teacher is also called a label in classification tasks, etc.

[0020] <Loss function setting> In order to allow learning to occur while interacting between multiple continuous models, we use a loss function L (e.g., mean square error or cross entropy error) with a regularization term added, L. reg When Z is a finite measure space, L is minimized as shown in the following equation (2). reg Define.

[0021]

number

[0022] <Minimization of the loss function> Consider minimizing the loss function L using the gradient method. N →A and parameter θ∈A N For L, the gradient of A at θ is ∇ θ Let L be defined as follows:

[0023] "There is a ξ∈A N There exists a δ∈A N and for any z ∈ Z,

[0024]

number

[0025] In this case, L defined in the above formula (2) reg The gradient of (θ) can be calculated as follows:

[0026]

number

[0027] ∇ θ L reg Using this, we construct a gradient method scheme as shown in the following equation (4).

[0028]

number

[0029] In reality, the elements (functions) of A cannot be handled on a computer, so the calculation is performed as follows: N A finite-dimensional subspace V of and a mapping P:A to V N → Take V, and ∇ θ L to P(∇ θ L). Also, we replace the hyperparameters λ and η t is chosen as a constant function. P is, for example, a set of suitable points z1,...,z m ∈Z. Also, since Z is compact, the sequence

[0030]

number

[0031]

number

[0032] <Example of hardware configuration of analysis device 10> An example of the hardware configuration of an analysis device 10 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, analysis device 10 according to this embodiment has an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0033] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the analysis device 10 does not necessarily have to have at least one of the input device 101 and the display device 102, for example.

[0034] The external I / F 103 is an interface with an external device such as a recording medium 103a. The analysis device 10 can read and write data from and to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), a SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0035] The communication I / F 104 is an interface for connecting the analysis device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as a hard disk drive (HDD) or a solid state drive (SSD). The processor 108 is an arithmetic device such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0036] The analysis device 10 according to this embodiment has the hardware configuration shown in Fig. 1 and is therefore capable of implementing a parameter optimization process, which will be described later. Note that the hardware configuration shown in Fig. 1 is merely an example, and the hardware configuration of the analysis device 10 is not limited thereto. For example, the analysis device 10 may have a plurality of auxiliary storage devices 107 and a plurality of processors 108, or may have various other hardware components in addition to the hardware components shown in the figure.

[0037] <Example of functional configuration of analysis device 10> An example of the functional configuration of the analysis device 10 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the analysis device 10 according to this embodiment has a data input unit 201, a parameter optimization unit 202, an output calculation unit 203, and a data output unit 204. Each of these units is realized, for example, by a process in which one or more programs installed in the analysis device 10 are executed by the processor 108. The analysis device 10 according to this embodiment also has a storage unit 205. The storage unit 205 is realized, for example, by the auxiliary storage device 107 or the like. Note that the storage unit 205 may be realized, for example, by a storage device connected to the analysis device 10 via a communication network.

[0038] The data input unit 201 inputs the learning data given to the analysis device 10. The parameter optimization unit 202 updates the parameter θ by equations (3) and (4) using the learning data input by the data input unit 201. The output calculation unit 203 calculates the output of f when the samples included in the learning data are input when the parameter optimization unit 202 calculates the gradient ∇ θ L reg When calculating reg . The data output unit 204 outputs the parameter θ updated by the parameter optimization unit 202. The storage unit 205 stores various information (for example, learning data, parameter θ, hyperparameters λ and η t etc.).

[0039] <Flow of Parameter Optimization Processing> The flow of the parameter optimization processing according to this embodiment will be described with reference to FIG. 3. Here, hereinafter, let M be a sufficiently large natural number and ε be a sufficiently small positive real number.

[0040] The data input unit 201 inputs the given learning data (generally, a large number of learning data) (step S101).

[0041] The parameter optimization unit 202 initializes t = 0 and initializes the parameter θ t = θ0 (step S102). Note that the parameter optimization unit 202 may initialize the parameter θ0 by, for example, random numbers or the like.

[0042] The parameter optimization unit 202 determines whether t < M or ||L(θ t )|| > ε (step S103). If the parameter optimization unit 202 determines that t < M or ||L(θ t )|| > ε, it proceeds to step S104, and if not (that is, if t ≥ M and ||L(θ t )|| ≤ ε), it proceeds to step S107.

[0043] The parameter optimization unit 202 calculates the gradient ∇ by equation (3) θ Lreg (Step S104). At this time, the parameter optimization unit 202 calculates ∇ in the formula (3). θ L to P(∇ θ In addition, when the parameter optimization unit 202 calculates the formula (3), the output of f when the sample included in the learning data is input is calculated by the output calculation unit 203, so that P(∇ θ Calculate L).

[0044] Next, the parameter optimization unit 202 calculates the parameter θ t+1 is updated (step S105).

[0045] Next, the parameter optimization unit 202 updates t to t+1 (step S106).

[0046] The data output unit 204 outputs the parameter θ t (Step S107). The data output unit 204 outputs the parameter θ to a predetermined output destination (for example, the storage unit 205, another program, another device connected via a communication network, etc.). t This will give the optimal parameter θ=θ t As a result, the optimal f=f θ is obtained.

[0047] <Application Examples> An application example of the analysis device 10 according to this embodiment will be described below.

[0048] <Application Example 1: Density estimation using normalizing flow> Let Z be a compact probability space and D be a probability measure on Z. Let Ω be a probability space, and let R be a random variable X on Z×Ω. N_0 (Here, R is the set of all real numbers, and "N_0" represents "N0".) For any ω∈Ω, X(·,ω) is continuous.

[0049] Then, for any z∈Z, if the random variable X(z,·) follows a normal distribution, then the random variable

[0050]

number

[0051] The samples given as training data are x1, , x n ∈R N_0 The loss function L is based on the idea of ​​maximizing the likelihood.

[0052]

number

[0053] <Application Example 2: Few-shot learning> Few-shot learning is a method for learning a model from a small number of samples. Meta-learning is a method for improving the accuracy of few-shot learning. In meta-learning, a meta-model is learned in advance using learning data for multiple tasks, and features common to tasks are learned. By using these features, a model for a new task can be efficiently learned, improving accuracy, even if the number of samples for the new task is small. For example, in the method proposed in Reference 1, a mapping Z that maps a task to a representation in a low-dimensional space and a mapping Θ that converts the representation in the low-dimensional space into the parameters of the model are learned as meta-models. The accuracy of few-shot learning can be further improved by combining such a meta-model with the parameter optimization process shown in Figure 3.

[0054] As an example, consider a classification problem. Let samples be x1, ,x n ∈R N_0 In addition, sample x i The label for y i ∈R N_(K+1) Furthermore, x i Let X be a constant function that takes values ​​in i , y i Let Y be a constant function that takes values ​​in i In this case, sample x i ∈R N_0 and its label y i ∈R N_(K+1) For a constant function X i to a constant function Y i f = f defined in the above equation (1) is used to transform θ Learn.

[0055] We first learn Z and Θ in the metamodel, and then we use the low-dimensional representation z new Get z new Let the neighborhood of Z be Z, and set the initial value θ0 of the model parameters for the new task as a mapping of Θ restricted to Z. For the loss function L, we use the cross-entropy error

[0056]

number

[0057] <Evaluation Results> The results of evaluation of the analysis device 10 according to this embodiment will be described below.

[0058] Assume that two types of 100 samples are given, as shown in the left and right figures of Figure 4. In this case, the density functions of the distributions that the two types of 100 samples each follow are estimated by the method described in Application Example 1.

[0059] Let Z = [-4,4] x [-4,4],

[0060]

number

[0061]

number

[0062] In addition, the random variable X(z,·) follows a normal distribution with mean z and standard deviation 1.

[0063] In this case, when the sample shown in the left diagram of Figure 4 is given, the density function estimated by the method explained in Application Example 1 is shown in the left diagram of Figure 5, with the average z i The right panel of Fig. 5 shows the results of averaging the outputs of nine models using normal distributions with a standard deviation of 1 and 10, respectively. Similarly, the results of estimating the density function using the method explained in Application Example 1 when the samples shown in the right panel of Fig. 4 are given are shown in the left panel of Fig. 6, with the average z i The results of training nine models using normal distributions with a standard deviation of 1 separately and averaging their outputs are shown in the right diagram of Figure 6. The method for obtaining the results shown in the right diagrams of Figures 5 and 6 corresponds to the method described in Application Example 1, in which both λ and the hyperparameter μ of the kernel ridge regression in Equation (2) are set to zero.

[0064] 5 and 6, it can be seen that the density function can be estimated more smoothly in the left diagram than in the right diagram. This is because the analysis device 10 according to this embodiment continuously connects multiple models and utilizes the continuous nature of the function.

[0065] The present invention is not limited to the above-described embodiments specifically disclosed, and various modifications, changes, combinations with known technologies, etc. are possible without departing from the scope of the claims.

[0066] [References] Reference 1: Rusu, AA, Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., and Hadsell, R. Meta-learning with latent embedding optimization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019. [Explanation of symbols]

[0067] 10 Analysis device 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 201 Data Entry Department 202 Parameter Optimization Department 203 Output Calculation Unit 204 Data output section 205 Storage section

Claims

1. An input unit configured to input learning data for a neural network model f, the neural network model having a parameter θ composed of a plurality of elements whose values ​​are in a space A of all continuous functions in a compact space Z; Using the learning data, a function L including a predetermined loss function L is calculated. ｒｅｇ a parameter optimization unit configured to learn the parameter θ that minimizes An analysis device having the above configuration.

2. The function L ｒｅｇ is expressed by a mapping that maps the parameter θ to an element of A, and includes a regularization term that is expressed by an integral of the loss function L over Z.

3. The parameter optimization unit is The number of elements of the parameter θ is N, A Ｎ Let V be a finite-dimensional subspace of Ｎ →V, the gradient ∇ of the loss function L with respect to the parameter θ θ Instead of L, use P (∇ θ L) to calculate the function L with respect to the parameter θ. ｒｅｇ The gradient of ∇ θ L ｒｅｇ Calculate The gradient ∇ θ L ｒｅｇ The analysis device according to claim 1 , wherein the parameter θ is learned by updating the parameter θ using

4. The analysis device according to claim 3 , wherein the mapping P is a mapping defined by regression including kernel ridge regression.

5. An input step of inputting learning data for a neural network model f having a parameter θ composed of a plurality of elements whose values ​​are in a space A of all continuous functions in a compact space Z; Using the learning data, a function L including a predetermined loss function L is calculated. ｒｅｇ a parameter optimization procedure to learn the parameters θ that minimize A program that causes a computer to execute the following.

Images