Calculation device, calculation method, and program

Abstract

A calculation device comprising: an input unit for inputting a function; an arithmetic unit for executing an arithmetic operation by using a neural network that causes a Koopman operator to act on the function; and an output unit for outputting a function converted from the function using the neural network.

Description

Calculation device, calculation method, and program

[0001] The present invention relates to neural networks.

[0002] Neural networks are applied to a wide range of fields and are actively researched. In particular, neural networks with functions as input and output have been proposed in recent years (see Non-Patent Document 1). Research on these neural networks has focused on finding solutions to differential equations.

[0003] Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89):1-97, 2023.

[0004] Existing neural networks that use functions as input and output are designed to handle functions in low-dimensional spaces such as two or three dimensions. However, in practical applications, there are cases where functions in high-dimensional spaces are desired as input and output, such as neural networks that use neural networks as input and output (models that transfer neural networks to neural networks).

[0005] The present invention has been made in consideration of the above points, and aims to provide a technology that enables calculations using a neural network that can use functions in a high-dimensional space as input and output.

[0006] According to the disclosed technology, there is provided a computing device comprising: an input unit for inputting a function; a calculation unit for performing calculations using a neural network that applies a Koopman operator to the function; and an output unit for outputting a function converted from the function using the neural network.

[0007] The disclosed technology provides a technology that enables calculations to be performed using a neural network that can use functions in a high-dimensional space as input and output.

[0008] It is a diagram showing an outline of processing. It is a diagram showing the configuration of a computing device 100. It is a diagram showing the configuration of a neural network. It is a diagram showing experimental results. It is a diagram showing an example of the hardware configuration of a computing device 100.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] (Outline of the embodiment) This embodiment relates to a neural network that receives a function as an input and outputs a function, as shown in Fig. 1. As described above, existing neural networks that use functions as input and output are designed to handle functions in low-dimensional spaces such as two or three dimensions, but there is a demand for functions in high-dimensional spaces as input and output.

[0011] In this embodiment, a linear operator that expresses function composition, called a Koopman operator, is used to realize a neural network that can take functions in a high-dimensional space as input and output.

[0012] Specifically, we construct a neural network with functions as input and output by transforming functions in high-dimensional spaces using the Koopman operator. Furthermore, when using the Koopman operator, we make it possible to apply the Koopman operator defined in infinite-dimensional spaces by using existing methods for approximating it in finite-dimensional spaces. This makes it possible to find solutions using neural networks even when functions in high-dimensional spaces are used as input and output. Furthermore, by making the space that approximates the Koopman operator dependent on the variables of the output function, we increase the diversity of functions that can be output, thereby achieving improved accuracy.

[0013] (Device Configuration Example) Fig. 2 shows a configuration example of a computing device 100 that performs calculations using a neural network according to this embodiment. As shown in Fig. 2, the computing device 100 includes an input unit 120, a calculation unit 130, an output unit 140, and a data storage unit 150.

[0014] The data storage unit 150 stores model parameters of a neural network. A function is input from the input unit 120. The calculation unit 130 uses the model parameters read from the data storage unit 150 to perform neural network calculations on the input function, and obtains a transformed function that is the output from the neural network. The transformed function is output from the output unit 140.

[0015] In this embodiment, the neural network is a neural network f, which will be described later, and more specifically, a fixed Koopman net or a dependent Koopman net, which will be described later.

[0016] The neural network of this embodiment will be described in detail below. Note that in the text of the specification, a_x means that x is a subscript of a, and a^x means that x is a superscript of a.

[0017] (Problem formulation) Let L be a natural number representing the number of layers in a neural network. For j = 1, ..., L, ｊ d ｊ－１ Let d be the dimension vector. ｊ is a nonlinear mapping to a dimensional vector, and the reproducing kernel Hilbert space H ｊ The Koopman operator K above ｆ＿ｊ Here, the Koopman operator is a function v such that v has f ｊis a linear operator that composes from the right (Reference 1: Kawahara, Y. Dynamic mode decomposition with reproducing kernels for Koopman spectral analysis. In Proceedings of Advances in Neural Information Processing Systems 30 (NIPS), 2016.).

[0018] σ j Let be a nonlinear operator (corresponding to an activation function) that transforms a function into another function, and consider a neural network f defined as follows for a function g:

[0019] From the definition of the Koopman operator, the output f(g) can be expressed as follows: An image of the neural network below is shown in Figure 3.

[0020] (Fixed Koopman net) Since the Koopman operator is defined in an infinite-dimensional space, it is difficult to express it on a computer. Therefore, in this embodiment, an actual algorithm is constructed using an existing method for approximating the Koopman operator in a finite-dimensional space.

[0021] For j=0,...,L, M ｊ is a natural number, and z ｊ = [z ｊ １ , ..., z ｊ Ｍ＿ｊ ]. However, M ｊ is M ｊ－１ Let j=1,...,M ｊ Assume that the following formula holds for

[0022] φ ｊ is the reproducing kernel Hilbert space H ｊ Let V be the feature map of ｊ (z ｊ ) is spanned by the following H ｊ Let us consider the subspace of

[0023] V ｊ (z ｊ ) orthonormal basis q ｊ １ , ..., q ｊ Ｍ＿ｊ is constructed using QR decomposition or principal component analysis. ｊ (z ｊ ) = [q ｊ １ , ..., q ｊ Ｍ＿ｊ ] and Φ(z ｊ ) are defined below.

[0024] Also, R ｊ (z ｊ ) into Φ(z ｊ ) R ｊ (z ｊ ) = Q ｊ (z ｊ ) ｊ ×M ｊ Let Q be the matrix. ｊ (z ｊ ) Q ｊ (z ｊ ) ＊ becomes a projection operator, where ＊ represents the adjoint operator. Therefore, the Koopman operator K ｆ＿ｊ is approximated by the following formula:

[0025] G ｊ (z ｊ ) to k ｊ (z ｊ ｉ , z ｊ ｌ ) as the (i, l) component ｊ＋１ ×M ｊ where k ｊ Is H ｊ Then, it can be calculated as in the following equation (3).

[0026] (3) The right side of the equation is K ｊ (z ｊ , z ｊ－１ ) Let K in equation (1) ｆ＿ｊ is replaced by equation (2), and then τ ｊ A, Mｊ Let M be the dimensional vector. ｊ By using a nonlinear mapping to a vector of dimensions, we can construct a fixed Koopman net defined as follows: x is the variable of the output function.

[0027] However, the following relationship is used:

[0028] (Dependent Koopman net) The problem with the fixed Koopman net is that there is a nonlinearity with respect to x. ～ G Ｌ (z Ｌ , x) (the left side of the above equation). Therefore, there is a limit to the functions that can be expressed as output functions. ｊ (z ｊ ) and K ｊ (z ｊ , z ｊ－１ ) also depends on x, which increases the diversity of functions that can be output.

[0029] P ｊ d ｊ ×d ０ Let V be the matrix of ｊ (z ｊ , x) into H spanned by ｊ Let us consider the subspace of

[0030] V ｊ (z ｊ , x) ｊ １ , ..., q ｊ Ｍ＿ｊ＋１ is constructed using QR decomposition or principal component analysis. ｊ (z ｊ , x) = [q ｊ １ , ..., q ｊ Ｍ＿ｊ＋１ ] and Φ(z ｊ , x) are defined below.

[0031] Also, R ｊ (z ｊ, x) into Φ(z ｊ , x)R ｊ (z ｊ , x) = Q ｊ (z ｊ , x) (M ｊ +1) × (M ｊ +1) matrix. Koopman operator K ｆ＿ｊ is approximated by the following equation (4).

[0032] G ｊ (z ｊ , x) for i=1,...,M ｊ＋１ ,l=1,...,M ｊ For k ｊ (z ｊ ｉ , z ｊ ｌ ) with i=M ｊ＋１ +1, l=1,...,M ｊ For k ｊ (P ｊ x, z ｊ ｌ ) for i=1,...,M ｊ＋１ , l=M ｊ k for +1 ｊ (z ｊ ｉ , P ｊ x), i=M ｊ＋１ +1, l=M ｊ For +1, k ｊ (P ｊ x, P ｊ x) as the (i, l) component (M ｊ＋１ +1) × (M ｊ +1) matrix. In this case, it can be calculated as in the following equation (5).

[0033] (5) The right side of the equation is K ｊ (z ｊ , z ｊ－１ , x). K in equation (1) ｆ＿ｊ is replaced with equation (5), and then τ ｊ A, M ｊ Let M be the dimensional vector. ｊBy using a nonlinear mapping to convert the vector into a dimensional vector, a dependent Koopman net defined as follows is constructed.

[0034] (Experiment) The technology according to this embodiment can be particularly applied to meta-learning. Meta-learning is a method for learning common knowledge from multiple tasks. A task is expressed as a function, and this is used as the input of a fixed Koopman net or a dependent Koopman net to obtain a new function as the output.

[0035] This allows us to obtain a model for solving the input task as a function. Here, we use a technique called LEO, introduced in Reference 2 (Rusu, A.A., Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., and Hadsell, R. Meta-learning with latent embedding optimization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.), to transfer the task to a neural network for solving that task and express it as a function of the task.

[0036] Here, we consider the classification problem of image data called miniImageNet. As in Reference 2, images were first converted into 640-dimensional vectors through feature extraction. We then input the neural network for solving the task obtained using LEO into a dependent Koopman net, and considered a learning problem to ensure that the output would be a better model for solving the task.

[0037] L=2, d ０ = d １ = d ２ = 640,M ０ =M １ =M ２ = 30, and a dependent Koopman net was constructed.ｊ is set to element-wise ReLU, and H ｊ The positive definite kernel that constitutes the above is set as follows:

[0038] Here, || ||1 represents the 1-norm. The loss function is cross categorical entropy, and training was performed using the Adam optimizer.

[0039] Figure 4 shows the test performance (accuracy rate) of the output of the dependent Koopman net after 200 epochs of training. Test performance was evaluated for three types of test datasets, and the results are shown as the mean ± standard deviation. As shown in Figure 4, the performance is better when using the dependent Koopman net than when using LEO alone.

[0040] (Hardware Configuration Example) The computing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on a cloud.

[0041] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0042] Fig. 5 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 5 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0043] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0044] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0045] (Summary, Effects, etc. of the Embodiments) As described above, the technology described in the present embodiments makes it possible to perform calculations using neural networks that can use functions in high-dimensional spaces as inputs and outputs. Furthermore, the technology according to the present embodiments contributes to improving the performance of neural networks, and is therefore applicable to a wide range of industrial fields where analysis using neural networks is used.

[0046] The following additional notes are provided regarding the above-described embodiments.

[0047] <Additional Notes> (Additional Item 1) A computing device comprising: an input unit that inputs a function; a calculation unit that performs an operation using a neural network that applies a Koopman operator to the function; and an output unit that outputs a function transformed from the function using the neural network. (Additional Item 2) The computing device according to Additional Item 1, wherein the Koopman operator in the neural network is a Koopman operator approximated in a finite-dimensional space using a projection operator. (Additional Item 3) The computing device according to Additional Item 1, wherein the neural network is configured to make a space that approximates the Koopman operator dependent on variables of the function to be output. (Additional Item 4) A computing method executed by a computing device, comprising: an input step of inputting a function; a calculation step of performing an operation using a neural network that applies a Koopman operator to the function; and an output step of outputting the function transformed from the function using the neural network. (Supplementary Item 5) A non-transitory storage medium storing a program for causing a computer to function as each unit in the computing device according to any one of Supplementary Items 1 to 3.

[0048] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0049] 100 Calculation device 120 Input unit 130 Arithmetic unit 140 Output unit 150 Data storage unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A computing device comprising: an input unit for inputting a function; a calculation unit for performing calculations using a neural network that applies a Koopman operator to the function; and an output unit for outputting a function converted from the function using the neural network.

2. The computing device according to claim 1, wherein the Koopman operator in the neural network is a Koopman operator approximated in a finite-dimensional space using a projection operator.

3. The computing device according to claim 1, wherein the neural network is configured to make the space that approximates the Koopman operator dependent on the variables of the output function.

4. A calculation method executed by a calculation device, comprising: an input step of inputting a function; a calculation step of performing a calculation using a neural network that applies a Koopman operator to the function; and an output step of outputting a function converted from the function using the neural network.

5. A program for causing a computer to function as each unit in the computing device according to any one of claims 1 to 3.

Images