Information processing device, information processing method, brain model, and program

Abstract

To provide efficient and versatile learning techniques. [Solution] The information processing device (1) includes an acquisition unit (11) that acquires one or more input values, and a learning unit (12) that executes a learning process by referring to a loss function that is a model function that has the one or more input values ​​as arguments and is defined by the logarithm of a model function that satisfies a predetermined normalization condition.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a brain model, and a program. [Background technology]

[0002] Since the usefulness of deep neural networks was demonstrated, deep learning and other machine learning technologies have made great progress and are being applied to various fields (e.g., Non-Patent Document 1). However, it is difficult to say that a fundamental understanding of the overall learning process, including machine learning, has progressed. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] G. E. Hinton, S. Osindero, and Y. W. Teh, Neural Computation 18, 1527 (2006) Summary of the Invention [Problem to be solved by the invention]

[0004] As mentioned above, due to the difficulty of fundamentally understanding the learning process, the machine learning techniques known to date have a large empirical element rather than a fundamental one. For this reason, algorithms that are not necessarily efficient or that work well only under certain circumstances have been widely used.

[0005] One aspect of the present invention is based on the inventor's essential knowledge of the learning process, and provides a more efficient and versatile learning technique than has been previously known. [Means for solving the problem]

[0006] In order to solve the above problem, an information processing device according to one embodiment of the present invention includes an acquisition unit that acquires one or more input values, and a learning unit that executes a learning process by referring to a loss function that is a model function that takes the one or more input values ​​as arguments and is defined by the logarithm of a model function that satisfies a predetermined normalization condition.

[0007] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a prediction unit that executes a prediction process using a model function that has one or more input values ​​as arguments and that is trained by referring to a loss function defined by the logarithm of the model function that satisfies a predetermined normalization condition, and an output unit that outputs a prediction result obtained by the prediction process.

[0008] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a network including a plurality of nodes directly or indirectly connected to each other, and a learning unit that trains each of the plurality of nodes using a loss function that is defined to be localized in space and time with respect to the node.

[0009] In order to solve the above problems, a brain model according to one embodiment of the present invention includes a plurality of cells that are directly or indirectly connected to each other, and each of the plurality of cells is trained using a loss function that is defined to be spatially and temporally localized with respect to the cell.

[0010] In order to solve the above problem, an information processing method according to one embodiment of the present invention includes an acquisition step of acquiring one or more input values, and a learning step of executing a learning process by referring to a loss function that has the one or more input values ​​as arguments and is defined by the logarithm of a model function that satisfies a predetermined normalization condition.

[0011] In order to solve the above problems, an information processing method according to one aspect of the present invention includes a prediction step of performing a prediction process using a model function that has one or more input values ​​as arguments and that has been trained with reference to a loss function defined by the logarithm of the model function that satisfies a predetermined normalization condition, and an output step of outputting a prediction result obtained by the prediction process.

[0012] The information processing device according to each aspect of the present invention may be realized by a computer. In this case, the program of the information processing device that realizes the information processing device on a computer by causing the computer to operate as each part (software element) of the information processing device, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention. Effect of the Invention

[0013] According to one aspect of the present invention, an efficient and versatile learning technique can be provided. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of a configuration of an information processing device according to an embodiment of the present invention; [Diagram 2] 1 is a flowchart showing a flow of processing by an information processing apparatus according to an embodiment of the present invention. [Diagram 3] 1 is a flowchart showing a flow of processing by an information processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Diagram 5] FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Figure 6] FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Figure 7]FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Figure 8] 11 is a graph for explaining the effect of a processing example according to an embodiment of the present invention. [Figure 9] 11 is a graph for explaining the effect of a processing example according to an embodiment of the present invention. [Figure 10] 11 is a graph for explaining the effect of a processing example according to an embodiment of the present invention. [Figure 11] FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Figure 12] FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Figure 13] FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Figure 14] FIG. 1 is a diagram for explaining a processing example according to an embodiment of the present invention, and is a diagram showing an example of the configuration of a neural network. [Figure 15] FIG. 2 is a diagram illustrating an example of a configuration of a model according to an embodiment of the present invention. [Figure 16] FIG. 2 is a diagram for explaining the time evolution of a model according to an embodiment of the present invention. [Figure 17] FIG. 2 is a diagram for explaining a nonlinear model according to an embodiment of the present invention. [Figure 18] 1 is a block diagram showing an example of a configuration of an information processing device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] [Embodiment 1] An embodiment of the present invention will be described in detail below. Fig. 1 is a block diagram showing an example of the configuration of an information processing device 1 according to this embodiment. As shown in Fig. 1, the information processing device 1 includes, as an example, a control unit 10, a storage unit 20, a communication unit 30, and an input / output unit 40.

[0016] (Communication unit 30) The communication unit 30 communicates with one or more devices external to the information processing device 1. The communication unit 30 transmits data supplied from the control unit 10 to the external device, and supplies data received from the external device to the control unit 10. As an example, the communication unit 30 supplies input data INPUT acquired from an external device to the control unit 10, and transmits output data OUTPUT derived by the control unit 10 to the external device.

[0017] (Input / output section 40) As an example, the input / output unit 40 is configured to include at least one of input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. Alternatively, the input / output unit 40 may be configured to have input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel connected thereto. In this configuration, the input / output unit 40 accepts input of various information from the connected input device to the information processing device 1. Also, the input / output unit 40 outputs various information to the connected output device under the control of the control unit 10A. An example of the input / output unit 40 is an interface such as a Universal Serial Bus (USB).

[0018] (Storage unit 20) The storage unit 20 stores various data referenced by the control unit 10 and various data generated by the control unit 10. As an example, the storage unit 20 stores: Input value (input data) INPUT acquired by the acquisition unit 11 described later A model function MF that is the object of learning by the learning unit 12 described later Loss function LF which is referenced in the learning process by the learning unit 12 described later Output value (output data) generated or derived by the control unit 10 etc. are stored. In this embodiment, the term "model function MF" may include the meaning of "parameters that define the model function MF." Furthermore, the "model function" may also be simply called a "model."

[0019] (Control unit 10) As shown in FIG. 1, the control unit 10 includes an acquisition unit 11, a learning unit 12, a prediction unit 13, and an output unit 14.

[0020] (Acquisition part 11) The acquisition unit 11 acquires one or more input values ​​INPUT. The input value may be simply referred to as input or as input data. The input value may also be expressed algebraically using x, such as input value x. The multiple input values ​​may also be considered to constitute a vector with the input values ​​as components, and therefore the input values ​​may also be referred to as an input vector.

[0021] (Learning Section 12) The learning unit 12 learns a model function with reference to a loss function. As an example, the learning unit 12 learns a loss function Φ(x) defined by a logarithm (for example, a natural logarithm) of a model function Φ(x) that has one or more input values ​​x as arguments and satisfies a predetermined normalization condition.

number

number

number

[0022] As an example, the learning unit 12 optimizes the model function by minimizing the loss function. Note that the appropriateness of the loss function does not limit this embodiment, and for example, a loss function with the sign of the above (Equation 1) inverted may be used. In such a case, the learning process by the learning unit 12 can be expressed as including a process of optimizing the model function by maximizing the loss function.

[0023] Here, the model function Φ(x) is the object of optimization in the learning process. Although more details will be described later, according to the findings of the inventors, the essence of the learning process can be expressed as optimizing the model function Φ(x) so as to reproduce the probability P(x) of the input value x. In other words, the essence of the learning process can be expressed as the process of constructing a model function Φ(x) that is as close as possible to the probability P(x) of the input value x.

[0024] As will be described in detail later, according to the findings of the inventors, The loss function L is defined using the logarithm of the model function Φ(x), Minimize the loss function L (maximize it if you invert the sign of the loss function) This theoretically guarantees that the model function Φ(x) approaches the probability P(x) of the input value x.

[0025] It should be noted that the above expression of the loss function L does not include any sums of values ​​or indexes, for example. For example, the Kullback=Leibler divergence (KL divergence) (also called relative entropy), which is often used as a loss function, can be expressed as follows, for example, using the probability distributions P(i) and Q(i) for value i:

number

[0026] On the other hand, the loss function (Formula 1) according to this embodiment is defined, for example, without including any sums related to values ​​or indexes. Therefore, by performing learning with reference to the loss function (Formula 1) according to this embodiment, it is possible to significantly reduce calculation costs compared to the conventional method.

[0027] (Prediction Section 13) The prediction unit 13 executes a prediction process using a model function that has one or more input values ​​as arguments and that has been trained with reference to a loss function defined by the logarithm of a model function that satisfies a predetermined normalization condition. As an example, the prediction unit 13 executes a prediction process using the model function Φ(x) trained (optimized) by the above-mentioned learning unit 12. Then, the prediction unit 13 derives an output value (output data) OUTPUT that indicates the result of the above-mentioned prediction process.

[0028] (Output section 14) The output unit 14 outputs the output value (output data) OUTPUT generated or derived by the control unit 10, for example, via the input / output unit 40. For example, the output unit 14 outputs a prediction result by the prediction process executed by the prediction unit 13. The output unit 14 may also be configured to output the model function Φ(x) itself learned (optimized) by the learning unit 12.

[0029] The prediction unit 13 and the output unit 14 perform prediction processing using the model (model function) suitably learned by the learning unit 12, and can suitably output the prediction result.

[0030] (Processing flow by information processing device) Next, an example of the flow of processing by the information processing device 1 will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a flow diagram showing an example of the flow of learning processing by the information processing device 1.

[0031] (Step S11) In step S11, the acquisition unit 11 acquires one or more input values ​​INPUT. The process by the acquisition unit 11 has been described above, and therefore will not be described here.

[0032] (Step S12) Next, in step S12, the learning unit 12 executes a learning process by referring to a loss function (Formula 1) that is a model function Φ(x) having the one or more input values ​​x as arguments and that is defined by the logarithm (natural logarithm as an example) of the model function Φ(x) that satisfies a predetermined normalization condition. The process by the learning unit 12 has been described above, so a description thereof will be omitted here.

[0033] As described above, the loss function (Formula 1) according to this embodiment is defined, for example, without including any sums related to values ​​or indexes. Therefore, according to the above processing flow for performing learning with reference to the loss function (Formula 1) according to this embodiment, the calculation cost can be significantly reduced compared to the conventional method.

[0034] FIG. 3 is a flow diagram showing an example of the flow of the inference process (prediction process) by the information processing device 1.

[0035] (Step S13) In step S13, the prediction unit 13 executes the prediction process using a model function that has one or more input values ​​as arguments and that has been trained with reference to a loss function defined by the logarithm of a model function that satisfies a predetermined normalization condition. As an example, the prediction unit 13 executes the prediction process using the model function Φ(x) trained (optimized) by the learning unit 12 described above.

[0036] (Step S14) Next, in step S14, the output unit 14 outputs the prediction result of the prediction process executed by the prediction unit 13. According to the information processing device 1 that executes the above-described processing flow, the prediction process is performed using the model (model function) that has been suitably learned by the learning unit 12, and the prediction result can be suitably output.

[0037] (Processing example) The following describes various processing examples performed by the information processing device 1. Note that the processing examples given below are examples of processing that are expressed as executable based on the inventor's knowledge, but these specific processing examples do not limit the present embodiment.

[0038] (Processing example 1) In this example, the acquisition unit 11 of the information processing device 1 acquires an input value INPUT including one or more data values ​​and teacher labels associated with the data values. As an example, the input value INPUT acquired by the acquisition unit 11 may be configured to include a set of input data {x1, x2,...} and a set of teacher labels {t1, t2,...}. In other words, the input value INPUT may be configured to include one or more pairs (x, t) of input data x and a corresponding teacher label t. The input value INPUT configured in this way can be used, as an example, as learning data in a supervised classification problem. However, the wording does not limit this example.

[0039] The model function in this example can be expressed as Φ(x,t) using the input data x and the corresponding teacher label t. In other words, the model function Φ in this example is, for example, a function that takes the input data x included in the input value and the corresponding teacher label t as arguments.

[0040] The learning unit 12 according to the present embodiment calculates a loss function defined by the logarithm (as an example, the natural logarithm) of a model function Φ(x, t) that uses input data x included in the input value INPUT and a corresponding teacher label t as arguments and satisfies a predetermined normalization condition.

number

number

[0041] An example of the model function (also simply called a model) according to this example is the neural network shown in FIG. 4. As shown in FIG. 4, in this example, an input value x is input to an input layer, and an output layer has the same number of nodes as the types of teacher labels (sometimes called classes, etc.). In the example of FIG. 4, the sum of the node values ​​in the output layer is normalized by a softmax function, and the model function Φ(x, t) is defined as the value of the corresponding node in the output layer. In this case, the loss function (Equation 5) according to this example is the same as the cross entropy by one-hot vectors.

[0042] (Processing example 2) In the above-described processing example 1, the truth label value is discrete, but this does not limit the present embodiment. As described in this example, the truth label value may be continuous.

[0043] As an example, the input value INPUT acquired by the acquisition unit 11 is configured to include a set of input data {x1, x2, . . .} and a set of teacher labels {t1, t2, . . .}, and each teacher label t i may take a continuous value. In other words, the input value INPUT may be configured to include one or more pairs (x, t) of input data x and a corresponding teacher label t that is a continuous value. The input value INPUT configured in this manner can be used, for example, as learning data in a regression problem. However, this wording does not limit this example.

[0044] The learning unit 12 according to the present embodiment is a model function Φ(x, t) that uses input data x included in the input value INPUT and a corresponding teacher label t as arguments, and is a loss function defined by the logarithm (as an example, the natural logarithm) of the model function Φ(x, t) that satisfies a predetermined normalization condition.

number

number

[0045] In this example, since the normalization condition for the model function Φ(x,t) includes an integral as described above, it is not preferable to use the model as described in the processing example 1 as it is.

[0046] In the following description of this example, some approximations and assumptions are used. As an example, a Gaussian distribution function Φ(x,t) is used as the model function Φ(x,t).

number

[0047] FIG. 5 shows a model according to this example, where n=2. More specifically, μ is

number

number

[0048] In the above example, the loss function L(Φ(x,t)) is

number

[0049] As shown in FIG. 5, the learning unit 12 according to the present embodiment inputs input data x included in the input value INPUT to a model (model function Φ(x,t)) and calculates each parameter (μ1, μ2, σ 11 ,σ 12 ,σ 22The values ​​of the above parameters are determined so as to minimize (or maximize when the sign of the loss function is inverted) the loss function L(Φ(x,t)) given by Equation 12.

[0050] Thus, in this example, The number of model functions Φ(x,t) is, for example, one or more parameters (in the above example, μ1, μ2, σ 11 ,σ 12 ,σ 22 ) is a distribution function defined by The learning process according to this example can be expressed as including a process of determining the one or more parameters by minimizing or maximizing the loss function L(Φ(x,t)).

[0051] As an example, if Σ with a tilde is approximated by a unit matrix, the loss function in Equation 12 becomes

number

[0052] As will be described in more detail later, from the viewpoint of the information processing device 1 according to this embodiment, the conventionally used mean square error is a distribution (distribution function) defined by the input data x and the corresponding t, and corresponds to approximating the true distribution P(t|x) of the input value by a Gaussian distribution having a certain variance. Conventionally, the mean square error has sometimes been used for data for which such an approximation should not be used, which has caused learning to fail. In this example, more general-purpose learning is possible by selecting an appropriate probability distribution function for the data used for learning.

[0053] (Processing example 3) In this processing example, processing executed by the information processing device 1 and processing related to a parameter estimation problem will be described. However, this wording does not limit this example.

[0054] First, let us consider some data {x1, x2, . . . , x N} is given. If we assume that these data follow a distribution defined by some parameters, there is a known problem of how to estimate the values ​​of these parameters (also called the parameter estimation problem). In the past, maximum likelihood estimation methods have often been used for such problems.

[0055] As an example, if we denote the set of parameters by θ and P(x|θ) by the probability distribution of x parameterized by θ, then the above parameter estimation problem can be solved as N} can be expressed as the problem of finding a parameter θ that best explains

[0056] This is the likelihood function L given by * (θ), or the likelihood function L * Logarithm of (θ) logL * This can be expressed as the problem of maximizing (θ).

[0057]

number

number

number

number

number

[0058] The model according to this example is shown in Fig. 6. As shown in Fig. 6, the model according to this example does not have layers or links, and can be interpreted as the parameters θ themselves (in the example shown in Fig. 6, four-dimensional parameters θ1, θ2, θ3, and θ4).

[0059] As a process using the model function Φ(x) given as described above, the learning unit 12 according to this example performs the following process:

number

[0060] Thus, the model function Φ(x) in this example is It can also be expressed as a distribution function with one or more input values ​​x as arguments, It can also be expressed as one or more parameters (θ) themselves that define the distribution function.

[0061] Comparing the conventionally used likelihood function (Formula 15) with the loss function L(Φ(x)) (Formulas 16 and 19) referred to by the learning unit 12 according to the present example, the conventionally used likelihood function (Formula 15) is defined to include a sum over the entire data set (in the above example, the sum from index i=1 to N), whereas the loss function L(Φ(x)) referred to by the learning unit 12 according to the present example is defined without including such a sum. In other words, the loss function L(Φ(x)) referred to by the learning unit 12 according to the present example is defined for each x, and the learning unit 12 calculates the value of the loss function L(Φ(x)) for each x.

[0062] In this way, the learning unit 12 according to the present embodiment performs learning using a loss function that is defined without including a sum across the data set, and therefore the calculation cost can be significantly reduced compared to the conventional method.

[0063] (Expression of model functions by differentiation) Although more details will be described later, based on the knowledge obtained by the inventors, the above-mentioned model function may be expressed by differentiation. In other words, the information processing device 1 according to the present embodiment may perform processing with reference to a model function Φ(x) expressed as a derivative of a certain function.

[0064] (In the case of one variable) First, the case of one variable will be described below. In the case of one variable, as an example, the model function Φ(x) according to this embodiment is expressed as follows, using a certain function y(x):

number

[0065] In order to give it a probabilistic interpretation, the condition for the above function y is as follows:

number

[0066] The first condition in Equation 21 is achieved, for example, by making the weight coefficients only positive and making the activation function at each node a monotonically increasing function in a network model as shown in Figure 7. The second condition in Equation 21 is achieved, for example, by using a sigmoid function at the output node as shown in Figure 7. A model function Φ(x) that satisfies such conditions satisfies the normalization condition as follows.

number

[0067] In the above configuration, the loss function L(Φ(x)) is

number

[0068] Fig. 8 is a diagram showing an example of the results of the learning process by the learning unit 12 using the model function Φ(x) defined as above, and shows the results when the input data indicates a multimodal distribution. More specifically, Fig. 8 shows the results when the sample size of the input data is 4000 and the results when the sample size of the input data is 40.

[0069] In Figure 8, the histogram shows the distribution of the input data (a multimodal distribution in the example of Figure 8), with the curve shown in light gray showing the true probability distribution of the input data, and the curve shown in dark gray showing the probability distribution derived (estimated) by the above-mentioned learning process by the learning unit 12 (the distribution shown by the model function Φ(x)).

[0070] As can be seen from FIG. 8, by executing a learning process using the above model function obtained from the inventor's findings, good estimation results are obtained not only when the sample size of the input data is relatively large (for example, when the sample size shown in FIG. 8 is 4000), but also when the sample size of the input data is very small (for example, when the sample size shown in FIG. 8 is 40).

[0071] Fig. 9 is a diagram showing an example of the results of the learning process by the learning unit 12 using the model function Φ(x) defined as above, and shows the results when the input data shows a flat distribution. More specifically, Fig. 8 shows the results when the sample size of the input data is 1000 and the results when the sample size of the input data is 10. The legend is the same as in Fig. 8.

[0072] As can be seen from FIG. 9, by executing a learning process using the above model function obtained from the inventor's findings, good estimation results are obtained not only when the sample size of the input data is relatively large (for example, when the sample size shown in FIG. 9 is 1000), but also when the sample size of the input data is very small (for example, when the sample size shown in FIG. 9 is 10).

[0073] 10 is a diagram showing an example of the results of the learning process by the learning unit 12 using the model function Φ(x) defined as above, and shows the results when the input data shows an asymmetric distribution. More specifically, FIG. 10 shows the results when the sample size of the input data is 10000 and the results when the sample size of the input data is 100. The legend is the same as FIG. 8.

[0074] As can be seen from FIG. 10, by executing a learning process using the above model function obtained from the inventor's findings, good estimation results are obtained not only when the sample size of the input data is relatively large (for example, when the sample size shown in FIG. 10 is 10,000), but also when the sample size of the input data is very small (for example, when the sample size shown in FIG. 10 is 100).

[0075] In this way, according to the configuration of this embodiment, not only can the calculation cost in the learning process be significantly reduced, but also a model with high estimation accuracy can be constructed even when the data size of the input data is small. In addition, highly accurate learning is possible for any data without any prior knowledge, making the learning process much more versatile than conventional models. (Multiple variables case (part 1)) Next, the case of multiple variables will be explained as follows. First, the input data set {x1, x2,...} containing multiple variables is x = (a1, a2, , a n ) Here, each a i As an example, n is a continuous value, and n is the number of dimensions of the input data.

[0076] In the case of multiple variables, as an example, the model function Φ(x) according to this embodiment is expressed as follows, using a certain function y(x):

number

[0077] Here, the function y(x) may be, for example, a network model in which x is an input value and y is an output value, as shown in FIG.

[0078] In order to give it a probabilistic interpretation, the condition for the above function y is as follows:

number

[0079] The first condition in Equation 25 is achieved, for example, by making the weight coefficients only positive and making the activation function in each node a monotonically increasing function in a network model as shown in Figure 11. The second condition in Equation 25 is achieved, for example, by using a sigmoid function in the output node as shown in Figure 11. A model function Φ(x) that satisfies such conditions satisfies the normalization condition as follows.

number

[0080] In the above configuration, the loss function L(Φ(x)) is

number

[0081] In addition, the above example for multiple variables includes n-th order differentiation, so that the calculation cost increases as the dimension n of the input data increases. Below, we will explain a processing example that can solve this problem.

[0082] (Multiple variables case (part 2)) In the following processing example, a process using a Jacobian (Jacobian determinant) instead of the n-th order differential (Equation 24) used in the above example will be described.

[0083] In this embodiment, as an example, the model function Φ(x) according to this embodiment is expressed as follows, using a certain function y(x):

number

number

[0084] Here, the function y(x) may be, for example, a network model in which x is an input value and y is an output value, as shown in FIG.

[0085] In order to give it a probabilistic interpretation, the condition for the above function y is as follows:

number

number

[0086] In this example, the following condition is set for the function y:

number

[0087] Here, the function y(x) may be, for example, a network model with x as an input value and y as an output value, as shown in FIG. 13. The model includes multiple variables x=(a1, a2, . . . , a n ) is input, and the multivariate y(x)=(b1,b2,...,b n ) is output. As shown in FIG. 13, the model has only links (edges) that satisfy the condition i≧j as links (edges) from the jth node in a layer to the ith node in the next layer. When expressed using the arrangement shown in FIG. 13, there are no links (edges) from lower nodes to higher nodes. The configuration in the network shown in FIG. 13 corresponds to the condition of the first formula in the above formula 32. Also, as shown in the second formula in the above formula 32, the values ​​of the links (edges) for i=j are all positive.

[0088] In the above-mentioned configuration, the model function Φ(x) according to this embodiment is expressed as follows:

number

number

[0089] In this way, the Jacobian in this example is expressed as the product of the diagonal elements of a triangular matrix, and the loss matrix is ​​expressed as the sum of the logarithms of the diagonal elements of the triangular matrix, so that the calculation cost is on the order of n. In addition, the positive definiteness of the Jacobian is guaranteed by the second equation of the above-mentioned equation 32.

[0090] Therefore, according to the learning unit 12 that performs the processing according to this example, the calculation cost required for the learning processing can be significantly reduced.

[0091] (Application to regression problems) In the following, a processing example will be described in which the above-mentioned representation of the model function by differentiation is applied to a general regression problem. Note that the description of the notation that has already been given will be omitted as appropriate.

[0092] In this embodiment, as an example, the model function Φ(x,t) according to this embodiment is expressed as follows, using a certain function y(x,t):

number

[0093] In this way, the learning unit 12 learns a plurality of input values ​​(x, t) (where t=(a1, a2, . . . , a n ) and a plurality of functions y(x,t)=(b1,b2,...,b nIn other words, the model function Φ(x) may be expressed as a Jacobian defined by at least a part of the plurality of input values ​​(x, t) and a plurality of functions y that use at least a part of the plurality of input values ​​(x, t) as arguments.

[0094] Here, the function y(x, t) may be, for example, a network model in which (x, t) is an input value and y is an output value, as shown in FIG.

[0095] In order to give it a probabilistic interpretation, the condition for the above function y(x,t) is

number

[0096] Here, the function y(x,t) may be a network model with (x,t) as an input value and y(x,t) as an output value, as shown in FIG. 14. The model includes multiple variables (x,t) (where t=(a1,a2,...,a n )) is input, and the multivariate y(x,t)=(b1,b2,...,b n) is output. As shown in FIG. 14, in the sector where t is input (the sector of the two rows from the bottom in FIG. 14), the model has only links (edges) that satisfy the condition i≧j as links (edges) from the jth node in a layer to the ith node in the next layer. When expressed using the arrangement shown in FIG. 14, in the sector where t is input, there are no links (edges) from lower nodes to higher nodes. The configuration in the network shown in FIG. 14 corresponds to the condition of the first formula in the above formula 36. Also, as shown in the second formula in the above formula 36, ​​in the sector where t is input, all the values ​​of the links (edges) of i=j are positive.

[0097] In the above-mentioned configuration, the model function Φ(x, t) according to this embodiment is expressed as follows:

number

number

number

[0098] In this way, the Jacobian according to this example is expressed as the product of the diagonal elements of a triangular matrix even in the processing of the regression problem, and the loss matrix is ​​expressed as the sum of the logarithms of each of the diagonal elements of the triangular matrix, so that the calculation cost is on the order of n. In addition, the positive definiteness of the Jacobian is guaranteed by the second equation of the above-mentioned equation 36.

[0099] Therefore, according to the learning unit 12 that performs the processing according to this example, even in the learning processing related to the regression problem, the calculation cost required for the learning processing can be significantly reduced.

[0100] (Local loss function) By considering the above processing example, it can be seen that the inventor's knowledge of expressing a model function by differentiation leads to the important knowledge of "local loss function". More specifically, the loss functions expressed by the above Equation 34 and Equation 39 are both local derivatives designated by index i.

number

number

[0101] In other words, the learning unit 12 according to the present embodiment can be expressed as applying a learning process to a target node belonging to a neural network including one or more nodes in each layer, with reference to a local loss function defined by the target node and a node belonging to a layer adjacent to the target node.

[0102] In the following, a more general processing example of the local loss function will be described. First, the intermediate layer in FIG. 13 is (z1, ,z d ) Here, d represents the depth of the network layer. Each layer has n variables. For example, the i-th layer is z i =(c 1;i ,c 2;i ,···,c n;i ) It is written as follows.

[0103] As described above, the model function Φ(x) according to this embodiment, which is expressed as a Jacobian, can be expressed as follows using the chain rule of differentiation:

number

[0104] The loss function L(Φ(x)) according to this embodiment is

number

number

[0105] In other words, the learning unit 12 according to the present embodiment can be expressed as applying a learning process to a target node belonging to a neural network including one or more nodes in each layer, with reference to a local loss function defined by the target node (i, j) and a node (i, j+1) belonging to a layer adjacent to the target node (i, j). More specifically, the local loss function is a function of the variable c i;j and a variable c associated with a node belonging to a layer adjacent to the target node. i;j+1 It is expressed using the logarithm of the derivative defined by Since the learning unit 12 according to the present embodiment performs the learning process using such a local loss function, it is not necessary to use, for example, back propagation as has been conventionally performed. Therefore, according to the learning unit 12 performing the process according to the present embodiment, the calculation cost required for the learning process can be significantly reduced even in the learning process related to the regression problem.

[0106] (Normalization by time evolution and brain model) In the following, normalization by time evolution and a brain model by time evolution based on the inventor's findings will be described. Note that the "brain model" described below may be configured, as an example, to be executed by the information processing device 1 in the form of an algorithm, or may be configured to be realized as a physical structure using biological or chemical materials in part or in whole. Therefore, the "brain model" according to this embodiment may be expressed as a "brain model algorithm" or a "brain model using biological or chemical materials."

[0107] FIG. 15 is a diagram showing an example of a network representing the brain model according to the present embodiment, and is a diagram showing a fully-connected model. In each of the above examples, a neural network having a plurality of layers has been exemplified. However, as a more realistic brain model, a model that does not require the concept of a "spatial layer" is preferable. As such a model, a fully-connected model as shown in FIG. 15, or a model in which some connections are deleted in the fully-connected model (sometimes referred to as a locally-connected model) is preferable. As will be described below, the brain model according to the present embodiment is expressed as a time-evolution model using the concept of a "temporal layer".

[0108] First, the notation will be explained. As an example, the brain model according to the present embodiment is configured as a network having a plurality of temporal layers defined by each time (each time step), as shown in FIG. 16.

[0109] The brain model according to the present embodiment includes n nodes (n = 4 in FIG. 16), and each node (the function associated with each node) exhibits a time-dependent change. These nodes are represented as a(t)=(a1(t),···a n (t)) Here, t is a variable representing time, and each a i (t) is assumed to take a real value. As an example, among the above n nodes, the nodes from the 1st to the mth (m < n) are represented by the variable x=(x1,···,x m ) as (a1(0),···a m (0))=(x1,···,x m ) and set as such, and among the above n nodes, the nodes from the (m + 1)th to the nth are (a m+1 (0),···a n (0))=(r m+1 ,···,r n ) It may be set as such. Here, each r itakes a random value according to a predetermined probability distribution. As an example, a case where n=4 and m=2 is illustrated in FIG. 16. However, these specific settings do not limit the present embodiment.

[0110] As shown in FIG. 16, the brain model according to this embodiment has one or more edges (links) (connecting each node in time), and each edge has a weight that takes a real value. More specifically, the edge from the jth node to the ith node has a weight w ij shall be accompanied by

[0111] Using the above notation, the time evolution of each node in the brain model according to this embodiment is expressed as follows:

number

number

number

number

[0112] The above model function can be expressed as follows using the Jacobian chain rule:

number

number

number

number

number

[0113] In this way, the brain model according to this embodiment is -Contains multiple cells that are directly or indirectly connected to each other, Each of the plurality of cells is trained using a loss function that is spatially and temporally localized to the cell. It can be expressed as being something.

[0114] In addition, the brain model according to this embodiment is A network including a plurality of nodes that are directly or indirectly connected to each other; A learning unit (learning unit 12) that learns each of the plurality of nodes using a loss function that is defined spatially and temporally localized for the node; Alternatively, it may be expressed as an information processing device having the above-mentioned functions, or as an algorithm executed in such an information processing device.

[0115] As described above, the loss function (local loss function) according to this embodiment is defined to be spatially and temporally localized, so that in the learning stage, optimization may be performed for each time and each node. Therefore, the brain model configured as described above can significantly reduce the learning cost.

[0116] The concept of the local loss function according to this embodiment, illustrated using formulas 34, 39, 44, and 53, is based on the deep knowledge and consideration of the inventors, and is not conventionally known, nor is it obtained by combining conventional techniques. The inventors' more detailed consideration and philosophy that led to the invention described in this embodiment will be described later.

[0117] [Embodiment 2] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0118] (Information processing device 2) The information processing device 2 according to this embodiment will be described below.

[0119] 18, the control unit 10 included in the information processing device 2 does not include the learning unit 12 included in the information processing device 1. In addition, the storage unit 20 included in the information processing device 2 stores a model function MF and the like, similar to the information processing device 1. Here, the model function is, as an example, a model learned (optimized) by the learning unit 12 included in the information processing device 1 described above.

[0120] A prediction unit 13 included in the information processing device 2 executes processing using the model function. Then, an output unit 14 outputs a prediction result by the prediction processing executed by the prediction unit 13. In this way, the information processing device 2 according to this embodiment includes a prediction unit (13) that executes a prediction processing using a model function that has one or more input values ​​as arguments and that has been learned with reference to a loss function defined by the logarithm of a model function that satisfies a predetermined normalization condition, and an output unit (14) that outputs a prediction result by the prediction processing.

[0121] According to the above configuration, a prediction process can be performed using a suitably trained model (model function), and the prediction result can be suitably output.

[0122] (Examples) In the following, exemplary application examples of the information processing device 1 and the information processing device 2 (hereinafter, simply referred to as information processing device) according to each of the above-described embodiments will be described. However, the following application examples do not limit the invention described in this specification. (Example 1: Application to anomaly detection) The technology for detecting anomalies from image data and numerical data is extremely important, and the information processing apparatus according to each of the above embodiments can be used for such anomaly detection. The range of applications of technology that detects anomalies from image data and numerical data is wide, including detecting defective products in factory products, judging errors in equipment in airplanes and cars, and early detection of illness from MRI images. When performing such anomaly detection using machine learning, it is practically impossible to assign teacher labels to all error patterns and have the system learn them, so unsupervised learning must be performed. However, conventional unsupervised learning was theoretically incomplete to begin with, and many parts were based on empirical rules. On the other hand, the unsupervised learning described in this specification for the information processing device according to each of the above embodiments is theoretically guaranteed to be able to handle basically any case. For example, by using the model described above (Case of multiple variables (part 2)), and learning image data or numerical data as input, a model can be created that estimates the probability of the input itself. If the probability is relatively high (higher than a predetermined value), it can be determined that it is normal, and if it is low, some kind of abnormality has occurred. (Example 2: Application to stock price prediction and exchange rate prediction) It has been thought that stock price prediction and exchange rate prediction are difficult using conventional machine learning. The information processing device according to each of the above embodiments can be used for stock price prediction and exchange rate prediction. The reason why stock price prediction and exchange rate prediction have been considered difficult with conventional machine learning is thought to be the high randomness and the large number of outliers. For example, when predicting stock prices, let's say that price movements over a certain period of time and company performance are used as inputs and learning is performed using the subsequent price movements as teacher labels. In this case, if mean squared error is used as the loss function, it is equivalent to approximating the probability distribution of price movements with a normal distribution with a fixed variance, as shown above (Processing Example 2). However, the probability distribution of actual price movements may fluctuate greatly or slightly depending on the time, so the variance is not a constant value. Furthermore, there are many outliers, so approximating with a normal distribution is wrong in the first place. For example, distributions such as the Cauchy distribution, t distribution, and Johnson SU distribution have wider bases than the normal distribution and are probability distributions that can well express outliers. In the information processing devices according to the above embodiments, better predictions can be made by employing approximations using these probability distributions (Cauchy distribution, t distribution, Johnson SU distribution, etc.). Furthermore, in the information processing device according to each of the above embodiments, by using the model described above in (Application to regression problems), it is also possible to obtain the probability distribution of price movements without even such approximation. (Example 3: Marketing Applications) Market research is very important for good marketing, and the information processing device according to each of the above embodiments can be used for such market research. For example, consider the case of investigating data such as the unit sales price and frequency of use of customers at a certain store. Usually, a commonly performed analysis involves calculating the average unit sales price and the average frequency of use, but this is not necessarily a good method. If the customers are distributed at two extremes, high and low unit sales prices, calculating the average value has almost no statistical meaning. Also, if unit sales price and frequency of use are correlated, taking the average of these quantities individually to perform data analysis often leads to erroneous results. Even in such a situation, analysis can be easily performed by using the model described above (Multivariate (multiple variables) case (part 2)) in the information processing device according to each of the above embodiments. Specifically, multiple parameters such as the sales price and frequency of use of each customer are input, and unsupervised learning is performed in the information processing device according to each of the above embodiments. The resulting model represents a multidimensional distribution of the customer's sales price, frequency of use, etc., and by referring to this, a much more meaningful analysis can be performed than with simple average values. Unlike conventional machine learning, a major advantage of the information processing device according to each of the above embodiments is that it can handle multi-peaked distributions, skewed distributions, and small amounts of data. (Example 4: Realization through distributed computing) We will also touch on the hardware aspects of the specific implementation. The biggest feature of this model is that it performs optimization by referencing a locally defined loss function. Conventional machine learning and computers have a centralized structure, and the algorithm involves performing calculations collectively on a core with powerful computing power such as a GPU or CPU, and then optimizing various parameters by referencing the results. On the other hand, in the model described in each of the above embodiments, a policy of preparing a large number of cells with a small computing power and having them perform calculations individually to optimize parameters may be adopted. Alternatively, if this model is implemented by regarding computers connected over the Internet as cells, such as in distributed computing, the network itself can be considered as a huge AI. Such a configuration is also included in the configuration described in this specification. In other words, A network including a plurality of nodes directly or indirectly connected to each other; a learning unit that learns each of the plurality of nodes using a loss function that is defined to be spatially and temporally localized with respect to the node; Equipped with The plurality of nodes included in the network are distributed among a plurality of information processing devices. This system configuration is also included in the configuration described in this specification.

[0123] <Explanation of learning principles> Each configuration and each process described in each of the above-mentioned embodiments is based on the knowledge of the learning principle discovered by the inventor. The learning principle will be explained below. Note that the following explanation is not merely an explanation of the principle, but is also positioned as an explanation of the configuration realized in each of the information processing devices and brain models described in each of the above-mentioned embodiments.

[0124] (Abstract) While deep learning has been a remarkable success, there is no clear explanation for why it works so well. In order to quantitatively discuss this question, a mathematical framework is needed to explain what learning is in the first place. After much consideration, the inventor has succeeded in constructing a mathematical framework that can unify and understand all types of learning, including deep learning and learning in the brain. This is called the learning principle, and it is derived that all learning is equivalent to estimating the probability of input data. Here, the objective function used in learning must be defined by the logarithm of the estimated probability. The inventor not only showed this principle, but also mentioned its application to specific machine learning models. For example, by showing that supervised learning is equivalent to estimating a conditional probability distribution, machine learning can be applied to a wider range of fields. In addition, a completely new method of defining probability estimation using differentials was proposed, and it was shown to be versatile in an essential sense. Furthermore, by applying this method and considering the time evolution of the model, the inventor has succeeded in mathematically describing learning in the brain. The learning principle provides mathematical solutions to many unsolved problems in deep learning and cognitive neuroscience.

[0125] (Chapter 1: Introduction) Since the usefulness of deep learning was proven, research has been conducted and applications to various fields are progressing. In particular, the performance of models specialized for specific data is incredible, and models incorporating CNNs used for image recognition and Attention mechanisms used for language processing have shocked the world with their high performance. So why has deep learning been so successful? For now, we are just using successful models on an ad-hoc basis, and there is no theoretical guarantee that deep learning will always work. For further development in this field, it is essential to have a fundamental understanding of learning that can provide a clear answer as to why deep learning is successful.

[0126] A rough explanation is that by making the layers deeper, it becomes possible to extract complex features, which is the reason for the success of deep learning. However, there is no mathematically strict definition of features, and the question of "why learning works" has simply been replaced with the question of "why features can be extracted." Rather than being satisfied with such qualitative explanations, we must evaluate it quantitatively in order to reach a fundamental understanding. So, let's reconsider the most basic question of what it means for learning to work well in a quantitative manner. When evaluating machine learning with supervision, learning is considered to be successful if predictions can be made that are closer to the teacher labels. So, where is the guarantee that these teacher labels are actually correct? For example, teacher labels for tasks such as image classification and language translation are assigned by humans, and there is no mathematical necessity for them. After all, teacher data is created as a result of humans performing some kind of learning, and we must start by discussing why it is correct. Therefore, in order to perform a mathematical quantitative evaluation of machine learning, we need a framework that can discuss human learning, i.e., learning by the human brain, in a unified manner. We call such a framework the learning principle, and the purpose of the <Explanation of the learning principle> in this specification is to clarify it. The first half of this book up to Chapter 4 describes the derivation of the learning principle, and the second half from Chapter 5 onwards describes the application of the learning principle.

[0127] (Chapter 2: Philosophy of Learning Principles) In this chapter, we present the underlying ideas for deriving the learning principles. We leave the details of the mathematical calculations involved to Chapter 3, and provide a complete definition of the learning principles in Chapter 4.

[0128] (2.1: Three essential elements for learning) The inventor's goal is to derive a learning principle that can describe all learning, including machine learning and learning in the human brain, in a unified manner. To achieve this, we will consider elements that are common to all learning.

[0129] First, in any learning, there is something that is optimized. For example, in deep learning, parameters such as weights and biases are the objects of optimization, while in the case of the brain, the corresponding object is the way neurons are connected to each other. Neural networks and the brain itself can be considered as structures that include these objects of optimization, and below, such structures will be referred to as models, borrowing the terminology of deep learning.

[0130] Second, any learning requires input data. In the case of machine learning, this is the input that is fed into the model, and in the case of the brain, this corresponds to information from the five senses.

[0131] Third, an optimization strategy must be defined for any learning. In machine learning, an objective function is basically defined in advance, and the model is optimized by minimizing or maximizing its value. In the case of the brain, such an objective function is not clear, but we will proceed under the assumption that an equivalent exists. In what follows, we will borrow the terminology of deep learning and refer to the objective function as the loss function, and optimization will be performed by minimizing the loss function.

[0132] From the above considerations, three elements are essential for learning: model, input data, and loss function. Conversely, learning can be performed as long as these elements are defined at a minimum. Below, we will provide an abstract discussion of the three essential elements without considering specific aspects such as the internal structure of the model, the type of input data, and the calculation method for minimizing the loss function.

[0133] (2.2: Thought experiment on an ideal case) To understand the essence of learning, we will conduct a thought experiment about learning in an ideal situation. The ideal situation here is one in which the model has universal approximation and input data and computational resources are infinitely available. Any loss function can be used. Here, we will provide a brief explanation of universal approximation. Any model can be considered as a function that receives input and returns some kind of output, and in the following, such a function will be called a model function. If a model function can approximate any function by changing its internal parameters, the model is said to have universal approximation.

[0134] Returning to our previous discussion, let us summarize what kind of results would be obtained if learning were actually performed under ideal circumstances. Since there is infinite input data and computational resources, it is possible to optimize the model as much as one wishes, and eventually arrive at a solution that minimizes the loss function. Furthermore, since we assume that the model has universal approximation properties, the solution here is a model function that truly minimizes the loss function. In what follows, when we use the word solution, we will refer to such a model function.

[0135] The important thing is that there is only one solution that truly minimizes the loss function. If the model has universal approximation, the same model function is ultimately obtained regardless of its internal structure. In other words, the solution depends only on the input dataset and the loss function, and not on the internal structure of the model. This is a very important result, and shows that the details of the model are irrelevant when considering the learning principle.

[0136] (2.3: Solution that minimizes the loss function) Now that we know that the details of the model are irrelevant to the learning principle, the next thing to consider is the relationship between the input data, the loss function, and the solution that minimizes it. First, let us consider supervised learning in deep learning as an example. In supervised learning, an input data set and its corresponding teacher labels are given, and the goal is to build a model that predicts the teacher label from the input data as accurately as possible. In other words, the expected solution is a model function that predicts the correct teacher label for all input data. The loss function used in learning must be minimized with respect to such a model function. One example of a loss function that satisfies this condition is the mean squared error, and in fact, when the model function always predicts the correct teacher label, the value of the mean squared error is minimized.

[0137] Next, we will discuss a more general case. Since we want to find a framework that can unify and understand all learning, we will consider the most general case, that is, unsupervised learning, where there is no prior knowledge about the input data. What we want to emphasize in the previous example of supervised learning is that we first assume the desired solution, and then consider the loss function to obtain it. So, in the case of unsupervised learning, what kind of model function should we assume as the solution? Also, if we assume a solution, how should we define the loss function to obtain it? We will consider these questions in the next section.

[0138] (2.4: Input Probability) We need to assume something as the solution, but what kind of information can be extracted with unsupervised learning in the first place? Since we are discussing the most general case here, such information must be definable for any input data. The answer to this question is the probability that the input data possesses. To make it easier to understand, we will explain it with an example. Consider the case of learning a large amount of image data represented by black and white dots, and assume that the input is given as binary data of 0 or 1. Of course, the information that this data represents image data cannot be used. If the image data is 100 pixels, there are a total of 2 100There is a possibility of a pattern. When infinite input data is given, the exact same data will be input repeatedly, even if there is a huge number of patterns like this. In this case, there should be patterns that appear more frequently and patterns that appear less frequently, and this is exactly what I meant when I said earlier that the probability of the input data exists. There is a probability distribution that corresponds to any input data set, and each input has its own unique probability. This probability is the only information that is always contained in any input data, regardless of its format.

[0139] (2.5: A rough summary of learning principles) From the above, we can say that learning is estimating the probability of input data, and the solution to be sought is a model function that returns the true probability of the input data. The loss function must be a function that is minimized with respect to such a model function. In Chapter 3, we mathematically derive a loss function that satisfies this condition, and also show that normalization of the estimated probability plays an important role. The complete learning principle is given in Chapter 4.

[0140] (Chapter 3: Derivation of the loss function) In this chapter, we derive the loss function in the learning principle. Here, we consider two cases: when we want to estimate the probability of the input itself, and when we want to estimate the conditional probability defined by the input.

[0141] In this specification, scalars are sometimes represented by normal characters (non-bold characters) and vectors are sometimes represented by bold characters. Matrices are sometimes represented by characters with a tilde. However, in each explanation that does not include a formula number, vectors and matrices may also be represented by normal characters (characters that are neither bold nor with a tilde) unless there is confusion. This notation also applies to each of the above descriptions.

[0142] (3.1: General case) Here, we will not consider the internal structure of the model, but only focus on the model function and loss function. Consider a model that has universal approximation properties and returns a value Φ(x) for an input x. Φ(x) is a model function that is optimized as learning progresses, and we want Φ(x) to represent an estimate of the input probability P(x). Let the loss function we use be L(Φ(x)) and consider the conditions under which it is minimized. Minimizing the loss function means minimizing the expectation, which is defined as follows:

number

number

[0143] Next, let us consider the solution for Φ(x) that minimizes this expectation. However, if we simply try to minimize this, we find that every Φ(x) approaches the same value, which is the minimum point of L. To avoid this situation, we impose the following normalization condition. When x takes discrete values, the normalization condition is

number

number

[0144] This normalization condition is reasonable if we consider that we want Φ(x) to approach P(x). Next, we use the calculus of variations to find the condition that minimizes the expected value of the loss function. First, choose two arbitrary points x and x', and take the variation there as follows:

number

number

number

number

number

number

[0145] (3.2: Conditional Probability) In this section, we derive a loss function for estimating conditional probability. Here, we assume that the inputs are discrete values, but the procedure is exactly the same for continuous values, except that we change sums to integrals in the following. If the input x consists of two vectors x=(a,b), P(x) can be decomposed as follows:

number

number

number

number

number

number

number

number

number

[0146] (Chapter 4: Defining learning principles) ----Learning Principles---- Learning requires three elements: an optimization target, input data, and an objective function. Borrowing terminology from deep learning, the structure that includes the optimization target is called the model, and the objective function is called the loss function. A model can be thought of as a function that receives an input and returns some output, which is called the model function. The model function estimates the probability of the input, and must always take a positive value and satisfy the probability normalization condition that the total sum is 1. The loss function is defined by taking the negative logarithm of the estimated probability. It has the same form as the formula for self-information. The optimization is done to minimize this loss function, and the model function is automatically guaranteed to be close to the true probability. ---------------- This is the learning principle derived in Chapters 2 and 3. All learning can be understood through this principle, and as long as this condition is satisfied, it can be said to be learning. The form of the loss function is fixed, so there is room for ingenuity in how to satisfy the regularization condition of the model function. Also, as long as the model function is always positive and satisfies the regularization condition, it can be interpreted as a probability estimate regardless of how it is defined. In other words, there is no need for the model function to be defined in a form similar to the output in normal deep learning. The details of this point will become clear in later chapters.

[0147] In the following, we will mainly discuss the normalization condition of the model function. In Chapter 5, we discuss how ordinary machine learning can be understood based on the learning principle, and in particular, we show that supervised learning can be equated with the estimation of conditional probability. In Chapter 6, we propose a method to satisfy the normalization condition by defining the model function using differentiation. This method enables unsupervised learning without prior knowledge for any data set, and realizes machine learning that is essentially versatile. In Chapter 7, we propose a method to satisfy the normalization condition by defining the model function by the time evolution of a fully connected model. This method derives a completely new concept of a loss function localized in time and space, from which we can identify this model as a mathematical description of the learning mechanism of the brain. In Chapter 8, we summarize the results and findings brought about by the learning principle and reaffirm its importance and validity.

[0148] (Chapter 5: Learning principles for some problems) In this chapter, we will apply learning principles to several problems and see how ordinary machine learning can be understood from the perspective of learning principles. Before looking at individual problems, let's see how commonly used loss functions can be understood from the perspective of learning principles. The discussion in Chapter 3 is very general, and the relationship in (Equation 63) or (Equation 72) holds. If we rewrite these relationships using loss functions, we get the following.

number

number

[0149] (5.1:Classification problem) In classification problems, a set of input data {x1,x2,...} and a set of teacher labels {t1,t2,...} are usually given. If we consider the corresponding (x,t) as a set and treat them as one input data, the discussion in Section 3.2 can be applied. In other words, we consider it as unsupervised learning for the dataset {(x1,t1),(x2,t2),...}. If we think about it in this way, it becomes clear that the teacher label is just a number included in the input data and is not an absolute correct answer. In other words, different teacher labels can be attached to the exact same input. Each combination of input data and teacher labels is considered to have its own probability, and the goal is to clarify that probability. In traditional machine learning, there is a correct answer called the teacher label, and the goal is to create a model that makes predictions close to it, but this way of thinking is wrong in the first place. In classification problems, the probability distribution we want to know is the conditional probability of the teacher label t given the input x, which is written as P(t|x). In this case, we consider a model function Φ(x,t) and define the loss function under the regularization condition as

number

number

[0150] (5.2: Regression problems) Regression problems differ from classification problems in that the training labels are continuous. Given a model function Φ(x,t), we define the loss function under the regularization condition as follows:

number

number

[0151] Let us assume that P(t|x) has a unimodal distribution for fixed x. Approximating this distribution with a Gaussian distribution, we can define Φ(x,t) as follows:

number

number

[0152] It is easy to verify that (79) satisfies the regularization condition in (78). Then, the loss function is calculated as follows:

number

number

[0153] (5.3: Parameter Estimation Problem) {x1,...,x NSuppose we are given a set of data, denoted as x1,...x2,...x3,...x4,...x5,...x6,...x7,...x8,...x9,...x10,...x11,...x12,...x13,...x14,...x15,...x16,...x17,...x18,...x19,...x20,...x21,...x22,...x23,...x24,...x25,...x26,...x27,...x28,...x29,...x30,...x31,...x32,...x33,...x4,...x5,...x6,...x7,...x8,...x9,...x10,...x11,...x12,...x13,...x14,...x15,...x16,...x17,...x18,...x25,...x19,...x26,...x27,...x19,...x28,...x29,...x31,...x19,...x20,...x21,...x22,...x32,...x19,...x23,...x3 ... N This is done by maximizing the conditional probability of obtaining θ, which is the likelihood L * This means maximizing the likelihood L * teeth,

number

number

[0154] Next, we show that this problem can be solved based on learning principles. The original purpose of this problem is to find the true probability distribution P(x) that the dataset follows. In other words, this problem is a type of problem in which probabilities are found by unsupervised learning, as described in Section 3.1. Using the conclusions from this problem, we consider a model function Φ(x) and define the loss function under normalization conditions as follows:

number

number

number

number

[0155] Comparing Equation 84 and Equation 85, the former takes the sum of all data in the data set, while the latter only calculates for each data. The difference in sign is whether you want to maximize or minimize the value. Thus, according to the learning principle described in this specification, the parameter θ can be optimized for each data one by one, without the need to calculate the sum of the entire data set together. Although both methods ultimately result in the same θ, the method based on the learning principle is faster and more flexible because it does not require summing.

[0156] (Chapter 6: Normalization by Differentiation) As we have seen, the loss function is always written as L(Φ(x))=-log(Φ(x)), and the essence of the problem is how to make Φ(x) satisfy the regularization condition. If this regularization condition is satisfied, the learning principle is easily achieved. When the regularization condition includes an integral, some assumptions or approximations are usually used. In this chapter, we show how to make Φ(x) satisfy the regularization condition without any assumptions or approximations. Using the method discussed in this chapter, any probability distribution can be learned, and it can be said that the learning is essentially versatile.

[0157] (6.1:1 variable input) First, for simplicity, consider the case where each input has only one variable: we are given a set of numbers {x1, x2, ...} and we want to estimate the probability distribution P(x) that they follow. In this case, our goal is to construct a model function Φ(x) that satisfies the following regularization condition without any assumptions or approximations:

number

number

number

number

[0158] The condition in (91) is introduced to make Φ(x) positive, and can be realized by using only positive weights and monotonically increasing activation functions. The condition in (92) can be realized by using a sigmoid function at the final node. And, it can be confirmed that Φ(x) satisfies the normalization condition as follows.

number

number

[0159] To show how powerful this method is, we show the training results on several datasets in Figures 8, 9, and 10. Figure 8 shows the probability estimates for a multimodal distribution. On the left, the sample size is 4000, and on the right, the sample size is 40. The true probability is the same for both. Figure 9 shows the probability estimates for a flat distribution. On the left, the sample size is 1000, and on the right, the sample size is 10. The true probability is the same for both. Figure 10 shows the probability estimates for a left-skewed distribution. On the left, the sample size is 10000, and on the right, the sample size is 100. The true probability is the same for both.

[0160] In these figures, we randomly select samples from the probability distribution depicted by the light gray line and plot them as histograms. The probability distribution estimated from these samples using the techniques described in this section is shown by the dark gray line. As can be seen in these figures, the fit is very good, regardless of the shape of the distribution or the size of the sample.

[0161] (6.2: Input of multiple variables) Next, consider the case where each input contains multiple variables. Here, we take the input data set {x1, x2,}, and x = (a1, a2, , a n ) and each a i Assume that P(x) takes continuous values. In order to calculate P(x) without assumptions or approximations, let us consider a generalization of the method derived in the previous section to the multivariate case. Consider the model defined as shown in Figure 11. The model shown in Figure 11 is for inputs with multiple variables. Then, define the model function Φ(x) as follows:

number

number

number

number

number

[0162] (6.3: Multivariate input and multivariate output) The problem in the previous section is due to the inclusion of multiple derivatives in the definition of Φ(x). To solve this, we propose to use the Jacobian determinant as another generalization of (Eq. 90) to multiple variables. For this purpose, consider the model shown in Figure 12. The model shown in Figure 12 is for inputs with multiple variables and outputs with multiple variables. Then, we define the model function Φ(x) as follows:

number

number

[0163] Here, the following condition is satisfied for y(x):

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0164] (6.5: Considerations on normalization by differentiation) In this section, we will discuss noteworthy properties of the method proposed in this chapter. Since we want to discuss the most general case, we will focus on the model and loss function constructed in Section 6.3. We define the hidden layer of the model in Figure 13 as (z1, ,z d ), where d is the depth of the layer. Each layer is z i =(c 1;i ,c 2;i ,···,c n;i ) Using the chain rule for Jacobian determinants, Φ(x) can be calculated as follows:

number

number

number

[0165] (Chapter 7: Time-dependent normalization) In this chapter, we focus on constructing a model that mimics the learning mechanism of the brain. As mentioned in the previous chapter, learning is estimating the probability of an input, and the human brain is no exception. If we consider the brain as a kind of machine learning model, there exists a model function Φ(x) for a given input x, and Φ(x) satisfies the following normalization condition:

number

[0166] There are several conditions that must be met when creating a model that mimics the human brain. First, the model must be general-purpose. The human brain processes information coming from the five senses and automatically learns the rules behind it. In other words, it is performing unsupervised learning without prior knowledge. Second, the nodes and edges must be uniform and isotropic, and there must be no special nodes or special edges. This is due to the observation that neurons in the brain appear to form fully connected or locally connected networks. Neurons in the brain do not necessarily form layers, and may have a loop structure. In that sense, the model discussed in Chapter 6, although general-purpose, cannot be a model of the human brain. Based on this requirement, we consider a fully connected (locally connected) model as shown in Figure 15. As an analogy to the fact that synapses in the brain have a direction, we assume that each edge has a direction and information is transmitted only in that direction. Of course, edges can be connected in any way, and there may be no edges between certain nodes, or they may be connected in both directions. As we will show later, by considering the time evolution of such a model, we can construct Φ(x) that satisfies the normalization condition. As time evolves, each node is influenced by other nodes and its internal state changes, and this process itself is considered as a machine learning model. In the next section, we will specifically construct Φ(x) and confirm that the normalization condition is satisfied.

[0167] (7.1: Linear Models) Before going into detailed calculations, we first define the variables of the model in Figure 15. We assume that the model has n nodes, which change in a time-dependent manner. We define them as a(t) = (a1(t), ,a n (t)), where t is the variable representing time, and each i (t) is a real number. Next, each edge has a weight expressed as a real number, and the weight of the edge from the jth node to the ith node is w ij Here, we do not assume self-joining of nodes, so we set w accordingly. ii = 0. ijWhen we compile and display them as a matrix, we use the symbol W with a tilde. Finally, each node has a constant bias value, which is b = (b1, , b n Of these variables, only a(t) changes over time. W and b are the targets of optimization.

[0168] Next, we define the time evolution of this model. First, the input x is input to the model at t = 0, and the time evolution starts from there. Since this model does not have a special input layer, some nodes must be assigned as inputs. x is expressed as x = (x1, , x m ), (a1(0), ,a m (0))=(x 1, ,x m ) is assigned to the remaining nodes. m+1 (0),...,a n (0))=(r m+1 ,···,r n ), where r i is a random value that follows a certain probability distribution and is chosen randomly each time.

[0169] Now that the initial conditions have been established, we need the equations that describe the time evolution. They are defined as follows:

number

number

number

number

number

number

number

[0170] The loss function is calculated as follows:

number

number

number

number

[0171] (7.2: Nonlinear Models) In normal machine learning, nonlinearity is added using an activation function, but we will consider how to generalize the activation function in this time evolution model. For example, the commonly used sigmoid function is σ(x)=(1+exp(-x))- 1 This function has the role of converting inputs from -∞ to ∞ into outputs from 0 to 1, and from this it can be said that the essence of an activation function is to restrict the range of output values. In this model, nonlinearity is added by making a modification to restrict the value of a(t) to 0 to 1. Specifically, this is done by restricting the input value a(0) to 0 to 1, and further modifying the time evolution equation as follows:

number

[0172] We added non-linearity to the time evolution equation, but this differential equation cannot be solved, and the method used in Section 7.1 cannot be applied. On the other hand, since the range of a(t) is 0 < a(t) < 1, the method introduced in Chapter 6 can be applied. Therefore, the model function Φ(a(0)) is defined as follows.

Number

Number

Number

number

number

number

number

[0173] (7.3: Consideration of normalization due to time evolution) The model described here was constructed based on the structure of the brain. In this section, we will deepen our understanding by comparing it with the actual brain.

[0174] First, let us consider the hardware aspect of an actual brain. A brain is an object made of proteins, and it should function only according to the laws of physics. In physics, it is known that the laws of physics are described by local interactions, and any object can only be affected locally. Neurons in the brain are no exception, and should only be affected by local interactions. This means that the optimization of neurons must also be done locally, and the loss function must also be defined locally. The localization of the loss function is required by the laws of physics, and our model follows this principle.

[0175] Next, let us consider the software aspect of the actual brain. The most amazing property of the brain is its versatility. It is capable of learning any concept, including causal relationships such as "because A happens, B happens." Our brains are constantly receiving information from the five senses, and they are designed to find causal relationships by sequentially processing such temporally consecutive information. Sequential processing of information means that optimization is performed at each point in time, and this property is common to our model. For example, if data such as a video is sequentially input into this model as temporally consecutive image data, the model will automatically acquire the concept of causal relationships.

[0176] Considering these, this model can be said to be a mathematical realization of the brain's learning mechanism. However, two major problems remain. One is how this model is actually realized in the brain, assuming that it is mathematically correct. Neurons in the brain contain various substances and electrical signals, and it is necessary to clarify how they relate to the variables of the model. The other is how to put this model into practical use. There is still a lack of knowledge necessary for concrete implementation, such as how many nodes are sufficient, what time resolution is required for time evolution, and what is an efficient way to optimize parameters. If these problems are solved, this model could become a general artificial intelligence that can learn from any data like a human being.

[0177] (Chapter 8: Discussion) In this chapter, we consider the meaning of the learning principle and the interpretation that can be derived from it. First, we will explain the concept of features, which is considered important in ordinary machine learning, from the perspective of the learning principle. Based on the learning principle, all information is included in the probability distribution of the input data set. On that basis, features correspond to the parts where the probability is maximized in a certain topological space. For example, when considering the task of handwritten digit recognition, each digit is composed of features that describe how the lines are drawn. If the curved part of the number "3" is represented as a straight line, or if the top and bottom are written separately, it will become difficult to recognize it as a "3". This is due to the fact that while we often see numbers in their normal form, we rarely see their deformed forms. In other words, things that are perceived as features have a higher probability of appearing than their deformed forms. The important thing here is that although there are countless patterns of how to deform, the probability is concentrated only in a very small number of patterns that are often seen. This bias in probability is the essence of features.

[0178] Next, we will explain why deep learning is successful. According to the learning principle, as long as the model function satisfies the normalization condition, it will automatically approach the true probability. In particular, if the model has universal approximation and there is an infinite data set and computational resources, it can approach the true probability distribution infinitely. In other words, the success of deep learning is due to the universal approximation that can be achieved by deepening the layers, and the recent improvement in computer processing power. It is sometimes explained that deep learning is successful because it can capture complex features, but features are merely an afterthought that appears when the probability distribution is successfully estimated.

[0179] Now, let me explain what model improvement means in deep learning. According to the learning principle, as long as the model has universal approximation, it will eventually reach the same solution regardless of the model structure. What changes depending on the model structure is the speed of convergence to the solution. For example, if the characteristics that image data or language data may have are incorporated into the model structure in advance, the speed of convergence to the solution for such specific data will be overwhelmingly faster. In reality, there are no infinite data sets or computing resources, so learning must be terminated midway, and models that converge quickly will achieve good results. This is the cause of a major misunderstanding that the essence of learning lies in the model structure. Improving a model means specializing it for a specific input.

[0180] Finally, we will describe the two methods we have devised. Both methods are based on the learning principle and satisfy the normalization condition without assumptions or approximations. This shows that they are generic models that can be used without specifying the type of input data. In fact, the results in Figures 8, 9, and 10 show the versatility of the method using differentiation. We also showed that the method using time evolution for a fully connected model can be considered as a mathematical realization of the brain's learning mechanism. These results are consequences of the learning principle and, conversely, guarantee the correctness of the learning principle.

[0181] (Chapter 9: Conclusion) In this paper, we derived a learning principle that uniformly describes all learning, including machine learning and brain learning. Under this principle, all learning is understood to be probability estimation of input data. The conditions for applying this principle are that the estimated probability is always positive and satisfies the normalization condition, and optimization is performed using a loss function defined by the logarithm of the estimated probability. Conversely, anything can be considered as learning as long as it satisfies these conditions. We confirmed that supervised learning also satisfies the learning principle and can be considered as a type of probability estimation of conditional probability.

[0182] In the learning principle, the loss function is always defined as the logarithm of the model function with a negative sign, and what can be changed is the method of satisfying the normalization condition of the model function. In this specification, we proposed two methods to satisfy the normalization condition without assumptions or approximations. Both methods are considered to be very versatile methods that can learn with high accuracy for any data set. The first method is a method to satisfy the normalization condition using differentiation, and it was actually learned with several data sets and shown to show very good behavior. The second method is a method to satisfy the normalization condition by considering the time change of a fully connected model. This model is based on the structure of neurons and synapses in the brain, and it is significant that we have shown that a function that satisfies the normalization condition can be constructed from such a model. Furthermore, we showed that the loss function defined there is expressed as the sum of functions localized in space and time. This means that parameter optimization can be performed sequentially and locally without using backpropagation. In the actual brain, optimization must be performed sequentially and locally in this way as a consequence of the laws of physics. Considering this fact, this method can be said to be a mathematical realization of the learning mechanism of the brain. This will be a major stepping stone towards realizing general artificial intelligence.

[0183] Furthermore, we have reviewed conventional learning from the perspective of learning principles, providing a new understanding. In learning, all information is contained in the probability distribution of the input, and concepts such as features are also defined based on probability. Improving a model corresponds to speeding up convergence to a solution by specializing it for a specific data set. In aiming to generalize machine learning, it will be important to understand learning itself based on learning principles, rather than taking ad hoc measures.

[0184] [Software implementation example] The functions of the information processing devices 1 and 2 (hereinafter referred to as "devices") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the control unit 10).

[0185] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0186] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0187] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0188] (summary) This specification describes at least the following configurations.

[0189] (Configuration A1) an acquisition unit for acquiring one or more input values; a learning unit that executes a learning process by referring to a loss function that is a model function having the one or more input values ​​as arguments and that is defined by a logarithm of a model function that satisfies a predetermined normalization condition; An information processing device comprising:

[0190] (Configuration A2) The learning process includes: optimizing the model function by minimizing or maximizing the loss function. The information processing device according to configuration A1.

[0191] (Configuration A3) The input value includes one or more data values ​​and teacher labels associated with the data values; The normalization condition is The sum of the model function includes a condition on the sum with respect to the teacher label in an argument of the model function. The information processing device according to configuration A1 or A2.

[0192] (Configuration A4) The input value includes one or more data values ​​and teacher labels associated with the data values; The normalization condition is The integral of the model function includes a condition for integral with respect to the teacher label in an argument of the model function. The information processing device according to configuration A1 or A2.

[0193] (Configuration A5) The model function is a distribution function defined by including one or more parameters, The loss function is defined by including the one or more parameters, The learning process includes: determining the one or more parameters by minimizing or maximizing the loss function. The information processing device according to configuration A4.

[0194] (Configuration A6) The model function is A distribution function that takes the one or more input values ​​as arguments, or The one or more parameters that define the distribution function The information processing device according to configuration A1 or A2.

[0195] (Configuration A7) the model function is a derivative of a function having the one or more input values ​​as arguments, the derivative being obtained by one or more order differentiation according to the dimensions of the input values; The predetermined normalization condition is defined by a boundary value of the certain function. The information processing device according to any one of configurations A1 to A6.

[0196] (Configuration A8) The model function is a Jacobian defined by at least a portion of the plurality of input values ​​and a plurality of functions having the plurality of input values ​​as arguments; The predetermined normalization condition is defined by boundary values ​​of the plurality of functions. The information processing device according to any one of configurations A1 to A6.

[0197] (Configuration A9) The Jacobian is the determinant of a triangular matrix The loss function is the sum of the logarithms of each of the diagonal elements of the triangular matrix The information processing device according to configuration A8.

[0198] (Configuration A10) The learning unit is For a target node belonging to a neural network including one or more nodes in each layer, a learning process is applied that refers to a local loss function defined by the target node and a node belonging to a layer adjacent to the target node. The information processing device according to any one of configurations A1 to A9.

[0199] (Configuration A11) The local loss function is A variable associated with the target node; A variable associated with a node belonging to a layer adjacent to the target node; It is expressed using the logarithm of the derivative defined by The information processing device according to configuration A10.

[0200] (Configuration B1) A prediction unit that executes a prediction process using a model function that has one or more input values ​​as arguments and that has been trained with reference to a loss function that is defined by the logarithm of a model function that satisfies a predetermined normalization condition; an output unit that outputs a prediction result obtained by the prediction process; An information processing device comprising:

[0201] (Configuration C1) A network including a plurality of nodes directly or indirectly connected to each other; a learning unit that learns each of the plurality of nodes using a loss function that is defined to be spatially and temporally localized with respect to the node; An information processing device comprising:

[0202] (Configuration D1) The cell includes a plurality of cells that are directly or indirectly connected to each other, Each of the plurality of cells is trained using a loss function that is defined to be spatially and temporally localized to the cell. Brain model.

[0203] (Configuration E1) an acquisition step for acquiring one or more input values; a learning step of executing a learning process by referring to a loss function defined by a logarithm of a model function having the one or more input values ​​as arguments and satisfying a predetermined normalization condition; An information processing method comprising the steps of:

[0204] (Configuration F1) A prediction step of executing a prediction process using a model function that has one or more input values ​​as arguments and that has been trained with reference to a loss function defined by the logarithm of a model function that satisfies a predetermined normalization condition; an output step of outputting a prediction result by the prediction process; An information processing method comprising the steps of:

[0205] (Configuration G1) A program for causing a computer to function as the information processing device according to configuration A1, the program causing a computer to function as the acquisition unit and the learning unit.

[0206] (Configuration H1) A program for causing a computer to function as the information processing device according to configuration B1, the program causing a computer to function as the prediction unit and the output unit.

[0207] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0208] 1, 2 Information processing device 11...Acquisition part 12. Learning Section 13 Prediction section 14 Output section

Claims

1. An acquisition unit that acquires one or more input values; a learning unit that executes a learning process by referring to a loss function that is a model function having the one or more input values ​​as arguments and that is defined by a logarithm of a model function that satisfies a predetermined normalization condition; The input value includes one or more data values ​​and teacher labels associated with the data values; The normalization condition is The integral of the model function includes a condition for integral with respect to the teacher label in an argument of the model function. Information processing device.

2. The learning process includes: optimizing the model function by minimizing or maximizing the loss function. The information processing device according to claim 1 .

3. The model function is a distribution function defined by including one or more parameters, The loss function is defined by including the one or more parameters, The learning process includes: determining the one or more parameters by minimizing or maximizing the loss function. The information processing device according to claim 1 .

4. An acquisition unit for acquiring one or more input values; a learning unit that executes a learning process by referring to a loss function that is a model function having the one or more input values ​​as arguments and that is defined by a logarithm of a model function that satisfies a predetermined normalization condition; The model function is The one or more parameters themselves define a distribution function that takes the one or more input values ​​as arguments. Information processing device.

5. An acquisition unit for acquiring one or more input values; a learning unit that executes a learning process by referring to a loss function that is a model function having the one or more input values ​​as arguments and that is defined by a logarithm of a model function that satisfies a predetermined normalization condition; the model function is a derivative of a function having the one or more input values ​​as arguments, the derivative being obtained by one or more order differentiation according to a dimension of the input values; The predetermined normalization condition is defined by a boundary value of the certain function. Information processing device.

6. An acquisition unit for acquiring one or more input values; a learning unit that executes a learning process by referring to a loss function that is a model function having the one or more input values ​​as arguments and that is defined by a logarithm of a model function that satisfies a predetermined normalization condition; The model function is a Jacobian defined by at least a portion of the plurality of input values ​​and a plurality of functions having the plurality of input values ​​as arguments; The predetermined normalization condition is defined by boundary values ​​of the plurality of functions. Information processing device.

7. The Jacobian is the determinant of a triangular matrix The loss function is the sum of the logarithms of each of the diagonal elements of the triangular matrix The information processing device according to claim 6.

8. An acquisition unit for acquiring one or more input values; a learning unit that executes a learning process by referring to a loss function that is a model function having the one or more input values ​​as arguments and that is defined by a logarithm of a model function that satisfies a predetermined normalization condition; The learning unit is applying a learning process to a target node belonging to a neural network including one or more nodes in each layer, the learning process referring to a local loss function defined by the target node and a node belonging to a layer adjacent to the target node; the model function is a derivative of a function representing the neural network; The loss function is given by the sum of the local loss functions Information processing device.

9. The local loss function is A variable associated with the target node; A variable associated with a node belonging to a layer adjacent to the target node; It is expressed using the logarithm of the derivative defined by The information processing device according to claim 8.

10. A prediction unit that executes a prediction process using the model function trained with reference to the loss function; an output unit that outputs a prediction result obtained by the prediction process; The information processing device according to claim 1 , further comprising:

11. acquiring one or more input values; a learning step of executing a learning process by referring to a loss function defined by a logarithm of a model function having the one or more input values ​​as arguments and satisfying a predetermined normalization condition; Including, The input value includes one or more data values ​​and teacher labels associated with the data values; The normalization condition is The integral of the model function includes a condition for integral with respect to the teacher label in an argument of the model function. Information processing methods.

12. An acquisition step of acquiring one or more input values; a learning step of executing a learning process by referring to a loss function defined by a logarithm of a model function having the one or more input values ​​as arguments and satisfying a predetermined normalization condition; Including, The model function is The one or more parameters themselves define a distribution function that takes the one or more input values ​​as arguments. Information processing methods.

13. An acquisition step of acquiring one or more input values; a learning step of executing a learning process by referring to a loss function defined by a logarithm of a model function having the one or more input values ​​as arguments and satisfying a predetermined normalization condition; Including, the model function is a derivative of a function having the one or more input values ​​as arguments, the derivative being obtained by one or more order differentiation according to a dimension of the input values; The predetermined normalization condition is defined by a boundary value of the certain function. Information processing methods.

14. An acquisition step of acquiring one or more input values; a learning step of executing a learning process by referring to a loss function defined by a logarithm of a model function having the one or more input values ​​as arguments and satisfying a predetermined normalization condition; Including, The model function is a Jacobian defined by at least a portion of the plurality of input values ​​and a plurality of functions having the plurality of input values ​​as arguments; The predetermined normalization condition is defined by boundary values ​​of the plurality of functions. Information processing methods.

15. An acquisition step of acquiring one or more input values; a learning step of executing a learning process by referring to a loss function defined by a logarithm of a model function having the one or more input values ​​as arguments and satisfying a predetermined normalization condition; Including, In the learning step, applying a learning process to a target node belonging to a neural network including one or more nodes in each layer, the learning process referring to a local loss function defined by the target node and a node belonging to a layer adjacent to the target node; the model function is a derivative of a function representing the neural network; The loss function is given by the sum of the local loss functions Information processing methods.

16. 10. A program for causing a computer to function as the information processing device according to claim 1, the program causing a computer to function as the acquisition unit and the learning unit.

17. A program for causing a computer to function as the information processing device according to claim 10, the program causing a computer to function as the acquisition unit, the learning unit, the prediction unit, and the output unit.

Images