Training device and inference device

Abstract

One aspect of the present invention is a training device provided with a control unit for training a MIMO neural network, which is a neural network comprising: an input layer having N unit nodes (N is an integer 2 or greater), each unit node being a group of nodes of an input layer and receiving one unit input data that is data in a predetermined format; and an output layer that outputs a predefined amount of values corresponding to each of the N unit input data inputted to the input layer. The calculation amount of the MIMO neural network is less than N times the calculation amount of a SISO neural network which is a neural network including an input layer having one unit node and an output layer for outputting a value of the amount corresponding to one unit input data inputted to the input layer. The calculation amount of the neural network is the total number of times of arithmetic processing required for execution of the neural network, and the unit input data is the set of one or a plurality of values.

Description

Learning and inference devices

[0001] The present invention relates to a learning device and an inference device.

[0002] In recent years, research has been conducted into solving inverse problems using neural networks.

[0003] Ben Mildenhall, et al. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis." arXiv:2003.08934v1, (2020)

[0004] However, until now, only neural networks that represent one-input, one-output functions have been used, and if multiple pieces of data were to be input into a neural network, the only option was to run the neural network multiple times, which meant that the amount of calculations increased in proportion to the amount of data to be input.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for suppressing an increase in the amount of calculation.

[0006] One aspect of the present invention is a learning device comprising: a control unit that performs training of a MIMO neural network, which is a neural network comprising an input layer having N (N is an integer greater than or equal to 2) unit nodes, which are a group of nodes in the input layer and each node accepts one unit input data, which is data in a predetermined format; and an output layer that outputs a value of a predefined quantity corresponding to each of the N unit input data input to the input layer; wherein the computational complexity of the MIMO neural network is less than N times the computational complexity of a SISO neural network, which is a neural network comprising an input layer having one unit node and an output layer that outputs a value of the quantity corresponding to one unit input data input to the input layer; the computational complexity of the neural network is the number of calculation processes required to execute the neural network; and the unit input data is a set of one or more values.

[0007] One aspect of the present invention is an inference device comprising: a control unit that performs training of a MIMO neural network, which is a neural network comprising an input layer having N (N is an integer greater than or equal to 2) unit nodes, each unit node being a group of nodes in the input layer that accepts one unit input data, which is data in a predetermined format; and an output layer that outputs a predefined quantity value corresponding to each of the N unit input data input to the input layer; wherein the computational complexity of the MIMO neural network is less than N times the computational complexity of a SISO neural network, which is a neural network comprising an input layer having one unit node and an output layer that outputs the quantity value corresponding to one unit input data input to the input layer; the computational complexity of the neural network is the number of calculation processes required to execute the neural network; and the unit input data is a set of one or more values.

[0008] The present invention makes it possible to suppress an increase in the amount of calculation.

[0009] An explanatory diagram for explaining an inference system according to an embodiment. A first diagram showing an example of experimental results according to an embodiment. A second diagram showing an example of experimental results according to an embodiment. A diagram showing an example of the hardware configuration of a learning device according to an embodiment. A diagram showing an example of the flow of processing executed by a learning device according to an embodiment. A diagram showing an example of the hardware configuration of an inference device according to an embodiment. A diagram showing an example of the flow of processing executed by an inference device according to an embodiment.

[0010] 1 is a diagram illustrating an inference system 100 according to an embodiment. The inference system 100 includes a learning device 1 and an inference device 2.

[0011] The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92 connected via a bus.

[0012] The control unit 11 executes a learning process. The learning process is a process for learning a learning object. The learning object in the learning process is a MIMO (Multi-input Multi-output) neural network. A MIMO neural network is a type of neural network. The input layer of a MIMO neural network has N unit nodes (N is an integer equal to or greater than 2).

[0013] A unit node is a group of nodes in the input layer of a MIMO neural network that accepts one unit of input data. Unit input data is data in a predetermined format and is a set of one or more values. Unit input data is data in a predetermined format and is, for example, a set of one or more values. Unit input data is, for example, a multidimensional vector. Unit input data is, for example, a multidimensional vector in which at least one of the components is a continuous value. Unit input data is, for example, a set of multidimensional vectors. Unit input data is, for example, a set of multidimensional vectors in which at least one of the components is a continuous value.

[0014] Therefore, for example, if the input layer of a MIMO neural network has 3 × N nodes, a unit node is a group of 3 nodes. In this case, the input layer of the MIMO neural network can accept N units of input data represented by a three-dimensional vector.

[0015] The output layer of the MIMO neural network outputs the value of a defined quantity corresponding to each of the N units of input data input to the input layer of the MIMO neural network. The defined quantity is a quantity defined in advance. The defined quantity may be, for example, a physical quantity, a quantity of a predetermined modality, time-series data, or a combination thereof. The defined quantity may be, for example, a color, opacity, waveform, or a combination thereof. Opacity is also called volume density.

[0016] Here, some definitions are made to further explain MIMO neural networks. The number of calculations required to execute each hidden layer of a neural network, whose total number of hidden layers (including the output layer) is I (I is an integer equal to or greater than 1), is defined as f(i) (i is an integer equal to or greater than 1 and equal to or less than I). The number of calculations required to execute a neural network is defined as the sum of f(i) from i=1 to i=I. Therefore, the number of calculations required to execute a neural network means the total number of calculations required to execute all hidden layers of the neural network.

[0017] For ease of explanation, the number of calculation processes required to execute a neural network will be referred to simply as the computational complexity. Note that the computational complexity may be expressed by any index that can express the computational complexity. For example, the computational complexity may be expressed by the number of floating-point operations (FLOPs) or the number of multiply-accumulate operations (MACs).

[0018] More specifically, FLOPs represents the number of calculation processes when one unit of processing is a single arithmetic operation such as addition, subtraction, multiplication, etc. On the other hand, MACs represents the number of calculation processes when one unit of processing is a multiplication and accumulation operation. For example, if the input size and output size of a certain layer of a multilayer perceptron (MLP) are f ｉｎ and f ｏｕｔ Then, the weight and bias parameters of a layer are f ｉｎ ×f ｏｕｔ and f ｏｕｔ is.

[0019] In this case, the FLOPs of this layer are 2×f ｉｎ ×f ｏｕｔ +f ｏｕｔ In this equation, the first term, 2×f ｉｎ ×f ｏｕｔ represents the FLOPs when the input is multiplied by the weight, and the second term f ｏｕｔrepresents the FLOPs when the bias is added. Also, in the layer of the multilayer perceptron, the MACs when the weight is applied to the input are f ｉｎ ×f ｏｕｔ is.

[0020] Similarly, in the case of other neural networks such as a convolutional neural network (CNN), a recurrent neural network (RNN), an attention network, or a transformer network, once the input and network structure are determined, the FLOPs or MACs are uniquely determined. Now, a further explanation of the MIMO neural network will be provided.

[0021] A MIMO neural network is a neural network whose computational complexity is less than N times that of a SISO (Single-input Single-output) neural network. A SISO neural network is a neural network that includes an input layer with one unit node and an output layer that outputs a defined quantity value corresponding to one unit input data input to the input layer. Therefore, a SISO neural network is a neural network that accepts one unit input data.

[0022] Therefore, for example, a MIMO neural network may be a neural network in which (N-V) (V is an integer between 2 and N) of the N unit nodes in the input layer of the MIMO neural network are each connected to a network that is the SISO neural network minus the input layer and the output layer, and the remaining V unit nodes are connected to one network that is the SISO neural network minus the input layer and the output layer.

[0023] Therefore, for example, a MIMO neural network may be a neural network in which the remaining network, excluding the input layer and the output layer from the MIMO neural network, is the same as the remaining network, excluding the input layer and the output layer from the SISO neural network.

[0024] <Effects of MIMO Neural Networks> The effects of MIMO neural networks will now be described. Generally, the computational complexity of a neural network is proportional to the number of arithmetic operations required to execute the network. In the case of a MIMO neural network, even though it accepts N unit input data and outputs values ​​corresponding to each unit input data, some or all of the computational operations in the intermediate layer that processes each unit node are shared among two or more unit nodes. Therefore, the number of arithmetic operations required by a MIMO neural network is less than N times the number of arithmetic operations required to execute a SISO neural network. As a result, in the case of a MIMO neural network, even if the number of unit input data increases N times, the computational complexity does not increase N times, but is less than N times. Therefore, a MIMO neural network can suppress an increase in the computational complexity. This is an example of the effect of a MIMO neural network.

[0025] <More Specific Explanation of SISO Neural Network> To facilitate understanding of MIMO neural networks, first, a more specific explanation of SISO neural networks will be given.

[0026] (Specific example when the dimension of unit input data is one-dimensional) A case where the dimension of unit input data input to the SISO neural network is one-dimensional will be described. The one-dimensional unit input data input to the SISO neural network is, for example, sound. In this case, at time x １ (x １ The SISO neural network expresses the function of the following equation (1), which converts the time x (x is a real number) into the signed magnitude s (s is a real number) of the sound. １ is an example of a one-dimensional coordinate system for a time series.

[0027]

[0028] (Specific example of when the dimension of unit input data is two-dimensional) A case where the dimension of unit input data input to the SISO neural network is two-dimensional will be described. The two-dimensional unit input data input to the SISO neural network is, for example, an image. In this case, the two-dimensional coordinates x ２ The SISO neural network represents a function that converts (each coordinate value is a real number) into a pixel value c.

[0029] The pixel value c represents, for example, a color. The color may be represented by any value that can represent the color. For example, the color may be represented in any color space. The color space may be a three-dimensional color space such as RGB, HSV, or HSB, or a four-dimensional color space such as RGBA or CMYK. The color may be represented, for example, in grayscale, or may be represented by a binary value indicating one of two colors, such as white or black.

[0030] For example, if the pixel value c represents a position in a three-dimensional color space such as RGB, then the two-dimensional coordinate x ２ The function for converting the pixel value c is expressed by the following equation (2).

[0031]

[0032] (Example of a case where the dimension of unit input data is three-dimensional) A case where the dimension of unit input data input to the SISO neural network is three-dimensional will be described. The three-dimensional unit input data input to the SISO neural network is, for example, three-dimensional space data. In this case, three-dimensional coordinates x ３ The SISO neural network expresses a function that converts a point (each coordinate value is a real number) into a value c of a point in three-dimensional space. The value c of a point in three-dimensional space represents, for example, a color.

[0033] For example, if the value c of a point in three-dimensional space represents a position in a three-dimensional color space such as RGB, then the three-dimensional coordinate x ３The function that converts into the value c of a point in three-dimensional space is expressed by the following equation (3).

[0034]

[0035] The three-dimensional unit input data is not limited to three-dimensional spatial data, and may be any three-dimensional data. Therefore, the three-dimensional unit input data may be, for example, a moving image. When the three-dimensional unit input data is a moving image, the function expressed by the SISO neural network converts the three-dimensional coordinates, which are a combination of two-dimensional coordinates indicating the vertical and horizontal coordinates of the image and one-dimensional coordinates indicating time, into colors.

[0036] (Specific example when the dimension of unit input data is four-dimensional) A case where the dimension of unit input data input to the SISO neural network is four-dimensional will be described. Four-dimensional unit input data input to the SISO neural network is, for example, four-dimensional spatial data. In this case, the four-dimensional coordinate x ４ The SISO neural network expresses a function that converts a point (each coordinate value is a real number) into a point value c in three-dimensional space. The four-dimensional coordinate is expressed, for example, as a combination of three-dimensional coordinates (vertical, horizontal, and depth directions) in the three-dimensional space and one-dimensional coordinates related to a time series such as time.

[0037] For example, if the value c of a point in three-dimensional space represents a position in a three-dimensional color space such as RGB, then the four-dimensional coordinate x ４ The function for converting into the value c of a point in three-dimensional space is expressed by the following equation (4).

[0038]

[0039] The four-dimensional unit input data may be any four-dimensional data.

[0040] (Example of a case where the dimension of unit input data is six) A case where the dimension of unit input data input to the SISO neural network is six will be described. Six-dimensional unit input data input to the SISO neural network is, for example, a three-dimensional coordinate x ３ and the viewing direction d. The viewing direction d is a pair of the two-dimensional sphere (S2 ) is three-dimensional data. ２ is a three-dimensional real space (R ３ ) and represents the space on a unit sphere with a size of 1 and centered at the origin in three-dimensional space. In this case, the function expressed by the SISO neural network is expressed, for example, by the following equation (5).

[0041]

[0042] σ represents the volume density, and c represents the value of a point in three-dimensional space, such as color. In the example of equation (5), color and volume density are output, but the output does not necessarily have to be both color and volume density, and may be only one of them.

[0043] (Example of a case where the dimension of unit input data is seven) A case where the dimension of unit input data input to the SISO neural network is seven will be described. The seven-dimensional unit input data input to the SISO neural network is, for example, a four-dimensional coordinate x ４ and the line of sight direction d. The line of sight direction d is three-dimensional data.

[0044] In this case, the output of the SISO neural network may be, for example, a pair of color and volume density, or may be only one of color and volume density.

[0045] (Specific example when the dimension of unit input data is M-dimensional) A case where the dimension of unit input data input to a SISO neural network is M-dimensional (M is an integer equal to or greater than 1) will be described. The function expressed by such a SISO neural network is expressed by the following equation (6).

[0046]

[0047] In any of the examples of SISO neural networks, from the example in which the unit input data has one dimension to the example in which the unit input data has M dimensions, the defined quantity output by the SISO neural network does not necessarily have to be signed size, color, or volume density. Furthermore, the defined quantity output by the SISO neural network does not necessarily have to be one-dimensional, three-dimensional, or four-dimensional data. As described above, the defined quantity output by the SISO neural network can be any quantity as long as it is a predefined quantity.

[0048] In any of the examples of the SISO neural network, from the example in which the dimension of the unit input data is one-dimensional to the example in which the dimension of the unit input data is M-dimensional, the unit input data does not need to be coordinates, time, or gaze direction. The unit input data may be data in which the coordinate or gaze direction values ​​have been converted using a value conversion technique such as positional encoding or positional embedding.

[0049] Furthermore, the unit input data does not necessarily have to be coordinates, time, or gaze direction, or data after conversion of these values, but may be any predetermined feature that represents a point specified by coordinates, time, or gaze direction, or any feature that represents a point extracted using a parameterized function such as a neural network.

[0050] <More Specific Description of MIMO Neural Network> Next, a more specific description of the MIMO neural network will be given. For simplicity of explanation, the description will be given taking as an example a case where the dimension of unit input data input to the MIMO neural network is M (M is an integer equal to or greater than 1). Furthermore, for simplicity of explanation, it will be assumed that the number of unit nodes in the input layer is N. ｐ (N ｐ is an integer of 2 or more).

[0051] The function expressed by such a MIMO neural network is expressed by the following equation (7).

[0052]

[0053]

[0054]

[0055] Here, the vector in equation (8) is k (k is 1 or more, N ｐ (below) the Nth element ｐ is a vector representing an M-dimensional coordinate input to the k-th element of the pre-ordered unit nodes. The M-dimensional coordinate is a generalized coordinate including the line of sight direction, etc.

[0056] The vector in equation (9) is k (k is 1 or more, N ｐ (below) the Nth element ｐ k represents the estimation result of the MIMO neural network for the value input to the kth element among the k pre-ordered unit nodes. The estimation result is, for example, the value of a defined quantity such as color. Note that, for simplicity of explanation, the above description has been given using an example in which the defined quantity corresponding to each unit input data is the color of three-dimensional data, but the defined quantity corresponding to each unit input data does not necessarily have to be color. Furthermore, the defined quantity corresponding to each unit input data does not necessarily have to be three-dimensional data. As described above, the defined quantity corresponding to each unit input data may be any quantity as long as it is a pre-defined quantity.

[0057] <N input to the MIMO neural network ｐ Technique for Determining Unit Input Data> Here, N input data to the MIMO neural network are ｐ An example of the technique for determining the number of unit input data is described below. ｐ The unit input data may be, for example, N randomly selected from a set of unit input data prepared in advance. ｐ are unit input data.

[0058] N input to the MIMO neural networkｐ The unit input data are, for example, N pieces of unit input data selected from a set of unit input data prepared in advance according to a predetermined rule for selecting unit input data (hereinafter referred to as "selection rule"). ｐ It may be unit input data.

[0059] An example of the selection rule will be described below. The selection rule is, for example, a rule that selects a point on a line determined by a predetermined rule, such as a rule that selects a point on a ray in a line-of-sight direction determined by a predetermined rule. Note that the line is a line in an M-dimensional topological space that includes a set of unit input data. The topological space is, for example, a vector space.

[0060] The selection rule may be, for example, a rule that selects a point on a plane in an M-dimensional topological space that includes the set of unit input data and that is determined by a predetermined rule. Note that the plane determined by the predetermined rule is, for example, a plane specified by a user.

[0061] The selection rule may be, for example, a rule that selects a point in a space of dimensions less than M determined by a predetermined rule, which is a space of dimensions less than M within an M-dimensional topological space containing a set of unit input data. Note that the space of dimensions less than M determined by a predetermined rule is, for example, a space designated by a user.

[0062] The selection rule may be, for example, a rule that selects points in the vicinity of a point determined by a predetermined rule in a space of dimensions M or less determined by a predetermined rule, the space being a space of dimensions M or less in an M-dimensional topological space containing a set of unit input data. Note that the points in the vicinity of a point determined by a predetermined rule are Np points closest to the point determined by the predetermined rule. The points determined by the predetermined rule are, for example, points specified by a user.

[0063] For ease of understanding, an example of a function expressed by a MIMO neural network is shown in equation (10) when the unit input data has three dimensions.

[0064]

[0065] Equation (10) is an equation obtained when M is 3 in equation (7). While the above description has been given using an example in which the defined quantity corresponding to each unit input data is the color of three-dimensional data, the defined quantity corresponding to each unit input data does not necessarily have to be the color of three-dimensional data. As described above, the defined quantity corresponding to each unit input data may be any quantity as long as it is a predefined quantity. For example, it may be four-dimensional data that combines color and volume density. An example of a function expressed by a MIMO neural network when the dimensions of the unit input data are six is ​​expressed, for example, by the following equation (11).

[0066]

[0067] Value d ｋ is N ｐ represents a value indicating the gaze direction input to the k-th element among the k pre-ordered unit nodes. Note that, in the above description, an example has been given in which the defined quantity corresponding to each unit input data is a combination of color and volume density, which are four-dimensional data, but the defined quantity corresponding to each unit input data does not necessarily have to be a combination of color and volume density, which are four-dimensional data. As described above, the defined quantity corresponding to each unit input data may be any quantity as long as it is a pre-defined quantity. For example, it may be color, which is three-dimensional data.

[0068] <MIMO Neural Network Accepting Auxiliary Data Input> Note that N ｐ When the values ​​indicating the gaze direction input to the unit nodes are all the same, an example of a function expressed by the MIMO neural network may be expressed by, for example, the following equation (12).

[0069]

[0070] Equation (12) shows that three-dimensional unit input data and auxiliary data are input to the MIMO neural network instead of the six-dimensional unit input data represented by equation (11). The auxiliary data is data different from the unit input data and represents information common to all unit input data input to the MIMO neural network. In the example of equation (12), the auxiliary data is three-dimensional data representing the line of sight.

[0071] Equation (12) is an example where the dimension of the unit input data is three-dimensional and the dimension of the auxiliary data is three-dimensional, but the dimension of the unit input data may be other than three, and the dimension of the auxiliary data may also be other than three.

[0072] In this way, N ｐ For a MIMO neural network that processes Q-dimensional (Q is an integer greater than or equal to 2) unit input data, if there is information common to all the input unit input data, a modified MIMO neural network can also output similar results.

[0073] The modified MIMO neural network is a MIMO neural network that includes a second-type input layer instead of the first-type input layer. The first-type input layer is an N-type input layer that receives Q-dimensional unit input data. ｐ The second type input layer is an input layer with N unit nodes satisfying Q = L + H. ｐ N that accepts L-dimensional unit input data ｐ The input layer includes unit nodes and a set of nodes that accept H-dimensional auxiliary data, where L and H are integers between 0 and Q, inclusive, and satisfy Q=L+H.

[0074] Therefore, the modified MIMO neural network has N ｐ Instead of Q-dimensional unit input data, N ｐ In other words, the modified MIMO neural network has an input layer that receives L-dimensional unit input data and auxiliary data. ｐThe MIMO neural network is an input layer having unit nodes, the input layer further having a set of nodes for accepting auxiliary data.

[0075] <Conversion Processing> The control unit 11 may perform a conversion processing on the output result of the MIMO neural network. The conversion processing is a process of obtaining the value of another predefined quantity (i.e., another defined quantity) based on the value of the defined quantity output by the MIMO neural network. The conversion processing may be a process with predetermined content, or a process whose content is parameterized by a neural network or the like and whose content can be optimized by learning.

[0076] Such conversion processing is, for example, processing for converting three-dimensional spatial data into the values ​​of each pixel of a two-dimensional image when the defined quantity output by the MIMO neural network is three-dimensional spatial data.

[0077] Here, the exemplary process will be described using a specific example in which the function expressed by the MIMO neural network is the first exemplary function. The exemplary process is a process in which three-dimensional spatial data is obtained using the MIMO neural network, and then pixel values ​​of a two-dimensional image are obtained by performing a transformation process on the obtained data.

[0078] The first exemplary function is a function expressed by Equation (12), and the independent variable of the function is N ｐ The dependent variables of the first exemplary function are the three-dimensional coordinates indicating the positions of the N objects and the data indicating the line of sight. ｐ The color and opacity at each position.

[0079] A ray of light in the line of sight direction in three-dimensional Euclidean space can be expressed as, for example, r(t) = o + td, where o is a position vector representing the position of the origin of the line of sight located in three-dimensional Euclidean space. Specifically, the position of the origin of the line of sight is the position of the camera capturing the subject.

[0080] d is a vector representing the line of sight direction. In other words, it is an example of data representing the line of sight direction. ｎ More than t ｆ The following real values ​​are given: t ｎ and ｆ What is t ｎ <t ｆ It is a predetermined real value that satisfies the relationship: r(t) = o + td ｎ and t ｆ As you can see by substituting each, r(t ｎ ) is r(t ｎ ) and r(t ｆ ) represents the point on the line segment connecting the origin of the line of sight. ｆ ) is r(t ｎ ) and r(t ｆ It represents the point on the line segment connecting the two points that is farthest from the origin of the line of sight.

[0081] Note that the three-dimensional coordinates indicating the position are input as independent variables of the first exemplary function. ｐ Specifically, the positions are r(t ｎ ) and r(t ｆ ) and N ｐ It is an individual point.

[0082] The values ​​input to the independent variables of the first example function are expressed by, for example, the following equations (13) and (14).

[0083]

[0084]

[0085] t ｊ is the number of coordinates input to the independent variables of the first example function. ｐ Among these positions, the j-th position from the closest to the origin of the line of sight is defined as r(t ｊ ) represents r(t ｊ ) are the values ​​of the independent variables.

[0086] The color output by the first example function is expressed by, for example, the following equation (15).

[0087]

[0088] The opacity output by the first example function is expressed by, for example, the following equation (16).

[0089]

[0090] Since the data d indicating the gaze direction does not depend on t, it is used as auxiliary data in equation (12). If the data d indicating the gaze direction depends on t, the function expressed by the MIMO neural network is equation (11) instead of equation (12). Even if the data d indicating the gaze direction does not depend on t, the function expressed by the MIMO neural network may be equation (11) instead of equation (12).

[0091] By the way, the number of positions whose coordinates have been input to the first example function is N ｐ However, N ｐ More than N Ｌ In some cases, we want to obtain the output of the MIMO neural network for N positions. Ｌ The positions of N ｐ The outputs of the MIMO neural network may be obtained for each set.

[0092] In addition, N Ｌ N ｐ If it is not divisible by N, padding is performed on the endpoints of the points on the ray in the line of sight. Ｌ N ｐ Specifically, padding processing may involve overlapping the values ​​of the endpoints, extending the points after the endpoints to the interval before the endpoints, or inserting a predetermined value. Ｌ N ｐ Since it is a multiple of N Ｌ N ｐ In this way, N ｐ N is more than Ｌ For each position, the color and opacity are also obtained using a MIMO neural network.

[0093] The position where the coordinates are input to the first example function may be randomly determined according to a predetermined distribution such as a uniform distribution. The position where the coordinates are input to the first example function may also be determined based on the importance of each position obtained by another predetermined technique such as a neural network. Furthermore, the position where the coordinates are input to the first example function may also be determined by another technique such as a neural network without using a parameter such as importance.

[0094] N on the ray of the line of sight d obtained by implementing a MIMO neural network that represents the first example function Ｌ Each color c at each position ｊ Using Volume Rendering, we apply the opacity σ ｊ By integrating based on the above, the color C(r) of the pixel when the light ray r seen from the line of sight d is projected onto a two-dimensional plane can be obtained. Ｌ Color c at each position ｊ and opacity σ ｊ Volume rendering, which obtains the color C(r) of a pixel when a ray r seen from a line of sight d is projected onto a two-dimensional plane based on the above equation, is an example of a conversion process.

[0095] N Ｌ Color c at each position ｊ and opacity σ ｊ Based on this, Volume Rendering obtains the color C(r) of a pixel when a ray r seen from a line of sight d is projected onto a two-dimensional plane, and is specifically expressed by the following equation (17):

[0096]

[0097]

[0098]

[0099]

[0100] In this way, three-dimensional spatial data is obtained using a MIMO neural network, and then pixel values ​​of a two-dimensional image are obtained by performing a transformation process on the obtained data.

[0101] The two-dimensional image is obtained by performing the example process on all rays within the viewing angle of the camera that are required to obtain values ​​for all pixels. More specifically, the two-dimensional image to be generated is obtained by performing the example process on pixels of the two-dimensional image to be generated that have not been subjected to the example process, for all pixels of the two-dimensional image to be generated.

[0102] Note that a process similar to the exemplary process can also be applied to M-dimensional data such as four-dimensional spatiotemporal data. An example of application to four-dimensional spatiotemporal data is a process for obtaining a moving image by repeating a process in which a MIMO neural network outputs four-dimensional spatiotemporal data and a conversion process is performed based on the output.

[0103] The process of obtaining a moving image will now be described in more detail. The MIMO neural network receives, for example, four-dimensional coordinates or a combination of four-dimensional coordinates and data indicating line-of-sight direction, and outputs color and opacity. The conversion process obtains pixel values ​​of a two-dimensional image at each time based on the color and opacity of each position in three-dimensional space at each time. The moving image is obtained by repeating this combination of the MIMO neural network and the conversion process.

[0104] Such a process of repeating a combination of a MIMO neural network and a conversion process is executed, for example, by the control unit 11. Note that the control unit 11 may execute only the MIMO neural network. Alternatively, the control unit 11 may execute the combination of a MIMO neural network and a conversion process, and the control of the repetition may be executed by another device.

[0105] It should be noted that the above repetition does not necessarily require that the conversion process be performed after the execution of the MIMO neural network. Therefore, for example, it is also possible to first repeatedly execute only the MIMO neural network, and then perform a conversion process only once to obtain the value of a desired defined quantity, such as a moving image, based on each result obtained in the repetition. Such a conversion process is also a process of obtaining the value of another defined quantity based on the value of the defined quantity obtained by the MIMO neural network.

[0106] <More Detailed Explanation of the Learning Process> Up to this point, we have explained the MIMO neural network. As described above, the control unit 11 executes the learning process. That is, the control unit 11 performs learning with the MIMO neural network as the learning target. In the learning process, the MIMO neural network that is the learning target is executed, and based on the results of the execution, the MIMO neural network that is the learning target is updated.

[0107] In the training process, for example, the MIMO neural network to be trained is updated based on the output result itself of the MIMO neural network to be trained, or the MIMO neural network to be trained may be updated based on the result of a conversion process performed on the output of the MIMO neural network to be trained.

[0108] In the learning process, the MIMO neural network to be trained may be updated based on the results obtained by repeatedly performing a conversion process on the output of the MIMO neural network to be trained, such as the above-mentioned video. Furthermore, as described above, it is also possible to obtain the value of the defined quantity by performing a conversion process after the repetition of the MIMO neural network to be trained. In such a case, in learning, the MIMO neural network to be trained may be updated based on the value of the defined quantity obtained by performing a conversion process after the repetition of the MIMO neural network to be trained.

[0109] In this way, in either case, in the learning process, the MIMO neural network to be learned is updated based on the output of the MIMO neural network to be learned.

[0110] The learning technique executed in the learning process may be any technique that improves the estimation accuracy of the MIMO neural network that is the learning target. Therefore, the learning technique executed in the learning process may be, for example, supervised learning or unsupervised learning. The learning technique executed in the learning process may be, for example, a learning technique that uses a generative model such as a generative adversarial network, a variational autoencoder (VAE), a diffusion model, an autoregressive model, a flow model, or a combination thereof.

[0111] The data to be prepared in advance for learning differs depending on the learning technique. The data to be prepared in advance for learning is so-called learning data. For example, the data to be prepared in advance for learning is a set of data to be input to a MIMO neural network and data indicating the true value of the output of the MIMO neural network. The data indicating the true value of the output of the MIMO neural network is a type of so-called correct answer data.

[0112] Furthermore, the data that should be prepared in advance for learning is, for example, a pair of data to be input to a MIMO neural network and data indicating the true value of the result of the conversion process. The data indicating the true value of the result of the conversion process is also a type of so-called correct answer data. A technique in which such a pair of data to be input to a MIMO neural network and correct answer data is prepared in advance is, for example, supervised learning such as regression.

[0113] The data to be prepared in advance for learning may be, for example, only ground truth data. Techniques that use such data include unsupervised learning such as generative adversarial networks, variational autoencoders, and diffusion models.

[0114] In the learning, the MIMO neural network may be updated so as to approach the distribution of correct answer data when a set of prepared correct answer data is categorized according to a predetermined rule.In the learning, instead of the distribution, the MIMO neural network may be updated for each piece of data input to the MIMO neural network so as to reduce the difference between the result estimated by the MIMO neural network based on the input data and the correct answer data corresponding to the input data.

[0115] <More Specific Example of Learning Process> (More Specific First Example) A more specific first example of the learning process will be described using an example in which the dimension of unit input data is M. Here, the description will be given using an example in which the learning data is paired data consisting of data input to a MIMO neural network to be learned and correct answer data. Hereinafter, when one piece of learning data includes one piece of unit input data, that one piece of unit input data will be referred to as input learning data.

[0116] For the sake of simplicity, the first example will be described in more detail using the M-dimensional coordinates ^x, which indicate a position in a predetermined M-dimensional topological space. ｊ Ｍ When the input learning data includes the above, the correct answer data is the definition quantity ^o of the position indicated by the M-dimensional coordinates. ｊ Here, ^A represents the letter A with a caret (or circumflex).

[0117] In a more specific first example, the optimization of the MIMO neural network is performed by the following first type procedure. In the first type procedure, the control unit 11 first selects N training data from a training data group, which is a set of training data, according to a predetermined rule such as random. ｐ The training data group is a set of data prepared in advance. If the number of training data included in the training data group is N, ｐ If the number is less than N, the same training data may be selected multiple times. ｐ A process of selecting training data is performed, and N ｐ pieces of training data are selected.

[0118] In the first type procedure, the control unit 11 then ｐ The control unit 11 inputs the input training data containing each of the training data into the MIMO neural network. ｐ For each input training data, an output of the MIMO neural network is obtained.

[0119] ^x ｊ Ｍ The output o of the MIMO neural network for input training data including ｊ In the first type procedure, the control unit 11 then ｊ and correct data ^o ｊ The MIMO neural network is updated so as to reduce the difference between the distribution of the output of the MIMO neural network and the distribution of the correct data.

[0120] The distribution of data refers to the distribution of the data within the topological space that contains the data. In other words, the distribution of data within the topological space is the arrangement of the data within the topological space.

[0121] The objective function used in such optimization is, for example, a function expressed by the following equation (21): In optimization, the objective function is minimized.

[0122]

[0123] The distance index used does not necessarily have to be the L2 distance, and any distance index such as the L1 distance, the Lp distance, or the hinge distance may be used, or a combination of multiple distance indexes from these may be used.

[0124] Note that the example of equation (21) shows an objective function for one output of the MIMO neural network. In training, similar objective functions may be used for other outputs of the MIMO neural network.

[0125] An example of optimizing a MIMO neural network using iterative calculations such as a gradient method will be described below. First, the value of the objective function in equation (21) is calculated for a set of outputs from the MIMO neural network, which is a predetermined number of sets (this set is called a batch; the number of samples included in a batch is called the batch size). Then, the MIMO neural network is optimized to minimize the integration of these outputs. Note that integration refers to averaging or summing.

[0126] (More specific second example) A more specific second example of the learning process will be described using an example in which the dimension of the unit input data is M. Here, by executing the transformation process, one definition quantity is obtained based on multiple outputs obtained from the MIMO neural network, and the obtained definition quantity and the correct answer data ^O are compared. ｊ A case where the MIMO neural network is updated based on the above will be described.

[0127] In this case, the training data is the correct answer data ^O ｊ and a plurality of input training data. A set of input training data including training data in a more specific second example can be expressed by the following formula (22).

[0128]

[0129] Each element x in equation (22) （ｊ、ｋ） Ｍ (k is an integer between 1 and K, inclusive. K is an integer greater than or equal to 1.) are input learning data.

[0130] Thus, the second more specific example is an example of learning in which an output of the MIMO neural network is obtained for each of a plurality of input training data, and the value of the defined quantity is obtained by a conversion process based on the obtained plurality of outputs.

[0131] In a more specific second example, the optimization of the MIMO neural network is performed by the following second type procedure.

[0132] In the second type procedure, first, the control unit 11 selects one piece of training data from the training data group. Next, the control unit 11 divides the K pieces of input training data included in the selected training data into N ｐ Divide into groups of K and N. ｐ If it is not divisible by, the control unit 11 performs the padding process described above.

[0133] Next, the control unit 11 executes the MIMO neural network for each set. At this time, those corresponding to the padded points are removed. As a result, the control unit 11 obtains K outputs of the MIMO neural network. The obtained K outputs can be expressed mathematically as follows:

[0134]

[0135] Next, the control unit 11 performs a conversion process on the K outputs. ｊ Next, the control unit 11 calculates the obtained value O ｊ and correct data ^O ｊ To reduce the difference between ｊ Distribution of and correct data ^O ｊ The MIMO neural network is optimized so as to reduce the difference between the values ​​of the transformation process and the values ​​of the transformation process. If the transformation process is a parameterized process, the transformation process may also be optimized when the MIMO neural network is updated.

[0136] The objective function used for optimization is, for example, a function expressed by the following equation (24).

[0137]

[0138] The distance index used does not necessarily have to be the L2 distance, and any distance index such as the L1 distance, the Lp distance, or the hinge distance may be used, or a combination of multiple distance indexes from these may be used.

[0139] Note that the example of equation (24) shows an objective function for one result obtained by the conversion process. In learning, a similar objective function may be used for other outputs obtained by the conversion process.

[0140] An example of optimizing a MIMO neural network using iterative calculations such as the gradient method will be described below. First, the value of the objective function (22) is calculated for a set of outputs from the MIMO neural network, which is a set of a predetermined number of outputs. Then, the MIMO neural network is optimized to minimize the integration of these outputs.

[0141] (More Specific Third Example) A more specific third example of the learning process will be described. In the more specific learning process of the third example, learning is performed in which the input learning data is not included in the learning data. In this case, values ​​randomly sampled according to a predetermined distribution, such as a uniform distribution or a normal distribution, are input to the MIMO neural network. Values ​​sampled using a predetermined parameterized function, such as a neural network, may also be input to the MIMO neural network.

[0142] In addition, when values ​​sampled using a parameterized predetermined function are input to a MIMO neural network, when optimizing the MIMO neural network, not only the MIMO neural network but also the parameterized predetermined function that performs the sampling may be optimized.

[0143] Furthermore, the MIMO neural network may be input with values ​​estimated by a predetermined function that estimates corresponding input training data based on correct answer data included in the training data. The predetermined function that estimates corresponding input training data based on correct answer data is, for example, a neural network.

[0144] In a more specific third example, the optimization of the MIMO neural network is performed by the following third type of procedure.

[0145] In the third procedure, first, the control unit 11 ｐ Next, the control unit 11 calculates the obtained N ｐ Input training data into the MIMO neural network to obtain N ｐ Next, the control unit 11 optimizes the MIMO neural network so as to reduce the difference between each of the obtained outputs and the correct data corresponding to each output, or to reduce the difference between the distribution of the outputs and the distribution of the correct data.

[0146] The objective function used for optimization is, for example, a function based on a Generative Adversarial Network (GAN). In this case, a parameterized classifier is used, and the classifier is optimized so that it can distinguish between the output of the MIMO neural network and the correct data included in the training data. Furthermore, the MIMO neural network is optimized so as to generate an output that can deceive the classifier. Note that the parameterized classifier is, for example, configured by a neural network.

[0147] Note that optimization does not necessarily have to be performed based on a generative adversarial network. For example, optimization may be performed using a generative model such as a variational autoencoder, a diffusion model, an autoregressive model, or a flow model. Optimization may also be performed using any combination of these generative models, including a generative adversarial network.

[0148] (More Specific Fourth Example) A more specific fourth example of the learning process will be described. In the learning process of the more specific fourth example, learning is also performed in which input learning data is not included in the learning data. The learning process of the more specific fourth example is a process performed by the following fourth type procedure.

[0149] In the fourth type procedure, first, the control unit 11 performs the N ｐ Next, the control unit 11 divides the obtained K pieces of input learning data into N ｐ Divide into groups of K and N. ｐIf it is not divisible by, the control unit 11 performs the padding process described above.

[0150] Next, the control unit 11 executes the MIMO neural network for each set. At this time, those corresponding to the padded points are removed. As a result, the control unit 11 obtains K outputs of the MIMO neural network. The obtained K outputs can be expressed mathematically as in the above formula (23).

[0151] Next, the control unit 11 performs a conversion process on the K outputs. ｊ Next, the control unit 11 calculates the obtained value O ｊ and correct data ^O ｊ To reduce the difference between ｊ Distribution of and correct data ^O ｊ The MIMO neural network is optimized so as to reduce the difference between the values ​​of the transformation process and the values ​​of the transformation process. If the transformation process is a parameterized process, the transformation process may also be optimized when the MIMO neural network is updated.

[0152] The objective function used for optimization is, for example, a function based on a Generative Adversarial Network (GAN). In this case, a parameterized classifier is used, and the classifier is optimized so that it can distinguish between the output of the MIMO neural network and the correct data included in the training data. Furthermore, the MIMO neural network is optimized so as to generate an output that can deceive the classifier. Note that the parameterized classifier is, for example, configured by a neural network.

[0153] Note that optimization does not necessarily have to be performed based on a generative adversarial network. For example, optimization may be performed using a generative model such as a variational autoencoder, a diffusion model, an autoregressive model, or a flow model. Optimization may also be performed using any combination of these generative models, including a generative adversarial network.

[0154] Note that in the more specific first to fourth examples, transformation optimization may be performed. Transformation optimization is performed when the unit input data is transformed data and the function used to obtain the transformed data (hereinafter referred to as the "transformation function") is parameterized. Transformation optimization is a process for optimizing not only the MIMO neural network but also the transformation function when optimizing the MIMO neural network. The transformed data is data in which coordinate or gaze direction values ​​have been transformed using value transformation techniques such as positional encoding or positional embedding.

[0155] Furthermore, in the more specific first to fourth examples, extraction function optimization may be performed. Extraction function optimization is performed when the unit input data is an arbitrary feature representing a point and the function used to extract the feature (hereinafter referred to as the "extraction function") is parameterized. Extraction function optimization is a process that optimizes not only the MIMO neural network but also the extraction function when optimizing a MIMO neural network.

[0156] <First Example of Learning Process for Further Improving Estimation Accuracy of Learning Object> In the learning process, the MIMO neural network of the learning object may be trained under a first constraint. The first constraint is that two or more different N ｐ The constraint is that for each output of the learning target for each of the input training data or less, even if the unit node to which each input training data is input is swapped, the difference between the outputs of the MIMO neural network corresponding to each input training data must be smaller than a predetermined difference. By updating the difference to be smaller than the predetermined difference, for example, the difference approaches 0.

[0157] In the training process, the MIMO neural network of the training target may be trained under a second constraint. The second constraint is a constraint that the difference between the output of the training target in response to first input training data input to the training target when the first input training data is input to the training target together with second input training data different from the first input training data and when the first input training data is input to the training target together with third input training data different from the first input training data and the second input training data is smaller than a predetermined difference. The constraint that the difference is smaller than the predetermined difference is, for example, a constraint that the difference should approach zero.

[0158] In the learning process, both the first constraint and the second constraint may be used. A more specific example of the learning process using the first constraint will be described.

[0159] First, the control unit 11 uses a MIMO neural network to ｐ From the first ordered set, which is an ordered set of ordered input training data, N ｐ Next, the control unit uses the MIMO neural network to obtain N outputs corresponding to each input training data from a second ordered set that includes the same input training data as the first ordered set but has a different order of the input training data. ｐ Get outputs.

[0160] Next, the control unit 11 updates the MIMO neural network so that the difference between the outputs obtained from the first ordered set and the outputs obtained from the second ordered set, which are obtained from the same input training data, is smaller than a predetermined difference.

[0161] This is an example of the learning process using the first constraint. Next, a more specific example of the learning process using the second constraint will be described.

[0162] First, the control unit 11 uses a MIMO neural network to ｐ From the first set, which is a set of ordered input training data, N ｐNext, the control unit uses the MIMO neural network to obtain N outputs corresponding to each input training data from the second set. ｐ The second set contains input training data that is partly the same as and partly different from the input training data contained in the first set, and has the same number of elements as the first set.

[0163] The second set may include fewer types of input learning data than the first set by duplicating some of the input learning data included in the first set. Thus, if the input learning data included in the first set is, for example, eight types of input learning data in total, namely, data D1, data D2, data D3, data D4, data D5, data D6, data D7, and data D8, the input learning data included in the second set may be four types of eight input learning data, namely, data D1, data D1, data D2, data D2, data D3, data D3, data D3, data D4, and data D4.

[0164] Next, the control unit 11 updates the MIMO neural network so that the difference between the outputs obtained from the first set and the outputs obtained from the second set, which are obtained from the same input training data, is smaller than a predetermined difference.

[0165] This is an example of the learning process using the second constraint. A more specific example of the learning process using the first constraint and the second constraint will be described.

[0166] First, the control unit 11 uses the MIMO neural network to select N corresponding to each input training data from the first ordered set. ｐ Next, the control unit uses the MIMO neural network to obtain N outputs corresponding to each input training data from the third ordered set. ｐ outputs are obtained. The third ordered set is an ordered set that satisfies the first set condition or the second set condition and has the same number of elements as the first set. The first set condition is a condition that the included input training data is partly the same as and partly different from the input training data included in the first set.

[0167] The second set condition is a condition that at least a portion of the input training data that is the same as that contained in the first ordered set is in a different order from that in the first ordered set. Therefore, for example, if one ordered set is an ordered set of {E1, E2, E3, E4, E5} and the other ordered set is an ordered set of {E6, E1, E2, E3, E4}, then if one ordered set is the first ordered set, the other ordered set is the third ordered set. This is because one ordered set and the other ordered set have E1, E2, E3, and E4 in common, but E5 and E6 are not common.

[0168] In one ordered set, E1 is the first element, E2 is the second element, E3 is the third element, and E4 is the fourth element. However, in the other ordered set, E1 is the second element, E2 is the third element, E3 is the fourth element, and E4 is the fifth element. In other words, in this way, one ordered set and the other ordered set in the example contain some common and some different input training data, and the common input training data is in a different order.

[0169] For the same reason, if one ordered set is {F1, F2, F3, F4, F5, F6, F7, F8} and the other ordered set is {F2, F2, F4, F4, F6, F6, F8, F8}, then if one ordered set is the first ordered set, the other ordered set is the third ordered set.For the same reason, if one ordered set is {G1, G2, G3, G4} and the other ordered set is {G3, G3, G3, G3}, then if one ordered set is the first ordered set, the other ordered set is the third ordered set. For the same reason, for example, if one ordered set is {H1, H2, H3, H4, H5, H6, H7, H8} and the other ordered set is {H2, H2, H2, H5, H5, H6, H8, H8}, then if one ordered set is the first ordered set, the other ordered set is the third ordered set.

[0170] Next, the control unit 11 updates the MIMO neural network so that the difference between the outputs obtained from the first set and the outputs obtained from the third set, which are obtained from the same input training data, is smaller than a predetermined difference.

[0171] This is an example of a learning process using the first and second constraints.

[0172] <Effects of the First and Second Constraints> In the case of a SISO neural network, there is only one input that explains the output, so other inputs provided do not affect the output. However, in the case of a MIMO neural network, there are multiple inputs that explain one output.

[0173] Therefore, inputs other than the input corresponding to the output may also affect the output. In other words, if you want to obtain an output O1 for an input X1, and there are multiple inputs to the MIMO neural network, including the input X1, the output O1 may change depending on the contents of the inputs other than the input X1.

[0174] Thus, in the case of a MIMO neural network, there may be cases where the output uncertainty is higher than in the case of a SISO neural network.

[0175] The first constraint increases the likelihood that the same information will be output even if the input destination of the information changes. The second constraint increases the likelihood that the same output will be obtained for the same input training data even if the information on which the MIMO neural network is based changes. Therefore, the first and second constraints have the effect of reducing the uncertainty of the MIMO neural network.

[0176] <Second Example of Learning Process for Further Improving Estimation Accuracy of Learning Object> In the learning process, learning may be performed to reduce the types of input learning data to be input to the MIMO neural network. For example, when the MIMO neural network is ｐ When the number of unit nodes accepted is larger than the number of unit nodes, the same input learning data may be input to some of the unit nodes.

[0177] <Effects of Inputting the Same Input Learning Data to Some Unit Nodes> By inputting the same input learning data to some unit nodes, the number of types of input that affect the output is reduced, thereby reducing the uncertainty described above.

[0178] Furthermore, inputting the same input learning data to some unit nodes may be used together with, for example, application of the first constraint or the second constraint. For example, when comparing the first output and the second output in the comparison of outputs under the first constraint or the second constraint, a strong constraint may be applied to the first output and a weak constraint may be applied to the second output.

[0179] The first output is the output obtained when the MIMO neural network processes more diverse input training data, and the second output is the output obtained when the MIMO neural network processes less diverse input training data.

[0180] Note that a strong constraint refers to a process of increasing the weight of an objective function when there is an objective function that makes the difference smaller than a predetermined difference, and a weak constraint refers to a process of decreasing the weight of an objective function when there is an objective function that makes the difference smaller than a predetermined difference.

[0181] In this way, a MIMO neural network that processes a larger variety of input training data can be made to approach a MIMO neural network that processes a smaller variety of input training data.

[0182] <Third Example of Learning Process for Further Improving Estimation Accuracy of the Learning Object> The learning process may be performed using a SISO neural network as a teacher network. In this case, the MIMO neural network is trained with the assistance of the SISO neural network. Specifically, the MIMO neural network may be updated so that the difference between the output obtained by the MIMO neural network for input training data and the output obtained by the SISO neural network for the same input training data becomes smaller.

[0183] In this learning process, the SISO neural network may or may not be updated when the MIMO neural network is updated.

[0184] Although examples have been described so far in which the input training data includes coordinates, the input training data is not limited to coordinates and does not have to be coordinates as long as it is a feature that indicates a point in a predetermined topological space.

[0185] As an example, a case where the predetermined topological space is a three-dimensional space will be described. Before using a MIMO neural network, the three-dimensional spatial data is represented based on discrete sample points. The discrete sample points may be, for example, discrete sample points on three-dimensional coordinates such as voxels or meshes, a combination of grid points on two-dimensional coordinates and discrete sample points on one-dimensional coordinates, or a combination of three discrete sample points on one-dimensional coordinates.

[0186] As a result, feature amounts of points on the discrete sample points of the three-dimensional spatial data are calculated. Note that the calculation of feature amounts may be performed by the control unit 11, or the control unit 11 may acquire feature amounts calculated by a device other than the control unit 11 or by a person.

[0187] The representation of the three-dimensional spatial data may be parameterized using a neural network, etc. In this case, the representation of the three-dimensional spatial data may be optimized simultaneously with the MIMO neural network, or may be optimized independently of the MIMO neural network.

[0188] The output of a MIMO neural network when calculating feature quantities of points on discrete sample points of three-dimensional spatial data will be described below. The output of the MIMO neural network is obtained by applying interpolation or the like to the feature quantities of points on the discrete sample points of the three-dimensional spatial data to obtain approximate feature quantities of given points, and then inputting input training data including at least these feature quantities into the MIMO neural network.

[0189] In this way, even when feature quantities are used as input training data, the MIMO neural network can obtain output faster than the SISO neural network.

[0190] The learning of the MIMO neural network is executed until a predetermined termination condition for the termination of learning (hereinafter referred to as the "learning termination condition") is satisfied. The learning termination condition may be, for example, any condition related to the termination of learning. The learning termination condition may be, for example, that the MIMO neural network has been updated a predetermined number of times, or that the change in the MIMO neural network due to the update is smaller than a predetermined change. The learning termination condition may also be that learning has been performed using all of the training data prepared in advance.

[0191] The inference device 2 executes an inference process. The inference process is a process of performing inference using the trained MIMO neural network obtained by the learning device 1. The trained MIMO neural network is the MIMO neural network at the time when the learning termination condition is satisfied. If a conversion process is performed in the learning process, the inference process may or may not use the conversion process. Even if the conversion process is not used, the output of the trained MIMO neural network can be obtained. Therefore, for example, if a user wants the output of the trained MIMO neural network rather than the result of the conversion process, the inference process executes the trained MIMO neural network, and the conversion process does not need to be performed.

[0192] The MIMO neural network may be configured in any manner as long as it is partially or entirely composed of neural networks. The neural networks constituting the MIMO neural network may be any neural network, such as a multilayer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), an attention network, or a transformer network. The neural networks constituting the MIMO neural network may also be a combination of these neural networks.

[0193] <About the Experiment> An example of the results of an experiment using a MIMO neural network will be described. In the experiment, two data sets, Data Set A and Data Set B, were used.

[0194] Dataset A contains 300 images of an object taken from all around. Of these images, 100 images were used for training, and 200 images were used for performance evaluation. There were eight types of objects, and a MIMO neural network was trained for each object, and the performance of each trained MIMO neural network was then evaluated.

[0195] Dataset B contains images of scenes taken from the front. The number of images varies from scene to scene, ranging from 20 to 62 images. Of these images, one-eighth of the images were used for performance evaluation, and the remaining images were used for training. There were eight types of scenes, and a MIMO neural network was trained for each scene, after which the performance of each trained MIMO neural network was evaluated.

[0196] In the experiment, a learning process was performed in which an exemplary process was executed in which the function expressed by the above-mentioned MIMO neural network was the first exemplary function, and the performance of the trained MIMO neural network obtained as a result of the learning process was evaluated.

[0197] More specifically, the MIMO neural network used in the experiment was a MIMO neural network that accepted 3D coordinates and gaze direction as input training data and output color and opacity. The MIMO neural network was a neural network that performed the above-mentioned position encoding in the input layer, and the subsequent parts were multilayer perceptrons (MLPs).

[0198] Since opacity is independent of gaze direction, opacity calculations were performed using only 3D coordinate information, whereas color calculations were performed using both 3D coordinate and gaze direction information, since color depends on gaze direction.

[0199] The conversion process was a process of obtaining a two-dimensional image by integrating the color and opacity output from the MIMO neural network.

[0200] In the experiment, four techniques, Technique 1 to Technique 4, were compared. Technique 1 was a technique that used a SISO neural network. Technique 2 was a technique that used a MIMO neural network, but did not use the first example of the learning process that further improves the estimation accuracy of the learning object to the third example of the learning process that further improves the estimation accuracy of the learning object.

[0201] Technique 3 is a technique using a MIMO neural network, and is a technique using a first example of a learning process that further improves the estimation accuracy of the learning object and a second example of a learning process that further improves the estimation accuracy of the learning object.Technology 4 is a technique using a MIMO neural network, and is a technique using a third example of a learning process that further improves the estimation accuracy of the learning object.

[0202] In the experiments, eight evaluation indices were used: PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index Measure), LPIPS (Learned Perceptual Image Patch Similarity), #Run (Number of Running), FLOPs (Number of Floating-Point Operations), I-time (Inference Time), T-time (Training Time), and #Params (Number of Parameters).

[0203] In the experiment, the difference between the image obtained by the trained SISO neural network or the trained MIMO neural network and the true image was evaluated by PSNR. The larger the PSNR value, the higher the performance of the trained SISO neural network or the trained MIMO neural network used to estimate the image.

[0204] In the experiment, the difference between the image obtained by the trained SISO neural network or the trained MIMO neural network and the true image was evaluated by SSIM. The larger the SSIM value, the higher the performance of the trained SISO neural network or the trained MIMO neural network used to estimate the image.

[0205] In the experiment, the difference between the image obtained by the trained SISO neural network or the trained MIMO neural network and the true image was evaluated by LPIPS. The smaller the LPIPS value, the higher the performance of the trained SISO neural network or the trained MIMO neural network used to estimate the image.

[0206] #Run indicates the number of times the trained SISO neural network or trained MIMO neural network is run to generate one pixel. Therefore, a smaller value of #Run indicates a smaller number of times the neural network is run.

[0207] FLOPs indicates the total number of floating-point multiplication and addition operations performed in a SISO neural network or a MIMO neural network to generate one pixel. Therefore, a smaller FLOPs value indicates a smaller amount of calculation.

[0208] I-time indicates the time required to generate one image. Therefore, the smaller the I-time value, the shorter the generation time.

[0209] T-time indicates the time required to train a SISO neural network or a MIMO neural network. Therefore, a smaller T-time value indicates a shorter training time.

[0210] #Params indicates the number of parameters of the SISO neural network or MIMO neural network. Therefore, a smaller value of #Params indicates a smaller model size.

[0211] FIG. 2 is a first diagram showing an example of experimental results in the embodiment. FIG. 3 is a second diagram showing an example of experimental results in the embodiment. Among the results shown in FIG. 2 and FIG. 3, the reduction in the amount of calculation is shown in FLOPs and I-time. For Technique 2, Technique 3, and Technique 4, N ｐ When the number of inputs and outputs is the same, a MIMO neural network with the same number of inputs and outputs is used. Therefore, the values ​​of FLOPs are all the same in Technique 2, Technique 3, and Technique 4, and the values ​​of I-time are also all the same in Technique 2, Technique 3, and Technique 4.

[0212] As described above, in the case of a MIMO neural network, N ｐ Although the MIMO neural network receives unit input data and outputs values ​​corresponding to each unit input data, the amount of calculation required to execute the MIMO neural network once is N times the amount of calculation required to execute the SISO neural network once. ｐIn the experimental results, the amount of calculation required to run a neural network once can be calculated in FLOPs or #Runs. This value is N times the amount of calculation required to run the above MIMO neural network once. ｐ Therefore, in the case of MIMO neural networks, N ｐ Experimental results show that there is no significant increase in the amount of computation compared to the increase rate of

[0213] I-time, which represents the time it takes to generate one image, is also ｐ As the number of iterations increases, the results show that the MIMO neural network reduces the computational complexity compared to the SISO neural network.

[0214] Next, we will explain #Params, which is an evaluation index that represents the model size. In the case of a MIMO neural network, N ｐ This shows that there is no significant increase in model size compared to the increase in

[0215] The experimental results for PSNR, SSIM, and LPIPS, which are evaluation indices representing image quality, are explained. ｐ However, in technique 3, the degradation of image quality tends to be less than in technique 2, and N ｐ It was shown that when = 2, it was equivalent to technique 1. Technique 4 also tended to have less degradation in image quality than technique 2.

[0216] Thus, these results show that the first example of the learning process that further improves the estimation accuracy of the learning object to the third example of the learning process that further improves the estimation accuracy of the learning object are effective in improving the estimation accuracy of the MIMO neural network.

[0217] In addition, in Technique 3, it was found that using the first example of the learning process that further improves the estimation accuracy of the learning object and the second example of the learning process that further improves the estimation accuracy of the learning object resulted in better image quality than using only one of them.

[0218] Next, we will explain the experimental results for T-time, an evaluation index that represents the time required for learning. The experimental results show that among techniques 1 to 4, technique 2 took the shortest time, and that technique 2 not only sped up image generation but also sped up learning. Technique 3 required more time for learning than technique 2 due to the increased constraints, but was still equivalent to technique 1.

[0219] In Technique 4, it is necessary to train the MIMO neural network after training the SISO neural network. The experimental results show this, and among Techniques 1 to 4, Technique 4 requires the longest training time.

[0220] 4 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. The learning device 1 includes a control unit 11 having a processor 91 and a memory 92 connected by a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0221] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0222] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 acquires learning data, for example, via the interface unit 12. The control unit 11 executes, for example, a learning process. The control unit 11 executes a MIMO neural network in the learning process. The control unit 11 may also execute a conversion process in the learning process. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is reading.

[0223] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits learning data. In such a case, the interface unit 12 acquires learning data by communicating with the device that transmits the learning data.

[0224] The external device is, for example, the inference device 2. In such a case, the interface unit 12 makes the trained MIMO neural network available to the inference device 2 via wired or wireless communication.

[0225] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, touch panel, microphone, etc. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the input devices of the interface unit 12 accept input of various information to the learning device 1 via wired or wireless connections. Note that the various data acquired by the interface unit 12 described above does not necessarily have to be acquired by the communication interface of the interface unit 12, but may be input to the input devices.

[0226] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the learning device 1. The interface unit 12 may include a sound output device such as a speaker. The interface unit 12 outputs, for example, information input to the interface unit 12.

[0227] The storage unit 13 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores, for example, various information generated by the operation of the control unit 11. The storage unit 13 stores, for example, information acquired by the interface unit 12. The storage unit 13 stores, for example, a MIMO neural network.

[0228] 5 is a diagram showing an example of the flow of processing executed by the learning device 1 according to the embodiment. The control unit 11 acquires a set of learning data (step S101). Next, the control unit 11 executes a learning process using the set of learning data acquired in step S101 until a learning termination condition is satisfied (step S102).

[0229] 6 is a diagram showing an example of the hardware configuration of the inference device 2 in an embodiment. The inference device 2 is equipped with a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected by a bus, and executes a program. By executing the program, the inference device 2 functions as a device including the control unit 21, an interface unit 22, and a memory unit 23.

[0230] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the inference device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0231] The control unit 21 controls the operation of each functional unit included in the inference device 2. The control unit 21 acquires data to be inferred, for example, via the interface unit 22. The control unit 21 executes inference processing. In the inference processing, the control unit 21 executes a trained MIMO neural network. The control unit 21 may also execute a conversion processing in the inference processing. The control unit 21 acquires, for example, information stored in the memory unit 23. Specifically, the process of acquiring information stored in the memory unit 23 is reading.

[0232] The interface unit 22 includes a communication interface for connecting the inference device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits data to be inferred. In such a case, the interface unit 22 acquires the data to be inferred by communicating with the device that transmits the data to be inferred.

[0233] The external device is, for example, the learning device 1. In this case, the control unit 21 uses the trained MIMO neural network via a wired or wireless connection with the interface unit 22. The interface unit 22 may obtain the trained MIMO neural network from the learning device 1 via a wired or wireless connection.

[0234] The interface unit 22 may be configured to include input devices such as a mouse, keyboard, touch panel, microphone, etc. The interface unit 22 may be configured as an interface that connects these input devices to the inference device 2. In this way, the input devices of the interface unit 22 accept input of various information to the inference device 2 via wired or wireless connections. Note that the various data acquired by the interface unit 22 described above does not necessarily have to be acquired by the communication interface of the interface unit 22, but may be input to an input device.

[0235] The interface unit 22 outputs, for example, various types of information. The interface unit 22 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 22 may be configured as an interface that connects these display devices to the inference device 2. The interface unit 22 may include, for example, a sound output device such as a speaker. The interface unit 22 outputs, for example, information input to the interface unit 22.

[0236] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the inference device 2. The storage unit 23 stores, for example, various information generated by the operation of the control unit 21. The storage unit 23 stores, for example, information acquired by the interface unit 22. The storage unit 23 stores, for example, a trained MIMO neural network.

[0237] 7 is a diagram showing an example of the flow of processing executed by the inference device 2 in the embodiment. The control unit 21 acquires data to be inferred (step S201). Next, the control unit 21 executes inference processing on the data to be inferred acquired in step S201 (step S202).

[0238] The learning device 1 of the embodiment configured as described above performs learning of a MIMO neural network, which makes it possible to suppress an increase in the amount of calculation, as described in <Effects of the MIMO neural network>.

[0239] Furthermore, the inference device 2 of this embodiment configured as described above performs inference using the trained MIMO neural network obtained by the learning device 1. This makes it possible to suppress an increase in the amount of calculation.

[0240] Furthermore, the inference system 100 of the embodiment configured as described above includes the learning device 1. Therefore, an increase in the amount of calculation can be suppressed.

[0241] (Modification) The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0242] The inference device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0243] Note that all or part of the functions of the control unit 11, interface unit 12, and memory unit 13, and the control unit 21, interface unit 22, and memory unit 23 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The program may be transmitted via a telecommunications line.

[0244] The control unit 21 is an example of an inference unit. The first input learning data is an example of first unit input data. The second input learning data is an example of second unit input data. The third input learning data is an example of third unit input data.

[0245] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0246] REFERENCE SYSTEM 100 1 Learning device 2 Inference device 11 Control unit 12 Interface unit 13 Memory unit 21 Control unit 22 Interface unit 23 Memory unit 91 Processor 92 Memory 93 Processor 94 Memory

Claims

1. A learning device comprising: a control unit that performs training of a MIMO neural network, which is a neural network comprising an input layer having N (N is an integer of 2 or more) unit nodes, which are a group of nodes in the input layer and accept one unit input data that is data in a predetermined format, and an output layer that outputs a pre-defined quantity value corresponding to each of the N unit input data input to the input layer; wherein the calculation amount of the MIMO neural network is less than N times the calculation amount of a SISO neural network, which is a neural network comprising an input layer having one unit node and an output layer that outputs the quantity value corresponding to one unit input data input to the input layer; the calculation amount of the neural network is the total number of calculation processes required to execute the neural network; and the unit input data is a set of one or more values.

2. The learning device of claim 1, wherein the learning is performed under the constraint that, for each output of the MIMO neural network for each of two or more and N or less different unit input data input to the MIMO neural network, even if the unit node to which each unit input data is input is swapped, the difference in the output of the MIMO neural network corresponding to each unit input data is smaller than a predetermined difference.

3. The learning device described in claim 1, wherein the constraint is that the difference in the output of the MIMO neural network in response to first unit input data input to the MIMO neural network is smaller than a predetermined difference between a case in which the first unit input data is input to the MIMO neural network together with second unit input data different from the first unit input data, and a case in which the first unit input data is input to the MIMO neural network together with third unit input data different from the first unit input data and the second unit input data.

4. An inference device comprising: a control unit that performs training of a MIMO neural network, which is a neural network comprising an input layer having N (N is an integer of 2 or more) unit nodes, which are a group of nodes in the input layer and accept one unit input data that is data in a predetermined format, and an output layer that outputs a pre-defined quantity value corresponding to each of the N unit input data input to the input layer, wherein the calculation amount of the MIMO neural network is less than N times the calculation amount of a SISO neural network, which is a neural network comprising an input layer having one unit node and an output layer that outputs the quantity value corresponding to one unit input data input to the input layer, the calculation amount of the neural network is the total number of calculation processes necessary to execute the neural network, and the unit input data is a set of one or a plurality of values.

Images