Estimation device, learning device, estimation method, learning method, and program

Abstract

An estimation device according to the present invention comprises an estimation unit that estimates results of image capturing with an image capturing device using an estimation model that estimates 3D images of objects captured by the image capturing device on the basis of hole location information indicating the location of the hole in an aperture diaphragm in the image capturing device provided with an aperture diaphragm, wherein estimation by the estimation unit uses information indicating that the size of the hole is not non-zero.

Description

[Technical Field]

[0001] The present invention relates to an estimation device, a learning device, an estimation method, a learning method, and a program. [Background technology]

[0002] An image is a two-dimensional representation of a three-dimensional world. The inverse problem, i.e., the recovery or estimation of three-dimensional information corresponding to a given two-dimensional image, has long been a topic of interest in the fields of computer vision and computer graphics. This problem is expected to be solved in various fields, such as robotics, content generation, and image editing, and has long been the subject of intense research.

[0003] In recent years, a neural network-based method called Neural Radiance Fields (NeRF) (see Non-Patent Document 1) has been proposed to solve this problem. NeRF has neural networks c and σ that estimate the color c(p, d) and volume density σ(p) of a point in three dimensions when the coordinates p of the point and the line of sight d for observing the point are given as input.

[0004] This neural network is used to calculate the color and volume density for each point in 3D space, and then the 3D points are projected onto a 2D plane by accumulating the color of each point along a ray starting from the viewpoint, weighted by the volume density. By performing this operation for each pixel in the image, a 2D image is generated.

[0005] In this way, NeRF uses a model that explicitly describes the relationship between the 3D world and 2D images, enabling it to learn a model that can reproduce the 3D world through the process of fitting the model to real 2D images. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. Summary of the Invention [Problem to be solved by the invention]

[0007] As described above, the mathematical models proposed so far estimate and output the results of camera photography. However, these mathematical models assume a pinhole camera. Therefore, when using these mathematical models to estimate the results of photography using a camera with a non-zero aperture, the accuracy of the estimation may be poor.

[0008] Specifically, when photographing with a camera with a non-zero aperture, blurring can occur in areas outside the depth of field. However, previously proposed mathematical models assume a pinhole camera, and therefore are unable to express this blurring. As a result, a discrepancy can occur between the actual image and the generated image. This, in turn, can lead to a discrepancy between the actual 3D information (depth, etc.) and the estimated 3D information. This issue is not limited to cameras, but is a common problem for all imaging devices.

[0009] In view of the above circumstances, an object of the present invention is to provide a technique for suppressing deterioration in accuracy in estimating the results of photography by a photography device. [Means for solving the problem]

[0010] One aspect of the present invention is an estimation device that includes an estimation unit that estimates the results of photography by an imaging device using an estimation model that estimates a three-dimensional image of a subject photographed by the imaging device based on hole position information that indicates the position of a hole in the aperture of the imaging device, and the estimation by the estimation unit uses information that indicates that the size of the hole is not non-zero.

[0011] One aspect of the present invention is a learning device that includes a learning unit that learns an estimation model that estimates a three-dimensional image of an object photographed by an imaging device equipped with an aperture based on hole position information that indicates the position of a hole in the aperture of the imaging device, and the learning uses one or more learning data that include input data that is data input to a mathematical model of the object to be learned and output data that is data used for comparison with the output of the mathematical model of the object to be learned, the input data including hole position information and the output data including a two-dimensional image of the object to be learned, and the mathematical model of the object to be learned in the learning is updated so as to reduce the difference between the set of estimation results by the mathematical model of the object to be learned and the set of output data.

[0012] One aspect of the present invention is an estimation method that includes an estimation step of estimating a result of photography by an imaging device using an estimation model that estimates a three-dimensional image of a subject photographed by the imaging device based on hole position information that indicates the position of a hole in the aperture of the imaging device, and the estimation by the estimation step uses information that indicates that the size of the hole is not non-zero.

[0013] One aspect of the present invention is a learning method including a learning step of learning an estimation model that estimates a three-dimensional image of an object photographed by an imaging device equipped with an aperture, based on hole position information that indicates the position of a hole in the aperture of the imaging device, wherein the learning uses one or more pieces of learning data including input data that is data input to a mathematical model of the object to be learned and output data that is data used for comparison with the output of the mathematical model of the object to be learned, the input data including hole position information, and the output data including a two-dimensional image of the object to be learned, and the mathematical model of the object to be learned in the learning is updated so as to reduce the difference between the set of estimation results by the mathematical model of the object to be learned and the set of output data.

[0014] One aspect of the present invention is a program for causing a computer to function as either the estimation device or the learning device. [Effects of the Invention]

[0015] According to the present invention, it is possible to suppress deterioration in accuracy in estimating the results of photography by a photography device. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is an explanatory diagram illustrating an outline of an estimation system according to an embodiment. [Figure 2] FIG. 3 is an explanatory diagram illustrating an example of a projection rule according to the embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a control unit included in the learning device according to the embodiment. [Figure 5] 10 is a flowchart showing an example of a flow of processing executed by a learning device according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of an estimation apparatus according to an embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a control unit included in the estimation device according to the embodiment. [Figure 8]1 is a flowchart showing an example of a flow of processing executed by an estimation device according to an embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a result of an experiment according to the embodiment. [Figure 10] FIG. 10 is a first diagram showing an example of a result of estimation by the estimation device 2 in the modified example. [Figure 11] FIG. 2 is a second diagram showing an example of the result of estimation by the estimation device 2 in the modified example. DETAILED DESCRIPTION OF THE INVENTION

[0017] (Embodiment) FIG. 1 is an explanatory diagram illustrating an overview of an estimation system 100 according to an embodiment. Prior to describing the estimation system 100, an image captured in a two-dimensional image will be described. A two-dimensional image is a two-dimensional image obtained by capturing an image using a capturing device equipped with an aperture. The capturing device is, for example, a camera. In such a case, the two-dimensional image is, for example, a photograph. The capturing device may be, for example, a depth camera. The two-dimensional image may be, for example, a depth image. Even when the capturing device is a depth camera, the two-dimensional image does not have to be a depth image and may be a photograph.

[0018] The image captured in a 2D image is thus obtained by photography, and can be said to be the result of a 3D image being projected onto a 2D plane. Therefore, if an inverse projection corresponding to the projection that transforms a 3D image into a 2D image is obtained, the 3D image corresponding to the image captured in the 2D image can be obtained as an inverse projection of the image captured in the 2D image. Obtaining a 3D image specifically means obtaining the volume density and color of the 3D image at each position in 3D space.

[0019] The definition of volume density is used in the technical field of obtaining three-dimensional information corresponding to a two-dimensional image. Therefore, volume density is the probability that a ray of light will not penetrate.

[0020] Next, a description will be given of the estimation system 100. The estimation system 100 includes a learning device 1 and an estimation device 2.

[0021] The learning device 1 learns the three-dimensional image estimation model until a predetermined condition for ending learning (hereinafter referred to as a "learning end condition") is satisfied. Learning refers to machine learning. The learning end condition may be any condition for ending learning, and may be, for example, a condition that the mathematical model has been updated a predetermined number of times. The learning end condition may be, for example, a condition that the change in the mathematical model due to the update is smaller than a predetermined change. The mathematical model at the time when the learning end condition is satisfied is a trained mathematical model.

[0022] The 3D image estimation model is a mathematical model that estimates a 3D image of an object photographed by the photographing device based on at least hole position information. As described above, the photographing device includes an aperture. The 3D image estimation model may be a mathematical model that estimates a 3D image of an object photographed by the photographing device based further on hole orientation information.

[0023] The hole position information is information that indicates the position of the aperture hole of the imaging device. The position of the aperture hole may be indicated in any way as long as it is indicated so that the position of the aperture hole can be distinguished from other positions. Therefore, the position of the aperture hole may be indicated by, for example, the position of the center of the aperture hole.

[0024] The hole position information may indicate the position of the diaphragm hole in any manner as long as it indicates the relationship between the position of the diaphragm hole and the position of the subject to be photographed. Therefore, the hole position information may be, for example, information indicating the position of the diaphragm hole using a coordinate system to which information indicating the position where the contour of the subject to be photographed is attached.

[0025] The hole orientation information indicates the orientation of the aperture hole. The hole orientation is perpendicular to the surface of the aperture hole. Estimating a three-dimensional image specifically means estimating the volume density and color of the three-dimensional image at each position in three-dimensional space.

[0026] The 3D image estimation model includes processing based on information indicating the size of the aperture of the aperture (hereinafter referred to as "hole size information") and information indicating the focal length of the imaging device (hereinafter referred to as "focal length information"). Therefore, the 3D image estimation model is a mathematical model that estimates a 3D image of the subject being photographed based on the size of the aperture of the aperture indicated by the hole size information and the focal length included in the focal length information. The size of the aperture of the aperture is, for example, the radius of the aperture of the aperture.

[0027] In the 3D image estimation model, the orientation, size, or focal length of the aperture hole may be a parameter updated through learning, or may be a predetermined value given in advance. Specifically, the orientation, size, or focal length of the aperture hole may be set for each piece of input data included in the learning data described below, or the orientation, size, or focal length of the aperture hole may be set for each piece of output data included in the learning data. Furthermore, the orientation, size, or focal length of the aperture hole may be included in the 3D image estimation model as one of the parameters updated through learning. Furthermore, a single value may be set for the orientation, size, or focal length of the aperture hole, or the orientation, size, or focal length of the aperture hole may be set to represent the distribution of the parameter.

[0028] The learning data used in training a 3D image estimation model includes input data and output data. The input data is data input to the mathematical model to be trained. The output data is data used for comparison with the output of the mathematical model to be trained. Hereinafter, the learning data used in the 3D image estimation model will be referred to as 3D learning data. The mathematical model to be trained in training using 3D learning data is the 3D image estimation model.

[0029] The output data in the 3D training data includes a 2D image of the photographed object (hereinafter referred to as a "2D image of the object"). The input data in the 3D training data is information including at least hole position information. Note that the hole position information included in the input data in the 3D training data may be set to a single value, or a value sampled from a predetermined distribution may be set. Furthermore, the hole position information may be set for each piece of output data included in the training data, or the hole position information may be set independently of the output data. Furthermore, the value of the hole position information may be estimated from each piece of output data. Furthermore, the hole position information may be optimized simultaneously with the training of the 3D image estimation model as one of the parameters updated by training.

[0030] The input data in the three-dimensional learning data may include hole orientation information. However, the hole orientation information does not necessarily have to be included in the input data of the three-dimensional learning data. If the input data of the three-dimensional learning data does not include hole orientation information, the hole orientation information may be stored in advance in a predetermined storage device such as the storage unit 14 described below. In such a case, when the three-dimensional image estimation model is executed, the hole orientation information may be read from the predetermined storage device and used for estimation by the three-dimensional image estimation model.

[0031] In the learning of the 3D image estimation model, a 3D image estimation process, a 2D image estimation process, and an update process are executed. The 3D image estimation process is a process for estimating a 3D image of the object being photographed by executing the 3D image estimation model. The 3D image estimation process is executed on input data including 3D learning data.

[0032] The two-dimensional image estimation process is a process for obtaining an estimated result image based on a three-dimensional image estimated by the three-dimensional image estimation process. The estimated result image is a two-dimensional image obtained by an imaging device in which the aperture is located at the position indicated by the hole position information. The estimated result image is a two-dimensional image according to the content of the two-dimensional image estimation process, and may be, for example, a photograph or a depth image.

[0033] The two-dimensional image obtained by the imaging device is the result of imaging by the imaging device. Therefore, the two-dimensional image estimation process can also be said to be a process of estimating the result of imaging by an imaging device in which the aperture is located at the position indicated by the hole position information, based on the three-dimensional image estimated by the three-dimensional image estimation process. Furthermore, the estimated result image is a two-dimensional image obtained by the two-dimensional image estimation process. Therefore, the estimated result image is a two-dimensional image obtained based on the result of estimation of the three-dimensional image estimation model.

[0034] The two-dimensional image estimation process may be any process that estimates an estimation result image based on a three-dimensional image estimated by the three-dimensional image estimation process.

[0035] The two-dimensional image estimation process may be, for example, a process of obtaining a three-dimensional image according to a predetermined rule for obtaining a three-dimensional image from a two-dimensional image (hereinafter referred to as an "inverse projection rule"), and then obtaining a two-dimensional image based on the three-dimensional image according to a predetermined rule for obtaining a two-dimensional image (hereinafter referred to as a "projection rule"). Therefore, the two-dimensional image estimation process is, for example, a process of obtaining a three-dimensional image from a two-dimensional image (e.g., a photograph or a depth image) according to an inverse projection rule, and then obtaining the same or a different type of two-dimensional image (e.g., a photograph or a depth image) from the three-dimensional image according to a projection rule. An example of a process of obtaining a three-dimensional image according to an inverse projection rule is a process of obtaining hole position information based on a two-dimensional image, and then obtaining a three-dimensional image from the hole position information by three-dimensional image estimation process.

[0036] The process of obtaining a two-dimensional image according to the projection rule may be, for example, a process of executing a previously obtained two-dimensional image estimation model. The two-dimensional image estimation model is a mathematical model that estimates a two-dimensional image according to the projection rule.

[0037] Here, an example of the projection rule will be described.

[0038] <Example of projection rules> FIG. 2 is an explanatory diagram illustrating an example of a projection rule in an embodiment. Note that, hereinafter, the terms "ray" and "direction of ray" are used, and the definitions of each term are those used in the technical field in which three-dimensional information corresponding to a given two-dimensional image is obtained. That is, a ray refers to the path along which light propagates. The direction of a ray refers to the positive direction of the ray. The positive direction of a ray is the direction in which the subject is viewed from the aperture.

[0039] In the projection rule (hereinafter referred to as the "exemplary rule") explained using the explanatory diagram of FIG. 2, the shape of the aperture hole is circular. Note that, as an example, the case where the shape of the hole is circular will be explained here, but it may be any shape such as a regular polygon. The exemplary rule is a rule that uses information indicating the size of the aperture hole, and is a rule that expresses a phenomenon that occurs when photographing with an imaging device. In the exemplary rule, a ray that passes through the aperture is a ray whose origin is a position vector o' expressed by the following equation (1). The origin of a ray means the starting point of a vector that indicates the direction of the ray.

[0040]

number

[0041] Here, vector o is a position vector indicating the center of the aperture hole. Vector u is a vector that is orthogonal to vector o and has a magnitude between 0 and s, where s is the radius of the aperture hole. Therefore, vector o' is a position vector that indicates a position within a circle with center o and radius s.

[0042] The direction d' of the ray at the origin o' is expressed by the following equation (2) using the center o of the aperture hole.

[0043]

number

[0044] Vector d is a vector that indicates the direction of the aperture hole. Value f represents the distance to the focal plane. Therefore, value f is a non-negative real number. The focal plane is defined as a plane where a ray convergence point exists. A ray convergence point is a point where a group of rays passing through the aperture converge. A ray group is defined as multiple rays. In the example in Figure 2, the ray convergence point is point P1. In the example in Figure 2, the focal plane is plane H1.

[0045] From equations (1) and (2), if vector o, vector u, and distance f are given, it is possible to calculate the origin o' and direction d'. Furthermore, once the origin o' and direction d' are calculated, a vector representing a ray r' originating from point o' can also be obtained. The ray r' originating from point o' is expressed by the following equation (3).

[0046]

number

[0047] t is t n More than t f The following real numbers are used: t n and t f What is t n <t f are real numbers in the relationship t n and t f The term "area" refers to a range that includes the area where the light rays intersect with the three-dimensional image of the subject being photographed, and includes the area that includes the area where the light rays intersect with the three-dimensional image of the subject being photographed.

[0048] Once equation (3) is obtained, the color C(r') and depth Z(r') of the pixel on the image plane corresponding to ray r' can be obtained by executing a process called Volume Rendering, which is expressed by the following equations (4) to (6). Note that the image plane corresponding to ray r' is plane H1 in Figure 2. In other words, the image plane corresponding to ray r' is the focal plane.

[0049] In the following formulas (4) to (6), for simplicity of expression, r is used instead of r' and d is used instead of d'.

[0050]

number

[0051]

number

[0052]

number

[0053] c(p, d) is a value indicating the color at position p and direction d. σ(p) indicates the volume density at position p.

[0054] Note that equations (4) to (6) require integral calculations, but this may be difficult to perform. This is because integrals are defined for continuous quantities, but it is difficult for a computer to handle continuous quantities. Therefore, instead of integrating equations (4) to (6), a computer may obtain an approximation of the integrals of equations (4) to (6) using discretized points. That is, for example, the integrals may be approximately calculated for the discretized points. For example, the integrals may be approximately calculated for points obtained by dividing the integral range at predetermined intervals. Furthermore, for example, the distribution of points may be weighted based on the results of a previous calculation, and the integrals may be approximately calculated for the points obtained as a result of resampling.

[0055] Regarding the integration, for example, for each discretized representative origin o', the color and depth of the pixel for the corresponding ray r' may be calculated, and the average of the obtained multiple colors and depths may be obtained instead of the integration result. The representative origin is a point located in the aperture hole, selected according to a predetermined rule.

[0056] Therefore, the representative origin may be, for example, a point selected randomly from among the points located in the aperture, or may be a point selected according to a rule at predetermined intervals from among the points located in the aperture.Furthermore, points located in the aperture may be selected with a focus on points that are likely to have an object on a light ray originating from that point.

[0057] The example rule shows the execution of a full integration process. The full integration process is a process of integrating at least the pixel color C(r') for all rays r' that satisfy the condition that the magnitude of the vector u is greater than or equal to 0 and less than s. In the full integration process, the depth Z(r') may also be integrated for all rays r' that satisfy the condition that the magnitude of the vector u is greater than or equal to 0 and less than s.

[0058] However, in the example rule, it is not necessary to perform integration for all rays r' that satisfy the condition that the magnitude of the vector u is greater than or equal to 0 and less than s. The depth may be, for example, the depth Z(r) obtained for the central ray r. Note that the central ray r is a ray that originates from the center of the aperture hole.

[0059] An example rule is one in which the information indicating the color or depth obtained for each pixel in this way is output as a two-dimensional image.

[0060] In the case of a rule such as the example rule that indicates that the value of each pixel is obtained using not only a value obtained based on a single ray but also a value obtained based on a group of rays, a two-dimensional image obtained according to that rule expresses the influence of the depth-of-field effect (i.e., the bokeh effect). In a two-dimensional image obtained in this manner, the image is in focus where all the rays entering the aperture intersect at a single point, and blur occurs where the group of rays is spread out. Hereinafter, a rule that indicates that the value of each pixel is obtained using not only a value obtained based on a single ray but also a value obtained based on a group of rays is referred to as a bokeh effect estimation rule.

[0061] The process expressed by equations (1) to (6) and the total integration process may also be included in the three-dimensional image estimation model. In such a case, the process expressed by equations (1) to (6) and the total integration process are examples of processes based on hole size information and focal length information. Now, let's return to the explanation of Figure 1.

[0062] The update process is a process of updating the 3D image estimation model so as to reduce the difference between the set of 2D images obtained by the 2D image estimation process (hereinafter referred to as "estimated 2D images") and the set of target 2D images. Updating the mathematical model means updating the parameter values ​​of the mathematical model. Note that a set here refers to a collection of data with one or more elements.

[0063] Specifically, the update process may update the 3D image estimation model so as to reduce the difference between the estimated 2D image and the target 2D image while maintaining a one-to-one correspondence between them, or may update the 3D image estimation model so as to reduce the difference between the estimated 2D image group and the target 2D image group as a whole. Note that the estimated 2D image group is a collection of estimated 2D images with one or more elements, and the target 2D image group is a collection of target 2D images with one or more elements.

[0064] Specifically, when minimizing the difference while assuring one-to-one correspondence between the estimated 2D image and the target 2D image, the 3D image estimation model may be trained using a loss function based on any distance criterion. The loss function may be, for example, a function based on the L2 distance, a function based on the L1 distance, or a function based on the Wasserstein distance. Alternatively, the loss function may be a hinge function that allows a difference of a certain value or less. A combination of these loss functions may also be used.

[0065] When minimizing the difference between the estimated 2D image group and the target 2D image group as a whole, the 3D image estimation model may be trained using a loss function based on any generative model. The generative model may be, for example, a generative adversarial network (GAN), a variational autoencoder (VAE), a flow model, a diffusion probabilistic model, or an autoregressive model. Furthermore, a combination of these generative models may also be used.

[0066] Note that learning of a 3D image estimation model using a GAN is an example of learning (hereinafter referred to as "competitive learning") in which the estimation unit 211 described later is used as a generator, and a classifier that distinguishes between a set of estimation results by the generator and a set of output data is included, and the generator and classifier learn the learning object according to mutually competing optimization conditions. That is, the learning of the 3D image estimation model may be performed by, for example, competitive learning, and a GAN may be used as the competitive learning.

[0067] The estimation device 2 estimates a 2D image of the object using the 3D image estimation model obtained by the learning device 1. For simplicity of the following explanation, the estimation system 100 will be described using an example in which the 3D learning data includes output data.

[0068] 3 is a diagram showing an example of the hardware configuration of a learning device 1 according to an embodiment. The learning device 1 includes a control unit 11 having a processor 91, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and a memory 92, which are connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0069] More specifically, the processor 91 reads out a program stored in the storage unit 14 and stores the read out program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0070] The control unit 11 controls the operations of various functional units included in the learning device 1. The control unit 11 executes, for example, a three-dimensional image estimation process, a two-dimensional image estimation process, and an update process.

[0071] Input unit 12 includes input devices such as a mouse, keyboard, and touch panel. Input unit 12 may be configured as an interface that connects these input devices to learning device 1. Input unit 12 accepts input of various information to learning device 1. For example, a user's instruction to start learning is input to input unit 12. For example, three-dimensional learning data is input to input unit 12.

[0072] The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits 3D learning data. The communication unit 13 acquires the 3D learning data by communicating with the device that transmits the 3D learning data. Note that the sources of the input data and output data of the 3D learning data may be different devices.

[0073] The storage unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 14 stores various information related to the learning device 1. The storage unit 14 stores information input via, for example, the input unit 12 or the communication unit 13. The storage unit 14 stores, for example, a 3D image estimation model. The storage unit 14 stores, for example, a trained 3D image estimation model. The storage unit 14 may or may not store, for example, hole orientation information in advance.

[0074] The output unit 15 outputs various types of information. The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the learning device 1. The output unit 15 outputs information input to the input unit 12 or the communication unit 13, for example.

[0075] 4 is a diagram showing an example of the configuration of the control unit 11 included in the learning device 1 according to the embodiment. The control unit 11 includes a learning unit 111, an input control unit 112, a communication control unit 113, a memory control unit 114, and an output control unit 115.

[0076] The learning unit 111 learns the 3D image estimation model. Therefore, the learning unit 111 executes 3D image estimation processing, 2D image estimation processing, and update processing. The input control unit 112 controls the operation of the input unit 12. The communication control unit 113 controls the operation of the communication unit 13. The memory control unit 114 controls the operation of the memory unit 14. The output control unit 115 controls the operation of the output unit 15.

[0077] 5 is a flowchart showing an example of the flow of processing executed by the learning device 1 in this embodiment. One or more pieces of 3D learning data are input to the input unit 12 or the communication unit 13 (step S101). Next, the learning unit 111 executes a 3D image estimation process on each piece of input data included in each piece of 3D learning data (step S102).

[0078] Next, the learning unit 111 executes a two-dimensional image estimation process (step S103). By executing the two-dimensional image estimation process, a two-dimensional image obtained by an imaging device in which an aperture is located at the position indicated by the hole position information is estimated as an estimated result image based on the result of estimation by the three-dimensional image estimation process. The hole position information is information included in the input data included in the three-dimensional learning data.

[0079] Next, the learning unit 111 executes an update process (step S104). In the update process, the 3D image estimation model is updated based on the difference between the set of estimation result images obtained in step S103 and the set of 2D images of the shooting target so as to reduce the difference. The 2D images of the shooting target are included in the 3D learning data as output data.

[0080] Next, the learning unit 111 determines whether the learning end condition is satisfied (step S105). If the learning end condition is satisfied (step S105: YES), the process ends. On the other hand, if the learning end condition is not satisfied (step S105: NO), the process returns to step S101.

[0081] 6 is a diagram illustrating an example of a hardware configuration of an estimation device 2 according to an embodiment. The estimation device 2 includes a control unit 21 having a processor 93 such as a CPU or a GPU and a memory 94 connected via a bus, and executes a program. By executing the program, the estimation device 2 functions as a device including the control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.

[0082] The control unit 21 controls the operations of various functional units included in the estimation device 2. The control unit 21 executes, for example, a trained three-dimensional image estimation model.

[0083] The input unit 22 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 22 may be configured as an interface that connects these input devices to the estimation device 2. The input unit 22 accepts input of various information to the estimation device 2. For example, a command to start estimation from a user is input to the input unit 22.

[0084] To the input unit 22, for example, information to be input to the trained three-dimensional image estimation model (hereinafter referred to as "input information") is input. The input information is the same type of information as the input side data included in the three-dimensional training data. Therefore, the input information includes at least hole position information. When the input side data of the three-dimensional training data includes hole orientation information, the input information further includes hole orientation information.

[0085] The communication unit 23 includes a communication interface for connecting the estimation device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless communication. The external device is, for example, a device that has transmitted hole position information. The communication unit 23 acquires input information by communicating with the device that has transmitted the input information.

[0086] The storage unit 24 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 24 stores various information related to the estimation device 2. The storage unit 24 stores information input via, for example, the input unit 22 or the communication unit 23. The storage unit 24 stores, for example, a trained three-dimensional image estimation model. The storage unit 24 may or may not store, for example, hole orientation information in advance.

[0087] The output unit 25 outputs various types of information. The output unit 25 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 25 may be configured as an interface that connects these display devices to the estimation device 2. The output unit 25 outputs information input to the input unit 22 or the communication unit 23, for example.

[0088] 7 is a diagram illustrating an example of the control unit 21 included in the estimation device 2 according to the embodiment. The control unit 21 includes an estimation unit 211, an input control unit 212, a communication control unit 213, a storage control unit 214, and an output control unit 215.

[0089] The estimation unit 211 executes the trained three-dimensional image estimation model. More specifically, the estimation unit 211 estimates a three-dimensional image based on input information by executing the trained three-dimensional image estimation model. After executing the trained three-dimensional image estimation model, the estimation unit 211 further executes two-dimensional image estimation processing. By executing the two-dimensional image estimation processing, the estimation unit 211 obtains the results of shooting using an imaging device equipped with an aperture that satisfies the conditions indicated by the input information, based on the three-dimensional image estimated by the trained three-dimensional image estimation model.

[0090] The input control unit 212 controls the operation of the input unit 22. The communication control unit 213 controls the operation of the communication unit 23. The memory control unit 214 controls the operation of the memory unit 24. The output control unit 215 controls the operation of the output unit 25.

[0091] 8 is a flowchart showing an example of the flow of processing executed by the estimation device 2 in the embodiment. Input information is input to the input unit 22 or the communication unit 23 (step S201). That is, the input unit 22 or the communication unit 23 receives at least the input of hole position information.

[0092] Next, the estimation unit 211 uses the trained three-dimensional image estimation model to estimate the result of shooting by the imaging device in which the aperture is located at the position indicated by the input information (step S202). More specifically, the estimation unit 211 first executes the trained three-dimensional image estimation model, and then executes two-dimensional image estimation processing to estimate the result of shooting by the imaging device in which the aperture is located at the position indicated by the input information. Next, the output control unit 215 controls the operation of the output unit 25 to cause the output unit 25 to output the estimation result obtained in step S202 (step S203).

[0093] The estimation system 100 of this embodiment configured as described above includes a learning device 1. The learning device 1 updates, through learning, a 3D image estimation model that includes processing based on information indicating the size of the aperture and information indicating the focal length of the imaging device. Therefore, the estimation system 100 can suppress deterioration in the accuracy of estimating the results of imaging by the imaging device even when a dataset includes blurred images.

[0094] (Variation) The three-dimensional image estimation model may estimate a three-dimensional image according to the object being photographed. That is, the three-dimensional image estimation model may include a latent variable z, which is a quantity indicating the object being photographed, as one of the parameters updated by learning. In such a case, the three-dimensional image estimation model includes information for identifying the object being photographed (hereinafter referred to as "object identification information"). The object identification information may be included in the input data.

[0095] The latent variable z may follow any predetermined distribution, such as a Gaussian distribution, a uniform distribution, a binomial distribution, a multinomial distribution, etc. The value of the latent variable z may be estimated using a neural network or the like when additional information such as an image is given.

[0096] Even if the 3D image estimation model includes a latent variable z, which is a quantity that identifies the subject of the photograph, as one of its parameters, the 2D image estimation process is the same as when the 3D image estimation model does not include a latent variable z, which is a quantity that identifies the subject of the photograph, as one of its parameters.

[0097] The latent variable z may also be used during estimation by the estimation device 2. That is, the input information may include the latent variable z.

[0098] The machine learning method used to train the three-dimensional image estimation model may be any method as long as it is a machine learning method that can update the three-dimensional image estimation model using three-dimensional training data.

[0099] The machine learning method used to train the 3D image estimation model may be, for example, a method of updating the 3D image estimation model so as to reduce the difference between a set of estimated 2D images and a set of target 2D images. If the machine learning method used to train the 3D image estimation model is a method of reducing the difference while establishing a one-to-one correspondence between estimated 2D images and target 2D images, the method may be a method of training the 3D image estimation model using a loss function based on any distance criterion. The loss function may be, for example, a function based on the L2 distance, a function based on the L1 distance, or a function based on the Wasserstein distance. The loss function may also be a hinge function that allows a difference below a certain value. A combination of these loss functions may also be used.

[0100] When minimizing the difference between the estimated 2D image group and the target 2D image group as a whole, the 3D image estimation model may be trained using a loss function based on any generative model. The generative model may be, for example, a GAN, a VAE, a flow model, a diffusion probabilistic model, or an autoregressive model. Furthermore, a combination of these generative models may also be used.

[0101] When learning using a GAN-based method, the loss function is expressed, for example, by the following equation (7).

[0102]

number

[0103] Among the symbols in equation (7), I r ~p r The symbol (I) represents the target 2D image distribution p r (I) Based on the target 2D image I r Among the symbols in equation (7), z~p g The symbol (z) represents the latent variable distribution p grepresents the process of sampling the latent variable z based on (z).

[0104] As mentioned above, the latent variable distribution p g (z) follows a predetermined arbitrary distribution such as a Gaussian distribution, a uniform distribution, a binomial distribution, or a multinomial distribution. In such cases, parameters representing the shape of the distribution, such as the mean and variance, may be included in the 3D image estimation model as learnable parameters and optimized during training. The value of z may be estimated using a neural network or the like when additional information, such as an image, is provided.

[0105] Among the symbols in formula (7), D represents a classifier in GAN. That is, the symbol D represents a classifier that distinguishes between real images and generated images. The classifier D is optimized to increase the accuracy of distinguishing between real images and generated images by maximizing the value of formula (7).

[0106] Among the symbols in formula (7), G represents a generator in GAN. The generator G is optimized to reduce the accuracy of classification by the classifier D by minimizing the value of formula (7). By optimizing under the competitive condition where one maximizes and the other minimizes, the generator G can generate an image that the classifier D will not determine to be a real image. The estimation device 2 is an example of the generator G.

[0107] In learning using a GAN-based method, the loss function does not necessarily have to be a loss function based on cross-entropy such as Equation (7). The loss function may be a loss function based on any predetermined distance criterion. For example, the loss function may be a function based on L2 distance, L1 distance, or Wasserstein distance. The loss function may also be a hinge function that allows differences below a certain value. A combination of these loss functions may also be used.

[0108] When the loss function is Equation (7), for example, log(1-D(G(z))) is minimized when optimizing G. However, instead of minimizing log(1-D(G(z))), -logD(G(z)) may be minimized.

[0109] Furthermore, in equation (7), learning may be performed while independently sampling the aperture size s, the focal length f of the imaging device, and the latent variable z. For example, when the aperture size s and the latent variable z are independently sampled, equation (7) can be replaced with the following equation (8):

[0110]

number

[0111] In equation (8), the generator G is expressed as G(z, s) to make it clear that G depends on s.

[0112] p g (s) may follow any predetermined distribution, such as a half-normal distribution, a uniform distribution with positive values, a binomial distribution, or a multinomial distribution. In such cases, parameters that represent the shape of the distribution, such as the mean and variance, may be included in the 3D image estimation model as learnable parameters and optimized during training. The value of s may be estimated using a neural network or the like when additional information, such as an image, is provided.

[0113] <Experimental Results> In the experiment, a depth estimator (hereinafter referred to as "target model depth estimator") trained using photographs and depth images obtained based on a trained 3D image estimation model was used to evaluate the performance of the learning device 1 and the estimation device 2. Specifically, a 3D image was first obtained using the trained 3D image estimation model, and then a pair of photographs and depth images were estimated by a 2D image estimation process. Next, the pair of photographs and depth images were used as training data to train the target model depth estimator, which converts photographs into depth images.

[0114] Then, depth images estimated from the evaluation photographs using a target model depth estimator (hereinafter referred to as "target model depth images") were used for evaluation. A mathematical model based on a pinhole camera was used as the comparison technology (i.e., the baseline method). Specifically, a depth estimator (hereinafter referred to as "baseline depth estimator") was trained using photographs and depth images obtained based on this mathematical model, and depth images estimated from the evaluation photographs using the baseline depth estimator (hereinafter referred to as "baseline depth images") were used for evaluation.

[0115] In the experiment, the degree of match between the estimated depth image (i.e., the target model depth image or the baseline depth image) and a predetermined standard was used as an index for evaluating the learning device 1. The degree of match was measured using SIDE (Scale-Invariant Depth Error). The smaller the SIDE value, the higher the degree of match and the better the performance.

[0116] Specifically, a depth image (hereinafter referred to as a "reference depth image") estimated from an evaluation photograph using a depth estimator known to have high performance and trained using a large number of diverse stereo images as training data was used as a predetermined reference. Therefore, in the experiment, a depth image (i.e., a target model depth image or a baseline depth image) estimated by a depth estimator trained based on a mathematical model to be evaluated (i.e., a target model depth estimator or a baseline depth estimator) was compared with the reference depth image, and an evaluation was made in which the higher the degree of match, the higher the performance of the mathematical model to be evaluated.

[0117] FIG. 9 is a diagram showing an example of experimental results in a modified example. FIG. 9 shows experimental results for a "baseline" and a trained 3D estimation model (hereinafter referred to as a "target model") obtained by the learning device 1 through learning using the above-described GAN-based method. The "baseline" technology is a technology that is compared with the target model. FIG. 9 shows that three types of datasets, "flower images," "bird images," and "face images," were used as datasets during both learning and estimation. "Flower images" refers to images of flowers. "Bird images" refers to images of birds. "Face images" refers to images of faces.

[0118] Figure 9 shows that the target model has a smaller SIDE value than the comparison technology for both types of datasets. In other words, Figure 9 shows that the target model has higher estimation accuracy than the comparison technology for both types of datasets.

[0119] Note that the dataset used during learning or estimation may include blurred images.

[0120] Note that the case where the aperture hole size s and the focal length f are sampled independently, and the case where the focal length f and the latent variable z are sampled independently, are the same as the case where the hole size s and the latent variable z are sampled independently. Also, the case where s, f, and z are sampled independently is the same as the case where the hole size s and the latent variable z are sampled independently.

[0121] In this way, by learning while sampling each variable independently, a separate representation for each variable is obtained, and as a result, the generator G can control each variable independently.

[0122] The significance of the case where learning is performed while s and z are sampled independently will be explained. If z is fixed and only s is changed, the generator G can change only the depth-of-field effect while keeping the image content fixed. Specifically, the image content refers to content other than the depth-of-field effect. Also, if s is fixed and only z is changed, the generator G can change only the image content while keeping the depth-of-field effect fixed.

[0123] The three-dimensional image estimation model is configured, for example, by a neural network. The three-dimensional image estimation model is, for example, a neural network that estimates color and volume density. Such a neural network may be, for example, a neural network that estimates color and volume density using different neural networks.

[0124] Furthermore, the neural network for estimating color and volume density may be a neural network in which the neural network for estimating color and the neural network for estimating volume density share at least a portion. For example, the neural network for estimating color and volume density may be a neural network in which the first half of the network estimates volume density and the second half of the network estimates color.

[0125] Furthermore, in training a 3D image estimation model including a latent variable z, the latent variable z of the neural network that estimates color and the latent variable z of the neural network that estimates volume density may be sampled independently. Also, some of these latent variables z may be sampled independently, while others may be sampled jointly. Note that a 3D estimation model including a latent variable z refers to a 3D estimation model that includes the latent variable z as one of the parameters updated by training.

[0126] In addition, the color and volume density of each position in a predefined three-dimensional space that includes a three-dimensional image may be estimated using the same neural network regardless of the position, or may be estimated using a neural network according to the position in the space. For example, they may be estimated using different neural networks for the foreground and the background.

[0127] When executing the neural network for the background, Inverted Sphere Parameterization may be used as the coordinate system. This is because by using Inverted Sphere Parameterization in the background rather than the foreground, it is possible to densely sample points in the vicinity and sparsely sample points in the distance, thus enabling efficient representation of a wide range. Note that the definition of the foreground in the three-dimensional space is the image closer to the viewpoint. Therefore, for example, if the two-dimensional image is a portrait image, the foreground in the three-dimensional space is the image where the person is located, and the definition of the background in the three-dimensional space is the image farther from the viewpoint. Therefore, for example, if the two-dimensional image is a portrait image, the background in the three-dimensional space is the background behind the person.

[0128] <Details of the effect of using Inverted Sphere Parameterization> (x´, y´, z´, 1 / w)(x´ 2 +y´ 2 +z´ 2 =1, 0 <= 1 / w <= 1), when this coordinate system is used, a linearly equidistant coordinate system (x, y, z) is used. On the other hand, in the case of Inverted Sphere Parameterization, the position of a point is represented by the direction (x´, y´, z´) seen from the center of the circle and the distance 1 / w. Here, when w is linearly equidistant, 1 / w is dense near the center of the circle and sparse as it moves away. Therefore, when Inverted Sphere Parameterization is used, it has the effect of densely sampling points in the vicinity and sparsely sampling points in the distance.

[0129] In addition, since the foreground contains the main object, it is sampled densely, while the background does not contain the main object, so the amount of calculation can be reduced by sampling sparsely.In particular, in the background, the further away the object is, the smaller it becomes, so there is little degradation in image quality caused by sampling sparsely.

[0130] When the 3D image estimation model includes a latent variable z as one of the parameters updated by learning, the neural network for estimating color is expressed as c(p, d, z), for example. That is, the neural network for estimating color is expressed as a function that depends on p, d, and z, for example.

[0131] In this case, the neural network for estimating the volume density is expressed as σ(p, z). That is, the neural network for estimating the volume density is expressed as a function that depends on p and z, for example. Note that p represents the position of the aperture hole, and d represents the orientation of the aperture hole.

[0132] The latent variable z in the neural network for estimating color and the neural network for estimating volume density does not necessarily have to be the same, and may be different. For example, if the neural network for estimating color uses c(p, d, z c ) and the neural network that estimates the volume density is σ(p, z σ ) may also be used. c and z σ The above does not necessarily mean that all of the components are different, and some components may be shared.

[0133] The neural network for estimating color may be, for example, c(p, z), that is, the neural network for estimating color may be expressed as a function that depends on p and z, for example.

[0134] In addition, in training the 3D image estimation model, in addition to the size s of the aperture, the focal length f of the imaging device, and the latent variable z, the position p of the aperture may also be sampled independently. In this case, it is possible to train separate representations of the size s of the aperture, the focal length f of the imaging device, the latent variable z, and the position p of the aperture.

[0135] In such cases, the 3D image estimation model uses volume rendering to represent the depth of field effect and the change in position p in a unified framework. Therefore, by simultaneously learning the depth of field effect and position p, the estimation accuracy of the 3D image estimation model is improved.

[0136] Furthermore, by using the 3D image estimation model trained in this way, the user can obtain a 2D image while independently controlling the variables s, f, z, and p.

[0137] Although the above describes the case where s, f, z, and p are all sampled independently while learning, it is also possible to sample only some of them independently while learning. In this case, it is possible to learn separated expressions for the independently sampled items.

[0138] It should be noted that the estimation device 2 does not necessarily need to estimate a 2D image using a 3D image estimation model. The estimation device 2 may obtain a 2D image in any way as long as it obtains the 2D image using a 3D image estimated by a mathematical model that satisfies predetermined model conditions based on hole position information, hole size information, and focal length information in accordance with the blur effect estimation rules. It should be noted that the estimation device 2 may also perform estimation based on hole orientation information in addition to hole position information, hole size information, and focal length information.

[0139] As described above, the bokeh effect estimation rules express the influence of the depth of field effect (i.e., the bokeh effect) in a two-dimensional image obtained in accordance with the rules. Therefore, an estimation device 2 that obtains a two-dimensional image using a three-dimensional image that satisfies the model conditions based on hole position information, hole size information, and focal length information in accordance with the bokeh effect estimation rules can obtain a two-dimensional image that expresses the influence of the depth of field effect, even without using a three-dimensional image estimation model. Therefore, such an estimation device 2 can suppress deterioration in the accuracy of estimating the results of photography by a photography device compared to techniques that do not follow the bokeh effect estimation rules.

[0140] The model conditions include a condition that the color and volume density of a 3D image of the object captured by the imaging device are estimated based on the hole position information. Such a 3D image estimation model is a mathematical model that estimates the color c(p) and volume density σ(p) based on the position p.

[0141] The model condition may further include a condition that the color and volume density of the 3D image of the object captured by the imaging device are estimated based on not only the hole position information but also the hole orientation information. Such a 3D image estimation model is a mathematical model that estimates the color c(p, d) and volume density σ(p, d) based on the position p and the orientation d.

[0142] As described above, c(p, d) indicates the color at position p and orientation d, and σ(p) indicates the volume density at position p. A mathematical model that satisfies the model conditions is, for example, the above-mentioned 3D image estimation model. A mathematical model that satisfies the model conditions may be, for example, the mathematical model described in Non-Patent Document 1 above, which is a mathematical model obtained by learning assuming a pinhole camera. Furthermore, a mathematical model that satisfies the model conditions may be the mathematical model obtained by learning assuming a camera with an aperture.

[0143] The model condition may further include a condition that the color and volume density of the 3D image of the object photographed by the photographing device are estimated based on not only the hole position information but also the photographing object identification information. Such a 3D image estimation model is a mathematical model that estimates the color c(p,z) and volume density σ(p,z) based on the position p and the latent variable z.

[0144] The model condition may further include a condition that the color and volume density of the 3D image of the object photographed by the photographing device are estimated based on not only the hole position information but also the hole orientation information and the photographing object identification information. Such a 3D image estimation model is a mathematical model that estimates the color c(p, d, z) and the volume density σ(p, z) based on the position p, the orientation d, and the latent variable z.

[0145] <About the size of the aperture hole in 2D image estimation processing> The two-dimensional image estimation process is performed both during learning by the learning device 1 and during estimation by the estimation device 2. The two-dimensional image estimation process is a process of estimating a two-dimensional image from a three-dimensional image according to a projection rule, as described above. The exemplary rule was explained as an example of a projection rule, and in the exemplary rule, the size of the aperture hole was used in estimating the two-dimensional image. In fact, the size of the aperture hole in the exemplary rule does not need to be non-zero and can be zero. An imaging device with an aperture hole size of zero is a pinhole camera.

[0146] Therefore, for example, the size of the aperture hole may be zero in the projection rule in the two-dimensional image estimation process performed by learning device 1, and the size of the aperture hole may be non-zero in the projection rule in the two-dimensional image estimation process performed by estimation device 2. Also, for example, the size of the aperture hole may be non-zero in the projection rule in the two-dimensional image estimation process performed by learning device 1, and the size of the aperture hole may be zero in the projection rule in the two-dimensional image estimation process performed by estimation device 2.

[0147] Furthermore, for example, the size of the aperture hole may be non-zero in the projection rule in the two-dimensional image estimation process performed by the learning device 1, and the size of the aperture hole may also be non-zero in the projection rule in the two-dimensional image estimation process performed by the estimation device 2. In other words, it is sufficient that the size of the aperture hole is non-zero in the two-dimensional image estimation process performed by at least one of the learning device 1 and the estimation device 2.

[0148] This is because, in either case, estimation by the estimation device 2 includes information about the size of the aperture, which is non-zero. As a result, the estimation device 2 can estimate a blurred image, which is the image expected by the user of the estimation device 2. Therefore, the estimation device 2 configured in this way can suppress deterioration in the accuracy of estimating the results of photography by the photography device.

[0149] <When "Learning: Aperture size is non-zero"> A blurred image can be estimated by setting the size of the aperture in the projection rule to a non-zero value in the 2D image estimation process executed by the learning device 1. This makes it possible to estimate an image close to a blurred image even if the output data of the 3D learning data contains a blurred image, making learning easier (making it easier to bring the estimation result closer to the learning data), and as a result, improving the 3D image estimation accuracy of the 3D image estimation model. Furthermore, as the accuracy of the 3D image estimation model increases, the estimation accuracy of the 2D image estimated based on it also increases.

[0150] <When "Estimation: Aperture size is non-zero"> A blurred image can be estimated by setting the size of the aperture hole in the projection rule to a non-zero value in the 2D image estimation process executed by the estimation device 2. As a result, when attempting to reproduce the results of photography using a camera with an aperture, specifically when attempting to reproduce the depth of field effect caused by manipulating the focus position or changing the aperture size, the device can express the blurred effect, thereby improving the estimation accuracy of the 2D image.

[0151] In this way, if information indicating that the size of the aperture hole is not non-zero is used in the estimation by the estimation device 2, it is possible to suppress deterioration in the accuracy of estimating the result of image capture by the image capture device. Specifically, the estimation by the estimation device 2 is performed by the estimation unit 211. Therefore, the estimation by the estimation device 2 described in the modified example is also performed in step S202. In this case, the processing of step S202 is processing in which the estimation unit 211 estimates the result of image capture by the image capture device in which the aperture is located at the position indicated by the input information, using a mathematical model that satisfies the model conditions instead of the trained three-dimensional image estimation model.

[0152] A user using such an estimation device 2 can obtain a two-dimensional image with a changed degree of blur by changing the size indicated by the hole size information. Also, a user using such an estimation device 2 can obtain a two-dimensional image with a changed focus position by changing the focal length indicated by the focal length information. Furthermore, by using such an estimation device 2, a user can also obtain a depth image.

[0153] FIG. 10 is a first diagram showing an example of the result of estimation by the estimation device 2 in the modified example. Image G101 in FIG. 10 shows an estimated depth image. Image G102 in FIG. 10 shows the estimated images in order of blurriness. FIG. 10 shows that the estimation device 2 can estimate a blurred image. Specifically, it shows that the estimation device 2 can obtain a two-dimensional image with a changed blurriness by changing the size indicated by the hole size information.

[0154] FIG. 11 is a second diagram showing an example of the result of estimation by the estimation device 2 in the modified example. Image G103 in FIG. 11 shows an estimated depth image. Image G104 in FIG. 11 shows the estimated images in order of focus position. FIG. 11 shows that the estimation device 2 can estimate a blurred image. Specifically, it shows that the estimation device 2 can obtain a two-dimensional image with a changed focus position by changing the focal length indicated by the focal length information.

[0155] The input unit 22 and the communication unit 23 are an example of an input information acquisition unit. The three-dimensional image estimation model is an example of an estimation model.

[0156] Note that the learning device 1 and the estimation device 2 do not necessarily have to be configured in a single housing. The learning device 1 and the estimation device 2 may each be implemented using a plurality of information processing devices communicably connected via a network. In this case, the functional units of the learning device 1 and the estimation device 2 may be distributed and implemented across the plurality of information processing devices.

[0157] The learning device 1 and the estimation device 2 do not necessarily have to be implemented as different devices, but may be implemented as a single device.

[0158] All or part of the functions of the learning device 1 and the estimation device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0159] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0160] 100...estimation system, 1...learning device, 2...estimation device, 11...control unit, 12...input unit, 13...communication unit, 14...memory unit, 15...output unit, 111...learning unit, 112...input control unit, 113...communication control unit, 114...memory control unit, 115...output control unit, 21...control unit, 22...input unit, 23...communication unit, 24...memory unit, 25...output unit, 211...estimation unit, 212...input control unit, 213...communication control unit, 214...memory control unit, 215...output control unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. an estimation unit that estimates a result of photography by an imaging device using an estimation model that estimates a three-dimensional image of a subject photographed by the imaging device based on hole position information that indicates the position of a hole in the aperture of the imaging device; Equipped with The estimation by the estimation unit uses information indicating that the size of the hole is non-zero. Estimation device.

2. An estimation unit that estimates the results of photography by an imaging device using an estimation model that estimates a three-dimensional image of a subject photographed by the imaging device based on hole position information that indicates the position of a hole in the aperture of the imaging device, the imaging device being equipped with an aperture; Equipped with The estimation by the estimation unit uses information indicating that the size of the hole is not non-zero, The estimation unit further performs estimation using a latent variable that is a quantity that identifies the target. Estimation device.

3. the estimation model is obtained by learning, In the learning, one or more learning data are used, the learning data including input data that is data input to a mathematical model to be learned and output data that is data used for comparison with an output of the mathematical model to be learned, The input data includes hole position information, the output data includes a two-dimensional image of a subject being photographed; the mathematical model of the learning target in the learning is updated so as to reduce a difference between a set of results of estimation by the mathematical model of the learning target and the set of output data; The estimation device according to claim 2 .

4. The learning using the estimator as a generator; a classifier that distinguishes between a set of results of estimation by the generator and a set of output data, The generator and the classifier are learning devices that learn a learning object according to mutually competing optimization conditions. The estimation device according to claim 3 .

5. the estimation unit further performs estimation using information indicating the orientation of the aperture hole. The estimation device according to claim 2 .

6. The estimation model is composed of a neural network. The estimation device according to any one of claims 1 to 5.

7. a learning unit that learns an estimation model that estimates a three-dimensional image of a target photographed by an imaging device, the imaging device including the aperture, based on hole position information that indicates the position of a hole in the aperture of the imaging device; Equipped with In the learning, one or more learning data are used, the learning data including input data that is data input to a mathematical model to be learned and output data that is data used for comparison with an output of the mathematical model to be learned, The input data includes hole position information, the output data includes a two-dimensional image of a subject being photographed; the mathematical model of the learning target in the learning is updated so as to reduce a difference between a set of results of estimation by the mathematical model of the learning target and the set of output data; The learning uses information indicating that the size of the hole is non-zero. Learning device.

8. The learning a generator that estimates the result of photography by the photography device using the estimation model; a classifier for identifying a set of the estimation results and a set of the output data; The generator and the classifier are learning devices that learn a learning object according to mutually competing optimization conditions. The learning device according to claim 7 .

9. an estimation step of estimating a result of photography by an imaging device using an estimation model that estimates a three-dimensional image of an object photographed by the imaging device based on hole position information indicating the position of a hole in the aperture of the imaging device; and The estimation in the estimating step uses information indicating that the size of the hole is non-zero. Estimation method.

10. a learning step of learning an estimation model that estimates a three-dimensional image of an object photographed by an imaging device equipped with a diaphragm based on hole position information indicating the position of a hole in the diaphragm of the imaging device; and In the learning, one or more learning data are used, the learning data including input data that is data input to a mathematical model to be learned and output data that is data used for comparison with an output of the mathematical model to be learned, The input data includes hole position information, the output data includes a two-dimensional image of a subject being photographed; the mathematical model of the learning target in the learning is updated so as to reduce a difference between a set of results of estimation by the mathematical model of the learning target and the set of output data; The learning uses information indicating that the size of the hole is non-zero. How to learn.

11. A program for causing a computer to function as either one of the estimation device according to any one of claims 1 to 5 and the learning device according to claim 7 or 8.

Images