Learning device, generation device, learning method, generation method, and program

Abstract

A learning device according to the present invention includes: an acquisition unit that acquires, as training data, a training image that is image-captured from each of a plurality of image-capturing directions, line-of-sight information of each training image, and a corresponding point map representing a corresponding point on each training image corresponding with that on another image; and a learning unit that, using the training data with a designated viewpoint position and line-of-sight direction as input, learns a model for outputting an image of the designated viewpoint position and line-of-sight direction and the corresponding point map by causing a deep neural network to output, for each pixel, the density of an object in three-dimensional coordinates sampled in the line-of-sight direction recorded in the pixel and whether or not the corresponding point is set, and compositing the same in the line-of-sight direction, thereby determining a color of the pixel.

Description

Learning device, generation device, learning method, generation method, and program

[0001] The technology disclosed herein relates to a learning device, a generating device, a learning method, a generating method, and a program.

[0002] In recent years, new viewpoint image synthesis technology using deep neural networks (DNNs) has attracted attention. New viewpoint image synthesis technology synthesizes an image from a new viewpoint based on images from multiple viewpoints. In particular, since the proposal of NeRF (Non-Patent Document 1), the related technology has made remarkable progress.

[0003] NeRF uses a Radiance Field and volume rendering. The Radiance Field represents density and color distribution in space. The density distribution of the Radiance Field represents the three-dimensional shape of an object. By performing volume rendering of the Radiance Field from an arbitrary viewpoint, it is possible to generate an image from that viewpoint. NeRF is characterized by using a Radiance Field (Neural Radiance Field) represented by a DNN and learning the Neural Radiance Field based on images from multiple viewpoints.

[0004] When estimating a 3D shape from images, including NeRF, the relative imaging positions (external camera parameters) between multiple images are required. If the positional relationship is not acquired during imaging, the extrinsic parameters can be estimated using corresponding points that appear in common between the images as landmarks.

[0005] Mildenhall, B., Srinivasan, PP, Tancik, M., Barron, JT, Ramamoorthi, R. and Ng, R.: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Vol. 65, No. 1, New York, NY, USA, Association for Computing Machinery (2021).

[0006] In images with little texture, there are few structures that serve as a reference for the positional relationship between images, making it difficult to estimate the shape and location of three-dimensional objects. For example, errors can occur even when external parameter estimation is performed using Colmap (Internet search <URL: https: / / colmap.github.io / >). Similarly, with NeRF, there are cases where there is insufficient information to determine the density distribution during Radiance Field learning, resulting in an erroneous hazy distribution.

[0007] The disclosed technology has been made in consideration of the above points, and aims to provide a learning device, generation device, learning method, generation method, and program that can accurately generate an image at a specified viewpoint position and line of sight direction, even for an image with little texture.

[0008] A first aspect of the present disclosure is a learning device that includes: an acquisition unit that acquires, as teacher data, training images captured from each of a plurality of imaging directions, gaze information for each training image, and a correspondence point map that indicates corresponding points in each training image that can be associated with other images; and a learning unit that uses the teacher data to learn a model for outputting an image at a specified viewpoint position and gaze direction and the correspondence point map by inputting a specified viewpoint position and gaze direction, and determining the color of the pixel by outputting, using a deep neural network, the density of an object in three-dimensional coordinates sampled in the gaze direction and recorded in that pixel, and whether or not it is a corresponding point, and combining the output in the gaze direction.

[0009] A second aspect of the present disclosure is a generation device that includes an acquisition unit that acquires a specified viewpoint position and gaze direction, and an estimation unit that generates an image of the specified viewpoint position and gaze direction using a trained model for outputting an image of the specified viewpoint position and gaze direction by using a deep neural network to output, for each pixel, the density of an object in three-dimensional coordinates sampled in the gaze direction recorded in that pixel, and whether or not it is a corresponding point, and combining the output in the gaze direction to determine the color of the pixel, based on the trained model.

[0010] A third aspect of the present disclosure is a learning method, in which an acquisition unit acquires, as teacher data, training images captured from each of a plurality of imaging directions, gaze information for each training image, and a correspondence point map representing corresponding points in each training image that can be associated with other images, and a learning unit uses the teacher data to learn a model for outputting an image at a specified viewpoint position and gaze direction and the correspondence point map by inputting a specified viewpoint position and gaze direction, and outputting, using a deep neural network, the density of an object in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not it is a corresponding point, and combining the output data in the gaze direction to determine the color of the pixel.

[0011] A fourth aspect of the present disclosure is a generation method, in which an acquisition unit acquires a designation of a viewpoint position and a gaze direction, and an estimation unit uses the designated viewpoint position and gaze direction as input, and outputs, using a deep neural network, the density of an object in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not it is a corresponding point, and synthesizes the results in the gaze direction to determine the color of the pixel, thereby generating an image at the designated viewpoint position and gaze direction using a trained model for outputting an image at the designated viewpoint position and gaze direction.

[0012] A fifth aspect of the present disclosure is a learning program for causing a computer to function as the learning device of the first aspect or the generation device of the second aspect.

[0013] According to the disclosed technology, even if the image has little texture, it is possible to generate an image at a specified viewpoint position and line of sight direction with high accuracy.

[0014] FIG. 1 is a schematic block diagram of an example of a computer that functions as a learning device and a generation device of this embodiment. FIG. 2 is a block diagram showing the functional configuration of the learning device of this embodiment. FIG. 3 is a diagram for explaining an example of a model learning method by the learning device of the first embodiment. FIG. 4 is a block diagram showing the functional configuration of the generation device of this embodiment. FIG. 5 is a diagram for explaining an example of an image generation method by the generation device of the first embodiment. FIG. 6 is a flowchart showing the flow of learning processing of this embodiment. FIG. 7 is a flowchart showing the flow of generation of this embodiment. FIG. 8 is a diagram for explaining an example of a model learning method by the learning device of the first embodiment.

[0015] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0016] <Outline of Embodiment of Disclosed Technology> In the disclosed technology, during Radiance Field learning in NeRF, corresponding points that are easily associated with other images are temporarily added, thereby enabling normal learning even for images with little texture.

[0017] The deep neural network structure of NeRF described in Non-Patent Document 1 inputs arbitrary viewpoint position coordinates and gaze direction, and outputs the density and color information of each pixel corresponding to them. A typical training image has three channels (R, G, B) or one channel (luminance), and the number of color channels output from the deep neural network is the same as the number of channels in the training image.

[0018] In this embodiment, the Radiance Field represented by the NeRF DNN is learned by adding a corresponding point map created for each training image, thereby supplementing information about the shape and arrangement of objects and improving the accuracy of density distribution estimation.

[0019] In this embodiment, corresponding points are areas that are commonly reflected in multiple images (hereinafter referred to as corresponding points) that are determined by the human eye or by using an algorithm appropriate for the image, and a corresponding point map is a marking of corresponding points on an image.

[0020] The marking format is to change the values ​​of only the corresponding points on an image that is painted in a uniform color with the same field of view and resolution as the training image. Rather than marking a single pixel in each corresponding point, it is also possible to mark parts within a certain range along the shape of an object's corners or other shapes where a correspondence can be determined. Also, if matching between corresponding points is difficult, it is also possible to mark each corresponding point with a different value. It is also possible to use multiple colors by using multiple channels in the corresponding point map.

[0021] In this manner, in this embodiment, in addition to a grayscale one-channel or RGB three-channel image, a correspondence point map configured with one or more layers is input as an additional channel of the training image of the NeRF.

[0022] First Embodiment Configuration of a Learning Device According to a First Embodiment FIG. 1 is a block diagram showing the hardware configuration of a learning device 10 according to this embodiment.

[0023] 1, the learning device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0024] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, a learning program is stored in the ROM 12 or the storage 14. The learning program may be a single program, or a group of programs consisting of multiple programs or modules.

[0025] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0026] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various inputs including a combination of training images captured from each of a plurality of imaging directions, line-of-sight information for each training image, and a correspondence point map representing corresponding points in each training image that can be associated with other images. For example, the input unit 15 receives input of a combination of training images captured by an imaging device from each of a plurality of imaging directions, line-of-sight information for each training image that includes internal parameters and external parameters of the imaging device, and a manually prepared correspondence point map representing corresponding points in each training image that can be easily associated with other images.

[0027] The display unit 16 is, for example, a liquid crystal display, and displays various information including the processing results. The display unit 16 may be a touch panel type and function as the input unit 15.

[0028] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0029] Next, a description will be given of the functional configuration of the learning device 10. Fig. 2 is a block diagram showing an example of the functional configuration of the learning device 10.

[0030] As shown in FIG. 2, the learning device 10 functionally comprises an acquisition unit 20, a learning unit 22, and a model storage unit 24.

[0031] The acquisition unit 20 acquires, as teacher data, a plurality of training images captured from a plurality of imaging directions, line-of-sight information for each training image, and a correspondence point map showing corresponding points in each training image that can be associated with other images. Here, the plurality of training images are images of a common imaging range captured from a plurality of imaging directions.

[0032] The learning unit 22 uses training data to learn a model for outputting an image of the specified viewpoint position and gaze direction as input, and the corresponding point map by outputting the density of the object in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not it is a corresponding point using a deep neural network, and synthesizing the output in the gaze direction to determine the color of the pixel.

[0033] Specifically, the learning unit 22 learns a DNN that expresses a Radiance Field as a model so as to optimize a loss function that includes the difference between the image output by the model and the training image of the teacher data, and the difference between the corresponding point map output by the model and the corresponding point map of the teacher data.

[0034] More specifically, as shown in FIG. 3 , a DNN that represents a Radiance Field is trained. In FIG. 3 , white blocks represent data, diagonally shaded blocks represent parts of the DNN, and shaded blocks represent processing other than the DNN. First, internal parameters (e.g., optical axis, focal length, angle of view, resolution) and external parameters (e.g., three-dimensional viewpoint position, gaze direction) included in the gaze information of the training image are input, and the gaze direction recorded by the pixel from the viewpoint position is determined for each pixel coordinate. Then, three-dimensional coordinates are sampled on a line in the gaze direction, and position encoding is performed on the sampled coordinates. The results of the position encoding are input to an MLP (Multi-Layer Perceptron), and one channel of the MLP output is separated to generate data representing the density of objects in three-dimensional coordinates sampled for each pixel on a line in the gaze direction from the viewpoint position to the pixel. As an example of an implementation of this DNN, the MLP may have eight layers, each with 256 channels. Furthermore, the remaining channels of the MLP output are input to an RGB estimation MLP and a correspondence point map estimation MLP. As an example, this RGB estimation MLP may have two layers, each with 128 and 3 channels. As an example, this correspondence point map estimation MLP may have two layers, each with 128 and 1 channel. The output of the RGB estimation MLP is data representing color information for each pixel. The output of the correspondence point map estimation MLP is data representing whether each pixel is a corresponding point. The output of the RGB estimation MLP and data representing the density of objects for each pixel are input to RGB channel rendering to generate an image. The output of the correspondence point map estimation MLP and data representing the density of objects for each pixel are input to correspondence point map rendering to generate a correspondence point map.

[0035] A loss is calculated between the image generated by RGB channel rendering and the training image, and the parameters of the MLP and the RGB estimation MLP are updated to optimize the calculated loss. Also, a loss is calculated between the correspondence point map generated by correspondence point map rendering and the correspondence point map corresponding to the training image, and the parameters of the MLP and the correspondence point map estimation MLP are updated to optimize the calculated loss.

[0036] For example, the loss for learning the model is calculated by applying the same calculation method to each term of the loss function for each channel (RGB, etc.) of the training image in accordance with the base technology to the channels of the additional corresponding point map. For example, ΔR ２ +ΔG ２ +ΔB ２ +ΔF ２ where F is the channel of the corresponding point map.

[0037] The learning unit 22 stores the learned model in the model storage unit 24 .

[0038] <Configuration of a Generating Apparatus According to a First Embodiment> FIG. 1 is a block diagram showing the hardware configuration of a generating apparatus 50 according to a first embodiment.

[0039] 1, like the learning device 10, the generation device 50 includes a CPU 11, a ROM 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other so as to be able to communicate with each other via a bus 19. A generation program is stored in the ROM 12 or the storage 14.

[0040] The input unit 15 is used to input various information including the designation of the viewpoint position and line of sight direction of the image to be generated.

[0041] Next, a description will be given of the functional configuration of the generating device 50. FIG.

[0042] As shown in FIG. 4, the generating device 50 functionally includes a model storage unit 52, an acquisition unit 54, and an estimation unit 56.

[0043] The model storage unit 52 stores a trained model similar to that of the model storage unit 24 of the learning device 10.

[0044] The acquisition unit 54 acquires the designation of the viewpoint position and the line of sight direction.

[0045] The estimation unit 56 receives a specified viewpoint position and gaze direction as input, and generates an image of the specified viewpoint position and gaze direction using a trained model for outputting an image of the specified viewpoint position and gaze direction by using a deep neural network to output the density of objects in three-dimensional coordinates sampled for each pixel in the gaze direction recorded in that pixel, and whether or not they are corresponding points. Here, the generated image is an image of an imaging range common to the training image, captured at the specified viewpoint position and gaze direction.

[0046] Specifically, as shown in Fig. 5, an image is generated using a DNN that represents a Radiance Field. In Fig. 5, white blocks represent data, diagonally shaded blocks represent parts of the DNN, and shaded blocks represent processes other than the DNN. Dotted blocks represent parts that are not used when generating the image.

[0047] First, using specified external parameters (e.g., the three-dimensional viewpoint position, the line of sight direction) and internal parameters (e.g., the optical axis, focal length, angle of view, and resolution) as input, the line of sight direction from the viewpoint position to the pixel is calculated for each pixel coordinate. Then, three-dimensional coordinates are sampled on a line in the line of sight direction, and position encoding is performed on the sampled coordinates. The results of the position encoding are input to an MLP, and one channel of the MLP output is separated to generate data representing the density of objects at the three-dimensional coordinates sampled on a line in the line of sight direction from the viewpoint position to the pixel in question for each pixel. The remaining channels of the MLP output are input to an RGB estimation MLP. The output of the RGB estimation MLP is used as data representing color information for each pixel. The output of the RGB estimation MLP and the data representing the density of objects for each pixel are input to RGB channel rendering to generate an image.

[0048] In this way, when the DNN that constitutes the Radiance Field is trained, the channels of the correspondence point map are output and training based on the training map is performed, but when generating an image from an arbitrary viewpoint, the channels of the correspondence point map are not output. This makes it possible to use supplementary information when training the Radiance Field, while generating an image by eliminating the additional correspondence point map that is unnecessary during image generation.

[0049] <Operation of the Learning Device According to the First Embodiment> Next, the operation of the learning device 10 according to this embodiment will be described.

[0050] 6 is a flowchart showing the flow of the learning process by the learning device 10. The learning process is performed by the CPU 11 reading a learning program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. It is also assumed that the learning device 10 has input thereto a combination of training images captured from each of a plurality of imaging directions, line-of-sight information for each training image, and a correspondence point map showing corresponding points for each training image that can be associated with other images. The learning process is an example of a learning method.

[0051] In step S100, the CPU 11, as the acquisition unit 20, acquires as teacher data a combination of training images captured from each of multiple imaging directions received by the input unit 15, gaze information for each training image, and a correspondence point map representing corresponding points for each training image that can be associated with other images.

[0052] In step S102, the CPU 11, as the learning unit 22, uses the teacher data to input the specified viewpoint position and line of sight, and outputs the density of the object in three-dimensional coordinates sampled in the line of sight recorded in each pixel, and whether or not it is a corresponding point, using a deep neural network, and synthesizes these in the line of sight to determine the color of the pixel, thereby learning a model for outputting an image of the specified viewpoint position and line of sight direction and the corresponding point map, and stores the learned model in the model storage unit 24, and ends the learning process.

[0053] <Operation of Generating Device According to First Embodiment> Next, the operation of the generating device 50 according to this embodiment will be described.

[0054] 7 is a flowchart showing the flow of generation by the generation device 50. The generation process is performed by the CPU 11 reading out the generation program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. In addition, a trained model trained by the learning device 10 is stored in the model storage unit 52 of the generation device 50. It is also assumed that the viewpoint position and line of sight direction of the image to be generated have been input to the generation device 50. The generation process is an example of a generation method.

[0055] In step S110, the CPU 11 functions as the acquisition unit 54 to acquire the designation of the viewpoint position and the line of sight direction.

[0056] In step S112, the CPU 11, as the estimation unit 56, takes the specified viewpoint position and gaze direction as input, and generates an image of the specified viewpoint position and gaze direction using a trained model for outputting an image of the specified viewpoint position and gaze direction by using a deep neural network to output the density of objects in three-dimensional coordinates sampled for each pixel in the gaze direction recorded in that pixel, and whether or not they are corresponding points, and then terminates the generation process.

[0057] As described above, the learning device according to the first embodiment acquires, as training data, training images captured from multiple imaging directions, gaze information for each training image, and a correspondence point map representing corresponding points in each training image that can be associated with other images. Using a specified viewpoint and gaze direction as input, the learning device outputs, via a deep neural network, the density of objects in 3D coordinates sampled along the gaze direction and recorded for each pixel, as well as whether or not they are corresponding points, and synthesizes the results along the gaze direction to determine the color of the pixel, thereby learning a model for outputting an image for the specified viewpoint and gaze direction and the correspondence point map. This makes it possible to accurately generate an image for the specified viewpoint and gaze direction, even for images with little texture.

[0058] Furthermore, the generation device according to the first embodiment generates an image at a specified viewpoint position and gaze direction using a trained model trained by the above-described learning device, thereby enabling accurate generation of an image at a specified viewpoint position and gaze direction even for an image with little texture.

[0059] Second Embodiment Next, a second embodiment will be described. The learning device and the generating device of the second embodiment have the same configuration as those of the first embodiment, so the same reference numerals are used and the description thereof will be omitted.

[0060] The first embodiment has been described as being based on the technology of Non-Patent Document 1, but the second embodiment is based on the BARF (Bundle-adjusting neural radiation fields) technology of Non-Patent Document 2.

[0061] BARF is an applied technology that enhances the convenience of NeRF. NeRF in Non-Patent Document 1 requires extrinsic parameters of training images when learning a Radiance Field. BARF optimizes the extrinsic parameters of training images in parallel with learning a Radiance Field, so it does not require extrinsic parameters of training images when learning a Radiance Field.

[0062] [Non-patent document 2]: Lin, CH, Ma, WC, Torralba, A., & Lucey, S. (2021). Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 5741-5751).

[0063] The input unit 15 of the learning device 10 of the second embodiment receives a combination of training images captured by an imaging device from each of multiple imaging directions, gaze information for each training image including internal parameters of the imaging device, and a manually prepared correspondence point map that represents corresponding points for each training image that can be easily matched with other images.

[0064] The learning unit 22 learns a model based on the training data and estimates external parameters of the imaging device for each training image.

[0065] Specifically, the system inputs internal parameters (e.g., optical axis, focal length, angle of view, resolution) included in the line-of-sight information of the training image and initial or estimated values ​​of external parameters (e.g., three-dimensional viewpoint position, line-of-sight direction) to first determine the line-of-sight direction from the viewpoint position to the pixel for each pixel coordinate. Then, it samples three-dimensional coordinates on a line along the line of sight direction, and performs position encoding on the sampled coordinates. The results of the position encoding are input to an MLP, and one channel of the MLP output is separated to represent, for each pixel, the density of objects at the three-dimensional coordinates sampled on a line along the line of sight direction from the viewpoint position to the pixel. The remaining channels of the MLP output are input to an RGB estimation MLP and a corresponding point map estimation MLP. The output of the RGB estimation MLP is used as data representing color information for each pixel. The output of the corresponding point map estimation MLP is used as data representing whether each pixel is a corresponding point. The output of the RGB estimation MLP and data representing the density of the object for each pixel are input to RGB channel rendering to generate an image. The output of the correspondence point map estimation MLP and data representing the density of the object for each pixel are input to correspondence point map channel rendering to generate a correspondence point map.

[0066] The loss between the image generated by RGB channel rendering and the training image is calculated, and the parameters of the MLP and RGB estimation MLP are updated. The loss between the correspondence point map generated by correspondence point map rendering and the correspondence point map corresponding to the training image is calculated, and the parameters of the MLP and correspondence point map estimation MLP are updated. In parallel with the model training, the extrinsic parameters are estimated.

[0067] Other configurations and operations of the learning device 10 are the same as those in the first embodiment, and therefore will not be described. Also, the configurations and operations of the generation device 50 are the same as those in the first embodiment, and therefore will not be described.

[0068] Third Embodiment Next, a third embodiment will be described. The learning device and the generating device of the third embodiment have the same configuration as those of the first embodiment, so the same reference numerals are used and the description thereof will be omitted.

[0069] The third embodiment differs from the first embodiment in that the training images are treated as four-channel image data including a corresponding point map.

[0070] The learning unit 22 of the learning device 10 of the third embodiment learns a DNN that expresses a Radiance Field as a model so as to optimize a loss function that includes the difference between an image output by the model and a training image of the teacher data, and the difference between a corresponding point map output by the model and a corresponding point map of the teacher data.

[0071] Specifically, a DNN that expresses a Radiance Field is trained as shown in Fig. 8. In Fig. 8, white blocks represent data, diagonally shaded blocks represent parts of the DNN, and shaded blocks represent processes other than the DNN.

[0072] Using internal parameters (e.g., optical axis, focal length, angle of view, resolution) and external parameters (e.g., three-dimensional viewpoint position, viewpoint direction) included in the viewpoint information of the training image as input, the viewpoint direction from the viewpoint position to the pixel is first calculated for each pixel coordinate. Then, three-dimensional coordinates are sampled on a line in the viewpoint direction, and position encoding is performed on the sampled coordinates. The results of the position encoding are input to an MLP (Multi-Layer Perceptron), and one channel of the MLP output is separated to obtain data representing the density of objects at three-dimensional coordinates sampled on a line in the viewpoint direction for each pixel. This MLP has eight layers, each with 256 channels.

[0073] The remaining channels of the MLP output are input to the MLP. The output of the MLP is a four-channel image representing the color information for each pixel and whether it is a corresponding point. This MLP has two layers, with 128 and 4 channels for each layer. The output of the MLP and data representing the density of objects for each pixel are input to four-channel rendering to generate a four-channel image. The loss between the four-channel image generated by the four-channel rendering and the four-channel images of the training image and the corresponding point map is calculated, and the parameters of each MLP are updated.

[0074] In addition, the estimation unit 56 of the generation device 50 takes a specified viewpoint position and gaze direction as input, and uses a DNN to output color and density distribution in the gaze direction (which can be calculated by calculating only for sampled points), and then synthesizes the output color and density for each pixel to render an image, thereby generating an image for the specified viewpoint position and gaze direction using a trained model for outputting an image for the specified viewpoint position and gaze direction.

[0075] Specifically, using specified external parameters (e.g., the three-dimensional viewpoint position, the line of sight direction) and internal parameters (e.g., the optical axis, focal length, angle of view, and resolution) as inputs, the line of sight direction from the viewpoint position to the pixel is first calculated for each pixel coordinate. Then, three-dimensional coordinates are sampled on a line in the line of sight direction, and position encoding is performed on the sampled coordinates. The results of the position encoding are input to an MLP, and one channel of the MLP output is separated to represent data representing the density of objects at the three-dimensional coordinates sampled on a line in the line of sight direction from the viewpoint position to the pixel for each pixel. The remaining channels of the MLP output are also input to the MLP. The output of the MLP is treated as a four-channel image. The output of the MLP and the data representing the object density for each pixel are input to four-channel rendering to generate a four-channel image. Then, a process is performed to delete the corresponding point map channel from the four-channel image.

[0076] The other configurations and operations of the learning device 10 and the generation device 50 are the same as those in the first embodiment, and therefore will not be described.

[0077] Furthermore, the present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

[0078] For example, although the above description has been given taking the case where the generation device and the learning device are configured separately, the present invention is not limited to this and the generation device and the learning device may be configured as a single device.

[0079] In addition, various processes executed by the CPU after reading the software (program) in the above embodiment may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and dedicated electrical circuits, which are processors having a circuit configuration designed specifically for executing specific processes, such as ASICs (Application Specific Integrated Circuits). Furthermore, the learning process and generation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0080] In addition, in each of the above embodiments, the learning program and the generation program are described as being pre-stored (installed) in the storage 14, but this is not limiting. The programs may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network.

[0081] The following additional notes are provided regarding the above-described embodiments.

[0082] (Supplementary Item 1) A learning device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to acquire, as teacher data, training images captured from each of a plurality of imaging directions, gaze information for each training image, and a correspondence point map representing corresponding points in each training image that can be associated with other images, and to use the teacher data to learn a model for outputting an image of a specified viewpoint position and gaze direction and the correspondence point map by inputting a specified viewpoint position and gaze direction, and outputting, via a deep neural network, the density of an object in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not it is a corresponding point, and combining the output data in the gaze direction to determine the color of the pixel.

[0083] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute a learning process, wherein the learning process is configured to acquire, as teacher data, training images captured from each of a plurality of imaging directions, gaze information for each training image, and a correspondence point map indicating corresponding points in each training image that can be associated with other images, and to use the teacher data to learn a model for outputting an image at the specified viewpoint position and gaze direction and the correspondence point map by inputting a specified viewpoint position and gaze direction, and outputting, using a deep neural network, the density of an object in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not it is a corresponding point, and combining the output data in the gaze direction to determine the color of the pixel.

[0084] (Supplementary Item 3) A generating device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to acquire a designation of a viewpoint position and a line of sight direction, and to generate an image of the designated viewpoint position and line of sight direction using a trained model that uses a deep neural network to output, for each pixel, the density of an object in three-dimensional coordinates sampled in the line of sight direction recorded in that pixel, and whether or not the points are corresponding points, using the designated viewpoint position and line of sight direction as input.

[0085] (Supplementary Item 4) A non-transitory storage medium storing a program executable by a computer to execute a generation process, wherein the generation process is configured to: acquire a designation of a viewpoint position and a line of sight direction; and, using the designated viewpoint position and line of sight direction as input, generate an image of the designated viewpoint position and line of sight direction using a trained model for outputting, by a deep neural network, the density of objects in three-dimensional coordinates sampled for each pixel along the line of sight direction recorded in that pixel, and whether or not they are corresponding points.

[0086] REFERENCE SIGNS LIST 10 Learning device 11 CPU 13 RAM 14 Storage 15 Input unit 16 Display unit 20 Acquisition unit 22 Learning unit 24 Model storage unit 50 Generation device 52 Model storage unit 54 Acquisition unit 56 Estimation unit

Claims

1. A learning device including: an acquisition unit that acquires training images captured from each of a plurality of imaging directions, gaze information for each training image, and a corresponding point map that indicates corresponding points in each training image that can be associated with other images as teacher data; and a learning unit that uses the teacher data to learn a model for outputting an image in a specified viewpoint position and gaze direction and the corresponding point map by inputting a specified viewpoint position and gaze direction, and determining the color of the pixel by outputting, using a deep neural network, the density of an object in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not it is a corresponding point, and synthesizing the results in the gaze direction.

2. A learning device as described in claim 1, wherein the learning unit learns the model so as to optimize a loss function including the difference between the image output by the model and the training image of the teacher data, and the difference between the corresponding point map output by the model and the corresponding point map of the teacher data.

3. A generating device including: an acquisition unit that acquires a designated viewpoint position and gaze direction; and an estimation unit that receives the designated viewpoint position and gaze direction as input, and generates an image in the designated viewpoint position and gaze direction using a trained model for outputting an image in the designated viewpoint position and gaze direction by determining the color of the pixel by outputting the density of objects in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not they are corresponding points using a deep neural network and synthesizing them in the gaze direction.

4. A learning method in which an acquisition unit acquires training images captured from each of a plurality of imaging directions, gaze information for each training image, and a corresponding point map showing corresponding points of each training image that can be associated with other images as teacher data, and a learning unit uses the teacher data to learn a model for outputting an image of a specified viewpoint position and gaze direction and the corresponding point map by inputting a specified viewpoint position and gaze direction, outputting the density of an object in three-dimensional coordinates sampled in the gaze direction recorded in each pixel for each pixel, and whether or not it is a corresponding point using a deep neural network, and synthesizing the data in the gaze direction to determine the color of the pixel.

5. A method for generating an image in the specified viewpoint position and gaze direction using a trained model for outputting an image in the specified viewpoint position and gaze direction, in which an acquisition unit acquires a designated viewpoint position and gaze direction, and an estimation unit receives the designated viewpoint position and gaze direction as input, and outputs, using a deep neural network, the density of objects in three-dimensional coordinates sampled in the gaze direction recorded in each pixel and whether or not they are corresponding points, and synthesizes the results in the gaze direction to determine the color of the pixel, thereby generating an image in the specified viewpoint position and gaze direction.

6. A program for causing a computer to function as the learning device according to claim 1 or 2, or the generating device according to claim 3.

Images