Image synthesis device and image synthesis method

Abstract

An image synthesis device according to one aspect of the present invention is equipped with a training unit, a density noise reduction processing unit, and an image synthesis unit. The training unit trains a neural network using training data based on a first image captured by a first camera sensitive to a first wavelength range and a second image captured by a second camera sensitive to a second wavelength range, thereby generating a trained model. Here, the neural network is configured to output color and density in response to input of viewpoint data including a viewpoint position and a line-of-sight direction. The density noise reduction processing unit reduces noise in density information obtained by entering the viewpoint data into the trained model. The image synthesis unit renders a sampling point on the basis of color information obtained by entering the viewpoint data into the trained model and the noise-reduced density information, thereby generating a new viewpoint image.

Description

Image synthesis device and image synthesis method

[0001] One aspect of the present invention relates to an image synthesis device and an image synthesis method.

[0002] New viewpoint image synthesis technology is a technology that uses images from multiple viewpoints to synthesize an image from a new viewpoint. Neural Radiance Field (NeRF) reconstructs a three-dimensional (3D) image using a deep neural network (DNN). This technology trains a DNN that returns color and density for given data (3D point coordinates, line of sight direction of each pixel). This DNN is called NeRF (see Patent Document 1).

[0003] There are many transparent objects around us. We often see things like glass cups and acrylic screens, and transparent materials are also used in buildings. However, because transparent materials transmit visible light, it is difficult to capture their shapes using technologies that perform 3D reconstruction from visible light images, such as NeRF, or surveying technologies that use near-infrared light, such as LiDAR.

[0004] Dex-NeRF is a technology that extends NeRF to accommodate transparent objects (see Non-Patent Document 2). This technology focuses on the fact that even transparent objects reflect light depending on the viewing angle. To find such angles, it is necessary to devise a lighting environment that increases the number of angles at which the camera can observe reflections from transparent objects.

[0005] A method for 3D reconstructing transparent objects by optimizing the NeRF using images acquired by thermography (hereinafter referred to as "thermoimages") is known (see Patent Document 3). Because transparent objects are often opaque to long-wavelength infrared light, thermoimages can be used to train DNNs. However, this method faces the challenge of accurately estimating the camera pose (image capture position), and it is not possible to acquire visible light information in the first place.

[0006] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", [Retrieved June 13, 2024], Internet<URL: https: / / www.ecva.net / papers / eccv_2020 / papers_ECCV / papers / 123460392.pdf> “Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects”, [online], [Retrieved June 13, 2024], Internet<URL:https: / / https: / / arxiv.org / abs / 2110.14217> Seiichi Tagawa, Tomio Echigo, Minoru Eito, "3D Thermography Generation from NeRF for Vehicle-Type Robots", Information Processing Society of Japan SIG Technical Report, Vol. 2024-CVIM-237 No. 23, March 3, 2024

[0007] When something appears transparent to the human eye, it means that it "transmits visible light." While the wavelength range that can be seen varies from person to person, visible light is generally electromagnetic waves with a wavelength of 0.4 to 0.8 μm. LiDAR is becoming popular as a means of seeing objects using light other than visible light, but the near-infrared light used by LiDAR also has a wavelength close to that of visible light and often passes through transparent objects such as glass. For this reason, it is difficult to reconstruct 3D images of transparent objects based on data acquired by a camera or LiDAR.

[0008] Visible light images are important for estimating the type and properties of objects because they contain a wealth of texture and detailed shape information. For example, they can be used to determine whether a surface is paved based on texture and color, and are also suitable for reading semantic information written on road signs. While it is desirable to use visible light images as much as possible to train NeRF, handling transparent objects is difficult. Existing technologies have issues with estimating lighting environments and viewpoints, and technological innovation is needed.

[0009] The present invention has been made in light of the above circumstances, and aims to provide a technique that makes it possible to easily generate accurate 3D images that include transparent objects.

[0010] An image synthesis device according to one aspect of the present invention includes a learning unit, a density noise reduction processing unit, and an image synthesis unit. The learning unit trains a neural network with training data based on a first image captured by a first camera sensitive to a first wavelength range and a second image captured by a second camera sensitive to a second wavelength range, thereby generating a trained model. Here, the neural network is configured to output color and density in response to input of viewpoint data including a viewpoint position and a line of sight. The density noise reduction processing unit reduces noise in density information obtained by inputting the viewpoint data into the trained model. The image synthesis unit renders sampling points based on color information obtained by inputting the viewpoint data into the trained model and the noise-reduced density information, thereby generating a new viewpoint image.

[0011] According to one aspect of the present invention, it is possible to easily generate accurate 3D images including transparent objects.

[0012] Fig. 1 is a block diagram showing an example of an image synthesis device according to an embodiment. Fig. 2 is a functional block diagram showing an example of an image synthesis device 10. Fig. 3 is a diagram showing an example of a DNN architecture. Fig. 4 is a diagram showing an example of a DNN architecture.

[0013] NeRF is a method for 3D reconstruction using a deep neural network (DNN) based on multi-view images. The NeRF DNN is optimized to represent an object as a 3D distribution of color and density using images of the object as training data.

[0014] NeRF is an example of a neural network configured to output color and density in response to input viewpoint data including a viewpoint position and a line of sight. In an embodiment, a visible light image obtained by a visible light camera and a thermal image obtained by a thermal camera are provided to the NeRF for learning, and a trained model is generated. When arbitrary viewpoint data is provided to this trained model to reconstruct a new viewpoint image, noise contained in the density information is appropriately removed, thereby generating an accurate new viewpoint image including transparent objects.

[0015] Compared to the sensors of ordinary cameras (response wavelength range of approximately 0.4 μm to 0.7 μm) and LiDAR (response wavelength range of 0.9 μm to 1.5 μm), the sensors used in thermography have a response wavelength range of around 5 μm to 10 μm, making them capable of observing long-wavelength infrared light. In this wavelength range, information related to temperature and emissivity can be observed. Everyday transparent materials such as glass and acrylic panels are transparent to wavelengths around the visible light band, but are often opaque to long-wavelength infrared light. Therefore, by using a camera equipped with a sensor that can capture the long-wavelength range, it is possible to capture the shape of even transparent objects as long as there is a difference in temperature and emissivity between them and surrounding objects.

[0016] The visible light camera is an example of a first camera that is sensitive to the visible light range (first wavelength range), and the thermal camera is an example of a second camera that is sensitive to the second wavelength range (infrared range). That is, the transmittance of the second wavelength range through a transparent object is lower than the transmittance of the first wavelength range through a transparent object.

[0017] 1 is a block diagram showing an example of an image synthesis device according to an embodiment. The image synthesis device 10 is a computer that includes a central processing unit (CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13, a storage 14, and a communication interface (I / F) 17. The blocks are connected to each other via a bus 19 so that they can communicate with each other.

[0018] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls each block and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14.

[0019] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of storage devices such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system and various data. The communication I / F 17 is an interface for wired or wireless communication with other devices.

[0020] 2 is a functional block diagram showing an example of the image synthesis device 10. In FIG. 2, the image synthesis device 10 includes, as functional blocks according to the embodiment, an acquisition unit 100, an estimation unit 101, an image synthesis unit 102, a learning unit 103, and a density noise reduction processing unit 104. These functional blocks are realized by the CPU 11 reading out a learning program stored in the ROM 12 or storage 14, and loading and executing the program in the RAM 13.

[0021] The image synthesis device 10 also includes a learning data storage unit 200 , a viewpoint data storage unit 201 , a point information storage unit 202 , a DNN parameter storage unit 203 , an estimated value storage unit 204 , and a generated image storage unit 205 .

[0022] The acquisition unit 100 acquires point information relating to points on the line of sight from a plurality of training images observed in different wavelength ranges. The point information is information with five-dimensional elements, including the three-dimensional coordinates of points on the line of sight of the camera and a vector (e.g., a unit vector in polar coordinates) representing the direction of the line of sight. The point information is recorded for each pixel. If point information is not recorded, the acquisition unit 100 can generate point information for each pixel based on the image coordinates of each pixel and the camera parameters.

[0023] In this embodiment, the training images are visible light images acquired by a visible light camera and thermographic images acquired by a thermal camera. That is, the acquisition unit 100 reads the internal parameters and external parameters of the camera that captured the training images from the training data storage unit 200. The internal parameters include the focal length and lens distortion coefficient of the camera. The internal parameters can be acquired by, for example, acquiring them as data from storage that records camera specification information and camera settings.

[0024] The extrinsic parameters represent the position and orientation of the camera. A set of the camera position and orientation is automatically estimated for each frame of the training image. The initial values ​​of the extrinsic parameters are random numbers, etc. The acquisition unit 100 then calculates the line of sight direction of each pixel from the internal parameters and the extrinsic parameters, and acquires point information related to points on the line of sight.

[0025] The acquisition unit 100 also reads out viewpoint data from the viewpoint data storage unit 201. The viewpoint data is the internal and external parameters of the camera when the captured image was captured. The acquisition unit 100 then calculates the line of sight direction of each pixel from the viewpoint data and acquires point information related to points that exist on the line of sight. The viewpoint data (camera pose) is data that is contained in each frame of the training data (training image), and the acquisition unit 100 obtains point information by sampling the area visible in a given field of view.

[0026] Here, the resolution and image size of the training images are set arbitrarily. Furthermore, the point intervals and ranges related to the point information acquired by the acquisition unit 100 are set to values ​​used in NeRF or general volume rendering. The acquisition unit 100 stores, in the point information storage unit 202, the point information acquired for a plurality of training images observed in different wavelength ranges.

[0027] The estimation unit 101 inputs the point information acquired from the captured image by the acquisition unit 100 to a density estimation multilayer perceptron to estimate a density estimation value. That is, the estimation unit 101 reads the point information acquired from the captured image by the acquisition unit 100 from the point information storage unit 202. Then, the estimation unit 101 inputs the point information acquired from the captured image by the acquisition unit 100 to the density estimation multilayer perceptron to estimate a density estimation value related to volume rendering. Parameters of the density estimation multilayer perceptron are stored in the DNN parameter storage unit 203.

[0028] The density estimation multilayer perceptron is a multilayer perceptron (MLP) that estimates a density estimate. The density estimation multilayer perceptron is trained by the training unit 103. In the embodiment, the color estimation multilayer perceptron outputs a density estimate σ, which is an estimate of the density distribution.

[0029] The color estimation multi-layer perceptron is a DNN independent of the density estimation multi-layer perceptron, and is a multi-layer perceptron that estimates color estimates. In the embodiment, the color estimation multi-layer perceptron outputs a color distribution. The color estimation multi-layer perceptron is trained by the training unit 103.

[0030] The estimation unit 101 inputs point information acquired from a visible light image to a color estimation multilayer perceptron related to the wavelength range of visible light, and estimates the RGB values ​​of the point related to the point information. The estimation unit 101 also inputs point information acquired from a thermographic image obtained by a thermal camera to a color estimation multilayer perceptron related to the wavelength range of infrared light, and estimates the reflected luminance, etc. of the point related to the point information in that wavelength range. The estimation unit 101 stores the density estimation value σ and the color estimation value for each wavelength range in the estimation value storage unit 204.

[0031] The training unit 103 trains the NeRF using training data based on visible light images and thermographic images to generate a trained model. Specifically, the training unit 103 trains a density estimation multilayer perceptron using point information acquired from multiple training images by the acquisition unit 100 and the multiple training images. The training unit 103 also trains a color estimation multilayer perceptron associated with a specific wavelength range using point information acquired from training images observed in the specific wavelength range and the training images observed in the specific wavelength range. During the training process, the training unit 103 calculates the difference between the training data and the generated image as a loss, and back-propagates the loss to the DNN to update the parameters. Here, only color images are used as training data, and depth is not used. The training unit 103 then stores the trained parameters of the density estimation multilayer perceptron and the color estimation multilayer perceptron in the DNN parameter storage unit 203. The DNN estimates viewpoint images identical to the training images and performs image synthesis. Each time the learning is repeated, the DNN parameters are updated, and the estimated / synthesized image gradually approaches the correct answer.

[0032] The density noise reduction processing unit 104 reduces noise in the density estimate value σ obtained by inputting viewpoint data into the trained model. For example, noise in the density distribution σ can be reduced by setting a predetermined threshold in advance and determining the point closest to the viewpoint position among sampling points with a density equal to or greater than this threshold as the depth. For example, according to Dex-NeRF in Non-Patent Document 2, without noise reduction processing, the influence of views with and without reflections may be mixed, resulting in a ghost-like, "blurred" appearance. In contrast, noise can be reduced by determining the depth as the point closest to the front with a density equal to or greater than the threshold.

[0033] The image synthesis unit 102 generates a new viewpoint image by providing arbitrary viewpoint data to the trained model generated by the training unit 103 and performing volume rendering. That is, the image synthesis unit 102 reads the noise-reduced density estimate σ and color estimates based on images from the visible light camera and the thermal camera. Then, the image synthesis unit 102 provides this information to the trained NeRF and generates a new viewpoint image by rendering sampling points. That is, the image synthesis unit 102 provides arbitrary viewpoint data to the trained NeRF and performs volume rendering to generate a new viewpoint image. The generated new viewpoint image is stored in the generated image storage unit 205.

[0034] Figure 3 shows an example of a DNN architecture. In Figure 3, blocks surrounded by double lines represent multilayer perceptrons. Blocks surrounded by solid lines represent input data to or output data from the DNN. Blocks surrounded by dotted lines represent processing on the data.

[0035] In Figure 3, a given camera pose (viewpoint data) is positionally encoded into 3D coordinates and gaze direction within the field of view, and the viewpoint data is input to a first multilayer perceptron (MLP). The output of this first MLP is input to MLPs for the color and density of visible light and the color of thermography. Here, a transparency α (alpha channel) may be newly set for the color of visible light (RGB) and the color of thermography.

[0036] The visible light color / transparency MLP outputs RGB color estimates. The density MLP outputs a density estimate σ. The thermo color / transparency MLP outputs a thermo color estimate. The density estimate σ has noise suppressed by the density noise reduction processing unit 104 (Figure 2) and is passed to the image synthesis unit 102. A visible light (RGB) volume rendering image is then generated based on the RGB color estimates and the density estimate σ using a density noise reduction rendering process. A thermo image volume rendering image is also generated based on the thermo color estimate and the density estimate σ using a density noise reduction rendering process. "Density noise reduction rendering" includes the synthesis of both color images and depth images. Here, the color image / depth image is generated by normal / noise reduction rendering, respectively.

[0037] As described above, in the embodiment, color images (for each different camera) are rendered using a DNN structure similar to that of the normal NeRF, but density (depth) images are determined by searching based on a threshold. This makes it possible to learn and estimate the density (representing the 3D shape) of a transparent object using the DNN structure of NeRF without losing the color representation of the object behind the transparent object.

[0038] As a result, according to the embodiment, it is possible to easily generate accurate 3D images including transparent objects, and it is also possible to perform 3D reconstruction of transparent objects by integrating information from visible light images and thermal images and taking advantage of the advantages of thermal images.

[0039] It should be noted that the present invention is not limited to the above-described embodiment. For example, the ultraviolet wavelength range may be used as the wavelength range for observing the training images. Alternatively, both the ultraviolet wavelength range and the infrared wavelength range may be used. In other words, the training data for training NeRF is not limited to general visible light images or thermographic images, but polarized images, etc., may also be used. Therefore, the camera is not limited to a visible light camera or a thermal camera. In other words, other cameras can be combined as long as clear image data can be obtained with either camera. For example, an ultraviolet camera that captures ultraviolet images may be combined, or images obtained by LiDAR (Light Detection and Ranging) may also be combined.

[0040] Alternatively, a DNN architecture such as that shown in FIG. 4 may be employed. That is, when visible light image rendering, thermographic image rendering, and depth image rendering are performed in a single process, only the noise in the depth image is reduced. Meanwhile, the visible light and thermographic images are combined without noise reduction processing. This allows for learning similar to the original NeRF. Incidentally, if a DNN is trained using an image combined using noise-reduced rendering, it will be unable to learn the color in the area deeper than the surface defined by the cutoff. The architecture of FIG. 4 can avoid such problems.

[0041] Furthermore, the learning process executed by the CPU in the above embodiment by loading software (programs) may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as application-specific integrated circuits (ASICs) that are processors with circuit configurations specifically designed to execute specific processes. The learning process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0042] In the above embodiment, the learning program is pre-stored (installed) in the ROM 12 or the storage 14, but the present invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0043] Furthermore, in the implementation stage, the components of this invention can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0044] 10...Image synthesis device 11...CPU 12...ROM 13...RAM 14...Storage 19...Bus 100...Acquisition unit 101...Estimation unit 102...Image synthesis unit 103...Learning unit 104...Density noise reduction processing unit 200...Learning data storage unit 201...Viewpoint data storage unit 202...Point information storage unit 203...DNN parameter storage unit 204...Estimated value storage unit 205...Generated image storage unit.

Claims

1. An image synthesis device comprising: a learning unit that trains a neural network configured to output color and density in response to input viewpoint data including a viewpoint position and a line of sight using training data based on a first image captured by a first camera sensitive to a first wavelength range and a second image captured by a second camera sensitive to a second wavelength range to generate a trained model; a density noise reduction processing unit that reduces noise in density information obtained by inputting viewpoint data into the trained model; and an image synthesis unit that renders sampling points based on color information obtained by inputting the viewpoint data into the trained model and the noise-reduced density information to generate a new viewpoint image.

2. The image synthesis device according to claim 1, wherein the density noise reduction processing unit reduces noise in the density information by processing the sampling points having a density equal to or greater than a predetermined threshold value, the point closest to the viewpoint position, as the depth.

3. The image synthesis device according to claim 1, wherein the transmittance of the second wavelength range through a transparent object is lower than the transmittance of the first wavelength range through the transparent object.

4. An image synthesis method by a computer having a memory unit that stores a program and a processor that executes instructions of the program, the image synthesis method comprising the steps of: training a neural network configured to output color and density in response to input of viewpoint data including a viewpoint position and a line of sight, using training data based on a first image captured by a first camera sensitive to a first wavelength range and a second image captured by a second camera sensitive to a second wavelength range, to generate a trained model; reducing noise in density information obtained by inputting the viewpoint data into the trained model; and rendering sampling points based on the color information obtained by inputting the viewpoint data into the trained model and the density information with the noise reduced, to generate a new viewpoint image.

Images