Image synthesis device and image synthesis method

Abstract

An image synthesis device according to one embodiment of the present invention is provided with a training unit and an image synthesis unit. The training unit trains a neural network that maps viewpoint information, which includes a viewpoint position and a line-of-sight direction, onto a parameter of a luminance field, the neural network being trained using practice data that is based on a first image captured by a first sensor sensitive to a first wavelength region and a second image captured by a second sensor sensitive to a second wavelength region. The image synthesis unit generates a new viewpoint image by performing volume rendering by giving arbitrary viewpoint information to a luminance field generated by training the neural network.

Description

Image synthesis device and image synthesis method

[0001] One aspect of the present invention relates to an image synthesis device and an image synthesis method.

[0002] New viewpoint image synthesis technology is a technology that uses images from multiple viewpoints to synthesize an image from a new viewpoint. In recent years, new viewpoint image synthesis technology using deep learning (DNN: Deep Neural Network) has been attracting attention. One such technology is Neural Radiance Field (NeRF) (see Non-Patent Document 1).

[0003] NeRF is a technology that trains a DNN to represent 3D scenes from photographs, enabling the synthesis of photo-realistic images from any viewpoint. This technology trains a DNN that returns color and density for given (3D point coordinates, line of sight direction of each pixel). This DNN is called NeRF.

[0004] While NeRF has primarily been used for visible light images, it is also beneficial to apply it to special images such as polarized light images and invisible light images. For example, objects that appear to have the same color or texture or appear transparent in visible light images can sometimes be distinguished using invisible light images or polarized light images. By aligning, superimposing, and combining various types of images, a wide range of properties of the target object can be captured. For example, combining visible light images and thermographic images obtained with different image sensors has been proposed (see Non-Patent Document 3).

[0005] BARF (Bundle-Adjusting Neural Radiance Fields), which applies NeRF, is known (see Non-Patent Document 2). BARF adds to NeRF a function for estimating the external camera parameters of the training images (point information: viewpoint information, including the three-dimensional position of the viewpoint and the line of sight direction). While NeRF requires the external camera parameters of the training images when learning the radiance field, BARF optimizes the external parameters of the training images in parallel with learning the radiance field. Therefore, BARF does not require the external parameters of the training images when learning the radiance field.

[0006] Non-Patent Document 3 proposes extending the BARF DNN by adding a color-output multilayer perceptron (MLP). In Non-Patent Document 3, images from multiple sensors are synthesized by a front-stage MLP, and then images from each sensor are synthesized by two color-output MLPs. With this configuration, the color-output MLP is trained with images from individual sensors, while the main front-stage MLP is trained with images from both sensors. Non-Patent Document 3 trains an extended BARF based on images from two sensors that are unrelated in terms of their positional relationship. By applying the camera pose estimation function of BARF, it is possible to train the MLP to output colors from two sensors in a registered state from the same viewpoint.

[0007] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", [Retrieved February 6, 2024], Internet<URL: https: / / www.ecva.net / papers / eccv_2020 / papers_ECCV / papers / 123460392.pdf> Lin, C.H., Ma, W.C., Torralba, A., & Lucey, S. (2021). BARF: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 5741-5751). [Retrieved February 6, 2024], Internet <URL: https: / / openaccess.thecvf.com / content / ICCV2021 / papers / Lin_BARF_Bundle-Adjusting_Neural_Radiance_Fields_ICCV_2021_paper.pdf> Kana Kurata, Mayuko Watanabe, Ryuichi Tanida, Jun Shimamura "Application of NeRF for Integrated Processing of Thermography and RGB Images", [Retrieved February 6, 2024], Internet<URL: https: / / cvim.ipsj.or.jp / MIRU2023 / program / >

[0008] In Non-Patent Document 3, images from two sensors were alternately learned, which may make the NeRF unsuitable for learning depending on the characteristics of the sensors. For example, a visible light camera can clearly capture the shape of an object, so the NeRF outputs a clear shape after learning. On the other hand, the object shapes captured by some sensors, such as thermography, are somewhat vague, so even when combined with a visible light camera, the shape of the output image is also somewhat blurred. Furthermore, it was inefficient because it was necessary to re-learn all images from both sensors every time the sensor combination was changed.

[0009] The present invention has been made in light of the above circumstances, and aims to provide a technique that enables efficient generation of new viewpoint images using multiple sensors.

[0010] An image synthesis device according to one aspect of the present invention includes a learning unit and an image synthesis unit. The learning unit trains a neural network that maps viewpoint information, including a viewpoint position and a line of sight direction, to parameters of a luminance field using training data based on a first image captured by a first sensor sensitive to a first wavelength range and a second image captured by a second sensor sensitive to a second wavelength range. The image synthesis unit applies arbitrary viewpoint information to the luminance field generated by training the neural network and performs volume rendering to generate a new viewpoint image.

[0011] According to one aspect of the present invention, it is possible to efficiently generate a new viewpoint image using multiple sensors.

[0012] FIG. 1 is a block diagram showing an example of an image synthesis device according to an embodiment. FIG. 2 is a functional block diagram showing an example of an image synthesis device 10. FIG. 3 is a diagram showing an example of a DNN architecture. FIG. 4 is a diagram showing an example of a DNN structure and a processing flow according to an embodiment. FIG. 5 is a diagram showing another example of a DNN structure and a processing flow according to an embodiment.

[0013] In this embodiment, we describe an improved technique for training a DNN, which is an extension of the BARF DNN structure, using image data acquired by a different sensor. BARF has a neural network that maps viewpoint information, including viewpoint position and gaze direction, to luminance field parameters. In this embodiment, an MLP is added at an appropriate position in this neural network, and this extended DNN is trained by providing visible light images acquired by an image sensor and thermographic images acquired by a thermal sensor.

[0014] The training data is not limited to visible light images or thermographic images. Furthermore, the sensors are not limited to image sensors or thermo sensors. In other words, other sensors can be combined as long as they can obtain clear image data. For example, a sensor that captures ultraviolet images can be combined, or images captured by LiDAR (Light Detection and Ranging). The DNN in Non-Patent Document 3 can be trained using a first image captured by a first sensor sensitive to a first wavelength range and a second image captured by a second sensor sensitive to a second wavelength range. The image sensor is an example of a sensor sensitive to the visible light range (first wavelength range), and the thermo sensor is an example of a second sensor sensitive to the second wavelength range (infrared range). Here, the first sensor is a sensor that can clearly capture the shape of an object. Examples include visible light and near-infrared sensors. The second sensor does not have this restriction, and a thermo sensor or the like can also be assigned. By setting the first wavelength range to a range with shorter wavelengths than the second wavelength range, the first sensor can obtain image data that is clearer than that obtained by the second sensor.

[0015] 1 is a block diagram showing an example of an image synthesis device according to an embodiment. The image synthesis device 10 is a computer that includes a central processing unit (CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13, a storage 14, and a communication interface (I / F) 17. The blocks are connected to each other via a bus 19 so that they can communicate with each other.

[0016] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls each block and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14.

[0017] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of storage devices such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system and various data. The communication I / F 17 is an interface for wired or wireless communication with other devices.

[0018] 2 is a functional block diagram showing an example of the image synthesis device 10. In FIG. 2, the image synthesis device 10 includes, as functional blocks according to the embodiment, an acquisition unit 100, an estimation unit 101, an image synthesis unit 102, and a learning unit 103. These functional blocks are realized by the CPU 11 reading out a learning program stored in the ROM 12 or storage 14, and loading and executing the program in the RAM 13.

[0019] The image synthesis device 10 also includes a learning data storage unit 200 , a viewpoint data storage unit 201 , a point information storage unit 202 , a DNN parameter storage unit 203 , an estimated value storage unit 204 , and a generated image storage unit 205 .

[0020] The acquisition unit 100 acquires point information relating to points on the line of sight from a plurality of training images observed in different wavelength ranges. The point information is information with five-dimensional elements, including the three-dimensional coordinates of points on the line of sight of the sensor and a vector (e.g., a unit vector in polar coordinates) representing the direction of the line of sight. The point information is recorded for each pixel. If point information is not recorded, the acquisition unit 100 can generate point information for each pixel based on the image coordinates of each pixel and the camera parameters.

[0021] In this embodiment, the training images are a visible light image obtained by an image sensor and a thermographic image obtained by a thermosensor. The image sensor acquires images in the visible light wavelength range (e.g., about 0.4 μm to about 0.78 μm). The thermosensor acquires images in the near-infrared to thermal infrared wavelength range (e.g., about 0.78 μm to about 14 μm).

[0022] Specifically, the acquisition unit 100 reads out the internal parameters and external parameters of the sensor that captured the training image from the training data storage unit 200. The internal parameters include the focal length and lens distortion coefficient of the sensor, etc. The internal parameters can be acquired by a method such as acquiring the sensor specification information or acquiring the parameters as data from the sensor.

[0023] The extrinsic parameters represent the position and orientation of the sensor. The extrinsic parameters are automatically estimated as a set of sensor positions and orientations for each frame of the training image. The initial values ​​of the extrinsic parameters are random numbers, etc. The acquisition unit 100 then calculates the line of sight direction of each pixel from the internal parameters and the extrinsic parameters, and acquires point information related to points on the line of sight.

[0024] The acquisition unit 100 also reads out viewpoint information from the viewpoint data storage unit 201. The viewpoint information is the internal and external parameters of the sensor when the captured image was captured. The acquisition unit 100 then calculates the line of sight direction of each pixel from the viewpoint information and acquires point information related to points that exist on the line of sight.

[0025] Here, the resolution and image size of the training images are set arbitrarily. Furthermore, the intervals and ranges of points related to the point information acquired by the acquisition unit 100 are set to values ​​used in NeRF, BARF, or general volume rendering. The acquisition unit 100 stores, in the point information storage unit 202, the point information acquired for a plurality of training images observed in different wavelength ranges.

[0026] The estimation unit 101 inputs the point information acquired from the captured image by the acquisition unit 100 to a density estimation multilayer perceptron to estimate a density estimation value. That is, the estimation unit 101 reads the point information acquired from the captured image by the acquisition unit 100 from the point information storage unit 202. Then, the estimation unit 101 inputs the point information acquired from the captured image by the acquisition unit 100 to the density estimation multilayer perceptron to estimate a density estimation value related to volume rendering. Parameters of the density estimation multilayer perceptron are stored in the DNN parameter storage unit 203.

[0027] The density estimation multilayer perceptron is a multilayer perceptron (MLP) that estimates a density estimate. In an embodiment, the color estimation multilayer perceptron outputs a density distribution of a luminance field. The density estimation multilayer perceptron is trained by a training unit 103.

[0028] The color estimation multi-layer perceptron is a DNN independent of the density estimation multi-layer perceptron, and is a multi-layer perceptron that estimates color estimates. In this embodiment, the color estimation multi-layer perceptron outputs a color distribution of a luminance field. The color estimation multi-layer perceptron is trained by a training unit 103.

[0029] The estimation unit 101 inputs point information acquired from a visible light image to a color estimation multilayer perceptron related to the wavelength range of visible light, and estimates the RGB values ​​of the point related to the point information. The estimation unit 101 also inputs point information acquired from a thermographic image obtained by a thermosensor to a color estimation multilayer perceptron related to the wavelength range of infrared light, and estimates the reflected luminance, etc. of the point related to the point information in that wavelength range. The estimation unit 101 stores the density estimation value σ and the color estimation value for each wavelength range in the estimation value storage unit 204.

[0030] The learning unit 103 trains a DNN that extends BARF using training data based on visible light images and thermography images, i.e., the DNN described in Non-Patent Document 3. That is, the learning unit 103 trains a density estimation multilayer perceptron using point information acquired from multiple training images by the acquisition unit 100 and the multiple training images. The learning unit 103 also trains a color estimation multilayer perceptron associated with a specific wavelength range using point information acquired from training images observed in the specific wavelength range and the training images observed in the specific wavelength range. The learning unit 103 then stores the parameters of the trained density estimation multilayer perceptron and color estimation multilayer perceptron in the DNN parameter storage unit 203.

[0031] The image synthesis unit 102 generates a new viewpoint image by providing arbitrary viewpoint information to the luminance field generated by training the DNN of Non-Patent Document 3 and performing volume rendering. That is, the image synthesis unit 102 reads the density estimate σ and color estimates based on images from the visible light camera and the thermo sensor from the estimate value storage unit 204, and provides the arbitrary viewpoint information to the trained DNN (extended BARF) and performs volume rendering to generate a new viewpoint image. The generated new viewpoint image is stored in the generated image storage unit 205.

[0032] FIG. 3 is a diagram illustrating an example of a DNN architecture. In FIG. 3, point information based on a visible light image from a visible light sensor is input to a multilayer perceptron included in the DNN. The multilayer perceptron then outputs color estimates and density estimates in the visible light wavelength range. An image observed in the visible light wavelength range is then generated from the color estimates and density estimates. Note that the numbers written on the multilayer perceptron in FIG. 3 represent an example of the number of channels in each layer.

[0033] The structure shown in Figure 3 is disclosed in Non-Patent Document 3, and is obtained by branching the color and density estimates of the visible light image and the thermographic image from the BARF network structure (Non-Patent Document 2). The density information of the luminance field is shared between the rendering of both the visible light image and the thermographic image, while the color information is used only for the rendering of the corresponding one.

[0034] Non-Patent Document 3 performs the process of learning visible light images from a visible light sensor and then learning thermographic images from a thermosensor x arbitrary number of times while maintaining the DNN structure of Non-Patent Document 3. In this case, the RGB estimation MLP is not updated, but the MLPs in the previous eight layers are updated, which may change the color output for the visible light sensor.

[0035] In the same DNN structure, it is conceivable to stop updating the parameters of the MLP in the previous stage after completing learning of the visible light sensor image. However, while only the thermography estimated output MLP is updated through learning with the thermosensor image, the thinness of the MLP layers means that the visible light sensor image has a strong influence, and textures not included in the thermosensor image may appear. Increasing the number of layers in the thermography estimated output MLP increases expressiveness, but this is undesirable because it is expected to lead to a phenomenon similar to the degradation of synthesis results when the color output MLP is made thicker in BARF.

[0036] It is also possible to provide separate MLPs in front of the [RGB Estimation MLP & Density] and [Thermography Estimation Output MLP]. However, in this case, the problem of not being able to properly learn the thermosensor image arises. Compared to learning BARF using only the thermosensor, the difference is that density information is not updated. This may make learning more difficult than learning BARF using only the thermosensor.

[0037] 4 is a diagram showing an example of a DNN structure and processing flow according to an embodiment. The DNN shown in Fig. 4 includes an MLP (first multilayer perceptron: approximately eight layers) that outputs a density distribution of a luminance field, a color output MLP(1) (second multilayer perceptron) that outputs a color distribution of a luminance field based on a visible light image, an MLP (third multilayer perceptron) generated by copying the parameters of the first multilayer perceptron (MLP), and a color output MLP(2) (fourth multilayer perceptron) that is provided after the output layer of the MLP and outputs a color distribution of a luminance field based on a thermographic image.

[0038] In Figure 4, the acquired 3D point coordinates and gaze direction of each pixel are positionally encoded within the NeRF to obtain encoded gaze information. Then, after training the neural radiance field using images from one sensor (e.g., a visible light sensor), color information from images from the other sensor (e.g., a thermosensor) is added. That is, while preserving the training results for the visible light sensor image, a portion of the DNN (the preceding MLP) is copied and reused for training the thermosensor image. With this configuration, the preceding MLP for the thermography estimated output MLP is pre-trained using visible light sensor images. This eliminates the problems inherent in existing DNN structures and enables the synthesis of clear images from new viewpoints using multiple sensors.

[0039] FIG. 5 illustrates another example of a DNN structure and processing flow according to an embodiment. In addition to the configuration of FIG. 4 , the DNN shown in FIG. 5 includes a first filter provided after the output layer of the first multilayer perceptron (MLP) and a second filter provided between the third multilayer perceptron and the color output MLP (2). These filters extract and emphasize specific frequency components in the output of the density estimation MLP and the output of the color output MLP (2) for thermography estimation. This enables the BARF learning process, which learns from low-frequency components to high-frequency components. Alternatively, since the thermosensor starts from a pre-trained state, learning up to high-frequency components may be performed from the beginning, unlike BARF.

[0040] As described above, according to the embodiment, color information from other sensor images (e.g., thermographic images from a thermosensor) can be added to the NeRF, which has learned color and density from images from a sensor that can clearly capture the shape of an object, such as a visible light camera. This makes it possible to efficiently synthesize new viewpoint images from multiple sensors.

[0041] The present invention is not limited to the above-described embodiment. For example, the wavelength range for observing the learning image may be the ultraviolet wavelength range. Alternatively, both the ultraviolet wavelength range and the infrared wavelength range may be used.

[0042] In the embodiment, the image synthesis device 10 includes the estimation unit 101 and the image synthesis unit 102. However, this is not limiting. The image synthesis device 10 does not necessarily have to include the estimation unit 101 and the image synthesis unit 102.

[0043] Furthermore, the learning process executed by the CPU in the above embodiment by loading software (programs) may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as application-specific integrated circuits (ASICs) that are processors with circuit configurations specifically designed to execute specific processes. The learning process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0044] In the above embodiment, the learning program is pre-stored (installed) in the ROM 12 or the storage 14, but the present invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0045] Furthermore, in the implementation stage, the components of this invention can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0046] 10...Image synthesis device 11...CPU 12...ROM 13...RAM 14...Storage 19...Bus 100...Acquisition unit 101...Estimation unit 102...Image synthesis unit 103...Learning unit 200...Learning data storage unit 201...Viewpoint data storage unit 202...Point information storage unit 203...DNN parameter storage unit 204...Estimated value storage unit 205...Generated image storage unit.

Claims

1. An image synthesis device comprising: a learning unit that trains a neural network that maps viewpoint information, including viewpoint position and line of sight, to luminance field parameters using training data based on a first image captured by a first sensor sensitive to a first wavelength range and a second image captured by a second sensor sensitive to a second wavelength range; and an image synthesis unit that provides arbitrary viewpoint information to the luminance field generated by training the neural network and performs volume rendering to generate a new viewpoint image.

2. The image synthesis device of claim 1, wherein the neural network comprises: a first multilayer perceptron that outputs a density distribution of the luminance field; a second multilayer perceptron that outputs a color distribution of the luminance field based on the first image; a third multilayer perceptron generated by transferring parameters of the first multilayer perceptron; and a fourth multilayer perceptron provided after the output layer of the third multilayer perceptron and that outputs a color distribution of the luminance field based on the second image.

3. The image synthesis device according to claim 2, wherein the neural network comprises: a first filter disposed after the output layer of the first multilayer perceptron; and a second filter disposed between the third multilayer perceptron and the fourth multilayer perceptron.

4. An image synthesis method by a computer having a memory unit that stores a program and a processor that executes instructions of the program, the image synthesis method comprising the steps of: training a neural network that maps viewpoint information including viewpoint position and line of sight to parameters of a luminance field using training data based on a first image captured by a first sensor sensitive to a first wavelength range and a second image captured by a second sensor sensitive to a second wavelength range; and generating a new viewpoint image by volume rendering the luminance field generated by training the neural network based on arbitrary viewpoint information.

Images