Optical information learning generation apparatus, optical information learning generation method, and program

Abstract

To provide an optical information learning generation apparatus which, in generation of a two-dimensional image from optional viewpoints of a three-dimensional scene using different cameras, contributes making it possible to generate a two-dimensional image with same color regardless of difference of colors resulting from used cameras.SOLUTION: An optical information learning generation apparatus according to the present invention has an optical information learning estimator including an optical learning estimation unit for learning optical information by using, as a teacher data, a two-dimensional image of a three-dimensional scene acquired by a sensor, a sensor identifier input unit for receiving a sensor identifier for use in identifying the sensor, a spectral characteristic conversion unit, and a spectral characteristic setting unit for setting spectral characteristics to the spectral characteristic conversion unit. The spectral characteristic setting unit sets a learning spectral characteristic corresponding to the sensor identifier to the spectral characteristic conversion unit. The optical learning estimation unit outputs optical information to the succeeding part. The spectral characteristic conversion unit, in accordance with the learning spectral characteristics, output a color value based on the optical information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an optical information learning generation device, an optical information learning generation method, and a program.

Background Art

[0002] Regarding the display of a two-dimensional image of a three-dimensional scene, the following documents can be cited.

[0003] Patent Document 1 relates to a method for reconstructing color and depth information of a scene.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The following analysis is provided by the inventor of the present invention.

[0006] NeRF (Neural Radiance Fields), a learning-based 3D modeling technology, is rapidly spreading in a wide field dealing with three-dimensional (3D) shapes. In NeRF, it is assumed that a two-dimensional image of a three-dimensional scene acquired using a single camera is input in the form of an RGB signal as a teacher image.

[0007] However, even when acquiring 2D images of the same 3D scene, the acquired 2D images often have different color tones due to differences in the individual or type of camera. Also, when multiple cameras are used in combination, the same 2D image of a single 3D scene may have different color tones depending on the camera. In such cases, the use of multiple cameras in combination may result in a decrease in the accuracy of the 2D images of the 3D scene generated using the learned model. Alternatively, an individual model may be required for learning.

[0008] The present invention aims to provide an optical information learning generation device, an optical information learning generation method, and a program that contribute to enabling the generation of 2D images of the same color tone regardless of the difference in the color tone of the cameras used in generating 2D images of an arbitrary viewpoint of a 3D scene using different cameras.

Means for Solving the Problems

[0009] According to a first aspect of the present invention, an optical information learning inference device including an optical learning inference unit that learns optical information using a 2D image of a 3D scene acquired by a sensor as teacher data, a sensor identifier input unit that receives a sensor identifier for identifying the sensor, a spectral characteristic conversion unit, including a spectral characteristic setting unit that sets spectral characteristics for the spectral characteristic conversion unit, wherein the spectral characteristic setting unit sets learning spectral characteristics corresponding to the sensor identifier for the spectral characteristic conversion unit, the optical learning inference unit outputs optical information to a subsequent stage, and the spectral characteristic conversion unit can provide an optical information learning generation device that outputs a color value based on the optical information according to the learning spectral characteristics.

[0010] According to a second aspect of the present invention, a computer of an optical information learning generation device receives a sensor identifier for identifying a sensor, As spectral characteristics, setting learning spectral characteristics corresponding to the sensor identifier; Using the two-dimensional image of the three-dimensional scene acquired by the sensor as teacher data to learn optical information; Outputting optical information to a subsequent stage; An optical information learning and generation method can be provided, which includes outputting a color value based on the optical information according to the learning spectral characteristics. This method is associated with a specific machine, namely a computer that executes the above method.

[0011] According to a third aspect of the present invention, a program can be provided that causes a computer of an optical information learning and generation device to: Receive a sensor identifier for identifying a sensor; As spectral characteristics, perform a process of setting learning spectral characteristics corresponding to the sensor identifier; Using the two-dimensional image of the three-dimensional scene acquired by the sensor as teacher data to perform a process of learning optical information; Perform a process of outputting optical information to a subsequent stage; Perform a process of outputting a color value based on the optical information according to the learning spectral characteristics.

[0012] These programs can be recorded on a computer-readable storage medium. The storage medium can be non-transitory, such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, etc. The present invention can also be embodied as a computer program product.

Advantages of the Invention

[0013] According to the present invention, it is possible to provide an optical information learning and generation device, an optical information learning and generation method, and a program that contribute to enabling the generation of a secondary image with the same color tone in the generation of a secondary image of an arbitrary viewpoint of a three-dimensional scene using different cameras, regardless of the difference in the color tone of the cameras used.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0015] In the present disclosure, the drawings may be associated with one or more embodiments. Also, each of the embodiments described below can be combined with other embodiments as appropriate, and the present invention is not limited by each embodiment.

[0016] First, an overview of one embodiment will be described with reference to the drawings. Note that the reference numerals in the drawings appended to this overview are for convenience and are appended to each element as an example to assist understanding, and are not intended to limit the present invention to the illustrated aspects. Also, the connection lines between the blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional ones. The one-way arrows schematically show the flow of the main signals (data) and do not exclude bidirectionality.

[0017] FIG. 1 is a block diagram showing an example of the configuration of an optical information learning generation device according to the present disclosure. Referring to FIG. 1, the optical information learning generation device 100 includes an optical information learning inference unit 110, a spectral characteristic conversion unit 120, a spectral characteristic setting unit 130 that sets spectral characteristics for the spectral characteristic conversion unit 120, and a sensor identifier input unit 140. The optical information learning inference unit 110 includes, for example, an optical learning inference unit 112 that learns optical information using a two-dimensional image of a three-dimensional scene acquired by a sensor that is a two-dimensional camera as teacher data. In the present embodiment, the spectral characteristic refers to, as an example, a function representing the light sensitivity of red (R), green (G), and blue (B) with respect to the wavelength of light (referred to as an R function, a G function, and a B function, respectively). For example, the sum obtained by multiplying the element of optical information corresponding to the wavelength of the original optical signal by the light sensitivity indicated by the R function is the red (R) value, the sum obtained by multiplying the element of optical information corresponding to the wavelength of the original optical signal by the light sensitivity indicated by the G function is the green (G) value, and the sum obtained by multiplying the element of optical information corresponding to the wavelength of the original optical signal by the light sensitivity indicated by the B function may be the blue (B) value.

[0018] The sensor identifier input unit 140 receives, for example, a sensor identifier that identifies a sensor that is a two-dimensional camera. The spectral characteristic setting unit 130 sets learning spectral characteristics corresponding to the sensor identifier for the spectral characteristic conversion unit 120. The optical learning inference unit 112 outputs optical information 1121 to the subsequent stage. The spectral characteristic conversion unit 120 outputs a color value based on the optical information according to the learning spectral characteristics. The color value is, for example, an RGB value.

[0019] Even if two-dimensional images are acquired using different sensors, the spectral characteristic setting unit 130 sets spectral characteristics based on the sensor identifier of the sensor that acquired the two-dimensional image, so that it is possible to prevent learning using spectral characteristics of a sensor different from the sensor that acquired the two-dimensional image. At the time of inference, if two-dimensional image generation is performed using the inference spectral characteristics, a secondary image having the same color tone can be generated regardless of the difference in the color tone of the camera used.

[0020] Therefore, according to one embodiment, it is possible to contribute to enabling the generation of secondary images of the same color tone regardless of the difference in the color tone of the cameras used in the generation of secondary images of an arbitrary viewpoint of a three-dimensional scene, and an optical information learning generation apparatus, an optical information learning generation method, and a program can be provided.

[0021] [First Embodiment] Next, the first embodiment will be described in detail with reference to the drawings. FIG. 3 is a block diagram showing an example of the configuration of an image generation system according to the present disclosure. Further, FIG. 2 is a block diagram showing an example of the configuration of a conventional image generation system. Referring to FIG. 2, the conventional image generation system 10A includes an optical information learning generation apparatus 100A, a rendering apparatus 200, a two-dimensional camera (also referred to as a sensor) 300, a two-dimensional image display apparatus 400, and an observation viewpoint input apparatus 500 for a three-dimensional scene. The two-dimensional camera 300 acquires a two-dimensional image of the three-dimensional scene 1000.

[0022] First, an example of the operations of the conventional image generation system 10A during learning and inference will be described. During the learning of the conventional image generation system 10A, the optical information learning generation apparatus 100A learns the spatial structure and optical information using the two-dimensional image of the three-dimensional scene acquired by the two-dimensional camera 300 as teacher data by the optical information learning inference unit 110 and the error detection unit 150. Although the two-dimensional camera 300 is connected to the error detection unit 150 by a line, it is not necessary for the two-dimensional camera 300 and the error detection unit 150 to be directly connected, and any configuration may be used as long as the image captured by the two-dimensional camera 300 is input to the error detection unit 150.

[0023] The optical information learning and inference device 110 includes a density learning and inference unit 111, an optical learning and inference unit 112, and an RGB value output unit 113. As an example, from the rendering device 200, the three-dimensional coordinates 201 are input to the density learning and inference unit 111, and the observation orientation 202 is input to the optical learning and inference unit 112. The density learning and inference unit 111 outputs the density 1111 to the rendering device 200 and outputs an intermediate representation 1112 representing the spatial structure to the optical learning and inference unit 112. The optical learning and inference unit 112 takes the observation orientation 202 and the intermediate representation 1112 as inputs and outputs optical information 1121. The RGB value output unit 113 converts the optical information 1121 into an RGB value (also referred to as a color value) 1131 and outputs it to the rendering device 200. The rendering device 200 renders and outputs a two-dimensional image 203 based on the density 1111 and the RGB value 1131. Note that the RGB value output unit 113 is set with fixed spectral characteristics, for example, but not limited to, the spectral characteristics of the two-dimensional camera 300. Note that the density learning and inference unit 111 is a sub-concept of the intermediate representation learning and inference unit that performs learning and inference of the intermediate representation, and may have any configuration as long as it outputs the intermediate representation 1112.

[0024] The shooting viewpoint 502 of the two-dimensional camera 300 is input to the observation viewpoint input device 500 of the three-dimensional scene and is input to the rendering device 200 as the observation viewpoint 503. The rendering device 200 outputs the three-dimensional coordinates 201 and the observation orientation 202 to the optical information learning and inference device 110 based on the observation viewpoint 503. Note that in the present disclosure, the rendering device 200 is described as including the rendering function of the two-dimensional image 203 and the control function of outputting the three-dimensional coordinates 201 and the observation orientation 202 to the optical information learning and inference device 110. However, the control function of outputting the three-dimensional coordinates 201 and the observation orientation 202 to the optical information learning and inference device 110 may be configured to be executed by a control device separate from the rendering device 200.

[0025] The error detection unit 150 detects the error between the 2D image 203 rendered by the rendering device 200 and the 2D image (also referred to as the camera image) 301 of the 3D scene acquired by the 2D camera 300. Based on the error 151 output by the error detection unit 150, the optical information learning inference device 110 adjusts the parameters of the density learning inference unit 111 and the parameters of the optical learning inference unit 112 to generate a learned model. The generated learned model includes a first learned model in the density learning inference unit 111 and a second learned model in the optical learning inference unit 112.

[0026] Next, an example of the operation during inference of the conventional image generation system 10A will be described. During inference of the conventional image generation system 10A, as an example, the user's observation viewpoint 501 for the 3D scene 1000 is input to the observation viewpoint input device 500 of the 3D scene, and is input to the rendering device 200 as the observation viewpoint 503. Based on the observation viewpoint 503, the rendering device 200 outputs the 3D coordinates 201 and the observation orientation 202 to the learned model created as described above.

[0027] Based on the user's observation viewpoint 503 of the 3D scene input from the rendering device 200, with the 3D coordinates and the observation orientation of the 3D scene 1000 as inputs, the second learned model in the optical learning inference unit 112 outputs optical information 1121, and the RGB value output unit 113 converts the optical information 1121 into an RGB value 1131 and outputs it to the rendering device 200. On the other hand, density 1111 is output from the first learned model in the density learning inference unit 111 to the rendering device 200. The rendering device 200 renders the 2D image of the 3D scene 1000 observed from the user's observation viewpoint 501 based on the density 1111 and the RGB value 1131, and outputs it to the 2D image display device 400. The 2D image display device 400 displays the 2D image 203 of the 3D scene 1000 observed from the user's observation viewpoint 501.

[0028] Next, the configuration and operation of the image generation system according to the first embodiment will be described with reference to FIG. 3. The first embodiment is an example in the case of using a two-dimensional camera (sensor) 300 and a second two-dimensional camera 310, and the color tones of the two-dimensional camera 300 and the second two-dimensional camera 310 are assumed to be different. In FIG. 3, components denoted by the same reference numerals as those in FIG. 2 represent the same components, and the description thereof will be omitted.

[0029] Referring to FIG. 3, the image generation system 10 includes an optical information learning generation device 100, a rendering device 200, a two-dimensional camera (also referred to as a sensor) 300, a two-dimensional image display device 400, and an observation viewpoint input device 500 for a three-dimensional scene. The second two-dimensional camera 310 is included. The two-dimensional camera 300 and the second two-dimensional camera 310 acquire two-dimensional images of the three-dimensional scene 1000. In the present disclosure, the rendering device 200 is described as including a rendering function for the two-dimensional image 203 and a control function for outputting the three-dimensional coordinates 201 and the observation orientation 202 to the optical information learning inference device 110. However, the control function for outputting the three-dimensional coordinates 201 and the observation orientation 202 to the optical information learning inference device 110 may be configured to be executed by a control device different from the rendering device 200.

[0030] The optical information learning generation device 100 of the image generation system 10 includes an optical information learning inference device 110, a spectral characteristic conversion unit 120, a spectral characteristic setting unit 130 for setting spectral characteristics in the spectral characteristic conversion unit 120, a sensor identifier input unit 140, an error detection unit 150, and a physical optical output unit 170 including a wavelength output unit 160. The optical information learning inference device 110 and the error detection unit 150 are the same as the optical information learning inference device 110 and the error detection unit 150 of the optical information learning generation device 100A of the conventional image generation system 10A described in FIG. 2. Also, the second two-dimensional camera 310 is assumed to have a different color tone from the two-dimensional camera 300.

[0031] Next, an example of the learning operation according to the first embodiment will be described with reference to the image generation system 10 according to the present disclosure shown in FIG. 3.

[0032] The differences between the operation during learning of the image generation system 10 according to the present disclosure shown in FIG. 3 and the operation during learning of the conventional image generation system 10A shown in FIG. 2 are as follows.

[0033] During learning, in the image generation system 10 according to the present disclosure shown in FIG. 3, as an example, the sensor identifier input unit 140 receives a sensor identifier ID 302 for identifying the two-dimensional camera (sensor) 300 from the two-dimensional camera 300 and outputs it to the spectral characteristic setting unit 130 as the sensor identifier ID 141. The spectral characteristic setting unit 130 sets the learning spectral characteristics corresponding to the sensor identifier ID 302 for the spectral characteristic conversion unit 120 according to the sensor identifier ID 141. Although the two-dimensional camera 300 is connected to the error detection unit 150 and the sensor identifier input unit 140 by lines, it is not necessary for the two-dimensional camera 300 to be directly connected to the error detection unit 150 and the sensor identifier input unit 140. Any configuration in which the image captured by the two-dimensional camera 300 and the sensor identifier (ID) are input to the error detection unit 150 and the sensor identifier input unit 140 is acceptable. The shooting viewpoint 502 of the two-dimensional camera 300 is input to the observation viewpoint input device 500 of the three-dimensional scene.

[0034] Also, when using the image 311 acquired by the second two-dimensional camera 310 with a different color tone from the two-dimensional camera 300, as an example, the image 311 acquired by the second two-dimensional camera 310 is input to the error detection unit 150. Also, the sensor identifier ID 312 of the second two-dimensional camera 310 is input to the sensor identifier input unit 140, and the sensor identifier input unit 140 outputs it to the spectral characteristic setting unit 130 as the sensor identifier ID 141. The spectral characteristic setting unit 130 sets the learning spectral characteristics corresponding to the sensor identifier ID 312 for the spectral characteristic conversion unit 120 according to the sensor identifier ID 141. When using the second two-dimensional camera 310, the shooting viewpoint 512 of the second two-dimensional camera 310 is input to the observation viewpoint input device 500 of the three-dimensional scene.

[0035] The spectral characteristic conversion unit 120 converts the optical information 1121 into RGB values (also referred to as color values) 121 according to the learning spectral characteristics corresponding to the set sensor identifier ID 141, and outputs the result to the rendering device 200. The rendering device 200 renders and outputs a two-dimensional image 203 based on the density 1111 and the RGB values 121.

[0036] The wavelength output unit 160 of the physical optics output unit 170 converts the optical information 1121 output by the optical learning inference unit 112 of the optical information learning inference device 110 into a wavelength vector 161, and outputs it to the spectral characteristic conversion unit 120 as the output 171 of the physical optics output unit 170. The wavelength vector 161 includes the levels for each wavelength of light as elements of the vector. Note that when the elements of the wavelength vector 161 are arranged in wavelength order, it is optimal because of good computational efficiency. For example, during convolution operations, it can reflect the influence of physically close wavelength elements appropriately, enabling efficient operation. However, it is not necessarily required to be arranged in wavelength order. As long as the correspondence between the subsequent spectral characteristic conversion unit 120 and each element of the wavelength vector is established, the operation is possible. Also, the unit of the elements of the wavelength vector does not have to be wavelength, and depending on the field of use, it may be replaced with a suitable representation for expressing color or spectrum. For example, the reciprocal of wavelength or frequency can also be mathematically treated in the same way, so these may be adopted.

[0037] The wavelength output unit 160 includes a neural network. During learning, it takes the error output by the error detection unit 150 described with reference to FIG. 2 as input, adjusts the parameters of the neural network, and creates a physical optics output model. The created physical optics output model converts the optical information 1121 into a wavelength vector 161 that includes the levels for each wavelength of light as elements of the vector, that is, into a wavelength spectrum. The spectral characteristic conversion unit 120 generates RGB values (color values) 121 from the levels for each wavelength of light output from the wavelength output unit 160, that is, from the wavelength vector 161, according to the set spectral characteristics.

[0038] That is, in the first embodiment, optical information 1121 is converted into a wavelength vector 161, and the converted RGB values 121 are input to the rendering device 200 by the spectral characteristic conversion unit 120 with learning spectral characteristics set for the wavelength vector 161. In the first embodiment, the spectral characteristics refer to functions representing the light sensitivities of red (R), green (G), and blue (B) lights with respect to the wavelength of light (hereinafter referred to as the R function, G function, and B function, respectively, as an example). That is, the sum of multiplying the elements of the wavelength vector 161 corresponding to the wavelength of the original optical signal by the light sensitivity indicated by the R function is the red (R) value, the sum of multiplying the elements of the wavelength vector 161 corresponding to the wavelength of the original optical signal by the light sensitivity indicated by the G function is the green (G) value, and the sum of multiplying the elements of the wavelength vector 161 corresponding to the wavelength of the original optical signal by the light sensitivity indicated by the B function is the blue (B) value.

[0039] The learning operation during the learning of the image generation system 10 according to the present disclosure shown in FIG. 3 is the same as the learning operation of the conventional image generation system 10A described with reference to FIG. 2, except that the wavelength vector 161 output by the wavelength output unit 160 is sent to the spectral characteristic conversion unit 120 as the output 171 of the physical optics output unit 170, and during the learning, the physical optics output unit 170 takes the error output by the error detection unit 150 as an input and adjusts the parameters of the neural network of the wavelength output unit 160 to create a physical optics output model.

[0040] That is, similar to the operation during the learning of the conventional image generation system 10A described with reference to FIG. 2, during the learning of the image generation system 10 according to the present disclosure shown in FIG. 3, based on the error 151 output by the error detection unit 150, the optical information learning inference device 110 adjusts the parameters of the density learning inference unit 111 and the parameters of the optical learning inference unit 112 to generate a learned model. The generated learned model includes a first learned model in the density learning inference unit 111 and a second learned model in the optical learning inference unit 112.

[0041] Next, the operation during the inference of the first embodiment will be described with reference to FIG. 3.

[0042] The differences between the operation during inference of the conventional image generation system 10A shown in FIG. 2 and the operation during inference of the image generation system 10 according to the present disclosure shown in FIG. 3 are as follows.

[0043] During inference of the image generation system 10 according to the present disclosure shown in FIG. 3, optical information 1121 is converted into a wavelength vector 161 by a physical optics output model in the wavelength output unit 160 of the physical optics output unit 170, and the wavelength vector 161 is sent to the spectral characteristic conversion unit 120 as the output 171 of the physical optics output unit 170. Except for this point, it is the same as the inference operation of the conventional image generation system 10A described with reference to FIG. 2.

[0044] That is, in the image generation system 10 according to the present disclosure shown in FIG. 3, the spectral characteristic setting unit 130 sets the spectral characteristics for inference in the spectral characteristic conversion unit 120 according to the user's input 132 for specifying spectral characteristics for inference. The spectral characteristics for inference may be, for example, ideal fixed spectral characteristics. Based on the 3D coordinates and observation azimuth of the 3D scene, which are generated based on the user's observation viewpoint 503 of the 3D scene input from the rendering device 200, the second learned model in the optical learning inference unit 112 outputs the optical information 1121, and the optical information 1121 is converted into a wavelength vector 161 by a physical optics output model in the wavelength output unit 160 of the physical optics output unit 170, and the wavelength vector 161 is sent to the spectral characteristic conversion unit 120 as the output 171 of the physical optics output unit 170. The spectral characteristic conversion unit 120 with the spectral characteristics for inference set converts the output 171 into an RGB value (also referred to as a color value) 121 and outputs it to the rendering device 200.

[0045] And, similar to the operation during inference of the conventional image generation system 10A shown in FIG. 2, during inference of the image generation system 10 according to the present disclosure shown in FIG. 3, the rendering device 200 renders a 2D image of the 3D scene 1000 observed from the user's observation viewpoint 503 and outputs it to the 2D image display device 400. The 2D image display device 400 displays a 2D image of the 3D scene 1000 observed from the user's observation viewpoint 501.

[0046] In the first embodiment, during learning, the spectral characteristic setting unit 130 sets learning spectral characteristics corresponding to the sensor identifier ID in the spectral characteristic conversion unit 120 and learns. Thus, even if there are differences in the color tones of the two-dimensional camera (sensor) 300 and the second two-dimensional camera (sensor) 310, the optical information 1121 obtained from the two-dimensional images acquired from the same shooting viewpoint of the three-dimensional scene can be made the same. Also, during inference, the spectral characteristic conversion unit 120 sets inference spectral characteristics and performs inference. By setting arbitrary spectral characteristics as the inference spectral characteristics, regardless of the color tone of the two-dimensional camera (sensor) 300 or 310 at the time of shooting when acquiring the two-dimensional image of the three-dimensional scene, a two-dimensional image can be generated according to the inference spectral characteristics.

[0047] In this way, according to the first embodiment, even when different two-dimensional cameras (sensors) are used, regardless of the difference in the color tones of the cameras used, a two-dimensional image of the three-dimensional scene 1000 observed from the user's observation viewpoint 501 can be displayed according to the inference spectral characteristics.

[0048] Therefore, according to the first embodiment, it contributes to making it possible to generate two-dimensional images of the same color tone regardless of the difference in the color tones of the cameras used in generating two-dimensional images of an arbitrary viewpoint of a three-dimensional scene using different cameras, and an optical information learning generation device, an optical information learning generation method, and a program can be provided.

[0049] Note that NeRF, at present, does not assume an expansion of the types of cameras, and for example, an IR (Infrared) camera, a spectral camera, a LiDAR (Light Detection and Ranging) camera, etc. cannot be used. According to the first embodiment, for example, when an IR camera, a spectral camera, a LiDAR (Light Detection and Ranging) camera, etc. are used, spectral characteristics such as those of an IR camera, a spectral camera, a LiDAR (Light Detection and Ranging) camera, etc. may be set as the learning spectral characteristics.

[0050] In addition, for the spectral characteristics for inference, it is also possible to set the spectral characteristics of a camera of a completely different type from the camera for which the spectral characteristics for learning were set. For example, when learning using an optical camera, the spectral characteristics of the optical camera are set as the spectral characteristics for learning, and at the time of inference, the spectral characteristics of an IR camera, a spectral camera, or a LiDAR (Light Detection and Ranging) camera may be set as the spectral characteristics for inference.

[0051] [Second Embodiment] Next, the second embodiment will be described in detail with reference to the drawings. FIG. 4 is a block diagram showing an example of the configuration of the image generation system according to the present disclosure. FIG. 4 shows an example of the configuration in the case where an optical reflection output unit 162 is provided in the physical optical output unit 170 in the example of the configuration of the image generation system according to the present disclosure shown in FIG. 3. In FIG. 4, components denoted by the same reference numerals as in FIG. 3 represent the same components.

[0052] In FIG. 4, the observation direction 202 in the optical information learning inference device 110 is the same as the observation direction 202 output from the rendering device 200 and input to the optical learning inference unit 112, and the output of the wavelength output unit 160 is output to the optical reflection output unit 162 in the observation direction 202.

[0053] The optical reflection output unit 162 generates a wavelength vector with three dimensions by adding two, the incident angle and the reflection angle, to each spectrum of the one-dimensional wavelength vector based on the observation direction 202, and outputs it as the output 171 of the physical optical output unit 170 to the spectral characteristic conversion unit 120. That is, the physical optical output unit 170 includes an optical reflection output unit 162 having a function of converting to an optical reflection model (BRDF) of light. Note that the angular distribution of the BRDF may be approximated and held as the coefficients of the spherical harmonic function.

[0054] The learning operation during learning is the same as the learning operation of the first embodiment described with reference to FIG. 3, except that the optical reflection output unit 162 adds two angles, the incident angle and the reflection angle, to the wavelength vector 161 output by the wavelength output unit 160 to form a three-dimensional wavelength vector, which is output as the output 171 of the physical optics output unit 170 to the spectral characteristic conversion unit 120. Also, the operation during inference is the same as the inference operation of the first embodiment described with reference to FIG. 3, except that the optical reflection output unit 162 adds two angles, the incident angle and the reflection angle, to the wavelength vector 161 output by the wavelength output unit 160 to form a three-dimensional wavelength vector, which is output as the output 171 of the physical optics output unit 170 to the spectral characteristic conversion unit 120.

[0055] Therefore, according to the second embodiment, in generating a secondary image of an arbitrary viewpoint of a three-dimensional scene using different cameras, it contributes to enabling the generation of secondary images of the same color tone regardless of the difference in color tone of the cameras used and considering the incident angle and the reflection angle. An optical information learning generation device, an optical information learning generation method, and a program can be provided.

[0056] [Third Embodiment] Next, the third embodiment will be described in detail with reference to the drawings. FIG. 5 is a block diagram showing an example of the configuration of an image generation system according to the present disclosure. FIG. 5 shows an example of the configuration in the case where an ambient light input unit 163 and an ambient light calculation unit 164 are provided in the physical optics output unit 170 in an example of the configuration of the image generation system according to the present disclosure shown in FIG. 3. In FIG. 5, components denoted by the same reference numerals as those in FIG. 3 represent the same components. The third embodiment will be described with reference to an example of the configuration of the image generation system according to the present disclosure shown in FIG. 5.

[0057] In FIG. 5, the observation direction 202 in the optical information learning inference device 110 is the same as the observation direction 202 output from the rendering device 200 and input to the optical learning inference unit 112, and the observation direction 202 is output to the ambient light input unit 163 of the physical optics output unit 170.

[0058] The ambient light input unit 163 receives the ambient light direction (lighting direction 165 and lighting color 166). The wavelength vector 161 output by the wavelength output unit 160 and the output of the ambient light input unit 163 are sent to the ambient light calculation unit 164. The wavelength vector including the ambient light direction calculated by the ambient light calculation unit 164 is output to the spectral characteristic conversion unit 120 as the output 171 of the physical optics output unit 170. Note that a learning or inference function may be provided in the ambient light input unit 163 or the ambient light calculation unit 164.

[0059] The learning operation during learning in the third embodiment is the same as the learning operation of the first embodiment described with reference to FIG. 3, except that the wavelength vector 161 output by the wavelength output unit 160 and the output of the ambient light input unit 163 are sent to the ambient light calculation unit 164, and the wavelength vector including the ambient light direction calculated by the ambient light calculation unit 164 is output to the spectral characteristic conversion unit 120 as the output 171 of the physical optics output unit 170. Also, the operation during inference is the same as the inference operation of the first embodiment described with reference to FIG. 3, except that the wavelength vector obtained by including the ambient light direction calculated by the ambient light calculation unit 164 in the wavelength vector 161 output by the wavelength output unit 160 is output to the spectral characteristic conversion unit 120 as the output 171 of the physical optics output unit 170.

[0060] Therefore, according to the third embodiment, it contributes to enabling the generation of secondary images of an arbitrary viewpoint of a 3D scene using different cameras with the same color tone, regardless of the difference in the color tone of the cameras used and considering the ambient light direction (lighting direction and lighting color). An optical information learning generation device, an optical information learning generation method, and a program can be provided.

[0061] [Fourth Embodiment] Next, the fourth embodiment will be described in detail with reference to the drawings. In the first to third embodiments shown in FIGS. 3 to 5, the spectral characteristic conversion unit 120 specified the spectral characteristics based on the identifier ID of the 2D camera.

[0062] On the other hand, the spectral characteristic conversion unit 120 is configured by a neural network, the actual wavelength vector (wavelength spectrum) of the camera image captured by the two-dimensional camera 300 is obtained, input to the neural network of the spectral characteristic conversion unit 120, and the output image of the neural network of the spectral characteristic conversion unit 120 and the camera image (correct data) captured by the two-dimensional camera are made to have no difference. By learning the coefficients of the neural network of the spectral characteristic conversion unit 120, a learned spectral characteristic conversion unit model can also be generated.

[0063] Spectral characteristics equal to those of the two-dimensional camera can be obtained by learning the camera image. During the learning of the first to third embodiments, a learned spectral characteristic conversion unit model may be set in the spectral characteristic conversion unit 120 as the spectral characteristics of the two-dimensional camera used.

[0064] As described above, according to the fourth embodiment, the spectral characteristics of the two-dimensional camera can be obtained by learning.

[0065] [Fifth Embodiment] Next, the fifth embodiment will be described in detail with reference to the drawings. FIG. 6 is a block diagram showing an example of the configuration of the image generation system according to the present disclosure. In FIG. 6, components denoted by the same reference numerals as in FIG. 3 represent the same components, and the description thereof will be omitted. The fifth embodiment is different from the first embodiment in that the physical optical output unit 170 including the wavelength output unit 160 is deleted, and the optical information 1121 output from the optical learning inference unit 112 of the optical information learning inference device 110 is input to the spectral characteristic conversion unit 120. Other points are the same as the example of the configuration of the image generation system described in FIG. 3.

[0066] In the first embodiment, the optical information 1121 is converted into the wavelength vector 161 by the wavelength output unit 160 of the physical optical output unit 170 and input to the spectral characteristic conversion unit. On the other hand, the operation during the learning of the fifth embodiment is configured such that the optical information 1121 is directly input to the spectral characteristic conversion unit 120 without passing through the physical optical output unit 170.

[0067] Therefore, the operation during inference in the fifth embodiment is the same as the operation during inference in the first embodiment, except that the optical information 1121 is input to the spectral characteristic conversion unit 120.

[0068] According to the fifth embodiment, in generating a secondary image of an arbitrary viewpoint of a three-dimensional scene using different cameras, it is possible to generate a secondary image of the same color tone regardless of the difference in the color tone of the cameras used and considering the ambient light direction. It is possible to provide an optical information learning generation device, an optical information learning generation method, and a program that contribute to this.

[0069] [Sixth Embodiment] Next, the sixth embodiment will be described in detail with reference to the drawings. In the block diagram showing an example of the configuration of the image generation system according to the present disclosure described in FIGS. 4 and 5, the physical optical output unit 170 is described as including the wavelength output unit 160. However, in the sixth embodiment, as in the configuration described in the block diagram showing an example of the configuration of the image generation system according to the present disclosure in FIG. 6, in FIGS. 4 and 5, the wavelength output unit 160 of the physical optical output unit 170 is deleted, and the optical information 1121 is configured to be input to the optical reflection output unit 162, and the optical information 1121 is configured to be input to the ambient light calculation unit 164.

[0070] Therefore, according to the sixth embodiment, in generating a secondary image of an arbitrary viewpoint of a three-dimensional scene using different cameras, it is possible to generate a secondary image of the same color tone regardless of the difference in the color tone of the cameras used and considering the ambient light direction. It is possible to provide an optical information learning generation device, an optical information learning generation method, and a program that contribute to this.

[0071] As described above, each embodiment of the present invention has been explained. However, the present invention is not limited to the above-described embodiments, and further modifications, substitutions, and adjustments can be made without departing from the basic technical idea of the present invention. For example, the network configuration shown in each drawing, the configuration of each element, and the expression form of the message are examples for assisting in the understanding of the present invention, and are not limited to the configurations shown in these drawings. Further, "A and / or B" is used to mean at least either A or B.

[0072] Also, the procedures shown in the first to sixth embodiments described above can be realized by a program that causes a computer (9000 in FIG. 7) functioning as an optical information learning generation device according to the present invention to realize the functions of the optical information learning generation device. Such a computer is exemplified by a configuration including a CPU (Central Processing Unit) 9010, a communication interface 9020, a memory 9030, and an auxiliary storage device 9040 in FIG. 7. That is, the CPU 9010 in FIG. 7 may execute the control program of the optical information learning generation device and perform update processing of each calculation parameter held in the auxiliary storage device 9040 or the like.

[0073] The memory 9030 is a RAM (Random Access Memory), a ROM (Read Only Memory), or the like.

[0074] That is, each part (processing means, function) of the optical information learning generation device shown in the first to sixth embodiments described above can be realized by a computer program that causes the processor of the above computer to execute each of the above processes using its hardware.

[0075] Finally, the preferred forms of the present invention will be summarized. [First Form] The optical information learning generation device may include an optical information learning inference device including an optical learning inference unit that learns optical information using a two-dimensional image of a three-dimensional scene acquired by a sensor as teacher data. The optical information learning and generation device may include a sensor identifier input unit that receives a sensor identifier for identifying the sensor. The optical information learning and generation device may include a spectral characteristic conversion unit. The optical information learning and generation device may include a spectral characteristic setting unit that sets spectral characteristics for the spectral characteristic conversion unit. The spectral characteristic setting unit of the optical information learning and generation device preferably sets learning spectral characteristics corresponding to the sensor identifier for the spectral characteristic conversion unit. The optical learning inference unit of the optical information learning and generation device preferably outputs optical information to the subsequent stage. The spectral characteristic conversion unit of the optical information learning and generation device preferably outputs a color value based on the optical information according to the learning spectral characteristics. [Second form] The optical information learning and generation device according to the first form may further include a physical optical output unit including a wavelength output unit that converts the optical information into a wavelength vector. In the optical information learning and generation device according to the first form, the spectral characteristic conversion unit preferably outputs the color value based on the optical information by converting the wavelength vector output by the wavelength output unit into the color value and outputting it. [Third form] In the optical information learning and generation device according to the first or second form, the three-dimensional coordinates and observation azimuth of the three-dimensional scene generated based on the shooting viewpoint of the two-dimensional image of the three-dimensional scene acquired by the sensor are preferably input to the optical information learning inference device. The optical information learning and generation device according to the first or second form may further include an error detection unit that detects an error between a two-dimensional image rendered using the color value output by the spectral characteristic conversion unit based on the three-dimensional coordinates and the observation azimuth and the two-dimensional image of the three-dimensional scene acquired by the sensor. In the optical information learning and generation device according to the first or second form, based on the error output by the error detection unit, it is preferable that the optical information learning inference device adjusts the parameters of the optical learning inference unit and generates a learned model. [Fourth Embodiment] In the optical information learning and generation apparatus according to the third embodiment, it is preferable that the spectral characteristic setting unit sets the inference spectral characteristics for the spectral characteristic conversion unit. In the optical information learning and generation apparatus according to the third embodiment, it is preferable that the learned model outputs the optical information using, as inputs, the three-dimensional coordinates and the observation orientation of the three-dimensional scene generated based on the observation viewpoint of the user of the input three-dimensional scene. In the optical information learning and generation apparatus according to the third embodiment, it is preferable that the spectral characteristic conversion unit outputs color values based on the optical information according to the inference spectral characteristics. [Fifth Embodiment] In the optical information learning and generation apparatus according to the fourth embodiment, it is preferable that the spectral characteristic setting unit sets, for the spectral characteristic conversion unit, one of the plurality of inference spectral characteristics. [Sixth Embodiment] In the optical information learning and generation apparatus according to the second embodiment, it is preferable that the physical optical output unit includes an optical reflection output unit having a function of converting to an optical reflection model (BRDF) of light. [Seventh Embodiment] In the optical information learning and generation apparatus according to the second embodiment, it is preferable that the physical optical output unit includes an ambient light input unit and an ambient light calculation unit. [Eighth Embodiment] In the optical information learning and generation apparatus according to the first embodiment, it is preferable that the spectral characteristic conversion unit has a learning function and learns the sensor response characteristics of the sensor. [Ninth Embodiment] The optical information learning and generation method may include the computer of the optical information learning and generation apparatus receiving a sensor identifier for identifying a sensor. The optical information learning and generation method may include the computer of the optical information learning and generation apparatus setting, as spectral characteristics, learning spectral characteristics corresponding to the sensor identifier. The optical information learning and generation method may include the computer of the optical information learning and generation apparatus learning optical information using, as teacher data, a two-dimensional image of a three-dimensional scene acquired by the sensor. The optical information learning generation method may include the computer of the optical information learning generation device outputting optical information to a subsequent stage. The optical information learning generation method may include the computer of the optical information learning generation device outputting a color value based on the optical information according to the learning spectral characteristics. [Tenth form] The program may cause the computer of the optical information learning generation device to execute a process of receiving a sensor identifier for identifying a sensor. The program may cause the computer of the optical information learning generation device to execute a process of setting, as spectral characteristics, learning spectral characteristics corresponding to the sensor identifier. The program may cause the computer of the optical information learning generation device to execute a process of learning optical information using, as teacher data, a two-dimensional image of a three-dimensional scene acquired by the sensor. The program may cause the computer of the optical information learning generation device to execute a process of outputting optical information to a subsequent stage. The program may cause the computer of the optical information learning generation device to execute a process of outputting a color value based on the optical information according to the learning spectral characteristics. Note that, similar to the first form, the ninth and tenth forms can be expanded to the second to eighth forms.

[0076] Note that the disclosures of the above patent documents are incorporated herein by reference. Within the scope of the entire disclosure of the present invention (including the claims), modifications and adjustments of the embodiments or examples can be made based on the basic technical idea. Also, within the scope of the disclosure of the present invention, various combinations or selections of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. That is, the present invention naturally includes all the disclosures including the claims and various modifications and corrections that could be made by those skilled in the art according to the technical idea. In particular, for the numerical ranges described in this document, any numerical value or small range included within the range should be construed as specifically described even in the absence of separate description. Furthermore, each disclosure item of the above-cited documents, as necessary and in accordance with the spirit of the present invention, is considered to be included in the disclosure of the present application as a part of the disclosure of the present invention, and can be used in combination with the description items of this document, either in part or in whole.

Explanation of Reference Signs

[0077] 10, 10A Image generation system 100, 100A Optical information learning generation device 110 Optical information learning inference device 111 Density learning inference unit 112 Optical learning inference unit 113 RGB value output unit 120 Spectral characteristic conversion unit 130 Spectral characteristic setting unit 140 Sensor identifier input unit 150 Error detection unit 160 Wavelength output unit 162 Optical reflection output unit 163 Ambient light input unit 164 Ambient light calculation unit 170 Physical optics output unit 200 Rendering device 300 2D camera (sensor) 310 Second 2D camera (sensor) 400 2D image display device Observation viewpoint input device for 500 three-dimensional scenes 1000 Three-dimensional scene 9000 Computer 9010 CPU 9020 Communication interface 9030 Memory 9040 Auxiliary storage device

Claims

1. An optical information learning and inference device including an optical learning and inference unit that learns optical information using a two-dimensional image of a three-dimensional scene acquired by a sensor as teacher data, a sensor identifier input unit that receives a sensor identifier for identifying the sensor, a spectral characteristic conversion unit, including a spectral characteristic setting unit that sets spectral characteristics for the spectral characteristic conversion unit, wherein the spectral characteristic setting unit sets learning spectral characteristics corresponding to the sensor identifier for the spectral characteristic conversion unit, the optical learning and inference unit outputs optical information to a subsequent stage, and the spectral characteristic conversion unit outputs a color value based on the optical information according to the learning spectral characteristics, an optical information learning and generation device.

2. further including a physical optics output unit including a wavelength output unit that converts the optical information into a wavelength vector, wherein the spectral characteristic conversion unit outputs the color value based on the optical information by converting the wavelength vector output by the wavelength output unit into the color value and outputting it, the optical information learning and generation device according to claim 1.

3. the three-dimensional coordinates and observation azimuth of the three-dimensional scene generated based on the shooting viewpoint of the two-dimensional image of the three-dimensional scene acquired by the sensor are input to the optical information learning and inference device, further including an error detection unit that detects an error between a two-dimensional image rendered using the color value output by the spectral characteristic conversion unit based on the three-dimensional coordinates and the observation azimuth and the two-dimensional image of the three-dimensional scene acquired by the sensor, wherein based on the error output by the error detection unit, the optical information learning and inference device adjusts parameters of the optical learning and inference unit to generate a learned model, the optical information learning and generation device according to claim 1 or 2.

4. the spectral characteristic setting unit sets inference spectral characteristics for the spectral characteristic conversion unit, using the three-dimensional coordinates and observation azimuth of the three-dimensional scene generated based on the observation viewpoint of the user of the input three-dimensional scene as input, the learned model outputs the optical information, and the spectral characteristic conversion unit outputs a color value based on the optical information according to the inference spectral characteristics, the optical information learning and generation device according to claim 3.

5. the spectral characteristic setting unit sets one of a plurality of inference spectral characteristics for the spectral characteristic conversion unit, the optical information learning and generation device according to claim 4.

6. The optical information learning and generating apparatus according to claim 2, wherein the physical optical output unit includes an optical reflection output unit having a function of converting light into an optical reflection model (BRDF).

7. The optical information learning and generating apparatus according to claim 2, wherein the physical optical output unit includes an ambient light input unit and an ambient light calculation unit.

8. The optical information learning and generating apparatus according to claim 1, wherein the spectral characteristic conversion unit has a learning function and learns the sensor response characteristics of the sensor.

9. A computer of an optical information learning and generating apparatus, receiving a sensor identifier for identifying a sensor; setting, as spectral characteristics, learning spectral characteristics corresponding to the sensor identifier; learning optical information using a two-dimensional image of a three-dimensional scene acquired by the sensor as teacher data; outputting optical information to a subsequent stage; An optical information learning and generating method, including outputting a color value based on the optical information according to the learning spectral characteristics.

10. A program for causing a computer of an optical information learning and generating apparatus to perform a process of receiving a sensor identifier for identifying a sensor; perform a process of setting, as spectral characteristics, learning spectral characteristics corresponding to the sensor identifier; perform a process of learning optical information using a two-dimensional image of a three-dimensional scene acquired by the sensor as teacher data; perform a process of outputting optical information to a subsequent stage; perform a process of outputting a color value based on the optical information according to the learning spectral characteristics.

Images