Three-dimensional modeling apparatus, three-dimensional modeling system, three-dimensional modeling method, and program

Abstract

This technology provides the ability to obtain physically accurate and consistent rendering results in scenes containing objects with multiple reflective and refractive layers. [Solution] The three-dimensional modeling device comprises a model that estimates the path and intensity of light rays to represent refraction and reflection, and a rendering unit that generates incident light under arbitrary lighting conditions and performs ray tracing using a model trained with training data to generate a rendered image of the object to be observed under arbitrary lighting conditions.

Description

Technical Field

[0001] The present invention relates to the technologies of three-dimensional modeling devices, three-dimensional modeling systems, three-dimensional modeling methods, and programs.

Background Art

[0002] Research and development of 3D (three-dimensional) modeling and rendering technologies for objects are underway. For example, NeRF (Neural Radiance Fields) that represents a scene is learned from a set of still images observing the scene, and an image of the scene observed from an arbitrary position is rendered by obtaining and integrating the radiance on a linear line of sight (ray) using the learned NeRF. Note that NeRF is a neural network that returns the radiance at an arbitrary coordinate and observation direction in the scene (see, for example, Non-Patent Document 1).

[0003] In addition, by expanding the concept of NeRF, three-dimensional shape estimation based on silhouette images and estimation of Radiance Field are separately modeled, and learning is performed end-to-end with an image as an input, enabling reconstruction and rendering of a 3D scene considering transmission, reflection, and refraction. NeRRF has been proposed (see, for example, Non-Patent Document 2). When using NeRRF, ray tracing is performed on the estimated three-dimensional shape, and rendering is performed by obtaining and integrating the radiance by the Radiance Field on the traced ray.

[0004] In addition, as an extended technology of NeRF, LB-NeRF (Light Bending Neural Radiance Fields for Transparent Medium) has been proposed (see, for example, Non-Patent Document 3). When using the technology of LB-NeRF, an Offset Field that represents the amount of change when the path of a ray changes due to refraction is introduced, enabling learning of a consistent model from an image observing an object where refraction occurs.

[0005] Furthermore, NeRFrac (Neural Radiance Fields through Refractive Surface) has been proposed as an extension of NeRF (see, for example, Non-Patent Document 4). The Refractive Field is designed to estimate the distance from the origin of the incident ray to the refractive surface, and directly estimates the intersection point between the ray and the refractive surface. When using the NeRFrac technique, the position of the refractive surface is determined based on the Refractive Field, ray tracing is performed, and rendering is performed by calculating the radiance on the traced ray using the Radiance Field and integrating the results. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] NeRF, https: / / www.matthewtancik.com / nerf, <Internet search>, 2024.9.17 [Non-Patent Document 2] Xiaoxue Chen, Junchen Liu, Hao Zhao, Guyue Zhou, Ya-Qin Zhang: NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields [Non-Patent Document 3] T. Fujitomi, K. Sakurada, R. Hamaguchi, H. Shishido, M. Onishi and Y. Kameda, "LB-NERF: Light Bending Neural Radiance Fields for Transparent Medium," 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 2022, pp. 2142-2146, doi: 10.1109 / ICIP46576.2022.9897642. [Non-Patent Document 4] Yifan Zhan, Shohei Nobuhara, Ko Nishino, Yinqiang Zheng:NeRFrac: Neural Radiance Fields through Refractive Surface [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the technology described in Non-Patent Document 1 has a problem in that it cannot accurately reproduce the behavior of light that needs to be tracked during observation of objects that exhibit transmission, refraction, and reflection, because the behavior of light that needs to be tracked is complex. Furthermore, the technology described in Non-Patent Literature 2 has the problem that it cannot accurately estimate the shape from the silhouette of complex objects where light repeatedly reflects and refracts over multiple layers, making it impossible to learn a consistent model. The technology described in Non-Patent Literature 2 has the problem that it cannot learn in scenes where a silhouette cannot be obtained, such as when the scene cannot be observed from all sides. The technology described in Non-Patent Literature 2 has the problem that it is necessary to retrain part of the model in order to perform relighting, which is rendering under different lighting conditions than those at the time of shooting. Furthermore, in the technology described in Non-Patent Document 3, in complex scenes where light is repeatedly reflected and refracted over multiple layers, the brightness value of each pixel in the observed image is the result of the superposition of light rays that have followed such complex paths. Therefore, in order to train a consistent model, a huge amount of training data is required depending on the number of layers. Furthermore, the technology described in Non-Patent Document 4 has the problem that the refractive surface is a single layer and the refractive index must be known. Thus, conventional techniques had the problem of not being able to obtain physically accurate and consistent rendering results in scenes containing objects with multiple reflective and refractive layers.

[0008] This invention has been made in view of the circumstances described above, and provides a technique that enables obtaining physically accurate and consistent rendering results in scenes including objects with multiple reflective and refractive layers. [Means for solving the problem]

[0009] A three-dimensional modeling apparatus according to one aspect of the present invention comprises a model for estimating the path and intensity of light rays that represent refraction and reflection, and a rendering unit that generates incident light under arbitrary lighting conditions and performs ray tracing using the model trained with training data to generate a rendered image of an object to be observed under arbitrary lighting conditions.

[0010] A three-dimensional modeling system according to one aspect of the present invention is a three-dimensional modeling system comprising: a light irradiation device capable of irradiating a light ray or a beam of light rays having a certain spread into a scene including an object to be measured; an observation device for photographing the scene; a learning device for learning one of the following models: a model capable of estimating the path of a light ray including refraction and reflection, a model capable of estimating the intensity of a light ray for any coordinate in the space through which the light ray passes, and a model that represents the shape and reflection characteristics of an object to be observed; and a three-dimensional modeling device that generates incident light under arbitrary lighting conditions and performs ray tracing using the learned model to generate a rendered image under arbitrary lighting conditions.

[0011] A three-dimensional modeling method according to one aspect of the present invention is a three-dimensional modeling method in a three-dimensional modeling apparatus that includes a model for estimating the path and intensity of a light ray that represents refraction and reflection, wherein each of the models is a model learned using learning data which is a set of the irradiation direction or position of the light ray, the observation position of the observed light in the camera coordinate system, and the intensity of the observed light, and the way in which a light ray output from a light ray irradiation device undergoes transmission, refraction, reflection, and branching in a scene at a certain moment, and the rendering unit generates incident light under arbitrary lighting conditions and performs ray tracing using the model learned using the learning data to generate a rendered image of the object to be observed under arbitrary lighting conditions.

[0012] A program according to one aspect of the present invention is a program for causing a computer to function as the above-mentioned three-dimensional modeling apparatus. [Effects of the Invention]

[0013] This invention makes it possible to obtain physically accurate and consistent rendering results in scenes that include objects with multiple reflective and refractive layers. [Brief explanation of the drawing]

[0014] [Figure 1]This is a diagram showing a configuration example of the three-dimensional modeling system according to the embodiment. [Figure 2] This is a diagram showing an outline of the hardware configuration example of the three-dimensional modeling apparatus applied to the embodiment. [Figure 3] This is a diagram showing input / output data and processing outline during model learning according to the embodiment. [Figure 4] This is a flowchart of the processing procedure during model learning. [Figure 5] This is a flowchart of the processing procedure during rendering. [Figure 6] This is a diagram showing a configuration example of the three-dimensional modeling system according to the first example. [Figure 7] This is a diagram showing a configuration example of the three-dimensional modeling system according to the second example.

Embodiments for Carrying Out the Invention

[0015] Embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a configuration example of the three-dimensional modeling system according to the present embodiment. As shown in FIG. 1, the three-dimensional modeling system 1 includes, for example, an irradiation device 2, an observation device 3, a three-dimensional modeling device 4, a model learning device 5, and a rendering device 6. The three-dimensional modeling device 4 includes, for example, a model 41, a rendering unit 42, a storage unit 43, a ray tracing unit 44, a control unit 45, an acquisition unit 46, and an output unit 47. The model 41 includes, for example, a path estimation model 411 (first model) and an intensity estimation model 412 (second model). The model learning device 5 includes, for example, an acquisition unit 51, a virtual ray generation unit 52, a learning unit 53, and a storage unit 54.

[0016] The rendering device 6 includes, for example, a virtual camera 61, a virtual ray generation unit 62, an image generation unit 63, and an image output unit 64. The rendering device 6 may also be included in, for example, the three-dimensional modeling device 4, in which case the rendering unit 42 may have the functions of the virtual camera 61, the virtual ray generation unit 62, the image generation unit 63, and the image output unit 64.

[0017] The irradiation device 2 and the three-dimensional modeling device 4 are connected via a wired or wireless network (NW). The observation device 3 and the three-dimensional modeling device 4 are also connected via a wired or wireless network (NW). The three-dimensional modeling device 4 and the model learning device 5 are connected to each other via a wired or wireless network (NW).

[0018] Irradiation device 2 is an illumination device capable of irradiating a light ray or a beam of light ray with a certain spread into the scene, such as a laser irradiation device or a projector. Irradiation device 2 irradiates the object to be measured with a light ray according to the instructions of the three-dimensional modeling device 4.

[0019] Observation device 3 is, for example, a camera or an event-based vision camera. An event-based vision camera is a camera that outputs information about changes in light (objects) at high speed on a pixel-by-pixel basis. During measurement, observation device 3 photographs the object being illuminated by the light ray from illumination device 2 and outputs the captured image data or observation data including event data to the three-dimensional modeling device 4. During learning, observation device 3 observes the object being illuminated by the light ray from illumination device 2 and outputs the observation data to the three-dimensional modeling device 4 and the model learning device 5.

[0020] (Three-dimensional modeling device) The three-dimensional modeling device 4 uses the observation data captured by the observation device 3 and the model 41, which has been trained by the model learning device 5, to perform three-dimensional modeling of the target object.

[0021] Model 41 is a model that estimates the path and intensity of light rays representing refraction and reflection. For example, it is a combination of a model that estimates the path of light rays involving transmission, refraction, and reflection, and a model that estimates the intensity of light rays at each position along the path. Model 41 may be stored on an external device of the three-dimensional modeling device 4, or it may be stored in the cloud. Model 41 is trained by the model learning device 5.

[0022] The path estimation model 411 is a model capable of estimating the path taken by a light ray input to any point in space at any angle. For example, the path estimation model is based on a Refractive Field that represents refraction and reflection. In this case, the Refractive Field has parameters for each position in the field: the normal direction and the relative refractive index (the ratio of the refractive index of the incident medium to the refractive index of the outgoing medium when incident from the same direction as the normal direction) (or the normal direction and relative refractive index are returned when a position is input to the function). Therefore, the path estimation model 411 can estimate the path of a light ray input to any position in space at any angle by determining reflection, refraction, and branching due to reflection or refraction based on the normal direction and relative refractive index returned by the Refractive Field. The path estimation model 411 is based on other models that can estimate the state of the reflective and refractive surfaces and their reflective and refractive properties of any observed object, such as a combination of a Signed Distance Function (SDF), which is the distance to the nearest object surface for each coordinate, and refractive parameters, a voxel field with density on the object surface and inside as parameters, or the three-dimensional shape of the object and its material parameters.

[0023] The intensity estimation model 412 is a model that can estimate the intensity of a light ray based on radiance, color, transparency, attenuation rate, density, absorptance, or a combination thereof, for any point in space. For example, the intensity estimation model 412 is based on a Radiance Field that estimates the transparency of any point in space. By tracing the path of a light ray estimated by the path estimation model 411 and attenuating the intensity based on the values ​​returned by the Radiance Field at each point along the path, it is possible to estimate how the light ray attenuates along the path. Furthermore, the path estimation model 411 and the intensity estimation model 412 may be based on the same Refractive Field or Radiance Field. For example, if the Refractive Field returns the normal direction and relative refractive index corresponding to the coordinates, and light absorption can be ignored for the medium being measured, the direction of reflected and refracted light can be calculated using the Fresnel equation, and at the same time, their intensity can be calculated. In this case, path estimation and intensity estimation can be performed based on the same Refractive Field. Alternatively, if the Radiance Field returns the density corresponding to the coordinates, the relative refractive index and normal direction can be determined from the change in density along the path, and similarly, the direction of reflected and refracted light can be calculated using the Fresnel equation, and at the same time, their intensity can be calculated. In this case, path estimation and intensity estimation can be performed based on the same Radiance Field.

[0024] The rendering unit 42 generates a rendered image under arbitrary lighting conditions by generating incident light under arbitrary lighting conditions and performing ray tracing using the trained model 41 and the ray tracing unit 44.

[0025] The memory unit 43 stores, for example, thresholds, formulas, programs, etc., used by the control unit 45 and the rendering unit 42.

[0026] The ray tracing unit 44 determines the position, direction, and intensity of a ray input to the model 41 from an arbitrary direction and position, which is output as a ray after passing through, refraction, reflection, and the resulting branching.

[0027] The control unit 45, for example, instructs the start and end of irradiation by the irradiation device 2. The control unit 45 also controls the observation device 3. During learning, the control unit 45 updates the parameters of the model 41 using the parameters acquired by the acquisition unit 46 from the model learning device 5.

[0028] The acquisition unit 46 acquires observation data captured by the observation device 3. The acquisition unit 46 also acquires parameters output by the model learning device 5.

[0029] The output unit 47 outputs the rendering result from the rendering unit 42 to an external device. The external device may be, for example, an image display device, a smartphone, a mobile terminal, a tablet terminal, or a dedicated terminal.

[0030] During training, the model learning device 5 trains model 41. For example, the model learning device 5 trains model 41 so that the characteristics of the virtual observed light obtained by injecting an input light ray with the same parameters as the training data into the model match those of the training data.

[0031] (Model learning device) The acquisition unit 51 acquires the observation data output by the observation device 3.

[0032] The virtual ray generation unit 52 generates virtual rays used during the training of model 41. A virtual ray is a virtual ray used during training.

[0033] The learning unit 53 trains the model 41 using the observation data output by the observation device 3. The learning unit 53 uses the training data to determine each parameter of the model 41 and outputs the determined parameters to the three-dimensional modeling device 4.

[0034] The memory unit 54 stores, for example, algorithms, programs, thresholds, etc., necessary for learning.

[0035] (Rendering device) During rendering, the virtual camera 61 virtually photographs the object that is illuminated by the virtual rays generated by the virtual ray generation unit 62.

[0036] The virtual ray generation unit 62 generates a set of virtual rays for rendering. For example, the virtual ray generation unit 62 generates a set of virtual rays by setting up virtual lighting for rendering and determining the number, direction, and intensity of rays emitted from that lighting into the scene.

[0037] The image generation unit 63 generates a rendered image under arbitrary lighting conditions by performing ray tracing using the model 41 with the virtual rays generated by the virtual ray generation unit 62. The image generation unit 63 performs ray tracing on all virtual rays using the ray tracing unit 44 and renders the image based on the obtained intensity.

[0038] The image output unit 64 outputs the rendered image generated by the image generation unit 63 to an external device.

[0039] The configuration shown in Figure 1, the configuration of each device, and the functions and operations described above are examples only and are not limited to these.

[0040] All or part of the functions of the three-dimensional modeling apparatus 4, model learning apparatus 5, and rendering apparatus 6 in the above-described embodiment may be implemented using a computer. In that case, the functions may be implemented by recording a program for implementing these functions on a computer-readable recording medium, loading the program recorded on this recording medium into a computer system, and executing it. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such a case. Furthermore, the above-mentioned program may be for implementing part of the functions described above, or it may be a program that can implement the above-mentioned functions in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0041] Furthermore, Model 41 may be implemented using multiple information processing devices (e.g., servers) connected to each other via a network. In this case, the path estimation model 411 and intensity estimation model 412 of Model 41 may be implemented in a distributed manner across multiple information processing devices.

[0042] (Example of hardware configuration for a 3D modeling system) Figure 2 is a schematic diagram of an example hardware configuration of a three-dimensional modeling apparatus applied to this embodiment. The three-dimensional modeling apparatus 4 comprises a processor 401, main memory 402, communication interface 403, auxiliary storage device 404, input / output interface 405, and internal bus 406. The processor 401, main memory 402, communication interface 403, auxiliary storage device 404, and input / output interface 405 are connected to each other via the internal bus 406 so as to be able to communicate with each other. For example, the acquisition unit 46 may be configured using the communication interface 403 or the input / output interface 405. The output unit 47 may also be configured using the communication interface 403 or the input / output interface 405. Furthermore, for example, the storage unit 43 may be configured using the main memory 402 or the auxiliary storage device 404. Furthermore, the rendering unit 42 may be configured using the processor 401, main memory 402, and auxiliary storage device 404. Furthermore, the control unit 45 may be configured using the processor 401, main memory 402, and auxiliary storage device 404. Furthermore, the model 41 may be configured using, for example, the main memory 402.

[0043] (Dataset) Next, I will explain the dataset used for training. During training, the observation device 3 observes how a light ray emitted from a light ray irradiator at a given moment in time undergoes transmission, refraction, reflection, and branching within the scene. A dataset consists of a set of data representing the direction or position of the light ray irradiation, the observed position in the camera coordinate system, and the intensity of the observed light.

[0044] (Learning Methods) Next, we will explain an example of how to train Model 41. The model learning device 5 learns a model 41 that represents the reflection, refraction, and intensity of light in the observed space. Model 41 consists of, for example, a path estimation model 411 and an intensity estimation model 412.

[0045] While the explanation will describe an example using training data, the data used for training is not limited to this example. Data obtained by irradiating various objects with light rays and measuring the results may also be used as training data.

[0046] Furthermore, during learning, when light incident from a certain position undergoes repeated reflection, refraction, and branching within the field before reaching the outside of the field, the ray tracing unit 44 refers to the path estimation model 411 at the incident position to calculate the directions of the reflected and refracted light. Then, the ray tracing unit 44 performs ray tracing by repeating the same calculation at the next coordinate to which the calculated reflected and refracted light reaches. Furthermore, during training, the ray tracing unit 44 similarly estimates the intensity of light along the paths of reflected and refracted light based on the intensity estimation model 412. For example, if the intensity estimation model 412 is based on a Radiance Field that returns transparency for any coordinate, the ray tracing unit 44 can represent attenuation by multiplying the current ray intensity of the intensity estimation model 412 by the transparency while tracing the path of the light ray.

[0047] As shown in Figure 3, the model learning device 5 generates irradiation parameters identical to those of the training data and inputs the generated irradiation parameters into the model 41. The training data includes the characteristics of the observed light when a light ray is irradiated onto multiple objects. The characteristics of the observed light include the number of observed light beams, the coordinates and intensity of each observed light beam. The model 41 outputs the characteristics of the observed light obtained by incident on an input light ray based on the irradiation parameters. The model learning device 5 trains the model 41 so that its output matches the training data. The model learning device 5 considers the output of the model 41 to match the training data if the difference between the training data and the output of the model 41 is, for example, within a predetermined value. Figure 3 shows the input / output data and processing overview during model learning in this embodiment.

[0048] Furthermore, during learning, it is also possible to define that an element disappears when its intensity falls below a certain level. In this case, it is possible that no observation light is obtained for the input light ray, but this result can also be used for learning. In addition, the learning unit 53 may learn the intensity threshold at which an element should be treated as disappearing as a parameter.

[0049] Furthermore, with conventional technology, training such a model using only images observed with a regular camera requires a vast number of sample images for the same reasons as LB-NeRF. In contrast, in this embodiment, a set of emitted rays corresponding to a given incident ray can be observed and used for learning, making it possible to learn with a small number of samples.

[0050] Next, we will explain an example of the processing procedure in training Model 41. Figure 4 is a flowchart of the processing procedure during model training. Note that the processing procedures and content described below using Figure 4 are not limited to this example.

[0051] (Step S1) The acquisition unit 51 acquires training data.

[0052] (Step S2) The learning unit 53 generates irradiation parameters that are the same as those obtained from the training data. Note that the training data may also contain irradiation parameters. In this case, the learning unit 53 extracts the irradiation parameters from the training data.

[0053] (Step S3) The learning unit 53 inputs the generated irradiation parameters to the model 41 and obtains the characteristics of the observed light output by the model 41.

[0054] (Step S4) The learning unit 53 compares the output of model 41 with the training data.

[0055] (Step S5) The learning unit 53 determines whether or not learning has finished. The acquisition unit 51 determines that learning has finished, for example, if all the learning data has been learned and the difference between the output of the model 41 and the learning data is within a predetermined value. If learning has finished (Step S5; YES), the learning unit 53 proceeds to the process in Step S6. If learning has not finished (Step S5; NO), the learning unit 53 returns to Step S1.

[0056] (Step S6) The learning unit 53 outputs the learned parameters of model 41 to model 41 and updates model 41. Note that the model 41 update may be performed by the control unit 45.

[0057] (Processing during rendering)

[0058] Next, an example of the rendering process performed by the rendering device 6 or rendering unit 42 will be described. Figure 5 is a flowchart of the rendering process. Note that the following processing steps and contents described using Figure 5 are not limited to those shown.

[0059] (Step S11) The virtual ray generation unit 62 or rendering unit 42 generates incident light under arbitrary lighting conditions. The virtual ray generation unit 62 or rendering unit 42 generates a set of virtual rays for rendering. For example, the virtual ray generation unit 62 or rendering unit 42 generates a set of virtual rays by setting up virtual lighting for rendering and determining the number, direction, and intensity of rays emitted from that lighting into the scene.

[0060] (Step S12) The image generation unit 63 generates a rendered image under arbitrary lighting conditions by performing ray tracing using the model 41 with the virtual rays generated by the virtual ray generation unit 62. Alternatively, the rendering unit 42 generates a rendered image under arbitrary lighting conditions by performing ray tracing using the model 41 with the generated virtual rays. The image generation unit 63 or the rendering unit 42 performs ray tracing on all virtual rays using the ray tracing unit 44 and renders the image based on the intensity obtained.

[0061] (Step S13) The ray tracing unit 44 estimates the ray trace and the intensity of light along the ray path for each virtual ray based on the model 41.

[0062] (Step S14) The image output unit 64 outputs the rendering result to an external device. Alternatively, the rendering unit 42 outputs the rendering result to an external device.

[0063] As described above, according to this embodiment, in addition to learning the model based on the results of actually observing the irradiation state of the light ray from the irradiation device 2, by inputting irradiation parameters, which are the irradiation conditions, into the model 41, it is possible to render images under any illumination conditions without retraining. Note that the object of observation is not limited to objects through which light rays are transmitted or refracted, but may also be an object through which light rays are not transmitted or an object through which light rays are not refracted.

[0064] <First Example> In the first embodiment, we describe an example where the observation device 3 is a camera and the illumination device 2 is a laser. Figure 6 shows an example of the configuration of the three-dimensional modeling system of the first embodiment. As shown in Figure 6, the three-dimensional modeling system 1A includes, for example, an illumination device 2, an observation device 3, a three-dimensional modeling device 4, and a model learning device 5A. The three-dimensional modeling apparatus 4 includes, for example, a model 41, a rendering unit 42A, a storage unit 43, a ray tracing unit 44, a control unit 45, an acquisition unit 46, and an output unit 47. The model 41 includes, for example, a path estimation model 411 (first model) and an intensity estimation model 412 (second model). The model learning device 5A includes, for example, an acquisition unit 51, a virtual ray generation unit 52, a learning unit 53A, and a storage unit 54.

[0065] The three-dimensional modeling system 1A may also include a rendering device 6A instead of the rendering unit 42A. The rendering device 6A may include, for example, a virtual camera 61, a virtual ray generation unit 62, an image generation unit 63, and an image output unit 64. The rendering device 6 may also be included in, for example, the three-dimensional modeling system 4.

[0066] (Processing details) The learning unit 53A trains model 41 using the observation device 3 as a camera and the illumination device 2 as a laser. Specifically, the learning unit 53A simultaneously trains the path estimation model 411 and the intensity estimation model 412 using the same process as in Figure 4. The rendering unit 42A (or rendering device 6A) performs rendering using the model 41 that has been learned for any lighting conditions.

[0067] As a result, according to the first embodiment, the path of light, including reflection and refraction, can be learned from the positions and directions of multiple observed emitted light beams. Furthermore, according to the first embodiment, the attenuation caused by repeated reflection and refraction can be learned from the intensity of multiple observed emitted light beams.

[0068] Alternatively, instead of using the intensity of the emitted light, the ratio of the intensity s' of other emitted light to the strongest intensity s among the observed emitted light may be used as training data. This allows for the training of a consistent model even when the illumination intensity or camera sensitivity is unknown, or when observations are made with multiple cameras with different sensitivities.

[0069] Alternatively, by arranging the observation device 3 and the illumination device 2 according to the epipolar constraint, line illumination may be used as the illumination device 2 to obtain more samples in a single observation. Furthermore, the object to be observed may be placed on a turntable and scanned all around. Or, a reflective surface such as a mirror may be placed at an arbitrary position to obtain more samples in a single observation.

[0070] <Second Example> In the second embodiment, the observation device 3 illustrates an example of an event-based vision camera. Figure 6 shows an example of the configuration of the three-dimensional modeling system of the second embodiment. As shown in Figure 7, the three-dimensional modeling system 1B includes, for example, an illumination device 2, an observation device 3, a three-dimensional modeling device 4A, a model learning device 5B, and a virtual event generation device 7. The three-dimensional modeling apparatus 4 includes, for example, a model 41, a rendering unit 42A, a storage unit 43, a ray tracing unit 44, a control unit 45, an acquisition unit 46, and an output unit 47. Model 41 includes a path estimation model 411 (first model) and an intensity estimation model 412 (second model). The model learning device 5B includes, for example, an acquisition unit 51, a virtual ray generation unit 52, a learning unit 53B, and a storage unit 54.

[0071] The virtual event generator 7 generates events from observed light. For example, the virtual event generator 7 counts an event as occurring when the intensity of the observed emitted light is above a threshold. In the event-based vision camera, each pixel operates asynchronously and outputs the brightness change of each pixel with a time resolution of microseconds. The event-based vision camera illuminates an object with a light ray and observes the reflected light. Assuming that the object does not transmit light and there is no scattering by the surrounding environment, when light is shone on only one pixel of the illumination device 2, scattered light in a minute area on the surface of the corresponding object is observed by the event-based vision camera, and an event occurs at the corresponding pixel. In this embodiment, scenes including objects with multiple reflective and refractive layers are also targeted, so the virtual event generator 7 virtually generates these events. The virtual event generator 7 may also be configured as being included in the three-dimensional modeling device 4A.

[0072] In the second embodiment, instead of the intensity of the emitted light, the number of events that occur based on the observed intensity of the emitted light is used as training data. For this reason, the learning unit 53B acquires the number of events from the observation device 3 as training data during training. The training data set is a set of the direction or position of the light beam, the observed position in the camera coordinate system, and the number of events.

[0073] Furthermore, it is also possible to determine the actual light intensity by capturing not only the event at the moment of illumination, but also the number of events in the several microseconds before and after the moment of illumination, for example, by analyzing the distribution of event occurrence times. This is because the latency at the time of event occurrence depends on the light intensity. In this case, the light intensity may be determined using not only the observed events but also the parameters used during imaging.

[0074] In the second embodiment, since the change in brightness value is captured rather than the absolute amount, it is easier to capture light with low light intensity, and emitted light that has undergone many reflections and refractions and has been severely attenuated can also be used for learning. Furthermore, in the second embodiment, observations can be performed with high temporal resolution, allowing observations to be made while rapidly changing the irradiation position, and enabling observation of a large number of samples in a short time.

[0075] <Other examples> The embodiments and examples described above are merely examples and are not limited thereto. For example, a SPAD (Single Photon Avalanche Diode) sensor may be used to measure the probability (histogram) of a photon being detected at each coordinate of the sensor for a given incident light, and this can be used for learning. A SPAD sensor is a pixel structure that utilizes "avalanche multiplication," which amplifies electrons from a single incident photon, and is a sensor that can detect even weak light. Alternatively, multiple measuring devices may be used in combination. For example, a laser may be used as the irradiation device 2, an event vision sensor as the observation device 3, and a camera as yet another measuring device, and the observation data from these devices may be used to train a model.

[0076] As described above, this embodiment and each of the embodiments enable accurate modeling of complex optical properties. For example, this embodiment and each of the embodiments enable more detailed and accurate modeling of the optical properties of an object in which light is repeatedly reflected and refracted over multiple layers. Furthermore, according to this embodiment and each of the embodiments, observation conditions can be relaxed. For example, according to this embodiment and each of the embodiments, modeling can be performed even when it is not possible to obtain a silhouette, such as when the scene cannot be observed from all sides. Furthermore, according to this embodiment and each of the embodiments, highly accurate shape estimation is possible. For example, according to this embodiment and each of the embodiments, it is possible to estimate the precise 3D shape of an object with a complex shape involving multiple internal reflections and refractions, which was difficult with shape estimation based on silhouette images. Furthermore, according to this embodiment and each of its embodiments, simple relighting can be performed. For example, according to this embodiment and each of its embodiments, images can be rendered under any lighting conditions without retraining.

[0077] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0078] The configuration of this embodiment can be used in a wide range of industries that require high-quality real-time visualization, such as 3D modeling equipment, rendering equipment, the film industry, video games, virtual reality (VR), augmented reality (AR), interior design, and product design. [Explanation of Symbols]

[0079] 1,1A,1B…Three-dimensional modeling system, 2…Irradiation device, 3…Observation device, 4,4A…Three-dimensional modeling device, 5,5A,5B…Model learning device, 6,6A…Rendering device, 7…Virtual event generation device, 41,41A…Model, 42…Rendering unit, 43…Storage unit, 44…Ray tracking unit, 45…Control unit, 46…Acquisition unit, 47…Output unit, 48…Three-dimensional shape estimation network, 411…Path estimation model, 412…Intensity estimation model, 51…Acquisition unit, 52…Virtual ray generation unit, 53,53A,53B…Learning unit, 54…Storage unit, 61…Virtual camera, 62…Virtual ray generation unit, 63…Image generation unit, 64…Image output unit

Claims

1. A model that estimates the path of light rays and the intensity of light rays to represent refraction and reflection, A rendering unit that generates incident light under arbitrary lighting conditions and performs ray tracing using the model trained with training data to generate a rendered image of the object to be observed under arbitrary lighting conditions, A three-dimensional modeling device equipped with the following features.

2. The aforementioned model, The model is trained using the training data, which consists of a set of light rays emitted from a light irradiator at a certain moment in time, undergoing transmission, refraction, reflection, and branching in the scene, and a set of light rays emitted in a certain direction or position, the observed position of the observed light in the camera coordinate system, and the intensity of the observed light. The three-dimensional modeling apparatus according to claim 1.

3. The aforementioned model includes a ray tracking device that determines the position, direction, and intensity at which a ray input from any direction and any position is output as a ray after branching within the observed object. The rendering unit generates incident light under arbitrary lighting conditions and performs ray tracing using the information obtained by the ray tracing device to generate a rendered image under arbitrary lighting conditions. A three-dimensional modeling apparatus according to claim 1 or claim 2.

4. The system includes a virtual event generator that counts an event as occurring when the intensity of the observed emitted light is above a threshold and outputs the number of events. The aforementioned model, The model is trained using the number of events output by the aforementioned virtual event generator as training data. At a given moment, a light ray emitted from a light ray irradiator in a certain direction or position undergoes transmission, refraction, reflection, and branching within the scene, and this is observed as an event. The model is trained using the training data, which consists of the direction or position of the light ray irradiation, the observed position of the observed light in the camera coordinate system, and the intensity of the observed light. A three-dimensional modeling apparatus according to claim 1 or claim 2.

5. A light irradiation device capable of irradiating a beam of light or a beam of light with a certain spread into a scene including an object to be measured, An observation device for photographing the aforementioned scene, A learning device that learns one of the following models: a model capable of estimating the path of a light ray including refraction and reflection, a model capable of estimating the intensity of a light ray for any coordinate in the space through which the light ray passes, and a model that represents the shape and reflection characteristics of an observed object. A three-dimensional modeling apparatus that generates incident light under arbitrary lighting conditions and performs ray tracing using the trained model described above to generate a rendered image under arbitrary lighting conditions, A three-dimensional modeling system equipped with the following features.

6. The aforementioned model includes a ray tracking device that determines the position, direction, and intensity at which a ray input from an arbitrary direction and position is output as a ray after branching within the object being observed. The three-dimensional modeling apparatus generates incident light under arbitrary lighting conditions and performs ray tracing, using the information obtained by the ray tracing apparatus, thereby generating a rendered image under arbitrary lighting conditions. The three-dimensional modeling system according to claim 5.

7. A three-dimensional modeling method in a three-dimensional modeling apparatus, comprising a model for estimating the path and intensity of light rays that represent refraction and reflection, The aforementioned model is trained using training data that consists of a set of data: the direction or position of the light beam emitted from a light irradiator at a certain moment in time, the observed transmission, refraction, reflection, and branching of the light beam in the scene, and the observed position of the observed light in the camera coordinate system and the intensity of the observed light. The rendering unit generates incident light under arbitrary lighting conditions and performs ray tracing using the model trained with the training data to generate a rendered image of the object to be observed under arbitrary lighting conditions. Three-dimensional modeling methods.

8. Computers, A program for causing a three-dimensional modeling apparatus to function as described in claim 1 or claim 2.

Images