Rendering device and its program

Abstract

To provide a rendering apparatus capable of generating a natural CG image even from a non-high precision model.SOLUTION: A rendering apparatus 1 includes: a first physical base rendering unit 10 which applies physical base rendering with first virtual lighting; a second physical base rendering unit 11 which applies physical base rendering with second virtual lighting which is different from the first virtual lighting; an image base rendering unit 12 which applies image base rendering on an image base model; and a pixel value calculation unit 13 which calculates pixel values of corresponding pixels of an output CG from pixel values of corresponding pixels between a first physical base CG, a second physical base CG, and an image base CG.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a rendering device and a program therefor. [Background technology]

[0002] A physically based model models not only the shape of a subject but also optical properties such as reflection, refraction, and transmission at surfaces and interfaces, and internal light scattering. A known method of rendering this physically based model is physically based rendering, which generates a CG (Computer Graphics) image by simulating the behavior of light rays based on the optical properties using a computer or device (see, for example, Patent Document 1). Patent Document 1 discloses a method of measuring subsurface scattering of the face and forming a facial skin reflectance model that combines a bidirectional reflectance function, an albedo map, and a translucency map.

[0003] Image-based rendering is also known, which synthesizes an image from a virtual viewpoint by performing geometric transformation and interpolation on a real-life image. In image-based rendering, a known technique uses, in addition to the real image, a texture map that associates coordinates on the surface of a geometric model of a subject with coordinates on the real-life image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-277748 Summary of the Invention [Problem to be solved by the invention]

[0005] Physically based rendering can generate natural-looking CG images if the shape and optical properties of the subject are modeled with sufficient accuracy. However, trying to model real-world objects or living things with high accuracy requires sensors to measure various physical quantities, and physically based rendering requires enormous amounts of time and computational costs for sensing and analysis.

[0006] For example, when measuring the bidirectional reflectance distribution function (BRDF), it is necessary to illuminate the object surface with light close to a point light source or coded pattern light from various angles and capture the reflected images from various angles. This requires either a mechanism to change two or more of the camera position, light source position, or object orientation, or multiple cameras or light sources, resulting in a large-scale device. Furthermore, when measuring the bidirectional reflectance distribution function, information cannot be acquired with just one capture.

[0007] Furthermore, physically based rendering has limitations in describing the shapes and optical properties of natural objects and living things, and the parts that cannot be fully described appear as errors in the CG. In particular, physically based rendering can fall into the so-called "uncanny valley," a cognitive unnaturalness that is the final step just before photorealism, and this presents a barrier.

[0008] On the other hand, image-based rendering allows for the generation of natural-looking CG images, which avoids the "uncanny valley" problem, since it also incorporates the effects of light, which cannot be fully expressed using physical models. However, image-based rendering does not strictly model optical properties, making it difficult to change lighting conditions. In particular, for subjects with specular reflection, the reflection and glossiness change depending on the viewing angle, but image-based rendering is generally unable to express such reflections and glossiness.

[0009] As described above, both physically based rendering and image based rendering have their advantages and disadvantages, and it is difficult to generate natural CG images from models that are not highly accurate.

[0010] Therefore, an object of the present invention is to provide a rendering device and a program therefor that can generate natural CG images even from models that are not highly accurate. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems, the rendering device of the present invention is a rendering device that renders an output CG image from a model of a subject using physics-based rendering and image-based rendering, and is configured to include a first physics-based rendering unit, a second physics-based rendering unit, an image-based rendering unit, and a pixel value calculation unit.

[0012] According to this configuration, the first physically-based rendering unit generates a first physically-based CG image by applying physically-based rendering to the physically-based model under predetermined first virtual lighting conditions. The second physically-based rendering unit generates a second physically-based CG image by applying physically-based rendering to the physically-based model under a second virtual lighting condition different from the first virtual lighting condition.

[0013] The image-based rendering unit generates an image-based CG image by applying image-based rendering to the image-based model. The pixel value calculation unit calculates pixel values ​​of corresponding pixels in the output CG image from pixel values ​​of corresponding pixels in the first physically based CG image, the second physically based CG image, and the image-based CG image.

[0014] As described above, the rendering device calculates pixel value components that cannot be fully reproduced by the first physical rendering through calculations between the image-based CG image and the second physically-based CG image, and interpolates them into the first physically-based CG image. This allows the rendering device to generate natural CG images even from models that are not highly accurate.

[0015] The present invention can also be realized by a program for causing a computer to function as the rendering device described above. [Effects of the Invention]

[0016] According to the present invention, natural CG images can be generated even from models that are not highly accurate. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a rendering device according to an embodiment. [Figure 2] In the embodiment, (a) is a first physically-based CG, (b) is a second physically-based CG, (c) is an image-based CG, (d) is an output CG, and (e) is an explanatory diagram for explaining an expected output. [Figure 3] FIG. 1 is an explanatory diagram illustrating the principle of a rendering device in an embodiment. [Figure 4] 2 is a flowchart showing an example of the operation of the rendering device of FIG. 1; DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to the following. Furthermore, the same means will be given the same reference numerals, and their description may be omitted.

[0019] (Embodiment) [Example of rendering device configuration] An example of the configuration of a rendering device 1 according to an embodiment will be described with reference to FIG. The rendering device 1 uses physics-based rendering and image-based rendering to render an output CG (output CG image) from a model of a subject. As shown in Fig. 1, the rendering device 1 includes a first physics-based rendering unit 10, a second physics-based rendering unit 11, an image-based rendering unit 12, and a pixel value calculation unit 13.

[0020] The first physically-based rendering unit 10 generates a first physically-based CG image (a first physically-based CG image) by applying physically-based rendering to a physically-based model under a predetermined first virtual illumination (a first virtual illumination condition). In this embodiment, the first physically-based rendering unit 10 receives a physically-based model, first virtual illumination, and virtual camera parameters, and performs physically-based rendering based on the received physically-based model, first virtual illumination, and virtual camera parameters to generate a first physically-based CG image. The first physically-based rendering unit 10 then outputs the generated first physically-based CG image to the pixel value calculation unit 13.

[0021] A physically based model models not only the shape of an object, but also the reflection, refraction, attenuation, scattering, etc. of light on the surface and inside of the object. The virtual camera parameters are information such as the position, orientation, and angle of view of a virtual camera that virtually captures an image of an object described in a physics-based model.

[0022] The first virtual lighting is information about light illuminating an object described in the physically-based model. For example, the first virtual lighting includes the number of light sources, the shape of the light-emitting portion of the light source (e.g., point light source, line light source, or surface light source), the color or spectral characteristics of the light source, the light distribution characteristics of the light source, the optical axis direction of the light source, and the arrangement of the light source. The first virtual lighting may also be expressed in accordance with specific lighting devices such as ambient light, directional light, area light, point light, or spot light. Furthermore, the first virtual lighting may also be expressed in accordance with global rendering of lighting, such as global illumination.

[0023] Here, the first physics-based rendering unit 10 can use known physics-based rendering. For example, physics-based rendering can be expressed using parameters or maps such as albedo, specular reflectance, roughness, and microfacets. Alternatively, physics-based rendering can be used to approximate the reflectance characteristics of an object surface using a relatively small number of parameters, such as Lambert shading or Phong shading.

[0024] The second physically-based rendering unit 11 generates a second physically-based CG (second physically-based CG image) by applying physically-based rendering to the physically-based model under second virtual lighting (second virtual lighting conditions) different from the first virtual lighting. In this embodiment, the second physically-based rendering unit 11 receives the physically-based model, the second virtual lighting, and virtual camera parameters, and performs physically-based rendering on the physically-based model based on the input second virtual lighting and virtual camera parameters to generate a second physically-based CG. The second physically-based rendering unit 11 then outputs the generated second physically-based CG to the pixel value calculation unit 13.

[0025] Here, information about the light illuminating an object (virtual lighting) differs between the first physics-based rendering unit 10 and the second physics-based rendering unit 11. That is, the first physics-based rendering unit 10 and the second physics-based rendering unit 11 apply the same physics-based rendering to the same physics-based model using the same virtual camera parameters.

[0026] In this embodiment, the first virtual illumination is the illumination condition in the output CG (output CG image) that is ultimately desired to be generated, that is, the illumination condition of the output CG that the rendering device 1 ultimately outputs. The second virtual lighting is an approximation of the lighting conditions when the image-based model described below was generated. For example, if the image-based model was generated by illuminating the subject from various azimuths and elevation angles using multiple lighting devices, the second virtual lighting may be approximated as ambient light. Also, if the image-based model was generated by illuminating a relatively small number of lighting devices (e.g., 10 or less), the second virtual lighting may be approximated by the same number and placement of area lights or point lights as the lighting devices. Note that both the first virtual lighting and the second virtual lighting are set manually, and the reasons for the differences between the two will be described later.

[0027] Note that "approximating" the second virtual illumination refers to approximating all or part of the number of light sources, the shape of the light-emitting portion of the light source (e.g., point light source, line light source, or surface light source), the color or spectral characteristics of the light source, the light distribution characteristics of the light source, the optical axis direction of the light source, the arrangement of the light source, etc. to the illumination conditions when the image-based model was generated. Naturally, "approximating" the second virtual illumination also includes making the illumination conditions the same as when the image-based model was generated.

[0028] The image-based rendering unit 12 applies image-based rendering to the image-based model to generate an image-based CG (image-based CG image). In this embodiment, the image-based rendering unit 12 receives the image-based model and virtual camera parameters, and performs image-based rendering on the image-based model based on the input virtual camera parameters to generate the image-based CG. The image-based rendering unit 12 then outputs the generated image-based CG to the pixel value calculation unit 13.

[0029] Here, the image-based rendering unit 12 can use known image-based rendering. For example, the image-based rendering unit 12 generates image-based CG by appropriately combining a method of transforming a real-life image based on virtual camera parameters, a method of synthesizing multiple real-life images (including transformed images) using blending, mosaicing, multiple-frame super-resolution, or the like, and a method of projecting a model generated based on real-life images.

[0030] The image-based model is a model based on actual images taken under multiple illumination lights, a shadowless lamp, or a planar illumination light. The above-mentioned methods of transforming and synthesizing real-life images, the method of projecting an image-based model generated from the real-life images, and the pixel value information or texture information extracted from the real-life images are called image-based CG.

[0031] For example, the image-based rendering unit 12 may use an image-based model including a texture map that associates coordinates on a shape model such as a polygon with image coordinates of a real-life image, and the shape model. In this case, the real-life image may be taken from a single viewpoint, or a sequence of images taken from multiple viewpoints may be used, and the images to be used may be changed depending on the location on the shape model.

[0032] The image-based rendering unit 12 may also generate image-based CG by using, for example, a point cloud and assigning pixel values ​​of a real-life image to each point that constitutes the point cloud. In this case, the real-life image may be an image taken from a single viewpoint, or a sequence of images taken from multiple viewpoints may be used, and the images used may be changed depending on the points that constitute the point cloud.

[0033] The image-based rendering unit 12 may also generate an image-based CG image of an intermediate viewpoint by interpolating an image sequence captured from multiple viewpoints. In this case, the pixel value calculation unit 13 (described later) may perform local correspondence between the images constituting the image sequence and multiply this correspondence vector by a predetermined factor to determine the pixel value of the intermediate viewpoint.

[0034] Furthermore, the image-based rendering unit 12 may use a sequence of images captured from multiple viewpoints to perform local correspondence between two or more images, and may identify the three-dimensional coordinates of the points for which the local correspondence has been performed by triangulating based on the camera parameters of the cameras that captured each of the two or more images. In this case, the pixel value calculation unit 13 may project the identified three-dimensional coordinates using desired camera parameters to identify corresponding pixel coordinates in the output CG. Furthermore, the pixel values ​​of the corresponding points in one or more images among the two or more images used to determine the identified three-dimensional coordinates may be used (for example, by averaging the pixel values).

[0035] The pixel value calculation unit 13 calculates the pixel values ​​of corresponding pixels in the output CG from the pixel values ​​of corresponding pixels in the first physics-based CG input from the first physics-based rendering unit 10, the second physics-based CG input from the second physics-based rendering unit 11, and the image-based CG image input from the image-based rendering unit 12.

[0036] <Calculation by pixel value calculation unit> The calculation performed by the pixel value calculation unit 13 will be described in detail below. Pixels at the same image coordinates (x, y) in the first physics-based CG, second physics-based CG, image-based CG, and output CG are defined as corresponding pixels. The pixel value of the corresponding pixel in the first physics-based CG is P(x, y), the pixel value of the corresponding pixel in the second physics-based CG is Q(x, y), the pixel value of the corresponding pixel in the image-based CG is R(x, y), and the pixel value of the corresponding pixel in the output CG is S(x, y). The pixel values ​​may be scalars (e.g., grayscale images) or vectors (e.g., color images consisting of red, green, and blue components).

[0037] Here, the pixel value calculation unit 13 calculates the pixel value of the corresponding pixel of the output CG using the pixel value P of the corresponding pixel of the first physical base CG, the pixel value Q of the corresponding pixel of the second physical base CG, and the pixel value R of the corresponding pixel of the image base CG, using the binomial operation μ as shown in the following equation (1).

[0038]

number

[0039] For example, the pixel value calculation unit 13 performs the conversion of the following equation (2) on the binary operator μ and the function f to calculate the pixel value S(x, y) of the corresponding pixel in the output CG.

[0040]

number

[0041] In addition, Q -1 is the inverse of Q under the binary operation μ. -1 means the reciprocal of each vector component when the binary operation μ is the Hadamard product and Q is a D-dimensional vector (where D is an integer greater than or equal to 1). That is, Q -1 is expressed by the following formula (3). In particular, when D=1, that is, when the binary operation μ is a modulo operation and Q is a scalar, Q -1 On the other hand, if the binary operation μ is addition or subtraction, then Q -1 means Q.

[0042]

number

[0043] The function f(v) is independent for each component as shown in the following equation (4), and the function f d Assume that (v) is strictly monotonically increasing with respect to v (d is an integer between 1 and D).

[0044]

number

[0045] Furthermore, the function f(v) may be expressed as the following equation (5): In this case, equation (1) is expressed as the following equation (6).

[0046]

number

number

[0047] Here, the binomial operation μ in equation (1) can be performed using the addition and subtraction of equation (7) or the modulus operation of equation (9). For example, the pixel value calculation unit 13 calculates the pixel value of the corresponding pixel in the output CG using the binomial operation μ, which is addition and subtraction, as shown in the following equation (7). That is, the pixel value calculation unit 13 performs the calculation expressed by the following equation (8) to calculate the pixel value S(x, y) of the corresponding pixel in the output CG.

[0048]

number

number

[0049] Furthermore, the pixel value calculation unit 13 may calculate the pixel value of the corresponding pixel of the output CG using the binomial operation μ, which is a modulo operation, as shown in the following equation (9) for pixels and color components whose Q components are not zero. That is, the pixel value calculation unit 13 performs the calculation expressed by the following equation (10) to calculate the pixel value S(x, y) of the corresponding pixel of the output CG.

[0050]

number

number

[0051] Here, the d-th components of P, Q, R and S are respectively P d , Q d , R d and S d Then, the pixel value S is calculated by the following equation (11): d Calculate the following.

[0052]

number

[0053] Exceptionally, Q d For the component (x, y)=0, the pixel value S is calculated using one of the following equations (12) to (15). d (where m and n are positive integers).

[0054]

number

[0055] <Principles of rendering devices> The principle of the rendering device 1 will be described with reference to FIGS. Here, an example will be described in which CG rendering of a patterned sphere (e.g., a billiard ball) is performed. The accuracy of the physics-based rendering performed by first physics-based rendering unit 10 and second physics-based rendering unit 11 is inferior to the accuracy of the image-based rendering performed by image-based rendering unit 12. When generating the image-based model, the subject is illuminated with uniform light from all around.

[0056] Figure 2(a) shows the first physically based CG. In this first physically based CG, the first physically based rendering produces an image with shadows and a glossy appearance, but the image is blurred due to the lack of precision of the physically based model.

[0057] Figure 2(b) shows the second physically-based CG. In this second physically-based CG, virtual lighting was used, shining uniform light from all around the subject, to approximate the lighting used when the image-based model was acquired. As a result, the second physically-based CG renders an image without shadows or gloss, revealing only the subject's pattern. Additionally, the second physically-based CG is blurred due to the lack of precision in the physically-based model.

[0058] Figure 2(c) shows an image-based CG. This image-based CG is generated from a real-life image taken by illuminating the subject with uniform light from all around. Image-based CG renders images without shadows or glossiness, but as mentioned above, the image-based model is more accurate than the physically-based model, so a high sense of detail can be achieved.

[0059] Figure 2(d) shows the output CG calculated using equation (6) for the GCs in Figures 2(a) to 2(c). This output CG is of high quality, with shading, gloss, and no blurring. This output CG is an approximation of the CG shown in Figure 2(e) that would be obtained if the physics-based model were sufficiently accurate.

[0060] As shown in Figure 3, the first physics-based CG reproduces the shading of the output CG under the desired lighting, but the degradation of the texture shape due to shape errors is significant. The image-based CG reduces the degradation of the texture shape due to shape errors, but reproduces the shading of the image-based model. The second physics-based CG reflects the shading of the image-based model, and the degradation of the texture shape due to shape errors is significant.

[0061] Therefore, in the rendering device 1, by adding the first physically-based CG and the image-based CG and subtracting the second physically-based CG, an output CG is obtained that reproduces the shadows under the desired lighting and has little deterioration of the texture shape due to shape errors. As described above, the rendering device 1 makes it possible to generate CG that combines texture and detail even when the performance of physically-based rendering is inferior to the performance of image-based rendering.

[0062] [Example of rendering device operation] An example of the operation of the rendering device 1 will be described with reference to FIG. As shown in FIG. 4, in step S1, the first physically-based rendering unit 10 generates a first physically-based CG by applying physically-based rendering to a physically-based model using first virtual lighting. In step S2, second physically-based rendering unit 11 generates a second physically-based CG by applying physically-based rendering to the physically-based model with second virtual lighting. In step S3, the image-based rendering unit 12 applies image-based rendering to the image-based model to generate image-based CG. The processes in steps S1 to S3 can be executed in parallel.

[0063] In step S4, the pixel value calculation unit 13 calculates the pixel values ​​of corresponding pixels of the output CG from the pixel values ​​of corresponding pixels of the first physical base CG in step S1, the second physical base CG in step S2, and the image base CG in step S3.

[0064] [Actions and Effects] As described above, the rendering device 1 calculates pixel value components that cannot be fully reproduced by the first physical rendering through calculations between the image-based CG and the second physically-based CG, and complements them in the first physically-based CG. This enables the rendering device 1 to generate natural CG images even from models that are not highly accurate. Furthermore, the rendering device 1 can obtain realistic output CG even if the accuracy of the physically based model is reduced, so that it is possible to reduce the number of devices and processing time required for modeling, and the amount of data for the model.

[0065] That is, the rendering device 1 complements pixel value components that cannot be fully reproduced in the first physically-based CG with the difference components between the image-based CG and the second physically-based CG. As a result, when the image-based CG has finer texture components than the second physically-based CG, the rendering device 1 can complement the difference components in the first physically-based CG, improving the sense of detail.

[0066] Furthermore, the rendering device 1 uses subtraction to calculate the difference component and addition to interpolate. In this case, the rendering device 1 has low computational cost for the difference calculation, which allows for improvements in the device size and processing speed.

[0067] Furthermore, the rendering device 1 uses division to calculate the difference component and multiplication to interpolate. In this case, since the effects of reflection on the surface of an object and scattering, reflection, and transmission inside the object are generally applied multiplicatively, the rendering device 1 enables physically more accurate interpolation by performing multiplication and division in the pixel value calculation unit.

[0068] Furthermore, by approximating the second virtual lighting to the lighting environment at the time of generating the image-based model, the rendering device 1 can approximate the shading of the second physically-based CG to the shading of the image-based CG, thereby eliminating the influence of shading in the interpolation. As a result, the rendering device 1 can specialize the interpolated information to information related to the sense of detail, such as the presence or absence of detailed structures.

[0069] Furthermore, the rendering device 1 uses an image-based model based on real-life images captured under multiple illuminations, a shadowless lamp, or planar illumination, making it possible to generate image-based CG with reduced influence from illuminations. In this case, the pixel value calculation unit 13 performs calculations between the image-based CG with reduced influence from illuminations and a second physically-based CG with shading similar to that of the image-based CG, thereby enabling highly accurate interpolation that eliminates adverse effects dependent on illuminations and improving the image quality of the output CG.

[0070] (Variation) Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention.

[0071] In the above embodiment, the first virtual lighting is the lighting condition in the output CG, and the second virtual lighting is an approximation of the lighting condition when the image-based model was generated. However, this is not limited to this. In other words, as long as the second virtual lighting is different from the first virtual lighting, both lighting conditions can be set arbitrarily. For example, by intentionally changing the second virtual lighting, special visual effects such as edge emphasis and strong gloss can be obtained.

[0072] In the above embodiment, the first physically-based rendering unit and the second physically-based rendering unit are described as being independent, but they may also be a common unit. That is, they may be implemented as a common circuit or program, and physically-based rendering may be performed by switching between the first virtual lighting and the second virtual lighting at different times, with the results being the first physically-based CG and the second physically-based CG.

[0073] In the above-described embodiment, the rendering device is described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to function as the rendering device. This program may be distributed via a communication line or written to a recording medium such as a CD-ROM or flash memory. [Explanation of symbols]

[0074] 1. Rendering Device 10. First Physically Based Rendering Unit 11 Second Physically Based Rendering Unit 12 Image-based rendering section 13 Pixel value calculation unit

Claims

1. 1. A rendering device that renders an output CG image from a model of a subject using physically based rendering and image based rendering, comprising: a first physically-based rendering unit that generates a first physically-based CG image by applying the physically-based rendering to a physically-based model under a predetermined first virtual lighting condition; a second physically-based rendering unit that generates a second physically-based CG image by applying the physically-based rendering to the physically-based model under a second virtual lighting condition different from the first virtual lighting condition; an image-based rendering unit that generates an image-based CG image by applying the image-based rendering to an image-based model; a pixel value calculation unit that calculates pixel values ​​of corresponding pixels in the output CG image from pixel values ​​of corresponding pixels in the first physically-based CG image, the second physically-based CG image, and the image-based CG image; A rendering device comprising:

2. the first virtual lighting condition is a lighting condition in the output CG image, 2. The rendering apparatus of claim 1, wherein the second virtual lighting conditions approximate lighting conditions under which the image-based model was generated.

3. The pixel value calculation unit calculates a pixel value P of a corresponding pixel in the first physical-based CG image, a pixel value Q of a corresponding pixel in the second physical-based CG image, and a pixel value R of a corresponding pixel in the image-based CG image using a binomial operation μ as shown in the following equation (1) (where Q －１ is the inverse of Q in the binary operation μ), [Equation 1] 3. The rendering device according to claim 1, further comprising: a calculation unit for calculating pixel values ​​of corresponding pixels in the output CG image.

4. The pixel value calculation unit uses the binomial operation μ, which is addition and subtraction, to calculate, as shown in the following equation (2): [Equation 2] 4. The rendering device according to claim 3, wherein pixel values ​​of corresponding pixels in the output CG image are calculated.

5. The pixel value calculation unit calculates, for pixels and color components whose Q components are not zero, the binomial operation μ, which is a modular operation, as shown in the following equation (3): [Equation 3] 4. The rendering device according to claim 3, wherein pixel values ​​of corresponding pixels in the output CG image are calculated.

6. 6. The rendering device according to claim 1, wherein the image-based model is a model based on actual images captured using a plurality of illumination lights, a shadowless lamp, or a planar illumination light.

7. A program for causing a computer to function as the rendering device according to any one of claims 1 to 6.

Images