Image processing apparatus, object data generation apparatus, image processing method, and object data generation method

Abstract

This image processing device places an object 120 in a virtual space 136 to be displayed, and sets a view screen 130 corresponding to a viewpoint. The image processing device tracks a ray passing through pixels 132 included in an image of the object 120, and uses a position P and direction ω of incidence of the ray into the object 120 to read out a distribution of degrees of light source contribution that represent the magnitudes of influence on the image color of the object by a plurality of light sources set in different directions with respect to the object, the distribution of degrees of contribution being acquired in advance. The image processing device determines the pixel values of the pixels 132 by calculating a weighted sum of luminances of light from in-scene light sources 134a, 134b, and 134c with the degrees of light source contribution obtained with respect to the corresponding directions serving as weights.

Description

Technical Field

[0001] This invention relates to an image processing apparatus that generates a display image, an object data generation apparatus that generates data related to an object to be displayed, an image processing method and an object data generation method using the apparatuses, and a data structure of an object model.

Background Art

[0002] With the recent improvement in information processing technology and image display technology, it has become possible to experience the video world in various forms. For example, by displaying a panoramic video on a head-mounted display and displaying an image corresponding to the user's line of sight, it is possible to enhance the sense of immersion in the video world or improve the operability of applications such as games. In addition, by displaying image data streamed from a server having abundant resources, users can enjoy high-definition moving images and game screens regardless of location and scale.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Regardless of the purpose and display format of image display, how to efficiently draw and display an image is always an important issue. For example, in a mode where the viewpoint and line of sight have degrees of freedom and a three-dimensional object is shown from various angles, high responsiveness is required for the change in display with respect to the movement of the viewpoint. The same applies when the three-dimensional object moves or deforms. On the other hand, to display a high-quality image, it is necessary to increase the resolution or perform complex calculations, which increases the load of image processing. As a result, a delay is likely to occur in the change of the image of the object that should originally be represented.

[0004] This invention has been made in view of such problems, and its object is to provide a technique for displaying an image including a three-dimensional object with low latency and high quality.

Means for Solving the Problems

[0005] In order to solve the above problems, an aspect of the present invention relates to an image processing apparatus. This image processing apparatus is an image processing apparatus that generates a display image of a space including an object, and stores a distribution of light source contributions representing the magnitude of the influence of a plurality of light sources in different directions on the object on the color of the image of the object, in association with the model data of the object, in an object model storage unit. When drawing an image of the object on the display image, a drawing unit determines the pixel value of the image by calculating a weighted sum of the luminance of the light reflected from the object by the in-scene light source set in the space, using the light source contribution obtained for the corresponding direction as a weight. An output unit outputs data of the display image including the image of the object. It is characterized by comprising these components.

[0006] Another aspect of the present invention relates to an object data generation apparatus. This object data generation apparatus is an object data generation apparatus that generates data related to an object for use in generating a display image. By repeatedly performing a sampling process a predetermined number of times, which arranges a plurality of light sources in different directions with respect to the object in a virtual space and traces rays from a virtual camera that observes the object to find the light source at the arrival destination, a light source contribution distribution acquisition unit obtains a distribution of light source contributions representing the magnitude of the influence of the plurality of light sources on the color of the image of the object. A light source contribution distribution data storage unit stores the distribution of light source contributions in association with the combination of the incident position and incident direction of the ray on the surface of the object. It is characterized by comprising these components.

[0007] Still another aspect of the present invention relates to an image processing method. The image processing method includes: a step of reading out, from a storage unit that stores, in association with model data of an object, a distribution of light source contributions representing the magnitude of the influence of a plurality of light sources in different directions on an object on the color of an image of the object, by an image processing apparatus that generates a display image of a space including the object; a step of determining a pixel value of the image by calculating a weighted sum of the luminance reflected from the object by an in-scene light source set in the space, using the light source contribution obtained for the corresponding direction as a weight, when drawing the image of the object on the display image; and a step of outputting data of the display image including the image of the object.

[0008] Still another aspect of the present invention relates to an object data generation method. The object data generation method includes: a step of obtaining a distribution of light source contributions representing the magnitude of the influence of a plurality of light sources on the color of an image of an object, by repeatedly performing a sampling process a predetermined number of times, in which a plurality of light sources are arranged in different directions with respect to the object in a virtual space and rays from a virtual camera that observes the object are traced to obtain the light source reached; and a step of storing the distribution of the light source contributions in a storage unit in association with a combination of an incident position and an incident direction of a ray on the surface of the object.

[0009] Yet another aspect of the present invention relates to a data structure of an object model. The data structure of this object model is a data structure of an object model used by an image processing apparatus to generate a display image including an object, and the image processing apparatus includes data representing the shape of an object arranged in the space to be displayed, and a distribution of light source contribution degrees representing the magnitude of the influence of a plurality of light sources in different directions on the color of the image of the object. When the image processing apparatus draws an image of the object, the weighted sum of the luminance reflected from the object by the in-scene light sources set in the space is calculated using the obtained light source contribution degree in the corresponding direction as a weight, and the distribution of the light source contribution degrees for determining the pixel value of the image is associated therewith.

[0010] In addition, any combination of the above components, and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a computer program, a data structure, a recording medium, etc. are also effective as aspects of the present invention.

Advantages of the Invention

[0011] According to the present invention, an image including a three-dimensional object can be displayed with low latency and high quality.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Embodiments for Carrying Out the Invention

[0013] FIG. 1 shows a configuration example of an image display system to which the present embodiment can be applied. The image display system 1 includes image processing apparatuses 10a, 10b, 10c that display images according to user operations, and a content server 20 that provides image data for display. Input devices 14a, 14b, 14c for user operations and display devices 16a, 16b, 16c for displaying images are connected to the image processing apparatuses 10a, 10b, 10c, respectively. The image processing apparatuses 10a, 10b, 10c and the content server 20 can establish communication via a network 8 such as a WAN (Wide Area Network) or a LAN (Local Area Network).

[0014] The image processing apparatuses 10a, 10b, 10c, the display devices 16a, 16b, 16c, and the input devices 14a, 14b, 14c may be connected either by wire or wirelessly. Alternatively, two or more of these devices may be integrally formed. For example, in the figure, the image processing apparatus 10b is connected to a head-mounted display that is the display device 16b. Since the head-mounted display can change the viewing field of the displayed image according to the movement of the user wearing it on the head, it also functions as the input device 14b.

[0015] Also, the image processing apparatus 10c is a mobile terminal and is integrally configured with the display device 16c and the input device 14c which is a touch pad covering its screen. Thus, the external shape and connection form of the illustrated devices are not limited. The number of the image processing apparatuses 10a, 10b, 10c and the content server 20 connected to the network 8 is also not limited. Hereinafter, the image processing apparatuses 10a, 10b, 10c are collectively referred to as the image processing apparatus 10, the input devices 14a, 14b, 14c are collectively referred to as the input device 14, and the display devices 16a, 16b, 16c are collectively referred to as the display device 16.

[0016] The input device 14 may be any one or a combination of common input devices such as a controller, a keyboard, a mouse, a touch pad, a joystick, etc., and supplies the content of the user operation to the image processing device 10. The display device 16 may be a common display such as a liquid crystal display, a plasma display, an organic EL display, a wearable display, a projector, etc., and displays the image output from the image processing device 10.

[0017] The content server 20 provides the image processing device 10 with the data of the content accompanied by image display. The type of the content is not particularly limited, and may be any of an electronic game, a viewing image, a web page, a video chat by an avatar, etc. The image processing device 10 may realize the display process while acquiring the content data used for display from the content server 20, or may read out the previously acquired content data from the internal storage device and use it for display. In the latter case, the source of the content data is not limited to the content server 20, and may be a recording medium.

[0018] In the present embodiment, at least a part of the display target is composed of 3D objects. The 3D object may be created manually or captured (measured) as long as a 3D model for drawing can be obtained. For example, the image processing device 10 displays how the object of the display target is viewed from a free angle in response to a user operation via the input device 14. Alternatively, the image processing device 10 may control the viewpoint and line of sight with respect to the object according to the progress of the game or the like. For example, the viewpoint may be defined as content data as in a movie.

[0019] The three-dimensional object may be stationary or moving. In any case, the image processing apparatus 10 generates a display image representing a space including the three-dimensional object at a predetermined rate, and causes the display device 16 to display the image. In the field of three-dimensional computer graphics, by more accurately representing the physical phenomena occurring in the space to be displayed, it has become possible to create a more immersive image representation. For example, in addition to the light from the light source, by accurately calculating the various light propagations reaching the virtual viewpoint, such as diffuse reflection and specular reflection on the object surface, it is possible to more realistically represent the changes in color and brightness due to the movement of the viewpoint and the object itself. Ray tracing is known as a physically based rendering that realizes this.

[0020] FIG. 2 is a diagram for explaining the outline of general ray tracing. First, it is assumed that in the virtual space 110 to be displayed, an object 108, light sources 116a, 116b, and 116c are arranged. When the viewpoint 100 for viewing the virtual space is determined, a view screen 102 is set at a position and posture corresponding to the position and the direction of the line of sight. The image of the object 108 in the display image is formed by the light emitted from the light sources 116a, 116b, and 116c, reflected or transmitted by the object 108, and reaching the view screen 102.

[0021] Ray tracing is a method of obtaining the apparent color of the surface of the object 108 by tracing the path of the light in the reverse direction and using it as the pixel value. For example, a virtual ray (light ray) 106 passing through the pixel 104 on the view screen 102 from the viewpoint 100 reaches the point 112 on the surface of the object 108. Generally, the luminance (radiance) L(r,ω) of the light reflected in the direction ω at the point r on the object surface is represented by the following rendering equation.

[0022]

Equation

[0023] Here, L e(r, ω) is the luminance of the light emitted by the object itself at point r. f r (r, ω, ω i ) is the bidirectional reflectance distribution function (BRDF: Bidirectional Reflection Distribution Function) representing the light reflection characteristics at point r, and the luminance L of the incident light in the incident direction ω i of the incident light, i (r, ω i ) represents the ratio of the reflected light in the direction ω with respect to it. This function depends on the material and surface roughness of the object. Also, Θ is the angle formed by the normal direction of the object surface and the reflection direction ω at point r, and S i is the range that the incident direction ω of the light at point r i can take.

[0024] In the illustrated environment, the rendering equation of Equation 1 represents the relationship between the intensity of the light reaching the pixel 104 where the ray is generated (left side) and the intensity of the incident light from the light sources 116a, 116b, and 116c to point 112 (right side). That is, the pixel value of pixel 104 can be determined using the rendering equation with point 112 on the object 108 as point r. However, since it is difficult to analytically solve the integral of S i , the incident direction ω i is sampled by the Monte Carlo method, and the integral value is statistically estimated.

[0025] Such a method is known as path tracing (see, for example, James T. Kajiya, "The rendering equation", ACM SIGGRAPH Computer Graphics, Volume 20, Issue 4, August 1986, pp. 132-150). The more the number of samplings is increased, the closer the obtained value is to the true integral value, and as a result, a high-quality image with less noise can be generated. However, there are time constraints for displaying the image rendered on the fly with low latency. Also, BRDF is a function that holds under the assumption that the incident position of light is sufficiently close to the emission position r. On the other hand, in a translucent object such as human skin, subsurface scattering (SSS) occurs where the incident light once enters the object and is scattered, and then exits from another position.

[0026] Substances where subsurface scattering occurs generally have a high density of the medium and go through a complex scattering process before exiting the surface again, making it difficult to accurately estimate. Therefore, a bidirectional scattering surface reflectance distribution function (BSSRDF) that models the emitted light using only the emission position and direction as parameters has been proposed (see, for example, Henrik Wann Jensen, et al., "A Practical Model for Subsurface Light Transport", (USA), Proceedings of the 28th annual conference on Computer graphics and interactive techniques, SIGGRAPH'2001, 2001, pp. 511-518).

[0027] Alternatively, a method has been proposed to pseudo-track subsurface scattering by means of a random walk (for example, "Arnold for maya user guide / subsurface", [online], 2021, Autodesk, [searched on October 27, 2021], Internet <URL:https: / / docs.arnoldrenderer.com / display / A5AFMUG / Subsurface>, Cedar 'zedar' T., "Physical Rendering with Random Walk Sub-Surface Scattering", [online], January 9, 2019, LinkedIn, [searched on October 27, 2021], Internet <URL:https: / / www.linkedin.com / pulse / physical-rendering-random-walk-sub-surface-scattering-zedar-thokme / refer to).

[0028] In any case, while physically-based rendering that reflects various phenomena in this way can achieve high realism, the more precision is pursued, the more complex the calculation becomes, resulting in a huge amount of required resources, or the display may be delayed in response to changes in the viewpoint or object. Therefore, in the present embodiment, by acquiring in advance data representing how much the light source in each direction affects the image of the object, it is possible to generate a high-quality image while reducing the amount of calculation during operation. Hereinafter, the distribution of the magnitude of the influence on the image of the object depending on the direction of the light source is referred to as the "light source contribution distribution".

[0029] FIG. 3 is a diagram for explaining a method of obtaining a light source contribution degree distribution in the present embodiment. First, in a virtual space, a plurality of virtual light sources (for example, light sources 122a, 122b, and 122c) are arranged in different directions with respect to the object 120. In the illustrated example, a large number of light sources are arranged at equal intervals on the surface of a sphere 124 having a predetermined radius centered on the center of gravity of the object 120. However, the arrangement rule and the number of arranged light sources are not limited. Hereinafter, the light sources arranged for obtaining the light source contribution degree distribution are referred to as "light sources for obtaining contribution degree", and the light sources set in the space of the display target during operation are referred to as "light sources in the scene".

[0030] Naturally, the object 120 is an object to be displayed by means as shown in FIG. 2 during operation, and information necessary for general computer graphics drawing such as shape, color, and material is set as an object model. For the virtual space composed of such an object 120 and the light sources for obtaining contribution degree, a virtual camera 126 for observing the object 120 is set. In the present embodiment, the virtual observation subject when obtaining the light source contribution degree distribution is referred to as a "virtual camera", which is distinguished from the "viewpoint" for the display image during operation.

[0031] The light source contribution degree distribution is a statistically estimated ratio of the light emitted from each light source for obtaining contribution degree among the light that reaches the virtual camera 126 through the object 120, and the calculation model is similar to the general ray tracing described in FIG. 2. That is, also in obtaining the light source contribution degree distribution, rays are generated for each pixel corresponding to the sensor of the virtual camera 126, and the paths thereof are traced. One of the rays 128 is shown in the figure.

[0032] By obtaining the light source contribution distribution in advance at a timing different from the operation stage of actually generating the display image, calculations of various physical phenomena such as subsurface scattering can be performed in detail over time. In this way, the ray path is traced with high accuracy, and when a ray reaches any of the light sources for contribution acquisition, one count is given to that light source. By repeating this sampling process a predetermined number of times and increasing the count of the light sources reached by the rays, the count finally given to each light source is proportional to the ray arrival probability.

[0033] Since the object 120 and the virtual camera 126 are fixed, the point where the ray 128 first reaches the surface of the object 120, that is, the incident position and its incident direction, are the same regardless of the number of sampling times. Here, let the incident position on the object 120 be P and the incident direction be ω. Among the images of the object 120 seen from the virtual camera 126, it can be said that the magnitude of the influence (contribution) on the color at the position of P is greater for the light sources with a higher probability of the ray 128 finally reaching.

[0034] Let the count representing the number of times the ray 128 reaches the i-th light source be n i , and when the total number of samplings is N, the arrival probability of the ray 128 to the i-th light source is n i / N, and simply this value is taken as the contribution w i to the color of the image. Alternatively, as will be described later, the reflectance and attenuation rate at the object 120 etc. in the process until the ray 128 reaches the light source may be reflected in the contribution w i . In either case, the contribution distribution W of the I light sources from 1 to I is W = {w 1 , w 2 , w 3 , ··· w i , ··· w I} represented by a sequence. This sequence is the distribution of the contribution to the color of the image at the position P, obtained for each pair of the position P on the surface of the object 120 and the incident direction ω of the ray.

[0035] The sequence representing the light source contribution distribution W is obtained by varying various combinations of the position P on the surface of the object 120 and the incident direction ω of the ray. Hereinafter, the combination of (P, ω) may be referred to as the "incident parameter". For example, if the virtual camera 126 is fixed and rays are generated for each pixel, the incident parameters corresponding to the number of pixels, and thus the light source contribution distributions corresponding thereto, can be obtained. Further, by changing the direction (the direction of the optical axis) of the virtual camera 126 with respect to the object 120 to obtain the light source contribution distribution, the incident parameters, and thus the light source contribution distributions corresponding thereto, can be obtained in a number equal to the product of the number of pixels and the number of directions set for the virtual camera 126 as described above.

[0036] FIG. 4 illustrates the data structure of the light source contribution distribution. As shown in the figure, the light source contribution distribution data 80 is data that associates the incident parameter (P, ω) with the sequence representing the light source contribution distribution W. The incident parameter (P, ω) is obtained in a number equal to the product of the number of pixels and the number of directions set for the virtual camera 126 as described above. Note that either the incident position P or the incident direction ω may overlap.

[0037] The light source contribution distribution is a sequence representing the respective contributions w of the light sources for obtaining contributions for each incident parameter. i The light source contribution distribution data 80 is stored in a storage device in association with the model data of the object 120 or as part of the model data, and is used for drawing the object during display. When the object 120 is deformed, similar contribution distribution data may be obtained for each state, and appropriate data may be selected and used according to the shape of the object during display.

[0038] FIG. 5 is a diagram for explaining a method of drawing an image of an object using data of a light source contribution distribution. First, in a virtual space 136 to be displayed, an object 120, in-scene light sources 134a, 134b, and 134c are arranged. A view screen 130 is set corresponding to a viewpoint with respect to this, and a position P where a ray generated for each pixel first reaches the surface of the object 120 and its incident direction ω are acquired as actual incident parameters. Then, referring to the data of the light source contribution distribution, a light source contribution distribution that has been acquired for the same or similar incident parameters is obtained.

[0039] The light source contribution distribution shows, as a probability, the characteristics of the subsequent behavior of the ray that has reached the position on the surface of the object 120. That is, since a part of the processing that is computationally intensive, such as sampling for estimating the integral value in the rendering equation of Equation 1 or simulation of subsurface scattering, is performed in advance, the processing can be lightened by using the result. Specifically, the weighted sum of the luminances of the light from the in-scene light sources 134a, 134b, and 134c is obtained with the contribution w i obtained for the contribution acquisition light sources 138a, 138b, and 138c in the same direction as the weight.

[0040] Here, the luminance of the light from the in-scene light sources is the luminance of the light emitted from each in-scene light source and reflected on the surface of the object 120. As shown in the following formula, the weighted sum is the luminance L(P, ω) of the light emitted in the reverse direction of the direction ω at the position P on the surface of the object 120.

[0041]

Equation

[0042] Here, L iis the luminance of the in-scene light source in the direction of the i-th light source among the light sources for luminance acquisition, and has the values of the three primary colors of R (red), G (green), and B (blue). Note that the summation symbol means the sum of all in-scene light sources. Also, ρ(P) (0 < ρ < 1.0) is the reflectance at the position P on the surface of the object 120, and determines the color of the object 120 itself according to the RGB components. By performing the calculation of Equation 2 for each RGB channel, the pixel value corresponding to the ray (for example, pixel 132) is obtained.

[0043] Accordingly, the luminance L of the in-scene light sources 134a, 134b, and 134c i allows the pixel value representing the image at the position P to be obtained by accurate and simple calculations. For example, if the luminance L i of a certain light source 134b is extremely stronger than others, an image in which the influence of the light source 134b is greatly reflected can be obtained even if its contribution degree w i is small. Also, when the light source 134b is a red light, the luminance of the R channel increases, so the image also becomes reddish. Even if the brightness and color of the in-scene light sources 134a, 134b, and 134c change over time, the color of the image can be immediately reflected. Thus, in this embodiment, by obtaining only the contribution degree w j in advance with a fine granularity, easy application is possible regardless of the state and changes of the in-scene light sources.

[0044] In the examples described so far, it was assumed that the light from all the in-scene light sources 134a, 134b, and 134c is reflected only once on the surface of the object 120 and reaches the view screen 130. In this case, since the luminance is reduced at the same reflectance regardless of the in-scene light sources, the arrival probability n i / N of the ray to the light source for contribution degree acquisition is taken as the contribution degree w i , and an accurate pixel value can be obtained by multiplying the reflectance ρ(P) uniformly as in Equation 2 during operation. On the other hand, considering the case where multiple reflections occur due to the presence of the object 120 itself or other objects, since the reduction ratio of the luminance due to reflection varies depending on the ray path, it is considered that the error becomes large in a uniform calculation.

[0045] FIG. 6 is a diagram for explaining a method of obtaining a light source contribution degree distribution considering multiple reflections. The figure shows an object 120, light sources 122a, 122b, 122c for obtaining contribution degrees, and a virtual camera 126 at the stage of obtaining the light source contribution degree distribution. In actuality, a large number of light sources for obtaining contribution degrees may be arranged around the object 120 as shown in FIG. 3. Also in this figure, the position where the ray 128 from the virtual camera 126 is reflected is represented by a black rectangle (for example, rectangle 139). As shown by the arrow, the ray 128 from the virtual camera 126 first reflects at the same position P on the object 120, and then, while reflecting variously by sampling, a part of it reaches one of the light sources 122a, 122b, 122c for obtaining contribution degrees.

[0046] In addition to occurring at other locations on the object 120, when multiple objects are set at the stage of obtaining the light source contribution degree distribution, reflections can also occur on an object different from the object 120. The tracing of such a ray path itself can be realized by a method similar to general ray tracing. On the other hand, focusing on the contribution degree, if, for example, more reflections occur probabilistically in the path to a certain light source, or if the possibility of reflection with a low reflectivity material is high, even if the arrival probability of the ray to that light source is high, the contribution degree to the color of the image is considered not to be so high.

[0047] Based on this, in the procedure described in FIG. 3, each time a ray reaches one of the light sources for obtaining contribution degrees, the reduction in luminance due to reflection in that path is reflected in the count given. Specifically, the product of the reflectivities for the number of reflections is calculated as the remaining ratio of luminance. For example, in FIG. 6, the ray reaching the light source 122a for obtaining contribution degrees reflects at three points with reflectivities ρ(1), ρ(2), ρ(3) respectively after reflecting at position P. At this time, instead of the above-mentioned 1 count, the remaining ratio ρ(1)*ρ(2)*ρ(3) is given to the light source 122a for obtaining contribution degrees that the ray has reached.

[0048] By repeating the same sampling and finally dividing the sum of the counts given to each light source by the total number of samplings N, the contribution degree w i is obtained. Among the n i rays reaching the i-th light source, the number of reflections other than at position P in the path of the L-th ray is A L , and the reflectance at the k-th reflection among them is ρ L (k). Then, the contribution degree w i of the i-th light source is expressed as follows.

[0049]

Equation

[0050] According to this definition, the greater the rate of decrease in luminance due to reflection in the path of each ray, the smaller the contribution degree w i of the i-th light source at its arrival destination. Also, similar to the above, the greater the number of times the ray reaches the light source, the greater the contribution degree w i . Since the reflectance ρ L (k) has RGB values, the contribution degree w i expressed by Equation 3 also has RGB channels. Note that by not including the reflection at position P in the calculation of the contribution degree w i , Equation 2 can be used to determine the pixel value during display image generation regardless of the number of reflections. By applying the reflectance ρ(P) at position P during display image generation, the detailed shape of the object 120 such as the texture can be reflected in the image with high accuracy.

[0051] When calculating the scattering of rays at the surface of an object by means of random walk or the like, the degree of attenuation of luminance due to the scattering may be estimated and reflected in the contribution degree. The color of the object itself is determined by the fact that the attenuation rate of the amount of light due to the interaction with the material particles depends on the wavelength band. By separately tracking the RGB rays and obtaining the degree of attenuation corresponding to the color and material of the object for each, the contribution degree of each light source taking into account the surface scattering can be obtained for each primary color. In this case, the attenuation rate of luminance due to a single collision with the material particles on the object surface is used. The attenuation rate depends on the color and material of the object and has RGB components.

[0052] When the ray 128 from the virtual camera 126 is scattered at the object surface, the greater the light attenuation ratio, the smaller the count given to the finally reached light source. Here, the light attenuation ratio increases as the number of collisions with the material particles increases and as the attenuation rate due to a single collision increases. In other words, the smaller these values are, the greater the remaining ratio of luminance. For example, if the attenuation rate of luminance due to a single collision of light of a certain color is γ (0 < γ < 1.0) and it is emitted from the object surface after C collisions, the remaining ratio (1 - γ) C is multiplied by the count. Since this value corresponds to the reflectance in a single reflection, it can be incorporated instead of the reflectance ρ L (k) in the total product of Equation 3.

[0053] Contribution degree w iRegardless of the definition, as shown in FIG. 3, by densely arranging the light sources for obtaining contribution degrees around the object 120, a corresponding light source for obtaining contribution degrees exists regardless of the state of the light sources in the scene, and thus the accuracy of the contribution degree distribution to be read can be improved. Also, the virtual camera is set to observe the object 120 from as many directions as possible to obtain the light source contribution degree distribution. Thus, regardless of the viewpoint and line of sight with respect to the display image, the contribution degree distribution corresponding to the ray can be read with high accuracy. As an example, it is conceivable to arrange 331 light sources for obtaining contribution degrees around the entire circumference of the object 120 and set the virtual camera to observe the center from 90 directions.

[0054] FIG. 7 shows the internal circuit configuration of the image processing apparatus 10. The image processing apparatus 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are interconnected via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes a communication unit 32 composed of a peripheral device interface such as USB or IEEE1394 and a network interface for wired or wireless LAN, a storage unit 34 such as a hard disk drive or a non-volatile memory, an output unit 36 for outputting data to the display device 16, an input unit 38 for inputting data from the input device 14, and a recording medium drive unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0055] The CPU 22 controls the entire image processing apparatus 10 by executing the operating system stored in the storage unit 34. The CPU 22 also executes various programs read from a removable recording medium and loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has the functions of a geometry engine and a rendering processor, performs rendering processing according to a rendering command from the CPU 22, and stores the display image in a frame buffer (not shown). Then, the display image stored in the frame buffer is converted into a video signal and output to the output unit 36. The main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing.

[0056] FIG. 8 shows the configuration of the functional blocks of the image processing apparatus 10 according to the present embodiment. As described above, the image processing apparatus 10 may perform general information processing such as advancing an electronic game or communicating with the content server 20. In FIG. 8, particular attention is paid to the function of generating a display image. Note that at least a part of the functions of the image processing apparatus 10 shown in FIG. 8 may be implemented in a display device 16 such as a head-mounted display. Alternatively, at least a part of the functions of the image processing apparatus 10 may be implemented in the content server 20.

[0057] Also, the functional blocks shown in FIG. 8 and FIG. 9 described later can be realized in terms of hardware by the configuration of the CPU, GPU, various memories, etc. shown in FIG. 7, and in terms of software, by a program that exhibits functions such as a data input function, a data holding function, an image processing function, and a communication function, which are loaded from a recording medium or the like into a memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0058] The image processing apparatus 10 includes a viewpoint information acquisition unit 50 that acquires information related to a viewpoint with respect to a display image, a space construction unit 54 that constructs a space composed of objects to be displayed, a drawing unit 52 that draws the display image, an object model storage unit 56 that stores data of an object model used for drawing, and an output unit 62 that outputs the data of the display image to a display device 16.

[0059] The viewpoint information acquisition unit 50 acquires information related to a viewpoint with respect to the space to be displayed. For example, when the display device 16 is a head-mounted display, the output values of an acceleration sensor built in the head-mounted display are sequentially acquired, and thereby the posture of the user's head is acquired. Further, a light-emitting marker (not shown) is provided outside the head-mounted display, and the captured image thereof is acquired from an imaging device (not shown), thereby acquiring the position of the head in the real space.

[0060] Alternatively, an imaging device (not shown) for capturing an image corresponding to the user's field of view may be provided on the head-mounted display side, and the position and posture of the head may be acquired by a technique such as SLAM (Simultaneous Localization and Mapping). If the position and posture of the head can be acquired in this way, the position of the user's viewpoint and the direction of the line of sight can be approximately specified. When the display device 16 is a flat panel display or the like and the viewpoint and the line of sight are defined in advance, the viewpoint information acquisition unit 50 acquires the information from a content program or the like.

[0061] The space construction unit 54 arranges objects in the space to be displayed and operates the objects as necessary according to user operations or programs. For example, the space construction unit 54 arranges an object corresponding to a place that is the stage of the game to be displayed as a background, and then makes an object representing a game character appear. At this time, the user may be able to control the operation of the object via the input device 14. The space construction unit 54 also arranges in-scene light sources according to a program or the like.

[0062] The drawing unit 52 sets a view screen corresponding to the viewpoint and line of sight acquired by the viewpoint information acquisition unit 50, and generates an image representing the state of the space of the display target constructed by the space construction unit 54 at a predetermined rate. The object model storage unit 56 stores model data of objects used when the space construction unit 54 constructs a space or when the drawing unit 52 generates a display image. The model data may be general data that defines the shape, material, color, etc. of the object.

[0063] However, in the present embodiment, the object model storage unit 56 stores the light source contribution degree distribution data 60 for at least a part of the objects to be displayed. As described above, the light source contribution degree distribution data 60 represents the distribution of the contribution degrees of a plurality of light sources that irradiate light on the object from different directions to the color of the image of the object. The color of the image of the object depends on the position in the image and the direction of observing it. As a result, as described above, the light source contribution degree distribution is acquired for each incident parameter (P, ω) of the ray from the observation point. Since the light source contribution degree distribution is object-specific data, it is stored in association with the data of a general object model.

[0064] The drawing unit 52 includes a light source contribution processing unit 58 that uses the light source contribution degree distribution data 60 to draw an image for the object associated with the data. Specifically, as described with reference to FIG. 5, when a ray from the view screen reaches the object, the light source contribution degree distribution corresponding to the incident parameter is read from the object model storage unit 56 by the light source contribution processing unit 58. Then, the pixel value is determined by calculating the weighted sum of the luminance of the light from the light sources in the scene with the corresponding contribution degree as the weight for each channel of RGB.

[0065] For objects in the space representing the object for which the light source contribution distribution data 60 is not associated, the drawing unit 52 draws the image thereof by ordinary ray tracing or the like. When the display device 16 is a head-mounted display and the display image is to be viewed stereoscopically, the drawing unit 52 draws images for the viewpoints of the left eye and the right eye, respectively.

[0066] The output unit 62 sends the data of the display image generated by the drawing unit 52 to the display device 16 at a predetermined rate. When generating a stereo image for stereoscopic viewing, the output unit 62 generates and outputs an image obtained by connecting them left and right as the display image. In the case of a head-mounted display configured to view the display image through an eyepiece lens, the output unit 62 may perform correction on the display image in consideration of the distortion caused by the lens. As a result, a moving image in which the object is represented with high image quality is displayed on the display device 16 with low latency.

[0067] FIG. 9 shows a functional block of an apparatus for generating data of the light source contribution distribution. The object data generation apparatus 200 may be a part of the image processing apparatus 10 in FIG. 8 or the content server 20 in FIG. 1, or may be provided independently as an apparatus for generating data used for display. Further, the generated data of the light source contribution distribution, the object model used for generation, electronic content including a program of an electronic game, etc. may be stored in a recording medium or the like so that it can be loaded into the main memory in the image processing apparatus 10 during operation. The internal circuit configuration of the object data generation apparatus 200 may be the same as the internal circuit configuration of the image processing apparatus 10 shown in FIG. 7.

[0068] The object data generation apparatus 200 includes a condition setting unit 210 for setting conditions for obtaining the light source contribution distribution, an object model storage unit 214 for storing model data of the target object for obtaining the light source contribution distribution, a light source contribution distribution acquisition unit 216 for obtaining data of the light source contribution distribution for the target object, and a light source contribution distribution data storage unit 218 for storing the obtained data of the light source contribution distribution.

[0069] The condition setting unit 210 determines the number and arrangement of the contribution acquisition light sources, and controls the direction of the virtual camera with respect to the object. As described above, it is desirable for the condition setting unit 210 to arrange a predetermined number of contribution acquisition light sources so as to uniformly surround the object, and control the virtual camera to observe the object from uniform directions. However, depending on the configuration of the space to be actually displayed, there may be a bias in the positions of the contribution acquisition light sources and the direction of the virtual camera. For example, the distribution of the contribution acquisition light sources and the direction of the virtual camera may be limited according to the distribution of the light sources in the scene and the movable range of the viewpoint and line of sight.

[0070] Also, the number of contribution acquisition light sources and the number of settings of the virtual camera direction may be optimized according to the size of the object and the required fineness of the image. Qualitatively, the smaller the number of contribution acquisition light sources and the number of settings of the virtual camera direction, the smaller the size of the light source contribution distribution data can be. The condition setting unit 210 may perform these settings according to a predefined rule by itself, or may perform them by accepting a specified input from a content creator or the like.

[0071] The light source contribution distribution acquisition unit 216 arranges the contribution acquisition light sources around the object according to the settings of the condition setting unit 210, and then sets the virtual camera. Then, rays are generated for each pixel, and the contribution of each light source is obtained based on the arrival probability and the reduction ratio of the luminance due to the reflection and subsurface scattering that occur in the path. A general ray tracing method can be applied to the tracking of the generated rays. However, as described above, by simulating physical phenomena over time considering subsurface scattering and the like, and performing a large number of samplings at the arrival destination, the quality of the display image during operation can be improved.

[0072] The light source contribution distribution acquisition unit 216 can obtain the light source luminance distribution specific to an object by reflecting the shape, color, material, etc. of the object stored in the object model storage unit 214 in the calculation. The light source contribution distribution acquisition unit 216 sequentially obtains the light source contribution distribution by changing the direction of the virtual camera variously according to the setting of the condition setting unit 210. As a result, for the incident parameters (P, ω) corresponding to each of the generated rays, a light source contribution distribution W is generated. The light source contribution distribution acquisition unit 216 stores the acquired light source contribution distribution in the light source contribution distribution data storage unit 218 in association with the incident parameters.

[0073] That is, in the light source contribution distribution data storage unit 218, a plurality of data sets associating the incident parameters (P, ω) with the light source contribution distribution for each object are stored. The data is finally associated with the object model data of the object stored in the object model storage unit 214 and stored in the object model storage unit 56 of the image processing apparatus 10 shown in FIG. 8.

[0074] Next, the operation of the apparatus realized by the configuration described so far will be described. FIG. 10 is a flowchart showing the processing procedure for the image processing apparatus 10 to generate a display image in the present embodiment. This flowchart starts in a state where an application such as a game is started by a user operation or the like. Note that the image processing apparatus 10 may perform information processing specific to the application in parallel with the processing shown in the figure, and appropriately construct or change the space of the display target including the object. The space includes, for example, an object (target object) associated with the data of the light source contribution distribution, other objects, and in-scene light sources.

[0075] First, the viewpoint information acquisition unit 50 of the image processing apparatus 10 acquires information related to the viewpoint with respect to the display space, and the drawing unit 52 sets a corresponding view screen (S10). As described above, the viewpoint may be fixed, or may be changed according to the movement of the head-mounted display or the like. Then, the drawing unit 52 selects one pixel on the view screen, generates and tracks a ray passing through the pixel from the viewpoint (S12). When the ray reaches an object associated with the data of the light source contribution distribution (Y in S14), the light source contribution processing unit 58 reads out the light source contribution distribution corresponding to the incident parameter of the ray from the object model storage unit 56 (S16).

[0076] Then, the light source contribution processing unit 58 determines the pixel value by calculating the weighted sum of the RGB luminances representing the light from the light sources in the scene, using the corresponding contribution as the weight (S18). When the ray does not reach an object associated with the data of the light source contribution distribution (N in S14), the drawing unit 52 determines the pixel value by directly tracking the ray (S20). That is, the drawing unit 52 determines the pixel value by general ray tracing as described in FIG. 2.

[0077] The drawing unit 52 repeats the processing of S12 to S20 for all the pixels on the view screen (N in S22). After determining the pixel values of all the pixels (Y in S22), the output unit 62 outputs the data to the display device 16 as the data of the display image (S24). If there is no need to end the display (N in S26), the image processing apparatus 10 performs the processing of S10 to S24 for the next time step. The image processing apparatus 10 repeats this, and when it becomes necessary to end the display, it ends all the processing (Y in S26).

[0078] FIG. 11 is a flowchart showing a processing procedure in which the object data generation device 200 generates data of the light source contribution degree distribution in the present embodiment. This flowchart is started when an object to be the generation target of the light source contribution degree distribution is designated by a content creator or the like. Assume that the model data of the object is stored in the object model storage unit 214.

[0079] The light source contribution degree acquisition unit 216 of the object data generation device 200 arranges a light source for contribution degree acquisition around the object in the virtual space (S30), and then arranges a virtual camera so as to include the object in the field of view (S32). The number, arrangement of the light sources for contribution degree acquisition, and the direction of the virtual camera follow the settings by the condition setting unit 210. Then, the light source contribution degree acquisition unit 216 selects one pixel of the virtual camera, generates a ray passing through the pixel from the optical center (S34), and traces it by the ray tracing method (S36). The light source contribution degree acquisition unit 216 traces the ray in consideration of various physical phenomena in the object, and finally gives a count to the luminance acquisition light source reached by the ray according to a predetermined rule (S38).

[0080] The light source contribution degree acquisition unit 216 performs sampling processing for obtaining the light source at the arrival destination for a predetermined number of times for one ray (N in S40, S36, S38). When the sampling reaches the predetermined number of times (Y in S40), the light source contribution degree acquisition unit 216 calculates the contribution degree based on the count given to each light source, and stores it in the light source contribution degree distribution data storage unit 218 as data of the contribution degree distribution together with the incident parameters of the ray (S42).

[0081] The light source contribution degree acquisition unit 216 repeats the processes of S34 to S42 for all pixels (N in S44). When the light source contribution degree distribution can be stored for the rays passing through all pixels (Y in S44), the light source contribution degree acquisition unit 216 changes the direction of the virtual camera (S32), and repeats the processes of S34 to S44 (N in S46). When the light source contribution degree distribution is obtained for all directions that the virtual camera should take, the entire process ends (Y in S46).

[0082] In the example of FIG. 5, an example in which the light source for obtaining contribution degrees and the light source in the scene correspond one-to-one was shown, but the present embodiment is not limited thereto. FIG. 12 is a diagram for explaining an example of a method by which the image processing apparatus 10 associates the light source for obtaining contribution degrees with the light source in the scene in the present embodiment. (a) shows a case where one light source 140 in the scene exists so as to cover the directions of a plurality of light sources for obtaining contribution degrees (four light sources 142 for obtaining contribution degrees in the figure). In this case, the image processing apparatus 10 associates the light source 140 in the scene with the plurality of light sources 142 for obtaining contribution degrees. That is, the image processing apparatus 10 calculates a weighted sum of the luminance of the light source 140 in the scene as the respective luminances of the light sources 142 for obtaining contribution degrees in the calculation of the pixel value.

[0083] Even when the light source is not explicitly shown in the display viewing field, such as sunlight, the luminance given to the light source for obtaining contribution degrees can be appropriately set according to the direction set for the light source. For example, as model data, a method of pseudo-expressing the background with a light load by preparing an HDRI (High Dynamic Range Image) environment map 144 as shown in (b) is known. According to ray tracing using the environment map 144, the background itself serves as a light source, and an illumination environment close to natural light can be expressed.

[0084] Since the position on the image represents azimuth information in the environment map 144, the combination of the azimuth and brightness can be obtained by region division. For example, if the environment map 144 is divided into regions as shown by the broken line in the figure and associated with the light sources for obtaining contribution degrees in the directions corresponding to the respective regions, it becomes the same as the case where the light source in the scene and the light source for obtaining contribution degrees correspond one-to-one. Therefore, by separately obtaining the brightness for each region and including it in the data of the environment map 144, the pixel value of the object image can be determined by the same method as in FIG. 5. Note that the division rule and the number of divisions of the environment map 144 are not limited to those shown in the figure, and in practice, it may be divided into more regions.

[0085] FIG. 13 is a diagram for explaining a method of drawing images of a plurality of objects using a light source contribution degree distribution. This example assumes that there are two objects 150a and 150b and their positional relationship does not change. In the figure, a light source for obtaining contribution degree (for example, light source 154 for obtaining contribution degree) arranged so as to surround the objects 150a and 150b is also shown. The light source for obtaining contribution degree can be regarded as the direction in which the contribution degree is obtained for the object.

[0086] That is, when there is an in-scene light source (for example, in-scene light source 155) on the half-line (for example, half-line 153) from the center of gravity of the objects 150a and 150b to the light source for obtaining contribution degree, the luminance L i of the in-scene light source is totaled with the contribution degree w i obtained for the corresponding light source for obtaining contribution degree as a weight. Note that the light sources for obtaining contribution degree may actually be distributed on the spherical surface as shown in FIG. 3. The same applies to FIGS. 14 and 15 described later.

[0087] When the positional relationship between the objects 150a and 150b is fixed, as shown in the figure, the ray 152 may reach the light source through both the object 150a and the object 150b. However, as long as their positional relationship does not change, the probability that the ray follows the path does not change, so they can be regarded as one object collectively. Therefore, the image processing apparatus 10 can determine the pixel values of the images of the objects 150a and 150b in the same manner as in the case of one object using the light source contribution degree distribution data acquired in the state shown in the figure in advance. The same applies when the number of objects is three or more as long as the positional relationship does not change.

[0088] FIG. 14 is a diagram for explaining another method of drawing images of a plurality of objects using a light source contribution distribution. This example assumes a case where the positional relationship between two objects 156a and 156b changes. In this case, as in FIG. 3, data on the light source contribution distribution is individually acquired for each of the objects 156a and 156b. While the objects are separated by a predetermined distance or more, the image processing apparatus 10 calculates, as described above, the weighted sum of the brightnesses of the light sources in the scene using the contribution of the corresponding light source for obtaining the contribution as a weight.

[0089] When the distance between the objects 156a and 156b falls below a predetermined value, the image processing apparatus 10 adjusts the brightness and color of the light sources in the scene in the direction where the other object exists in the calculation of the pixel values. For example, among the light sources in the scene that determine the pixel value of the image of the object 156a, a brightness is obtained such that at least one of the brightness and color of the light source in the direction of the other object 156b is adjusted, and this is used in the calculation of the pixel value. In the illustrated example, the light sources in the scene corresponding to the three light sources 158a for obtaining the contribution are the targets of adjustment. For example, as the distance between the objects approaches, the brightness of the light sources in the scene is decreased in the calculation. Alternatively, as the distance between the objects approaches, the color of the light sources in the scene is made closer to the color on the model that the other object 156b originally has in the calculation.

[0090] Similarly, when drawing the other object 156b, the brightness and color of the light sources in the scene corresponding to the light sources 158b for obtaining the contribution in the direction of the object 156a are adjusted to calculate the pixel values. Even if the other object is not the object to be drawn based on the light source contribution distribution, the change in the image due to its approach can be controlled in the same way. For example, when the shield 160 approaches the object 156a, the brightness and color of the light sources in the scene corresponding to the light source 162 for obtaining the contribution in that direction are adjusted to calculate the pixel value of the image of the object 156a.

[0091] Also in this case, as the shield 160 approaches, the luminance of the light sources in the scene is computationally decreased, or the color of the light sources in the scene is made closer to the color that the shield 160 originally has. In any case, the image processing apparatus 10 calculates the pixel values by adjusting the brightness and color of the light sources in the scene, which uses the data of the contribution degree distribution obtained in advance as weights, without changing the data of the contribution degree distribution. Thereby, it is possible to immediately represent the change in the color tone and the shading of the surface due to the approach of the objects to each other.

[0092] FIG. 15 is a diagram for explaining still another method of drawing images of a plurality of objects using the light source contribution degree distribution. In this example, in the space to be displayed, there are two objects 170a and 170b and an object 172 at the location where they exist. In the example of the figure, the object 172 at the location includes the ground, the forest, etc., but when they do not change significantly, it is possible to perform drawing using the light source contribution degree distribution collectively as shown in FIG. 13. On the other hand, the objects 170a and 170b are included in the set of light sources for obtaining the contribution degree of the object 172 at the location (for example, the light source 174 for obtaining the contribution degree).

[0093] Also in such a case, as shown in the figure, the light source contribution degree distribution data is prepared individually for each object. Thereby, the image processing apparatus 10 reads out the data of the light source contribution degree distribution associated with each object and determines the pixel values of each image. By obtaining the light source contribution degree distribution for each object regardless of the inclusion relationship, it is possible to prepare the light source contribution degree distribution with a granularity suitable for the characteristics such as the size and shape of the object, and optimize the balance of image quality, data size, and calculation load. In addition, since images can be drawn by the same process regardless of the number of objects existing in the space to be displayed, the versatility is increased.

[0094] Next, an example of the data format of the light source contribution degree distribution will be described. FIG. 16 is a diagram for explaining an example of the storage format of the light source contribution degree distribution data 60 in the object model storage unit 56 of the image processing apparatus 10. In this example, the light source contribution degree distribution data is stored in a three-dimensional space. First, a rectangular parallelepiped 182 including the three-dimensional model of the target object 180 is prepared, and it is divided into a predetermined number of rectangular parallelepipeds (voxels).

[0095] Among these, the voxels including the surface of the object 180 define the position P where the ray is incident among the incident parameters. That is, the voxel granularity corresponds to the granularity of the position P. For example, when defining the incident position P with a granularity obtained by dividing the object 180 into 100 parts in each of the three axial directions, the rectangular parallelepiped 182 is divided into 100 parts in each direction to obtain 10 6 ^3 voxels. One voxel stores the incident direction among the incident parameters, that is, the variation of the incident direction of the ray to the voxel position, and the corresponding light source contribution degree distribution.

[0096] For example, when the state of the virtual camera when obtaining the light source contribution degree distribution is 90 cases, the direction of the ray incident on a certain position P on the object surface is 90 cases. On the other hand, assuming that the contribution degree w for each light source for obtaining the contribution degree has RGB channels, and further assuming that the number of light sources for obtaining the contribution degree is 300, the contribution degree w per voxel i will be stored 90×300×3 times. Actually, in the data storage area, a list of the incident direction and the contribution degree distribution is stored at an address corresponding to the three-dimensional coordinates of the voxel. i In this case, the object data generation device 200 generates and tracks rays so as to be incident on the positions corresponding to the respective voxels on the surface of the object 180. At this time, the number of ray tracks N

[0097] (3D), that is, the calculation amount, becomes the following value. l N N l (3D) = the number of virtual camera states × the number of incident positions on the object surface × the number of samples Here, the number of samples is the number of tracking times when obtaining the arrival destination of one ray as a probability. Since the number of samples is counted regardless of whether it reaches the light source, the number of light sources for obtaining contribution degrees does not affect the calculation amount.

[0098] When the number of samples is set to 5000 times under the above conditions, N l (3D) is obtained as follows. N l (3D) = 90 × 100 6 × 5000 By performing such a large number of ray tracings in advance, the operations required during operation can be reduced accordingly, and both high image quality and high processing speed can be achieved.

[0099] According to the 3D-based data storage format as shown in the figure, since the 3D model data that is the source of the object 180 can be directly used to define the storage area, the load of preprocessing can be reduced. On the other hand, when the granularity of the incident position is made finer, the data amount increases rapidly. Also, among the voxels that make up the rectangular parallelepiped 182, storing the data of the contribution degree distribution is limited to the voxels including the object surface, so many voxels where data is not stored occur.

[0100] FIG. 17 is a diagram for explaining another example of the storage format of the light source contribution degree distribution data 60 in the object model storage unit 56 of the image processing apparatus 10. In this example, the light source contribution degree distribution data is stored as a 2D image. That is, the contribution degree distribution is stored in the image of the object seen from the virtual camera in each direction at the time of obtaining the contribution degree (for example, image 184). As shown in the figure, the image becomes an image of the object photographed from different directions depending on the direction of the virtual camera.

[0101] Here, the pixels of the image 184 define the incident position P of the ray among the incident parameters. Also, since each image defines the direction of the virtual camera, the incident direction of the ray is determined uniquely at the position of the pixels among them. That is, since one pixel in the image represents one incident parameter, one set of the corresponding light source contribution distribution is stored for each pixel. For example, if there are 90 states of the virtual camera when obtaining the contribution distribution and the number of light sources for contribution acquisition is 300, 90 images in which the light source contribution w i for each pixel is stored in 300×3 amounts are generated. Actually, in the data storage area, the contribution distribution is stored at the address associated with the two-dimensional coordinates of each image.

[0102] The number of ray traces N l (2D) by the object data generation device 200 for generating such data, that is, the computational amount becomes the following value. N l (2D) = the number of states of the virtual camera × the resolution of the image × the number of samples When the resolution of the virtual camera is 4096×4096 pixels and the number of samples is 5000 under the above conditions, N l is obtained as follows. N l (2D) = 90×4096 2 ×5000

[0103] Even with the data storage format based on 2D as shown in the figure, since the three-dimensional model data of the object 180 can be directly used to define the storage area, the load of preprocessing can be reduced. On the other hand, since the data of the contribution distribution is stored only in the pixels where the image of the object appears among the pixels constituting the image 184 etc., there are pixels where no data is stored.

[0104] FIG. 18 is a diagram for explaining still another example of the storage format of the light source contribution degree distribution data 60 in the object model storage unit 56 of the image processing apparatus 10. In this example, the light source contribution degree distribution data is stored as an image 186 obtained by UV-unfolding the object. The UV-unfolded image is data representing the surface of a three-dimensional object in two dimensions and is widely used as a means for expressing textures. The position in the image represented by the UV coordinates is associated with the three-dimensional position coordinates on the object surface.

[0105] Similar to the case of FIG. 17, each pixel of the image 186 defines the incident position P of the ray among the incident parameters. On the other hand, since the UV-unfolded image 186 can represent the entire surface of the object at once, it is not affected by the blind spots of the virtual camera, and it is not essential to prepare as many images as the number of directions of the virtual camera as in the case of FIG. 17. When storing the light source contribution degree distribution in one image 186, it is necessary to store the variations in the incident direction of the ray among the incident parameters and the corresponding light source contribution degree distributions in each pixel region.

[0106] Therefore, it is conceivable to divide each pixel region by the number of directions of the virtual camera and store each light source contribution degree distribution in the divided region. For example, as shown in the enlarged view on the right side of the image 186 in the figure, the region 188 of each pixel is first divided into eight regions. Each of the eight regions is a triangle corresponding to each face of a regular octahedron 190. In the figure, the correspondence between the face and the region is indicated by numbers. In this way, the directions of the entire sky can be defined according to the position of the pixel region 188.

[0107] For example, if the eight regions are further divided into eight parts and the light source contribution degree distribution is stored in each divided region, a set of light source contribution degree distributions for 64 incident directions can be stored per pixel of the image 186. Assuming the number of light sources for obtaining the contribution degree is 300, 64×300×3 light source contribution degrees w i are stored per pixel. Actually, in the data storage area, a list of contribution degree distributions is stored at the address associated with the UV coordinates of each image. The correspondence between the position within one pixel region and the incident direction is set separately.

[0108] However, the pixel division rule is not limited to this, and an appropriate rule may be set according to the number of virtual camera directions. Alternatively, the UV-unwrapped image 186 may be prepared in the number corresponding to the number of virtual camera directions. In this case, similar to FIG. 17, since each image can be associated with the virtual camera direction, the incident direction of the ray is determined uniquely at the position of the pixel among them. That is, since one pixel in the image represents one incident parameter, one set of the corresponding light source contribution degree distribution may be stored in each pixel.

[0109] In any case, the number of ray tracings by the object data generation device 200 is the same as that in the case of FIG. 17. However, the number of pixels is the resolution of the UV coordinates. In this case, since all pixels correspond to some part of the object surface, there are no pixels where data is not stored, and the storage area can be used efficiently.

[0110] As described above, the light source contribution degree distribution of the present embodiment is obtained for discrete incident parameters depending on the virtual camera direction and the number of pixels. Therefore, the incident parameters of the rays generated from the viewpoint with respect to the display space during operation do not always exactly match the incident parameters for which the light source contribution degree distribution was obtained. Therefore, the light source contribution degree processing unit 58 of the image processing device 10 may estimate the light source contribution degree distribution corresponding to the actual rays based on the light source contribution degree distribution read from the object model storage unit 56.

[0111] FIG. 19 is a diagram for explaining an example of a method for the image processing device 10 to estimate the light source contribution degree distribution corresponding to the rays during operation. The horizontal axis in the figure is the incident position P of the ray among the incident parameters, and the vertical axis is the incident direction ω of the ray. However, actually, the incident position P is a three-dimensional value and the incident direction ω is a two-dimensional value. In the object model storage unit 56 of the image processing device 10, as light source contribution degree distribution data 60, the light source contribution degree distributions W1, W2, W3, W4,... are stored in the form shown in FIGS. 16 to 18 for the discrete incident parameters indicated by the white circles in the figure.

[0112] The light source contribution processing unit 58 of the image processing apparatus 10 generates rays based on the viewpoints for the display image, and acquires the incident positions and incident angles on the object surface. When it is at the position indicated by the black circle, the light source contribution processing unit 58 reads out the light source contribution distributions W1, W2, W3, and W4 obtained for the incident parameters within a range 192 of a predetermined Euclidean distance from the position, and appropriately interpolates them to calculate the light source contribution distribution W for the actual ray. As a result, the light source contribution distribution can be obtained precisely with a finer granularity than the pixels of the virtual camera, and a more detailed image can be generated. Consequently, the data size of the light source contribution distribution prepared in advance can be reduced.

[0113] According to the present embodiment described above, in a system that displays an image including a three-dimensional object, the magnitudes of the influences of a plurality of light sources in different directions on the object on the image of the object are acquired in advance as contribution distributions. Then, during operation, the weighted sum of the luminances of the light sources in the scene is calculated using the contributions obtained in the corresponding directions as weights to determine the pixel values of the image of the object. As a result, processes such as calculating the behavior of light on the object surface or inside in ray tracing or performing sampling to search for the light sources reached by the rays a large number of times can be omitted, and a high-quality image can be represented with low latency.

[0114] Also, by acquiring the light source contribution distribution in advance, it becomes possible to accurately simulate complex physical phenomena such as subsurface scattering over time, and a more realistic image representation can be realized. By making the information acquired in advance the contribution of the light source according to the direction with respect to the object, it can be easily reflected in the display image regardless of the characteristics and colors of the light sources in the scene or even if changes occur during display. Furthermore, by positioning the light source contribution distribution as part of the object model, it can be used in a general-purpose manner regardless of the type of content or the number of objects.

[0115] As described above, the present invention has been described based on the embodiments. It is understood by those skilled in the art that the embodiments are illustrative, and various modifications are possible for each of the constituent elements and combinations of the processing processes, and such modifications are also within the scope of the present invention.

Explanation of Signs

[0116] 10 Image processing apparatus, 16 Display apparatus, 22 CPU, 24 GPU, 26 Main memory, 32 Communication unit, 34 Storage unit, 36 Output unit, 38 Input unit, 50 Viewpoint information acquisition unit, 52 Rendering unit, 54 Space construction unit, 56 Object model storage unit, 58 Light source contribution processing unit, 60 Light source contribution distribution data, 62 Output unit, 200 Object data generation apparatus, 210 Condition setting unit, 214 Object model storage unit, 216 Light source contribution distribution acquisition unit, 218 Light source contribution distribution data storage unit.

Industrial Applicability

[0117] As described above, the present invention can be used in various information processing apparatuses such as an image processing apparatus, a game apparatus, an image display apparatus, a mobile terminal, a personal computer, and a content server, and an information processing system including any of them.

Claims

1. An image processing apparatus for generating a display image of a space including an object, comprising: an object model storage unit that stores, in association with model data of the object, a distribution of light source contribution degrees representing the magnitudes of the influences of a plurality of light sources in different directions on the color of the image of the object; a drawing unit that determines a pixel value of the image by calculating a weighted sum of the luminance of light reflected from the object by an in-scene light source set in the space, using the light source contribution degree obtained for the corresponding direction as a weight, when drawing the image of the object on the display image; an output unit that outputs data of the display image including the image of the object; An image processing apparatus characterized by comprising the above.

2. The object model storage unit stores distributions of the plurality of light source contribution degrees in association with combinations of the incident position and incident direction of a ray from a virtual camera observing the object on the surface of the object, The drawing unit selects a distribution of the light source contribution degree used for determining a pixel value based on a combination of the incident position and incident direction of a ray from the viewpoint with respect to the display image on the surface of the object. The image processing apparatus according to claim 1, characterized by this.

3. The drawing unit associates a distribution of the brightness of the space obtained based on an environment map representing the background of the space with the luminance of the in-scene light source. The image processing apparatus according to claim 1 or 2, characterized by this.

4. The object model storage unit stores the light source contribution degree for each of the three primary colors, The drawing unit calculates the weighted sum for each of the three primary colors. The image processing apparatus according to any one of claims 1 to 3, characterized by this.

5. The object model storage unit stores a distribution of the light source contribution degree that collectively represents the magnitudes of the influences of images of a plurality of objects whose positional relationships do not change on the color, The drawing unit calculates pixel values of images of the plurality of objects using the distribution of the light source contribution degree that is collectively represented. The image processing apparatus according to any one of claims 1 to 4, characterized by this.

6. The drawing unit calculates the weighted sum by computationally adjusting the luminance of the in-scene light source in the direction of the other object according to the distance to the other object. The image processing apparatus according to any one of claims 1 to 5, characterized by this.

7. The image processing apparatus according to claim 6, wherein the drawing unit adjusts the luminance so that the color of the in-scene light source in the direction of the other object approaches the color on the model of the other object in response to the approach of the other object.

8. The image processing apparatus according to claim 6 or 7, wherein the drawing unit adjusts the luminance so that the brightness of the in-scene light source in the direction of the other object decreases in response to the approach of the other object.

9. The image processing apparatus according to any one of claims 1 to 8, wherein the drawing unit individually draws images of a plurality of objects arranged in the space using the distribution of the light source contribution degrees associated with each object.

10. The image processing apparatus according to any one of claims 1 to 9, wherein the drawing unit draws an image of an object for which the distribution of the light source contribution degree is not associated in the object model storage unit by ray tracing using the model data.

11. An object data generation device that generates data related to an object for use in generating a display image, a light source contribution degree distribution acquisition unit that obtains a distribution of light source contribution degrees representing the magnitude of the influence on the color of the image of the object for a plurality of light sources by repeating a sampling process a predetermined number of times, the sampling process including arranging a plurality of light sources in different directions with respect to the object in a virtual space and tracking rays from a virtual camera that observes the object to obtain the light sources at the arrival destinations; a light source contribution degree distribution data storage unit that stores the distribution of the light source contribution degrees in association with combinations of the incident positions and incident directions of the rays on the surface of the object; An object data generation device characterized by comprising the above.

12. The object data generation device according to claim 11, wherein the light source contribution degree distribution acquisition unit increases the light source contribution degree for a light source having a larger number of ray arrivals in the sampling process.

13. The object data generation device according to claim 11 or 12, wherein the light source contribution degree distribution acquisition unit decreases the light source contribution degree of the light source at the arrival destination of the ray as the rate of decrease in luminance due to reflection of the ray on the surface of the object is larger.

14. The object data generation device according to any one of claims 11 to 13, wherein the light source contribution distribution acquisition unit simulates the scattering of rays inside the object in the tracking of the rays.

15. The object data generation device according to claim 14, wherein the light source contribution distribution acquisition unit reduces the light source contribution of the light source at the arrival destination of the ray as the light attenuation ratio due to the collision between the ray and the material particles inside the object is larger.

16. The object data generation device according to any one of claims 11 to 15, wherein the light source contribution distribution data storage unit stores, in a voxel obtained by dividing a rectangular parallelepiped including the object, the distributions of the plurality of incident directions and the light source contributions in correspondence with the voxels including the surface of the object.

17. The object data generation device according to any one of claims 11 to 15, wherein the light source contribution distribution data storage unit stores, in each region of pixels representing an image of the object among a plurality of images representing the states of the object observed by the virtual camera from different directions, the corresponding distribution of the light source contributions.

18. The object data generation device according to any one of claims 11 to 15, wherein the light source contribution distribution data storage unit stores, in each region of pixels of an image obtained by UV-unfolding the object, the distributions of the plurality of incident directions and the light source contributions in correspondence with each other.

19. The object data generation device according to claim 18, wherein the light source contribution distribution data storage unit associates small regions obtained by dividing the pixel regions with the incident directions, and stores the distribution of the light source contributions in each of the small regions.

20. An image processing device that generates a display image of a space including an object, reads out the distribution of the light source contributions from a storage unit that stores the distribution of the light source contributions, which represents the magnitude of the influence of a plurality of light sources in different directions on the color of the image of the object, in association with the model data of the object; When drawing an image of the object on the display image, the weighted sum of the luminance of the light reflected by the object from the in-scene light sources set in the space is calculated using the light source contribution obtained for the corresponding direction as a weight, thereby determining the pixel value of the image; Outputting data of a display image including the image of the object; An image processing method characterized by including the above.

21. An object data generation device that generates data related to an object used for generating a display image, By arranging a plurality of light sources in different directions with respect to the object in a virtual space and repeatedly performing a sampling process for a predetermined number of times to track a ray from a virtual camera that observes the object and obtain the light source at the arrival destination, a distribution of light source contributions representing the magnitude of the influence on the color of the image of the object is obtained for the plurality of light sources; Associating the distribution of the light source contributions with the combination of the incident position and incident direction of the ray on the surface of the object, and storing the distribution of the light source contributions in a storage unit; An object data generation method characterized by including the above.

22. In a computer that generates a display image of a space including an object, A function of reading out a distribution of light source contributions representing the magnitude of the influence of a plurality of light sources in different directions with respect to the object on the color of the image of the object from a storage unit that stores the distribution of the light source contributions in association with the model data of the object; When drawing an image of the object on the display image, a function of determining the pixel value of the image by calculating the weighted sum of the luminance of the light reflected by the object from the in-scene light sources set in the space using the light source contribution obtained for the corresponding direction as a weight; A function of outputting data of a display image including the image of the object; A computer program characterized by realizing the above.

23. In a computer that generates data related to an object used for generating a display image, A function of obtaining a distribution of light source contributions representing the magnitude of the influence on the color of the image of the object for the plurality of light sources by arranging a plurality of light sources in different directions with respect to the object in a virtual space and repeatedly performing a sampling process for a predetermined number of times to track a ray from a virtual camera that observes the object and obtain the light source at the arrival destination; A function of storing the distribution of the light source contribution degree in a storage unit in association with a combination of the incident position and the incident direction of the ray on the surface of the object; A computer program characterized by realizing the above.

Images