Image processing device, image processing method, and data structure of three-dimensional object

Abstract

In the present invention, a map is prepared such that, on each of the surfaces of a cube 112 including an object 110, a distribution including a surface color value is projected when the object 110 is viewed from viewpoints corresponding to the surfaces. The cube 112 is arranged in a space to be displayed. When a ray generated for an arbitrary display viewpoint reaches the inside of the cube, the arrival to the object is determined by using a depth map 116, and the pixel value is determined by sampling the color value corresponding to the arrival point by using a color map 114.

Description

Image processing device, image processing method, and data structure of three-dimensional object

[0001] The present invention relates to an image processing apparatus and an image processing method for generating an image of a three-dimensional object, and a data structure of the three-dimensional object.

[0002] Recent advances in information processing and image display technologies have enabled people to experience the world of video in a variety of ways. For example, by displaying panoramic images on a head-mounted display and displaying images that correspond to the user's line of sight, it is possible to enhance the sense of immersion in the world of video and improve the operability of applications such as games. Furthermore, by displaying image data streamed from a server with abundant resources, users can enjoy high-definition video and game screens regardless of location or size.

[0003] Regardless of the purpose or display format of the image display, how to efficiently draw and display an image is always an important issue. For example, in a mode in which a three-dimensional object is displayed from various angles by allowing freedom of viewpoint and line of sight, high responsiveness to changes in display in response to movement of the viewpoint is required. On the other hand, displaying a high-quality image requires high resolution and complex calculations, which increases the load of image processing. The greater the freedom of viewpoint for an object, the more difficult it becomes to display a high-quality image with low latency.

[0004] The present invention has been made in view of these problems, and its purpose is to provide a technology that can generate high-quality images showing three-dimensional objects from any viewpoint with low delay.

[0005] To achieve the above object, one aspect of the present invention relates to an image processing device, comprising: a map data storage unit that stores map data for each surface constituting a solid containing an object existing in three-dimensional space, the map data projecting the distribution of predetermined parameters on the surface of the object as seen from a corresponding viewpoint; a rendering unit that generates rays to a viewscreen corresponding to a display viewpoint, refers to the map based on the position of the rays within the solid placed in three-dimensional space, and samples parameter values ​​to display an image of the object on the viewscreen; and an output unit that outputs data on the image of the object.

[0006] Another aspect of the present invention relates to an image processing method, comprising the steps of: reading from a storage unit map data in which, for each face constituting a solid containing an object existing in three-dimensional space, a distribution of predetermined parameters is projected onto the surface of the object as seen from a viewpoint corresponding to the face; generating a ray to a view screen corresponding to the display viewpoint, and referring to the map based on the position of the ray within the solid placed in three-dimensional space, sampling the parameter values, thereby displaying an image of the object on the view screen; and outputting the data of the image of the object.

[0007] Yet another aspect of the present invention relates to a data structure for a three-dimensional object, which associates information about the object with a plurality of maps projecting the distribution of predetermined parameters on the surface of the object as seen from a corresponding viewpoint for each face constituting the solid containing the object in three-dimensional space, and which is characterized in that an image processing device generates rays to a view screen corresponding to the display viewpoint, samples parameter values ​​by referring to the maps based on the position of the rays within the solid placed in three-dimensional space, and displays an image of the object on the view screen.

[0008] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also valid aspects of the present invention.

[0009] According to the present invention, an image including a three-dimensional object can be displayed with low delay and high quality.

[0010] 1 is a diagram illustrating an example of the configuration of an image display system to which the present embodiment can be applied. FIG. 1 is a diagram for explaining an outline of image rendering processing that can be used in the present embodiment. FIG. 2 is a flowchart illustrating a processing procedure for determining pixel values ​​when ray marching is adopted in the present embodiment. FIG. 3 is a diagram for explaining an outline of map data prepared in the present embodiment. FIG. 4 is a diagram for explaining a procedure for determining pixel values ​​of a display image using the map illustrated in FIG. 4. FIG. 5 is a diagram illustrating the internal circuit configuration of an image processing device in the present embodiment. FIG. 6 is a diagram illustrating functional blocks of an image processing device in the present embodiment. FIG. 7 is a diagram for explaining a method of referring to a depth map in the present embodiment. FIG. 8 is a diagram for explaining a method of optimizing a solid that defines a map surface based on the shape of an object in the present embodiment. FIG. 9 is a diagram illustrating functional blocks of a device that generates map data in the present embodiment. FIG. 10 is a diagram illustrating an example of a situation in which a projection unit in the present embodiment can omit map generation. FIG. 11 is a diagram illustrating another example of a situation in which a projection unit in the present embodiment can omit map generation. FIG. 12 is a diagram illustrating an example of the data structure of a map stored in a map data storage unit in the present embodiment. FIG. 13 is a diagram schematically illustrating how a map surface setting unit in the present embodiment divides a solid that defines a map surface. FIG. 14 is a diagram schematically illustrating an example of setting a solid that defines a map surface based on the space to be displayed in the present embodiment. 1 is a diagram for explaining a mode in which an image processing device according to the present embodiment draws an image by switching the layer of a map to be used in accordance with a change in a display viewpoint. FIG. 2 is a diagram showing a functional block configuration of an image processing device in consideration of reading out layered map data in the present embodiment. FIG. 3 is a diagram for explaining an example of a method in which a rendering unit determines pixel values ​​when color values ​​of a ray's arrival point are represented by a map on multiple surfaces in a color map in the present embodiment. FIG. 4 is a diagram for explaining another example of a method in which a rendering unit determines pixel values ​​when color values ​​of a ray's arrival point are represented by a map on multiple surfaces in a color map in the present embodiment.

[0011] 1 shows an example of the configuration of an image display system to which this embodiment can be applied. The image display system 1 includes image processing devices 10a, 10b, and 10c that display images in response to user operations, and a content server 20 that provides image data used for display. Input devices 14a, 14b, and 14c for user operations and display devices 16a, 16b, and 16c for displaying images are connected to the image processing devices 10a, 10b, and 10c, respectively. Communication can be established between the image processing devices 10a, 10b, and 10c and the content server 20 via a network 8 such as a WAN (World Area Network) or a LAN (Local Area Network).

[0012] The image processing devices 10a, 10b, and 10c may be connected to the display devices 16a, 16b, and 16c and the input devices 14a, 14b, and 14c either wired or wirelessly. Alternatively, two or more of these devices may be integrated. For example, in the figure, the image processing device 10b is connected to a head-mounted display, which is the display device 16b. The head-mounted display can change the field of view of the displayed image by the movement of the user wearing it on their head, so it also functions as the input device 14b.

[0013] Furthermore, the image processing device 10c is a portable terminal, and is integrally configured with a display device 16c and an input device 14c, which is a touchpad covering the display device 16c's screen. Thus, the external shape and connection form of the illustrated devices are not limited. The number of image processing devices 10a, 10b, 10c and content servers 20 connected to the network 8 is also not limited. Hereinafter, the image processing devices 10a, 10b, 10c will be collectively referred to as image processing devices 10, the input devices 14a, 14b, 14c as input device 14, and the display devices 16a, 16b, 16c as display device 16.

[0014] The input device 14 may be any one or a combination of general input devices such as a controller, keyboard, mouse, touchpad, or joystick, and supplies the content of user operations to the image processing device 10. The display device 16 may be a general display such as a liquid crystal display, plasma display, organic EL display, wearable display, or projector, and displays an image output from the image processing device 10.

[0015] The content server 20 provides the image processing device 10 with content data that accompanies image display. The type of content is not particularly limited and may be any of electronic games, decorative images, virtual spaces, etc. The image processing device 10 may perform display processing while acquiring content data to be used for display from the content server 20, or may read content data acquired in advance from an internal storage device and use it for display. In the latter case, the source of the content data is not limited to the content server 20, but may also be a recording medium.

[0016] In this embodiment, at least a portion of the display target is composed of a three-dimensional object. The three-dimensional object may be manually created, photographed, or measured, as long as three-dimensional shape and color information is available. For example, the image processing device 10 displays the display target object as viewed from any angle in response to a user operation via the input device 14. Alternatively, the image processing device 10 may control the viewpoint or line of sight with respect to the object in response to the progress of a game, etc. The three-dimensional object may be stationary, or may move or change its posture. The image processing device 10 may display a moving image by generating images including the three-dimensional object at a predetermined rate.

[0017] In the field of computer graphics, the technology for displaying three-dimensional objects has made great strides in both hardware and software. For example, path tracing, a type of ray tracing, reduces the amount of computation by probabilistically calculating the rendering equation that represents the relationship between rays and object surfaces using the Monte Carlo method.

[0018] In recent years, high-speed data structures such as BVH (Bounding Volume Hierarchy) have been used to improve the efficiency of ray-object intersection detection, making it possible to use polygon models for rasterization for ray tracing. However, as the model to be displayed becomes more complex and larger in scale, the computational load increases accordingly, making it difficult to represent realistic images that follow changes in viewpoint, line of sight, and object state. In this embodiment, the state of an object as viewed from multiple viewpoints is acquired in advance, thereby reducing the processing load during image generation.

[0019] 2 is a diagram for explaining an outline of the image rendering process that can be used in this embodiment. Here, as a simple example, a spherical object 106 is assumed to be the display target. That is, the image processing device 10 places the object 106, which is defined by a three-dimensional model, and a view screen 102 that corresponds to the viewpoint position and line of sight of the user 100 (hereinafter, sometimes referred to as the "display viewpoint") in a world coordinate system that defines the virtual space. Essentially, an image is generated by projecting the object 106 and background 108 onto the view screen 102.

[0020] By acquiring a display viewpoint at a predetermined rate in response to user operations, game progress, etc., and changing the position and orientation of the view screen 102 accordingly, it is possible to display moving images showing the object 106 from various distances and angles. In particular, this embodiment uses ray tracing as a base to achieve high-quality image representation. Ray tracing is generally a technology that generates rays that pass through each pixel on the view screen 102 from the viewpoint, and tracks the path of the rays while taking into account interactions with the object such as reflection, transmission, and refraction, thereby acquiring color information of the destination as pixel values.

[0021] In the example shown in the figure, ray 104a passing through pixel 103a reaches object 106, and the pixel value of pixel 103a is determined by obtaining the color of the point of arrival. Ray 104b passing through pixel 103b does not include object 106 in its path, and therefore the color of the point of arrival in background 108 is obtained as the pixel value of pixel 103b. In reality, however, ray 104a may take a complex path, such as reflecting off or passing through object 106 to reach another object, and then further reflecting or passing through.

[0022] Therefore, realistic image representation that reflects the material of the object and the surrounding environment is generally achieved by solving rendering equations that take into account the shape and reflection characteristics of each object, the position of the light source, etc. Ray marching is a method for efficiently tracking rays until they reach the object. Ray marching is a method for determining the final destination of a ray by advancing the ray according to a predetermined rule while checking the distance to the nearest object.

[0023] 3 is a flowchart showing the processing procedure for determining pixel values ​​when ray marching is employed. This flowchart shows the procedure for determining the value of one pixel in the image to be generated, and the illustrated procedure is repeated for all pixels to draw the entire image. Note that the illustrated processing procedure is merely an example, and is not intended to limit the present embodiment. For example, ray tracing methods other than ray marching can also be applied to this embodiment.

[0024] As described above, the image processing device 10 first sets a view screen corresponding to the display viewpoint in a virtual space in which an object to be displayed is located, and then generates a ray from the viewpoint through the target pixel (S10). This process actually corresponds to defining the direction in which the ray travels. Next, the image processing device 10 searches in all directions for the object closest to the ray position (first, the viewpoint) (S12).

[0025] If the distance to the nearest object is not close enough for the ray to be considered to have reached the object (N in S14), the image processing device 10 advances the ray according to a predetermined rule (S16). For example, the image processing device 10 may always advance the ray the same distance, or may advance the ray the distance to the nearest object. If the path length of the ray so far has not reached a preset upper limit (N in S18), the image processing device 10 searches for the nearest object at the destination (S12).

[0026] Thereafter, the same process is repeated, and when an object close enough to be considered as having been reached by the ray is detected (Y in S14), the image processing device 10 acquires the point reached (S20) and acquires the color value at the point reached as a pixel value (S22). On the other hand, when the path length of the ray reaches the upper limit in S18 (Y in S18), the image processing device 10 determines that the object is not included in the path of the ray. In this case, the image processing device 10 determines that a point on the background in the direction of the ray is the point reached, and acquires the color value as a pixel value (S20, S22).

[0027] In this embodiment, the processing speed is increased by generating in advance any of the information required in the illustrated procedure as a map representing the distribution on the surface of the object when viewed from multiple directions. For example, at least one of a depth map used for arrival confirmation in S14 and a color map used for color value acquisition in S22 is prepared. These maps can be prepared using conventional ray tracing or the like, with detailed data, over a long period of time.

[0028] For example, even complex models such as a human face can be expressed in detail by path tracing using a polygon mesh. However, the means for generating the map is not particularly limited, and it may be obtained by sensor measurement, photography, or the like. In addition to the color map, a material map representing the distribution of values ​​indicating the material of the object surface, an ambient occlusion map representing the distribution of values ​​of the shade of the shadow formed on the object surface, and the like may be prepared, and the final pixel value may be determined taking into account the influence of these maps. For example, the image processing device 10 may perform lighting processing, such as expressing the reflection of a light source according to the angle between the normal of the object surface and a ray using the light reflection coefficient represented as a material map.

[0029] More specifically, an ambient occlusion map is data that pre-calculates the degree to which light can indirectly reach each point on an object's surface from any direction, due to the influence of nearby shapes. Using an ambient occlusion map makes it easy to darken concave areas, such as the gaps between the fingers, or areas surrounded by other surfaces or recessed areas. Ambient occlusion maps can be created in advance using methods such as the Monte Carlo method in path tracing, starting from each point on the object's surface. By referencing the ambient occlusion map in combination with a material map and other methods to determine the final pixel values, higher quality images can be generated.

[0030] FIG. 4 is a diagram for explaining an outline of map data prepared in this embodiment. First, (a) shows an example of setting surfaces representing a map. In this embodiment, the map is basically represented on multiple surfaces constituting a solid that contains all or part of an object. Hereinafter, these surfaces may be referred to as "map surfaces." Note that the number of objects contained in one solid is not limited to one.

[0031] In the illustrated example, a cube 112 is set in three-dimensional space to contain a duck object 110, and the six faces that make up the cube are used as map surfaces. The map is obtained by projecting the distribution of color values ​​and other data on the surface of the object 110 onto each map surface. If the center of the cube is the origin and the axes perpendicular to the faces are the X, Y, and Z axes, respectively, the map corresponds to six different views of the object 110 when viewed from the positive and negative sides of the X axis, the positive and negative sides of the Y axis, and the positive and negative sides of the Z axis.

[0032] In the figure, the viewpoint on the positive side of each axis is indicated by "+" and the viewpoint on the negative side by "-", and the line of sight to each map surface is indicated by an arrow. Assuming a line of sight perpendicular to the map surface, the map of each surface corresponds to an image obtained by orthogonally projecting the object 110. However, the projection method in this embodiment is not limited to this.

[0033] (b) shows a schematic example of a map generated by the setting in (a). The color map 114 in the upper row is composed of six images formed by projecting the distribution of color values ​​of the object 110 onto the six map surfaces of the cube 112. Here, "color values" typically refer to a set of values ​​for the three primary colors red, green, and blue, but may also include an alpha value representing transparency. The depth map 116 in the lower row is composed of six images formed by projecting the distribution of distances from the six map surfaces of the cube 112 to the object 110.

[0034] By setting a map surface, and thus a viewpoint, to surround the object 110 and acquiring detailed parameter distributions visible from that point, it is possible to accommodate movement of the display viewpoint in all directions during display. For example, the depth map 116 also includes shape information on the back and bottom surfaces of the object 110, making it easy to confirm the distance to the object and acquire the arrival point regardless of the ray's incident direction. Furthermore, in the real world, even the same part of an object's color changes depending on the viewing direction, depending on factors such as the reflection of light from a light source. The color map 114 of this embodiment includes information on such color changes, making it possible to reproduce them in the displayed image.

[0035] In the illustrated example, the color map 114 and the depth map 116 are each two-tiered images, with maps corresponding to the three positive planes of the X, Y, and Z axes arranged in the upper tier and maps corresponding to the three negative planes arranged in the lower tier. For ease of understanding, symbols such as "X+" are used in the illustration, but these do not represent actual maps. Furthermore, the map configuration is not limited to that shown in the illustration. Furthermore, as described above, the types of maps prepared in this embodiment are not limited to those shown in the illustration.

[0036] Figure 5 is a diagram for explaining the procedure for determining pixel values ​​of a display image using the map illustrated in Figure 4. For ease of understanding, the relationship between object 110 and cube 112 shown in Figure 4(a) is represented two-dimensionally using the X and Y axes. Here, assume that a ray 122 propagated by ray marching is incident on face 120 of cube 112 and reaches point 124 that is a distance x away in the X-axis direction.

[0037] The image processing device 10 then acquires the distance value D indicated in the corresponding YZ coordinates from the depth map displayed on the surface 120. The image processing device 10 then compares the current position x of the ray 122 with the distance value D to the object 110, and determines whether or not the ray 122 has reached the object 110. For example, the image processing device 10 determines that the difference between D and x has not reached the threshold value for determining that the ray 122 has reached the object 110, and moves the ray 122 further forward.

[0038] By repeating this determination and ray progress, when it is determined that the ray 122 has reached the object 110, the image processing device 10 refers to the color map represented on the surface 120 and determines a pixel value by sampling color values ​​in the YZ coordinates of the arrival point 126 of the ray 122. In practice, a similar comparison is performed in three axis directions, and the arrival point of the ray 122 is identified on the condition that arrival has been determined for all axes. If the ray 122 goes outside the cube 112 without satisfying this condition, the image processing device 10 acquires the color value of the background as shown in FIG. 2, or performs similar processing on another cube.

[0039] By using a combination of a depth map and a color map in this way, high-quality images can be efficiently generated without imposing restrictions on the range of movement of the viewpoint. However, the map data used in this embodiment is not limited to these combinations. For example, ray marching may be performed using a conventional method without using a depth map, and only the color value of the arrival point may be obtained from the color map of this embodiment. Alternatively, if the shape of the object 110 as seen from a free viewpoint can be represented, it is sufficient to use only the depth map in situations where it is not necessary to represent it in color.

[0040] 6 shows the internal circuit configuration of the image processing device 10. The image processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are interconnected via a bus 30. An input / output interface 28 is also connected to the bus 30. Connected to the input / output interface 28 are a communication unit 32 including a peripheral device interface such as USB or IEEE 1394 or a network interface for a wired or wireless LAN, a storage unit 34 such as a hard disk drive or nonvolatile memory, an output unit 36 ​​that outputs data to the display device 16, an input unit 38 that inputs data from the input device 14, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, optical disk, or semiconductor memory.

[0041] The CPU 22 controls the entire image processing device 10 by executing an operating system stored in the storage unit 34. The CPU 22 also executes various programs read from removable recording media and loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has the functions of a geometry engine and a rendering processor, performs drawing processing in accordance with drawing commands from the CPU 22, and stores display images in a frame buffer (not shown). The GPU 24 then converts the display images stored in the frame buffer into video signals and outputs them to the output unit 36. The main memory 26 is composed of RAM (Random Access Memory) and stores programs and data required for processing.

[0042] Fig. 7 shows functional blocks of the image processing device 10 in this embodiment. As described above, the image processing device 10 may perform general information processing such as progressing through an electronic game and communicating with a server, but Fig. 7 focuses particularly on the function of generating a display image. Note that at least some of the functions of the image processing device 10 shown in Fig. 7 may be implemented in the content server 20.

[0043] 7 and the functional blocks shown in Figures 10 and 17 described below can be realized in terms of hardware by the configuration of the CPU, GPU, various memories, etc. shown in Figure 6, and in terms of software by programs that perform various functions such as data input function, data storage function, image processing function, and communication function, loaded into memory from a recording medium, etc. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0044] The image processing device 10 includes an input information acquisition unit 50 that acquires the content of user operations, a space data storage unit 52 that stores definition information for the space to be displayed, a space control unit 54 that controls the state of the space, a map data storage unit 56 that stores map data, a drawing unit 58 that generates an image using the map, and an output unit 60 that outputs the generated image.

[0045] The input information acquisition unit 50 acquires the content of a user operation via the input device 14. Possible content of a user operation includes selection of a display target or electronic content, movement of a display viewpoint, and input of various commands for electronic content. In this embodiment, attention is particularly focused on the operation of moving the display viewpoint. The display viewpoint may be directly operated by the user or may be changed as a result of content processing such as an electronic game. In the latter case, the input information acquisition unit 50 may acquire information related to the movement of the display viewpoint from a content processing function (not shown).

[0046] The spatial data storage unit 52 stores definition information relating to the placement and movement of objects in the three-dimensional space to be displayed. For example, the spatial data storage unit 52 stores information such as position coordinates in the three-dimensional space to be displayed and rules of movement corresponding to the information acquired by the input information acquisition unit 50, in association with the identification information of each object. The spatial data storage unit 52 may also store model data of objects and backgrounds other than the objects represented using the map.

[0047] The space control unit 54 controls the three-dimensional space to be displayed in accordance with the input information acquired by the input information acquisition unit 50. For example, the space control unit 54 places individual objects in a world coordinate system that defines the space, based on the information stored in the space data storage unit 52. Here, for objects that are drawn using a map, the space control unit 54 places the corresponding map surface in the world coordinate system. The correspondence and position / orientation relationships between the solids that define the map surface and the objects contained therein are stored in the map data storage unit 56.

[0048] The space control unit 54 may further change the state of the three-dimensional space in response to input information acquired sequentially. When the position or orientation of an object rendered using a map changes, the space control unit 54 changes the position or orientation of the three-dimensional object defining the map surface accordingly. The map data storage unit 56 stores, as a map, data obtained by projecting onto each surface the distribution of predetermined parameters on the surface of the object as seen from viewpoints set to correspond to each surface of the three-dimensional object containing the object.

[0049] More specifically, the map data storage unit 56 stores the shape and size of a solid that defines the map surface, the position and orientation relationship with the object, and data on the map itself, in association with the identification information of the object to be displayed. The correspondence between the area on the map surface and the map surface in the solid is determined in advance. As described above, the map data typically consists of a color map and a depth map, but either one may be used, or a material map, ambient occlusion map, etc. may also be stored.

[0050] In addition, the map data storage unit 56 may store any type or number of maps, such as a shadow map used for shading, as long as the map is a parameter representing the state of an object surface. Here, a shadow map refers to a depth map represented by regarding a light source as a viewpoint. By using a shadow map, it is possible to determine whether light from a light source directly reaches the surface of any object, thereby rendering a shadow.

[0051] Furthermore, while the map illustrated in FIG. 4 is data obtained by orthogonally projecting the distribution of color values ​​and the like onto each face of a cube containing the object, this is not intended to limit the scope of this embodiment. For example, the solid defining the map face may be a rectangular parallelepiped other than a cube, and the number of faces is not limited to tetrahedrons or octahedrons. Furthermore, the projection method onto each face is not limited to orthogonal projection; for example, the projection direction from the object to the face may be non-perpendicular. Furthermore, projection may be performed in multiple directions onto a single face to generate multiple maps.

[0052] The more projection directions there are, the fewer parts of the object that are not represented in the map due to occlusion, etc., and the more accurate the rendering. On the other hand, as the map data size increases, the amount of space required for transmission and storage increases, which can lead to poorer data access efficiency and increased rendering load. Therefore, it is preferable to determine the number of map planes and projection directions based on the processing capacity of the image processing device 10 and the accuracy required for the generated image.

[0053] The rendering unit 58 sets a view screen in the three-dimensional space to be displayed according to the display viewpoint and renders an image of the object. At this time, as shown in FIG. 3, the rendering unit 58 generates rays for each pixel of the view screen and determines the pixel value by identifying the point at which each ray reaches. In this embodiment, a solid that defines a map surface exists in the three-dimensional space. When a ray reaches the map surface, the rendering unit 58 reads the corresponding depth map from the map data storage unit 56 and uses it to determine whether the ray is inside or outside the object, thereby obtaining the point at which it reaches.

[0054] Next, the rendering unit 58 reads the corresponding color map from the map data storage unit 56 and determines the pixel value by sampling the color value of the ray's arrival point from the corresponding position. At this time, the rendering unit 58 may sample material information of the object at the ray's arrival point from the corresponding position in the material map and reflect this in the pixel value. For example, if specular reflection is dominant at the arrival point, the rendering unit 58 may further trace the path of the ray reflected at the arrival point to reflect the influence of another object or light source outside the map surface in the pixel value.

[0055] When the color value includes an alpha value, the rendering unit 58 may further trace rays that pass through the object to represent a translucent object. A general ray tracing technique can be applied to processing related to light reflection and transmission. The rendering unit 58 may also sample a value indicating the shade of a shadow at the point where the ray reaches from a corresponding position in the ambient occlusion map, and reflect the influence of the shadow in the pixel value.

[0056] Those skilled in the art will understand that any of a variety of maps used in deferred shading, etc., can be applied as the map of this embodiment. The rendering unit 58 generates a pixel string of a display image by sequentially storing the determined pixel values ​​in a frame buffer (not shown). The output unit 60 outputs the display image data written in the frame buffer as a video signal to the display device 16 at an appropriate timing.

[0057] 8 is a diagram illustrating a depth map reference method according to this embodiment. The four squares in the diagram represent, in two dimensions, a cube defining the map surface and the object 130. As described in FIG. 5, the depth map represents the distribution of distances from each map surface to the surface of the object 130.

[0058] As shown in the upper left, the map surface 132a on the left side of the figure shows the distance to the object 130 visible from the map surface 132a, as well as the distribution of distances to the surrounding opposing map surfaces, as indicated by the arrows. However, since the latter is known by the definition of a solid, it may be treated as invalid data for the map. Similarly, the upper map surface 132b, the lower map surface 132c, and the right map surface 132d also show the distribution of distances to the object 130 visible from each surface.

[0059] Furthermore, for the front and rear surfaces (not shown), the distribution of distances to the object 130 as seen from those surfaces is expressed, and six distributions form one set of depth maps. In this way, by preparing a depth map for each surface of the solid that contains the object 130, the shape of the object 130 can be expressed in all directions using Boolean operations used in CSG (Constructive Solid Geometry) expression, for example. CSG expression is known as a method of solid modeling. As a result, regardless of the position of a ray that reaches the interior of the cube, it is possible to accurately determine whether the object 130 is inside or outside the cube.

[0060] On the other hand, by optimizing the solid that defines the map surface based on the shape of the object, it may be possible to obtain a more accurate depth map, and ultimately improve the accuracy of the image that is drawn. Figure 9 is a diagram for explaining a method for optimizing the solid that defines the map surface based on the shape of the object. The squares in (a), (b), and (c) represent the cube that defines the map surface in two dimensions in the cross section of the object.

[0061] The object 140 in the figure is shaped like the letter "C." An example of such a shape is a roughly spherical vase with a narrow opening. In this case, at least a portion of the curved portion 144 inside the object 140 is invisible from any of the four map surfaces shown in the figure and the two map surfaces (not shown) located in front and behind the object 140. That is, as indicated by the arrows, the depth map represented by surface 146 shows the shape of part of the curved portion 144, but information related to the shape of the remaining portions is missing from the depth map.

[0062] In other words, based on the information stored in the depth map, the object will have the shape of the fake object 147 shown in (b), and the image will be drawn and displayed in that shape. To improve the drawing accuracy of the curved portion 144, it is possible to prepare depth maps with a wider variety of projection directions, as described above. Alternatively, it is possible to divide the solid that defines the map surface, as shown in (c).

[0063] 1C shows the cube shown in 1A divided horizontally and vertically to define eight cubes. Because the diagram is two-dimensional, the divided cube is shown as four squares. Each of the divided cubes 148a, 148b, 148c, and 148d represents a map for the partial objects 140a, 140b, 140c, and 140d contained therein on each face. In other words, this division generates maps for 48 faces. By dividing the object 140 in this manner, all of the surface information of the object 140, including the curved portion 144, can be represented as a map, thereby enabling the object 140 to be accurately rendered.

[0064] Fig. 10 shows functional blocks of a device for generating map data in this embodiment. The map data generating device 70 may be part of the image processing device 10 of Fig. 7, or may be provided independently as a device for generating data to be stored in the map data storage unit 56 of Fig. 7. Furthermore, the generated map and content data including information on the three-dimensional space to be displayed may be stored on a recording medium or the like, and may be loaded into the main memory of the image processing device 10 during operation. The content data may be stored in a storage device of the content server 20, and may be downloaded to the image processing device 10 during operation, or data required for display may be transmitted sequentially.

[0065] The internal circuit configuration of the map data generating device 70 may be the same as the internal circuit configuration of the image processing device 10 shown in Fig. 6. The map data generating device 70 includes a space data storage unit 72 that stores definition information of the space to be displayed, an object model storage unit 74 that stores model data of objects and backgrounds existing in the space, a map surface setting unit 76 that sets three-dimensional objects that define the map surface, a projection unit 78 that projects predetermined parameters onto the map surface, and a map data storage unit 80 that stores map data.

[0066] The spatial data storage unit 72 stores definition information relating to the placement and movement of objects in the three-dimensional space to be displayed. This information corresponds to the information stored in the spatial data storage unit 52 of the image processing device 10. The object model storage unit 74 stores three-dimensional model data of objects existing in the space to be displayed. This data may be model data used in general three-dimensional computer graphics, such as polygons representing the shape of the object and textures representing the surface appearance.

[0067] The map surface setting unit 76 sets a solid that defines the map surface. For example, the map surface setting unit 76 sets a solid of a predetermined shape that encompasses an object for which a map is to be generated among objects present in the space to be displayed. In this case, the map surface setting unit 76 may read model data of the target object from the object model storage unit 74 and, based on the model data, optimize the shape of the solid, the orientation of the solid relative to the object, whether to divide the solid, the number of divisions, etc. In this case, the map surface setting unit 76 may optimize the solid with a policy of minimizing the invisible portion of the object as viewed from the map surface.

[0068] The projection unit 78 generates a map by projecting the distribution of predetermined parameters on the object surface onto each surface of the solid set by the map surface setting unit 76. As described above, the parameters include, but are not limited to, distance values ​​from each surface, color values, materials, and shadow shading, and the number of parameters is also not limited. The projection unit 78 performs the projection using model data read from the object model storage unit 74.

[0069] At this time, the projection unit 78 may identify polygons that make up the object model but are not projected onto a map surface based on the polygon's identification number, etc. When a polygon is not projected onto any map surface, as described with reference to FIG. 9 , there is a portion of the object that is not represented as a map. Therefore, the projection unit 78 may request further optimization of the solid that defines the map surface by feeding back the identification number of the unprojected polygon to the map surface setting unit 76. In this case, the map surface setting unit 76 adjusts the shape or orientation of the solid or divides the solid based on the orientation of the unprojected polygon, etc.

[0070] Alternatively, the projection unit 78 may identify an area of ​​the texture that constitutes the object model that is not referenced when projecting it onto the map surface. In this case, since there will be a portion of the object that is not represented as a map, the projection unit 78 may feed back information about that area to the map surface setting unit 76. In this case, the map surface setting unit 76 may adjust the shape or orientation of the solid, or divide the solid, based on the direction of the object surface that corresponds to the unreferenced texture, etc.

[0071] The feedback by the projection unit 78 and the optimization of the solid by the map surface setting unit 76 may be performed only once or multiple times. In the latter case, the optimization process may be terminated, for example, when the proportion of unprojected polygons or textures falls below a predetermined standard. The projection unit 78 may reduce the portion of the map surface set by the map surface setting unit 76 that is not projected by adjusting the projection direction and re-projecting it. The projection unit 78 stores the generated map data in the map data storage unit 80 in association with information on the solid that defines the map surface.

[0072] The projection unit 78 may identify, from among the map surfaces set by the map surface setting unit 76, surfaces that will clearly not be referenced when drawing an image, and may omit map generation. Figure 11 illustrates an example of a situation in which the projection unit 78 can omit map generation. In this example, a cube 152 is set as a solid that defines a map surface for an object 150 consisting of a ground surface and trees. In this case, unless there is a special situation in which the back side of the ground surface 154 needs to be drawn, the map for the bottom surface 156 of the cube 152 will not be referenced.

[0073] Therefore, the projection unit 78 can omit generating a map of the bottom surface 156. The projection unit 78 may determine which map surfaces can be omitted on the basis that model data to be projected is not defined. Alternatively, when the map surface setting unit 76 sets a map surface, it may determine which map surfaces can be omitted on the basis of the shape of the object, such as whether the entire surface is included in the cross section of the object, and notify the projection unit 78 of the determined map surfaces. If the movable range of the display viewpoint is limited, the map surface setting unit 76 may identify parts of the object that will not be displayed due to the limitation, and notify the projection unit 78 of the corresponding surfaces as map surfaces that can be omitted.

[0074] FIG. 12 shows another example of a situation in which the projection unit 78 can omit map generation. In this example, a map surface is also set for an object 160 consisting of ground and trees, but the solids defining the map surface are multiple cubes (e.g., cubes 162a, 162b, and 162c). Note that the figure shows the object 160 and cross sections of the multiple cubes in two dimensions. By enclosing the object 160 with a collection of small cubes in this way, the invisible portion of the map surface is reduced, based on the same principle as in FIG. 9, and objects with complex shapes can be accurately rendered. Conversely, even if the size of the cubes remains constant, using a collection of cubes can be used to represent a map regardless of the size of the object.

[0075] The map surface setting unit 76 may form smaller cubes by dividing a cube that contains the entire object, as in the cube 152 shown in Fig. 11, or may arrange cubes of a predetermined size as shown in the figure so as to contain the object 160. In the former case, the map surface setting unit 76 excludes from the setting the outer cubes that do not contain the object.

[0076] In either case, if there is no need to display the interior of object 160, no map surface for the interior is required. Therefore, in the example shown, map surface setting unit 76 excludes the cube located in interior area 164 of the tree object from the settings. As explained in Figures 11 and 12, by omitting the setting of unnecessary map surfaces, the data size of the map stored in map data storage unit 80 can be reduced.

[0077] 13 illustrates an example of the data structure of a map stored in the map data storage unit 80. This example assumes that a cube containing a duck object, as shown in FIG. 4, is set up to generate six color maps and depth maps. As shown by the rectangular regions in (a) and (b), the map can be expressed as an array of pixels on a plane, and can be regarded as an image itself. Of these, the map in (a) is composed of a color map region 170 and a depth map region 172.

[0078] The color map area 170, for example, represents the distribution of color values ​​projected onto three planes on the positive side and three planes on the negative side of each of the X, Y, and Z axes in two stages. In this example, the right side of each stage indicates that the pixel values ​​have channels for red (R), green (G), blue (B), and alpha value (A). The depth map area 172 similarly represents the distribution of distance values ​​projected onto three planes on the positive side and three planes on the negative side of each of the X, Y, and Z axes in two stages, and indicates that the depth map area 172 has a channel for distance values ​​(D). As a result, the image in the color map area 170 is a color image, and the image in the depth map area 172 is grayscale. Note that pixel values ​​for areas of the map surface where no image is projected are set to 0.

[0079] In (b), the number of channels in the color map region 170 is reduced by increasing the number of channels in the depth map region 172 in (a), and the size of the entire map is reduced by representing distance values ​​on opposing map surfaces with the same pixel. That is, the map data in (b) is composed of a color map region 174 and a blended map region 176. In this example, the color map region 174, like the color map region 170 in (a), represents the distribution of color values ​​projected onto three planes on the positive and negative sides of the X, Y, and Z axes in two stages, but as shown on the right side of each stage, the pixel values ​​are composed of three channels: red (R), green (G), and blue (B).

[0080] The blending map region 176 represents the distribution of distance values ​​projected onto the three positive and three negative surfaces of the X, Y, and Z axes, respectively, using different channels of the same image. As shown in (a), when orthogonally projected onto the surfaces of a rectangular parallelepiped, including a cube, the image of the object has the same shape on the opposing surfaces. In other words, by inverting the bottom image in the depth map region 172 in (a), the distance value information for the opposing map surfaces can be combined into a single image, as in the blending map region 176.

[0081] The pixel values ​​in the blending map region 176 also include an alpha channel from the original color map. If the alpha value is the transmittance of the object itself, then the alpha values ​​orthogonally projected onto the faces of a rectangular parallelepiped, including a cube, will have the same distribution on the opposing faces. Therefore, in the blending map region 176, the alpha value information for the opposing faces can be collectively represented as one channel of pixel values. By standardizing the number of pixel value channels in this way regardless of the map type, the map data can be similarly compressed and encoded using conventional image compression techniques. In the example of (b), both the images in the color map region 174 and the blending map region 176 are color images.

[0082] 14 is a schematic diagram showing how the map plane setting unit 76 divides a solid that defines a map plane. In this example, a single cube 182 that encompasses the object 180 is first set. The size of the cube 182 may be set according to a predetermined rule in accordance with the size of the object 180. As a result, as shown in FIG. 13, six maps are generated, each representing the entire image of the object 180. In the figure, this cube 182, and therefore the map generated thereby, are designated as the zeroth layer and are denoted as "Level 0."

[0083] When division is necessary, the map surface setting unit 76 divides the cube 182 on the 0th layer in half along the three axes to generate eight cubes (e.g., cubes 184a and 184b). As described above, division becomes necessary when the portion of the surface of the object 180 that is not represented as a map exceeds a predetermined standard. As a result, a map with 48 surfaces is generated, each representing an image of a portion of the object 180. In the figure, these cubes, and therefore the maps generated from them, are the first layer and are denoted as "Level 1."

[0084] If further division is necessary, the map surface setting unit 76 may, for example, divide each of the eight cubes in the first layer in half along the three axes to generate a total of 64 cubes (e.g., cubes 186a, 186b, 186c, and 186d). This sets 384 map surfaces, but as described with reference to FIGS. 11 and 12, cubes that do not contain the object 180 or cubes located inside the object 180 can be excluded from the setting. As a result, maps are generated for the remaining map surfaces. In the figures, these cubes, and thus the maps generated therefrom, are designated as the second layer and are denoted as "Level 2."

[0085] The map surface setting unit 76 continues to divide the cube in the same manner as above, as necessary. As described above, the smaller the solids that define the map surface, the less information is lost in the map, even for objects 180 with complex shapes. In the illustrated example, the rule is to divide all cubes into eight equal parts, but in some cases, only some of the cubes may be divided, and the number of divisions at one time is not limited. Also, as described above, the solids that define the map surface are not limited to cubes.

[0086] The resolution of each map surface may be constant regardless of the number of divisions of the cube, i.e., the layer, or may be changed depending on the layer. When the resolution is constant, the number of pixels of each map surface of the 0th layer is n １ ×n ２ When , the number of pixels on each map surface of the first layer is n １ / 2×n ２ / 2, the number of pixels on each map surface in the second layer is n １ / 4×n ２ / 4. Note that n １ , n ２ is a natural number, and if the solid that defines the map surface is a cube, n １ = n ２ In addition, when dividing the cube, the map of the next layer may be generated by enlarging the map without actually reprojecting the parameters.

[0087] On the other hand, when the resolution is changed depending on the layer, for example, the number of pixels on each map surface is set to n １ ×n ２ In this case, the more the solid object is divided, the more detailed the map becomes, and the more detailed the image drawn when displayed. To utilize this characteristic, the map data generating device 70 may generate a map with multiple predetermined layers, regardless of whether or not there are invisible parts, and store the map in the map data storage unit 80. This allows the image processing device 10 to generate an image by using different layers of the map depending on the situation at the time of display.

[0088] For example, the image processing device 10 divides the distance range of the viewpoint for the object 180 into three, and draws an image using a map of the 0th layer when the actual display viewpoint is in the farthest range, the 1st layer when it is in the second farthest range, and the 2nd layer when it is in the closest range. This makes it possible to display an image in which the level of detail of the object increases as the display viewpoint gets closer.

[0089] The number of pixels on map surfaces in the same layer may be the same or different. For example, a portion of the surface of object 180 that has a uniform surface that changes appearance little as the viewpoint approaches is mapped at a constant resolution regardless of the layer. On the other hand, a complex portion that should be represented with increased detail as the viewpoint approaches is mapped with increased resolution as the division progresses. In this case, the projection unit 78 may adaptively adjust the resolution of each map surface according to the characteristics of the object's polygons and texture.

[0090] In the embodiments described so far, the solids defining the map surface are basically set based on the object, but the solids can also be set based on the three-dimensional space to be displayed. FIG. 15 schematically illustrates an example of setting the solids defining the map surface based on the space to be displayed. In this example, the display target is an indoor space 190 containing a bed, table, etc. The leftmost diagram illustrates a state in which 2 x 2 cubes (e.g., cubes 190a and 190b) are set for the space 190. Setting a solid based on the space 190 as illustrated corresponds to setting each of the partial areas obtained by dividing the space 190 as the solid.

[0091] As a result, similar to the case where an object is used as the reference, a map representing the surface information of one or more objects contained within the cube is generated for six faces per cube. However, in this case, parts of the object's surface that are not displayed during display, such as parts that are in contact with other objects, are not displayed in the map, thereby reducing data waste. However, a map generated based on a pre-constructed space does not allow the user to move only some objects during display. Therefore, it is conceivable to introduce a map into a mode in which a static space 190 is viewed from any viewpoint.

[0092] When setting a solid based on a space, the map surface setting unit 76 first generates a three-dimensional space 190 by arranging objects in the world coordinate system based on the information stored in the space data storage unit 72. Then, the map surface setting unit 76 arranges cubes so as to fill (or divide) the space 190. The rules for arranging the cubes with respect to the world coordinate system and the size of each cube are determined in advance, taking into account the characteristics of the space 190, etc. The map generation process by the projection unit 78 is as described above.

[0093] As in the case where an object is used as the reference, the projection unit 78 may divide a cube to generate a map with multiple levels with different numbers of divisions, such as the 0th level (Level 0), the 1st level (Level 1), the 2nd level (Level 2), etc. In the example shown in the figure, all cubes are divided into eight equal parts, but only some of the cubes may be divided, and the number of divisions at one time is not limited. Furthermore, the solid body defining the map surface is not limited to a cube.

[0094] The resolution of the map data may be constant regardless of the layer, or may vary depending on the layer. The number of pixels on the map surface at the same layer may be uniform or may vary. The map data generator 70 generates map data for multiple predetermined layers and stores it in the map data storage unit 80. This allows the image processing device 10 to generate images by using different map layers depending on the display situation.

[0095] 16 is a diagram illustrating a manner in which the image processing device 10 switches the map layer to be used in accordance with a change in the display viewpoint and draws an image. This example assumes a case in which, as shown in FIG. 15, maps with multiple layers having different numbers of divisions set based on the space are used. As shown in FIG. 16(a), when the display viewpoint 200a is relatively far from the space 190 to be displayed, the image processing device 10 draws the display image using, for example, the first layer map 202a. Note that in the figure, the map data is represented as a three-dimensional object set in the space 190.

[0096] As shown in (b), when the display viewpoint 200b moves to a position closer to the space 190 to be displayed than in (a), the image processing device 10 draws the display image using, for example, the second-layer map 202b. In this way, even for objects with complex shapes, information of which is missing due to invisible parts in the first-layer map 202a, it becomes possible to accurately draw them using the second-layer map 202b as the display viewpoint 200b approaches.

[0097] As shown in Figure 15, the more a solid is divided, the more solids there are, and therefore the number of map surfaces. However, the map required to render the image is limited to the field of view corresponding to the display viewpoint. In other words, the closer the display viewpoint is to the space 190, the narrower the map range will be. This characteristic means that the closer the display viewpoint is, the more compatible it is with using a map with a larger number of divisions. For example, in the example shown, the number of map surfaces required does not change as the viewpoint moves.

[0098] In other words, the data size of the map referenced during display can be kept roughly constant, regardless of the size of the space to be displayed or the number of map layers. As a result, high robustness can be achieved in terms of data access and rendering processing load, and ultimately display responsiveness, against these external factors. Furthermore, the boundary of the distance of the display viewpoint when switching map layers can be adjusted so that the number of map faces required to render the image is roughly constant. In this way, the number of map faces to be loaded into memory can always be kept below a certain number.

[0099] As long as the number of map planes required remains constant regardless of the distance of the display viewpoint, the map data size itself can be kept constant even if the number of pixels on each map plane remains constant, i.e., even if the resolution increases with each division. Therefore, even if the space to be displayed is huge, an image overlooking the entire space and an image showing a specific object in detail can be drawn by referencing maps of similar data size. As a result, even if the display viewpoint changes significantly, the displayed image can be changed naturally and stably through constant data access.

[0100] FIG. 17 shows the functional block configuration of the image processing device 10, taking into consideration the reading of hierarchical map data. Note that blocks having the same functions as those of the image processing device 10 shown in FIG. 7 are given the same reference numerals, and descriptions thereof will be omitted where appropriate. This example assumes a case in which a three-dimensional object defining the map surface is arranged based on the space to be displayed, as shown in FIG. 15. In this case, the spatial information of the display object, such as the placement of objects, is included in the map itself, so the spatial data storage unit 52 and spatial control unit 54 shown in FIG. 7 can be omitted. However, when a three-dimensional object is set based on an object, the spatial data storage unit 52 and spatial control unit 54 are combined.

[0101] The image processing device 10a includes an input information acquisition unit 50 that acquires the content of user operations, a target data determination unit 210 that determines the map hierarchy and area required for drawing, a compressed data storage unit 214 that stores compressed and encoded map data, a decoding and decompression unit 216 that decodes and decompresses the map data, a map data storage unit 218 that stores the decoded and decompressed map data, a drawing unit 58 that generates an image using the map, and an output unit 60 that outputs the generated image.

[0102] The input information acquisition unit 50 acquires the content of a user operation via the input device 14. Here, particular attention is paid to the operation of moving the display viewpoint. The target data determination unit 210 determines a map necessary for drawing an image. Specifically, the target data determination unit 210 determines the map hierarchy according to the distance from the display viewpoint to the space to be displayed, and then determines the map displayed on the map surface within the field of view according to the position of the viewpoint and the direction of the line of sight. The target data determination unit 210 may predict a necessary map according to the movement of the display viewpoint up to that point.

[0103] Compressed data storage unit 214 stores hierarchical map data generated by map data generation device 70. The map data is compressed and encoded. As shown in FIG. 13 , the map in this embodiment has a data structure similar to that of a general image, so a general compression and encoding method can be used. Decoding and decompression unit 216 decodes and decompresses the map data determined by target data determination unit 210 from the map data stored in compressed data storage unit 214, and stores the decompressed data in map data storage unit 218.

[0104] The map data storage unit 218 is, for example, the internal memory of the GPU 24 shown in FIG. 6 or a memory accessible by the GPU 24, and stores a map of the area required for drawing an image at that time, as well as map data for a predetermined range surrounding that area. The hierarchy and range of the required map are updated as the display viewpoint moves, and the data stored in the map data storage unit 218 also changes. However, as described above, according to this embodiment, only data of a substantially fixed size needs to be stored in the map data storage unit 218, regardless of the data size of the entire map or the position of the display viewpoint, enabling stable image drawing. The functions of the drawing unit 58 and the output unit 60 may be similar to those shown in FIG. 7.

[0105] Next, a method will be described in which the drawing unit 58 of the image processing device 10, 10a determines the pixel value of the arrival point of the ray. As described above, the drawing unit 58 acquires the arrival point of the ray on the surface of the object using the depth map, and determines the pixel value by sampling the color value at the corresponding position in the color map. Here, since the map is generated on a map surface that is set to surround the object, the color value of one arrival point may be represented on multiple maps depending on the shape of the object and its positional relationship with the map surface.

[0106] 18 is a diagram illustrating an example of a method in which the rendering unit 58 determines pixel values ​​when the color values ​​of the ray's arrival point are represented on multiple faces of a color map. In both (a) and (b) of the diagram, the faces of a cube 222 that contains an object 220 are set as map faces. For ease of understanding, the object 220 and the cube 222 are represented in two dimensions in the diagram.

[0107] The figure shows a state in which a ray 224 of a target pixel on the view screen reaches a point 226 on the surface of an object 220. The color value of the arrival point 226 is projected onto both the upper surface 228a and the right surface 228b of the cube 222. Because the color of an object changes depending on the viewing direction, depending on factors such as its position relative to the light source, the color values ​​projected onto surfaces 228a and 228b may differ even for the same arrival point 226. For this reason, the rendering unit 58 may, for example, determine the simple average of the two color values ​​as the pixel value. Alternatively, the rendering unit 58 may determine a prioritized surface depending on the situation and select its color value as the pixel value, or assign a greater weight to the color value and use a weighted average as the pixel value, thereby improving the accuracy of the rendering.

[0108] 10A is a diagram illustrating a method for determining a prioritized surface based on the proximity of the normal vector of the surface to the direction of the ray 224. Specifically, the drawing unit 58 determines the priority surface based on the normal vector n of the surface 228a that represents the color value of the arrival point 226. ａ , the normal vector n of the surface 228b ｂ , and the normal vector n that is closer to the direction of the ray 224, i.e., that has a smaller angle with the ray 224, is compared. ｂIn this case, the rendering unit 58 may adopt the color value of the surface 228b as the pixel value, or may assign a larger weight to the color value of the surface 228b and adopt the weighted average of the color values ​​of the surface 228b as the pixel value.

[0109] The same rule can be applied even when the color values ​​of the arrival point are represented by three or more surfaces. In this case, the drawing unit 58 may use the color value of the surface having the normal vector closest to the direction of the ray 224 as the pixel value, or may assign a greater weight to the surface having the normal vector closer to the direction of the ray 224 and use a weighted average of the color values ​​as the pixel value.

[0110] 10B is a diagram illustrating a method for determining a prioritized surface based on the proximity of the normal vector of the surface to the normal vector of the object at the destination point 226. Specifically, the drawing unit 58 determines the priority surface based on the proximity of the normal vector n ａ , the normal vector n of the surface 228b ｂ and compare the normal vector n of the object 220 at the arrival point 226 ｏ Normal vector n close to ａ The normal vector n of the destination point 226 is given priority. ｏ can be obtained based on the gradient of the distance values ​​at the corresponding positions in the depth map.

[0111] In this case, the drawing unit 58 may also use the color value of the surface 228a as the pixel value, or may assign a larger weight to the color value of the surface 228a and use the weighted average of the two as the pixel value. The same rule can be applied even when the color value of the destination point is represented on three or more surfaces. In this case, the drawing unit 58 uses the normal vector n of the object 220 ｏ Alternatively, the color value of the surface having the closest normal vector to the normal vector n ｏ A weighted average of the color values ​​may be used as the pixel value, with a larger weight being assigned to a surface having a normal vector closer to the normal vector.

[0112] 19 is a diagram illustrating another example of a method by which the drawing unit 58 determines pixel values ​​when the color values ​​of the ray's arrival point are represented on a map of multiple surfaces in a color map. This method is a combination of the methods (a) and (b) in FIG. 18. That is, in FIG. 18, the normal vector n of the surface 228a representing the color value of the arrival point 226 is ａ , the normal vector n of the surface 228b ｂ , and the direction of the ray 224 is compared with the normal vector n of the object 220 at the arrival point 226. ｏ Prioritize the color value of the face whose normal vector is closer to both of

[0113] In the state of FIG. 18, the ray direction vector L is the vertical axis, and the object normal vector n ｏ The horizontal axis represents the distribution of the weighting coefficient k given to the color value of one surface 228a. ａ , c ｂ Then, the final pixel value C is as follows: C=k×c ａ +(1-k) × c ｂ In the rectangular region representing the distribution of k, the upper end is the ray direction vector L, which is the normal vector n of the surface 228a. ａ The lower end is the normal vector n of the surface 228b. ｂ The right end is the normal vector of the object n ｏ is the normal vector n of the surface 228a ａ The left end is the normal vector n of the surface 228b. ｂ This is a state that matches.

[0114] For example, the state 230a is a state in which the ray direction vector L and the object normal vector n ｏ Both of these are normal vectors n ａ , the weight assigned to the color value of surface 228a is set to 1.0, and the weight assigned to the color value of surface 228b is set to 0.0. As a result, the final pixel value C becomes the color value of surface 228a. State 230b is a state where the ray direction vector L and the object normal vector n ｏ Both of these are normal vectors n ｂTherefore, the weight assigned to the color value of surface 228a is set to 0.0, and the weight assigned to the color value of surface 228b is set to 1.0. As a result, the final pixel value C becomes the color value of surface 228b.

[0115] The state 230c is a state in which the ray direction vector L and the object normal vector n ｏ Both of these are normal vectors n ａ and the normal vector n of the surface 228b ｂ Since the surface 228a is at an equal angle to both the surface 228a and the surface 228b, the weight assigned to the color value of the surface 228a is 0.5, and the weight assigned to the color value of the surface 228b is also 0.5. As a result, the final pixel value C is the simple average of the color values ​​of the surfaces 228a and 228b. In addition, the ray direction vector L, the object normal vector n ｏ The weighting coefficient k is adjusted based on a comprehensive evaluation of which of the two angles is closer to the normal vector of the surface 228a or 228b.

[0116] In the figure, the state indicated by the dashed line 232a is k = 0.75, the state indicated by the dashed line 232b is k = 0.5, and the state indicated by the dashed line 232c is k = 0.25. In the states between the lines, the weighting coefficient k is varied continuously or discontinuously. Note that the distribution of weighting coefficients shown in the figure is merely an example. Furthermore, since the number of dimensions of the distribution corresponds to the number of map surfaces on which the color values ​​of the final point are expressed, similar settings are made for spaces with three or more dimensions. The techniques shown in Figures 18 and 19 can also be applied to parameters that affect color values, such as material maps and ambient occlusion maps.

[0117] According to the embodiment described above, map data is prepared for each surface constituting a solid body that contains a three-dimensional object, projecting the distribution of predetermined parameters on the surface of the object as seen from the corresponding viewpoint. This allows for a high degree of freedom in the display viewpoint, while reducing the processing load in ray tracing and enabling the generation of an image of the object at high speed. Because the map can be generated in advance, it can be generated precisely, and as a result, the quality of the image from a free viewpoint is maintained.

[0118] When there are areas not shown in the map due to factors such as the shape of the object, the solid that defines the map surface is divided into smaller units to create a map. This reduces the amount of information missing from the map and enables the generation of high-quality images. It is also possible to prepare maps with multiple layers with different numbers of divisions of the solid, and switch between them depending on the distance of the display viewpoint. This allows the number of maps to be referenced to be kept below a certain level as the display viewpoint changes, while generating an image with a level of detail that corresponds to the distance.

[0119] The solids that define the map can be set based on the object, or can be set to divide the 3D space to be displayed. In this case, by preparing multiple layers of maps with different division numbers, the number of maps to be referenced can be kept constant regardless of the size of the space or the amount of movement of the display viewpoint. Even if the resolution of the map increases as the number of divisions increases, the small surface area of ​​the map prevents changes in the size of the data to be referenced. Therefore, it is possible to generate images at higher resolutions depending on the proximity of the display viewpoint without affecting the efficiency of data access.

[0120] The map in this embodiment has the same data structure as a general image, so image processing techniques and compression coding techniques that are designed for efficiency in general images can be applied, allowing for easy and efficient data transmission, management, reading, and various calculations.

[0121] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0122] As described above, the present invention can be used in various information processing devices such as image processing devices, game devices, image display devices, mobile terminals, personal computers, and head-mounted displays, as well as image display systems including any of these.

[0123] The present disclosure may include the following aspects. [Item 1] An image processing device comprising: a content server including a circuit configured to: store, for each face constituting a solid body enclosing an object existing in three-dimensional space, map data projecting a distribution of predetermined parameters on the surface of the object as seen from a viewpoint corresponding to the face of the object; generate rays to a viewscreen corresponding to a display viewpoint, and refer to the map based on the position of the ray inside the solid body placed in three-dimensional space to sample values ​​of the parameters, thereby displaying an image of the object on the viewscreen; and output data of the image of the object. [Item 2] The image processing device according to item 1, wherein the circuit stores, as the map, a depth map representing the distribution of distances from each face of the solid body to the surface of the object; and acquires arrival points of the rays on the surface of the object by referring to the depth map. [Item 3] The image processing device according to item 1 or 2, wherein the circuit stores a color map representing a distribution of color values ​​of the object as the map, and represents a color image of the object by obtaining color values ​​of arrival points of rays on the surface of the object by referring to the color map. [Item 4] The image processing device according to item 1 or 2, wherein the circuit stores map data obtained by projecting the distribution for each surface of the solid body corresponding to each of a plurality of partial regions obtained by dividing the three-dimensional space. [Item 5] The image processing device according to item 4, wherein the circuit stores map data for a plurality of layers with different numbers of divisions, and switches the layer of the map to be referenced depending on the distance of the display viewpoint. [Item 6] The image processing device according to item 5, wherein the map data storage unit stores map data including maps of different resolutions in the same layer.[Item 7] The image processing device according to item 3, wherein the circuit stores as the map a material map representing a distribution of values ​​indicating the material of the surface of the object, and reflects the value indicating the material at the destination point in the color image by referencing the material map. [Item 8] The image processing device according to item 7, wherein the circuit reflects the influence of another object and a light source outside the three-dimensional object in the color image by further tracing a ray path based on the value indicating the material at the destination point. [Item 9] The image processing device according to item 3, wherein the circuit stores as the map an ambient occlusion map representing a distribution of values ​​indicating the shading of a shadow formed on the surface of the object, and reflects the value indicating the shading at the destination point in the color image by referencing the ambient occlusion map. [Item 10] The image processing device according to item 1 or 2, wherein the circuit stores data of the map from which data for some of the faces constituting the three-dimensional object is omitted. [Item 11] The image processing device according to item 1 or 2, wherein the circuitry stores map data in which the distribution of the predetermined parameter on the surface of an object encompassed by a plurality of the solids is projected onto each of the faces that make up the plurality of solids. [Item 12] The image processing device according to item 11, wherein the circuitry stores the map data in which data for faces that make up one of the plurality of solids and that are located in an invisible portion of the object is omitted. [Item 13] The image processing device according to item 3, wherein, when the color value of the arrival point of the ray is projected onto a plurality of faces in the solid in the color map, the circuitry determines a color value to be adopted based on the proximity of at least one of a normal vector of the object at the arrival point and a direction vector of the ray to the normal vector of each of the plurality of faces.[Item 14] The image processing device according to item 3, wherein the circuit, when the color values ​​of the arrival point of the ray are projected onto multiple surfaces of the three-dimensional object in the color map, calculates a weighted average of the color values ​​with weighting based on the proximity of at least one of the normal vector of the object at the arrival point and the direction vector of the ray to the normal vector of each of the multiple surfaces. [Item 15] The image processing device according to item 2, wherein the circuit stores map data in which the distribution of distances is projected for each surface constituting a rectangular parallelepiped as the three-dimensional object, and data in which the distribution projected onto opposite surfaces of the rectangular parallelepiped is represented in different channels of a common plane. [Item 16] The image processing device according to item 15, wherein the circuit stores data including an α value representing the transparency of the object in one channel of the depth map. [Item 17] An image processing method comprising: reading from a storage unit map data projecting a distribution of predetermined parameters on the surface of an object as seen from a corresponding viewpoint for each face that makes up a solid that contains an object in three-dimensional space; generating rays to a viewscreen corresponding to a display viewpoint, and referencing the map based on the position of the ray within the solid placed in three-dimensional space to sample the values ​​of the parameters, thereby displaying an image of the object on the viewscreen; and outputting data of the image of the object. [Item 18] A recording medium having recorded thereon a program for causing a computer to implement the following functions: reading from a storage unit map data projecting a distribution of predetermined parameters on the surface of an object as seen from a corresponding viewpoint for each face that makes up a solid that contains an object in three-dimensional space; generating rays to a viewscreen corresponding to a display viewpoint, and referencing the map based on the position of the ray within the solid placed in three-dimensional space to sample the values ​​of the parameters, thereby displaying an image of the object on the viewscreen; and outputting data of the image of the object.[Item 19] A recording medium on which three-dimensional object data is recorded, in which, for each face constituting a solid containing an object existing in three-dimensional space, a plurality of maps projecting the distribution of predetermined parameters on the surface of the object as seen from a corresponding viewpoint are associated with information on the solid, and an image processing device generates rays to a view screen corresponding to the display viewpoint, samples the values ​​of the parameters by referencing them based on the position of the rays inside the solid placed in the three-dimensional space, and displays an image of the object on the view screen.

[0124] 1 Image display system, 10 Image processing device, 14 Input device, 16 Display device, 20 Content server, 22 CPU, 24 GPU, 26 Main memory, 32 Communication unit, 34 Storage unit, 36 Output unit, 50 Input information acquisition unit, 52 Spatial data storage unit, 54 Spatial control unit, 56 Map data storage unit, 58 Drawing unit, 60 Output unit, 70 Map data generation device, 72 Spatial data storage unit, 74 Object model storage unit, 76 Map surface setting unit, 78 Projection unit, 80 Map data storage unit, 210 Target data determination unit, 214 Compressed data storage unit, 216 Decoding and decompression unit, 218 Map data storage unit.

Claims

1. An image processing device comprising: a map data storage unit that stores map data projecting the distribution of specified parameters on the surface of an object seen from a corresponding viewpoint for each face that constitutes a solid that contains an object existing in three-dimensional space; a drawing unit that generates a ray to a view screen corresponding to a display viewpoint, references the map based on the position of the ray inside the solid placed in three-dimensional space, and samples the values ​​of the parameters to display an image of the object on the view screen; and an output unit that outputs data on the image of the object.

2. The image processing device described in claim 1, characterized in that the map data storage unit stores a depth map as the map, which represents the distribution of distances from each face of the solid to the surface of the object, and the drawing unit refers to the depth map to obtain the arrival points of rays on the surface of the object.

3. An image processing device as described in claim 1 or 2, characterized in that the map data storage unit stores a color map representing the distribution of color values ​​of the object as the map, and the drawing unit represents a color image of the object by obtaining the color value of the arrival point of a ray on the surface of the object by referring to the color map.

4. An image processing device as described in claim 1 or 2, characterized in that the map data storage unit stores the map data projected from the distribution for each surface of the three-dimensional object corresponding to each of a plurality of partial regions obtained by dividing the three-dimensional space.

5. The image processing device according to claim 4, characterized in that the map data storage unit stores data of the map at multiple levels with different numbers of divisions, and the drawing unit switches the level of the map to be referenced depending on the distance of the display viewpoint.

6. The image processing device according to claim 5, wherein said map data storage unit stores the map data including maps of different resolutions in the same layer.

7. The image processing device described in claim 3, characterized in that the map data storage unit stores a material map representing a distribution of values ​​indicating the material of the surface of the object as the map, and the drawing unit reflects the value indicating the material at the arrival point in the color image by obtaining it by referring to the material map.

8. The image processing device described in claim 7, characterized in that the drawing unit reflects the influence of other objects and light sources outside the solid body in the color image by further tracing the path of the ray based on the value indicating the material at the arrival point.

9. The image processing device described in claim 3, characterized in that the map data storage unit stores an ambient occlusion map representing the distribution of values ​​indicating the shade of the shadow formed on the surface of the object as the map, and the drawing unit reflects the value indicating the shade at the arrival point in the color image by obtaining it by referring to the ambient occlusion map.

10. An image processing device according to claim 1 or 2, characterized in that the map data storage unit stores the map data from which data for some of the faces constituting the solid body is omitted.

11. An image processing device as described in claim 1 or 2, characterized in that the map data storage unit stores map data in which the distribution of the specified parameters on the surface of an object encompassed by multiple solids is projected onto each of the faces that make up the multiple solids.

12. The image processing device according to claim 11, characterized in that the map data storage unit stores the map data in which data on the faces constituting the solids located in the invisible part of the object among the plurality of solids is omitted.

13. The image processing device described in claim 3, characterized in that when the color value of the arrival point of the ray is projected onto multiple surfaces of the solid body in the color map, the drawing unit determines the color value to be used based on the proximity of at least one of the normal vector of the object at the arrival point and the direction vector of the ray to the normal vector of each of the multiple surfaces.

14. The image processing device described in claim 3, characterized in that when the color values ​​of the ray's arrival point are projected onto multiple surfaces of the solid body in the color map, the drawing unit performs a weighted average of the color values ​​using weighting based on the proximity of at least one of the object's normal vector at the arrival point and the ray's direction vector to the normal vectors of each of the multiple surfaces.

15. The image processing device described in claim 2, characterized in that the map data storage unit stores map data in which the distribution of distances is projected onto each face constituting a rectangular parallelepiped as the three-dimensional object, and stores data in which the distribution projected onto the faces directly opposite each other in the rectangular parallelepiped is represented in different channels of a common plane.

16. The image processing device according to claim 15, wherein the map data storage unit stores data including an α value representing the transparency of the object in one channel of the depth map.

17. An image processing method comprising the steps of: reading from a memory unit, for each face constituting a solid containing an object existing in three-dimensional space, map data projecting the distribution of predetermined parameters on the surface of said object as seen from a corresponding viewpoint; generating a ray to a view screen corresponding to the display viewpoint, and referring to said map based on the position of the ray within said solid placed in said three-dimensional space, sampling the values ​​of said parameters, thereby displaying an image of the object on said view screen; and outputting data of the image of said object.

18. A computer program that causes a computer to perform the following functions: read out from a memory unit map data that projects the distribution of specified parameters on the surface of a solid that contains an object in three-dimensional space, as seen from each corresponding viewpoint, for each face that makes up the solid; generate rays to a view screen corresponding to the display viewpoint, refer to the map based on the position of the ray within the solid placed in three-dimensional space, and sample the values ​​of the parameters to display an image of the object on the view screen; and output data of the image of the object.

19. A data structure for a three-dimensional object, characterized in that for each surface constituting a solid containing an object existing in three-dimensional space, multiple maps projecting the distribution of specified parameters on the surface of the object as seen from the corresponding viewpoint are associated with the information on the solid, and an image processing device generates rays to a view screen corresponding to the display viewpoint, samples the values ​​of the parameters by referencing them based on the position of the ray within the solid placed in three-dimensional space, and displays an image of the object on the view screen.

Images