Image data generating device, display device, image display system, image data generating method, and image display method

Abstract

In this image data generation device 200, a viewpoint setting unit 262 sets a reference viewpoint on the basis of position and posture information on a head-mounted display 100, and a reference image drawing unit 264 generates a reference image with a field of view corresponding to the reference viewpoint. An additional data generation unit 268 acquires, as additional sampling data, the color information about a hidden portion that is not represented by the reference image, from a different viewpoint. The reprojection unit 276 of the head-mounted display 100 transforms the reference image into an image from the latest viewpoint, and determines the pixel value of the hidden portion by using the additional sampling data.

Description

[Technical field]

[0001] The present invention relates to an image data generating device that generates an image with a changing field of view, a display device that displays the image, an image display system, an image data generating method, an image display method, and a data structure of image data. [Background technology]

[0002] Image display systems that allow users to view a target space from any viewpoint are becoming widespread. For example, electronic content that realizes virtual reality (VR) by displaying an image according to the line of sight of a user wearing a head-mounted display in a virtual three-dimensional space is known. Using a head-mounted display can enhance the sense of immersion in the image world and improve the operability of applications such as games. A walk-through system has also been developed that allows a user wearing a head-mounted display to virtually walk around a space displayed as an image by physically moving around. Summary of the Invention [Problem to be solved by the invention]

[0003] Regardless of the type of display device, when allowing users to manipulate the viewpoint or line of sight, high responsiveness is required for image display. On the other hand, to realize more realistic image expression, the image processing load increases due to higher resolution and more complex calculations. When transferring image data from a device other than the display device, the transfer time is also required. As a result, a significant delay occurs before the image in the field of view corresponding to the operation is displayed, which reduces the sense of realism and makes the user feel uncomfortable. In particular, in the case of head-mounted displays, motion sickness caused by a delay in the image relative to head movement is also a problem.

[0004] The present invention has been made in view of the above problems, and has as its object to provide a technique capable of achieving both responsiveness and quality in image display. [Means for solving the problem]

[0005] One aspect of the present invention relates to an image data generating device, which includes a viewpoint setting unit that sets a reference viewpoint at a predetermined time step based on information related to a change in viewpoint with respect to a space to be displayed, a reference image drawing unit that draws a reference image that represents the space with a field of view corresponding to the reference viewpoint, an additional data generating unit that acquires color information of an occluded part of the space that is not represented in the reference image as additional sampling data by setting a viewpoint different from the reference viewpoint, and a transmitting unit that associates the reference image with the additional sampling data and transmits them to a display device.

[0006] Another aspect of the present invention relates to a display device, which displays a space to be displayed by changing a field of view based on information related to a change in a viewpoint, and is characterized by including an image data acquisition unit that acquires data of a reference image that displays the space in a field of view corresponding to the reference viewpoint and additional sampling data including color information of an occluded part of the space that is not displayed in the reference image, a reprojection unit that converts the reference image into an image in a field of view corresponding to the latest viewpoint and generates a display image by adding color information using the additional sampling data, and a display unit that outputs the display image.

[0007] Yet another aspect of the present invention relates to an image display system, which includes an image data generating device and a display device, and represents a space to be displayed by changing a field of view based on information related to a change in a viewpoint, the image data generating device includes a viewpoint setting unit that sets a reference viewpoint at a predetermined time step, a reference image drawing unit that draws a reference image representing the space with a field of view corresponding to the reference viewpoint, an additional data generating unit that acquires color information of an occluded part of the space not represented in the reference image as additional sampling data after setting a viewpoint different from the reference viewpoint, and a transmitting unit that transmits the reference image and the additional sampling data in association with each other, and the display device includes an image data acquiring unit that acquires the reference image and the additional sampling data, a reprojection unit that converts the reference image into an image with a field of view corresponding to the latest viewpoint and generates a display image by adding color information using the additional sampling data, and a display unit that outputs the display image.

[0008] Yet another aspect of the present invention relates to an image data generating method, which includes the steps of: setting a reference viewpoint at a predetermined time step based on information related to a change in viewpoint with respect to a space to be displayed; rendering a reference image representing the space with a field of view corresponding to the reference viewpoint; acquiring color information of an occluded part of the space that is not represented in the reference image as additional sampling data after setting a viewpoint different from the reference viewpoint; and transmitting the reference image and the additional sampling data to a display device in association with each other.

[0009] Yet another aspect of the present invention relates to an image display method, the image display method including a step of acquiring, by a display device which changes a field of view to display a space of a display object based on information related to a change in a viewpoint, data of a reference image which displays the space in a field of view corresponding to the reference viewpoint and additional sampling data including color information of an occluded part not displayed in the reference image, a step of converting the reference image into an image in a field of view corresponding to the latest viewpoint and generating a display image by adding color information using the additional sampling data, and a step of outputting the display image.

[0010] Yet another aspect of the present invention relates to a data structure of image data, which is image data for expressing a space to be displayed by changing a field of view based on information related to a change in a viewpoint, and is characterized in that data of a reference image expressing the space with a field of view corresponding to the reference viewpoint, position information of the reference viewpoint, color information of an occluded part of the space not shown in the reference image, and information related to the position of the occluded part from which the color information is acquired are associated with each other, which are acquired by an image data acquisition unit of a display device, and a reprojection unit converts the reference image into an image with a field of view corresponding to the latest viewpoint and generates a display image by adding color information as an image of the occluded part.

[0011] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also effective as aspects of the present invention. Effect of the Invention

[0012] According to the present invention, it is possible to achieve both responsiveness and quality in image display. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram showing an example of the appearance of a head mounted display according to an embodiment of the present invention; [Diagram 2] 1 is a diagram showing an example of the configuration of an image display system according to an embodiment of the present invention; [Diagram 3] 1 is a diagram for explaining an example of an image world that the image data generating device displays on a head-mounted display in this embodiment. FIG. [Figure 4] 10A to 10C are diagrams illustrating an example of the progress of the generation process of each frame in the image data generating device of the present embodiment and the display process in the head mounted display. [Diagram 5] 4A to 4C are diagrams for explaining changes in an image in response to changes in viewpoint in the present embodiment. [Figure 6] 2 is a diagram showing an internal circuit configuration of an image data generating device according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing an internal circuit configuration of a head mounted display according to the present embodiment. [Figure 8] 2 is a diagram showing a configuration of functional blocks of an image data generating device and a head mounted display according to the present embodiment. FIG. [Figure 9] 4A to 4C are diagrams illustrating an example of the structure of image data transmitted from an image data generating device to a head mounted display in the present embodiment. [Figure 10] 1A and 1B are diagrams for explaining the concept of a method in which a reprojection unit of a head mounted display in the present embodiment generates a display image. [Figure 11] 11A and 11B are diagrams for explaining the effect of reprojection using additional sampling data at a pixel level in the present embodiment. [Figure 12] 5 is a flowchart showing a processing procedure in which the image data generating device generates and transmits image data in the present embodiment. [Figure 13] 5 is a flowchart showing a processing procedure for displaying an image by the head mounted display in the present embodiment. [Figure 14] 1A and 1B are diagrams showing an actual reference image and an image assuming a latest viewpoint in the present embodiment. [Figure 15] FIG. 15 is a diagram showing a state in which the reference image shown in FIG. 14 is divided into patches. [Figure 16]16 is a diagram showing a state in which a patch formed as in FIG. 15 is warped based on the latest viewpoint, and a state in which pixel values ​​of a reference image are mapped onto the warped patch. FIG. [Figure 17] FIG. 13 is a diagram showing the results of determining pixel values ​​by filtering color information of the reference image, also for disocclusion regions. [Figure 18] 11A and 11B are diagrams showing results of determining pixel values ​​using a reference image and additional sampling data in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] This embodiment relates to a system that displays an image so that the field of view of the three-dimensional space of the display target changes according to the user's position and posture, user operation, etc. In this respect, the device that displays the image is not particularly limited to a head-mounted display, a wearable terminal, a mobile terminal, a flat display, a television receiver, etc., but hereinafter, a head-mounted display will be described as an example. FIG. 1 shows an example of the appearance of a head-mounted display 100. In this example, the head-mounted display 100 is composed of an output mechanism 102 and a mounting mechanism 104. The mounting mechanism 104 includes a mounting band 106 that goes around the head when worn by the user and secures the device.

[0015] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head mounted display 100, and includes a display panel inside so as to face the eyes when worn. The housing 108 may further include an eyepiece lens positioned between the display panel and the user's eyes when the head mounted display 100 is worn, which expands the user's viewing angle. The head mounted display 100 may further include a speaker or earphone at a position corresponding to the user's ear when worn. The head mounted display 100 also includes a motion sensor that detects the translational and rotational movements of the head of the user wearing the head mounted display 100, as well as the position and posture at each time.

[0016] In this example, the head mounted display 100 is provided with a stereo camera 110 on the front of the housing 108, and captures a video of the surrounding real space in a field of view corresponding to the user's line of sight. If the captured image is displayed immediately, so-called video see-through can be realized, in which the real space in the direction the user is facing can be seen as it is. Furthermore, AR (Augmented Reality) can be realized by drawing a virtual object on the image of a real object shown in the captured image. In addition, the position and orientation of the head mounted display 100 can be identified by analyzing the captured image using technology such as VSLAM (Visual Simultaneous Localization and Mapping).

[0017] 2 shows an example of the configuration of an image display system in this embodiment. In the configuration shown in (a), a head mounted display 100a is connected to an image data generating device 200a via a wireless LAN (Local Area Network) or HDMI (registered trademark), etc. The image data generating device 200a may further be connected to a server via a network such as a WAN (World Area Network). In this case, the image data generating device 200a may download electronic content such as games and moving images from the server. In the configuration shown in (b), a head mounted display 100b is connected to the image data generating device 200b via a network 160 such as a WAN.

[0018] The image data generating device 200a may be an information processing device owned by an individual and having functions other than image generation, such as a game device or a personal computer. The image data generating device 200b may be a server such as a game server, a video distribution server, or a cloud server that distributes electronic content to clients via the network 160. Hereinafter, these will be collectively referred to as the "image data generating device 200." The form of communication between the image data generating device 200 and the head mounted display 100, the number of head mounted displays 100 connected to the image data generating device 200, and the like are not particularly limited.

[0019] The image data generating device 200 identifies the position of the viewpoint and the direction of the line of sight based on the head mounted display 100 and the position and posture of the head of the user wearing the head mounted display 100, generates a display image in a corresponding field of view, and streams the image to the head mounted display 100. In this respect, the purpose of displaying the image may be various. For example, the image data generating device 200 may generate a virtual world in which the game is set as a display image while progressing through an electronic game, or may generate an image for viewing or providing information. In any case, the image data generating device 200 draws an image representing a three-dimensional space in a field of view corresponding to the viewpoint of the user wearing the head mounted display 100.

[0020] 3 is a diagram for explaining an example of an image world that the image data generating device 200 displays on the head mounted display 100 in this embodiment. In this example, a state is created in which the user 12 is in a room, which is a virtual space. As shown in the figure, objects such as walls, floors, windows, a table, and objects on the table are arranged in a world coordinate system that defines the virtual space. The image data generating device 200 defines a view screen 14 in the world coordinate system according to the position of the viewpoint and the direction of the line of sight of the user 12, and draws a display image by expressing the image of the object thereon.

[0021] By acquiring the position of the user's 12 viewpoint and the direction of his / her line of sight (hereinafter, these may be collectively referred to as "viewpoint") at a predetermined rate and changing the position and direction of the view screen 14 accordingly, it is possible to display an image in a field of view corresponding to the user's viewpoint. By generating stereo images with parallax and displaying them respectively in the left and right areas of the display panel, it is also possible to provide a stereoscopic view of the virtual space. This allows the user 12 to experience virtual reality as if he / she were in a room in the displayed world.

[0022] To realize such an embodiment, the image data generating device 200 repeats a series of processes, such as acquiring information related to the position and posture of the head mounted display 100 in real time, generating corresponding images (frames of a moving image), and transmitting the images to the head mounted display. Fig. 4 illustrates an example of the progress of the generation process of each frame in the image data generating device 200 and the display process in the head mounted display 100.

[0023] In the figure, the image generation process in the image data generating device 200 is shown on the upper part along a common horizontal time axis, and the display process in the head mounted display 100 is shown on the lower part along a single pentagonal arrow, with one frame indicated. In this example, the image data generating device 200 generates frames at a rate lower than the frame rate of display in the head mounted display 100. As an example, when the display rate is 120 fps, the image data generating device 200 generates and transmits frames at 60 fps. In this way, by making the frame generation cycle longer than the display update cycle, it is possible to generate high quality images over a longer period of time while suppressing the overall bit rate.

[0024] However, the relationship of the frame rates in this embodiment is not limited to this, and the generation and display may be at the same rate. In any case, a time lag occurs between the generation and display of each frame, and the time lag becomes larger as the generation rate is lowered or the data transfer time becomes longer depending on the transmission form as shown in the figure. Therefore, in the head mounted display 100, if the image is corrected to correspond to the user's viewpoint immediately before display, the effect of the time lag can be reduced. Such a correction to correspond to a change in viewpoint is called "reprojection."

[0025] In the illustrated example, the image data generating device 200 first determines the viewpoint of the image to be generated based on the position and orientation information acquired from the head mounted display 100 at time t0. Then, toward time t1, images 140a are generated in the corresponding field of view and transmitted sequentially to the head mounted display 100. The head mounted display 100 reprojects the transmitted images based on the position and orientation at that time (time t2), and outputs them to the display panel at time t3. In this example, reprojection corresponding to a viewpoint shift to the left is performed on the transmitted image 140a, resulting in the display of image 140b in which the object on the table is shifted to the right.

[0026] In this way, by absorbing the change in viewpoint that occurs between time t0 and t2 through the reprojection process, it is possible to continue displaying images with little delay in response to the user's movements. This allows the frame generation rate to be set independently of the display rate, and also provides the effects of improving image quality and reducing bit rate as described above. Hereinafter, the image generated by the image data generating device 200 will be called the "reference image," the viewpoint set in generating the reference image will be called the "reference viewpoint," and the viewpoint set by the head mounted display 100 during reprojection will be called the "latest viewpoint."

[0027] The image data generating device 200 sequentially sets the reference viewpoint at time steps corresponding to the frame generation rate, such as time t0, t1, .... The data on which the setting is based may vary depending on the type of display device and the purpose of display, as long as it is information that brings about a change in the viewpoint. For example, as shown in the figure, the position and orientation information of the head mounted display 100 may be used, or the contents of a user operation related to the viewpoint change may be obtained via an input device (not shown). Alternatively, information on the viewpoint determined by an application such as a game program in response to a user operation may be obtained. These are collectively regarded as "information related to the change in viewpoint". Naturally, the latest viewpoint is also determined based on any of these pieces of information at a time step corresponding to the frame rate of the display.

[0028] 5 is a diagram for explaining changes in an image in response to changes in the viewpoint. The diagram shows a three-dimensional space of a display target in which objects 142a, 142b, and a background 144 are arranged, and a bird's-eye view of the viewpoint. If a shaded region 146 of the viewing frustum is the field of view from the reference viewpoint, the image data generating device 200 generates a reference image that shows the state in which objects present within region 146 are projected onto the view screen 14a. That is, the reference image shows the images of the surfaces of objects 142a, 142b, and background 144 that are shown by thick lines.

[0029] When the viewpoint moves as shown by the arrow, the area 148 of the view frustum corresponding to the latest viewpoint also changes, so that the change in the image projected onto the view screen 14b may not only involve movement or deformation, but also the appearance of an occluded part. In the example shown in the figure, for example, a surface 150 of the object 142a, which is occluded from the reference viewpoint and is shown by a dashed line, becomes visible. The appearance of an occluded part due to a change in viewpoint in this way is called disocclusion. As shown in the figure, disocclusion occurs not only for other surfaces of one object 142a, but also for objects and backgrounds behind objects in the foreground.

[0030] When disocclusion occurs at the latest viewpoint, the reference image does not contain information about the relevant part, so accurate representation cannot be achieved by simple image transformation alone. In other words, when translation or rotation transformation is performed on the image represented in the reference image, a so-called "hole" part, whose pixel value cannot be determined, is generated. As a countermeasure, it is considered that the head mounted display 100 partially draws the image from the latest viewpoint to fill the hole.

[0031] However, in this case, for the rendering process, it becomes necessary to transmit spatial information of the display target, model data of each object, and the like from the image data generating device 200 to the head mounted display 100. For example, in a system that displays a high-definition image by ray tracing, it is necessary to transmit all of spatial geometry information, texture data, light source information, material information, and the like to the head mounted display 100. As a result, the size of the transmitted data increases and reprojection takes time, leading to an increase in the delay time until display.

[0032] Therefore, in this embodiment, the image data generating device 200 speculatively generates color information for a portion where disocclusion may occur. That is, the image data generating device 200 partially renders an image from a viewpoint other than the reference viewpoint, adds the pixel values, i.e., color information, to the data of the reference image, and transmits the result to the head mounted display 100. In the reprojection process, the head mounted display 100 corrects the transmitted reference image based on the latest viewpoint, and determines pixel values ​​for the portion where disocclusion has occurred using the additionally transmitted color information.

[0033] In a general environment, the area ratio where disocclusion occurs is small relative to the entire image, and the increase in data size is less than a few percent. Therefore, the impact on data transfer time can be significantly reduced compared to when the entire model data is transmitted and drawn on the head mounted display 100 side. In addition, by generating additional information using the abundant resources of the image data generating device 200, the impact on the required time due to speculative processing can also be reduced. Furthermore, since an increase in the processing load on the head mounted display 100 can be avoided, it is easy to realize even in a head mounted display 100 with low processing performance. Hereinafter, the color information additionally generated by the image data generating device 200 is called "additional sampling data", and a viewpoint other than the reference viewpoint set to generate the additional sampling data is called an "additional viewpoint".

[0034] 6 shows the internal circuit configuration of the image data generating device 200. The image data generating device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and a main memory 226. These components are connected to each other via a bus 230. An input / output interface 228 is further connected to the bus 230.

[0035] Connected to the input / output interface 228 are a communication unit 232 consisting of a peripheral device interface such as USB or IEEE1394, or a network interface for wired or wireless LAN, a storage unit 234 such as a hard disk drive or nonvolatile memory, an output unit 236 that outputs data to a display device (not shown) or the like, an input unit 238 that inputs data from an input device (not shown) or the like, and a recording medium drive unit 240 that drives a removable recording medium such as a magnetic disk, optical disk, or semiconductor memory.

[0036] The CPU 222 executes an operating system stored in the storage unit 234 to control the entire image data generating device 200. The CPU 222 also executes various programs that are read from a removable recording medium and loaded into the main memory 226 or downloaded via the communication unit 232. The GPU 224 has a geometry engine function and a rendering processor function, and performs image drawing processing in accordance with drawing commands from the CPU 222. The main memory 226 is composed of a RAM (Random Access Memory), and stores programs and data necessary for processing.

[0037] 7 shows the internal circuit configuration of the head mounted display 100. The head mounted display 100 includes a CPU 120, a main memory 122, a display unit 124, and an audio output unit 126. These units are connected to each other via a bus 128. An input / output interface 130 is further connected to the bus 128. A communication unit 132 consisting of a wired or wireless LAN network interface, a motion sensor 134, and a stereo camera 110 are connected to the input / output interface 130.

[0038] The CPU 120 processes information acquired from each unit of the head mounted display 100 via the bus 128, and also supplies display image and audio data acquired from the image data generating device 200 to the display unit 124 and audio output unit 126. The main memory 122 stores programs and data necessary for processing in the CPU 120. The display unit 124 is composed of a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of a user wearing the head mounted display 100. As described above, stereoscopic vision may be realized by displaying a pair of parallax images in areas corresponding to the left and right eyes.

[0039] The audio output unit 126 is composed of speakers or earphones provided at positions corresponding to the user's ears when the head mounted display 100 is worn, and allows the user to hear audio. The communication unit 132 is an interface for sending and receiving data to and from the image data generating device 200, and establishes communication using a known communication technology. Image data sent from the image data generating device 200 is displayed on the display unit 124 under the control of the CPU 120 via the communication unit 132.

[0040] The motion sensor 134 includes a gyro sensor and an acceleration sensor, and acquires the angular velocity and acceleration of the head mounted display 100 at a predetermined rate. As shown in Fig. 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints in a field of view corresponding to the user's viewpoint. Measurement values ​​by the motion sensor 134, data of images captured by the stereo camera 110, or data related to the position and orientation of the head mounted display 100 obtained therefrom are transmitted to the image data generating device 200 via the communication unit 132 at a predetermined rate.

[0041] Fig. 8 shows a configuration of functional blocks of the image data generating device 200 and the head mounted display 100 in this embodiment. As described above, the image data generating device 200 may perform general information processing such as progressing an electronic game and distributing moving images, but Fig. 8 focuses in particular on the function of generating a display image.

[0042] Moreover, the functional blocks shown in Fig. 8 can be realized in terms of hardware by the configuration of the CPU, GPU, various memories, sensors, etc. shown in Fig. 6 or 7, and in terms of software by programs that perform various functions such as a data input function, a data storage function, an image processing function, a communication function, etc., loaded into memory from a recording medium, etc. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof, and are not limited to any one of them.

[0043] The image data generating device 200 includes a state information acquisition unit 260 that acquires information related to the position and posture of the head mounted display 100, a viewpoint setting unit 262 that sets a reference viewpoint, a reference image drawing unit 264 that draws a reference image, an additional data generation unit 268 that generates additional sampling data, a scene data storage unit 266 that stores model data of the scene to be displayed, and an image data transmission unit 270 that transmits data of the generated image to the head mounted display 100.

[0044] The state information acquisition unit 260 acquires data such as the measured values ​​of the motion sensor and the captured images by the stereo camera 110 from the head mounted display 100, and calculates the position and orientation of the head mounted display 100 at a predetermined rate. Note that the head mounted display 100 may calculate the position and orientation information, and the state information acquisition unit 260 may acquire it at a predetermined rate. The viewpoint setting unit 262 sets the position of the viewpoint and the direction of the line of sight relative to the space of the display object as a reference viewpoint at a predetermined rate so as to correspond to the position and orientation acquired by the state information acquisition unit 260. Note that the viewpoint setting unit 262 may predict the position of the viewpoint and the direction of the line of sight at the timing when an image is displayed on the head mounted display 100 based on the history of the movement of the viewpoint up to that point, and set this as the reference viewpoint.

[0045] The reference image rendering unit 264 renders a reference image representing the space to be displayed in the field of view from the reference viewpoint set by the viewpoint setting unit 262 at a predetermined rate. This image may be the result of information processing such as a game. The scene data storage unit 266 stores data of an object model required for image rendering and information related to the progress of a scene. The reference image rendering unit 264 renders a reference image using data stored in the scene data storage unit 266 and data of a captured image transmitted from the head mounted display 100 as necessary.

[0046] The reference image rendering unit 264 preferably renders the reference image by ray tracing, which acquires color information by physical calculation based on the intersection judgment between the object and a ray passing through each pixel of the view screen from the reference viewpoint and the reflection characteristics of light. This makes it possible to render a high-definition image that accurately reflects the material of the object surface and the state of the light source. At this time, the reference image rendering unit 264 also generates depth information to the object surface represented by each pixel, that is, the distance value in the depth direction. When rendering an image by ray tracing, depth information can be easily acquired in the ray intersection judgment.

[0047] The depth information is used for reprojection in the head mounted display 100. 3D warping is known as a technique for correcting an image into an image viewed from a different viewpoint using depth information (see, for example, Andre Schollmeyer and four others, "Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic Rendering," IEEE Transactions on Visualization and Computer Graphics, January 25, 2017, Vol. 23, Issue 4, pp. 1332-1341; Niko Wismann and three others, "Accelerated Stereo Rendering with Hybrid Reprojection-Based Rasterization and Adaptive Ray-Tracing," 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), March 22, 2020, pp. 828-835).

[0048] In the above document, 3D warping is used for the purpose of generating one image from the other of a stereo image. On the other hand, in this embodiment, this technology is applied to reprojection in the head mounted display 100. That is, reprojection is performed by first inversely projecting each pixel into a virtual three-dimensional space using depth information, and then reprojecting it onto a view screen corresponding to the latest viewpoint immediately before display.

[0049] In this case, instead of reprojecting all pixels, the image plane is divided according to a predetermined rule to perform projective transformation in units of small regions (hereafter referred to as "patches"), and the pixel values ​​inside the small regions are determined in a similar manner to texture mapping, thereby improving processing efficiency. In the above document, the image before warping is divided so that the minimum area patch is used for parts where the depth changes discontinuously, such as the contours of objects, and the area of ​​the patch increases the further away from the minimum area. This allows for more precise transformation of contours and other areas that are greatly affected by changes in viewpoint. This technology is called adaptive grid warping.

[0050] In this embodiment, adaptive grid warping is also applied to efficiently perform 3D warping, thereby realizing display with lower delay. In this case, the reference image rendering unit 264 first obtains the depth for each pixel of the reference image, and then divides the image plane so that the area of ​​the patch changes according to the spatial change. Qualitatively, as described above, the area of ​​the patch is made smaller the closer it is to a location where the depth changes discontinuously. Information on which the area of ​​the patch is controlled may be the extraction result of the edge in the reference image, geometry information of the display space, etc., in addition to the depth. The reference image rendering unit 264 then extracts depth information at the vertices of each patch and sets it as a transmission target.

[0051] The additional data generating unit 268 extracts a portion in the reference image where disocclusion may occur, and acquires color information of the portion as additional sampling data at a predetermined rate. For example, the additional data generating unit 268 sets an additional viewpoint at a position shifted from the reference viewpoint according to a predetermined rule, extracts a portion of the display target space visible from the additional viewpoint that is not shown in the reference image, and acquires color information of the portion.

[0052] The additional viewpoint may be set by predicting the viewpoint at the time of reprojection of the same frame in the head mounted display 100, that is, the latest viewpoint, from the history of changes in the reference viewpoint up to that point in time. In this case, the additional data generation unit 268 specifies the time difference between the time at which the reference viewpoint is set and the time at which the latest viewpoint is set in the head mounted display 100, by a handshake with the head mounted display 100 or the like. Then, the latest viewpoint is predicted as the additional viewpoint by extrapolating the time difference from the change in the reference viewpoint over time.

[0053] Alternatively, the additional data generating unit 268 may first identify a part of the object that is likely to be visible after an infinitesimal time based on the shape of the object represented in the reference image, and set a viewpoint from which the part is visible as the additional viewpoint. Here, the part of the object that is likely to be visible after an infinitesimal time is likely to exist near the contour of the image in the reference image. When adaptive grid warping is applied, a patch with a minimum area is formed near the contour of the image. The additional data generating unit 268 may use this to process a part on the object that corresponds to an area where the area of ​​the patch is equal to or smaller than a predetermined value, and obtain color information when the part is viewed from another viewpoint. Note that, even when the latest viewpoint is predicted as the additional viewpoint, the contour area of ​​the image in the reference image may be extracted in the same manner, and color information may be obtained for the corresponding part on the object.

[0054] In any case, the additional data generating unit 268 can generate detailed and accurate color information for the additional sampling data by drawing a partial image seen from the additional viewpoint by ray tracing based on the data stored in the scene data storage unit 266. The additional data generating unit 268 generates position information of the additional viewpoint, partial color information from the viewpoint, and depth information of the portion as the additional sampling data. Note that the number of additional viewpoints may be one or more.

[0055] The image data transmission unit 270 sequentially transmits the reference image drawn by the reference image drawing unit 264 and the additional sampling data generated by the additional data generation unit 268 to the head mounted display 100 in association with each other. When a stereoscopic image is displayed on the head mounted display 100, the reference image drawing unit 264 and the additional data generation unit 268 perform the same process on both the left eye image and the right eye image. Alternatively, the reference image drawing unit 264 and the additional data generation unit 268 may generate data on either the left eye image or the right eye image, or an image from an intermediate viewpoint, and the head mounted display 100 may generate a final stereoscopic image by 3D warping.

[0056] The head mounted display 100 includes a status information transmitting unit 274 that transmits information relating to the position and posture to the image data generating device 200, an image data acquiring unit 272 that acquires image data transmitted from the image data generating device 200, a reprojection unit 276 that generates a display image by reprojection, and a display unit 282 that displays the image after reprojection.

[0057] 7 and / or the motion sensor 134, and transmits captured images and measurement value data to the image data generating device 200 at a predetermined rate. Alternatively, the state information transmitting unit 274 may calculate the position and orientation of the head mounted display 100 from the data, and transmit the calculated data to the image data generating device 200. The image data acquiring unit 272 acquires the reference image data and additional sampling data transmitted from the image data generating device 200 at a predetermined rate.

[0058] The reprojection unit 276 includes at least one of the stereo camera 110 and the motion sensor 134 in FIG. 7, and specifies the latest position and orientation of the head mounted display 100, sets the latest viewpoint, and reprojects the reference image accordingly. In detail, the reprojection unit 276 includes a pixel displacement processing unit 278 and a pixel value determination unit 280. The pixel displacement processing unit 278 determines the displacement destination of the pixels of the reference image corresponding to the change from the reference viewpoint to the latest viewpoint. That is, based on the depth information, the pixels of the reference image are first reverse-projected into a three-dimensional space, and then reprojected onto the view screen corresponding to the latest viewpoint. According to the above-mentioned 3D warping, the vertices of the patch are first displaced, and the displacement destination of the internal pixels is determined by the UV coordinates.

[0059] The pixel value determination unit 280 determines pixel values ​​of the display image by referring to color values ​​at corresponding positions in the reference image based on the positional relationship of pixels before and after displacement. The above processing by the pixel displacement processing unit 278 and the pixel value determination unit 280 can be realized in a similar manner to texture mapping using the reference image as texture. However, the pixel value determination unit 280 determines image values ​​more accurately by adding additional sampling data to the reference. In other words, by inserting additional sampling data at appropriate positions for a patch that has been stretched due to a change in viewpoint, the resolution of the reference patch is artificially improved.

[0060] This makes it possible to prevent the image from being inappropriately stretched or its contours from becoming unclear. The reprojection unit 276 may also perform appropriate operations such as generating a stereo image, correcting for distortion of the eyepiece, and correcting color differences. The display unit 282 sequentially displays the display images corresponding to the latest viewpoints generated by the reprojection unit 276 through reprojection on a display panel.

[0061] 9 shows an example of the structure of image data transmitted from the image data generating device 200 to the head mounted display 100. The image data 30 shown in the figure represents one frame of data generated by the image data generating device 200, and includes reference image data and additional sampling data. Note that the figure merely shows the types of information included in one frame of image data, and does not stipulate the order of data transmitted from the image data generating device 200 to the head mounted display 100.

[0062] The reference image data includes reference viewpoint information 32, pixel values ​​34, and patch information 36. The reference viewpoint information 32 indicates the position coordinates in three-dimensional space of the reference viewpoint set when drawing the reference image. The pixel values ​​34 indicate the values ​​of all pixels resulting from the drawing of the reference image, i.e., color information, in pixel column order. The patch information 36 indicates position information of patches obtained by dividing the image plane, and is composed of, for example, the position coordinates and depth of each vertex on the image plane.

[0063] The additional sampling data includes additional viewpoint information 38a, 38b and pixel information 40a, 40b for each additional viewpoint. The additional viewpoint information 38a, 38b indicates the position coordinates of the additional viewpoint in three-dimensional space. The pixel information 40a, 40b is composed of color information of a point on an object as seen from the additional viewpoint and position information of the point. Here, the "point" corresponds to one pixel of the view screen corresponding to the additional viewpoint, and is a location on an object that intersects with a ray in ray tracing.

[0064] The position information of the point may be represented by the position coordinates and depth on the view screen corresponding to the additional viewpoint, or may be represented by the position coordinates in three-dimensional space. In the latter case, the position information of the additional viewpoint may be omitted. There is no particular limit to the number of additional viewpoints or the number of points from which color information is obtained for one additional viewpoint. Considering that even at the same point on the object, a color change strictly occurs depending on the angle of the line of sight, the closer the additional viewpoint is to the latest viewpoint, the more accurately the disocclusion part can be expressed. On the other hand, the fewer the additional viewpoints and the points from which color information is obtained, the less the processing load. Therefore, the number of additional viewpoints and points may be optimized according to the accuracy required for display, the processing performance of the image data generating device 200, the communication bandwidth, the prediction accuracy of the viewpoint change, and the like.

[0065] 10 is a diagram for explaining the concept of a method in which the reprojection unit 276 of the head mounted display 100 generates a display image. The reference image 52 transmitted from the image data generating device 200 is a drawing of an object 50 existing in a three-dimensional space to be displayed, as viewed from a reference viewpoint 54. In the figure, a square image 56 is shown, with face A of the cubic object 50 being visible. In addition to the reference image 52, the image data generating device 200 acquires, as additional sampling data, for example, color information of face B, which is not visible from the reference viewpoint, of an image of the object 50 viewed from an additional viewpoint 58, and transmits the information in association with the data of the reference image.

[0066] To avoid complexity, the figure does not show a view screen corresponding to the additional viewpoint 58, but a view screen can be set in the same way as in normal ray tracing, and an image can be drawn for an area of ​​the space to be displayed that corresponds to a predetermined range of the contour of the image 56. Alternatively, the three-dimensional position coordinates of the point where a ray generated from the additional viewpoint 58 intersects on the surface B of the object 50, and color information obtained therefrom, can be used as additional sampling data. The reprojection unit 276 of the head mounted display 100 sets the latest viewpoint 60 based on the latest position and orientation, and then changes the image 56 appearing in the reference image 52 by 3D warping.

[0067] That is, based on the depth information, a surface A in three-dimensional space is virtually formed from the image 56 (for example, arrow a), and is reprojected onto the view screen 62 corresponding to the latest viewpoint 60 (for example, arrow b). As a result, an image 64 of the surface A of the object 50 is formed on the view screen 62. The conversion from position coordinates on the plane of the reference image to three-dimensional position coordinates, and the conversion from the three-dimensional position coordinates to position coordinates on the view screen 62 can actually be calculated at once using a general conversion matrix. Also, as described above, the vertices of the patch obtained by dividing the reference image 52 are converted, and for the pixels inside the patch, pixel values ​​of the image 64 and the like can be determined by sampling pixel values ​​of the corresponding positions in the reference image 52.

[0068] On the other hand, if surface B of object 50 is also visible from the latest viewpoint 60, reprojection unit 276 uses the additional sampling data to form its image 66. If the additional sampling data represents an image of surface B on the view screen corresponding to additional viewpoint 58, surface B is virtually formed in three-dimensional space based on the depth information, and warping is performed to project it onto the view screen 62 (e.g., arrows c and d). If color information is associated with the three-dimensional position coordinates on surface B, that point is projected onto the view screen 62 (e.g., arrow d).

[0069] This allows information about surface B of object 50, which is not included in reference image 52, to be accurately added by reprojection, making it possible to display a fine image with high accuracy. Note that for ease of understanding, the size of image 66 represented by the additional sampling data is exaggerated in the figure, but in reality it has a width of about one pixel to several pixels. Therefore, it becomes possible to display an image that achieves both responsiveness and image quality without significantly changing the amount of data transmitted from image data generating device 200 by using the additional sampling data.

[0070] Fig. 11 is a diagram for explaining the effect of reprojection using additional sampling data at the pixel level. The left side of the diagram shows four 2 x 2 pixels 72a, 72b, 72c, and 72d in the reference image. Of these, pixels 72a and 72b represent the image of a white object in the foreground, and pixels 72c and 72d represent the image of a gray background. In the reference image, the color of horizontally adjacent pixels 72a / 72c or pixels 72b / 72d changes significantly, so the actual contour 74 of the object is visible with pixel-level accuracy.

[0071] When 3D warping is performed in the reprojection unit 276 of the head mounted display 100, the area 70 surrounded by four adjacent pixels 72a, 72b, 72c, and 72d in the reference image may be stretched as shown on the right side of the figure. That is, since the foreground image has a larger displacement than the background image, the areas 76a and 76b are stretched in the horizontal direction in response to the horizontal movement of the viewpoint. Note that, according to the above-mentioned adaptive grid warping, the smallest patch is formed by the contour of the object, so the area 70 corresponds to the patch before warping, and the areas 76a and 76b correspond to the patch after warping.

[0072] Here, the area 76a shows the color distribution determined after warping without using additional sampling data, and the area 76b shows the color distribution determined after warping with additional sampling data. In reality, in both cases, pixel values ​​are determined for the pixel rows inside. In the case of the area 76a without using additional sampling data, only the reference image is used to determine the pixel values. Since only information that pixels 72a and 72b are white and pixels 72c and 72d are gray is obtained from the reference image, a wide range of gradation is obtained by filtering according to the distance from these four pixels. In other words, the color distribution is stretched as the area is stretched, and color bleeding, such as a blend of the background and foreground, is likely to occur.

[0073] On the other hand, when the color information of points 78a, 78b, and 78c is used as the additional sampling data, points 78a and 78b on the object side of the actual contour 74 are controlled to be white, and point 78c on the background side is controlled to be gray, so that the range in which colors bleed even after filtering can be significantly narrowed. The illustrated example shows the distribution of two colors, foreground and background, but as shown in Figure 10, even if a third color appears due to a newly seen surface, that color can naturally be displayed at the corresponding position. As a result, deterioration of image quality caused by reprojection, such as the original image being stretched or the boundary with another object becoming unclear, can be suppressed.

[0074] Next, the operation of the image display system that can be realized by the above configuration will be described. Fig. 12 is a flowchart showing the processing procedure in which the image data generating device 200 generates and transmits image data. First, the state information acquiring unit 260 acquires state information related to the position and orientation of the head mounted display 100 (S10). Next, the viewpoint setting unit 262 sets a reference viewpoint so as to correspond to the position and orientation (S12). Next, the reference image drawing unit 264 sets a view screen for the reference viewpoint and draws a reference image representing an object by ray tracing or the like (S14).

[0075] Furthermore, the reference image rendering unit 264 divides the image plane into patches based on the depth information of the reference image and extracts the depth information of the vertices (S16). Meanwhile, the additional data generating unit 268 sets an additional viewpoint different from the reference viewpoint (S18) and generates additional sampling data by acquiring color information of partial points on the object seen from there by ray tracing or the like (S20). As described above, the additional data generating unit 268 may set the additional viewpoint by predicting the latest viewpoint based on the history of viewpoint movement, or may specify a part where disocclusion is likely to occur from the shape of the object and set a viewpoint where the part can be viewed from a different angle as the additional viewpoint.

[0076] The portion where disocclusion is likely to occur may be an area where a patch of a predetermined area or less is formed as a result of dividing the reference image into patches in S16. The additional data generating unit 268 acquires color information of points on the object that are not shown in the reference image and are in the vicinity of the area as additional sampling data. Then, the image data transmitting unit 270 transmits the reference image data and the additional sampling data in association with each other to the head mounted display 100 as illustrated in FIG. 9 (S22). While it is not necessary to end the display due to a user operation or the like, the image data generating device 200 repeats the processes from S10 to S22 on a frame-by-frame basis (N in S24). When it is necessary to end the display, the image data generating device 200 ends all the processes (Y in S24).

[0077] 13 is a flowchart showing a processing procedure for the head mounted display 100 to display an image. First, the image data acquisition unit 272 acquires data of a reference image and additional sampling data from the image data generation device 200 in association with each other (S30). Next, the pixel displacement processing unit 278 of the reprojection unit 276 sets a latest viewpoint based on the latest position and orientation of the head mounted display 100, and acquires a displacement destination of pixels of the reference image corresponding to a change in viewpoint from the reference viewpoint (S32). This processing may be realized by 3D warping of the patch.

[0078] Next, the pixel value determination unit 280 determines the pixel value of the display image using the reference image and the additional sampling data (S34). Specifically, the pixel value determination unit 280 determines the pixel value of the display image using color information of the corresponding position in the reference image based on the positional relationship of the pixel before and after the displacement according to the change in the viewpoint. In addition, for the part where disocclusion occurs, the pixel value is determined using color information of the corresponding point in the additional sampling data. Both may be performed separately or simultaneously. In the former case, the "hole" part where the pixel value cannot be determined due to disocclusion is filled later using the additional sampling data. In the latter case, when the position on the reference image corresponding to the pixel of the display image is referred to, additional color information is inserted and filtering is performed. The reprojection unit 276 may also perform other corrections necessary for display as appropriate.

[0079] The display unit 282 sequentially outputs the data of the display image generated by the reprojection unit 276 through reprojection to the display panel for display (S36). As long as there is no need to end the display due to a user operation or the like, the head mounted display 100 repeats the processes from S30 to S36 on a frame-by-frame basis (N in S38). When it becomes necessary to end the display, the head mounted display 100 ends all the processes (Y in S38).

[0080] Figures 14 to 18 show the actual rendering results. First, (a) in Figure 14 is a reference image, which is the result of rendering a three-dimensional space in which cubes and cylinders are arranged. (b) is an image showing the same space as seen from a different viewpoint, which is rendered in the same way as (a). In what follows, we will assume that the viewpoint from which the image in (b) is obtained is the latest viewpoint, and show how an image equivalent to that image is created from the image in (a) by reprojection.

[0081] Fig. 15 shows the state in which the reference image shown in Fig. 14(a) is divided into patches. In this example, adaptive grid warping is assumed. Therefore, the dividing boundaries of the patches shown by white lines are set at higher density as they approach the contour of the object. The patches on the contour have the minimum area with 4 adjacent pixels as vertices, as shown in Fig. 11, for example.

[0082] FIG. 16(a) shows a state where the patch formed as in FIG. 15 is warped based on the latest viewpoint. FIG. 16(b) shows a state where pixel values ​​of the reference image are mapped onto the warped patch. In this image, areas 80 and 82 are disocclusion areas that are not shown in the reference image shown in FIG. 14(a). FIG. 17 shows the result of determining pixel values ​​for the disocclusion areas by filtering color information of the reference image. As described for the area 76a in FIG. 11, color bleeding occurs in the disocclusion area, the boundary between the foreground and background becomes unclear, and the faces of the cubic object become indistinguishable.

[0083] Fig. 18 shows the result of determining pixel values ​​using the reference image and additional sampling data according to this embodiment. As a result of obtaining appropriate color information for the disocclusion area using the additional sampling data, the boundary between the foreground and background becomes clear, and the newly seen faces of the cube are expressed with high accuracy. In other words, it can be seen that an image close to the original image shown in Fig. 14(b) can be obtained by reprojection.

[0084] According to the present embodiment described above, the image data generating device generates and transmits a reference image at a reference viewpoint according to viewpoint manipulation on the display side, and generates a display image corresponding to the latest viewpoint on the display device by reprojection. At this time, the image data generating device speculatively generates color information of an object viewed from a different viewpoint for an area where disocclusion may occur due to a viewpoint change from the reference viewpoint, and transmits the color information as additional sampling data. As a result, when disocclusion actually occurs during reprojection on the display device, accurate color information can be inserted into the additional sampling data, and an image from the nearest viewpoint can be displayed with high accuracy.

[0085] Reprojection uses 3D warping on a patch-by-patch basis by dividing the image surface. By dividing the patch so that the area becomes smaller the closer it is to the contour of the object's image, more accurate warping can be achieved near the contour, where accurate representation is required, and areas suitable for additional sampling can be identified along with the patch division. As a result, changes in the image near the contour that accompany changes in viewpoint can be represented efficiently and accurately. The use of additional sampling data increases the accuracy of reprojection, so the image generation rate can be set independently of the display frame rate. As a result, it is possible to draw high-quality reference images over a long period of time, reduce the size of the transferred data, and display high-definition images with good responsiveness.

[0086] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0087] 100 head mounted display, 110 stereo camera, 120 CPU, 122 main memory, 124 display unit, 132 communication unit, 134 motion sensor, 200 image data generating device, 222 CPU, 224 GPU, 226 main memory, 232 communication unit, 234 memory unit, 260 status information acquisition unit, 262 viewpoint setting unit, 264 reference image drawing unit, 266 scene data memory unit, 268 additional data generation unit, 270 image data transmission unit, 272 image data acquisition unit, 274 status information transmission unit, 276 reprojection unit, 278 pixel displacement processing unit, 280 pixel value determination unit, 282 display unit. [Industrial Applicability]

[0088] As described above, the present invention can be used in various information processing devices such as image data generating devices, electronic content servers, image distribution servers, head-mounted displays, game devices, image display devices, mobile terminals, and personal computers, as well as image processing systems that include any of these.

Claims

1. a viewpoint setting unit that sets a reference viewpoint at a predetermined time step based on information related to a change in viewpoint with respect to a space of a display object; a reference image rendering unit that renders a reference image representing the space in a field of view corresponding to the reference viewpoint; an additional data generating unit that acquires color information of an occluded portion of the space that is not shown in the reference image as additional sampling data after setting a viewpoint different from the reference viewpoint; a transmission unit that transmits the reference image and the additional sampling data to a display device in association with each other; Equipped with The image data generating device, characterized in that the additional data generating unit predicts a viewpoint when the display device performs display processing using the reference image, and acquires the color information from the predicted viewpoint.

2. 2. The image data generating device according to claim 1, wherein the additional data generating section acquires the color information after setting a plurality of viewpoints.

3. 3. The image data generating device according to claim 1, wherein the additional data generating unit includes information relating to the position of the occluded portion, from which the color information is obtained, in the additional sampling data.

4. the reference image rendering unit includes, in the data of the reference image, position information of vertices of patches obtained by dividing an image plane by areas based on a depth distribution of an object represented in the reference image, and position information of the reference viewpoint; 4. The image data generating device according to claim 1, wherein the additional data generating section extracts a target from which the color information is to be obtained based on an area of ​​the patch.

5. The image data generating device according to any one of claims 1 to 4, characterized in that the additional data generating unit obtains the color information by rendering an area in the space corresponding to an area of ​​a predetermined range including an outline of an image in the reference image from the different viewpoints by ray tracing.

6. A display device that changes a field of view to display a space of a display object based on information related to a change in viewpoint, an image data acquisition unit that acquires data of a reference image that represents the space in a field of view that corresponds to a reference viewpoint, and additional sampling data that includes color information of an occluded portion of the space that is not represented in the reference image; a reprojection unit that converts the reference image into an image in a field of view corresponding to a latest viewpoint, and generates a display image by adding color information using the additional sampling data; A display unit that outputs the display image; A display device comprising:

7. the image data acquisition unit acquires position information of vertices of patches obtained by dividing a plane of the reference image and position information of the reference viewpoint in association with the reference image; 7. The display device according to claim 6, wherein the reprojection section performs conversion corresponding to a change in a view screen from the reference viewpoint to the latest viewpoint in units of the patch.

8. 8. The display device according to claim 6, wherein the reprojection unit determines pixel values ​​of a display image by inserting color information indicated by the additional sampling data in filtering color values ​​with reference to the reference image.

9. The display device according to claim 6 or 7, characterized in that the reprojection unit determines pixel values ​​of the display image by referring to the reference image, and then determines pixel values ​​for parts where pixel values ​​cannot be determined due to occlusion using color information indicated by the additional sampling data.

10. An image display system including an image data generating device and a display device, which changes a field of view based on information related to a change in viewpoint to display a space of a display target, The image data generating device includes: a viewpoint setting unit for setting a reference viewpoint at a predetermined time step; a reference image rendering unit that renders a reference image representing the space in a field of view corresponding to the reference viewpoint; an additional data generating unit that acquires color information of an occluded portion of the space that is not shown in the reference image as additional sampling data after setting a viewpoint different from the reference viewpoint; a transmitting unit that transmits the reference image and the additional sampling data in association with each other, The display device includes: an image data acquisition unit that acquires the reference image and the additional sampling data; a reprojection unit that converts the reference image into an image in a field of view corresponding to a latest viewpoint, and generates a display image by adding color information using the additional sampling data; A display unit that outputs the display image; An image display system comprising:

11. setting a reference viewpoint at a predetermined time step based on information relating to a change in viewpoint with respect to a space of a display object; drawing a reference image representing the space in a field of view corresponding to the reference viewpoint; acquiring color information of an occluded portion of the space that is not shown in the reference image as additional sampling data after setting a viewpoint different from the reference viewpoint; transmitting the reference image and the additional sampling data to a display device in association with each other; Including, The image data generating method is characterized in that the step of acquiring the color information as additional sampling data includes predicting a viewpoint when the display device performs display processing using the reference image, and acquiring the color information from the predicted viewpoint.

12. A display device that changes a field of view to display a space of a display object based on information related to a change in a viewpoint, A step of acquiring data of a reference image representing the space in a field of view corresponding to a reference viewpoint, and additional sampling data including color information of an occluded portion not represented in the reference image; generating a display image by transforming the reference image into an image with a field of view corresponding to a current viewpoint and adding color information using the additional sampling data; outputting the display image; 2. An image display method comprising:

13. A function of setting a reference viewpoint at a predetermined time step based on information related to a change in viewpoint with respect to the space of the display object; a function of drawing a reference image representing the space in a field of view corresponding to the reference viewpoint; a function of acquiring color information of an occluded portion of the space that is not shown in the reference image as additional sampling data after setting a viewpoint different from the reference viewpoint; a function of transmitting the reference image and the additional sampling data to a display device in association with each other; This is realized on a computer. A computer program characterized in that the function of acquiring the color information as additional sampling data predicts the viewpoint when the display device performs display processing using the reference image, and acquires the color information from the predicted viewpoint.

14. A display device that changes a field of view to display a space of a display object based on information related to a change in a viewpoint, A function of acquiring data of a reference image representing the space in a field of view corresponding to a reference viewpoint, and additional sampling data including color information of an occluded portion not represented in the reference image; a function of converting the reference image into an image with a field of view corresponding to a current viewpoint, and generating a display image by adding color information using the additional sampling data; a function of outputting the display image; A computer program characterized by causing a computer to execute the above.

Images