Image transmission device and image transmission method

Abstract

This image transmission device sets a viewscreen 150 which is curved around a visual line 162 of a user 160 as the central axis, and creates an original image 152 in which the magnification rate of an image is amplified toward the center of the image, by changing the density, according to an angle formed relative to the central axis, in a direction in which color information is acquired in a three-dimensional space of an object to be displayed. A display control unit sets a viewscreen 154, determines a pixel value by performing sampling from appropriate positons in the original image 152, and creates a display image 156 having distortion corresponding to an ocular lens.

Description

Technical Field

[0001] The present invention relates to an image transmission device that displays an image by data transmission, and an image transmission method.

Background Art

[0002] Display devices of UHDTV (Ultra HDTV) having a resolution 4 times and 16 times that of HDTV (High Definition Television) have been put into practical use. In addition, a display system that gives a sense of immersion in the image world by showing all-round video in a free field of view corresponding to the line of sight of a user wearing a head-mounted display has also become widespread. With the advent of 5G (the fifth-generation mobile communication system), data transmission with wide bandwidth and low latency has become possible, and high-quality image content can be enjoyed regardless of the environment.

Summary of the Invention

Problems to be Solved by the Invention

[0003] As described above, in various electronic contents, the resolution and viewing angle of an image, and thus the number of pixels constituting one frame, tend to increase more and more. However, such an increase in data volume causes various problems such as a shortage of transmission bandwidth, an increase in memory cost and calculation cost, and an increase in power consumption. In particular, since it takes time to process and transmit data for one frame, the frame rate may not increase, but rather the image quality may deteriorate, or the change in the field of view may be delayed with respect to the movement of the user's line of sight, giving a sense of discomfort.

[0004] The present invention has been made in view of such problems, and an object thereof is to provide a technology that can display an image with a large resolution and viewing angle easily and with high quality.

Means for Solving the Problems

[0005] To solve the above problems, an aspect of the present invention relates to an image transmission device. This image transmission device is an image transmission device that transmits data of an original image used for generating a display image in parallel with the display, and includes one or more processors having hardware. The one or more processors determine the value of each pixel on the view screen of the original image by obtaining the color information of the corresponding azimuth in the space of the display target to generate the original image. At this time, the color information is obtained after changing the interval of the azimuth so as to depend on the angle formed with the central axis of the view screen, and the data of the original image is transmitted to a device that generates a display image.

[0006] Another aspect of the present invention relates to an image transmission method. This image transmission method is an image transmission method that transmits data of an original image used for generating a display image in parallel with the display. The value of each pixel on the view screen of the original image is determined by obtaining the color information of the corresponding azimuth in the space of the display target to generate the original image. At this time, the color information is obtained after changing the interval of the azimuth so as to depend on the angle formed with the central axis of the view screen, and the data of the original image is transmitted to a device that generates a display image.

[0007] In addition, any combination of the above components, and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a computer program, a data structure, a recording medium, etc. are also effective as aspects of the present invention.

Advantages of the Invention

[0008] According to the present invention, an image with a high resolution and a large viewing angle can be displayed with high quality and easily.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0010] FIG. 1 shows a configuration example of an image display system to which the present embodiment can be applied. The image display system 1 includes display control devices 10a, 10b, 10c that display images according to user operations, and an image transmission device 20 that provides image data for display. Input devices 14a, 14b, 14c for user operations and display devices 16a, 16b, 16c for displaying images are connected to the display control devices 10a, 10b, 10c, respectively. The display control devices 10a, 10b, 10c and the image transmission device 20 can establish communication via a network 8 such as a WAN (World Area Network) or a LAN (Local Area Network).

[0011] The display control devices 10a, 10b, 10c, the display devices 16a, 16b, 16c, and the input devices 14a, 14b, 14c may be connected either by wire or wirelessly. Alternatively, two or more of these devices may be integrally formed. For example, in the figure, the display control device 10b is connected to a head-mounted display that is the display device 16b. The head-mounted display can change the viewing field of the displayed image according to the movement of the user wearing it on the head, and thus also functions as the input device 14b.

[0012] Also, the display control device 10c is a mobile terminal and is integrally configured with the display device 16c and the input device 14c which is a touch pad covering its screen. Thus, the external shape and connection form of the illustrated devices are not limited. The number of the display control devices 10a, 10b, 10c and the image transmission device 20 connected to the network 8 is also not limited. For example, the image transmission device 20 may be a cloud server or a content server in a system such as cloud gaming. Hereinafter, the display control devices 10a, 10b, 10c may be collectively referred to as the display control device 10, the input devices 14a, 14b, 14c may be collectively referred to as the input device 14, and the display devices 16a, 16b, 16c may be collectively referred to as the display device 16.

[0013] The input device 14 may be any one or a combination of common input devices such as a controller, a keyboard, a mouse, a touch pad, and a joystick, and supplies the content of the user operation to the display control device 10. The display device 16 may be a common display such as a liquid crystal display, a plasma display, an organic EL display, a wearable display, and a projector, and displays the image output from the display control device 10.

[0014] The image transmission device 20 provides the display control device 10 with the data of the content accompanied by the image display. The type of the content is not particularly limited, and may be any of electronic games, viewing images, television broadcasts, electronic conferences, video chats, and the like. The display control device 10 basically realizes the display process while acquiring the content data used for display from the image transmission device 20. Alternatively, the image transmission device 20 and the display control device 10 may cooperate to generate the content data to disperse the load such as the drawing process.

[0015] FIG. 2 shows an example of the display image assumed in the present embodiment. The image transmission device 20 prepares the data of the wide-angle image 130 that represents the space to be displayed with a wide-angle view such as an omnidirectional view. Since it becomes more difficult to represent the image world in a plane as the viewing angle becomes wider, the wide-angle image 130 actually needs to be represented by a sphere, a polyhedron, or in the form of an orthomorphic cylindrical projection method having latitude / longitude axes. The display device 16 outputs the display image 132 generated using a part of the data of the wide-angle image 130 to the display panel. When the display device 16 is a head-mounted display, the viewing field of the display image 132 is changed so as to correspond to the movement of the user's head.

[0016] At this time, as in the case of the display image 132, stereoscopic vision can be realized by displaying stereoscopic images with parallax in the left and right regions corresponding to the user's left and right eyes. Further, when the head-mounted display includes an eyepiece lens for expanding the field of view, the display image 132 is given an inverse distortion corresponding to the distortion aberration and chromatic aberration of the eyepiece lens so that an undistorted image is visually recognized when viewed through the eyepiece lens. For example, in the case of a lens in which the four sides of the image appear concave like a spool, the left and right images of the display image 132 are each curved in a barrel shape.

[0017] Hereinafter, an image given a distortion corresponding to the eyepiece lens will be referred to as a "distorted image". The distorted image basically has the characteristic that the amount of image distortion increases as the distance from the position intersecting the optical axis of the eyepiece lens in the image plane increases. Note that the wide-angle image 130 may be an image drawn by computer graphics, a photographed image, or a combination thereof. The wide-angle image 130 may also be a still image or a moving image with the still images as frames in chronological order. For example, the object represented by the wide-angle image 130 may move or change based on the operation of the user viewing the display image 132.

[0018] The data source of the wide-angle image 130 may be, in addition to the image transmission device 20, a drawing mechanism inside the display control device 10 or an imaging device connected to the display control device 10. Hereinafter, the image transmission device 20 will mainly be described as the data source. In any case, the closer the angle of view of the wide-angle image 130 approaches a full circle, the wider the range within which the user can freely look around, enhancing the sense of immersion in the image world. On the other hand, in order to realize a more immersive video experience, it is desirable to prepare a high-definition wide-angle image 130. However, the larger the angle of view, the larger the data volume. As a result, the costs of transmission, storage, and calculation increase, and delays until display are likely to occur.

[0019] Therefore, in the present embodiment, the data format of an image that serves as the basis for the display image 132, such as the wide-angle image 130 (hereinafter referred to as the "original image"), is devised to achieve both low-latency display and high-definition image visibility. Specifically, from the beginning, the same distortion as that of the display image 132 is applied to the original image. That is, the image transmission device 20 generates a wide-angle distorted image assuming the user's line of sight direction as the optical axis and transmits it to the display control device 10. The display control device 10 generates the display image 132 using a part of the transmitted data and outputs it to the display device 16.

[0020] FIG. 3 is a diagram for explaining the distribution of the amount of information in the image plane when displaying a distorted image. The view screen 414 is a screen for expressing an image of central projection without distortion and is used for generating an image to be displayed on a general flat panel display. The view screen 426 is a screen for expressing a distorted image based on an eyepiece lens. The figure shows a state where both screens are viewed from the side together with the user's viewpoint 424.

[0021] The view screen 414 is configured by a plane having a predetermined angular field, for example, with the user's line of sight as the central axis C. In this case, the image of the object 425 to be displayed is represented in a state uniformly reduced at a reduction ratio corresponding to the distance from the view screen 414 regardless of the height from the central axis C. On the other hand, the distorted image assuming an eyepiece lens has the same equidistant projection as the image taken by a fish-eye lens or a property similar thereto. As a result, the view screen 426 has a curved shape as shown in the figure. The detailed shape of the view screen 426 depends on the lens design and is not particularly limited in the present embodiment. Hereinafter, mainly assuming equidistant projection, it will be described as a spherical view screen.

[0022] As is clear from the figure, in the region of the angular range 428 near the optical axis (central axis C) of the view screen 426, the area difference from the corresponding region of the view screen 414 is small, whereas the area ratio with respect to the view screen 414 decreases as the angular range moves away from the optical axis. For this reason, in the central region 434 of the image, there is almost no difference in the size of the image between the central projection image and the distorted image, whereas in the peripheral regions 432, 426, the image in the central projection image is significantly reduced in the distorted image. That is to say, it can be said that a part of the image represented by the central projection contains useless information that is not reflected in the distorted image.

[0023] Such a difference in the amount of information with the distorted image becomes even more prominent not only in the central projection image but also in the orthographic cylindrical projection image in which the image is enlarged above and below the image plane. Based on this, in the present embodiment, the original image is prepared in the form of a distorted image without going through images in forms such as central projection and orthographic cylindrical projection. Thereby, generation and transmission of information that becomes useless in the finally displayed distorted image are suppressed as much as possible. As described above, qualitatively, the magnification of the image in the distorted image changes depending on the distance from the position intersecting the optical axis. Utilizing this characteristic, in the present embodiment, an original image is generated in which the magnification of the image is further changed from the equidistant projection image depending on the distance from the position intersecting the central axis C corresponding to the line of sight.

[0024] As a human visual characteristic, it is known that the visual function of recognizing fineness decreases as the distance from the central visual field corresponding to the fovea increases. Therefore, when the central visual field is in the central region 434 of the display image as shown in the figure, a high-definition image is represented in that region, and even if the fineness is lowered as the distance from there increases, it will appear to be of high image quality in terms of visual recognition. Therefore, the image transmission device 20 represents the image with a magnification distribution that further amplifies the characteristic of the equidistant projection in which the magnification of the image decreases from the position intersecting the central axis C toward the periphery on the image, thereby appropriately distributing the amount of information with respect to the plane of the original image.

[0025] The display control device 10 generates a distorted image having the original magnification distribution as a display image by sampling the original image in which the in-plane distribution of the magnification is amplified in this way. As a result, a high-definition image can be represented in the central region 434, the data amount of the original image can be further reduced in portions that are less likely to be noticed by the user, and both high image quality and low latency can be achieved. Hereinafter, in the original image and the display image, a position that intersects the central axis C or the optical axis of the eyepiece lens is referred to as the "image center".

[0026] The above-described effects of the present embodiment become more prominent as the image center of the original image is closer to the image center of the display image and the user's line-of-sight direction. In a form in which the display device 16 is a head-mounted display and the visual field is changed in accordance with the movement of the user's head, the change in the user's line of sight is likely to be reflected in the movement of the head. Therefore, the image transmission device 20 continuously acquires the measurement results of the position and posture of the user's head from the display control device 10 or the like, and adaptively determines the direction in which the center of the face faces as the image center of the original image.

[0027] On the other hand, even in a display form in which the visual field is determined regardless of the user's movement, in view of the fact that the user's line of sight tends to gather at the center of the display image, it can be said that a similar effect can be obtained by using the image center of the display image as the image center of the original image. In this case, the image transmission device 20 may continuously acquire the display visual field determined by itself or the display visual field determined by the display control device 10 based on the display visual field determined by itself, and determine the center thereof as the image center of the original image. In the following description, mainly, an aspect in which the display visual field is changed in accordance with the movement of the user's head will be described.

[0028] Figure 4 shows the internal circuit configuration of the display control device 10. The display control device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are interconnected via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 is connected to a communication unit 32 that establishes communication with an image transmission device 20 and the like, a storage unit 34 such as a hard disk drive or a non-volatile memory, an output unit 36 that outputs data to the display device 16, an input unit 38 that inputs data from an input device 14 or an imaging device, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory. The communication unit 32 constitutes, for example, a peripheral device interface such as USB or IEEE1394, or a network interface for a wired or wireless LAN.

[0029] The CPU 22 controls the entire display control device 10 by executing an operating system stored in the storage unit 34. The CPU 22 also executes various programs read from a removable recording medium and loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 performs drawing processing according to a drawing command from the CPU 22 and stores the display image in a frame buffer (not shown). Then, the display image stored in the frame buffer is converted into a video signal and output to the output unit 36. The main memory 26 is constituted by a RAM (Random Access Memory) and stores programs and data necessary for processing. The circuit configuration of the image transmission device 20 may be substantially the same. In this case, the data of the acquired or generated image is transmitted to the display control device 10 via the communication unit 32.

[0030] FIG. 5 shows the configuration of the functional blocks of the image transmission device 20 and the display control device 10 in the present embodiment. In the present embodiment, the entity that generates the original image data that is the basis of the display image is not limited to the image transmission device 20, and at least a part of it may be borne by a local device on the display side such as the display control device 10 or the imaging device. However, hereinafter, it will mainly be described as the image transmission device 20. Further, the entity that forms the final display image from the image data is not limited to the display control device 10, and at least a part of it may be borne by the display device 16. However, hereinafter, it will mainly be described as the display control device 10.

[0031] Also, the image transmission device 20 and the display control device 10 may perform general information processing such as advancing an electronic game or outputting sound. However, in FIG. 5, particular attention is paid to the function of transmitting and displaying image data. The functional blocks shown in FIG. 5 can be realized in terms of hardware by configurations such as the CPU 22, GPU 24, and main memory 26 shown in FIG. 4, and in terms of software, by a program that exhibits various functions such as a data input function, a data holding function, an image processing function, and a communication function, which is loaded from a recording medium or the like into the main memory 26. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to any one of them.

[0032] The image transmission device 20 includes a gaze information acquisition unit 50 that acquires information related to the user's gaze, an original image data generation unit 52 that generates the data of the original image, and a data transmission unit 56 that transmits the data of the original image to the display control device 10. The gaze information acquisition unit 50 acquires information related to the direction of the user's gaze with respect to the image being displayed on the display device 16 from the display control device 10 at a predetermined rate. When the display device 16 is a head-mounted display and the visual field is changed in accordance with the movement of the user's head, the gaze information acquisition unit 50 acquires the information on the position and orientation of the head as the gaze information.

[0033] The original image data generation unit 52 generates data of the original image at a predetermined rate. For example, the original image data generation unit 52 draws the original image at a predetermined rate by computer graphics based on the model data and game programs stored therein. In this case, the original image data generation unit 52 may acquire the content of the user operation for a game or the like from the display control device 10. Alternatively, the original image data generation unit 52 may generate the original image using an image captured by an imaging device connected to the image transmission device 20, a content image prepared separately, or the like.

[0034] In any case, the original image data generation unit 52 generates a wide-angle distorted image with the position corresponding to the user's line-of-sight direction as the center of the image. For example, the original image data generation unit 52 draws the original image using ray tracing technology. Ray tracing is a method of generating virtual light rays (rays) passing through each pixel on the view screen from the viewpoint and determining the pixel value based on the color information of the arrival destination. Ray tracing has generally been regarded as a process with a large amount of calculation and heavy load. However, compared with rasterization that projects polygons, it has the advantage that much of the intermediate buffer memory such as the depth buffer is not required.

[0035] In recent years, depending on the content of the image, it has become possible to operate with low weight and low latency in terms of the amount of calculation and resources used. For example, since ray tracing enables drawing and transmission on a pixel-by-pixel basis, measures such as minimizing data updates for backgrounds that do not contribute to improving the image quality can reduce the load of the drawing process and transmission.

[0036] Alternatively, the original image data generation unit 52 may draw the original image using a software rasterizer. A software rasterizer is a technology that software-implements rasterization for projecting a 3D model to generate an image (see, for example, Tobais Ritschel, et al., "Perceptual rasterization for head-mounted display image synthesis", ACM, January 2019, Transactions on Graphics, Vol. 38, No. 4, Article 97). By projecting polygons subdivided into minimum units by the software rasterizer, pixel-by-pixel drawing can be performed without passing through an intermediate buffer, similar to ray tracing.

[0037] In any case, in the present embodiment, the original image data generation unit 52 directly draws a distorted original image by setting a curved view screen at a wide angle like the view screen 426 in FIG. 3. At this time, the original image data generation unit 52 controls so that the interval of the azimuth for acquiring color information changes depending on the angle formed with the central axis, although color information of objects in the azimuths equidistant from the viewpoint is originally acquired in the three-dimensional space of the display target. Thereby, on the original image, an image in which the magnification distribution of the image in equidistant projection is suitably adjusted can be obtained.

[0038] The original image data generation unit 52 generates the original image at a predetermined rate while appropriately changing the central axis and thus the image center based on the line-of-sight information acquired by the line-of-sight information acquisition unit 50. Thereby, even when the field of view changes, it is possible to continuously generate an image in which an image closer to the center is enlarged. As described above, when the field of view of the display image is fixed regardless of the user's movement or the field of view of the display image is changed based on the regulations in a program such as a game, the original image data generation unit 52 may generate the original image with the center of the field of view as the image center. In this case, the function of the line-of-sight information acquisition unit 50 can be omitted.

[0039] The data transmission unit 56 transmits the data of the original image to the display control device 10 at a predetermined rate. The transmission rate of the original image may be the same as or lower than the frame rate of the display on the display device 16. Further, as will be described later, the original image data generation unit 52 may generate data having a mipmap structure in which the original image is represented at a plurality of resolutions and hierarchically. In this case, the data transmission unit 56 may select and transmit the original image of the layer necessary for generating the display image in response to a request from the display control device 10. The data transmission unit 56 may further select and transmit only a partial area of the original image of a certain layer in response to a request from the display control device 10.

[0040] The display control device 10 includes a gaze information transmission unit 60 that transmits information related to the user's gaze, a viewpoint information acquisition unit 62 that acquires information on the user's viewpoint and gaze, an original image data acquisition unit 64 that acquires the data of the original image from the image transmission device 20, a display image generation unit 66 that generates a display image using the data of the original image, and an output unit 68 that outputs the data of the display image. The gaze information transmission unit 60 transmits information related to the direction of the user's gaze to the image transmission device 20 at a predetermined rate.

[0041] The viewpoint information acquisition unit 62 specifies the position of the user's viewpoint and the direction of the gaze at a predetermined rate. For example, when the display device 16 is a head-mounted display, at least one of a motion sensor and a camera that captures the surrounding space is mounted on the head-mounted display. The viewpoint information acquisition unit 62 acquires the measured values of the motion sensor and the data of the captured image of the camera from the display device 16, and based on this, obtains the position and posture of the head-mounted display, and thus the user's head.

[0042] Then, the viewpoint information acquisition unit 62 generates information on the viewpoint and gaze at a predetermined rate by setting the position of the face as the position of the viewpoint or the direction in which the face faces as the direction of the gaze. At least the gaze information among this is transmitted to the image transmission device 20 via the gaze information transmission unit 60. Note that various techniques for acquiring information on the user's viewpoint and gaze have been put into practical use, and any of them may be adopted in the present embodiment.

[0043] The original image data acquisition unit 64 acquires data of the original image with distortion from the image transmission device 20 at a predetermined rate. The display image generation unit 66 generates a display image at the frame rate for display using the data of the original image. For example, the display image generation unit 66 draws a display image using the technology of ray tracing. That is, the display image generation unit 66 arranges a view screen and the original image in a virtual space, and generates rays passing through each pixel on the view screen from the user's viewpoint. Then, the pixel value of the display image is determined by sampling the pixel value at the arrival point in the original image.

[0044] The view screen at this time also becomes the curved view screen 426 shown in FIG. 3, but has a narrower viewing angle than the view screen when generating the original image. Further, the display image generation unit 66 sets the view screen based on the latest viewpoint and line-of-sight information acquired by the viewpoint information acquisition unit 62. As a result, when the user's line of sight changes between the generation of the original image and the generation of the display image, the image center is displaced from the original image. However, since the displacement amount of the image center in a very short time is limited, the effect of being able to generate a high-definition display image in the central region while reducing the data amount can be sufficiently obtained.

[0045] By setting the view screen based on the latest viewpoint and line-of-sight information and generating the display image immediately before display by the display image generation unit 66, a low-latency field-of-view change is guaranteed against the movement of the user's head. From these facts as described above, even if the rate of generation and transmission of the original image is made somewhat lower than the frame rate of display, the influence on the user's visual recognition can be minimized, and at the same time, the data amount transmitted from the image transmission device 20 can be further reduced.

[0046] The display image generation unit 66 adjusts the sampling direction for each pixel in the view screen of the display image in the direction opposite to the direction in which the original image data generation unit 52 adjusted the reference direction during the generation of the original image. As a result, the display image generation unit 66 can generate an image of an original orthographic projection that represents the color information of objects in directions equidistant from the viewpoint in the three-dimensional space to be displayed. When realizing stereoscopic vision, the display image generation unit 66 sets viewpoints to the left and right with respect to the original image, and generates display images for the left eye and the right eye.

[0047] The output unit 68 sequentially outputs the data of the display image generated by the display image generation unit 66 to the display device 16 for display. Note that the display device 16 may further adjust the viewing field of the display image according to the latest information on the position and posture of the user's head. Also, the display image generation unit 66 or the display device 16 may correct the degree of image distortion for each primary color according to the chromatic aberration of the eyepiece.

[0048] FIG. 6 is a diagram for explaining the basic processing of generating the original image and the display image in the present embodiment. The figure shows a side view of the view screen 150 of the original image 152 and the view screen 154 of the display image 156, which are set with respect to the line of sight 162 of the user 160, together with schematic diagrams of the original image 152 and the display image 156. In this example, the image transmission device 20 sets the view screen 150 at a wide angle such as 220° centered on the direction of the line of sight 162, and then generates the original image 152. Preferably, the view screen 150 of the original image has a curved shape corresponding to the view screen 154 of the display image and thus to the eyepiece.

[0049] For example, if the view screen 154 of the display image is spherical, the view screen 150 of the original image is also made spherical. However, even if the shapes are somewhat different, the effects of the present embodiment can be obtained. When the view screen 150 is a curved shape such as a spherical surface, the images of objects at substantially the same distance on the axis of the line of sight 162 are in a state of being reduced as they are farther from the image center in the original image 152. In the present embodiment, such a change in the degree of reduction is referred to as the "distribution of magnification". The display control device 10 sets the view screen 154 with an angle of view that can be displayed on the head-mounted display, centered on the direction of the line of sight 162, and then generates the display image 156.

[0050] That is, the display image 156 is an image representing a partial area of the original image 152. When realizing stereoscopic vision, the display control device 10 generates images with parallax by setting view screens for the viewpoints of the left eye and the right eye respectively, and arranges them side by side to form the display image 156. By making the image center of the original image 152 substantially the same as the left and right image centers of the display image 156, an original image that matches the in-plane distribution of the amount of information presented in the display can be generated or transmitted, and the costs related to various processes and transmissions can be reduced. Also, the storage area for storing the data of the original image and the time required for memory access in the display control device 10 can be saved.

[0051] In the figure, the view screen 150 for the original image and the view screen 154 for the display image are shown in a state set based on the common line of sight 162. However, by adjusting to the latest line of sight direction at the time of setting each view screen, the image centers may be shifted. As described above, this ensures that the field of view of the display image follows the movement of the head with low latency.

[0052] FIG. 7 is a diagram for explaining in detail the original image generated by the image transmission device 20. In (a), an image 182 representing a checkerboard flat plate by equidistant projection is shown, and in (b), an original image 192 generated by the image transmission device 20 is illustrated. On the right side of (a) and (b), the positional relationships of the viewpoint 180, the view screen 172, and the flat plate 170 to be displayed are shown in a horizontal cross section. As described above, when pixels are arranged at equal intervals on the surface of the view screen 172, the range of the flat plate 170 represented by one pixel changes within the plane.

[0053] For example, in (a), the widths a and b of the regions of the flat plate 170 represented by regions 174a and 174b of the same size on the view screen 172 are such that a < b, and the farther away from the central axis 184, the wider they become. As a result, in the image 182 obtained by the view screen 172, the image is reduced as it moves away from the center of the image. In the present embodiment, the distribution of magnification is controlled so as to further amplify this characteristic. Specifically, as shown on the right side of (b), when representing a pixel at the position coordinates (u, v) on the view screen 172 which is the image plane, the angle θ formed by the three-dimensional vector 186 passing through the position coordinates (u, v) from the viewpoint 180 and the central axis 184 is converted by a predetermined function f(θ) to generate a pseudo vector 188.

[0054] Then, the pixel value at the position coordinates (u, v) is determined using the color information at the arrival point 190 of the pseudo vector 188. By controlling the angle of the pseudo vector depending on the original angle θ, even for pixels arranged at equal intervals on the view screen 172, the density of the position (direction) on an object such as the flat plate 170 that they represent can be varied in various ways. In the present embodiment, in particular, the closer to the central axis 184 (the smaller the angle formed with the central axis 184), the higher the density at which color information is acquired. As a result, an image can be generated in which the magnification of the image at the center of the image is even larger than that of the image 182, as in the original image 192 of (b).

[0055] FIG. 8 is a diagram for explaining the generation process of the original image by the image transmission device 20. As shown in (a), the image transmission device 20 sets a view screen in a predetermined angular range centered on the position where the central axis 184 corresponding to the user's line-of-sight direction intersects the surface of the sphere 194 centered on the viewpoint. The position coordinates (u, v) on the view screen indicate the longitude and latitude with respect to the central axis 184. In the present embodiment, the pseudo 3D vector rd for determining the pixel at the position coordinates (u, v) is obtained, for example, as follows.

[0056]

Equation

[0057] Here, the angular field of view of the view screen is set to 220°. Also, the angle th is a value obtained by normalizing the angle θ formed by the 3D vector 186 (3D vector rd in FIG. 8) corresponding to the position coordinates (u, v) shown in (b) of FIG. 7 and the central axis 184 with respect to the angular field of view of the view screen. The function f(th) is a function that generates a pseudo vector from the angle th, and is defined, for example, as follows.

[0058]

Equation

[0059] This function f(th) becomes 1 when th = 0, that is, at the image center, and when th = ±1, that is, at the edge of the view screen. Also, by appropriately setting the parameters a and d, the rate of change with respect to th can be controlled. For example, as shown by the solid line 320a in (b) of FIG. 8, by making the rate of change with respect to th smaller the closer to the central axis, images closer to the image center in the original image can be represented with higher resolution, and thus, as shown in (b) of FIG. 7, an original image 192 with a more magnified central portion can be generated. On the other hand, when the display control device 10 generates a display image, for example, by the following calculation, the 3D vector nor that references the original image is associated with the position coordinates (u, v) on the view screen of the display image.

[0060]

Number

[0061] Here, the function f(th) is the inverse function of the solid line 320a used for generating the original image, as shown by the broken line 320b in FIG. (b). As a result, an image with the magnification distribution of the image emphasized can be converted into the image of the original equidistant projection. By such a conversion process of the three-dimensional vector, the region corresponding to the fovea can be expressed with higher definition while reducing the amount of information of the image near the periphery. In the above operation, data reading corresponding to the mipmap structure is performed by the "LODBIAS" parameter. However, the present embodiment is not limited to this gist.

[0062] FIG. 9 shows a conceptual diagram of an example of the data structure of the original image having a mipmap structure. In this example, the original image data has a hierarchical structure consisting of a 0th layer 90, a 1st layer 92, a 2nd layer 94, and a 3rd layer 96 as the levels of detail (LOD). Although only four layers are shown in the figure, the number of layers is not limited to this. Also, the original image data shown in the figure has a hierarchical structure of a quadtree, and each layer is composed of one or more tile regions 98. All the tile regions 98 are formed in the same size with the same number of pixels.

[0063] The image transmission device 20 directly draws the original image based on the equidistant projection and further adjusted in magnification by the method described so far. Then, by reducing the original image in multiple stages, data of the mipmap structure as shown in the figure is generated. By appropriately switching the layer of the original image used for generating the display image according to the position of the viewpoint, the amount of data to be processed can be optimized, and a high-quality image can be visually recognized while suppressing the costs related to transmission, processing, and storage areas.

[0064] When the display device 16 is a head-mounted display, the image transmission device 20 determines the center of the original image in accordance with the movement of the user's head and generates the data of the original image with a structure as shown in the figure. The display control device 10 obtains the detail required for display according to the distance of the viewpoint or the like, and requests the image transmission device 20 to transmit the original image with the said detail. The display control device 10 may also predict the detail that will be required later according to the movement history of the viewpoint so far, and request the image transmission device 20 to transmit the data of the corresponding layer.

[0065] The display control device 10 may also request the data of the entire area of the original image in the required or predicted required layer, or may request the data of a part of the area in tile units by uv coordinates. In this way, by accurately narrowing down the data required for generating the display image based on the trend of the viewpoint, it is possible to achieve both the optimization of the processing load and the high quality of the display image.

[0066] In the examples described so far, it is assumed that the original image with the central axis corresponding to the user's line of sight is prepared, but it is conceivable to further prepare the original image with the central axis in a different direction. FIG. 10 is a diagram for explaining a mode in which two types of images with different central axes are used as the original images. The upper part conceptually shows the entire circumferential space 200 to be displayed, and the horizontal direction corresponds to the longitude. If the user's line of sight in the space 200 is directed in the direction 202, the image transmission device 20 generates the original image 208 for the wide-angle region 204a centered on the direction 202 as described above.

[0067] In this mode, the image transmission device 20 further generates the original image 210 for a wide-angle region 204b centered on the direction 206 opposite to the user's line of sight. Note that the region 204a and the region 204b may partially overlap. Also, the central axes of the two original images are not limited to the opposite directions. In any case, by representing the space at the same time and preparing additional original images with central axes different from the line-of-sight direction, the range of the image world covered by the original images can be expanded. As a result, even when the user's line of sight suddenly changes and objects outside the visual field of the original image 208 are included in the display visual field, or when transmission data is lost due to communication congestion or the like, the possibility of image loss or time consumption until display due to insufficient data is reduced.

[0068] Also, even during the period when the display image can be generated using the original image 208, there may be a case where information about objects behind the user is required. For example, when there is a mirror in the visual field of the display image, the reflection of the objects behind changes depending on the line of sight. Also, depending on the relationship between the objects and light sources behind and the line of sight, the reflected light and shadows of the objects within the visual field can change. When accurately expressing such detailed physical phenomena by techniques such as ray tracing, it is effective to generate and transmit the image behind as an original image. Hereinafter, the original image 208 centered on the user's line-of-sight direction is referred to as the "main image", and the original image 210 centered on a different direction is referred to as the "sub-image".

[0069] FIG. 11 shows an example of the data structure of the original images including two types of images with different central axes. As described above, in this embodiment, in the main image, the images farther from the center are represented in a reduced manner. As a result, on the display image, the images closer to the periphery are represented with lower resolution, and it conforms to the visual characteristics of humans that the visual function deteriorates as the distance from the central visual field increases. Developing this, since the sub-image is less likely to be immediately captured by the fovea centralis, it is considered that even if it is prepared at a low magnification as a whole, the visual recognition effect is small.

[0070] Therefore, in the present embodiment, the image transmission device 20 makes the size of the sub-image, and thus the resolution, smaller than that of the main image, and represents the sub-image in the empty area of the rectangular image plane where the main image is not represented. In the example of the figure, the original image data 220 is image data obtained by arranging four sub-images (for example, sub-image 224) so as to be in contact with the main image 222 around the four vertices of the circumscribed rectangle (square) of the main image 222, and cutting out only the area within the rectangle.

[0071] In the figure, for clarity, the areas of the sub-images excluded by the cutting are shown in grayscale. As a result, among the four areas (partial images) obtained by dividing the sub-image vertically and horizontally, the lower right area 226a, the lower left area 226b, the upper right area 226c, and the upper left area 226d are respectively stored in the empty areas at the four corners (upper left, upper right, lower left, lower right) of the rectangular area of the original image data 220. Thereby, the data of the sub-image can be included in a rectangular area having the same size as the case where only the main image is used as the original image, and the cost related to transmission and storage area can be suppressed.

[0072] FIG. 12 shows the relationship between the sizes of the main image and the sub-image when generating the original image with the structure of FIG. 11. As shown in FIG. 11, when the sub-image is represented in the empty area when only the main image is used as the original image, the size of the sub-image is uniquely determined with respect to the size of the main image. That is, as shown on the left side of the figure, if the radius of the main image is R, the rectangular area of the original image is a square with a side length of 2R. Therefore, the radius r of the sub-image is determined as follows. r=(L - 2R) / 2 =R(√2 1 / 2 - 1)

[0073] That is, by setting the radius r of the sub-image 224 to approximately 0.41 times the radius R of the main image 222, the sub-image can be transmitted and stored without changing the size of the original image. Even at such a magnification, it has sufficient effects for avoiding display defects and expressing reflections. Note that the image transmission device 20 may reduce the data of the original image having the structure shown in the figure in multiple stages to have a hierarchical structure as shown in FIG. 9. In this case, in accordance with the switching of the hierarchy of the main image, the resolution of the sub-image will also be optimized, and compared with hierarchizing the main image and the sub-image individually, data requirements and size control will become easier.

[0074] Also, since the region corresponding to the fovea of the user and requiring detail is included in the main image, the image transmission device 20 may generate a distorted image corresponding to the eyepiece lens without adjusting the magnification distribution for the sub-image. The image transmission device 20 may also be a distorted image without adjusting the magnification distribution for the main image. Also in this case, by making the sizes of the main image and the sub-image different and incorporating them into one rectangular region, the effect of transmitting and storing the original image with a wide angle of view without increasing the data can be obtained. Naturally, if the magnification distribution of the main image is adjusted as described above, the effect of further reducing the data amount while expressing the region corresponding to the fovea in more detail can be obtained.

[0075] FIG. 13 is a diagram for explaining the data flow in the image display system of the present embodiment. A display image 232 generated in the previous time step is being displayed on the display device 16. The display device 16 transmits measurement data for deriving the position and posture of the head of the user looking at it to the display control device 10 (S10). The display control device 10 specifies the position and posture of the head based on the measurement data, and transmits the direction in which the face is facing and the like as the line-of-sight direction to the image transmission device 20 (S12).

[0076] The image transmission device 20 sets a curved view screen centered on the transmitted line-of-sight direction and generates an original image (main image). At this time, the image transmission device 20 may further adjust the in-plane magnification distribution originally obtained by the curved view screen and appropriately distribute the amount of information. The image transmission device 20 may further set a curved view screen centered on a direction different from the line-of-sight direction and generate a sub-image. The image transmission device 20 may further reduce the generated original image in multiple stages to generate original image data 236 composed of multiple layers.

[0077] The image transmission device 20 transmits the data of the original image generated in this way to the display control device 10 (S14). When the original image is hierarchical data, the image transmission device 20 may use only a part of the data as the transmission target in response to a request from the display control device 10. The display control device 10 temporarily stores the data of the transmitted original image 234 in a storage area such as the main memory 26 and uses it to generate a display image 238. Specifically, the display control device 10 first identifies the position of the viewpoint and the direction of the line of sight based on the information on the position and posture of the user's head that is the latest at the time of generation. Then, by setting a curved view screen corresponding to them and sampling the original image 234, each pixel value of the display image 238 is determined.

[0078] When the magnification distribution is adjusted in the original image 234, the display control device 10 generates a distorted image having the original magnification distribution by applying the reverse adjustment. Then, the display control device 10 outputs the data of the generated display image 238 to the display device 16 (S16). The display device 16 appropriately corrects the data of the transmitted display image and outputs it to the display panel. For example, the display device 16 may further adjust the field of view of the display image 238 based on the position and posture of the user's head immediately before output to obtain the final display image 232.

[0079] In this case, the display control device 10 may represent the display image 238 at an angle of view wider than the actually displayed angle of view so as to be able to correct the field of view. In the above data flow, the transmission and output frequencies of each data may be the same or at least partially different. For example, as described above, the transmission rate of the original image data in S14 may be lower than the output rate of the display image data in S16. In this case, during the period when the original image data is not transmitted, for example, the display images for several frames are generated in the data cycle between the display device 16 and the display control device 10. Thereby, while generating the data of the original image with high quality, the display image can be generated at high speed.

[0080] According to the embodiment described above, in a system that displays an image using the data of the original image transmitted from the image transmission device, the image transmission device generates and transmits an original image having the same distortion as the distortion given to the display image according to the eyepiece lens. At this time, the image transmission device adjusts the magnification distribution in the plane of the original image so as to represent higher-density color information in the central region corresponding to the user's line of sight. Since the distorted image corresponding to the eyepiece lens originally has the characteristic that the image is enlarged closer to the center, such adjustment can be easily performed.

[0081] Thereby, in the display image restored to the original magnification distribution, an image with high resolution can be represented in the region corresponding to the fovea centralis where high definition is easy to recognize, and high image quality in visual recognition can be realized. On the other hand, since the data amount can be suppressed at the peripheral portion where the user is less likely to notice the low resolution, the data size of the original image can be reduced as a whole, and the transmission band, processing load, and storage area can be saved. As a result, improvement in image quality and low latency can be easily achieved simultaneously.

[0082] Also, an original image in another direction smaller than the original image in the user's line-of-sight direction is generated, and is stored in a single rectangular area to be the data of the original image. As described above, by using the original image as a distorted image, the contour becomes a curve, and an empty area is generated in principle in the orthogonal plane, so that the original image in another direction can be accommodated.

[0083] As a result, it is possible to prepare an original image that covers the entire day and night without increasing the data size or memory area, prevent missing parts in the displayed image, express the influence of light on objects in the background, and present a more immersive image world. Also, since the difference in resolution between the two types of original images is appropriately defined, the processing related to transmission and reference becomes easier than generating each data independently.

[0084] As described above, the present invention has been described based on the embodiments. It is understood by those skilled in the art that the embodiments are examples, and various modifications are possible for the combinations of their respective components and each processing process, and such modifications are also within the scope of the present invention.

Explanation of Reference Numerals

[0085] 10 Display control device, 16 Display device, 20 Image transmission device, 22 CPU, 24 GPU, 26 Main memory, 32 Communication unit, 34 Storage unit, 50 Line-of-sight information acquisition unit, 52 Original image data generation unit, 56 Data transmission unit, 60 Line-of-sight information transmission unit, 62 Viewpoint information acquisition unit 62, 64 Original image data acquisition unit, 66 Display image generation unit, 68 Output unit.

Industrial Applicability

[0086] As described above, the present invention can be used in various information processing devices such as content servers, game devices, head-mounted displays, display devices, mobile terminals, personal computers, etc., and image display systems including any of them.

Claims

1. An image transmission device that transmits data of an original image used for generating a display image in parallel with the display, comprising one or more processors having hardware, wherein the one or more processors, generate the original image by determining the value of each pixel on the view screen of the original image by obtaining the color information of the corresponding direction angle in the space to be displayed, and at this time, obtain the color information after changing the interval of the direction angles so as to depend on the angle formed with the central axis of the view screen, and transmit the data of the original image to the device that generates the display image, an image transmission device.

2. The one or more processors, obtain information related to the user's line of sight, set the view screen having the central axis corresponding to the line of sight, and change the view screen in accordance with the change of the line of sight, The image transmission device according to Claim 1.

3. The one or more processors, transmit the data of the original image to the device that generates the display image for output to a display device provided with an eyepiece, and set the view screen having a curved shape corresponding to the eyepiece, The image transmission device according to Claim 1.

4. The one or more processors, obtain the color information at a higher density as the angle formed with the central axis is smaller, The image transmission device according to Claim 1.

5. The one or more processors, in addition to the main image generated by the view screen having the central axis corresponding to the line of sight, generate a sub-image representing the space at the same time in another field of view by a view screen having a different central axis, and transmit, as the data of the original image, data representing the main image and the sub-image having a smaller size than the main image in one rectangular area, The image transmission device according to Claim 2.

6. The one or more processors, generate the sub-image by the view screen having a central axis in the direction opposite to the central axis corresponding to the line of sight, The image transmission device according to Claim 5.

7. The one or more processors, generate the main image and the sub-image by the curved view screen, and represent four partial images obtained by vertically and horizontally dividing the sub-image at the four corners of the rectangular area circumscribing the main image, The image transmission device according to Claim 5.

8. The one or more processors, change the interval of the direction angles for obtaining the color information only for the main image, The image transmission device according to claim 5.

9. The one or more processors: Transmit the data of the original image at a rate lower than the frame rate of the display. The image transmission device according to claim 1.

10. The one or more processors: Generate hierarchical structure data representing the original image at a plurality of resolutions, Transmit the data of the hierarchy requested by the device that generates the display image. The image transmission device according to claim 1.

11. An image transmission device that transmits the data of the original image used for generating a display image in parallel with the display, comprising one or more processors having hardware, The one or more processors: Acquire information related to the user's line of sight, Generate a main image generated by a curved view screen having a central axis corresponding to the line of sight and a sub-image representing the space at the same time in another field of view by a curved view screen having a different central axis, Transmit the data of the original image representing four partial images obtained by vertically and horizontally dividing the sub-image into four parts at the four corners of the rectangular region circumscribing the main image to the device that generates the display image. Image transmission device.

12. An image transmission method for transmitting the data of the original image used for generating a display image in parallel with the display, Determine the value of each pixel on the view screen of the original image by acquiring the color information of the corresponding direction angle in the space to be displayed to generate the original image. At this time, acquire the color information after changing the interval of the direction angles so as to depend on the angle formed with the central axis of the view screen, Transmit the data of the original image to the device that generates the display image. Image transmission method.

13. A computer for transmitting the data of the original image used for generating a display image in parallel with the display, A function of determining the value of each pixel on the view screen of the original image by acquiring the color information of the corresponding direction angle in the space to be displayed to generate the original image. At this time, acquire the color information after changing the interval of the direction angles so as to depend on the angle formed with the central axis of the view screen, and A function of transmitting the data of the original image to the device that generates the display image, A recording medium recording a program for realizing the above.

Images