IMAGE DISPLAY SYSTEM, DISPLAY CONTROL DEVICE, AND IMAGE DISPLAY METHOD

Abstract

An image transmission device 20a or 20b transmits, at a predetermined rate, data 166 of an entire original image captured by an imaging device 140 or 141 to a display control device 10a or 10b, obtains a signal for a brightness change event detected by the imaging device 140 or 141 for each pixel to generate event information 168, and asynchronously transmits the event information 168. The display control device 10a or 10b generates a display image by ray tracing at a frame rate higher than the frame rate of the transmitted original image and selectively updates the corresponding image in units of pixels by using the latest event information.

Description

[Technical field]

[0001] The present invention relates to an image display technique for displaying a moving image using a transmitted original image. [Background technology]

[0002] UHDTV (Ultra HDTV) display devices with resolutions four to 16 times that of HDTV (High Definition Television) are now in practical use. Display systems that give a sense of immersion in the image world by showing full-dome images with a free field of view that corresponds to the line of sight of the user wearing a head-mounted display are also becoming widespread. The advent of 5G (5th generation mobile communication system) has made it possible to transmit data at high bandwidth and low latency, allowing users to enjoy high-quality image content regardless of the environment. Summary of the Invention [Problem to be solved by the invention]

[0003] As mentioned above, there is a tendency for image resolution, viewing angle, and the number of pixels that make up one frame to continue to increase in various electronic contents. However, this increase in data volume causes various problems, such as strain on transmission bandwidth, increased calculation costs, and increased power consumption. In particular, if it takes time to process and transmit one frame of data, the frame rate may not increase and image quality may deteriorate, or the change in the field of view may be delayed in response to the user's eye movement, causing discomfort.

[0004] The present invention has been made in view of the above problems, and has as its object to provide a technology that can easily display images with high resolution and a wide viewing angle with high quality. [Means for solving the problem]

[0005] In order to solve the above problems, one aspect of the present invention relates to an image display system, which includes an image transmission device that transmits data of an original image, and a display control device that controls image display using the data of the original image, the image transmission device includes an event information generation unit that generates event information including position information of a pixel when a luminance change event occurs in any pixel in the original image, and a data transmission unit that transmits data of the entire original image at a predetermined frame rate and transmits the event information asynchronously, the display control device includes a display image drawing unit that generates data of a display image at the predetermined frame rate using the data of the entire original image, locally updates the display image based on the most recent event information, and an output unit that outputs the display image data to a display device.

[0006] Another aspect of the present invention relates to an image transmitting device that transmits data of an original image and realizes an image display using the original image at a transmission destination, and is characterized by including an event information generating unit that generates event information including position information of a pixel when a luminance change event occurs in any pixel in the original image, and a data transmitting unit that transmits data of the entire original image at a predetermined frame rate and asynchronously transmits the event information to locally update the display image at the transmission destination.

[0007] Yet another aspect of the present invention relates to a display control device that receives data of an original image and controls image display using the data of the original image, and is characterized by including an image data acquisition unit that acquires data of the entire original image at a predetermined frame rate, an event information acquisition unit that acquires event information including position information of a pixel in the original image, transmitted asynchronously with the data of the entire original image, in response to an event of a luminance change occurring in the pixel of the original image, the event information including position information of the pixel, a display image drawing unit that uses the data of the entire original image to generate data of a display image at the predetermined frame rate and locally updates the display image based on the most recent event information, and an output unit that outputs the data of the display image to a display device.

[0008] Yet another aspect of the present invention relates to an image display method, the image display method being characterized in that, in an image display system including an image transmission device that transmits data of an original image and a display control device that controls image display using the data of the original image, the image transmission device, when an event of a luminance change occurs in any pixel in the original image, generates event information including position information of the pixel, transmits data of the entire original image at a predetermined frame rate and transmits the event information asynchronously, generates data of a display image at a predetermined frame rate using the data of the entire original image, and locally updates the display image based on the most recent event information, and outputs the data of the display image to a display device.

[0009] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also effective as aspects of the present invention. Effect of the Invention

[0010] According to the present invention, images with a large resolution and viewing angle can be easily displayed with high quality. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration example of an image display system to which the present embodiment can be applied; [Diagram 2] 1A to 1C are diagrams showing examples of images transmitted and displayed in the present embodiment. [Diagram 3] FIG. 1 is a diagram illustrating a specific configuration of an image display system according to an embodiment of the present invention. [Figure 4] 2 is a diagram showing an internal circuit configuration of the display control device according to the present embodiment. [Diagram 5] 2 is a diagram showing the configuration of functional blocks of an image transmitting device and a display control device according to the present embodiment. FIG. [Figure 6]1 is a diagram for explaining a problem associated with the balance between the amount of information provided and the amount of information received when a flat display is used as the display destination. [Figure 7] 4 is a diagram for explaining the relationship between data transmitted by an image transmitting device and a display image generated by a display control device in the present embodiment. FIG. [Figure 8] 11 is a diagram showing another example of the relationship between data transmitted by the image transmitting device and a display image generated by the display control device in the present embodiment. FIG. [Figure 9] 10A and 10B are diagrams illustrating examples of data formats of panoramic images and event information prepared by the image transmission device in the present embodiment. [Figure 10] 11A and 11B are diagrams for explaining the relationship between the shape of a view screen and the amount of information provided. [Figure 11] 1A to 1C are diagrams for explaining a distorted image drawing process in the display control device according to the present embodiment. [Figure 12] FIG. 13 is a diagram showing a modified example of the image display system according to the present embodiment. [Figure 13] 11A and 11B are diagrams for explaining the relationship between partial updating based on event information and reprojection in the present embodiment. [Figure 14] 11A to 11C are diagrams for explaining a display image generation process in the case where a panoramic image is expressed by a cube map in the present embodiment. [Figure 15] 10A to 10C are diagrams for explaining a display image generation process in the case where a panoramic image is expressed in equirectangular projection in the present embodiment. [Figure 16] 11A and 11B are diagrams for explaining a display image generation process in the case where a panoramic image is expressed by equidistant projection in the present embodiment. [Figure 17] 1A and 1B are diagrams for explaining an embodiment in which a panoramic image is represented by equidistant projection and a plurality of resolution levels are provided. [Figure 18] 11A and 11B are diagrams for explaining a method for generating an equidistant projection image having a distribution in which the resolution decreases from the center to the periphery in the present embodiment. [Figure 19]10A and 10B are diagrams for explaining a display image generation process in the present embodiment when a panoramic image is displayed on the surface of a regular octahedron. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1 shows an example of the configuration of an image display system to which this embodiment can be applied. The image display system 1 includes display control devices 10a, 10b, and 10c that display images in response to user operations, and an image transmission device 20 that provides image data used for display. Input devices 14a, 14b, and 14c for user operations and display devices 16a, 16b, and 16c that display images are connected to the display control devices 10a, 10b, and 10c, respectively. The display control devices 10a, 10b, and 10c and the image transmission device 20 can establish communication via a network 8 such as a WAN (World Wide Network) or a LAN (Local Area Network).

[0013] The display control devices 10a, 10b, and 10c may be connected to the display devices 16a, 16b, and 16c and the input devices 14a, 14b, and 14c either by wire or wirelessly. Alternatively, two or more of these devices may be integrally formed. For example, in the figure, the display control device 10b is connected to a head-mounted display, which is the display device 16b. The head-mounted display can change the field of view of the displayed image by the movement of the user wearing it on the head, so it also functions as the input device 14b.

[0014] The display control device 10c is a mobile terminal, and is integrally configured with a display device 16c and an input device 14c, which is a touch pad covering the screen of the display device 16c. In this way, the external shape and connection form of the illustrated devices are not limited. The number of display control devices 10a, 10b, 10c and image transmission devices 20 connected to the network 8 is also not limited. For example, the image transmission device 20 may be a cloud server or a content server in a system such as cloud gaming. Hereinafter, the display control devices 10a, 10b, 10c may be collectively referred to as the display control device 10, the input devices 14a, 14b, 14c as the input device 14, and the display devices 16a, 16b, 16c as the display device 16.

[0015] The input device 14 may be any one or a combination of general input devices such as a controller, a keyboard, a mouse, a touchpad, or a joystick, and supplies the contents of user operations to the display control device 10. The display device 16 may be a general display such as a liquid crystal display, a plasma display, an organic EL display, a wearable display, or a projector, and displays an image output from the display control device 10.

[0016] The image transmission device 20 provides data of content accompanied by image display to the display control device 10. The type of the content is not particularly limited, and may be any of electronic games, ornamental images, television broadcasts, electronic conferences, video chats, and the like. The display control device 10 basically realizes display processing while acquiring content data used for display from the image transmission device 20. Alternatively, the image transmission device 20 and the display control device 10 may cooperate to generate content data, thereby distributing the load of drawing processing and the like.

[0017] 2 shows examples of images transmitted and displayed in this embodiment. (a) is an example in which the data of a captured image 130 is displayed as a displayed image 132 as is in a live broadcast, video chat, or the like. In this case, the source of the image data may be an image capturing device connected to the display control device 10, in addition to the image transmitting device 20. The display device 16 is a flat panel display.

[0018] 1B is an example in which a panoramic image 134 is prepared and a display image 136 is generated in a field of view corresponding to the movement of the head of a user wearing a head mounted display. In the case of a head mounted display, stereoscopic vision can be realized by displaying a stereo image having parallax in the left and right regions of a display panel corresponding to the left and right eyes of the user, as in the illustrated display image 136. In the case of a head mounted display equipped with an eyepiece for expanding the field of view, an inverse distortion corresponding to the distortion aberration and chromatic aberration of the eyepiece is given to the display image 136 so that an undistorted image is viewed when viewed through the eyepiece.

[0019] For example, in the case of a lens that makes the four sides of an image appear concave like a pincushion, the left and right images of the display image 136 are each curved into a barrel shape. Hereinafter, an image that has been given a distortion corresponding to the eyepiece lens will be referred to as a "distorted image." Note that the full-sky image 134 may be an image drawn by computer graphics, a photographed image, or a combination of both. The source of the image data may be the image transmission device 20, a drawing mechanism inside the display control device 10, or an imaging device connected to the display control device 10.

[0020] In the field of flat panel displays shown in (a), it has become possible to realize high-definition image representation with a high pixel count, such as UHDTV. In the field of head-mounted displays, preparing an image with a wide viewing angle, such as the panoramic image 134 shown in (b), makes it possible to look around the image world from any viewpoint, enhancing the sense of immersion. In either case, the number of pixels that make up one frame tends to increase. When the amount of data for one frame increases in this way, the time required for processing and transmission increases, making it difficult to achieve a high frame rate at the same time.

[0021] In addition, in a case where an image captured or drawn in real time is displayed, the delay time until display can become unnoticeable. For example, in an electronic game where an image is generated according to a user's operation, the delay time has a significant effect on the operability of the game. Furthermore, since the power consumption for data processing and transmission increases, a large-capacity storage battery may be required, particularly in the case of a head-mounted display, and the wearing comfort may be deteriorated due to heat generation.

[0022] Therefore, in this embodiment, the image is captured on a pixel-by-pixel basis and information is transmitted, thereby saving resources related to processing and transmission. Specifically, the data transmission source obtains the occurrence of a change event on a pixel-by-pixel basis and transmits only information related to the local change event via a bypass line, independent of the data for the entire image. The display control device 10 locally updates the part of the displayed image where the change event occurred based on the most recent transmitted information.

[0023] Here, a "change event" refers to a state in which a difference equal to or exceeds a threshold occurs in observed brightness or pixel value (hereinafter, simply referred to as an "event"). A data source such as image transmission device 20 transmits data such as captured image 130, panoramic image 134, or a part thereof at a predetermined rate, and transmits event information including the position of the pixel and the content of the change by bypass when a change event occurs. The display control device 10 generates frames of a display image at a predetermined rate, and locally updates the frames based on the transmitted event information.

[0024] Here, the display control device 10 uses ray tracing technology to draw the display image. Ray tracing is a technique for generating virtual rays that pass through each pixel on the view screen from the viewpoint, and determining pixel values ​​based on color information of the destination. In this embodiment, an image transmitted from the image transmission device 20 or the like is placed in front of the view screen, and the destination of the ray is obtained, thereby acquiring color information on the image. By employing this technique, it becomes possible to selectively draw only the pixel where an event has occurred.

[0025] Ray tracing is generally considered to be a computationally intensive and heavy-load process. However, compared to rasterization, which projects polygons, it has the advantage of not requiring much intermediate buffer memory such as a depth buffer. Depending on the content of the image, it can also be operated with low latency and light computational load and resource usage. Furthermore, since the amount of processing is proportional to the drawing area, limiting the update targets as mentioned above can increase the drawing efficiency while maintaining quality. Also, by enabling the transmission and drawing of information on a pixel-by-pixel basis, the transmission of data such as backgrounds that do not contribute to improving image quality can be kept to a minimum, reducing the processing and transmission load at the data source.

[0026] 3 illustrates a specific configuration of the image display system in this embodiment. As described above, in this embodiment, the entity that generates the image data that is the source of the display image is not limited to the image transmission device 20, and at least a part of it may be performed by a local device on the display side such as the display control device 10 or an imaging device, but hereafter, the image transmission device 20 will be mainly described. Also, the entity that forms the image data into a final display image using a ray tracing technique is not limited to the display control device 10, and at least a part of it may be performed by the display device 16, but hereafter, the display control device 10 will be mainly described. Hereinafter, the image transmitted by the image transmission device 20 will be referred to as an "original image" to be distinguished from the "display image" generated by the display control device 10.

[0027] The figure shows a state in which two image transmission devices 20a and 20b and two display control devices 10a and 10b are connected via any network such as LAN, WiFi (registered trademark), or 5G. Of these, the image transmission device 20a transmits a video of a speaker 160 captured by an imaging device 140 to the display control devices 10a and 10b, as shown in (a) of Fig. 2. Meanwhile, the image transmission device 20b captures a panoramic image 142 with an imaging device 141 and transmits at least a part of it to the display control devices 10a and 10b, as shown in (b) of Fig. 2.

[0028] In the figure, the panoramic image 142 is shown as a cross section of a spherical screen representing the image plane, but as described later, the method of expressing the panoramic image 142 (data format) is not particularly limited. The imaging device 141 may be configured as a combination of fisheye cameras on the front and back, or as a multi-lens camera. As described later, the image transmitting devices 20a and 20b may render images equivalent to the images captured by the imaging devices 140 and 141 using computer graphics, or may render computer graphics on the captured images.

[0029] The display control device 10a uses data of an original image 148 transmitted from the image transmission devices 20a and 20b to display a display image 150 on a flat panel display, which is the display device 16. On the other hand, the display control device 10b uses data of an original image 152 transmitted from the image transmission devices 20a and 20b to display a display image 154 on a head mounted display, which is the display device 16.

[0030] In the figure, the data of the original images 148, 152 are shown as part of a spherical screen representing the image plane, but the data format is not limited to this. For example, if the image of central projection captured by the imaging device 140 is used as the original image 148, the screen will be flat. Also, since the image displayed on the head mounted display is a distorted image as described above, the screen (view screen) of the display image 154 is shown as a substantially spherical surface.

[0031] In the case where an image having distortion, such as a panoramic image, is used as the original image 152, in this embodiment, a distorted display image 154 is directly generated from the original image 152. Specifically, as shown in the figure, the original image 152 is placed in front of a view screen in a curved screen shape according to the distortion. Then, the original image 152 is sampled from the curved view screen by ray tracing to generate the display image 154. This eliminates the need to temporarily generate a centrally projected image, making it possible to save memory for expanding the image and waiting time for expansion.

[0032] From the image transmitting devices 20a, 20b, data 166 of the entire original image is transmitted at a predetermined rate, as indicated by the bold line in the figure. Here, "the entire original image" refers to the entire area of ​​the frame of the original image, for example, the entire area of ​​the image within the field of view. However, the "frame" to be transmitted is not limited to the image within the field of view, but may include an area wider than the field of view that is used by the display control devices 10a, 10b to form the display image. Furthermore, from the image transmitting devices 20a, 20b, as indicated by the dashed line in the figure, event information 168 including the position of the pixel where the event occurred and the details of the change are transmitted by bypass every time an event occurs, as indicated by the dashed line in the figure.

[0033] Basically, the display control devices 10a and 10b use data 166 of the entire original image transmitted at a predetermined rate from the image transmission devices 20a and 20b to generate the entire display image and display it on the display device 16. At this time, the display control devices 10a and 10b may increase the frame rate of the display by interpolating frames of the original image, or may correct the field of view according to the latest position and posture of the head of a user 162b viewing the head mounted display. The display control devices 10a and 10b then selectively update the necessary parts of the display image using event information 168 transmitted asynchronously from the image transmission devices 20a and 20b.

[0034] The display control devices 10a and 10b may also transmit data 170a and 170b relating to the position or area on the screen that the users 162a and 162b are gazing at when viewing the display image to the image transmission devices 20a and 20b at a predetermined rate. In this case, the image transmission devices 20a and 20b exclude information on events that occur in the original image and that occur outside the area that the users 162a and 162b are gazing at from the information to be transmitted. This makes it possible to further limit the update targets in the transmission data and the display image without the users 162a and 162b being aware of this.

[0035] 4 shows the internal circuit configuration of the display control device 10. The display control device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are connected to one another via a bus 30. An input / output interface 28 is further connected to the bus 30. To the input / output interface 28, there are connected a communication unit 32 including a peripheral device interface such as USB or IEEE1394, or a network interface for wired or wireless LAN, a storage unit 34 such as a hard disk drive or nonvolatile memory, an output unit 36 ​​that outputs data to the display device 16, an input unit 38 that inputs data from the input device 14 or an imaging device, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0036] The CPU 22 executes an operating system stored in the storage unit 34 to control the entire display control device 10. The CPU 22 also executes various programs that are read from a removable recording medium and loaded into the main memory 26, or that are downloaded via the communication unit 32. The GPU 24 performs drawing processing according to drawing commands from the CPU 22, and stores a display image in a frame buffer (not shown). The GPU 24 then converts the display image stored in the frame buffer into a video signal and outputs it to the output unit 36. The main memory 26 is composed of a RAM (Random Access Memory), and stores programs and data required for processing. The circuit configuration of the image transmission device 20 may be substantially the same, and in this case, the data of the acquired or generated image is transmitted to the display control device 10 via the communication unit 32.

[0037] Fig. 5 shows the functional block configuration of the image transmission device 20 and the display control device 10 in this embodiment. Note that the image transmission device 20 and the display control device 10 may perform general information processing such as progressing an electronic game and outputting sound, but Fig. 5 focuses on the function of transmitting and displaying image data.

[0038] 5 can be realized in hardware by the configuration of the CPU 22, GPU 24, main memory 26, etc., shown in Fig. 4, and in software by a program that performs various functions such as a data input function, a data storage function, an image processing function, a communication function, etc., that is loaded into the main memory 26 from a recording medium, etc. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof, and are not limited to any one of them.

[0039] The image transmitting device 20 includes a gaze area acquiring unit 50 that acquires information related to the user's gaze area, an image data acquiring unit 52 that acquires data of an original image, and a data transmitting unit 56 that transmits the data of the original image to the display control device 10. The gaze area acquiring unit 50 acquires, at a predetermined rate, information indicating the position coordinates of the user's gaze point with respect to the screen of the display device 16, or information indicating a predetermined range of an area including the gaze point, from the display control device 10.

[0040] The image data acquisition unit 52 acquires data of the entire original image at a predetermined rate. The data may be generated by the image data acquisition unit 52 itself, or may be acquired from an external device (not shown). For example, the image data acquisition unit 52 acquires captured images at a predetermined rate from an imaging device connected to the image transmission device 20. Alternatively, the image data acquisition unit 52 draws the original image at a predetermined rate by computer graphics based on model data and a game program stored inside. In this case, the image data acquisition unit 52 may acquire the contents of a user's operation for a game or the like from the display control device 10.

[0041] Furthermore, when the display device 16 is a head-mounted display and the field of view is changed in accordance with the movement of the user's head, the image data acquisition unit 52 also acquires information related to the position and orientation of the head from the display control device 10 at a predetermined rate. In this case, the head-mounted display is equipped with at least one of a motion sensor and a camera that captures the surrounding space. The display control device 10 determines the position and orientation of the head-mounted display and, in turn, the user's head based on the measurement value of the motion sensor and the image captured by the camera, and transmits them to the image transmission device 20 at a predetermined rate. Note that various techniques for determining the position and orientation of the head-mounted display have been put into practical use, and any of these may be adopted in this embodiment.

[0042] As a result, the image data acquisition unit 52 prioritizes drawing an original image of an area that is within the user's field of view or is predicted to be within the user's field of view, for example, in a panoramic image showing a virtual world. When the image data acquisition unit 52 draws an image by itself, it can be considered as an "original image drawing unit." The image data acquisition unit 52 may also draw computer graphics on the captured image.

[0043] The image data acquisition unit 52 also includes an event information generation unit 54 that detects the occurrence of an event and generates event information to be transmitted. When the image to be displayed is a captured image, the event information generation unit 54 may acquire the event information from the imaging device. An event-driven sensor (EDS) is known that generates a signal indicating information when a change occurs in the intensity of light incident on an imaging element (see, for example, International Publication No. WO 2021 / 033251 and International Publication No. WO 2021 / 033252).

[0044] An EDS, also known as a neuromorphic camera, silicon retina, or dynamic vision sensor, is an image sensor that responds to events of local changes in brightness. Specifically, when a change occurs in the intensity of incident light at any sensor in the sensor array, it outputs a signal indicating the location, time, and whether the change is an increase or decrease. Since an EDS responds locally only to the pixel where an event occurs at the time that the event occurs, it is lighter in load, lower latency, and lower power consumption than image processing approaches that extract differences while scanning the entire image. In other words, it can output information with higher time resolution than processing the entire image.

[0045] Therefore, the event information generating unit 54 receives an event signal from the EDS equipped in the imaging device and generates event information in real time. For example, it generates information that associates the position of a pixel where the intensity of the incident light shows a change equal to or greater than a threshold value, the time of occurrence, and the content of the change. Here, the position of a pixel may be represented by the direction of a ray that passes through each pixel of the original image from a virtual viewpoint or the center of the optical axis of the camera.

[0046] The content of the change is not particularly limited as long as it is information that allows the display control device 10 to update the display image based on the content. For example, it may be a difference in pixel values ​​or a pixel value after the change. Alternatively, as described below, information used to draw an object represented by the pixel where the event occurred, such as depth information or model data, may be included in the event information so that the display control device 10 itself can draw the object.

[0047] The imaging device may be configured with an image sensor that captures general color images and an EDS separately, or may be a device that can output color images by accumulating data on events that have occurred for a certain period of time. In the former case, the positions of the two sensor arrays are associated, and the clock that generates the shooting time and the event occurrence time is synchronized. In the latter case, a lower frame rate and improved compression rate for color images can be expected, making it easier to transmit depth information and model data of the object itself where the event occurred.

[0048] On the other hand, when the image data acquisition unit 52 draws the original image by itself, the event information generation unit 54 generates event information in accordance with a program by which the image data acquisition unit 52 draws the original image. That is, the image data acquisition unit 52 determines the movement of objects in the image world before drawing the image by ray tracing or the like. Therefore, when drawing the image, the image data acquisition unit 52 can identify at which pixel an event will occur. By acquiring the information, the event information generation unit 54 generates event information in the same way as when an event signal from the EDS is used. When computer graphics are to be synthesized with a captured image, the event information generation unit 54 may generate event information based on both the event signal from the EDS and event information that becomes clear when the image is drawn.

[0049] The event information generating unit 54 may further select events to be transmitted as event information based on information related to the gaze point and gaze area acquired by the gaze area acquiring unit 50. For example, the event information generating unit 54 masks events that occur outside the gaze area to exclude them from transmission. The data transmitting unit 56 transmits data of the entire original image to the display control device 10 at a predetermined rate, and transmits the generated event information immediately when the event information generating unit 54 generates the event information.

[0050] The display control device 10 includes a gaze area transmission unit 60 that transmits information related to the user's gaze area, an image data acquisition unit 62 that acquires data of the entire original image from the image transmission device 20, an event information acquisition unit 64 that acquires event information from the image transmission device 20, a display image drawing unit 66 that draws a display image, and an output unit 74 that outputs data of the display image.

[0051] The gaze area transmission unit 60 acquires the position coordinates of the user's gaze point on the screen of the display device 16 or information related to a predetermined range of an area including the gaze point from the display device 16, and transmits it at a predetermined rate to the image transmission device 20. For this purpose, the display device 16 is provided with a gaze point detector (not shown). The gaze point detector is a general device that detects the point at which a person is gazing at an object he or she is looking at, for example, by photographing the reflection of infrared rays irradiated onto the eyeball.

[0052] The image data acquisition unit 62 acquires data of the entire original image from the image transmission device 20 at a predetermined rate. The event information acquisition unit 64 acquires event information from the image transmission device 20 each time. The display image drawing unit 66 includes a frame image drawing unit 70 that draws a frame of the display image, and an event information management unit 72 that changes the processing of the frame image drawing unit 70 in response to the occurrence of an event. The frame image drawing unit 70 uses the data of the entire original image acquired by the image data acquisition unit 62 to draw a frame of the display image at a predetermined rate using a method similar to ray tracing.

[0053] That is, the frame image rendering unit 70 generates a ray that passes through each pixel on the view screen from the user's viewpoint, and samples the pixel value of the arrival point in the original image placed in front of the ray to determine the pixel value of the displayed image. When an image is displayed on a flat panel display using a centrally projected original image, the pixel arrangement of the display screen and the pixel arrangement of the original image may be the same. Also, the frame image rendering unit 70 performs local updates by switching the processing content for the location where an event occurred based on the most recent event information.

[0054] For example, the frame image rendering unit 70 determines pixel values ​​of the display image for the location where an event has occurred based on pixel value information of the original image included in the event information. When the location where the event has occurred in the original image is associated with the direction of the ray, it becomes clear whether or not an event has occurred at the destination based on the relationship with the direction of the ray set in ray tracing. When an event has occurred at the destination, the frame image rendering unit 70 updates only the necessary portion of the display image by replacing the sampling destination with pixel value information included in the event information.

[0055] Alternatively, the frame image rendering unit 70 may render an image of the object in which the event has occurred using 3DCG based on the object's depth information and model data included in the event information. In this case, the frame image rendering unit 70 may perform general ray tracing that calculates reflection and refraction on the object's surface. In any case, in this embodiment, the rendering of the entire display image is based on ray tracing, so even with such a local response, there is little change in the processing content, making it easy to switch.

[0056] Note that the frame image drawing unit 70 draws frames of the display image at a rate higher than the frame rate of the original image in order to reflect the occurrence of an event in the display with high time resolution. The event information management unit 72 switches the drawing process in the frame image drawing unit 70 as described above, based on the event information acquired by the event information acquisition unit 64. That is, it determines whether an event has occurred at the destination of the set ray, and replaces the sampling destination or draws the object where the event has occurred in 3DCG.

[0057] The model data of the object may be stored inside the display control device 10. The output unit 74 outputs the data of the display image drawn by the display image drawing unit 66 to the display device 16 for display. As a result, an image showing in detail the change in the part where the event occurred is displayed at a frame rate higher than the transmission rate of the original image.

[0058] FIG. 6 is a diagram for explaining the problem of balancing the amount of information provided and the amount of information received when a flat panel display is used as the display destination. Here, the amount of information provided is the amount of information presented by the displayed image, and the amount of information received is the amount of information recognized by the brain of the user viewing it. The diagram shows the angle of view (or FOV: Field Of View) of a displayed image at an appropriate distance from the user's viewpoint 80. For example, in the case of an HDTV screen 81 with a horizontal pixel count of about 2000, the angle of view θ 2K In the case of UHDTV screens 82 and 84 with horizontal pixels of about 4000 and 8000, the angle of view θ 4K , θ 8K are approximately 61° and 100°, respectively.

[0059] For comparison, the figure also shows a screen 86 of a head mounted display, which has an angle of view of about 100°, the same as screen 84. In any case, as shown in the figure, the larger the screen of the display, the greater the difference in the angle of view from line of sight V. Meanwhile, as viewed from viewpoint 80, the number of pixels on the screen contained per unit angle increases the farther away from line of sight V. For example, when looking at screen 84, a wider range of pixels comes into view in area 90b away from line of sight V compared to area 90a close to line of sight V, even within the same range of angle θ.

[0060] In other words, the farther away from the line of sight V, the greater the density of the amount of information entering the field of view, and this difference becomes more pronounced as the angle of view widens and the maximum distance between the edge of the screen and line of sight V becomes greater. Meanwhile, it is known that human visual function has a distribution relative to the field of view. The central area on the retina corresponding to the center of the field of view is called the fovea, and has the highest visual acuity and ability to distinguish colors and shapes. The area of ​​the field of view within about 5° from the line of sight, including the area viewed from the fovea, is called the central visual field, the area within about 20° from the line of sight is called the useful visual field, and the area outside of that is called the peripheral visual field, and visual function declines in that order.

[0061] In other words, the amount of information received decreases the further away from the line of sight V. Considering that the density of the amount of information entering the field of view increases at positions farther from the line of sight V as mentioned above, displaying the entire image with a uniform spatial level of detail can be said to create an imbalance between the amount of information provided and the amount of information received. In reality, the line of sight V changes due to the movement of the image of an object, and the area that can be captured by high visual function moves on the screen. To deal with this, it is necessary to increase the frame rate and update the image in detail over time, at least for that area.

[0062] Therefore, in this embodiment, by focusing on the line of sight V and in turn on a region within a predetermined range from the gaze point to improve image quality, the imbalance between the amount of information provided and the amount of information received is corrected, and wasteful processing and transmission for regions away from the gaze point is eliminated. Specifically, as described above, information on events that occur outside the gaze area is excluded from the transmission targets. This also eliminates wasteful update processing in the display control device 10, making it possible to display at a high frame rate.

[0063] 7 is a diagram for explaining the relationship between data transmitted by the image transmission device 20 and the display image generated by the display control device 10. The diagram shows the passage of time horizontally, and the middle part of the diagram shows the arrangement of frames to be displayed. Here, the updated parts of the vertical line representing the entire frame are shown in bold. This example assumes that, as shown in FIG. 2(a), images captured in a remote location are displayed on a flat-panel display, such as for live coverage or video chat.

[0064] In this case, the data transmitted from the image transmitting device 20 is composed of data of the entire original image 180a, 180b, ... and event information 182a, 182b, 182c, ... as shown in the upper part. In the figure, for simplification, it is assumed that there is no change in the background and there is a change in the image of the speaker, and the image of the speaker on the image plane is shown as event information 182a, 182b, 182c, .... However, in reality, as described above, by detecting the occurrence of an event in pixel units, information is transmitted with higher spatial resolution and time resolution, such as the part of the lips that is speaking and the part of the eyes that is blinking. Also, the event information 182a, 182b, 182c, ... includes only events inside the gaze area 192.

[0065] Data of the entire original images 180a, 180b, ... is transmitted at a predetermined cycle, whereas the event information 182a, 182b, 182c, ... is transmitted in response to the occurrence timing of the event asynchronously with the transmission cycle of the entire original images 180a, 180b, .... The display control device 10 generates a frame of a display image using the transmitted data. That is, the display image 184a is generated using the data of the entire original image 180a. The display image 184a corresponds to the frame 188 in the middle row.

[0066] Then, the display control device 10 generates display images 186a, 186b, 186c, etc., in which only the necessary part 190 is updated, using the event information 182a, 182b, 182c, etc., transmitted during the period until the data of the entire original image 180b is next transmitted. Since the transmission timing of the event information 182a, 182b, 182c, etc., which are the source of the updated parts, varies, the display control device 10 periodically updates the frames using the latest event information. Then, the display control device 10 generates a corresponding display image 184b based on the data of the entire original image 180b transmitted next.

[0067] In this way, the display control device 10 realizes display at a high frame rate by interpolating between frames of the entire original images 180a, 180b, ... transmitted from the image transmission device 20 based on the latest event information. For example, the display control device 10 uses original images of 30 fps or 60 fps transmitted from the image transmission device 20 and displays images at 120 fps, 240 fps, etc. By updating only the part that the user is looking at and the part that has changed in pixel units, it has good affinity with the above-mentioned visual characteristics and can display changes with high definition and low delay by reducing the data size to be transmitted and processed.

[0068] Fig. 8 shows another example of the relationship between the data transmitted by the image transmitting device 20 and the display image generated by the display control device 10. In this example, as shown in Fig. 2(b), it is assumed that an area of ​​the panoramic image corresponding to the user's field of view is displayed on a head mounted display. In this case, the data transmitted from the image transmitting device 20 is composed of data of entire original images 200a, 200b, ... and event information 202a, 202b, 202c, ... as shown in the upper part.

[0069] By preparing a panoramic image as the original image, a continuous image world can be shown even if the user suddenly turns around. On the other hand, when limited to a certain time, the angle of view to be displayed on the head mounted display is limited to about 100 to 120 degrees as shown in Figure 6. Furthermore, as mentioned above, the further the area is from the line of sight, the lower the user's visual function becomes. Therefore, treating the entire image uniformly lacks a balance between the amount of information provided and the amount of information received, which is more noticeable than when a centrally projected image is displayed on a flat panel display.

[0070] The image transmission device 20 transmits data of the entire original image 200a, 200b, ... at a predetermined cycle, and transmits event information 202a, 202b, 202c, ... asynchronously. In the figure, animal characters are assumed to be moving, and the images of the characters on the image plane are shown as event information 202a, 202b, 202c, .... Note that in the figure, the entire original image 200a, 200b, ... is shown as a panoramic image, but the transmission target is not limited to this, and may be limited to an area corresponding to the user's field of view or an area predicted to be in the field of view when displayed. The event information 202a, 202b, 202c, ... may also be limited to events within the field of view or the gaze area.

[0071] The display control device 10 generates display images 204a, 204b, ... using the data of the entire original images 200a, 200b, ... and generates display images 206a, 206b, 206c, ... at a predetermined rate by updating only the necessary parts using the event information 202a, 202b, 202c, ... transmitted during that time. Here, the display control device 10 may perform reprojection to correct the field of view so as to correspond to the position and posture of the user's head at each time, and then generate distorted images for the left eye and the right eye. As described above, the position and posture of the user's head can be obtained by using the motion sensor and camera equipped in the head mounted display.

[0072] In this reprojection, the display control device 10 uses the event information 202a, 202b, 202c, ... to draw pixels where an event has occurred. Even in this case, even if the frame rate of the entire original image is lowered, a high-quality image can be displayed at a high frame rate such as 120 fps or 240 fps. In addition, it is possible to achieve both low latency in the change of the field of view in response to the movement of the user's head and low latency in the movement of the image in the gaze area. Since the event detection is performed independently of the user's movement, it is possible to accurately detect the movement in the world coordinate system by comparing it with the inter-frame difference of the displayed image.

[0073] 9 illustrates data formats of panoramic images and event information prepared by image transmission device 20. (a) in the figure illustrates equidistant cylindrical projection, and (b) illustrates equidistant projection format that directly represents the output of a fisheye camera. When capturing a panoramic image by shooting, a panoramic camera is used that combines two fisheye cameras, each with an angle of view of about 220°, on the front and back. When a fisheye camera is used, an equidistant projection image that projects a subject onto a spherical surface can be obtained, as shown in (b).

[0074] In the case of (a), such an equidistant projection image is first converted into equirectangular projection data 210a, and at least a part of the converted data is transmitted. In this case, event information 212a indicates the event occurrence position in position coordinates on the image plane of equirectangular projection. On the other hand, in this embodiment, equidistant projection data 210b as shown in (b) may be transmitted. In this way, there is no need to perform a process for converting the image into equirectangular projection data 210a or to prepare a buffer memory for this purpose, and data transmission with low delay can be achieved.

[0075] In this case, the event information 212b represents the occurrence position of the event in the position coordinates on the image plane in the equidistant projection. In other words, the event information can also be transmitted with low delay without coordinate conversion of the output from the EDS. In the case of (b), only the data of the field of view 214 of the equidistant projection data 210b may be transmitted as the data of the entire original image. Furthermore, for the event information 212b, only information on events that occurred in the gaze area 216 within the field of view 214 may be transmitted.

[0076] While Fig. 9 focuses on the transmission process from the image transmission device 20, the display format on the display device 16 side also affects the reduction of delay until display. Fig. 10 is a diagram for explaining the relationship between the shape of a viewscreen and the amount of information provided. The viewscreen 414 is a screen used when presenting a centrally projected image on a flat panel display. The viewscreen 426 is a screen used when presenting a distorted image taking into account an eyepiece lens. The figure shows both screens as viewed from the side, together with the user's viewpoint 424.

[0077] The view screen 414 is formed, for example, of a plane having a predetermined angle of view centered on the axis C. In this case, the image of the object 425 is displayed in a uniformly reduced state at a scale ratio according to the distance from the view screen 414, regardless of the height from the axis C. On the other hand, a distorted image that assumes an eyepiece lens has properties similar to an image captured by a fisheye lens, and as a result, the view screen 426 has a curved shape as shown in the figure. However, the detailed shape of the view screen 426 depends on the lens design.

[0078] As is clear from the figure, in an angular range 428 of the viewscreen 426 near the optical axis (axis C), the area difference with the corresponding region of the viewscreen 414 is small, whereas the area ratio to the viewscreen 414 decreases as the angular range moves away from the optical axis. For this reason, in a central region 434 of the image, there is almost no difference in the size of the image between the centrally projected image and the distorted image, whereas in peripheral regions 432 and 426, the image in the centrally projected image is significantly reduced in size in the distorted image. In other words, it can be said that a part of the image expressed by central projection contains unnecessary information that is not reflected in the distorted image.

[0079] This is not only true for centrally projected images, but is even more pronounced for equirectangular projection images, in which the image is enlarged above and below the image plane. Therefore, in the case of a head-mounted display, as explained in Fig. 9, by transmitting an equidistantly projected image without converting it into equirectangular projection data, it is possible to eliminate the generation and transmission of unnecessary information in the distorted image that is finally displayed.

[0080] 11 is a diagram for explaining the distorted image drawing process in the display control device 10. The horizontal direction of the figure indicates the passage of time, and the upper part of the figure indicates the arrangement of frames to be displayed. Here, each frame is represented as a curved view screen of a distorted image, and updated parts are indicated by thick lines. Note that the format of the data transmitted from the image transmission device 20 is not limited.

[0081] The display image drawing unit 66 of the display control device 10 draws the entire frame 220a of the display image at the timing when the data of the entire original image is transmitted. At this time, rays are generated that pass through each pixel on the curved view screen, and pixel values ​​are determined by obtaining color information of the arrival point in the original image. This makes it possible to directly draw the distorted frame 220a. In practice, an image pair consisting of an image for the left eye and an image for the right eye is generated, as shown in FIG. 8.

[0082] Until the next transmission of the data of the entire original image, the display image drawing unit 66 generates frames 222a, 222b, and 222c in which only the part where the event occurred is updated based on the event information. In the figure, only the updated parts of the frames 222a, 222b, and 222c are shown, but in reality, the area where no event occurs is also shown as an image. The direction from the optical center of the camera to the pixel where the event occurred can be determined from the position information of the pixel where the event occurred, which is included in the event information, and the camera optical system.

[0083] For example, the display image rendering unit 66 performs ray tracing using the event information only for the necessary parts based on the correspondence between the direction and the ray set at the time of rendering, and determines the pixel value. In this case, too, a distorted image can be directly rendered by using a curved view screen. Then, when the data of the entire original image is next transmitted, the display image rendering unit 66 generates the corresponding display image frame 220b. Through these procedures, it is possible to realize low-delay and high-definition image representation without generating unnecessary processing or transmission, even for distorted images.

[0084] 12 shows a modified example of the image display system in this embodiment. In this example, a server 252 that generates computer graphics using 3D model data 260 is introduced as the image transmission device 20. The server 252 may be connected to imaging devices 240a and 240b that capture panoramic images and wide-angle images, and may acquire the captured images or generate images that are synthesized with computer graphics. The server 252 may be a server in a cloud environment such as cloud gaming.

[0085] In this case, the server 252 also transmits data 266 of the entire original image to the client display control devices 10d, 10d at a predetermined rate, and transmits event information 268a, 268b via a bypass line. In the figure, as examples of the data format of the panoramic image, an equidistant projection image 242 output by a pair of fisheye cameras and a cube map image 270 are shown. The cube map is a two-dimensional map in which the panoramic image is projected onto the inner surface of a cube circumscribing the spherical screen representing the panoramic image and then developed into a development. In this way, the server 252 may use various data formats when representing the panoramic image.

[0086] In any case, of the display control devices 10c and 10d that receive any of these data, the display control device 10c connected to a flat panel display as the display device 16c generates and outputs a centrally projected display image 272. The display control device 10d connected to a head mounted display as the display device 16d generates and outputs a display image 274 consisting of distorted images for the left and right eyes. In addition, the display control devices 10c and 10d realize display at a high frame rate by updating only the necessary parts 276a and 276b in response to the transmission of the event information 268a and 268b.

[0087] In this way, the display control device 10 determines color information pixel by pixel by ray tracing using the transmitted image, thereby enabling display with a unified scheme without limiting the type and data format of the transmitted image and the type of display device 16. In addition, it is possible to locally draw an image based on event information, generate a distorted image without converting it into a centrally projected image, and perform fine control such as drawing some areas on an image, such as the area corresponding to the fovea of ​​the eye, at a higher resolution than other areas.

[0088] FIG. 13 is a diagram for explaining the relationship between partial updating based on event information and reprojection. In the case of a head mounted display, as described above, the display control device 10 performs reprojection to correct the image so as to correspond to the position and posture of the user's head immediately before display. For example, the image transmission device 20 acquires information related to the position and posture of the user's head from the display control device 10 to which the data is transmitted, and generates an image with a field of view corresponding to the information. The display control device 10 makes a final adjustment to the field of view according to the movement of the head during the time from the generation of the image to the display. As a result, the image is displayed with a field of view that follows the movement of the head.

[0089] The image 280 in the upper part of the figure represents an image world generated by the image transmission device 20. In this image world, a spherical object 282 is moving against the background of a belt-shaped object 283, such as the ground below. (t0), (t1), and (t2) attached to the object 282 represent the position of the object 282 at times t0, t1, and t2. In this image world, the display field of view at each time corresponding to the movement of the user's head is assumed to change as in fields of view 284a, 284b, and 284c.

[0090] On the other hand, if the frame rate of the entire original image by the image transmission device 20 is made lower than the display frame rate and only the data at times t0 and t2 is transmitted, the image of the field of view 284b including the object 282 at time t1 is not transmitted as a matter of course. Note that in practice, the image transmission device 20 makes reprojection possible by transmitting image data of a range wider than the field of view 284a as the data of the original image by predicting the movement of the user's head as described above.

[0091] The display control device 10 performs reprojection based on the transmitted data, and generates and outputs a display image as shown in the lower part. That is, the display frames for times t0 and t1 are generated (S10) using the image data transmitted corresponding to time t0, and the display frames for time t2 and the next are generated (S14) using the image data transmitted corresponding to time t2. Here, when the display frame for time t1 in which only the field of view is changed is generated using the image data for time t0, as shown in (a), the object 282 at time t1, which should originally appear in the field of view, is not displayed, even though the area occupied by the background object 283 changes appropriately.

[0092] Therefore, as described above, the image transmission device 20 transmits information on pixels whose luminance has changed due to the movement of the object 282 as event information, at a timing independent of the data transmission of the entire original image. This allows the display control device 10 to properly render an image of the object 282 rendered using the most recent event information in the display frame at time t1, as shown in (b) (S12). Note that for ease of understanding, the figure shows a major change, such as whether or not the object 282 appears in the display image, but in reality, both the frame rate of the original image and the frame rate of the display using it are significantly higher values. This allows very minute changes to be expressed with high definition and low delay.

[0093] When the event information includes a color change in a pixel where an event has occurred, the display control device 10 refers to color information included in the event information for rays to the pixel where the event has occurred in ray tracing. For other rays, the display control device 10 refers to color information of the original image transmitted corresponding to time t0. On the other hand, the display control device 10 may itself draw the object 282 without including color information in the event information.

[0094] 12, the server 252 generates data of a virtual world represented by computer graphics. In this case, the server 252 generates information related to the three-dimensional space to be rendered, such as object depth information, during the image rendering process. The server 252 then transmits the information related to the generated three-dimensional space, allowing the display control device 10 to render, for example, the movement of an object 282 from model data.

[0095] In this case, by also transmitting material information of the target object, accurate light source calculation processing based on the material, such as changes in specular reflection accompanying viewpoint movement, becomes possible. For example, for objects set close to the user or objects that interact with the user, the display control device 10 directly renders the objects, making it possible to efficiently express a realistic image world. This makes it possible to meet the needs of virtual stores that display virtual products such as jewelry, virtual try-on and virtual makeup that allow users to virtually try on items or experience them.

[0096] This aspect is not limited to computer graphics, but can also be applied to live-action images. In other words, the image transmission device 20 separately acquires and transmits depth information and material information of the subject, and the display control device 10 can use the information to draw only objects that have changed. Here, the depth information and material information may be estimated using deep learning or artificial intelligence, as well as using on-the-spot measurements or registered information obtained in advance. Technologies for estimating material information and depth information of a subject from a captured image have been rapidly developing in recent years, and any of these technologies may be adopted (see, for example, Abhimitra Meka and six others, "LIME: Live Intrinsic Material Estimation," 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), (USA), 2018, Vol. 1, pp. 6315-6324).

[0097] Next, examples of the representation format of the panoramic image that can be used in this embodiment are shown. In this embodiment, panoramic images taken by a fisheye camera or a multi-lens camera, or panoramic images drawn by computer graphics can be used. Since panoramic images cannot be represented at once by a general central projection image, a special data format is used to represent a three-dimensional image with inherent directional information on a plane, such as an image projected onto a polyhedron made up of a combination of planes, an equirectangular projection image, or an equidistant projection image.

[0098] In image display technology for viewing through an eyepiece, such as a head-mounted display, a part of such a unique image is cut out, converted into a central projection image, and then a distortion corresponding to the eyepiece is often applied. On the other hand, the image transmission device 20 of this embodiment transmits an original image in a data format other than central projection, and the display control device 10 reads pixel values ​​directly from the original image using a conversion formula corresponding to the data format, and draws a distorted display image.

[0099] This saves resources required for central projection. In addition, since the orientation information inherent in the original image can be transmitted as is, a display image corresponding to the direction of the user's line of sight can be easily generated. For example, when viewing a panoramic image from the same position as the imaging device, the coordinate system of the image centered on the viewpoint is the same, so it is sufficient to display the image in the same direction as the user's line of sight among the captured images. Furthermore, if the distortion aberration of the lens of the fisheye camera that captured the panoramic image and the eyepiece lens are approximately the same, the conversion for the distorted image can be minimized.

[0100] FIG. 14 is a diagram for explaining a display image generation process in the case where a panoramic image is expressed by a cube map. As described above, a cube map is an image 292 that is developed by projecting an image on the surface of a sphere 294 that represents the panoramic image onto each surface of a cube 290 that circumscribes the sphere 294. The image transmission device 20 first sets a ray that reaches each pixel of the six squares that constitute the cube 290 from the center of the sphere 294. Then, the image 292 is generated by setting the color of the surface of the sphere 294 through which the ray (vector rd) passes as the pixel value of the pixel. As a result, the image 292 indicates directional information by the six squares and the position within them. As described above, the image transmission device 20 may transmit only the data of the area of ​​the image 292 that is necessary for display. In this case, the data format of the image is the same.

[0101] The display control device 10 acquires the image 292 or a part of the data thereof, and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of a curved screen corresponding to the eyepiece. Then, the display control device 10 arranges the image 292 as a cube circumscribing a sphere 294 centered on the viewpoint, and samples the color on the image 292 corresponding to the set ray (vector nor) to determine the pixel value. For this process, for example, a cube map texture read command in the cube mapping function of OpenGL, which is a well-known computer graphics library, can be used. Meanwhile, in this embodiment, the event information is also transmitted in the same format.

[0102] That is, the position of the pixel where the event occurred is represented by position coordinates on the plane of the image 292. At this time, the image transmission device 20 may transmit a vector rd of a ray that reaches that pixel. In this case, when generating a new display frame, for example, if the vector rd transmitted as event information matches the ray vector nor set in generating the display frame, the display control device 10 samples the latest pixel information included in the event information and partially updates the image.

[0103] 15 is a diagram for explaining a display image generation process when a panoramic image is expressed by equirectangular projection. An image 302 by equirectangular projection is data developed by projecting an image on the surface of a sphere 294 representing the panoramic image onto the surface of a cylinder 300 that touches the equator. The image transmitting device 20 first sets a ray that reaches each pixel of the cylinder 300 from the center of the sphere 294. Then, the image 302 is generated by setting the color of the surface of the sphere 294 through which the ray (vector rd) passes as the pixel value of the pixel. As a result, the plane of the image 302 represents longitude u in the horizontal direction and latitude v in the vertical direction.

[0104] The display control device 10 acquires data of the image 302 or a part of it, and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the display control device 10 arranges the image 302 as a cylinder circumscribing the equator of a sphere 294 centered on the viewpoint, and samples the color on the image 302 corresponding to the set ray (vector nor) to determine the pixel value. The image transmission device 20 further transmits event information that indicates the position of the pixel where the event occurred by the position coordinates on the plane of the image 302. The process in which the display control device 10 partially updates the image in response to this may be similar to that described in FIG. 14.

[0105] FIG. 16 is a diagram for explaining the display image generation process when a panoramic image is expressed by equidistant projection. In this example, an imaging device is assumed in which fisheye cameras with a viewing angle of 220° are combined on the front and back. That is, the captured image is data on the surface of a sphere 306 centered on the optical center, excluding the range outside 220°. Note that the figure shows only the sphere 306 constituting the captured image of one of a pair of fisheye cameras. Also, the viewing angle of each fisheye camera is not limited to 180° or more. In the case of a fisheye camera, equidistant projection images 310a and 310b are obtained by displaying the output of the sensor array as is. However, it is not limited to the captured image, and equidistant projection images can also be generated by drawing them using computer graphics.

[0106] The image transmitting device 20 generates an image 308 consisting of equidistantly projected images 310a and 310b representing the space in the opposite direction. The horizontal direction of the plane of the image 308 represents longitude u, and the vertical direction represents latitude v. When generating the image 308 by ray tracing, the image transmitting device 20 can generate the image 308 by setting rays at equal intervals from the center of the sphere 294, i.e., the sphere 306, and setting the color of the surface of the sphere 294 (306) through which the vector rd passes as the pixel value.

[0107] The display control device 10 acquires data of the image 308 or a part of it, and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, in the image 308, the color at the position corresponding to the set ray (vector nor) is sampled to determine the pixel value. The image transmission device 20 further transmits event information in which the position of the pixel where the event occurred is expressed by position coordinates on the plane of the image 308. The process in which the display control device 10 partially updates the image in response to this may be similar to that described in FIG. 14.

[0108] Fig. 17 is a diagram for explaining an embodiment in which a panoramic image is represented by equidistant projection and multiple resolution levels are provided. The image used may be an image taken by a device combining two fisheye cameras with a viewing angle of 180° or more on the front and back, as in Fig. 16, or an image drawn by computer graphics in a similar format. Therefore, the original image is data on the surface of a sphere 306 excluding the range outside the viewing angle.

[0109] As explained in Figure 10, images displayed on a curved screen such as a fisheye lens are compressed as they move away from the center of the image, compared to centrally projected images. This characteristic matches the human visual characteristics, in which visual function decreases as the image moves away from the area corresponding to the fovea. Expanding on this, images that represent spaces outside the user's field of view are unlikely to be immediately captured by the fovea, so even if they are prepared at a low resolution, it is thought that the impact on visibility will be small.

[0110] Therefore, the image transmission device 20 independently controls the resolution of the two equidistant projection images that represent the space in the opposite direction. Specifically, each image 314a, 314b has a mipmap structure that is represented at multiple resolution levels. For example, when a user is looking in the direction represented by the image 314b, most of the area of ​​the image 314a is outside the field of view. Therefore, the image transmission device 20 preferentially transmits a low-resolution image for the image 314a as necessary.

[0111] This reduces the overall transmission data size and ensures low latency compared to when the two images 314a and 314b have the same resolution. In addition, the image can be displayed without insufficiency in response to high-speed line-of-sight movement, such as when the user suddenly turns around, and the image world can be viewed without discomfort by gradually increasing the resolution of image 314a as the line of sight approaches.

[0112] The process of generating a display image in the display control device 10 and the process of partially updating an image based on event information may be the same as those described above. In the figure, the panoramic image is expressed by two equidistantly projected images, but the number of equidistantly projected images may be three or more depending on the respective angles of view. In this case, too, the resolutions of the images may be controlled independently by constructing a mipmap structure for each image representing a different direction.

[0113] The image transmission device 20 may generate data in which the resolution of the equidistantly projected image is maximum at the center and decreases as the image moves away from the center. FIG. 18 is a diagram for explaining a method for generating an equidistantly projected image having a distribution in which the resolution decreases from the center to the periphery. First, an equidistantly projected image 318 is represented in two-dimensional coordinates of longitude u and latitude v as shown in (a). When two fisheye cameras are used as shown in FIGS. 16 and 17, two similar images are generated, but the target for which the resolution is given a distribution may be both of them, or may be only the image that the user mainly views.

[0114] The image transmission device 20 sets a ray that reaches each pixel on the plane of the image 318 from the optical center of the camera, and the color of the sphere surface through which the vector rd passes is set as the pixel value of the pixel. However, in this case, the area represented by one pixel is increased by decreasing the density of the ray as it moves away from the image center. The conversion from the position coordinates (u, v) to the vector rd on the plane of the image 318 with a field angle of 220° is realized, for example, by the following calculation.

[0115]

number

[0116] Here, the function f(th) is a monotonically increasing function in which the rate of change increases as the absolute value of the angle th corresponding to the distance from the image center to the position coordinates (u, v) increases, as shown by the dashed line 320a in FIG. 1(b). This allows the density of the ray represented by the vector rd to be reduced as the distance from the image center increases. As a result, the image 318 becomes data in which the area represented by one pixel varies depending on the position on the image plane. The display control device 10 converts the set ray vector nor into the position coordinates (u, v) on the image 318, for example, by the following calculation, and then performs sampling.

[0117]

number

[0118] Here, the function f(th) is the inverse function of the dashed line 320a used to generate the image 318, as shown by the solid line 320b in FIG. 3B. This allows data with different areas represented by one pixel to be represented as data of pixels evenly arranged on the display screen. This processing makes it possible to express the area corresponding to the fovea with higher resolution, reduce the amount of data transmitted, and improve low latency. Note that in the above calculation, the "LODBIAS" parameter is used to read out data corresponding to the mipmap structure, but this is not intended to limit the present embodiment.

[0119] 19 is a diagram for explaining the display image generation process when a panoramic image is displayed on the surface of a regular octahedron. In this case, an image on the surface of a sphere 294 representing the panoramic image is projected onto the surface of a regular octahedron 322 circumscribing it, and then an expanded image 324 is generated. That is, the image transmitting device 20 sets a ray that reaches each pixel of the regular octahedron 322 from the center of the sphere 294, and generates the image 324 by setting the color of the surface of the sphere 294 through which the ray (vector rd) passes as the pixel value of the pixel.

[0120] The display control device 10 acquires data of the image 324 or a part thereof, and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the image 324 is arranged as a regular octahedron circumscribing a sphere 294 centered on the viewpoint, and a pixel value is determined by sampling a color on the image 324 corresponding to the set ray (vector nor). The image transmission device 20 further transmits event information in which the position of the pixel where the event occurred is expressed by position coordinates on the plane of the image 324. The process in which the display control device 10 partially updates the image in response to this may be the same as that described in FIG. 14. Note that various methods other than those described above have been proposed for expressing a panoramic image with two-dimensional data, and any of them may be adopted in this embodiment.

[0121] According to the present embodiment described above, in a system that displays an image using data of an original image transmitted from an image transmitting device, the image transmitting device detects changes in luminance as events on a pixel-by-pixel basis and transmits the data of the entire original image asynchronously. The display control device that has acquired the data generates the entire region of a display frame using the data of the entire original image, and interpolates a display frame in which only the changed portion is updated on a pixel-by-pixel basis based on the event information. Here, the display control device enables updating on a pixel-by-pixel basis by using a ray tracing technique that sets rays corresponding to each pixel on the display image and acquires color information.

[0122] This allows the size of data to be transmitted to be significantly reduced, since detailed image movements can be accurately expressed on the displayed image even if the frame rate of the entire original image is lowered. By limiting the events to be transmitted to within the user's gaze area, the impact on visibility can be minimized and the data size can be further reduced. As a result, even when images with a wide viewing angle, such as omnidirectional images, are displayed on a head-mounted display or a display device with a high pixel count, such as a UHDTV, communication bandwidth and processing resources can be saved, and high-quality image expression with low power consumption and low latency can be achieved.

[0123] In addition, by adopting ray tracing, it is possible to directly generate a display image with distortion corresponding to the distortion aberration and chromatic aberration of the eyepiece lens from an image in which the original image is expressed in a distorted state, such as a panoramic image. As a result, a buffer memory for expanding a centrally projected image is not required, and the processing for image conversion can be simplified. Furthermore, by directly generating a display image with distortion from an equidistantly projected image, the amount of information can be maintained in a distribution suitable for human visual characteristics from transmission to display, eliminating room for unnecessary processing and information.

[0124] In a flat display, the same effect can be obtained by updating only the part of the gaze area where an event has occurred. In this way, this embodiment can be easily applied to any display format, such as a flat display or a head-mounted display, and to any type of displayed image, such as a photographed image, computer graphics, or an image obtained by combining these.

[0125] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0126] 10 display control device, 16 display device, 20 image transmission device, 22 CPU, 24 GPU, 26 main memory, 50 gaze area acquisition unit, 52 image data acquisition unit, 54 event information generation unit, 56 data transmission unit, 60 gaze area transmission unit, 62 image data acquisition unit, 64 event information acquisition unit, 66 display image drawing unit, 70 frame image drawing unit, 72 event information management unit, 74 output unit. [Industrial Applicability]

[0127] As described above, the present invention can be used in various information processing devices such as content servers, game devices, head-mounted displays, display devices, mobile terminals, and personal computers, as well as image display systems including any of these devices.

Claims

1. An image display system including an image transmission device that transmits data of an original image, and a display control device that controls an image display using the data of the original image, The image transmitting device includes: an event information generating unit that generates event information including position information of a pixel when a luminance change event occurs in any pixel in the original image; a data transmission unit that transmits data of the entire original image at a predetermined frame rate and transmits the event information asynchronously; Equipped with The display control device includes: a display image drawing unit that generates data of the display image at a predetermined frame rate by determining pixel values ​​of the display image by ray tracing with the original image as a destination, and locally updates the display image by replacing the destination pixel values ​​on a ray-by-ray basis based on the most recent event information; an output unit that outputs data of the display image to a display device; An image display system comprising:

2. 2. The image display system according to claim 1, wherein the display image drawing section generates the data of the display image at a frame rate higher than a frame rate of the entire original image to be transmitted.

3. The image transmission device further includes a gaze area acquisition unit that acquires information related to a gaze area of ​​a user with respect to the display image, 3. The image display system according to claim 1, wherein the event information generating section selects an event that occurs within the gaze area, and generates the event information relating to the selected event.

4. The image display system according to any one of claims 1 to 3, characterized in that the event information generation unit generates the event information by acquiring an event signal indicating the occurrence of the event from an imaging device equipped with an event-driven sensor that captures the original image.

5. The image transmission device further includes an original image drawing unit that draws the original image, 5. The image display system according to claim 1, wherein the event information generation unit generates the event information by acquiring information on an event occurrence specified in a drawing process by the original image drawing unit.

6. the event information generating unit includes information related to a pixel value of a pixel in which the event has occurred in the event information; 6. The image display system according to claim 1, wherein the display image drawing section locally updates the display image based on the information relating to the pixel values.

7. the event information generating unit includes, in the event information, information used for drawing an object represented by the pixel in which the event has occurred; 7. The image display system according to claim 1, wherein the display image drawing section draws the object by using the information used for drawing.

8. The output unit outputs data of the display image to a head mounted display; The image display system according to any one of claims 1 to 7, characterized in that the display image drawing unit corrects the field of view of the original image in the display image to correspond to the position and posture of the head of a user wearing the head mounted display, and performs local updates based on the event information.

9. the data transmission unit transmits data of the entire original image in which at least a part of the omnidirectional image is represented in a format other than the central projection; 9. The image display system according to claim 8, wherein the display image drawing unit draws a display image with a distortion corresponding to an eyepiece of the head mounted display by ray tracing the original image.

10. The image display system according to claim 9, characterized in that the data transmission unit acquires a plurality of equidistant projection images representing different directions as the panoramic image, generates data of a plurality of resolutions for each image, and transmits data of a selected resolution for each of the images.

11. the data transmission unit transmits data of the entire original image having a distribution in an area represented by one pixel; An image display system according to any one of claims 8 to 10, characterized in that the display image drawing unit uses data of the entire original image to generate data of the display image having a resolution distribution in which the resolution of the central portion is maximum.

12. A display control device that receives data of an original image and controls image display using the data of the original image, an image data acquisition unit that acquires data of the entire original image at a predetermined frame rate; an event information acquisition unit that acquires event information, which is transmitted asynchronously with the data of the entire original image in response to an event of a luminance change occurring in any pixel in the original image, and includes position information of the pixel; a display image drawing unit that generates data of the display image at a predetermined frame rate by determining pixel values ​​of the display image by ray tracing with the original image as a destination, and locally updates the display image by replacing the pixel values ​​of the destination on a ray-by-ray basis based on the most recent event information; an output unit that outputs data of the display image to a display device; A display control device comprising:

13. 1. An image display system including an image transmission device that transmits data of an original image, and a display control device that controls an image display using the data of the original image, The image transmitting device, generating event information including position information of a pixel when a luminance change event occurs in the original image; transmitting data of the entire original image at a predetermined frame rate and asynchronously transmitting the event information; The display control device, generating data of the display image at a predetermined frame rate by determining pixel values ​​of the display image by ray tracing with the original image as a destination, and locally updating the display image by replacing the destination pixel values ​​on a ray-by-ray basis based on the most recent event information; outputting data of the display image to a display device; 2. An image display method comprising:

14. A computer receives data of an original image and controls image display using the data of the original image, a function of acquiring data of the entire original image at a predetermined frame rate; a function of acquiring event information, which includes position information of a pixel in the original image, transmitted asynchronously with the data of the entire original image in response to a luminance change event occurring in the pixel in the original image; a function of determining pixel values ​​of a display image by ray tracing with the original image as a destination, thereby generating data of the display image at a predetermined frame rate, and locally updating the display image by replacing the destination pixel values ​​on a ray-by-ray basis based on the most recent event information; a function of outputting data of the display image to a display device; A computer program for realizing the above.

Images