IMAGE DISPLAY SYSTEM AND IMAGE DISPLAY METHOD

Abstract

To solve the problem in which data transmission costs increase when video quality is improved.SOLUTION: A state information acquisition unit 260 of an image generation device 200 acquires communication status, and state information on a head mounted display 100. An image generation unit 262 generates a display image composed of distorted images for a left eye and a right eye. A reduction processing unit 266 converts the display image into data with magnification different for each area according to the communication status and the like, and transmits the data from an output unit 270. An image size restoration unit 280 of the head mounted display 100 restores the transmitted data to the size of the display image, and causes a display unit 284 to display it.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image display system including an image generating device and a display device, and an image display method. [Background technology]

[0002] Image display systems that allow users to view a target space from any viewpoint are becoming widespread. For example, electronic content that realizes VR (virtual reality) by displaying an image according to the line of sight of a user wearing a head-mounted display in a virtual three-dimensional space is known. Using a head-mounted display can increase the sense of immersion in the video and improve the operability of applications such as games. A walk-through system has also been developed that allows a user wearing a head-mounted display to virtually walk around a space displayed as a video by physically moving around. Summary of the Invention [Problem to be solved by the invention]

[0003] Regardless of the type of display device or the degree of freedom of viewpoint, when the field of view changes or the displayed world moves, high responsiveness is required for image display. On the other hand, to realize more realistic image expression, higher resolution and complex calculations are required, which increases the cost of image processing and data transmission. As a result, the display cannot keep up with the movement of the field of view or the displayed world, which may result in a loss of realism or cause motion sickness.

[0004] In addition, when the display device is a head-mounted display, it is desirable to separate the system that draws the image from the head-mounted display and to transmit data wirelessly in order to facilitate the user's movement. However, the more the quality of the image world is pursued, the greater the amount of data that needs to be transmitted, and the greater the risk of display delays or transmission failures depending on the communication environment. In addition, if power consumption increases due to the load of communication and processing, the head-mounted display may become heavy or heat up due to factors such as the built-in battery, making it difficult to wear.

[0005] The present invention has been made in view of the above problems, and has an object to provide a technique that can display high-quality images with low latency. Another object of the present invention is to provide a technique that can improve the wearing comfort of a head-mounted display. [Means for solving the problem]

[0006] In order to solve the above problems, one aspect of the present invention relates to an image display system including an image generating device that generates data of a display image, and a head mounted display that receives and displays the data of the display image, the image generating device including a reduction processing unit that converts the display image into transmission data with different magnifications for each region of an image plane, and the head mounted display including an image size restoration unit that restores the transmission data to the size of the display image.

[0007] Another aspect of the present invention relates to an image display method, the image display method including a step of generating data of a display image by an image generating device, a step of converting the display image into transmission data having different magnifications for each region of an image plane, a step of transmitting the transmission data to a head mounted display, and a step of restoring the transmission data to a size of the display image by the head mounted display and displaying the restored data.

[0008] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also effective as aspects of the present invention. Effect of the Invention

[0009] According to the present invention, it is possible to achieve both responsiveness and quality of image display, and also to improve the wearing comfort of the head mounted display. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an example of the appearance of a head mounted display according to an embodiment of the present invention; [Diagram 2] 1 is a diagram illustrating an example of a configuration of an image display system according to an embodiment of the present invention. [Diagram 3] 1 is a diagram for explaining an example of an image world displayed on a head-mounted display by an image generating device according to an embodiment of the present invention. FIG. [Figure 4] 11 is a diagram showing an example of a processing procedure for generating an image to be displayed on a head mounted display in the present embodiment. FIG. [Diagram 5] 2 is a diagram showing an internal circuit configuration of the image generating device according to the present embodiment. FIG. [Figure 6] FIG. 2 is a diagram showing an internal circuit configuration of the head mounted display according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing a configuration of functional blocks of an image generating device and a head mounted display according to the present embodiment. [Figure 8] FIG. 4 is a diagram illustrating a magnification map set in the present embodiment. [Figure 9] 11 is a diagram for explaining a process in which a reduction processing unit of the image generating device reduces an image by referring to a magnification map in the present embodiment. FIG. [Figure 10] 10A to 10C are diagrams for explaining the permutation of image data output from a reduction processing unit in the present embodiment. [Figure 11]11 is a diagram for explaining a process in which an image size restoration unit of the head mounted display restores an image size by referring to a magnification map in this embodiment. FIG. [Figure 12] 11 is a diagram illustrating an example of a magnification map that is switched depending on the situation by the image generating device according to the present embodiment. FIG. [Figure 13] FIG. 13 is a diagram showing another example of a magnification map that can be used in the present embodiment. [Figure 14] 11 is a diagram for explaining an aspect in which a unit for controlling a magnification corresponds to an encoding method of image data in the present embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] This embodiment relates to an image display system in which a display device instantly displays moving image data that is streamed from an external device. In this respect, the type of display device is not particularly limited, but a head mounted display will be described as an example. Fig. 1 shows an example of the appearance of a head mounted display 100. In this example, the head mounted display 100 is composed of an output mechanism 102 and a mounting mechanism 104. The mounting mechanism 104 includes a mounting band 106 that goes around the head when worn by the user, thereby fixing the device in place.

[0012] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head mounted display 100, and includes a display panel inside so as to face the eyes when worn. The housing 108 also includes an eyepiece lens that is positioned between the display panel and the user's eyes when the head mounted display 100 is worn, and expands the user's viewing angle. The head mounted display 100 may also include a speaker or earphone at a position corresponding to the user's ear when worn. The head mounted display 100 also includes a motion sensor that detects the translational and rotational movements of the head of the user wearing the head mounted display 100, and thus the position and posture at each time.

[0013] In this example, the head mounted display 100 is equipped with a stereo camera 110 on the front of the housing 108, and captures video of the surrounding real space in a field of view corresponding to the user's line of sight. If the captured image is displayed immediately, the user can see the real space in the direction in which he or she is facing, which is known as video see-through. Furthermore, if a virtual object is drawn on the image of a real object captured in the captured image, augmented reality can be achieved.

[0014] 2 shows an example of the configuration of an image display system according to the present embodiment. The head mounted display 100 is connected to the image generating device 200 via an interface for connecting peripheral devices such as wireless communication or a USB. The image generating device 200 may further be connected to a server via a network. In this case, the server may provide the image generating device 200 with an online application such as a game in which multiple users can participate via the network.

[0015] The image generating device 200 identifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head mounted display 100, generates a display image so as to provide a corresponding field of view, and outputs the image to the head mounted display 100. In this respect, the purpose of displaying the image may be various. For example, the image generating device 200 may generate a virtual world in which the game is set as a display image while progressing through an electronic game, or may display still images or moving images for viewing or providing information regardless of whether the virtual world is the real world. Displaying a panoramic image with a wide angle of view centered on the user's viewpoint can give the user a sense of immersion in the displayed world.

[0016] 3 is a diagram for explaining an example of an image world that the image generating device 200 displays on the head mounted display 100 in this embodiment. In this example, a state is created in which the user 12 is in a room, which is a virtual space. As shown in the figure, objects such as walls, floors, windows, a table, and objects on the table are arranged in a world coordinate system that defines the virtual space. The image generating device 200 defines a view screen 14 in the world coordinate system according to the position of the viewpoint and the direction of the line of sight of the user 12, and draws a display image by expressing the image of the object thereon.

[0017] By acquiring the position of the user's 12 viewpoint and the direction of his / her line of sight (hereinafter, these may be collectively referred to as "viewpoint") at a predetermined rate and changing the position and direction of the view screen 14 accordingly, it is possible to display an image in a field of view corresponding to the user's viewpoint. By generating stereo images with parallax and displaying them respectively in the left and right areas of the display panel, it is also possible to provide a stereoscopic view of the virtual space. This allows the user 12 to experience virtual reality as if he / she were in a room in the displayed world.

[0018] Fig. 4 shows an example of a processing procedure for generating an image to be displayed on the head mounted display in the form shown in Fig. 3. First, an image 16 corresponding to the user's field of view is generated by projecting an object existing in the virtual world onto a view screen corresponding to the user's viewpoint. In reality, this processing involves converting the vertex coordinates of the object in the world coordinate system that defines the virtual world into the coordinate system of the view screen, and then mapping a texture onto the surfaces that make up the object. The image in question is originally a representation of the image that the user should view.

[0019] For stereoscopic viewing, a stereo image consisting of an image 18a for the left eye and an image 18b for the right eye is generated by shifting the images in the image 16 laterally by an amount corresponding to the distance between the left and right eyes, or by generating an image 16 for each eye. Then, each image is inversely corrected to match the distortion caused by the eyepiece lens, generating the final display image 22. Inverse correction here refers to a process of distorting the image in advance in the opposite direction to the distortion caused by the lens, so that the original image 16 is visible when viewed through the eyepiece lens. For example, in the case of a lens that causes the four sides of the image to appear concave like a pin spool, the image is curved into a barrel shape as shown in the figure.

[0020] Basically, the image generating device 200 periodically acquires information related to the position and posture of the head mounted display 100 and, in turn, the user's head, generates display images with a corresponding field of view, and sequentially transmits them to the head mounted display 100. The head mounted display 100 sequentially outputs the received images to a display panel, but due to the image rendering time, data transfer time, and the like, the field of view of the received images may not correspond to the actual viewpoint at the time of display.

[0021] As a countermeasure for this, the head mounted display 100 may have a reprojection function that corrects the field of view of an image immediately before display so as to correspond to the actual viewpoint. When realizing reprojection, the image generating device 200 generates and transmits an image by adding a margin area for reprojection to the field of view assumed from the acquired position and posture of the user's head. As a result, the head mounted display 100 appropriately cuts out and displays an image in a range corresponding to the viewpoint immediately before display (for example, see JP 2017-215875 A).

[0022] If the margin area is set wide, an image corresponding to the viewpoint at the time of display can be displayed without loss even when the user's head moves quickly. In a system in which an image is viewed through an eyepiece such as the head mounted display 100, it is desirable to generate an image with a high resolution that assumes a magnification ratio for the area enlarged by the lens. In order to maintain image quality in such a variety of situations, it becomes necessary to transmit a large amount of data from the image generating device 200 to the head mounted display 100, which hinders low latency and wireless communication.

[0023] Therefore, the image generating device 200 of this embodiment reduces the data size by transmitting a reduced version of a portion of each frame of a display image that is difficult to view in detail due to human visual characteristics and eyepiece lens characteristics. For example, the image generating device 200 reduces the image of an area that is a predetermined distance or more from the center of the left eye image and the right eye image shown in the left and right areas of the display image 22 in FIG. 4.

[0024] The head mounted display 100 enlarges the reduced area to return the image to its original size and displays it. In this case, the resolution of the reduced area is reduced, but considering that the visual characteristics of an average person are such that the ability to distinguish information decreases the further away from the gaze point, this is difficult to recognize. In addition, in an image distorted according to the eyepiece, the resolution is originally low at the periphery, so the impact on visibility is small.

[0025] Therefore, by appropriately selecting the area to be reduced in this way, it is possible to reduce the size of data to be transmitted while maintaining the quality as perceived by the user. In addition, by excluding data outside the user's field of view, i.e., data in areas filled with black in display image 22, from the data to be transmitted, the data size can be further reduced. The area to be reduced, its magnification, or its distribution may be changed according to the situation even within one video image.

[0026] For example, during periods when the user's head moves and the gaze point is not fixed, or during periods when the object to be displayed moves significantly and the image is perceived as blurred, widening the area to be reduced or increasing the degree of reduction is unlikely to be noticed. Also, during periods when communication conditions are deteriorating, by reducing the data size by widening the area to be reduced or increasing the degree of reduction, it is possible to avoid situations such as partial loss of the displayed image or blackouts. By controlling the data size accordingly in this way, it is possible to continue displaying detailed images for areas that are important to the user's perception.

[0027] 5 shows the internal circuit configuration of the image generating device 200. The image generating device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and a main memory 226. These components are connected to each other via a bus 230. An input / output interface 228 is further connected to the bus 230.

[0028] The input / output interface 228 is connected to a communication unit 232 consisting of a peripheral device interface such as USB or IEEE1394, a network interface for wired or wireless LAN, a storage unit 234 such as a hard disk drive or non-volatile memory, an output unit 236 that outputs data to the head mounted display 100, an input unit 238 that inputs data from the head mounted display 100, and a recording medium drive unit 240 that drives a removable recording medium such as a magnetic disk, optical disk, or semiconductor memory.

[0029] The CPU 222 executes an operating system stored in the storage unit 234 to control the entire image generating device 200. The CPU 222 also executes various programs that are read from a removable recording medium and loaded into the main memory 226 or downloaded via the communication unit 232. The GPU 224 has a geometry engine function and a rendering processor function, performs drawing processing in accordance with drawing commands from the CPU 222, and outputs the results to the output unit 236. The main memory 226 is composed of a RAM (Random Access Memory), and stores programs and data necessary for processing.

[0030] 6 shows the internal circuit configuration of the head mounted display 100. The head mounted display 100 includes a CPU 120, a main memory 122, a display unit 124, and an audio output unit 126. These units are connected to each other via a bus 128. An input / output interface 130 is further connected to the bus 128. A communication unit 132 consisting of a wired or wireless LAN network interface, a motion sensor 134, and a stereo camera 110 are connected to the input / output interface 130.

[0031] The CPU 120 processes information acquired from each unit of the head mounted display 100 via the bus 128, and supplies display image and audio data acquired from the image generating device 200 to the display unit 124 and audio output unit 126. The main memory 122 stores programs and data necessary for processing in the CPU 120.

[0032] The display unit 124 is composed of a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of a user wearing the head mounted display 100. As described above, stereoscopic vision may be realized by displaying a pair of parallax images in areas corresponding to the left and right eyes. The display unit 124 further includes a pair of lenses that are positioned between the display panel and the user's eyes when the head mounted display 100 is worn, and expand the user's viewing angle.

[0033] The audio output unit 126 is composed of speakers or earphones provided at positions corresponding to the user's ears when the head mounted display 100 is worn, and allows the user to hear audio. The communication unit 132 is an interface for sending and receiving data to and from the image generating device 200, and realizes communication by a known wireless communication technology such as Bluetooth (registered trademark). The motion sensor 134 includes a gyro sensor and an acceleration sensor, and acquires the angular velocity and acceleration of the head mounted display 100.

[0034] 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints in a field of view corresponding to the user's viewpoint. Measurement values ​​by the motion sensor 134 and data on the captured images by the stereo camera 110 are transmitted to the image generating device 200 via the communication unit 132 as necessary.

[0035] Fig. 7 shows a functional block configuration of the image generating device 200 and the head mounted display 100 in this embodiment. As described above, the image generating device 200 may perform general information processing such as progressing an electronic game and communicating with a server, but Fig. 7 focuses on the function of generating and transmitting a display image. At least a part of the functions of the image generating device 200 shown in Fig. 7 may be implemented in a server connected to the image generating device 200 via a network.

[0036] 7 can be realized in hardware by the configuration of the CPU, GPU, various memories, etc., shown in Figures 5 and 6, and in software by programs that perform various functions such as a data input function, a data storage function, an image processing function, a communication function, a display function, etc., loaded into memory from a recording medium, etc. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination of both, and are not limited to any one of them.

[0037] The image generating device 200 includes a status information acquisition unit 260 that acquires information relating to the real-time status of the head mounted display 100 and communication, an image generating unit 262 that generates a display image, an image data storage unit 264 that stores data used to generate the image, a reduction processing unit 266 that reduces the generated image as necessary, a magnification map storage unit 268 that stores a map showing the distribution of reduction magnifications, and an output unit 270 that outputs data of the generated image to the head mounted display 100.

[0038] 5, CPU 222, etc., and acquires information related to the position and orientation of the user's head transmitted from the head mounted display 100 at a predetermined rate. Alternatively, the state information acquisition unit 260 may acquire data such as the measured values ​​of the motion sensor 134 and images captured by the stereo camera 110 from the head mounted display 100 at a predetermined rate, and calculate the position and orientation of the user's head based on at least one of them. As a method for acquiring the position and orientation from the captured image, an existing technology such as V-SLAM (Visual Simultaneous Localization and Mapping) can be used.

[0039] The state information acquisition unit 260 may also acquire the communication status with the head mounted display 100 at a predetermined rate. Here, the communication status refers to the communication bandwidth and communication delay realized at that time point, etc. For example, the state information acquisition unit 260 acquires the delay time from when image data is transmitted until it arrives at the head mounted display 100 based on a response signal from the head mounted display 100.

[0040] Alternatively, the status information acquisition unit 260 acquires the ratio of data that has reached the head mounted display 100 out of the transmitted image data as a data arrival rate from the head mounted display 100. Based on this information, the status information acquisition unit 260 can acquire the amount of data that has been transmitted per unit time, that is, the available bandwidth.

[0041] The image generation unit 262 is composed of the CPU 222, GPU 224, etc. in Fig. 5, and generates display images at a predetermined rate using the procedure shown in Fig. 4. That is, the image generation unit 262 sets a view screen corresponding to the viewpoint based on information about the position and orientation of the user's head, and generates images for the left and right eyes by projecting objects that exist in a three-dimensional virtual space. Then, by applying distortion to each image according to the eyepiece of the head mounted display 100 and arranging them on the left and right of the image plane, an image like the display image 22 in Fig. 4 is generated. When realizing the above-mentioned reprojection, the image generation unit 262 generates an image by adding a margin area.

[0042] 5 and the like, and stores model data of the object to be displayed, data related to the configuration of the virtual space, and information related to distortion corresponding to the eyepiece of the head mounted display 100. Note that the display image generated by the image generating device 200 is not limited to a three-dimensional object, and may be a separately captured panoramic image, a real-time image captured by the stereo camera 110 of the head mounted display 100, or an image obtained by processing the image.

[0043] Alternatively, the image generating device 200 may obtain images of various contents such as cloud gaming and movies transmitted from a server (not shown) via a network, and decode and expand them on the fly. The reduction processing unit 266 is composed of the CPU 222, the GPU 224, etc. shown in Fig. 5, and reduces the display image (frame) generated by the image generating unit 262 at a different magnification for each region on the image plane. The reduction processing unit 266 also excludes regions outside the field of view of the eyepiece from the transmission target.

[0044] The magnification map storage unit 268 is composed of the main memory 226 in FIG. 5 or the like, and stores setting data of one or more magnification maps that indicate the distribution of reduction magnifications for an image plane. The magnification map may be in the form of a lookup table that associates the position coordinates of pixels in the image plane with the magnifications. The reduction processing unit 266 sequentially reduces the data of the image sequence supplied from the image generating unit 262 at the magnifications set in the magnification map. At this time, the reduction processing unit 266 may acquire at least one of the communication status with the head mounted display 100 and information related to the movement of the user's head from the state information acquisition unit 260, and switch the magnification map to be referred to based on the acquired information.

[0045] For example, in a state where it is determined that there is no need to suppress the data size, such as when the communication bandwidth is above a threshold, the reduction processing unit 266 refers to the area of ​​the region to be reduced or the magnification map with the minimum degree of reduction set by the magnification. The reduction processing unit 266 also refers to the area of ​​the region to be reduced or the magnification map with the minimum degree of reduction, even in a state where it is necessary to suppress degradation in image quality, such as when the angular velocity of the user's head is below a threshold. Note that the reduction processing unit 266 does not need to perform reduction processing in these states.

[0046] Then, as the communication band decreases and as the angular velocity of the head increases, the reduction processing unit 266 switches the reference to a magnification map with a larger area of ​​the region to be reduced and a larger degree of reduction. For this reason, the magnification map storage unit 268 stores the range of the communication band and the range of the angular velocity of the head in association with the magnification map to be adopted. Note that the parameters used as the basis for switching the reference magnification map are not limited to the communication band and the angular velocity of the head.

[0047] That is, any parameter may be used as long as it can derive the degree of "necessity to suppress data size" or the degree of "necessity to suppress degradation of image quality," and may be communication delay time, head speed or acceleration, speed or acceleration of the user's gaze point relative to the displayed image, object speed relative to the gaze point, magnitude of optical flow, etc. Any of these parameters may be used alone, or multiple parameters may be used in combination.

[0048] For example, the reduction processing unit 266 uses these parameters to quantify the degree of "necessity to reduce data size" and the degree of "necessity to reduce degradation of image quality" as scores. Since there is a trade-off between the "necessity to reduce data size" and the "necessity to reduce degradation of image quality", the magnification map to be referenced is uniquely determined based on the balance between the scores of both.

[0049] 6, the main memory 226, the output unit 236, etc., and sequentially transmits display image data after the reduction processing unit 266 reduces the data as necessary to the head mounted display 100. Note that the output unit 270 may compress and encode the data using an encoding method such as H.264 / MPEG-4 AVC or H.265 / MPEG-H HEVC, and then transmit the compressed data to the head mounted display 100 via streaming.

[0050] The head mounted display 100 includes a status information transmission unit 276 that transmits information related to the real-time status of the head mounted display 100, an image data acquisition unit 278 that acquires image data transmitted from the image generating device 200, an image size restoration unit 280 that restores the acquired data to the size of the displayed image, a magnification map memory unit 282 that stores a magnification map used for the restoration, and a display unit 284 that displays the image whose size has been restored.

[0051] 6, the motion sensor 134, the stereo camera 110, the communication unit 132, etc., and transmits information relating to the position and posture of the user's head, or measurement values ​​and captured image data for deriving the information, to the image generating device 200 at a predetermined rate. When adjusting the reduction ratio and its distribution according to the movement of the gaze point, the state information transmitting unit 276 transmits information relating to the position coordinates of the gaze point measured by a gaze point detector (not shown) to the image generating device 200.

[0052] The status information transmission unit 276 may also transmit data for deriving the communication status with the image generating device 200, i.e., a response signal, information such as the proportion of data received by the image data acquisition unit 278, etc., to the image generating device 200 as needed. The image data acquisition unit 278 is made up of the CPU 120, communication unit 132, main memory 122, etc. in Fig. 6, and acquires display image data transmitted from the image generating device 200. If the data is compression-encoded, it is decoded and expanded sequentially.

[0053] The image size restoration unit 280 is composed of the CPU 120, main memory 122, etc. of Fig. 6, and restores the data transmitted from the image generating device 200 to the size of the display image for each frame. That is, of the acquired data, the images of the reduced areas are enlarged and connected together to return to the original size of the display image. The magnification map storage unit 282 is composed of the main memory 122, etc. of Fig. 6, and stores the magnification map data that the image size restoration unit 280 refers to when restoring the image size.

[0054] The image size restoration unit 280 restores the size to the original size by referring to the same magnification map used by the reduction processing unit 266 of the image generating device 200 and enlarging the image by the reciprocal of the set magnification. For this reason, the magnification map is provided in advance with identification information that is common to the image generating device 200 and the head mounted display 100. The image generating device 200 embeds the identification information of the magnification map used in the reduction process in the header of the image data or the like, thereby enabling the head mounted display 100 to identify the appropriate magnification map.

[0055] When switching the magnification map during display, the image generating device 200 may transmit identification information of the magnification map to the head mounted display 100 each time. Note that the means for identifying the magnification map is not limited to transmitting and receiving identification information. For example, the head mounted display 100 may also associate the ranges of parameters representing various states with the magnification maps, and may identify the magnification map to be referenced by transmitting the actual values ​​from the image generating device 200 or by the image size restoration unit 280 acquiring them by itself.

[0056] 6, main memory 122, display unit 124, etc., and drives a display panel so as to display the image expanded in the memory restored by image size restoration unit 280. When realizing reprojection, display unit 284 acquires the position and orientation of the user's head at that time, corrects the image based on that, and outputs it to the display panel.

[0057] Fig. 8 shows a schematic diagram of a magnification map set in this embodiment. As described above, the magnification map 300 may be in the form of a lookup table or the like in which the reduction magnification corresponds to the position coordinates on the image plane, but in the figure, it is shown as a distribution of magnification on the image plane. In this example, an image plane consisting of an area 302a for the left eye and an area 302b for the right eye is assumed, as in the display image 22 in Fig. 4. The area is then divided into concentric circles centered on the image centers 304a and 304b, which correspond to the centers of the left and right lenses, and these are used as units for setting the magnification.

[0058] More specifically, the image is divided into a circular area A including the center 304a, 304b, a hollow circular area B outside of that, and a hollow circular area C further outside. Areas D and E outside area C are connected at the boundary between the left eye area 302a and the right eye area 302b as shown in the figure. Area F including the side of the image plane is outside the field of view and therefore has invalid pixel values. In the magnification map, a reduction magnification is associated with the pixels constituting each of the areas A to F thus divided.

[0059] For example, the magnifications (area ratios) of 1, 1 / 4, 1 / 16, etc. are set for the pixels constituting areas A, B, C, etc. This allows the left eye area 302a and the right eye area 302b to be set so that the degree of reduction increases as the distance range from the image centers 304a, 304b increases. Also, for the outermost area F, a setting is made to that effect so that invalid data is not transmitted.

[0060] Here, the width of regions B to E may be equal to the radius of region A, or may be determined independently as shown in the figure. For example, the width and magnification of each region may be optimized according to the amount of change in image height caused by refraction of the eyepiece. In addition, when multiple magnification maps are set, the divided regions as shown in the figure may not be changed, and the magnification set for each region may be different, or the number, size, and shape of the regions may be different.

[0061] Fig. 9 is a diagram for explaining the process in which the reduction processing unit 266 of the image generating device 200 reduces an image by referring to a magnification map. The left side of the diagram shows the magnification map to be used, and in this example, the magnification map 300 consisting of areas A to F shown in Fig. 8 is assumed. The right side of the diagram shows how the reduction processing unit 266 reduces the image according to the settings of each area, with the smallest rectangle representing one pixel area. The reduction processing unit 266 first divides the display image into pixel blocks of the same size, and reduces the image in units of that. In the example shown, one pixel block is 8 x 8 pixels.

[0062] In this example, the pixel blocks included in areas A and B are scaled down by 1, meaning the image is not reduced and remains at 8x8 pixels. The pixel blocks included in area C are reduced to 1 / 2 in both vertical and horizontal directions to become 4x4 pixels. The pixel blocks included in area D are reduced to 1 / 4 in both vertical and horizontal directions to become 2x2 pixels. The pixel blocks included in area E are reduced to 1 / 8 in both vertical and horizontal directions to become 1x1 pixels. Any existing method may be used for reduction, such as the bilinear method, nearest neighbor method, or bicubic method.

[0063] Furthermore, pixel blocks contained in area F are excluded from the transmission target. The reduction processing unit 266 refers to the magnification map and specifies whether to leave the pixel blocks of each area as they are or reduce them, and if so, the setting value of the magnification. If the communication situation, head movement, or other conditions change, the reduction processing unit 266 switches the referenced magnification map and changes the area to be reduced and the reduction magnification.

[0064] For example, if the communication conditions deteriorate, the reduction processing unit 266 also reduces the pixel blocks included in region B by a factor of 4. If the head movement stops, the reduction processing unit 266 excludes the pixel blocks included in region C from the reduction targets. Those skilled in the art will understand that the reduction ratio for each region can be adjusted in various ways so as to obtain a target data size.

[0065] Fig. 10 is a diagram for explaining the permutation of image data output from the reduction processing unit 266. When the pixel block is 8 x 8 pixels as shown in Fig. 9, the reduction processing unit 266 progresses the reduction process from the left end to the right for each pixel column of 8 rows. For example, as shown in the enlarged view in the middle of the figure, of the five pixel blocks in area 312, the leftmost pixel block is included in area F and is therefore excluded from the transmission target. The second and third pixel blocks from the left are included in area E and are therefore reduced to the size of one pixel according to the settings in Fig. 9.

[0066] Since the fourth and fifth pixel blocks are included in area D, they are reduced to a size of 2x2 pixels according to the settings in Fig. 9. In detail, the reduction processing unit 266 scans the pixels in raster order in units of pixel blocks, as indicated by an arrow (e.g., arrow 316) in each pixel block, and sequentially excludes data or uses the data as input values ​​for the reduction calculation. As a result, data is supplied in raster order in units of pixel blocks from the reduction processing unit 266 to the output unit 270. However, data in area F is not supplied.

[0067] In the illustrated example, as shown in the bottom row, data for one pixel in the second pixel block, one pixel in the third pixel block, four pixels in the fourth pixel block, four pixels in the fifth pixel block, etc. are output in that order. Note that when the reduction result is a multi-pixel result such as 2×2 pixels, data is output in raster order starting from the upper left pixel in each block, as indicated by an arrow in the reduced result block (e.g., arrow 318).

[0068] Similarly, in the case of an area that is not subject to reduction, such as area A, pixel values ​​are output in raster order within a block of 8×8=64 pixels. In this case, the output unit 270 may output all frames as intra frames using an encoding method such as H.265 / MPEG-H HEVC.

[0069] Fig. 11 is a diagram for explaining the process in which the image size restoration unit 280 of the head mounted display 100 restores the image size by referring to the magnification map. The left side of the figure shows the magnification map to be used, and in this example, the magnification map 300 consisting of the divided areas A to F shown in Fig. 8 is assumed. The magnification map 300 is the same as that used by the image generating device 200 for the reduction process. The right side of the figure shows how the image size restoration unit 280 restores the image size to its original size in accordance with the settings of each area.

[0070] The image size restoration unit 280 refers to the magnification map specified based on the identification information of the magnification map transmitted from the image generating device 200, and divides the image into pixel blocks based on the permutation of the transmitted pixels. At this stage, pixel blocks of various sizes exist, so the image size restoration unit 280 enlarges them as appropriate. In the example shown, the pixel blocks included in areas A and B remain unenlarged at 8×8 pixels. The pixels included in area C are expanded from 16 pixels to 4×4 pixels, and then enlarged twice in both the vertical and horizontal directions to make them 8×8 pixels.

[0071] For pixels included in region D, four pixels are expanded into 2x2 pixels, and then expanded four times in both vertical and horizontal directions to form 8x8 pixels. For pixels included in region E, one pixel is expanded eight times in both vertical and horizontal directions to form 8x8 pixels. Any existing method such as the bilinear method, nearest neighbor method, or bicubic method may be used for the expansion. Furthermore, since data for pixels included in region F is not transmitted, data for 8x8 pixels with invalid pixel values ​​is prepared in advance. The image size restoration unit 280 then forms a display image by sequentially connecting adjacent pixel blocks.

[0072] Fig. 12 illustrates an example of a magnification map that the image generating device 200 switches depending on the situation. As described above, the reduction processing unit 266 may switch the reference magnification map based on various real-time state information acquired by the state information acquisition unit 260. In the figure, the magnification map 300a is the same as the magnification map 300 shown in Fig. 8 and other figures. On the other hand, in the magnification map 300b, the area of ​​the region E, which is reduced to a large extent, is larger and the area of ​​the region A, which is not subject to reduction, is smaller than that of the magnification map 300a.

[0073] Therefore, qualitatively, the magnification map 300a is used when there is a high "need to suppress degradation in image quality," and the magnification map 300b is used when there is a high "need to suppress data size." For example, the reduction processing unit 266 performs reduction processing using the magnification map 300a in a normal state. Then, when it detects that the available communication bandwidth has fallen below a threshold, it switches the reference destination to the magnification map 300b. Thereafter, when the available communication bandwidth exceeds the threshold, the reduction processing unit 266 returns the reference destination to the magnification map 300a.

[0074] The bandwidth threshold for switching the reference destination to magnification map 300b and the bandwidth threshold for returning to magnification map 300a may be the same or different. By making the threshold different, it is possible to prevent the switching of the magnification map from occurring more frequently than necessary. The parameter that triggers the switching of the magnification map is not limited to the communication bandwidth, and may be various, such as the speed of the head or the point of gaze, or the speed of the object, as described above. A combination of multiple parameters may also be used.

[0075] The timing of the switch is not limited to when the parameter exceeds a threshold value, but may be when the scene being displayed changes, etc. Furthermore, although the figure shows an example of switching between two magnification maps, three or more magnification maps may be switched for three or more state transitions. Alternatively, one magnification map may be prepared and used to switch between reducing the image or not reducing the image as a whole.

[0076] Fig. 13 shows another example of a magnification map that can be used in this embodiment. The area symbols A to F in the figure are set to the same magnification as the magnification map 300 exemplified above. First, in (a), area D is not a hollow circle, but is an approximately quadrangular area (rounded rectangle) that straddles the areas for the left eye and the right eye. In addition, the center of area D is shifted downward from the center of the image plane, and the area ratio of the lower half is increased, so that the area of ​​area E, which is reduced to a large extent, is increased on the upper side.

[0077] Human vision has the characteristic of being more sensitive to objects below the top of the field of view than to objects above it. Therefore, instead of reducing the image symmetrically from top to bottom, by making the average degree of reduction in the lower half of the image smaller than that in the upper half, degradation of image quality becomes less noticeable even with the same data size. In the figure, the image plane has asymmetric regions above and below, but the regions for the left eye and the right eye may be asymmetric left and right, or may be asymmetric up, down, left and right. For example, the degree of reduction may be suppressed in the region where the face is facing or where the gaze point is moving.

[0078] In addition, the shape of the region is not limited, and may be a circle, a hollow circle, a rectangle, a rectangle with rounded corners, an ellipse, a hollow ellipse, a polygon, or the like, and any two or more of these shapes may be combined. (b) shows a magnification map consisting of only three types of regions C, E, and F, without setting a region A that is not subject to reduction. For example, when the angular velocity of the user's head is equal to or greater than a threshold value, the displayed object is perceived as blurred. Therefore, by using a magnification map as shown in the figure, the degree of reduction is relatively large even near the center of the image, so that the effect on recognition is small and the data size can be significantly reduced. In this way, the number of regions for which the magnification is set is not limited, but it is preferable to relatively reduce the degree of reduction near the center of the image for the left eye and the right eye.

[0079] In the above-mentioned aspects, the degree of reduction is generally greater the further away from the center of the left-eye and right-eye images. This allows the data size to be reduced while minimizing the impact on the image quality as perceived by the user due to the characteristics of human vision and the characteristics of viewing images through a lens. On the other hand, when an image is divided into blocks and compressed and encoded in units for transmission, the efficiency of the processing can be improved by using the units in the reduction process.

[0080] 14 is a diagram for explaining an aspect in which a unit for controlling a magnification ratio corresponds to an encoding method of image data. In the case of H.265 / MPEG-H HEVC, which is a video compression standard, an image of one frame is divided into slices or tiles of a predetermined size, and compression encoding is performed in units of the slices or tiles. Therefore, by performing the reduction process in this embodiment in units of the slices or tiles, a series of processes including reduction, compression encoding, transmission, decoding / expansion, and enlargement can be performed in the same units.

[0081] This reduces data waiting time and the time required to rearrange the data order due to different processing units, and enables display with lower delays. Figure (a) shows a magnification map assuming compression encoding is performed in slice units, which are formed by dividing the image plane into six horizontal slices. In this example, the top slice 320a and bottom slice 320b shown in gray are targeted for reduction. An appropriate reduction magnification is set for the slices to be reduced.

[0082] (b) is a magnification map assuming that compression encoding is performed in slice units obtained by dividing the image plane vertically into 12 slices. In this example, the leftmost slice 322a and the rightmost slice 322b of the left eye image, and the leftmost slice 322c and the rightmost slice 322d of the right eye image, shown in gray, are targets for reduction. In this way, by following the division rules of the encoding method and reducing the image in areas away from the center of the left eye and right eye images, it is possible to reduce the impact on recognition, suppress the data size, and further reduce delay.

[0083] In the figure, slices are divided into two types, those that are reduced and those that are not, but as in Figure 8, multiple reduction ratios may be assigned to slices so that the degree of reduction increases the farther away from the center of the image. Also, in this embodiment, the allocation of reduction ratios may be changed according to real-time status information. That is, in a situation where it is desired to suppress the data size, the overall degree of reduction may be increased, or the number of slices to be reduced may be increased.

[0084] Also in this case, as described with reference to (a) of Fig. 13, the reduction may be performed so that the magnification is distributed asymmetrically above and below the image plane, or between the left and right image regions for the left and right eyes. Furthermore, the reduction magnification may be controlled in units of tiles obtained by dividing a slice.

[0085] According to the embodiment described above, the image generating device transmits data in which one display image is magnified differently for each region, and the image is displayed on the display device at the original size. At this time, the magnification is set with a distribution that takes into account human visual characteristics. When using a head-mounted display that is viewed through an eyepiece, the distribution of magnification is set taking into account distortion caused by the lens, and data outside the field of view is excluded from the transmission target. This reduces the impact on recognition, suppresses the size of the transmitted data, reduces the cost required for processing and communication, and realizes reliable display with low latency.

[0086] In addition, by reducing information that is excessive in recognition, it is possible to increase the amount of information in the area that is magnified by the lens and in the area that the user can easily view in detail. Furthermore, by reducing the data size, it is also easy to increase the frame rate. As a result, the user can view a high-quality image without difficulty. In addition, by changing the magnification distribution settings, data transmission and image display can be performed in an optimal state according to the situation.

[0087] In addition, when using a head-mounted display, it is easy to make it wireless and simplify the internal structure, which allows for weight reduction and reduced heat generation, improving the comfort of wearing the device. Furthermore, when implementing a video distribution system via a server or network, such as cloud gaming, if the distribution data is similarly reduced, a low-latency, high-quality video experience can be easily achieved.

[0088] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0089] 100 head mounted display, 110 stereo camera, 120 CPU, 122 main memory, 124 display unit, 132 communication unit, 134 motion sensor, 200 image generating device, 222 CPU, 224 GPU, 226 main memory, 260 status information acquisition unit, 262 image generation unit, 264 image data storage unit, 266 reduction processing unit, 268 magnification map storage unit, 270 output unit, 276 status information transmission unit, 278 image data acquisition unit, 280 image size restoration unit, 282 magnification map storage unit, 284 display unit.

Claims

1. An image display system that displays a display image transmitted from an image generating device on a head mounted display, The image generating device comprises: a state information acquisition unit that acquires at least one of a communication status with the head mounted display and information related to a movement of the head mounted display; a reduction processing unit that divides an image plane of the display image into a plurality of regions by changing an area according to the information acquired by the state information acquisition unit, and reduces each of the regions by a different magnification to reduce the number of pixels; an output unit that compresses and encodes the display image data with a reduced number of pixels and transmits the compressed image data to the head mounted display; The head mounted display includes an image size restoration unit that restores the transmitted data to a size of a display image, An image display system characterized in that the reduction processing unit changes the area so that the amount of reduction in the number of pixels increases as the need for data size reduction based on the information acquired by the status information acquisition unit increases.

2. 2 . The image display system according to claim 1 , wherein the reduction processing unit changes the magnification ratio set for each of the plurality of regions in accordance with the information acquired by the state information acquisition unit.

3. 3. The image display system according to claim 1, wherein the reduction processing unit changes the number of the plurality of regions in accordance with the information acquired by the state information acquisition unit.

4. The reduction processing unit reduces the display image in units of pixel blocks of a predetermined size, 4. The image display system according to claim 1, wherein the image size restoration unit restores the size of the displayed image by enlarging the reduced pixel blocks and connecting the pixel blocks together.

5. An image generating unit that generates an image for a left eye and an image for a right eye, and generates the display images arranged in the left and right regions by applying a distortion corresponding to the eyepiece of the head mounted display, The image display system according to any one of claims 1 to 4, characterized in that the reduction processing unit sets the different magnifications to at least a first region including the center of each of the image for the left eye and the image for the right eye, a second region surrounding the first region, and a third region surrounding the second region as the multiple regions.

6. The image display system described in claim 5, characterized in that the reduction processing unit defines the multiple regions as regions formed by dividing the image plane with multiple concentric circles centered on the centers of the image for the left eye and the image for the right eye, and changes the radius of the multiple concentric circles depending on the information acquired by the status information acquisition unit.

7. 7. The image display system according to claim 5, wherein the reduction processing unit excludes an area of ​​the displayed image outside the field of view of the eyepiece from the area to be transmitted.

8. A magnification map storage unit stores a plurality of magnification maps each representing a distribution of magnifications to be set on the image plane, the reduction processing unit reduces the display image by switching and referring to the magnification map in accordance with the information acquired by the state information acquisition unit; The output unit transmits information related to the used magnification map to the head mounted display, 8. The image display system according to claim 1, wherein the image size restoration unit restores the size of the display image by referring to the used magnification map.

9. 9. The image display system according to claim 1, wherein the reduction processing unit reduces an average value of the degree of reduction in a lower half area of ​​the image plane to be smaller than that in an upper half area.

10. The output unit compresses and encodes the display image in units of blocks, 10. The image display system according to claim 1, wherein the reduction processing unit varies a magnification for each of the blocks.

11. An image generating device, generating data of a display image to be displayed on the head mounted display; acquiring at least one of information regarding a communication status with the head mounted display and information regarding a movement of the head mounted display; a step of dividing the image plane of the display image into a plurality of regions by changing the area according to the information acquired in the acquiring step, and reducing the number of pixels by reducing each of the regions at a different magnification; compressing and encoding the display image data with the reduced number of pixels and transmitting the compressed image data to a head mounted display; The head mounted display, A step of restoring the transmitted data to a size of a display image and displaying the data; Including, An image display method characterized in that the reduction step changes the area so that the amount of reduction in the number of pixels increases as the need for data size reduction based on the information acquired in the acquisition step increases.

12. A function for generating data for a display image to be displayed on a head mounted display; A function of acquiring at least one of information regarding a communication status with the head mounted display and information regarding a movement of the head mounted display; a function of dividing an image plane of the display image into a plurality of regions by changing an area according to the information acquired by the acquiring function, and reducing the number of pixels by reducing each of the regions at a different magnification; a function of compressing and encoding the display image data with the reduced number of pixels and transmitting the compressed image data to a head mounted display; This is realized on a computer. A computer program characterized in that the reducing function changes the area so that the greater the need to reduce data size based on the information acquired by the acquiring function, the greater the amount of reduction in the number of pixels.

Images