Image display system and image display method

Abstract

To allow a viewer to visually recognize an image matching the display quality of an image originally possessed by contents and a display device.SOLUTION: An image generation device causes a head-mounted display to display an image 310b in which the luminance is increased in comparison to an original image 310a by increasing an adjustment amount of the luminance distribution up to a target value B at the time t0 at which a light amount entering eyes of a user changes to such a degree that influences the action of visual cells. The image generation device gradually decreases the adjustment amount of the luminance distribution in a restoration period Δt such that an image 310c having the original luminance distribution is displayed at the subsequent time t1.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present invention relates to an image display system and an image display method for displaying an image of content.

Background Art

[0002] Conventionally, various technologies have been developed to improve image quality in video displays such as television broadcasts and distribution videos. In recent years, in addition to technologies for improving resolution and color gamut, technologies for processing HDR (High Dynamic Range) signals with an expanded luminance range are becoming widespread. Compared with conventional SDR (Standard Dynamic Range), HDR has a luminance tolerance range about 100 times wider, so that objects that are dazzling in the real world, such as reflected sunlight, can be represented more realistically in an image. Not only in television broadcasts and distribution videos, but also in the world of computer graphics such as game images, expressing in HDR gives a sense of reality to the virtual world.

[0003] On the other hand, a system that can detect the movement of the head of a user wearing a head-mounted display and represent a space of a display target in a corresponding field of view, thereby expressing a realistic image world, is widespread. Also, a walk-through system has been developed in which a user wearing a head-mounted display can virtually walk around in a space displayed as a video by physically moving. Technologies have also been proposed that utilize the dark adaptation of a person wearing a head-mounted display and reduce the luminance of the display unit to suppress the feeling of glare or extend the overall life of the device (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As described above, when the luminance range that can be correspondingly processed is expanded, it is possible to realistically represent high-luminance light such as strong reflected light, and in principle, it is possible to achieve high-definition representation with fine gradations even in low-luminance and medium-luminance regions. However, in the original image data, even if a large number of gradations are assigned to important objects on the content to produce a sense of resolution, due to various factors such as data conversion before display and human visibility, it may not look very high-definition or may even become difficult to view.

[0006] The present invention has been made in view of such problems, and its object is to provide a technique for allowing a viewer to visually recognize an image that matches the display quality of an image originally possessed by content or a display device.

Means for Solving the Problems

[0007] One aspect of the present invention relates to an image display system. This image display system includes a luminance distribution control unit that detects a change in the amount of light incident on the user's eyes that reduces visibility in the function of photoreceptor cells based on predetermined information and controls adjustment of the luminance distribution of an image according to a rule corresponding to the change in the amount of light, a luminance conversion unit that converts the luminance represented by an image value according to the rule, and an output unit that outputs data of the image whose luminance has been converted.

[0008] Another aspect of the present invention relates to an image display method. This image display method includes a step of detecting a change in the amount of light incident on the user's eyes that reduces visibility in the function of photoreceptor cells based on predetermined information, a step of controlling adjustment of the luminance distribution of an image according to a rule corresponding to the change in the amount of light, a step of converting the luminance represented by an image value according to the rule, and a step of outputting data of the image whose luminance has been converted.

[0009] In addition, any combination of the above components, as well as those obtained by converting the expression of the present invention among a method, an apparatus, a system, a computer program, a data structure, a recording medium, etc., are also effective as aspects of the present invention.

Advantages of the Invention

[0010] According to the present invention, it is possible to allow a viewer to visually recognize an image that matches the display quality of an image originally possessed by content or a display device.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0012] FIG. 1 shows an external appearance example of the head-mounted display 100. In this example, the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that wraps around the head when worn by the user to achieve fixation of the device. The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and a display panel is provided inside so as to face the eyes during wearing.

[0013] Further inside the housing 108, an eyepiece lens is provided that is located between the display panel and the user's eyes when the head-mounted display 100 is worn and expands the user's viewing angle. Also, the head-mounted display 100 may further include speakers or earphones at positions corresponding to the user's ears when worn. Also, the head-mounted display 100 incorporates a motion sensor to detect the translational and rotational movements of the head of the user wearing the head-mounted display 100, and thus the position and posture at each moment.

[0014] In this example, the head-mounted display 100 includes a stereo camera 110 on the front surface of the housing 108, and captures a moving image of the surrounding real space in a field of view corresponding to the user's line of sight. If the captured image is immediately displayed, so-called video see-through can be realized, where the user can directly see the state of the real space in the direction they are facing. Furthermore, if a virtual object is drawn on the image of a real object shown in the captured image, AR (augmented reality) can be realized.

[0015] FIG. 2 shows a configuration example of the image display system according to the present embodiment. The head-mounted display 100 is connected to the image generation device 200 by an interface for connecting a peripheral device such as wireless communication or USB. The image generation device 200 may be further connected to a server via a network. In that case, the server may provide the image generation device 200 with an online application such as a game in which a plurality of users can participate via the network.

[0016] The image generation device 200 specifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, and generates a display image so as to have a corresponding field of view and outputs it to the head-mounted display 100. The purpose of displaying the image may be various within this limit. For example, the image generation device 200 may generate a virtual world, which is the stage of an electronic game, as a display image while advancing the electronic game, or may display a still image or a moving image for viewing or information providing regardless of whether the virtual world is the real world. If a panoramic image is displayed with a wide angle of view centered on the user's viewpoint, a feeling of immersion in the displayed world can be given.

[0017] Note that part or all of the functions of the image generation device 200 may be provided inside the head-mounted display 100. When all the functions of the image generation device 200 are incorporated into the head-mounted display 100, the illustrated image processing system is realized by one head-mounted display 100.

[0018] Figure 3 illustrates an overview of the flow of image display processing performed by the image display system according to the present embodiment. First, the image generation device 200 draws an image in accordance with the program of the content to be processed (S10). Typically, the image generation device 200 generates a moving image by continuously drawing images at a predetermined rate. In this case, the image generation device 200 sequentially generates an image corresponding to the user's line of sight by determining the field of view of the image based on the latest position and orientation information of the head-mounted display 100, and thus the user's head, acquired at that time.

[0019] The process of S10 is determined for each content, and in the figure, it is represented by individual rectangles for the content such as "Game A", "Game B", "Game C",... By the process of S10, RGB luminance is given as a pixel value to each pixel constituting the image plane. Here, the image generation device 200 may perform conversion processing, such as tone mapping, to bring the pixel values determined by detailed calculations such as ray tracing within a displayable luminance range.

[0020] Then, the image generation device 200 performs correction processing necessary for display on the drawn image (S12). When the head-mounted display 100 is the display destination, the image generation device 200 performs distortion correction and reprojection in S12. Distortion correction is a correction in which the original image is distorted in the reverse direction according to the aberration of distortion so that an undistorted image is visually recognized when viewed through the eyepiece lens provided in the head-mounted display 100. Reprojection is a process of correcting the field of view of the image to the latest state in consideration of the movement of the head over time from the start of drawing in S10.

[0021] The image generation device 200 also performs, as necessary, color gamut conversion and the like according to the display characteristics of the head-mounted display 100. The head-mounted display 100 sequentially receives the data of the image corrected in this way, and generates a driving voltage corresponding to the RGB luminance represented by the pixel value (S14). Note that general processes such as quantization and inverse quantization of pixel values may be interposed in the transmission of image data from the image generation device 200 to the head-mounted display 100. Then, the head-mounted display 100 sequentially displays the image by driving the display panel with the generated driving voltage (S16).

[0022] FIG. 4 illustrates a conversion function (hereinafter referred to as a “tone curve”) when performing tone mapping in S10 of FIG. 3. This figure shows a representative tone curve, the Reinhard function, and realizes a conversion that suppresses gradation more as it goes from the low luminance region to the high luminance side in consideration of human visual characteristics. By appropriately setting and selecting the tone curve, even if the characteristics of the display at the display destination and the corresponding luminance range change, the image can be displayed in the same color tone intended by the content creator.

[0023] Here, the maximum value 1.0 of the tone curve corresponds to the upper limit of the luminance that the head-mounted display 100 can display. In the case of HDR, the maximum emission luminance is 1000 (cd / m 2 ) and so on. Also, according to the tone curve as shown in the figure, for example, in the low luminance and medium luminance regions smaller than luminance A, more gradations are assigned. If the displayable luminance range increases, in those regions in particular, an image expression with a sense of resolution can be achieved by fine color tone changes.

[0024] On the one hand, when the head-mounted display 100 has a structure that shields external light, for the user immediately after wearing, due to the sudden decrease in the amount of light entering the eyes, a switch in the function of photoreceptor cells from photopic vision to mesopic vision or scotopic vision occurs. FIG. 5 shows the relationship between the luminance range of light stimuli and the activity range of photoreceptor cells (see, for example, "Chapter 16: Human eye sensitivity and photometric quantities", [online], [searched on June 19, 2021], Internet <URL: https: / / www.ecse.rpi.edu / ~schubert / Light-Emitting-Diodes-dot-org / Sample-Chapter.pdf>).

[0025] As shown in the figure, at light stimuli of about 10 -2 (cd / m 2 ) or less, only rod cells are active, and at light stimuli of about 10 2 (cd / m 2 ) or more, only cone cells are active. In the intermediate luminance range, both rod cells and cone cells are active. Cone cells have high sensitivity to colors and the appearance of objects, and rod cells have high sensitivity to the brightness of light. Generally, the vision when mainly only rod cells are active is called scotopic vision, the vision when only cone cells are active is called photopic vision, and the vision when both are active is called mesopic vision. When transitioning from photopic vision to mesopic vision or scotopic vision, in vision, as an autonomous function of the retina with respect to the intensity of light, dark adaptation occurs where visibility once drops and then the eyes gradually get used to it and visibility increases.

[0026] FIG. 6 illustrates the change in the sensitivity of photoreceptor cells during dark adaptation. The figure shows the change in the minimum luminance of light that can be visually recognized with respect to the horizontal time axis. The smaller the minimum recognizable luminance, the higher the sensitivity. Starting from a state where the sensitivity is saturated as photopic vision, the time when the amount of light entering the eyes suddenly decreases, such as when the user wears the head-mounted display 100, is set to 0. At this time, first, the cone cells increase their sensitivity and saturate as shown by curve C, and then, with a delay, the rod cells increase their sensitivity and saturate as shown by curve R.

[0027] The time until the sensitivity of each cell saturates also depends on the difference in light quantity. As an example, the adaptation time of cone cells is about several tens of seconds to several minutes, and the adaptation time of rod cells is about 30 minutes to 1 hour. Note that after dark adaptation, if the light quantity entering the eyes suddenly increases, such as when the user removes the head-mounted display 100, a switch to photopic vision occurs. At this time, light adaptation occurs to reduce glare by lowering the sensitivity of photoreceptor cells. In the figure, light adaptation has occurred from 35 minutes. Light adaptation is achieved in a shorter period compared to dark adaptation.

[0028] Due to such changes in the function of photoreceptor cells with respect to changes in the light quantity entering the eyes, in a situation where adaptation is not sufficient, the visibility of the displayed image by the user temporarily decreases. In particular, the brighter the surrounding environment the user was looking at before wearing the head-mounted display 100, and the darker the displayed image, the greater the light quantity difference and the easier it is for the visibility to deteriorate. In recent years, technologies such as VSLAM (Visual Simultaneous Localization and Mapping) have been put into practical use to analyze the captured images by the stereo camera 110 and identify the movement of the user.

[0029] In this case, in order to obtain sufficient analysis accuracy, it is necessary to take pictures in an environment that is bright to a certain extent. Therefore, the difference in light quantity due to the attachment and detachment of the stereo camera 110 is likely to be large. As a result, it is conceivable that the entire displayed image appears darker to the user immediately after wearing the head-mounted display 100 than it actually is. In addition, the user may not be able to distinguish the change in the color tone of the low-luminance part, which is originally expressed in fine gradations, and may feel it is difficult to see or may have the impression that the image quality is not good, which may lead to an evaluation of the entire content.

[0030] The same phenomenon can occur not only immediately after wearing the head-mounted display 100, but also immediately after switching from a bright screen to a dark screen. Therefore, in the present embodiment, the luminance distribution of the display image is adjusted on the occasion that a difference sufficient to change the function of photoreceptor cells occurs in the amount of light incident on the user's eyes. Here, the "luminance distribution" refers to the distribution formed by the set of RGB luminances of the pixels constituting the image, and can also be regarded as a luminance histogram. Focusing on each pixel, the "adjustment of luminance distribution" is a conversion of luminance values according to a predetermined rule.

[0031] FIG. 7 is a diagram for explaining the adjustment effect of the luminance distribution in dark adaptation. The horizontal direction in the figure is the time axis, and similar to FIG. 6, the time when the amount of light suddenly decreases is set to 0. It is assumed that the luminance of the image as the content does not change. (a) schematically shows the transition of the image visually recognized by the user when the luminance is not adjusted. In this case, the visually recognized image becomes darker at time 0 when the amount of light decreases, and the visibility is gradually improved as the adaptation progresses.

[0032] (b) schematically shows the transition of the display image whose luminance distribution is adjusted according to the present embodiment. That is, at time 0 when the amount of light decreases, the image display system increases the luminance so as to cancel out the decrease in visibility as shown in (a). Actually, the luminance of the high-luminance region that is not easily affected by the deterioration of visibility may remain the same, and the luminance of the low-luminance and medium-luminance regions where poor visibility is expected may be increased, and the increase rate may be changed according to the original luminance. Then, as the visibility is restored by dark adaptation, the increased luminance distribution is gradually returned to the original state.

[0033] (c) schematically shows the transition of the image visually recognized by the user by adjusting the luminance distribution of the display image as in (b). By adjusting the luminance distribution of the image so as to cancel out the change in visibility, even if the amount of light incident on the eye changes, the luminance and gradation of the visually recognized image are maintained to some extent, and it is possible to continuously recognize a stable and high-quality image. When the timing at which the user wears the head-mounted display 100 is set as time 0, of course, the leftmost images in (a) and (c) are not recognized, but are shown as images that should be visually recognized in a state where the sensitivity of the photoreceptor cells is saturated.

[0034] FIG. 8 shows the internal circuit configuration of the image generation device 200. The image generation device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and a main memory 226. These components are interconnected via a bus 230. An input / output interface 228 is further connected to the bus 230.

[0035] Connected to the input / output interface 228 are a communication unit 232 composed of a peripheral device interface such as USB or IEEE1394 and a network interface for wired or wireless LAN, a storage unit 234 such as a hard disk drive or a non-volatile memory, an output unit 236 that outputs data to the head-mounted display 100, an input unit 238 that inputs data from the head-mounted display 100, and a recording medium drive unit 240 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0036] The CPU 222 controls the entire image generation apparatus 200 by executing the operating system stored in the storage unit 234. The CPU 222 also executes various programs read from a removable recording medium and loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has the functions of a geometry engine and a rendering processor, performs rendering processing according to a rendering command from the CPU 222, and outputs the result to the output unit 236. The main memory 226 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing.

[0037] FIG. 9 illustrates the internal configuration of the head-mounted display 100. The control unit 150 is a main processor that processes and outputs signals such as image signals and sensor signals, as well as commands and data. The stereo camera 110 supplies data of a captured image to the control unit 150 at a predetermined rate. The display panel 152 is composed of a light-emitting panel such as a liquid crystal or an organic EL and its control mechanism, and receives and displays an image signal from the control unit 150.

[0038] The communication control unit 154 transmits data input from the control unit 150 to the outside by wired or wireless communication via a network adapter or antenna (not shown). The communication control unit 154 also receives data from the outside by wired or wireless communication via a network adapter or antenna and outputs it to the control unit 150. The storage unit 160 temporarily stores data, parameters, operation signals, etc. processed by the control unit 150.

[0039] Sensor 162 includes a motion sensor, measures attitude information such as the rotation angle and inclination of the head-mounted display 100, and sequentially supplies it to the control unit 150. Sensor 162 may also include a contact sensor that detects when the user wears the head-mounted display 100, and an infrared camera that captures the state of the pupil of the user looking at the display panel as needed. Also in this case, sensor 162 sequentially supplies the acquired various information to the control unit 150. For example, the infrared camera may be part of a gaze point detector that detects the user's gaze point with respect to the displayed image.

[0040] The external input / output terminal interface 164 is an interface for connecting peripheral devices such as a USB (Universal Serial Bus) controller. The external memory 166 is an external memory such as a flash memory. The control unit 150 can supply image and audio data to the display panel 152 and an earphone or speaker (not shown) for output, or supply it to the communication control unit 154 for transmission to the outside.

[0041] FIG. 10 shows the configuration of the functional blocks of the image generation device 200 and the head-mounted display 100 in the present embodiment. As described above, the image generation device 200 may perform general information processing such as advancing an electronic game or communicating with a server, but in FIG. 10, particular attention is paid to the function of generating a display image. Note that at least a part of the functions of the image generation device 200 shown in FIG. 10 may be implemented in the head-mounted display 100. Alternatively, at least a part of the functions of the image generation device 200 may be implemented in a server connected to the image generation device 200 via a network.

[0042] Also, the functional blocks shown in FIG. 10 can be realized in terms of hardware by the configurations such as the CPU, GPU, control unit, various memories, sensors, etc. shown in FIG. 8 or FIG. 9, and in terms of software, they can be realized by a program that exhibits various functions such as a data input function, a data holding function, an image processing function, a communication function, etc. loaded from a recording medium or the like into a memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0043] The image generation device 200 includes an input data acquisition unit 260 that acquires data transmitted from the head-mounted display 100, an image drawing unit 262 that draws an image of the content, a content data storage unit 264 that stores content data, a luminance conversion unit 266 that converts the luminance given as a pixel value, a luminance distribution control unit 274 that controls the adjustment of the luminance distribution corresponding to the visual adaptation of a person, a luminance adjustment rule storage unit 268 that stores the adjustment rule of the luminance distribution, a correction unit 270 that performs correction for display, and an output unit 272 that outputs the data of the corrected image to the head-mounted display 100.

[0044] The input data acquisition unit 260 acquires, at a predetermined rate, data such as the measured values of the motion sensor and the images captured by the stereo camera 110 transmitted from the head-mounted display 100. The input data acquisition unit 260 also sequentially acquires from the head-mounted display 100 predetermined information necessary for luminance distribution control corresponding to visual adaptation. Here, the "predetermined information" may be any data that can detect the occurrence of a difference in the amount of light entering the user's eyes (hereinafter referred to as "light-dark conversion") to such an extent that it causes a decrease in visibility.

[0045] For example, the input data acquisition unit 260 acquires information indicating that the head-mounted display 100 has been attached or detached, which is acquired by the contact sensor. The input data acquisition unit 260 may acquire an image of the user's pupil captured by the infrared camera or pupil state information obtained therefrom. In addition, the input data acquisition unit 260 may acquire the content of a user operation related to a change in the display image, such as an operation on the content being displayed or a call to the system screen, from an input device (not shown).

[0046] The image drawing unit 262 acquires the position and orientation of the head-mounted display 100 at a predetermined rate and draws an image of the display target in the corresponding field of view. This process corresponds to S10 in FIG. 3. Here, the image drawing unit 262 determines pixel values composed of RGB elements by performing ray tracing, rasterization, etc. in accordance with the program of the content being executed. The content data storage unit 264 stores data necessary for drawing an image, such as object model data and data related to the configuration of the display world.

[0047] Note that the process performed by the image drawing unit 262 may be image generation. In addition to image drawing by computer graphics, processes such as decoding and expansion of actual video data and synthesis of actual video and computer graphics may also be performed. In any case, the content data storage unit 264 may store metadata indicating the timing at which the luminance of the entire display image changes, such as the timing of switching of the display scene, to a predetermined value or more.

[0048] The luminance conversion unit 266 converts the luminance range of the pixel values obtained as a result of drawing to a range suitable for the display panel of the head-mounted display 100. For example, the luminance conversion unit 266 internally holds a tone curve as shown in FIG. 4 and performs tone mapping using it. The luminance conversion unit 266 also adjusts the luminance distribution under the control of the luminance distribution control unit 274 when a light-dark conversion is detected. That is, the luminance conversion unit 266 converts each luminance of RGB, which is a pixel value, according to the rule determined by the luminance distribution control unit 274.

[0049] The luminance distribution control unit 274 detects or predicts the light and dark conversion based on the information obtained from the input data acquisition unit 260 or the like, and controls the luminance conversion unit 266 so that the luminance distribution of the pixel values changes according to the corresponding rule. The luminance distribution control unit 274 detects the timing of the light and dark conversion by, for example, at least any one of the following.

[0050] 1. Obtain from the detection result by the contact sensor that the user has worn the head-mounted display 100 2. Obtain from the change in the size of the user's pupil photographed by the infrared camera that the overall luminance (average luminance) of the displayed image has decreased 3. Obtain from the drawing result by the image drawing unit 262 or the metadata of the content that the overall luminance (average luminance) of the displayed image has decreased 4. Obtain from the drawing result by the image drawing unit 262 or the content of the user operation and metadata or the like the decrease in the screen luminance due to the switching of the content to be displayed or the switching between the content screen and the system screen

[0051] The luminance distribution control unit 274 may also predict the timing of the light and dark conversion by at least any one of the following. 1. Detect the activation of the head-mounted display 100 and predict that the user will wear the head-mounted display 100 2. Predict the timing when the overall luminance of the displayed image decreases from the metadata of the content

[0052] The luminance distribution control unit 274 not only detects the timing of the light and dark conversion as described above, but also obtains the difference in the amount of light in the light and dark conversion and reflects it in the degree of adjustment of the luminance distribution. Qualitatively, the luminance distribution control unit 274 increases the adjustment amount of the luminance distribution as the difference in the amount of light is larger. For example, the luminance distribution control unit 274 obtains the image photographed by the stereo camera 110 of the head-mounted display 100, and obtains the brightness of the environment where the user is based on its average luminance, luminance histogram, etc.

[0053] The brighter the environment where the user is, the greater the drop in the amount of light when the head-mounted display 100 is worn, so the visibility deteriorates more. Therefore, the luminance distribution control unit 274 increases the amount of increase in luminance when the head-mounted display 100 is worn as the surrounding environment becomes brighter. The difference in the amount of light may be determined based on, among other things, the difference in the size of the user's pupils, the rate of change, the average luminance of the display image, and the difference in the luminance histogram.

[0054] The "difference in the amount of light" acquired by the luminance distribution control unit 274 here does not need to be a precise physical quantity. That is, depending on the information used as the basis for the difference in the amount of light, such as the surrounding brightness, the degree may be defined in multiple levels or the like. At this time, the luminance adjustment rule storage unit 268 stores the levels of the difference in the amount of light and the adjustment rules for the luminance distribution in association with each other. The luminance distribution control unit 274 also controls the restoration process of the luminance distribution corresponding to visual adaptation. For example, if the difference in the amount of light during light and dark conversion is large, the time until the visibility is sufficiently improved by dark adaptation becomes longer. Therefore, the luminance distribution control unit 274 may slow down the restoration speed of the luminance distribution and make it restored over a long time as the difference in the amount of light during light and dark conversion is larger.

[0055] The luminance distribution control unit 274 may change either one or both of the adjustment amount of the luminance distribution and the restoration time based on the difference in the amount of light. Alternatively, both may be fixed values. The luminance distribution control unit 274 may further determine whether to restart the adjustment of the luminance distribution from the beginning or resume it from the middle according to the time from when the user removes the head-mounted display 100 until wearing it again, the surrounding brightness, and the like.

[0056] The correction unit 270 performs corrections necessary for display, such as distortion correction, reprojection, and color gamut correction, on the drawn image or the image subjected to luminance conversion. This process corresponds to S12 in FIG. 3. When the display image is to be viewed stereoscopically, the correction unit 270 may generate left-eye and right-eye images with parallax from the original reference image. Alternatively, the image drawing unit 262 may directly generate the left-eye and right-eye images. The output unit 272 sequentially sends the corrected image data supplied from the correction unit 270 to the head-mounted display 100. When viewing stereoscopically, the output unit 272 sends the data in such an order that the left-eye image is arranged in the left half of the image and the right-eye image is arranged in the right half of the image.

[0057] The head-mounted display 100 includes an output data transmission unit 284 that transmits data used for drawing the display image and data necessary for luminance distribution control to the image generation device 200, an image data acquisition unit 280 that acquires the image data transmitted from the image generation device 200, a light emission luminance control unit 282 that controls the light emission luminance of the display panel based on the image data, and a display unit 286 that drives the display panel to display an image.

[0058] The output data transmission unit 284 transmits data necessary for drawing the display image, such as the image captured by the stereo camera 110 and the measurement values of the motion sensors included in the sensor 162, to the image generation device 200 at a predetermined rate. The output data transmission unit 284 also sequentially transmits data necessary for luminance distribution control corresponding to visual adaptation to the image generation device 200. This data is, as described above, predetermined among the results of attachment / detachment detection of the head-mounted display 100, information representing the state of the pupil, the image of the surrounding environment, and the like.

[0059] The image data acquisition unit 280 acquires the data of the image transmitted from the image generation device 200. At this time, the image data acquisition unit 280 sequentially acquires the data of the pixel values sent by the image generation device 200 in raster order or the like, and supplies it to the emission luminance control unit 282. The emission luminance control unit 282 determines the drive voltage of the light-emitting element in the display panel based on the RGB luminance represented by each pixel value. This process corresponds to S14 in FIG. 3. The display unit 286 displays an image by sequentially causing the corresponding elements of the display panel to emit light with the drive voltage generated by the emission luminance control unit 282. This process corresponds to S16 in FIG. 3.

[0060] FIG. 11 illustrates the luminance conversion rule used by the luminance conversion unit 266 for luminance conversion under the control of the luminance distribution control unit 274. (a) and (b) show the luminance distribution without performing the adjustment related to visual adaptation by a dotted line, and show the change in the luminance distribution when adjusting by a thick line. Note that the maximum luminance before and after adjustment is normalized to 1.0. (a) is a conversion rule in which luminance below a predetermined luminance P1 where 0 < P1 < 1.0 is linearly increased by multiplying by a coefficient k = 1.0 / P1, and all luminance higher than P1 is set to the maximum luminance.

[0061] (b) is a conversion rule in which the luminance is increased by a curve with a maximum increase rate at a predetermined luminance P2 where 0 < P2 < 1.0. The function giving the curve is not limited, and it may be a power function such as a gamma curve or a tone curve as shown in FIG. 4. In any case, in the illustrated example, by surely increasing the luminance in the low luminance region and the middle luminance region, gradation can be preferentially assigned to the region that becomes difficult to see due to the light amount decrease at the light and dark transition. The predetermined luminance P1 and P2 may be determined according to important luminance regions and differences in light amount in the content.

[0062] In this embodiment, the "adjustment amount" of the luminance distribution is strictly a variable that depends on the luminance before adjustment. Conceptually, however, it may be the difference from the state without adjustment indicated by the dotted line, and various definitions are acceptable. For example, the "adjustment amount" may be the area of regions 300a and 300b surrounded by the thick line representing the conversion rule during adjustment and the dotted line when not adjusting (converting), that is, the integral value of the change amount of each luminance. Alternatively, the "adjustment amount" may be a coefficient k multiplied by the luminance before adjustment, the increase amount (maximum increase amount) after adjustment of predetermined luminances P1 and P2, etc. Also, the conversion rule shown in the figure is just an example, and it may be a discontinuous function depending on the luminance range, or it may be represented as a look-up table associating the luminance before and after conversion.

[0063] FIG. 12 schematically shows the state of temporal control of luminance distribution adjustment realized by the luminance distribution control unit 274. The upper part of the figure shows the temporal change of the adjustment amount of the luminance distribution with respect to the horizontal time axis. At time t0, when the conversion from light to dark occurs due to the decrease in light amount, the luminance distribution control unit 274 raises the adjustment amount of the luminance distribution from 0 to the target value B. Then, at time t1 after the restoration period Δt, the luminance distribution control unit 274 decreases the adjustment amount so that the state without adjustment, that is, the adjustment amount reaches 0.

[0064] In this embodiment, it is important to ensure that the color representation expressed by the RGB luminance is not impaired by the light and dark conversion. Therefore, the restoration period Δt is generally set to be the time until the sensitivity of color vision increases and saturates by cone cells during human visual adaptation. That is, Δt is about several tens of seconds, and as an example, Δt = 50 seconds. However, as described above, the restoration time Δt and the target value B of the adjustment amount may be changed depending on the difference in light amount in the light and dark conversion.

[0065] Also, the decreasing rate of the adjustment amount may be gradually increased as shown in the figure or may be fixed. According to such control, as shown in the lower part of the figure, before time t0, the original image 310a is displayed, and at time t1, it switches to the image 310a with increased luminance. Finally, by gradually decreasing the luminance, an image 310c equivalent to the original image 310a is reached, realizing the transition of the displayed image. As a result, as shown in (c) of FIG. 7, even when the light and dark conversion occurs, a stable and high-quality image can continue to be recognized.

[0066] In addition, when the light and dark conversion at time t0 is caused by wearing the head-mounted display 100, the images in the period before that cannot be visually recognized, so the luminance can be increased in advance. Also, when the user removes or reinstalls the head-mounted display 100 during the restoration period Δt, the luminance distribution control unit 274 may determine whether to re-adjust the luminance distribution according to that timing.

[0067] For example, if the head-mounted display 100 that has been removed is reinstalled within a predetermined time that can be regarded as a short time such as 5 seconds, the luminance distribution control unit 274 resumes the decrease from the adjustment amount at the time of removal. Alternatively, the decrease in the adjustment amount of the luminance distribution is continued as if there was no removal. This can avoid unnecessarily increasing the adjustment amount to the target value B even when the sensitivity of the cone cells has increased. On the other hand, in situations where it can be considered that the improvement in sensitivity has not progressed, such as when it has been removed for a longer time or when it is removed immediately after the head-mounted display 100 is first worn, the adjustment of the luminance distribution may be restarted from the beginning.

[0068] When the luminance distribution control unit 274 starts adjusting the luminance distribution upon the head-mounted display 100 being worn, it may continue to adjust the restoration period Δt regardless of other changes. For example, even if there is a transition from light to dark in the displayed image during the restoration period Δt, such as a change in the scene or content to be displayed, or a switch between the system screen and the content screen, the adjustment of the luminance distribution due to the wearing of the head-mounted display 100 is prioritized. This can avoid the unnatural increase in the luminance distribution when visual adaptation is progressing.

[0069] FIG. 13 illustrates the data structure of the luminance adjustment rule stored in the luminance adjustment rule storage unit 263 of the image generation device 200. In this example, the luminance adjustment rule table 320 associates the target value B of the adjustment amount of the luminance distribution, the restoration period Δt, and the conversion function of the luminance distribution with respect to the difference in the amount of light in the light-dark transition. Here, as described above, the difference in the amount of light is derived from the ambient brightness, the change in the pupil size, the change in the average luminance or the luminance histogram of the displayed image, etc., and in the example of the figure, three levels of "small", "medium", and "large" are set.

[0070] In this example, when the difference in the amount of light is "small", the setting is not to adjust the luminance distribution. That is, the target value B and the restoration period Δt are "0", and the conversion function is not set. When the difference in the amount of light is "medium", the target value B is 10%, the restoration period Δt is 30 seconds, and the conversion function is "F1". When the difference in the amount of light is "large", the target value B is 30%, the restoration period Δt is 50 seconds, and the conversion function is "F2". Here, the target value B indicates the maximum increase ratio from the original luminance as a percentage, but the unit varies depending on the definition of the adjustment amount. The conversion function is information for identifying the luminance conversion rule as illustrated in FIG. 11, and the final shape is determined by the combination with the target value B.

[0071] As shown in the figure, by restoring with a larger adjustment amount over time as the light amount difference in light and dark conversion is larger, it is possible to achieve an adjustment corresponding to visual adaptation. Also, when the head-mounted display 100 is worn in a dark environment or the like, if the light amount difference is small enough that no visual change occurs, the waste of processing can be saved by omitting the adjustment of the luminance distribution itself. In the example of the figure, the target value B of the adjustment amount, the restoration period Δt, and the conversion function are all set to depend on the light amount difference. However, the present embodiment is not limited thereto, and only some of the parameters may be changed. Also, the number of steps of the light amount difference is not limited.

[0072] Figure 14 shows variations in the timing of performing the adjustment process of the luminance distribution by light and dark conversion in the image display system. Here, as shown in FIG. 3, the display process of the image is roughly classified into four stages: image drawing (S10), correction process (S12), drive voltage generation (S14), and display (S16). Also, the adjustment process of the luminance distribution by light and dark conversion is named "VBA (Visual Brightness Adaptation)" and shown in individual blocks. (a) is the mode described so far, and the luminance conversion unit 266 adjusts the luminance distribution under the control of the luminance distribution control unit 274 before the correction by the correction unit 270.

[0073] The luminance conversion unit 266 originally has a function of adjusting the luminance range with a tone curve suitable for the content according to the corresponding luminance range of the head-mounted display 100 and the like. By having the luminance conversion unit 266 undertake part of the VBA function, the content creator can set the period during which VBA is effective and the adjustment rules as shown in FIG. 13 according to the content. On the other hand, there is a concern that the consistency of the adjustment mode may be lost in the platform of the entire image display system. Also, it is considered that it will be difficult to handle the case where it is desired to disable VBA due to control outside the content such as recording and distribution.

[0074] (b) is a mode in which the correction unit 270 of the image generation device 200 adjusts the luminance distribution under the control of the luminance distribution control unit 274. Since the correction process by the correction unit 270 is common to the content, according to this mode, the consistency of the adjustment mode is ensured in the platform of the entire image display system. Also, when the conversion function for adjustment is a power function, it becomes possible to perform simultaneous adjustment with the de-gamma / gamma process originally performed by the correction unit 270, that is, gamma correction, etc., without increasing the processing load. When realizing VBA with a gamma curve, the gamma value may be gradually brought closer to the original value during the restoration period.

[0075] However, in the mode of (b), it is necessary to incorporate the logic for enabling VBA into all functional blocks that output image data (video signals), for example, the functional blocks that generate images provided by systems outside the content. Also, due to recording and distribution, the adjustment by VBA will be reflected in the images viewed by users who do not wear the head-mounted display 100.

[0076] (c) is a mode in which the emission luminance control unit 282 of the head-mounted display 100 adjusts the luminance distribution. In this case, the luminance distribution control unit 274 may also be incorporated into the emission luminance control unit 282. Since the drive voltage generation process by the emission luminance control unit 282 is also common to the content, also by this mode, the consistency of the adjustment mode is ensured in the platform of the entire image display system. Also, for the adjustment of the final display image, it is not necessary to incorporate the logic for enabling VBA into other functional blocks. Furthermore, due to the processing inside the head-mounted display 100, it does not affect the recorded images and distribution images.

[0077] Furthermore, when the conversion function for adjustment is a power function, it becomes possible to perform simultaneous adjustment with gamma correction according to the characteristics of the display panel, etc. in the drive voltage generation process, without increasing the processing load. Thus, since the adjustment timing of the luminance distribution in the light and dark conversion is not limited, an appropriate implementation form is selected in view of the advantages and disadvantages as described above, the processing performance of each device, the required processing accuracy, etc.

[0078] In addition, when a head-mounted display 100 is provided with a user interface capable of adjusting the emission luminance separately from the luminance adjustment mechanism inside the content, it is desirable to disable the VBA function during the period when the operation is effective so that the control does not become complicated. Alternatively, by enabling both simultaneously, it may be possible to achieve both control according to visual adaptation and operation based on the user's subjectivity.

[0079] According to the embodiment described above, in a system that displays an image of content on a head-mounted display, the timing at which the amount of light entering the user's eyes changes is detected, and when a light amount difference that affects the function of photoreceptor cells occurs, the luminance distribution of the image is adjusted. As a result, it is possible to compensate on the display side for the decrease in visibility caused by the difference in the amount of light, and it is possible to continue to visually recognize an image that matches the original quality and performance of the content and the head-mounted display.

[0080] For example, in response to the drop in the amount of light caused by wearing a head-mounted display, by increasing the luminance in the low-luminance region or the mid-luminance region, it is possible to maintain the resolution of color representation regardless of the decrease in visibility. Also, within about several tens of seconds when the sensitivity of cone cells saturates in human visual adaptation, by gradually returning the adjustment amount to its original state, it is possible to efficiently act on color vision. As a result, especially in the case of content and head-mounted displays corresponding to a wide luminance range such as HDR, it is possible to provide a rich video experience without degrading the originally high image quality.

[0081] As described above, the present invention has been described based on the embodiments. It is understood by those skilled in the art that the embodiments are examples, and various modifications are possible for each combination of their respective components and each processing process, and such modifications are also within the scope of the present invention.

[0082] For example, in this embodiment, mainly focusing on the switching from bright vision to low-light vision or dark vision due to the decrease in the amount of light, the brightness of the image was increased. This mode is considered to produce a greater effect because the dark adaptation of the human eye is longer than the light adaptation, and general images are generally composed of medium or lower brightness, and the visibility becomes significantly difficult during the switching to low-light vision or dark vision. On the other hand, if the brightness distribution is adjusted in the opposite direction to the change in the amount of light entering the user's eyes, the same effect as in this embodiment can be obtained not only when the amount of light decreases but also when the visibility deteriorates when the amount of light increases.

[0083] That is, the technique for adjusting the brightness distribution can be similarly applied even when switching from low-light vision or dark vision to bright vision when the amount of light increases. For example, when switching from an overall dark image to a scene with many regions represented by high brightness, or when switching the display to another content or system screen having such an image, once the brightness is reduced and then gradually returned to the original brightness. Except for adjusting the brightness in the decreasing direction, it is the same as in this embodiment. This can prevent glare and difficulty in viewing caused by strong light incident on the photoreceptor cells in a state where the sensitivity has increased in low-light vision or dark vision.

Description of Reference Numerals

[0084] 100 Head-Mounted Display, 110 Stereo Camera, 150 Control Unit, 152 Display Panel, 154 Communication Control Unit, 160 Storage Unit, 162 Sensor, 200 Image Generation Device, 222 CPU, 224 GPU, 226 Main Memory, 234 Storage Unit, 236 Output Unit, 260 Input Data Acquisition Unit, 262 Image Drawing Unit, 264 Content Data Storage Unit, 266 Luminance Conversion Unit, 268 Luminance Adjustment Rule Storage Unit, 270 Correction Unit, 272 Output Unit, 274 Luminance Distribution Control Unit, 280 Image Data Acquisition Unit, 282 Emission Luminance Control Unit, 284 Output Data Transmission Unit, 286 Display Unit.

Claims

An image display system for displaying an image on a head-mounted display having a structure for shielding external light, comprising: an image drawing unit that draws the image and determines the luminance of each pixel; a luminance distribution control unit that, when the wearing of the head-mounted display is detected, detects a decrease amount of the amount of light incident on the user's eyes due to shielding of external light based on predetermined information, and controls to increase the luminance of each pixel in the image by an adjustment amount depending on the luminance at the time of drawing according to a rule corresponding to the decrease amount of the amount of light; a luminance conversion unit that converts the luminance of each pixel according to the control; an output unit that outputs data of the image with the converted luminance; and characterized by comprising the above. The image display system according to claim 1, wherein the luminance conversion unit converts the luminance of each pixel such that the adjustment amount becomes maximum at a predetermined value where the luminance at the time of drawing is greater than 0 and less than the maximum value.

3. The image display system according to claim 1 or 2, wherein the luminance conversion unit converts the luminance of each pixel such that the change rate of the luminance depends on the luminance at the time of drawing.

4. The image display system according to any one of claims 1 to 3, wherein the luminance conversion unit converts the luminance using a gamma curve.

5. The image display system according to any one of claims 1 to 4, wherein the luminance distribution control unit controls to increase the luminance by an adjustment amount corresponding to the decrease amount of the amount of light.

6. The image display system according to any one of claims 1 to 5, wherein the luminance distribution control unit controls to reach the luminance at the time of drawing by decreasing the luminance of each pixel after increasing it by the adjustment amount.

7. The image display system according to claim 6, wherein the luminance distribution control unit reaches the luminance at the time of drawing over a longer time as the decrease amount of the amount of light is larger.

8. The image display system according to any one of claims 1 to 7, wherein the luminance distribution control unit further controls to increase the luminance of each pixel when the decrease amount of the amount of light from the image being displayed incident on the eyes of the user wearing the head-mounted display exceeds a predetermined range, and to decrease the luminance of each pixel when the increase amount of the amount of light exceeds a predetermined range.

9. The image display system according to any one of claims 1 to 8, wherein the luminance distribution control unit detects the amount of change in the light quantity based on a change in the size of the pupil shown in an image captured by an infrared camera included in the head-mounted display.

10. The image display system according to claim 8, wherein the luminance distribution control unit detects the amount of change in the light quantity by obtaining a change in the luminance of an image being displayed based on a rendering result of the image or metadata of content for which the image is being displayed.

11. The image display system according to claim 8, wherein the luminance distribution control unit identifies a switch between content to be displayed or a switch between a content screen and a system screen based on the content of a user operation, and obtains a change in the luminance of an image due to the switch based on metadata of the content, thereby detecting the amount of change in the light quantity.

12. An image display method for displaying an image on a head-mounted display having a structure that shields external light, a step of rendering the image and determining the luminance of each pixel; a step of detecting, based on predetermined information, an amount of decrease in the amount of light incident on the user's eyes due to shielding of external light when wearing of the head-mounted display is detected; a step of controlling to increase the luminance of each pixel in the image by an adjustment amount depending on the luminance at the time of rendering according to a rule corresponding to the amount of decrease in the light quantity; a step of converting the luminance of each pixel according to the control; a step of outputting data of the image whose luminance has been converted; An image display method characterized by including the above.

13. On a computer for displaying an image on a head-mounted display having a structure that shields external light, a function of rendering the image and determining the luminance of each pixel; a function of detecting, based on predetermined information, an amount of decrease in the amount of light incident on the user's eyes due to shielding of external light when wearing of the head-mounted display is detected; a function of controlling to increase the luminance of each pixel in the image by an adjustment amount depending on the luminance at the time of rendering according to a rule corresponding to the amount of decrease in the light quantity; a function of converting the luminance of each pixel according to the control; a function of outputting data of the image whose luminance has been converted; A computer program characterized by realizing the above.

Images