Image display device, control method for image display device, and program

Abstract

Even in bright or dark environments that exceed the exposure control limits of the imaging unit that captures the real world, the system allows users to understand the real world environment by viewing the real-world image displayed on the display unit. [Solution] An HMD (image display device) 100 is provided with a display unit 106 positioned to cover the user's eyes and displaying an experiential image including a real-world image to the user. The HMD includes an imaging unit 102 that images the real world and acquires a real-world image, a gaze imaging unit 202 that images the area including the user's eyes and acquires an eye image, an eye-opening degree determination unit 122 that determines the degree of eye opening based on the eye image acquired by the gaze imaging unit 202, and an imaging control unit 108 that, when the exposure control limit in the imaging unit 102 is reached, controls the apparent brightness of the real-world image on the display unit 106 according to the degree of eye opening determined by the eye-opening degree determination unit 122.

Description

Technical Field

[0001] The present invention relates to an image display apparatus, a control method for an image display apparatus, and a program.

Background Art

[0002] There is a Mixed Reality (MR) technology that fuses a real space and a virtual space to allow a user to interact with a virtual object. As one of such MR technologies, a technology using an HMD (Head Mounted Display), which is an example of an image display apparatus, is known. In this technology, first, a real space observed by a user wearing the HMD on the head is imaged by a camera (imaging unit) mounted on the front face of the HMD to generate a real space image. Further, based on the position and orientation of the camera (imaging unit), a CG image, which is a virtual space image rendered (drawn) by Computer Graphics (CG), is generated. Then, a composite image obtained by composing the real space image and the virtual space image (CG image) is displayed on the display unit of the HMD as an experience image, so that the user can observe the experience image.

[0003] Patent Document 1 describes, with respect to an HMD, a technique for correcting an image to be displayed to a user based on the pupil position of the user's eye in order to reduce optical distortion inherent to an optical lens through which the user views the image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0005] However, the technology described in Patent Document 1 has a problem in that, in bright or dark environments that exceed the exposure control limit of the imaging unit that captures the real space, it is difficult to grasp the real space environment simply by looking at the real space image displayed on the display unit.

[0006] This invention has been made in view of these problems, and aims to enable users to understand the environment of the real space by viewing the real space image displayed on the display unit, even in bright or dark environments that exceed the exposure control limit of the imaging unit that captures the real space. [Means for solving the problem]

[0007] The present invention provides an image display device comprising a display unit positioned to cover a user's eyes and displaying an experiential image including a real-space image to the user, the display device comprising: a first imaging means for capturing a real-space image and acquiring the real-space image; a second imaging means for capturing an area including the user's eyes and acquiring an eye image; a determination means for determining the degree to which the user's eyes are wide open based on the eye image; and a control means for performing control to change the apparent brightness of the real-space image on the display unit according to the degree to which the eyes are wide open when the exposure control limit of the first imaging means is reached. [Effects of the Invention]

[0008] According to the present invention, even in bright or dark environments that exceed the exposure control limit of the imaging unit that captures images of the real space, the environment of the real space can be understood by looking at the real space image displayed on the display unit. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of a schematic configuration of an HMD device (image display device) according to the first embodiment. [Figure 2] This figure shows an example of the main functional configuration of an HMD device (image display device) according to the first embodiment. [Figure 3]This figure illustrates the processing details of the eye organ detection unit and the eye opening degree determination unit shown in Figure 2, in the HMD device (image display device) according to the first embodiment. [Figure 4] This figure illustrates the processing contents of the imaging control unit shown in Figure 2 in the HMD device (image display device) according to the first embodiment. [Figure 5] This flowchart shows an example of a processing procedure in the control method for an HMD device (image display device) according to the first embodiment. [Figure 6] This figure shows an example of the main functional configuration of an HMD device (image display device) according to the second embodiment. [Figure 7] This figure illustrates the processing details of the eye organ detection unit and the eye opening degree determination unit shown in Figure 6, in the HMD device (image display device) according to the second embodiment. [Figure 8] This figure illustrates the processing contents of the image correction control unit shown in Figure 6 in the HMD device (image display device) according to the second embodiment. [Figure 9] This flowchart shows an example of a processing procedure in the control method for an HMD device (image display device) according to the second embodiment. [Figure 10] This figure shows an example of the main functional configuration of an HMD device (image display device) according to the third embodiment. [Figure 11] This is a flowchart showing an example of a processing procedure in the control method for an HMD device (image display device) according to the third embodiment. [Modes for carrying out the invention]

[0010] The embodiments for carrying out the present invention will be described below with reference to the drawings.

[0011] (First embodiment) First, let me describe the first embodiment.

[0012] Figure 1 shows an example of a schematic configuration of an HMD device (image display device) 100 according to the first embodiment. The HMD device 100 shown in Figure 1 is an image display device that is worn on the user's head and positioned to cover the user's eyes, and includes a display unit that displays an experiential image including a real-world image to the user. Specifically, in this embodiment, an example will be described in which a composite image, which is a combination of a real-world image and a virtual-world image (CG image), is displayed as the experiential image to allow the user to experience mixed reality (MR).

[0013] As shown in Figure 1, the HMD device 100 includes a lens unit 101, an imaging unit 102, an imaging signal processing unit 103, an image signal processing unit 104, an image synthesis unit 105, a display unit 106, an eyepiece optical system 107, an imaging control unit 108, and an HMD system control unit 109. Furthermore, as shown in Figure 1, the HMD device 100 includes a gaze camera control unit 110, a gaze camera unit 111, a gaze detection unit 112, a memory unit 113, a power supply unit 114, an HMD operation unit 115, an HMD shake detection unit 116, a position estimation unit 117, and an HMD communication control unit 118. Furthermore, as shown in Figure 1, the HMD device 100 includes a microphone unit 119, a microphone control unit 120, an eye organ detection unit 121, an eye opening degree determination unit 122, a CG communication control unit 123, and a computer graphics generation unit (CG generation unit) 124.

[0014] The lens unit 101 is an optical system that guides light from the real-world environment surrounding the HMD device 100 to the imaging unit 102. The imaging unit 102 is a first imaging means that, based on the control of the imaging control unit 108, images the real-world environment surrounding the HMD device 100 and acquires a real-world image. The imaging unit 102 is configured such that pixels, including photoelectric conversion elements that convert light incident from the lens unit 101 into electrical signals related to the real-world image, are arranged in a two-dimensional manner.

[0015] The imaging signal processing unit 103 and the video signal processing unit 104 perform signal processing on the real space image output from the imaging unit 102 under the control of the HMD system control unit 109. The imaging signal processing unit 103 performs, for example, processing to remove fixed noise (noise from predetermined pixels of the imaging unit 102) from the real space image. Further, the video signal processing unit 104 performs, for example, processing to change the image format on the real space image processed by the imaging signal processing unit 103.

[0016] The image composition unit 105 composites the real space image processed by the video signal processing unit 104 and the CG image (virtual space image) generated by the CG generation unit 124 to generate a composite image. Under the control of the HMD system control unit 109, the display unit 106 displays the composite image generated by the image composition unit 105 to the user as an experience image. The eyepiece optical system 107 is an optical system for forming an image of the experience image displayed on the display unit 106 on the user's eye (more specifically, the retina of the eye).

[0017] The imaging control unit 108 performs control for imaging the real space in the imaging unit 102 under the control of the HMD system control unit 109. The imaging control unit 108 performs, for example, exposure control and the like in the imaging unit 102. The HMD system control unit 109 controls each component of the HMD device 100, comprehensively controls the operation of the HMD device 100, and performs various processes. This HMD system control unit 109 is configured to include, for example, a CPU.

[0018] The gaze camera control unit 110 controls the operation of the gaze camera unit 111 based on the control of the HMD system control unit 109. Based on the control of the gaze camera control unit 110, the gaze camera unit 111 processes to capture an image of the user's eyes by imaging the area including the user's eyes. Based on the control of the HMD system control unit 109, the gaze detection unit 112 estimates and detects which direction the user's gaze is directed using the image of the user's eyes captured by the gaze camera unit 111. In this embodiment, the gaze camera control unit 110, the gaze camera unit 111, and the gaze detection unit 112 constitute a gaze imaging unit, which is a second imaging means that captures an image of the user's eyes by imaging the area including the user's eyes.

[0019] The memory unit 113 stores various information (including programs, data, images, signals, etc.) necessary for various controls and processes in the HMD device 100. The memory unit 113 also stores various information (including data, images, signals, etc.) obtained through various controls and processes in the HMD device 100.

[0020] The power supply unit 114 supplies power to each component of the HMD device 100 according to its purpose via the HMD system control unit 109. The HMD operation unit 115 is a component operated by the user when they input an operation to the HMD device 100. The HMD operation unit 115 outputs an operation signal based on the user's operation input to the HMD system control unit 109. The HMD vibration detection unit 116 detects the amount of vibration applied to the HMD device 100. The HMD vibration detection unit 116 outputs a detection signal based on the detected amount of vibration applied to the HMD device 100, along with a real-space image processed by the video signal processing unit 104, to the position estimation unit 117. The position estimation unit 117 uses the detection signal output from the HMD vibration detection unit 116 and the real-space image to estimate the position of the HMD device 100.

[0021] The HMD communication control unit 118 controls communication between the HMD system control unit 109, the microphone control unit 120, and the CG communication control unit 123.

[0022] The microphone unit 119 converts sound into electrical signals (audio signals) based on the control of the microphone control unit 120. The microphone control unit 120 controls the operation of the microphone unit 119 based on the control of the HMD system control unit 109. The microphone control unit 120 also converts the audio signals obtained by the microphone unit 119 into audio stream data. The audio stream data obtained by the microphone control unit 120 is sent to the CG generation unit 124 along with the real-world image processed by the video signal processing unit 104 and stored in the storage unit 113.

[0023] The eye organ detection unit 121 detects the position of the user's eye organs using the user's eye image captured by the gaze camera unit 111. The eye opening degree determination unit 122 is a determination means that determines the degree of eye opening of the user based on the position of the user's eye organs detected by the eye organ detection unit 121. In this embodiment, the eye opening degree determination unit 122 determines the degree of eye opening of the user as a normal eye opening state, an open eye state, a squinted eye state, etc. In this case, the states from the greatest degree of eye opening to the least are the open eye state, the normal eye opening state, and the squinted eye state.

[0024] The CG communication control unit 123 controls communication with the HMD system control unit 109 via the HMD communication control unit 118. The CG generation unit 124 generates a CG image, which is a virtual space image, using various information (including data, images, signals, etc.) stored in the storage unit 113. The CG generation unit 124 then outputs the generated CG image to the image synthesis unit 105.

[0025] The following describes the processing overview of the HMD device 100 according to the first embodiment. The imaging control unit 108, for example, uses a program diagram to control exposure in order to set the exposure of the imaging unit 102 (exposure of the real-world image) to an appropriate brightness. The imaging control unit 108 is a control means that, when the exposure control limit of the imaging unit 102 is reached, controls the apparent brightness of the real-world image on the display unit 106 according to the degree of eye opening determined by the eye opening degree determination unit 122.

[0026] For example, in the first case where the real-world environment surrounding the HMD device 100 is dark enough to reach the low-luminance limit of the exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the eyes are wide open, the imaging control unit 108 performs the following processing. Specifically, in this first case, the imaging control unit 108 controls the apparent brightness of the real-world image on the display unit 106 to be brighter than the low-luminance limit of the exposure control in the imaging unit 102 (the low-luminance limit in the program diagram). More specifically, in this embodiment, in the first case described above, the imaging control unit 108 controls the apparent brightness of the real-world image on the display unit 106 by setting an ISO sensitivity higher than the low-luminance limit of the exposure control in the imaging unit 102. In this way, by setting a high ISO sensitivity that would not be set in the normal exposure control of the imaging unit 102, although noise increases, the exposure linkage range on the low-luminance side is widened, and a real-world image with appropriate brightness and high visibility can be displayed on the display unit 106. Conventionally, if the real-world environment around the HMD device 100 was too dark, the user could not confirm the real-world environment simply by viewing the overly dark real-world image displayed on the display unit 106, due to the low-luminance limit of the exposure control in the imaging unit 102. In this conventional case, the user had to remove the HMD device 100 to confirm the real-world environment around it. In contrast, in this embodiment, the user can confirm the real-world environment around the HMD device 100 by viewing the real-world image displayed on the display unit 106 while the HMD device 100 is still attached. In other words, in this embodiment, even if the real-world environment around the HMD device 100 is too dark for normal exposure control to handle, the user can confirm the environment by viewing a real-world image with appropriate brightness displayed on the display unit 106.

[0027] Furthermore, in the second case where the real-world environment surrounding the HMD device 100 is bright enough to reach the high-luminance limit of the exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the user is squinting, the imaging control unit 108 performs the following processing. Specifically, in this second case, the imaging control unit 108 controls the display unit 106 to darken the apparent brightness of the real-world image by exceeding the high-luminance limit of the exposure control in the imaging unit 102 (the high-luminance limit in the program diagram). More specifically, in this embodiment, in the second case described above, the imaging control unit 108 controls the display unit 106 to darken the apparent brightness of the real-world image by setting a shutter speed faster than the high-luminance limit of the exposure control in the imaging unit 102. By setting a shutter speed that is faster than that which would not be set in the normal exposure control of the imaging unit 102, although flickering increases, the high-luminance exposure linkage range is widened, and a real-world image with appropriate brightness and high visibility can be displayed on the display unit 106. Conventionally, if the real-world environment surrounding the HMD device 100 was too bright, the user could not confirm the real-world environment simply by viewing the overly bright real-world image displayed on the display unit 106, due to the high-brightness limit of the exposure control in the imaging unit 102. In this conventional case, the user had to remove the HMD device 100 to confirm the real-world environment surrounding the HMD device 100. In contrast, in this embodiment, the user can confirm the real-world environment surrounding the HMD device 100 by viewing the real-world image displayed on the display unit 106 while the HMD device 100 is still attached. In other words, in this embodiment, even if the real-world environment surrounding the HMD device 100 is too bright for normal exposure control to handle, the user can confirm the environment by viewing the real-world image with appropriate brightness displayed on the display unit 106.

[0028] Thus, in this embodiment, when the exposure control limit in the imaging unit 102 is reached, the imaging control unit 108 controls the apparent brightness of the real-world image on the display unit 106 according to the degree of eye opening determined by the eye-opening degree determination unit 122. That is, the imaging control unit 108 controls the apparent brightness of the real-world image on the display unit 106 beyond the normal exposure control limit in the imaging unit 102, and temporarily expands the exposure linkage range. As a result, even if the real-world environment around the HMD device 100 is too dark or too bright, the user can check the real-world environment around the HMD device 100 from the real-world image displayed on the display unit 106 while wearing the HMD device 100.

[0029] Figure 2 shows an example of the main functional configuration of the HMD device (image display device) 100 according to the first embodiment. In Figure 2, components similar to those shown in Figure 1 are denoted by the same reference numerals, and their detailed explanations are omitted. In the following description, the HMD device (image display device) 100 having the functional configuration shown in Figure 2 will be referred to as "HMD device (image display device) 100-1".

[0030] The HMD device 100-1 has the following functional configurations: lens unit 101, imaging unit 102, image synthesis unit 105, display unit 106, imaging control unit 108, position estimation unit 117, eye organ detection unit 121, eye opening degree determination unit 122, and CG generation unit 124, as shown in Figure 1. Furthermore, the HMD device 100-1 has the following functional configurations: image processing unit 201 and gaze imaging unit 202. Figure 2 illustrates the user's eye E.

[0031] The image processing unit 201 shown in Figure 2 includes, for example, the imaging signal processing unit 103 and the video signal processing unit 104 shown in Figure 1. The image processing unit 201 performs image processing on the real-world image output from the imaging unit 102.

[0032] The gaze imaging unit 202 shown in Figure 2 is configured to include, for example, the gaze camera control unit 110, gaze camera unit 111, and gaze detection unit 112 shown in Figure 1. The gaze imaging unit 202 is a second imaging means that captures an image of the user's eyes by imaging the area including the user's eyes.

[0033] In Figure 2, the eye organ detection unit 121 detects the position of the user's eye organs using the user's eye image acquired by the gaze imaging unit 202. Specifically, in this embodiment, the eye organ detection unit 121 detects the positions of the user's upper and lower eyelids. The eye-opening degree determination unit 122 then determines the degree of eye-opening based on the distance between the user's upper and lower eyelids detected by the eye organ detection unit 121. More specifically, the eye-opening degree determination unit 122 determines the normal eye-opening state, wide-open eye state, narrowed eye state, etc., based on the distance between the user's upper and lower eyelids detected by the eye organ detection unit 121 and the time that distance is maintained.

[0034] In this embodiment, as described above, the imaging control unit 108 uses a program diagram to control the exposure of the imaging unit 102. When the exposure control limit of the imaging unit 102 is reached, the imaging control unit 108 controls the apparent brightness of the real-world image on the display unit 106 according to the degree of eye opening determined by the eye-opening degree determination unit 122. For example, in the first case where the real-world environment surrounding the HMD device 100 is dark and has reached the low-luminance limit of the exposure control of the imaging unit 102, and the eye-opening degree determination unit 122 has determined that the user's eyes are wide open, the imaging control unit 108 performs the following processing. Specifically, in this first case, the imaging control unit 108 controls the apparent brightness of the real-world image on the display unit 106 to exceed the low-luminance limit of the exposure control of the imaging unit 102. More specifically, in this embodiment, in the first case described above, the imaging control unit 108 sets an ISO sensitivity higher than the low-luminance limit of the exposure control in the imaging unit 102 to control the apparent brightness of the real-space image on the display unit 106. Furthermore, in the second case, when the real-space environment surrounding the HMD device 100 is bright enough to reach the high-luminance limit of the exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the user is squinting, the imaging control unit 108 performs the following processing. Specifically, in this second case, the imaging control unit 108 controls the apparent brightness of the real-space image on the display unit 106 by exceeding the high-luminance limit of the exposure control in the imaging unit 102. More specifically, in this embodiment, in the second case described above, the imaging control unit 108 sets a shutter speed faster than the high-luminance limit of the exposure control in the imaging unit 102 to control the apparent brightness of the real-space image on the display unit 106.

[0035] Figure 3 is a diagram illustrating the processing contents of the eye organ detection unit 121 and the eye opening degree determination unit 122 shown in Figure 2 in the HMD device (image display device) 100 according to the first embodiment.

[0036] In Figure 3, the eye image 310 is an example of an eye image acquired by the gaze imaging unit 202 when the user's eyes are wide open. The eye organ detection unit 121 detects the positions of the user's upper eyelid 311 and lower eyelid 312 in the eye image 310. Then, for example, the eye-opening degree determination unit 122 calculates a distance β, which is the distance 313 between the user's upper eyelid 311 and lower eyelid 312 detected by the eye organ detection unit 121. Based on the calculated distance β(313), the eye-opening degree determination unit 122 determines that the user's eyes are wide open.

[0037] In Figure 3, the eye image 320 is an example of an eye image acquired by the gaze imaging unit 202 when the user's eyes are in a normal wide-open state. The eye organ detection unit 121 detects the positions of the user's upper eyelid 321 and lower eyelid 322 in the eye image 320. Then, for example, the eye-opening degree determination unit 122 calculates a distance α, which is the distance 323 between the user's upper eyelid 321 and lower eyelid 322 detected by the eye organ detection unit 121. Based on the calculated distance α(323), the eye-opening degree determination unit 122 determines that the user's eyes are in a normal wide-open state.

[0038] In Figure 3, the eye image 330 is an example of an eye image acquired by the gaze imaging unit 202 when the user is squinting. The eye organ detection unit 121 detects the positions of the user's upper eyelid 331 and lower eyelid 332 in the eye image 330. Then, for example, the eye-opening degree determination unit 122 calculates distance γ, which is the distance 333 between the user's upper eyelid 331 and lower eyelid 332 detected by the eye organ detection unit 121. Based on the calculated distance γ(333), the eye-opening degree determination unit 122 determines that the user is squinting as the degree of eye opening.

[0039] In Figure 3, graph 340 shows an example of the relationship between the distance between the upper and lower eyelids of the user's eyes, calculated by the eye-opening degree determination unit 122, and time. Specifically, in graph 340 of Figure 3, time is plotted on the horizontal axis and the distance between the upper and lower eyelids of the user's eyes is plotted on the vertical axis. The distance β(313) calculated using the eye image 310 in Figure 3 corresponds to distance β', which is distance 341 in graph 340. Similarly, the distance α(323) calculated using the eye image 320 in Figure 3 corresponds to distance α', which is distance 342 in graph 340. Furthermore, the distance γ(333) calculated using the eye image 330 in Figure 3 corresponds to distance γ', which is distance 343 in graph 340. In addition, in graph 340, the length of time 344 at distance β'(341) represents the length of time the user has their eyes wide open. The user's eyes can also be wide open due to emotions, etc. If the user maintains a wide-eyed state for a certain period of time (for example, a duration of 344), the eye-opening degree determination unit 122 determines that the user's eyes are wide open, thereby excluding eye-opening caused by emotions or other factors. This makes it possible to determine the degree of eye-opening that accurately reflects the user's intentions.

[0040] Figure 4 is a diagram illustrating the processing content of the imaging control unit 108 shown in Figure 2 in the HMD device (image display device) 100 according to the first embodiment. Specifically, Figure 4 is a diagram illustrating the control on the program diagram by the imaging control unit 108 to realize the degree of eye opening and the exposure linkage range that exceeds the limit.

[0041] In this embodiment, as described above, the imaging control unit 108 uses a program diagram to control the exposure of the imaging unit 102. In this case, the program diagram, as shown in Figure 4, includes an SV diagram 410 for controlling the ISO sensitivity of the imaging unit 102 and a TV diagram 420 for controlling the shutter speed and aperture of the imaging unit 102. In the SV diagram 410 and the TV diagram 420, 1 / 60, 1 / 500, 1 / 2000, and 1 / 4000 written horizontally indicate the shutter speed of the imaging unit 102. Also, in the SV diagram 410, ISO100, ISO1600, and ISO3200 written diagonally upward from the left end indicate the ISO sensitivity of the imaging unit 102. Figure 4 also illustrates the situation where the user has their eyes wide open 401, the situation where the user has their eyes wide open in a normal state 402, and the situation where the user is squinting 403.

[0042] In the program diagram shown in Figure 4, the low-luminance ISO sensitivity limit 411 is set as the low-luminance limit for exposure control in the imaging unit 102. Also in the program diagram shown in Figure 4, the high-luminance shutter speed limit 421 is set as the high-luminance limit for exposure control in the imaging unit 102. In this embodiment, in order to achieve an exposure linkage range that exceeds the exposure control limits in the imaging unit 102 according to the degree to which the user's eyes are wide open, it is necessary to change the low-luminance ISO sensitivity limit 411 and the high-luminance shutter speed limit 421.

[0043] In this case, simply changing the ISO sensitivity limit 411 on the low-luminance side to the higher ISO sensitivity 412 would increase noise, making the real-world image grainy and noisy, which would compromise the quality of the HMD device 100 as a product. In this embodiment, instead of always performing this ISO sensitivity change control, the ISO sensitivity change control is performed only when the external environment reaches the low-luminance side ISO sensitivity limit 411 and is dark, and the user is in a situation 401 where it is dark and their eyes are wide open. In this embodiment, by controlling the low-luminance side ISO sensitivity limit 411 to the higher ISO sensitivity 412 in this case, a noisy real-world image is not always displayed, minimizing the impact on product quality. Thus, even in situations where, conventionally, the external environment is too dark for the user to see without removing the HMD device 100, in this embodiment, the exposure can be adjusted, allowing the user to see the external environment while wearing the HMD device 100.

[0044] Furthermore, changing the high-brightness shutter speed limit 421 to a faster shutter speed 422 makes it possible to widen (expand) the exposure linkage range of the exposure control in the imaging unit 102. However, if the shutter speed of the imaging unit 102 is too fast, problems may arise where the real-world image appears to flicker excessively when a moving object passes in front of the imaging unit 102 or when the user moves the HMD device 100. Simply changing the high-brightness shutter speed limit 421 to a faster shutter speed 422 would compromise the product quality of the HMD device 100. In this embodiment, this shutter speed change control is not always performed, but only when the external environment is bright enough to reach the high-brightness shutter speed limit 421, and the user is squinting due to the glare 403. In this embodiment, by controlling the shutter speed limit 421 on the high-brightness side to a faster shutter speed 422, a constantly flickering image of the real world is eliminated, minimizing the impact on product quality. Thus, even in situations where, conventionally, the external environment is too bright for the user to see without removing the HMD device 100, this embodiment allows the exposure to adjust, enabling the user to see the external environment while wearing the HMD device 100.

[0045] Figure 5 is a flowchart showing an example of a processing procedure in the control method of the HMD device (image display device) 100 according to the first embodiment.

[0046] First, in step S501 of Figure 5, the gaze imaging unit 202 images the area including the user's eyes and acquires an image of the user's eyes.

[0047] Next, in step S502 of Figure 5, the eye organ detection unit 121 detects the position of the user's eye organs using the user's eye image acquired in step S501. Specifically, in this embodiment, the eye organ detection unit 121 detects the positions of the user's upper and lower eyelids.

[0048] Next, in step S503 of Figure 5, the eye-opening degree determination unit 122 determines the degree of eye-opening of the user based on the distance between the upper and lower eyelids of the user's eyes detected in step S502. More specifically, the eye-opening degree determination unit 122 determines the degree of eye-opening of the user's eyes based on the distance between the upper and lower eyelids of the user's eyes detected in step S502, and determines whether the eyes are in a normal open state, a wide-open state, or a squinted state.

[0049] Next, in step S504 of Figure 5, the imaging control unit 108 determines whether the degree to which the user's eyes are wide open, as determined in step S503, indicates that the eyes are wide open.

[0050] In step S504 of Figure 5, if the imaging control unit 108 determines that the degree to which the user's eyes are open, as determined in step S503, is not in a wide-open state (S504 / NO), the process proceeds to step S505. When the process proceeds to step S505 in Figure 5, the imaging control unit 108 determines whether the degree to which the user's eyes are wide open, as determined in step S503, indicates that the user is squinting.

[0051] In step S505 of Figure 5, if the imaging control unit 108 determines that the degree of eye opening determined in step S503 is not that the user is squinting (S505 / NO), the process proceeds to step S506. In this embodiment, proceeding to step S506 corresponds to the case in step S503 where the degree of eye opening determined to be that the user is in a normal wide-eyed state. When the process proceeds to step S506 in Figure 5, the imaging control unit 108 calculates the exposure for the imaging unit 102 based on the determination in step S503 that the degree to which the user's eyes are open is in a normal open-eye state.

[0052] Next, in step S507 of Figure 5, the imaging unit 102 performs exposure based on the exposure calculation in step S506.

[0053] Furthermore, in step S504 of Figure 5, if the imaging control unit 108 determines that the degree of eye opening of the user, as determined in step S503, is in a wide-open state (S504 / YES), the process proceeds to step S508. When the process proceeds to step S508 in Figure 5, the imaging control unit 108 calculates the exposure for the imaging unit 102 based on the determination result in step S503 that the user's eyes are wide open.

[0054] Next, in step S509 of Figure 5, the imaging control unit 108 determines whether or not the ISO sensitivity limit 411, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102 based on the exposure calculation in step S508, has been reached.

[0055] In step S509 of Figure 5, if the imaging control unit 108 determines that the ISO sensitivity limit 411, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102, has not been reached (S509 / NO), the process proceeds to step S510. When the process proceeds to step S510 in Figure 5, the imaging unit 102 performs exposure based on the exposure calculation in step S508.

[0056] Furthermore, in step S509 of Figure 5, if the imaging control unit 108 determines that the ISO sensitivity limit 411, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102, has been reached (S509 / YES), the process proceeds to step S511. When the process proceeds to step S511 in Figure 5, the imaging control unit 108 sets a higher ISO sensitivity value of 412, which exceeds the ISO sensitivity limit 411, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102.

[0057] Next, in step S512 of Figure 5, the imaging unit 102 performs exposure based on a higher ISO sensitivity value 412, which exceeds the ISO sensitivity limit 411, the upper limit of the low-luminance limit for exposure control in the imaging unit 102, set in step S511.

[0058] Furthermore, in step S505 of Figure 5, if the imaging control unit 108 determines that the degree to which the user's eyes are wide open, as determined in step S503, indicates that the user is squinting (S505 / YES), the unit proceeds to step S513. When the process proceeds to step S513 in Figure 5, the imaging control unit 108 calculates the exposure for the imaging unit 102 based on the determination in step S503 that the user's eyes are squinting.

[0059] Next, in step S514 of Figure 5, the imaging control unit 108 determines whether or not the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102 based on the exposure calculation in step S513, has been reached.

[0060] In step S514 of Figure 5, if the imaging control unit 108 determines that the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102, has not been reached (S514 / NO), the process proceeds to step S515. When the process proceeds to step S515 in Figure 5, the imaging unit 102 performs exposure based on the exposure calculation in step S513.

[0061] Furthermore, in step S514 of Figure 5, if the imaging control unit 108 determines that the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102 has been reached (S514 / YES), the process proceeds to step S516. When the process proceeds to step S516 in Figure 5, the imaging control unit 108 sets a value for a shutter speed 422 that exceeds the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102.

[0062] Next, in step S517 of Figure 5, the imaging unit 102 performs exposure based on a faster shutter speed 422 value that exceeds the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102, as set in step S516.

[0063] If the exposure process in step S507, step S510, step S512, step S515, or step S517 in Figure 5 is completed, proceed to step S518. When the process proceeds to step S518 in Figure 5, the image processing unit 201 performs image processing on the real-space image output from the imaging unit 102.

[0064] Next, in step S519 of Figure 5, the image synthesis unit 105 first synthesizes the real-world image processed in step S518 with the CG image (virtual space image) generated by the CG generation unit 124 to generate a composite image. Then, the display unit 106 displays the composite image generated by the image synthesis unit 105 as an experience image for the user.

[0065] Next, in step S520 in Figure 5, for example, the HMD system control unit 109 determines whether or not to terminate the display of the experience image on the display unit 106 based on the operation input of the HMD operation unit 115. In step S520 in Figure 5, for example, if the HMD system control unit 109 determines not to terminate the display of the experience image on the display unit 106 (S520 / NO), the process returns to step S501 and repeats the processing from step S501 onwards.

[0066] Furthermore, in step S520 of Figure 5, if, for example, the HMD system control unit 109 determines that it has finished displaying the experience image on the display unit 106 (S520 / YES), the process shown in the flowchart of Figure 5 is terminated.

[0067] As described above, the HMD device 100 according to the first embodiment is an image display device equipped with a display unit 106 positioned to cover the user's eyes and displaying an experiential image including a real-world image to the user. The HMD device 100 according to the first embodiment has an imaging unit 102, which is a first imaging means for capturing images of the real world and acquiring real-world images, and a gaze imaging unit 202, which is a second imaging means for capturing images of the area including the user's eyes and acquiring eye images. The HMD device 100 according to the first embodiment also has an eye-opening degree determination unit 122, which is a determination means for determining the degree to which the user's eyes are open based on the eye images acquired by the gaze imaging unit 202. Furthermore, in the HMD device 100 according to the first embodiment, the imaging control unit 108, which is a control means, performs the following control when the exposure control limit in the imaging unit 102 is reached. In other words, when the exposure control limit in the imaging unit 102 is reached, the imaging control unit 108 controls the apparent brightness of the real-space image on the display unit 106 according to the degree of eye opening determined by the eye opening degree determination unit 122. According to the configuration of the HMD device 100 as described in this first embodiment, even in bright or dark environments that exceed the exposure control limit of the imaging unit 102 that captures images of the real world, the user can understand the real world environment by looking at the real world image displayed on the display unit 106. As a result, the user can check the real world environment around the HMD device 100 by viewing the real world image displayed on the display unit 106 without removing the HMD device 100, while the HMD device 100 is still attached. The control of changing the apparent brightness of the real world image on the display unit 106 is performed according to the degree to which the user's eyes are wide open, and therefore it is performed only at the timing intended by the user. Therefore, while preventing malfunctions that the user did not anticipate, the user can maintain a good user experience (UX) in any brightness environment while wearing the HMD device 100. You can experience this.

[0068] (Second embodiment) Next, a second embodiment will be described. In the description of the second embodiment below, matters common to the first embodiment described above will be omitted, and matters that differ from the first embodiment described above will be explained.

[0069] The schematic configuration of the HMD device (image display device) according to the second embodiment is the same as the schematic configuration of the HMD device (image display device) 100 according to the first embodiment shown in Figure 1.

[0070] Figure 6 shows an example of the main functional configuration of the HMD device (image display device) 100 according to the second embodiment. In Figure 6, components similar to those shown in Figures 1 and 2 are denoted by the same reference numerals, and their detailed explanations are omitted. In the following description, the HMD device (image display device) 100 having the functional configuration shown in Figure 6 will be referred to as "HMD device (image display device) 100-2".

[0071] The HMD device 100-2 has the following functional configurations: lens unit 101, imaging unit 102, image synthesis unit 105, display unit 106, position estimation unit 117, eye organ detection unit 121, eye opening degree determination unit 122, CG generation unit 124, image processing unit 201, and gaze imaging unit 202, as shown in Figure 2. In this case, the eye organ detection unit 121 of the HMD device 100-2 has the functional configuration of a pupil detection unit 602 internally. Furthermore, the HMD device 100-2 has the functional configuration of an image correction control unit 601. The image correction control unit 601 shown in Figure 6 is configured to include, for example, the imaging control unit 108 and the HMD system control unit 109 shown in Figure 1.

[0072] The pupil detection unit 602 detects the pupil of the user's eye using the user's eye image acquired by the gaze imaging unit 202. The eye-opening degree determination unit 122 then determines the degree of eye-opening based on the size of the user's pupil detected by the pupil detection unit 602. More specifically, the eye-opening degree determination unit 122 determines the degree of eye-opening as follows: normal eye-open state (perceiving normal brightness), wide-open state (perceiving darkness), squinted state (perceiving glare), etc.

[0073] The image correction control unit 601 performs exposure control, for example, so that the brightness of the real-world image output from the imaging unit 102 is within a certain range. In the second embodiment, this exposure control is performed using a program diagram, similar to the first embodiment. The image correction control unit 601 is a control means that, when the exposure control limit in the imaging unit 102 is reached, changes the apparent brightness of the real-world image on the display unit 106 according to the degree of eye opening determined by the eye opening degree determination unit 122.

[0074] For example, in the first case where the real-world environment surrounding the HMD device 100 is dark enough to reach the low-luminance limit of exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the eyes are wide open, the image correction control unit 601 performs the following processing. Specifically, in this first case, the image correction control unit 601 controls the apparent brightness of the real-world image on the display unit 106 to exceed the low-luminance limit of exposure control in the imaging unit 102 (the low-luminance limit in the program diagram). More specifically, in this embodiment, in the first case described above, the image correction control unit 601 controls the image processing unit 201 to perform correction to brighten the real-world image using digital gain, thereby brightening the apparent brightness of the real-world image on the display unit 106. Due to this digital gain, the image processing unit 201 outputs a brighter real-world image, although the noise increases. In this first case, the low-luminance exposure linkage range is widened (expanded), and the exposure can track even in dark environments. In other words, in this embodiment, even if the real-world environment surrounding the HMD device 100 is a dark environment that cannot be tracked by normal exposure control, the user can confirm the environment by viewing a real-world image with appropriate brightness displayed on the display unit 106.

[0075] Furthermore, in the second case where the real-world environment surrounding the HMD device 100 is bright enough to reach the high-luminance limit of the exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the user is squinting, the image correction control unit 601 performs the following processing. Specifically, in this second case, the image correction control unit 601 controls the apparent brightness of the real-world image on the display unit 106 to darken it by exceeding the high-luminance limit of the exposure control in the imaging unit 102 (the high-luminance limit in the program diagram). More specifically, in this embodiment, in the second case described above, the image correction control unit 601 controls the image processing unit 201 to perform correction to darken the real-world image using gamma curve characteristics, thereby controlling the apparent brightness of the real-world image on the display unit 106. Specifically, the image processing unit 201 outputs a dark real-world image using gamma curve characteristics designed to suppress high luminance. In this second case, the exposure linkage range on the high-brightness side is widened (expanded), and the exposure can track even in bright environments. That is, in this embodiment, even if the real-world environment around the HMD device 100 is a bright environment that cannot be tracked by normal exposure control, the user can confirm the environment by viewing the real-world image with appropriate brightness displayed on the display unit 106.

[0076] Figure 7 is a diagram illustrating the processing content of the eye organ detection unit 121 and the eye opening degree determination unit 122 shown in Figure 6 in the HMD device (image display device) 100 according to the second embodiment. In Figure 7, eye images 710 to 730 obtained by capturing the area including the user's eyes with the gaze imaging unit 202 positioned at a bush-gazing position (diagonally downward position) are shown.

[0077] In Figure 7, the eye image 710 is an example of an eye image acquired by the gaze imaging unit 202 when the user's eyes are wide open. The pupil detection unit 602 included in the eye organ detection unit 121 detects the pupil of the user's eye in the eye image 710. Then, for example, the eye-opening degree determination unit 122 calculates the size β, which is the size 711 of the user's pupil detected by the pupil detection unit 602. Then, based on the calculated size β(711) of the user's pupil, the eye-opening degree determination unit 122 determines that the user's eyes are wide open.

[0078] In Figure 7, the eye image 720 is an example of an eye image acquired by the gaze imaging unit 202 when the user's eyes are in a normal wide-open state. The pupil detection unit 602 included in the eye organ detection unit 121 detects the pupil of the user's eye in the eye image 720. Then, for example, the eye-opening degree determination unit 122 calculates the size α, which is the size 721 of the user's eye pupil detected by the pupil detection unit 602. Based on the calculated size α(721) of the user's eye pupil, the eye-opening degree determination unit 122 determines the user's eye-opening degree as a normal wide-open state.

[0079] In Figure 7, the eye image 730 is an example of an eye image acquired by the gaze imaging unit 202 when the user is squinting. The pupil detection unit 602 included in the eye organ detection unit 121 detects the pupil of the user's eye in the eye image 730. Then, for example, the eye-opening degree determination unit 122 calculates the size γ, which is the size 731 of the user's pupil detected by the pupil detection unit 602. Then, based on the calculated size γ(731) of the user's pupil, the eye-opening degree determination unit 122 determines that the user is squinting as the degree of eye-opening.

[0080] In Figure 7, graph 740 shows an example of the relationship between the size of the user's pupil, calculated by the eye-opening degree determination unit 122, and brightness. Specifically, in graph 740 of Figure 7, brightness is plotted on the horizontal axis and the size of the user's pupil on the vertical axis. In the pupil size characteristic data 741 shown in graph 740 of Figure 7, the pupil size decreases as the brightness increases and increases as the brightness decreases. The eye-opening degree determination unit 122 uses this characteristic of pupil size characteristic data 741 to determine the degree of eye-opening. Specifically, for example, if the pupil size is larger than a first predetermined size, the eye-opening degree determination unit 122 determines that the user is feeling dim and that the user's eye-opening degree is wide open. Also, for example, if the pupil size is smaller than a second predetermined size, the eye-opening degree determination unit 122 determines that the user is feeling dazzled and that the user's eye-opening degree is narrowed. Furthermore, for example, the eye-opening degree determination unit 122 determines that if the pupil size is smaller than a first predetermined size and larger than a second predetermined size, the user perceives the brightness as normal, and the degree of eye opening of the user is determined to be a normal eye-opening state.

[0081] Figure 8 is a diagram illustrating the processing of the image correction control unit 601 shown in Figure 6 in the HMD device (image display device) 100 according to the second embodiment. Specifically, Figure 8 is a diagram illustrating the control on the program diagram for realizing the degree of eye opening and the exposure linkage range exceeding the limit by the image correction control unit 601. In Figure 8, the same reference numerals are used for components similar to those shown in Figure 4, and their detailed explanation is omitted.

[0082] In this embodiment, the image correction control unit 601 uses a program diagram to control the exposure of the imaging unit 102. In this case, the program diagram, as shown in Figure 8, includes an SV diagram 810 for controlling the ISO sensitivity of the imaging unit 102 and a TV diagram 820 for controlling the shutter speed and aperture of the imaging unit 102. In the SV diagram 810 and the TV diagram 820, 1 / 60, 1 / 500, 1 / 2000, and 1 / 4000 written horizontally indicate the shutter speed of the imaging unit 102. Also, in the SV diagram 810, ISO100 and ISO1600 written diagonally upward from the left end indicate the ISO sensitivity of the imaging unit 102. Figure 8 also illustrates the situation where the user has their eyes wide open 801, the situation where the user has their eyes wide open in a normal state 802, and the situation where the user is squinting 803.

[0083] In the program diagram shown in Figure 8, a low-luminance ISO sensitivity limit 811 is set as the low-luminance limit for exposure control in the imaging unit 102. Also in the program diagram shown in Figure 8, a high-luminance shutter speed limit 821 is set as the high-luminance limit for exposure control in the imaging unit 102. In this embodiment, in order to achieve an exposure linkage range that exceeds the low-luminance limit for exposure control in the imaging unit 102, it is necessary to raise the amount that becomes too dark because the exposure stops at the low-luminance ISO sensitivity limit 811 and cannot keep up, using the digital gain 812. Furthermore, in order to achieve an exposure linkage range that exceeds the high-luminance limit for exposure control in the imaging unit 102, it is necessary to change the amount that becomes too bright because the exposure stops at the high-luminance shutter speed limit 821 and cannot keep up, using the gamma curve characteristics. Specifically, in order to achieve an exposure linkage range that exceeds the high-brightness limit of exposure control in the imaging unit 102, the gamma curve characteristic data 831 shown in Figure 8 is changed from the normal gamma curve characteristic data 831 to the high-brightness suppression gamma curve characteristic data 832. Also, in the gamma curve characteristic data 830 shown in Figure 8, the horizontal axis represents the input and the vertical axis represents the output.

[0084] In this case, simply changing the low-luminance ISO sensitivity limit 811 and raising the brightness with the digital gain 812 would increase noise, making the real-world image grainy and noisy, which would compromise the quality of the HMD device 100 as a product. In this embodiment, instead of always performing this ISO sensitivity change control, the ISO sensitivity change control is performed only when the external environment reaches the low-luminance ISO sensitivity limit 811 and is dark, and the user is in a situation 801 where it is dark and their eyes are wide open. In this embodiment, by changing the low-luminance ISO sensitivity limit 811 and raising the brightness with the digital gain 812 in this case, a constantly noisy real-world image is not displayed, minimizing the impact on product quality. Thus, even in situations where, conventionally, the external environment is too dark for the user to see without removing the HMD device 100, in this embodiment, the exposure can be adjusted, allowing the user to see the external environment while wearing the HMD device 100.

[0085] Furthermore, on the high-brightness side, simply changing from the normal gamma curve characteristic data 831 to the high-brightness suppression gamma curve characteristic data 832 would reduce the contrast of the real-world image, resulting in problems with the product quality of the HMD device 100. In this embodiment, this control of changing the gamma curve characteristic data is not always performed, but rather the above-mentioned change control is performed only when the external environment reaches the high-brightness shutter speed limit 821 and is bright, and the user is squinting due to the glare 803. In this embodiment, by changing from the normal gamma curve characteristic data 831 to the high-brightness suppression gamma curve characteristic data 832 in this case, the display of a low-contrast real-world image is prevented, minimizing the impact on product quality. Thus, conventionally, even in situations where the external environment is too bright for the user to check the external environment without removing the HMD device 100, in this embodiment, the exposure can be adjusted, allowing the user to check the external environment while wearing the HMD device 100.

[0086] Figure 9 is a flowchart showing an example of a processing procedure in the control method of the HMD device (image display device) 100 according to the second embodiment.

[0087] First, in step S901 of Figure 9, the gaze imaging unit 202 captures an image of the user's eyes and obtains an image of the user's eyes.

[0088] Next, in step S902 of Figure 9, the pupil detection unit 602 included in the eye organ detection unit 121 detects the pupil of the user's eye using the user's eye image acquired in step S901.

[0089] Next, in step S903 of Figure 9, the eye-opening degree determination unit 122 determines the degree of eye-opening based on the size of the user's pupil detected in step S902. More specifically, the eye-opening degree determination unit 122 determines the degree of eye-opening as follows: normal eye-open state (normal pupil dilation), wide-open state (pupil dilation), and narrowed state (pupil constriction).

[0090] Next, in step S904 of Figure 9, the image correction control unit 601 determines whether the degree to which the user's eyes are wide open (the pupils are dilated) as determined in step S903 is the state of wide-open eyes (a state of dilated pupils).

[0091] In step S904 of Figure 9, if the image correction control unit 601 determines that the degree to which the user's eyes are wide open, as determined in step S903, is not in a state of wide-open eyes (a state of dilated pupils) (S904 / NO), the process proceeds to step S905. When the process proceeds to step S905 in Figure 9, the image correction control unit 601 determines whether the degree to which the user's eyes are wide open, as determined in step S903, indicates that the user is squinting (the pupils are closed).

[0092] In step S905 of Figure 9, if the image correction control unit 601 determines that the degree of eye opening determined in step S903 is not that of squinting (pupils are closed) (S905 / NO), the process proceeds to step S906. In this embodiment, proceeding to step S906 corresponds to the case in step S903 where the degree of eye opening is determined to be that of a normal eye-open state (normal pupil dilation). When the process proceeds to step S906 in Figure 9, the image correction control unit 601 calculates the exposure for the imaging unit 102 based on the determination in step S903 that the degree of eye opening of the user is in a normal eye-open state (normal pupil dilation state).

[0093] Next, in step S907 of Figure 9, the imaging unit 102 performs exposure based on the exposure calculation in step S906.

[0094] Furthermore, in step S904 of Figure 9, if the image correction control unit 601 determines that the degree of eye opening of the user, as determined in step S903, is in a wide-open state (pupil dilated state) (S904 / YES), the process proceeds to step S908. When the process proceeds to step S908 in Figure 9, the image correction control unit 601 calculates the exposure for the imaging unit 102 based on the determination in step S903 that the user's eyes are wide open (pupils are dilated).

[0095] Next, in step S909 of Figure 9, the image correction control unit 601 determines whether or not the ISO sensitivity limit 811, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102 based on the exposure calculation in step S908, has been reached.

[0096] In step S909 of Figure 9, if the image correction control unit 601 determines that the ISO sensitivity limit 811, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102, has not been reached (S909 / NO), the process proceeds to step S910. When the process proceeds to step S910 in Figure 9, the imaging unit 102 performs exposure based on the exposure calculation in step S908.

[0097] Furthermore, in step S909 of Figure 9, if the image correction control unit 601 determines that the ISO sensitivity limit 811, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102, has been reached (S909 / YES), the process proceeds to step S911. When the process proceeds to step S911 in Figure 9, the imaging unit 102 performs exposure based on the exposure calculation in step S908.

[0098] Next, in step S912 of Figure 9, the image correction control unit 601 increases the brightness using the digital gain 812 and sets the image quality settings for the real-world image to the image processing unit 201 so that the real-world image has an appropriate brightness.

[0099] Furthermore, in step S905 of Figure 9, if the image correction control unit 601 determines that the degree of eye opening of the user, as determined in step S903, is in a state of squinting (a state where the pupils are closed) (S905 / YES), the process proceeds to step S913. When the process proceeds to step S913 in Figure 9, the image correction control unit 601 calculates the exposure for the imaging unit 102 based on the determination in step S903 that the user's eyes are narrowed (pupils are closed).

[0100] Next, in step S914 of Figure 9, the image correction control unit 601 determines whether or not the shutter speed limit 821, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102 based on the exposure calculation in step S913, has been reached.

[0101] In step S914 of Figure 9, if the image correction control unit 601 determines that the shutter speed limit 821, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102, has not been reached (S914 / NO), the process proceeds to step S915. When the process proceeds to step S915 in Figure 9, the imaging unit 102 performs exposure based on the exposure calculation in step S913.

[0102] Furthermore, in step S914 of Figure 9, if the image correction control unit 601 determines that the shutter speed limit 821, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102 has been reached (S914 / YES), the process proceeds to step S916. When the process proceeds to step S916 in Figure 9, the imaging unit 102 performs exposure based on the exposure calculation in step S913.

[0103] Next, in step S917 of Figure 9, the image correction control unit 601 suppresses brightness (suppresses high brightness) using the gamma curve characteristic 830 and sets the image quality settings for the real-world image to the image processing unit 201 so that the real-world image has an appropriate brightness.

[0104] If the exposure process in step S907, step S910, or step S915 in Figure 9 is completed, proceed to step S918. Also, if the image quality setting process in step S912 or step S917 in Figure 9 is completed, proceed to step S918. When the process proceeds to step S918 in Figure 9, the image processing unit 201 performs image processing on the real-space image output from the imaging unit 102. At this time, if the image quality setting for the real-space image was set in step S912 or step S917 in Figure 9, the image processing unit 201 performs image processing on the real-space image based on that image quality setting.

[0105] Next, in step S919 of Figure 9, the image synthesis unit 105 first synthesizes the real-world image processed in step S918 with the CG image (virtual space image) generated by the CG generation unit 124 to generate a composite image. Then, the display unit 106 displays the composite image generated by the image synthesis unit 105 as an experience image for the user.

[0106] Next, in step S920 in Figure 9, for example, the HMD system control unit 109 determines whether or not to terminate the display of the experience image on the display unit 106 based on the operation input from the HMD operation unit 115. In step S920 in Figure 9, for example, if the HMD system control unit 109 determines not to terminate the display of the experience image on the display unit 106 (S920 / NO), the process returns to step S901 and repeats the processing from step S901 onwards.

[0107] Furthermore, in step S920 of Figure 9, if, for example, the HMD system control unit 109 determines that it has finished displaying the experience image on the display unit 106 (S920 / YES), the process shown in the flowchart of Figure 9 is terminated.

[0108] As described above, in the HMD device 100 according to the second embodiment, the pupil detection unit 602 detects the pupil of the user's eye using the user's eye image acquired by the gaze imaging unit 202. In addition, in the HMD device 100 according to the second embodiment, the eye-opening degree determination unit 122 determines the degree of eye-opening based on the size of the user's pupil detected by the pupil detection unit 602. Furthermore, in the HMD device 100 according to the second embodiment, the image correction control unit 601 performs the following control when the exposure control limit in the imaging unit 102 is reached. That is, when the exposure control limit in the imaging unit 102 is reached, the image correction control unit 601 performs control to change the apparent brightness of the real-space image on the display unit 106 according to the degree of eye-opening determined by the eye-opening degree determination unit 122. Specifically, the image correction control unit 601 performs the following processing in the first case where the real-world environment surrounding the HMD device 100 is dark enough to reach the low-luminance limit of the exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the eyes are wide open. In this first case, the image correction control unit 601 controls the apparent brightness of the real-world image on the display unit 106 to exceed the low-luminance limit of the exposure control in the imaging unit 102 (the low-luminance limit in the program diagram). More specifically, in the first case described above, the image correction control unit 601 controls the image processing unit 201 to perform correction to brighten the real-world image using the digital gain 812, thereby controlling the apparent brightness of the real-world image on the display unit 106. Furthermore, the image correction control unit 601 performs the following processing in a second case where the real-world environment surrounding the HMD device 100 is bright enough to reach the high-luminance limit of the exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the user is squinting. In this second case, the image correction control unit 601 controls the display unit 106 to darken the apparent brightness of the real-world image by exceeding the high-luminance limit of the exposure control in the imaging unit 102 (the high-luminance limit in the program diagram).More specifically, in the second case described above, the image correction control unit 601 controls the image processing unit 201 to perform correction that darkens the real-space image using the gamma curve characteristic 830, thereby controlling the apparent brightness of the real-space image on the display unit 106 to darken it. According to the configuration of the HMD device 100 in this second embodiment, even in bright or dark environments that exceed the exposure control limit of the imaging unit 102 that captures images of the real space, the user can understand the real space environment by looking at the real space image displayed on the display unit 106. As a result, the user can confirm the real space environment around the HMD device 100 by viewing the real space image displayed on the display unit 106 without removing the HMD device 100, while the HMD device 100 is still attached.

[0109] (Third embodiment) Next, a third embodiment will be described. In the description of the third embodiment below, matters common to the first and second embodiments described above will be omitted, and matters that differ from the first and second embodiments described above will be explained.

[0110] The schematic configuration of the HMD device (image display device) according to the third embodiment is the same as the schematic configuration of the HMD device (image display device) 100 according to the first embodiment shown in Figure 1.

[0111] Figure 10 shows an example of the main functional configuration of the HMD device (image display device) 100 according to the third embodiment. In Figure 10, components similar to those shown in Figures 1, 2, and 6 are denoted by the same reference numerals, and their detailed explanations are omitted. Furthermore, in the following description, the HMD device (image display device) 100 having the functional configuration shown in Figure 10 will be referred to as "HMD device (image display device) 100-3".

[0112] The HMD device 100-3 has the functional configuration of the lens unit 101, imaging unit 102, image synthesis unit 105, display unit 106, position estimation unit 117, eye organ detection unit 121, eye opening degree determination unit 122, CG generation unit 124, image processing unit 201, and gaze imaging unit 202 as shown in Figure 2. Furthermore, the HMD device 100-3 has the functional configuration of the head-mounted control unit 1001. The head-mounted control unit 1001 shown in Figure 10 is configured to include, for example, the imaging control unit 108 and the HMD system control unit 109 shown in Figure 1.

[0113] The head-mounted control unit 1001 performs exposure control, for example, so that the brightness of the real-world image output from the imaging unit 102 is within a certain range. In the third embodiment, this exposure control is performed using the program diagram shown in Figure 4, similar to the first embodiment. The head-mounted control unit 1001 is a control means that, when the exposure control limit in the imaging unit 102 is reached, changes the apparent brightness of the real-world image on the display unit 106 according to the degree of eye opening determined by the eye-opening degree determination unit 122.

[0114] For example, in the first case where the real-world environment surrounding the HMD device 100 is dark enough to reach the low-luminance limit of the exposure control of the imaging unit 102, and the eye-opening degree determination unit 122 determines that the eyes are wide open, the head-mounted control unit 1001 performs the following processing. Specifically, in this first case, the head-mounted control unit 1001 controls the apparent brightness of the real-world image on the display unit 106 to be brighter than the low-luminance limit of the exposure control in the imaging unit 102 (the low-luminance limit in the program diagram). More specifically, in this embodiment, in the first case described above, the head-mounted control unit 1001 controls the brightness of the display unit 106 to be brighter than normal, thereby brightening the apparent brightness of the real-world image on the display unit 106. In this embodiment, even if the real-world environment surrounding the HMD device 100 is a dark environment that cannot be followed by normal exposure control, the user can confirm the environment by viewing the real-world image displayed on the display unit 106 at an appropriate brightness.

[0115] Furthermore, in the second case where the real-world environment surrounding the HMD device 100 is bright enough to reach the high-brightness limit of the exposure control of the imaging unit 102, and the eye-opening degree determination unit 122 determines that the user is squinting, the head-mounted control unit 1001 performs the following processing. Specifically, in this second case, the head-mounted control unit 1001 controls the display unit 106 to dim the apparent brightness of the real-world image by exceeding the high-brightness limit of the exposure control in the imaging unit 102 (the high-brightness limit in the program diagram). More specifically, in this embodiment, in the second case described above, the head-mounted control unit 1001 controls the display unit 106 to dim the brightness of the display unit 106 more than usual, thereby dimming the apparent brightness of the real-world image on the display unit 106. In this embodiment, even if the real-world environment surrounding the HMD device 100 is bright enough that normal exposure control cannot keep up, the user can confirm the environment by viewing the real-world image displayed on the display unit 106 at an appropriate brightness.

[0116] Figure 11 is a flowchart showing an example of a processing procedure in the control method of the HMD device (image display device) 100 according to the third embodiment.

[0117] First, in step S1101 of Figure 11, the gaze imaging unit 202 images the area including the user's eyes and acquires an image of the user's eyes.

[0118] Next, in step S1102 of Figure 11, the eye organ detection unit 121 detects the position of the user's eye organs using the user's eye image acquired in step S1101. Specifically, in this embodiment, the eye organ detection unit 121 detects the positions of the user's upper and lower eyelids.

[0119] Next, in step S1103 of Figure 11, the eye-opening degree determination unit 122 determines the degree of eye-opening of the user based on the distance between the upper and lower eyelids of the user's eyes detected in step S1102. More specifically, the eye-opening degree determination unit 122 determines the degree of eye-opening of the user's eyes based on the distance between the upper and lower eyelids of the user's eyes detected in step S1102, and determines whether the eyes are in a normal open state, a wide-open state, or a squinted state.

[0120] Next, in step S1104 of Figure 11, the head-mounted control unit 1001 determines whether the degree to which the user's eyes are wide open, as determined in step S1103, is in a wide-open state.

[0121] In step S1104 of Figure 11, if the head-mounted control unit 1001 determines that the degree to which the user's eyes are open, as determined in step S1103, is not in a wide-open state (S1104 / NO), the process proceeds to step S1105. When the process proceeds to step S1105 in Figure 11, the head-mounted control unit 1001 determines whether the degree to which the user's eyes are wide open, as determined in step S1103, indicates that the user is squinting.

[0122] In step S1105 of Figure 11, if the head-mounted control unit 1001 determines that the degree of eye opening determined in step S1103 is not that the user is squinting (S1105 / NO), the process proceeds to step S1106. In this embodiment, proceeding to step S1106 corresponds to the case in step S1103 where the degree of eye opening determined to be that the user is in a normal wide-open state. When the process proceeds to step S1106 in Figure 11, the head-mounted control unit 1001 calculates the exposure for the imaging unit 102 based on the determination in step S1103 that the degree to which the user's eyes are open is in a normal open-eye state.

[0123] Next, in step S1107 of Figure 11, the imaging unit 102 performs exposure based on the exposure calculation in step S1106.

[0124] Furthermore, in step S1104 of Figure 11, if the head-mounted control unit 1001 determines that the degree of eye opening of the user, as determined in step S1103, is in a wide-open state (S1104 / YES), the process proceeds to step S1108. When the process proceeds to step S1108 in Figure 11, the head-mounted control unit 1001 calculates the exposure for the imaging unit 102 based on the determination result in step S1103 that the user's eyes are wide open.

[0125] Next, in step S1109 of Figure 11, the head-mounted control unit 1001 determines whether or not the ISO sensitivity limit 411, which is the upper limit of the low-luminance limit for exposure control in the imaging unit 102 based on the exposure calculation in step S1108, has been reached.

[0126] In step S1109 of Figure 11, if the head-mounted control unit 1001 determines that the ISO sensitivity limit 411, which is the upper limit of the low-luminance limit for exposure control of the imaging unit 102, has not been reached (S1109 / NO), the process proceeds to step S1110. When the process proceeds to step S1110 in Figure 11, the imaging unit 102 performs exposure based on the exposure calculation in step S1108.

[0127] Furthermore, in step S1109 of Figure 11, if the head-mounted control unit 1001 determines that the ISO sensitivity limit 411, which is the upper limit of the low-luminance limit for exposure control of the imaging unit 102, has been reached (S1109 / YES), the process proceeds to step S1111. When the process proceeds to step S1111 in Figure 11, the head-mounted control unit 1001 sets the brightness of the display unit 106 to be brighter than normal (increases the brightness above normal).

[0128] Next, in step S1112 of Figure 11, the imaging unit 102 performs exposure based on the exposure calculation in step S1108.

[0129] Furthermore, in step S1105 of Figure 11, if the head-mounted control unit 1001 determines that the degree to which the user's eyes are wide open, as determined in step S1103, indicates that the user is squinting (S1105 / YES), the process proceeds to step S1113. When the process proceeds to step S1113 in Figure 11, the head-mounted control unit 1001 calculates the exposure for the imaging unit 102 based on the determination in step S1103 that the user's eyes are squinting.

[0130] Next, in step S1114 of Figure 11, the head-mounted control unit 1001 determines whether or not the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control in the imaging unit 102 based on the exposure calculation in step S1113, has been reached.

[0131] In step S1114 of Figure 11, if the head-mounted control unit 1001 determines that the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control of the imaging unit 102, has not been reached (S1114 / NO), the process proceeds to step S1115. When the process proceeds to step S1115 in Figure 11, the imaging unit 102 performs exposure based on the exposure calculation in step S1113.

[0132] Furthermore, in step S1114 of Figure 11, if the head-mounted control unit 1001 determines that the shutter speed limit 421, which is the upper limit of the high-brightness limit for exposure control of the imaging unit 102, has been reached (S1114 / YES), the process proceeds to step S1116. When the process proceeds to step S1116 in Figure 11, the head-mounted control unit 1001 sets the brightness of the display unit 106 to be dimmer than normal (reduces the brightness to a lower level than normal).

[0133] Next, in step S1117 of Figure 11, the imaging unit 102 performs exposure based on the exposure calculation in step S1113.

[0134] If the exposure process in step S1107, step S1110, step S1112, step S1115, or step S1117 in Figure 11 is completed, proceed to step S1118. When the process proceeds to step S1118 in Figure 11, the image processing unit 201 performs image processing on the real-space image output from the imaging unit 102.

[0135] Next, in step S1119 of Figure 11, the image synthesis unit 105 first synthesizes the real-world image processed in step S1118 with the CG image (virtual space image) generated by the CG generation unit 124 to generate a composite image. Then, the display unit 106 displays the composite image generated by the image synthesis unit 105 as an experience image for the user. At this time, if the processing in step S1111 or step S1116 has been completed, the display unit 106 displays the experience image with the brightness set in step S1111 or step S1116.

[0136] Next, in step S1120 of Figure 11, for example, the HMD system control unit 109 determines whether or not to terminate the display of the experience image on the display unit 106 based on the operation input of the HMD operation unit 115. In step S1120 of Figure 11, for example, if the HMD system control unit 109 determines not to terminate the display of the experience image on the display unit 106 (S1120 / NO), the process returns to step S1101 and repeats the processing from step S1101 onwards.

[0137] Furthermore, in step S1120 of Figure 11, if, for example, the HMD system control unit 109 determines that it has finished displaying the experience image on the display unit 106 (S1120 / YES), the process of the flowchart shown in Figure 11 is terminated.

[0138] As described above, in the HMD device 100 according to the third embodiment, the eye-opening degree determination unit 122 determines the degree of eye-opening of the user based on the eye image acquired by the gaze-viewing unit 202. Furthermore, in the HMD device 100 according to the third embodiment, the head-mounted control unit 1001 performs the following control when the exposure control limit in the imaging unit 102 is reached. That is, when the exposure control limit in the imaging unit 102 is reached, the head-mounted control unit 1001 controls the apparent brightness of the real-world image on the display unit 106 according to the degree of eye-opening determined by the eye-opening degree determination unit 122. In the first case, when the real-world environment surrounding the HMD device 100 is dark and the low-luminance limit of the exposure control in the imaging unit 102 has been reached, and the eye-opening degree determination unit 122 has determined that the eyes are open, the head-mounted control unit 1001 performs the following processing. In the first case, the head-mounted control unit 1001 controls the apparent brightness of the real-space image on the display unit 106 by exceeding the low-luminance limit of the exposure control in the imaging unit 102 (the low-luminance limit in the program diagram). More specifically, in the first case described above, the head-mounted control unit 1001 controls the brightness of the display unit 106 to be brighter than normal (increases the brightness above normal), thereby brightening the apparent brightness of the real-space image on the display unit 106. In the second case, when the real-space environment surrounding the HMD device 100 is bright enough to reach the high-luminance limit of the exposure control in the imaging unit 102, and the eye-opening degree determination unit 122 determines that the user is squinting, the head-mounted control unit 1001 performs the following processing. In this second case, the head-mounted control unit 1001 controls the apparent brightness of the real-space image on the display unit 106 by exceeding the high-luminance limit of the exposure control in the imaging unit 102 (the high-luminance limit in the program diagram). More specifically, in the second case described above, the head-mounted control unit 1001 controls the brightness of the display unit 106 to be darker than normal (lower the brightness than normal), thereby darkening the apparent brightness of the real-space image on the display unit 106. According to the configuration of the HMD device 100 in this third embodiment, even in bright or dark environments that exceed the exposure control limit of the imaging unit 102 that captures images of the real space, the user can understand the real space environment by looking at the real space image displayed on the display unit 106. As a result, the user can confirm the real space environment around the HMD device 100 by viewing the real space image displayed on the display unit 106 without removing the HMD device 100, while the HMD device 100 is still attached.

[0139] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. This program and a computer-readable storage medium storing said program are included in the present invention.

[0140] The embodiments of the present invention described above are merely examples of how the invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0141] This embodiment includes the following configurations, methods, and programs. [Configuration 1] An image display device comprising a display unit positioned to cover the user's eyes and displaying an experiential image including a real-world image to the user, A first imaging means for capturing images of the real space and acquiring the image of the real space, A second imaging means for capturing an image of the area including the user's eyes and acquiring an eye image, A determination means for determining the degree to which the user's eyes are open, based on the aforementioned eye image, When the exposure control limit of the first imaging means is reached, a control means is provided to change the apparent brightness of the real-space image on the display unit according to the degree of eye opening, An image display device characterized by having the following features. [Configuration 2] The determination means determines the degree to which the eye is wide open based on the distance between the upper and lower eyelids of the eye detected from the eye image. The image display device according to configuration 1, characterized in that it is a display device. [Configuration 3] The determination means determines the degree of eye opening based on the size of the pupil of the eye detected from the eye image. The image display device according to configuration 1, characterized in that it is a display device. [Structure 4] The exposure control limit is the low-luminance limit. The control means, when the low-luminance limit is reached and the determination means determines that the eyes are wide open, performs control to increase the brightness beyond the low-luminance limit. An image display device according to any one of configurations 1 to 3, characterized by the above. [Composition 5] The control means performs control to increase the brightness by setting an ISO sensitivity higher than the low-brightness limit. The image display device according to configuration 4, characterized by the features described above. [Composition 6] The control means brightens the real-world image using digital gain and performs control to increase the brightness. The image display device according to configuration 4, characterized by the features described above. [Composition 7] The control means increases the brightness of the display unit and performs control to increase the brightness. The image display device according to configuration 4, characterized by the features described above. [Structure 8] The limit of the exposure control is the limit on the high-brightness side. The control means, when the high-brightness limit is reached and the determination means determines that the eyes are narrowed as a measure of how wide the eyes are open, performs control to dim the brightness beyond the high-brightness limit. An image display device according to any one of configurations 1 to 7, characterized by the above. [Composition 9] The control means performs control to dim the brightness by setting a shutter speed faster than the high-brightness limit. The image display device according to configuration 8, characterized by the above. [Configuration 10] The control means darkens the real-space image based on the gamma curve characteristics, thereby controlling the brightness. The image display device according to configuration 8, characterized by the above. [Composition 11] The control means performs control to dim the brightness of the display unit. The image display device according to configuration 8, characterized by the above. [Method 1] A control method for an image display device comprising a display unit positioned to cover the user's eyes and displaying an experiential image including a real-world image to the user, A first imaging step involves using a first imaging means to image the real space and acquire the real space image, A second imaging step involves using a second imaging means to image the area including the user's eyes and acquire an eye image; A determination step of determining the degree to which the user's eyes are open based on the aforementioned eye image, A control step is performed to change the apparent brightness of the real-space image on the display unit according to the degree of eye opening when the exposure control limit of the first imaging means is reached, A method for controlling an image display device, characterized by having the following features. [Program 1] A program for causing a computer to perform each step in the method for controlling an image display device described in Method 1. [Explanation of symbols]

[0142] 100: HMD device, 101: Lens unit, 102: Imaging unit, 105: Image synthesis unit, 106: Display unit, 108: Imaging control unit, 117: Position estimation unit, 121: Eye organ detection unit, 122: Eye opening degree determination unit, 124: CG generation unit, 201: Image processing unit, 202: Gaze imaging unit, 601: Image correction control unit, 602: Pupil detection unit, 1001: Head-mounted control unit, E: User's eye

Claims

1. An image display device comprising a display unit positioned to cover the user's eyes and displaying an experiential image including a real-world image to the user, A first imaging means for capturing images of the real space and acquiring the image of the real space, A second imaging means for capturing an image of the area including the user's eyes and acquiring an eye image, A determination means for determining the degree to which the user's eyes are open, based on the aforementioned eye image, When the exposure control limit of the first imaging means is reached, a control means is provided to change the apparent brightness of the real-space image on the display unit according to the degree of eye opening, An image display device characterized by having the following features.

2. The determination means determines the degree to which the eye is wide open based on the distance between the upper and lower eyelids of the eye detected from the eye image. The image display device according to feature 1.

3. The determination means determines the degree of eye opening based on the size of the pupil of the eye detected from the eye image. The image display device according to feature 1.

4. The exposure control limit is the low-luminance limit. The control means, when the low-luminance limit is reached and the determination means determines that the eyes are wide open, performs control to increase the brightness beyond the low-luminance limit. The image display device according to feature 1.

5. The control means performs control to increase the brightness by setting an ISO sensitivity higher than the low-brightness limit. The image display device according to feature 4.

6. The control means brightens the real-world image using digital gain and performs control to increase the brightness. The image display device according to feature 4.

7. The control means increases the brightness of the display unit and performs control to increase the brightness. The image display device according to feature 4.

8. The limit of the exposure control is the limit on the high-brightness side. The control means, when the high-brightness limit is reached and the determination means determines that the eyes are narrowed as a measure of how wide the eyes are open, performs control to dim the brightness beyond the high-brightness limit. The image display device according to feature 1.

9. The control means performs control to dim the brightness by setting a shutter speed faster than the high-brightness limit. The image display device according to feature 8.

10. The control means darkens the real-space image based on the gamma curve characteristics, thereby controlling the brightness. The image display device according to feature 8.

11. The control means performs control to dim the brightness of the display unit. The image display device according to feature 8.

12. A control method for an image display device comprising a display unit positioned to cover the user's eyes and displaying an experiential image including a real-world image to the user, A first imaging step involves using a first imaging means to image the real space and acquire the real space image, A second imaging step involves using a second imaging means to image the area including the user's eyes and to acquire an eye image; A determination step of determining the degree to which the user's eyes are open based on the aforementioned eye image, A control step is performed to change the apparent brightness of the real-space image on the display unit according to the degree of eye opening when the exposure control limit of the first imaging means is reached. A method for controlling an image display device, characterized by having the following features.

13. A program for causing a computer to perform each step in the control method for an image display device described in claim 12.

Images