Video display device and control method thereof

Abstract

To provide a smaller and lighter video display device that allows diopter adjustment according to point of fixation.SOLUTION: A video display device 101 comprises: first and second video display part 201a, 201b displaying videos corresponding respectively to a right eye and a left eye of a user; and first and second display optical systems 202a, 202b. First line of sight detection parts 203a, 204a and second line of sight detection parts 203b, 204b detect the line of sight of the user without the first and second display optical system 202a, 202b. The video display device 101 determines a position of a point of fixation of the user from the line of sight detection result, and calculates diopter adjustment amount corresponding to video parallax on the position of the point of fixation. First and second diopter adjustment parts 205a, 205b perform, on the basis of the diopter adjustment amount, diopter adjustment related to the first and second video display part 201a, 201b.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for presenting images by directing different images to the left and right eyes of a user. [Background technology]

[0002] An image display device with a function for detecting a user's line of sight can adjust the diopter by calculating the diopter adjustment amount corresponding to the parallax of the gaze point position. In Patent Document 1, the gaze detection is performed via a display optical system using a prism. A special prism is employed to prevent changes in optical power due to diopter adjustment from affecting the gaze detection. In addition, a configuration is employed that adjusts the diopter according to the user's gaze point, which reduces the sense of discomfort during stereoscopic viewing and can suppress motion sickness during viewing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 8-234141 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technology, the use of prisms results in large and heavy optical systems. When a user wears a video display device on their head, such as a head-mounted display, it is difficult to wear a device with a large optical system due to its weight. Even if it were possible to wear it, it would cause significant fatigue for the user, making it difficult to view for medium to long periods of time. An object of the present invention is to provide a smaller and lighter image display device that allows for diopter adjustment according to the position of the gaze point. [Means for solving the problem]

[0005] An image display device according to an embodiment of the present invention includes first and second image display units that display first and second images to a user's right and left eyes, respectively; first and second display optical systems that correspond to the first and second image display units, respectively; first and second gaze detection means that perform gaze detection for the first and second image display units, respectively, without using the first and second display optical systems; a calculation means that calculates a diopter adjustment amount corresponding to the parallax between the first image and the second image at the position of the gaze point obtained from the detection results of the first and second gaze detection means; and first and second diopter adjustment means that perform diopter adjustment for the first and second image display units in accordance with the diopter adjustment amount. first and second optical path changing means disposed on the user side relative to the first and second display optical systems, respectively; the first and second line-of-sight detection means each include an infrared illumination unit and a camera, and the camera of the first line-of-sight detection means and the camera of the second line-of-sight detection means are disposed in a region between an optical axis of the first display optical system and an optical axis of the second display optical system in a direction in which the first and second image display units are aligned. an optical axis of a camera provided in the first line-of-sight detection means is inclined at a predetermined angle with respect to the optical axis of the first display optical system, and an optical axis of a camera provided in the second line-of-sight detection means is inclined at a predetermined angle with respect to the optical axis of the second display optical system; the first line-of-sight detection means performs line-of-sight detection via the first optical path changing means; the second line-of-sight detection means performs line-of-sight detection via the second optical path changing means; the first and second display optical systems are constituted by flat optical elements whose focal lengths can be changed by an electric signal; the first and second display optical systems also serve as the first and second diopter adjustment means, respectively; the optical elements constituting the first display optical system are arranged in a region between the first optical path changing means and the first image display unit in the direction of the optical axis of the first display optical system; and the optical elements constituting the second display optical system are arranged in a region between the second optical path changing means and the second image display unit in the direction of the optical axis of the second display optical system. do. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide a smaller and lighter image display device that can adjust the visibility according to the position of the gaze point. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing the appearance of a video display device according to an embodiment of the present invention; [Figure 2] 1 is a cross-sectional view of an image display device according to a first embodiment, as viewed from above. [Figure 3] 1A and 1B are schematic diagrams illustrating the convergence angle of the eyeballs and the diopter during gaze. [Figure 4] 1 is a block diagram showing the overall system configuration of a video display device. [Figure 5] 10 is a flowchart illustrating operations from line-of-sight detection to diopter adjustment. [Figure 6] FIG. 10 is a cross-sectional view of an image display device according to a second embodiment, as viewed from above. [Figure 7] FIG. 10 is a cross-sectional view of an image display device according to a third embodiment, as viewed from above. DETAILED DESCRIPTION OF THE INVENTION

[0008] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings. In the embodiment, a head-mounted display is shown as an example of an image display device in which a pair of images having parallax are displayed on a plurality of image display units, respectively, and stereoscopic display is possible via a plurality of display optical systems.

[0009] [First Example] 1 is a schematic diagram showing the appearance of an image display device according to this embodiment. Image display device 101 has a belt part 101a for being worn on a user's head 102. The user observes images displayed by image display device 101 with right eye 103a and left eye 103b.

[0010] An image display device according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a schematic cross-sectional view of image display device 101 as viewed from above. In Fig. 2, the reference numerals of components corresponding to the user's right eye 103a are marked with an a, and the reference numerals of components corresponding to the user's left eye 103b are marked with a b.

[0011] The image display unit is composed of an image display unit 201a for the right eye and an image display unit 201b for the left eye. The image display unit is composed of a liquid crystal display, an organic EL (Electro Luminescence) display, or the like. The display optical system 202a is an optical system that guides the image on the image display unit 201a to the right eye 103a, and its optical axis is denoted as La. The display optical system 202b is an optical system that guides the image on the image display unit 201b to the left eye 103b, and its optical axis is denoted as Lb. For simplicity, both the display optical systems 202a and 202b in Figure 2 are shown as single lenses, but there are various optical systems such as optical systems with multiple lenses and optical systems that are thinned using Fresnel elements.

[0012] The gaze detection camera 203a detects the gaze of the right eye 103a, and its optical axis is denoted as LGa. The gaze detection camera 203b detects the gaze of the left eye 103b, and its optical axis is denoted as LGb. The gaze detection cameras 203a and 203b are arranged near the sides of the display optical systems 202a and 202b, respectively, and can capture the user's gaze without using the display optical systems 202a and 202b. The optical axis LGa of the gaze detection camera 203a is tilted at an angle θ1a with respect to the optical axis La of the display optical system 202a, which displays an image directly in front of the right eye 103a. Similarly, the optical axis LGb of the gaze detection camera 203b is tilted at an angle θ1b with respect to the optical axis Lb of the display optical system 202b, which displays an image directly in front of the left eye 103b. In this way, the gaze detection cameras 203a and 203b detect the gaze without using the display optical systems 202a and 202b. Therefore, unlike the conventional configuration, the display optical systems 202a and 202b have a simple configuration, and a lighter and smaller display optical system can be realized.

[0013] The infrared illumination unit 204a illuminates the gaze detection camera 203a at the right eye 103a, forming a corneal reflection image. Similarly, the infrared illumination unit 204b illuminates the gaze detection camera 203b at the left eye 103b, forming a corneal reflection image. An infrared LED (light-emitting diode) or the like is used for the infrared illumination units 204a and 204b. Although only one infrared illumination unit 204a and one infrared illumination unit 204b are shown in FIG. 2, in reality, multiple units are arranged to surround the display optical systems 202a and 202b, respectively.

[0014] In this way, the gaze detection cameras and infrared lighting units 204a and 204b can detect the gaze of the user's right and left eyes. In other words, the gaze detection cameras 203a and 203b and the infrared lighting units 204a and 204b constitute a gaze detection means.

[0015] The gaze detection method uses a well-known method that uses an image of the pupil of the eyeball and an image of the corneal reflection captured by infrared illumination, so a detailed description of this method will be omitted. Furthermore, the gaze detection means of this embodiment captures an image of the eyeball from a position tilted relative to the user's eye. Therefore, perspective distortion occurs in the pupil and corneal reflection images captured by the gaze detection camera, but by correcting this, the gaze can be detected accurately.

[0016] Diopter adjustment unit 205a adjusts the diopter by driving image display unit 201a in the direction of optical axis La. Diopter adjustment unit 205b adjusts the diopter by driving image display unit 201b in the direction of optical axis Lb. Diopter adjustment involves adjusting the power to match the visual acuity of the user's left and right eyes. Diopter adjustment units 205a and 205b have motors capable of linear drive. There are no limitations on the motor drive method, but since the motors are located near the user's ears when the image display device 101 is in use, a quiet motor is preferred; for example, an ultrasonic motor that uses friction or a voice coil motor that uses electromagnetic force may be used.

[0017] The diopter adjustment units 205a and 205b perform diopter adjustment based on the diopter adjustment amount at the position of the gaze point determined from the detection result by the line-of-sight detection means. The diopter is adjusted by changing the distance between the image display units 201a and 201b and the display optical systems 202a and 202b. The diopter adjustment is performed in real time every time the gaze point is changed, thereby reducing the user's dizziness and fatigue when watching videos. Details of this will be explained using FIG. 3.

[0018] FIG. 3 is a schematic diagram illustrating the convergence angle and diopter of the eyes during gaze. FIG. 3(A) shows the state of gaze in the real world, and FIG. 3(B) shows the state of gaze on the image display device 101. In FIG. 3(A), the observer is gazing at object point 301 located at a distance A1 in the real world. The line of sight Ga1 of the observer's right eye 103a and the line of sight Gb1 of the observer's left eye intersect at object point 301, forming an angle B1. Angle B1 corresponds to the convergence angle when the observer gazes at object point 301. At this time, the crystalline lens of the observer's eye changes so that the focus is adjusted to the distance A1, and as shown by light rays Da1 and Db1 in the figure, light rays emitted from object point 301 converge at approximately one point on the retina in the eye. In other words, in the real world, the distance corresponding to the convergence angle B1 and the distance A1 corresponding to the diopter always coincide with each other.

[0019] On the other hand, in FIG. 3(B), the observer is gazing at an object point 301 located at a distance A1 away on the image display device 101. The convergence angle B1 is the same as in the real world case shown in FIG. 3(A). The virtual image position 302 is the virtual image position when the observer observes the first image display unit 201a and the second image display unit 201b through the first display optical system 202a and the second display optical system 202b, respectively. The virtual image position 302 is located at a distance A2 (>distance A1) away from the eyes 103a and 103b. Regarding the diopter of the eyeball, the crystalline lens changes so that the focus is at the distance A2, and as shown by light rays Da2 and Db2 in the figure, light rays emitted from the virtual image position 302 converge at approximately one point on the retina inside the eyeball. This is the principle of stereoscopic vision by image display device 101, and by providing a convergence angle B1 with respect to virtual image position 302, the viewer visually perceives an object as if it were located at point 301. However, unlike the real world, in image display device 101, the distance corresponding to convergence angle B1 does not match the distance A2 corresponding to diopter adjustment. This is the so-called contradiction between convergence and diopter adjustment, and is the main cause of motion sickness and fatigue in the viewer.

[0020] Therefore, in this embodiment, as shown in FIG. 2, by changing the distance between the image display units 201a and 201b and the display optical systems 202a and 202b, the virtual image position 302 in FIG. 3B can be changed. For example, assume that the gaze detection means detects that the observer is gazing at point 301. The distance between the image display unit and the display optical system is changed so that the virtual image position 302, which is at distance A2, becomes distance A1. As a result, as shown in FIG. 3A, the distance corresponding to the convergence angle and the distance corresponding to the diopter coincide, allowing for more natural viewing and simultaneously reducing motion sickness and fatigue. Note that, when changing the distance between the image display unit and the display optical system, it is sufficient to drive at least one of the image display unit and the display optical system. That is, there are embodiments in which the image display unit or the display optical system is driven, and embodiments in which both the image display unit and the display optical system are driven.

[0021] Next, the operation of the image display device 101 will be described with reference to Figures 4 and 5. Figure 4 is a block diagram showing the overall system configuration of the image display device 101 of this embodiment. The arithmetic processing unit 401 of the image display device 101 includes a CPU (Central Processing Unit) and controls the overall system. The arithmetic processing unit 401 is connected to image display units 201a and 201b, gaze detection cameras 203a and 203b, infrared illumination units 204a and 204b, and diopter adjustment units 205a and 205b.

[0022] The video display device 101 includes a power supply 402, a wireless unit 403, and speakers 404a and 404b. By connecting to a network via the wireless unit 403, the user can view video content and the like on the video display device 101.

[0023] The operation from gaze detection to diopter adjustment will be described with reference to the flowchart of Fig. 5. A program corresponding to the flowchart of Fig. 5 is stored in a memory unit within the arithmetic processing unit 401. First, in S501, gaze detection is performed on each of the user's right eye 103a and left eye 103b using the gaze detection cameras 203a and 203b and the infrared illumination units 204a and 204b.

[0024] Next, in S502, the arithmetic processing unit 401 executes a process of determining the point of gaze from the line of sight of the right eye 103a and the line of sight of the left eye 103b. The calculation of the point of gaze uses an average value of the line of sight of the right eye and the left eye. In S503, the arithmetic processing unit 401 executes a process of calculating the diopter adjustment amount at the position of the point of gaze of the image displayed on the image display units 201a and 201b. Specifically, the parallax between the image at the position of the point of gaze of the right eye and the image at the position of the point of gaze of the left eye is calculated, and the diopter adjustment amount is derived from this parallax. For example, the image display device 101 has a data table that associates parallax with diopter adjustment amount, and can calculate the diopter adjustment amount corresponding to the parallax. Alternatively, the arithmetic processing unit 401 can calculate the diopter adjustment amount from the parallax using a mathematical formula that expresses the relationship between parallax and diopter adjustment amount.

[0025] In S504, the motors constituting the diopter adjustment units 205a and 205b are driven based on the calculated diopter adjustment amount. As a result, an operation is performed to change the distance between the image display units 201a and 201b and the display optical systems 202a and 202b. The above operation is performed every time the gaze point is changed, so it is possible to change the diopter by adjusting the diopter in real time.

[0026] The human gaze includes high-frequency micromovements known as fixational eye movements. S501 enables smooth gaze detection by removing the micromovement components. In this embodiment, the diopter adjustment units 205a and 205b can be driven independently for the right and left eyes, making it possible to accommodate users with different visual acuity in each eye. Furthermore, the gaze detection cameras 203a and 203b perform gaze detection without using the display optical systems 202a and 202b. This prevents the display optical system from becoming too large, and enables gaze detection that is less affected by diopter adjustments related to the display optical system to be achieved with a lightweight and compact configuration.

[0027] According to this embodiment, it is possible to provide a smaller and lighter image display device while realizing visibility adjustment according to the position of the gaze point.

[0028] [Second Example] With reference to FIG. 6, an image display device according to a second embodiment of the present invention will be described. In this embodiment, an example is shown in which a reflective member that serves as an optical path changing means is disposed between the display optical system and the eye in order to reduce the angle formed between the optical axis of the display optical system and the optical axis of the gaze detection camera. This configuration enables gaze detection with higher accuracy. Note that by using the same reference numerals as in the first embodiment, detailed explanations of these will be omitted and differences will be mainly explained. This method of omitting explanations will be the same in the embodiments described below.

[0029] 6 is a schematic cross-sectional view of the image display device 101 of this embodiment as seen from above. Reflecting members 601a and 601b are disposed between the user's eyes 103a and 103b and the display optical systems 202a and 202b, respectively, and function as optical path changing means.

[0030] 6, the optical axis of the optical system of gaze detection camera 203a relative to the user's right eye 103a is indicated by LGa2. The optical axis of the optical system of gaze detection camera 203b relative to the user's left eye 103b is indicated by LGb2. Gaze detection cameras 203a and 203b are disposed closer to the user than the sides of display optical systems 202a and 202b, respectively, and can capture the user's gaze via reflecting members 601a and 601b. Similar to the first embodiment, gaze detection cameras 203a and 203b can capture the user's gaze without the intervention of display optical systems 202a and 202b.

[0031] In this embodiment, the angle between the optical axis LGa2 and the optical axis La is denoted as θ2a, and the angle between the optical axis LGb2 and the optical axis Lb is denoted as θ2b. The gaze detection camera 203a captures the eye 103a from a position tilted at an angle θ2a with respect to the optical axis La. Similarly, the gaze detection camera 203b captures the eye 103b from a position tilted at an angle θ2b with respect to the optical axis Lb. The angles θ2a and θ2b are significantly smaller than the angles θ1a and θ1b shown in the first embodiment. Therefore, the perspective distortion occurring in the pupil and corneal reflection image captured by the gaze detection camera is significantly reduced. This allows for the acquisition of an image closer to the eyeball as viewed from the front, enabling more accurate gaze detection. In particular, when the eyeball rotates away from the gaze detection camera 203a or 203b, the perspective distortion of the pupil becomes significant. The configuration including the reflecting members 601a and 601b as in this embodiment is highly effective in improving the accuracy of line of sight detection.

[0032] The gaze detection cameras 203a and 203b perform gaze detection using infrared images generated by infrared illuminators 204a and 204b. The reflecting members 601a and 601b have the property of reflecting infrared light and transmitting visible light. This allows images to be presented to the user while suppressing light loss in the image display units 201a and 201b. Also, in this embodiment, the gaze detection cameras 203a and 203b perform gaze detection without using the display optical systems 202a and 202b, thereby achieving the same effects as in the first embodiment.

[0033] [Third Example] An image display device according to a third embodiment of the present invention will be described with reference to Fig. 7. Differences from the second embodiment will be explained below. In this embodiment, a liquid crystal optical element capable of changing the focus by electrical switching is used in the display optical system. With this configuration, the display optical system itself also serves as a diopter adjustment means, which is effective in reducing the size of the image display device.

[0034] 7 is a schematic cross-sectional view of the image display device 101 of this embodiment as seen from the top. Optical elements 701a and 701b constitute a display optical system, and the focal length can be changed by changing the orientation characteristics based on liquid crystal switching. Such optical elements include elements using a Fresnel structure and elements incorporating polarization characteristics.

[0035] The optical elements 701a and 701b that make up the display optical system not only guide the images on the image display units 201a and 201b to the user's eyes, but also serve as diopter adjustment means. Therefore, a linear drive motor that changes the distance between the image display units 201a and 201b and the optical elements 701a and 701b is not required, making it possible to further miniaturize the image display device 101. In this embodiment, the gaze detection cameras 203a and 203b perform gaze detection without using the optical elements 701a and 701b.

[0036] According to this embodiment, in addition to the effects of the second embodiment, a more compact image display device 101 can be realized by using flat optical elements 701a and 701b whose focal length can be changed by an electric signal.

[0037] In the above embodiment, the optical system does not become large as in the conventional example, but has a lighter and smaller configuration, and can achieve gaze detection without being affected by diopter adjustment of the display optical system. In other words, it is possible to achieve size and weight reduction while achieving diopter adjustment according to the position of the gaze point. Although the preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the spirit and scope of the present invention.

[0038] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0039] 101 Video display device 201a, 201b Video display section 202a,202b Display optical system 203a, 203b, 204a, 204b Gaze detection unit 205a,205b Diopter adjustment section 601a, 601b Reflecting member (optical path changing means) 701a, 701b Optical elements

Claims

1. first and second image display units for displaying first and second images to the right and left eyes of a user, respectively; first and second display optical systems corresponding to the first and second image display units, respectively; first and second line-of-sight detection means for respectively performing line-of-sight detection for the first and second image display units without using the first and second display optical systems; a calculation means for calculating a diopter adjustment amount corresponding to a parallax between the first image and the second image at a position of a gaze point acquired from detection results of the first and second gaze detection means; first and second diopter adjustment means for adjusting the diopter of the first and second image display units in accordance with the diopter adjustment amount; first and second optical path changing means disposed on the user side relative to the first and second display optical systems, respectively; the first and second line-of-sight detection means each include an infrared illuminator and a camera; a camera included in the first line-of-sight detection means and a camera included in the second line-of-sight detection means are disposed in a region between an optical axis of the first display optical system and an optical axis of the second display optical system in a direction in which the first and second image display units are aligned, the optical axis of the camera included in the first line of sight detection means is inclined at a predetermined angle with respect to the optical axis of the first display optical system, the optical axis of the camera of the second line of sight detection means is inclined at a predetermined angle with respect to the optical axis of the second display optical system, the first line-of-sight detection means performs line-of-sight detection via the first optical path changing means, the second line of sight detection means performs line of sight detection via the second optical path changing means, the first and second display optical systems are configured with flat optical elements whose focal lengths can be changed by electrical signals; the first and second display optical systems also function as the first and second diopter adjustment means, respectively; an optical element constituting the first display optical system is disposed in a region between the first optical path changing means and the first image display unit in a direction of an optical axis of the first display optical system; The optical elements constituting the second display optical system are disposed in a region between the second optical path changing means and the second image display unit in the direction of the optical axis of the second display optical system. A video display device characterized by:

2. The first and second image display units respectively display the first and second images having parallax, and perform stereoscopic display via the first and second display optical systems.

2. The image display device according to claim 1.

3. The first and second optical path changing means have the property of reflecting infrared light and transmitting visible light.

3. The image display device according to claim 1 or 2.

4. The first and second line-of-sight detection means each include a plurality of the infrared illumination units.

4. The image display device according to claim 1, wherein the image display device is a display device for displaying an image on a display screen.

5. The camera included in the first line-of-sight detection means and the camera included in the second line-of-sight detection means are disposed in a region between the first and second image display units in the direction in which the first and second image display units are arranged.

5. The image display device according to claim 1, wherein the image display device is a display device for displaying an image on a display screen.

6. Can be worn on the user's head 6. The image display device according to claim 1, wherein the image display device is a display device for displaying an image on a display screen.

7. a first and second image display unit for displaying a first image and a second image to a right eye and a left eye of a user, respectively; a first and a second display optical system corresponding to the first and second image display unit, respectively; a first and a second line-of-sight detection means for performing line-of-sight detection for the first and second image display unit, respectively; and a first and a second optical path changing means arranged on the user side of the first and second display optical systems, wherein the first and second line-of-sight detection means each include an infrared illumination unit and a camera, and the camera of the first line-of-sight detection means and the camera of the second line-of-sight detection means are arranged in an area between an optical axis of the first display optical system and an optical axis of the second display optical system in a direction in which the first and second image display units are aligned, and the optical axis of the camera of the first line-of-sight detection means is inclined at a predetermined angle with respect to the optical axis of the first display optical system, an optical axis of a camera provided in the line-of-sight detection means is tilted at a predetermined angle with respect to the optical axis of the second display optical system, the first line-of-sight detection means performs line-of-sight detection via the first optical path changing means, the second line-of-sight detection means performs line-of-sight detection via the second optical path changing means, the first and second display optical systems are constituted by flat optical elements whose focal lengths can be changed by an electric signal, the first and second display optical systems also serve as first and second diopter adjustment means, respectively, the optical elements constituting the first display optical system are arranged in a region between the first optical path changing means and the first image display unit in the direction of the optical axis of the first display optical system, and the optical elements constituting the second display optical system are arranged in a region between the second optical path changing means and the second image display unit in the direction of the optical axis of the second display optical system, a gaze detection step of performing gaze detection for the first and second image display units, respectively, without using the first and second display optical systems; a calculation step of calculating a diopter adjustment amount corresponding to a parallax between the first image and the second image at a position of a gaze point acquired from a detection result of the line-of-sight detection step; a diopter adjustment step of adjusting the diopter of the first and second image display units by the first and second diopter adjustment means in accordance with the diopter adjustment amount. A control method for a video display device.

Images