Image display device

Abstract

[Problem] To acquire a high-definition captured image and display a natural and high-quality composite image in an image display device. [Solution] This image display device includes: an imaging device that generates a captured image by imaging an outside world through an imaging optical system 103; a display element 106 that displays a composite image of the captured image and a virtual image; a display optical system 107 that guides light from the display element to an eye of an observer; an acquisition means 108 that acquires a gaze distance that is a distance to a gaze point of the observer in the outside world; and a control means 205 that controls focusing of the imaging device according to the gaze distance. The pupil diameter of the imaging optical system is 1.3-4.0 mm inclusive.

Description

Image display device

[0001] The present invention relates to an image display device that displays images using a video camera and a display element.

[0002] Mixed reality (MR) technology and augmented reality (AR) technology display a composite image that combines a real image, which is an image of the outside world captured by a video camera, with a virtual image created by computer graphics (CG) or the like. Examples of video see-through image display devices that perform such a display include those disclosed in Patent Documents 1 and 2.

[0003] Japanese Patent Application Laid-Open No. 2001-133725 U.S. Patent No. 10,733,800

[0004] In a video see-through display device, in order to display a high-quality composite image that is natural to humans, it is necessary to increase the resolution of not only the virtual image but also the captured image. However, Patent Documents 1 and 2 do not disclose a specific method for acquiring such high-resolution captured image.

[0005] The present invention provides an image display device that can acquire high-definition captured images and display natural, high-quality composite images.

[0006] An image display device according to one aspect of the present invention includes an imaging device that generates an image by capturing an image of the outside world through an imaging optical system, a display element that displays a composite image of the captured image and a virtual image, a display optical system that directs light from the display element to an observer's eye, an acquisition means that acquires a gaze distance that is the distance to the observer's gaze point in the outside world, and a control means that controls focusing of the imaging device in accordance with the gaze distance, wherein the pupil diameter of the imaging optical system is 1.3 mm or more and 4.0 mm or less.

[0007] According to the present invention, it is possible to acquire a high-definition captured image and display a natural, high-quality composite image.

[0008] 1 is a diagram showing the configuration of an image display device according to the first embodiment. FIG. 1 is a diagram showing the relationship between the convergence angle and the distance to the point of gaze. FIG. 2 is a diagram showing focusing in the first embodiment. FIG. 3 is another diagram showing focusing in the first embodiment. FIG. 4 is a diagram showing the relationship between the pupil diameter of the imaging optical system and the best focus MTF. FIG. 5 is a diagram showing the relationship between the pupil diameter of the imaging optical system and the depth of field. FIG. 6 is a diagram showing the MTF of the imaging optical system according to the first embodiment. FIG. 7 is a diagram showing a real image according to the first embodiment. FIG. 8 is another diagram showing a real image according to the first embodiment. FIG. 9 is a diagram showing a composite image according to the first embodiment. FIG. 10 is a diagram showing the relationship between the focal length of the imaging optical system and the object distance as camera parameters according to the first embodiment. FIG. 11 is a diagram showing the relationship between the object distance and the number of pixels of the image sensor according to the first embodiment. FIG. 12 is a diagram showing the MTF of the imaging optical system according to the second embodiment. FIG. 13 is a diagram showing a real image according to the second embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] Fig. 1 shows the configuration of a video see-through type HMD (Head Mounted Display) 100 as an image display device according to Example 1. Fig. 1 is a vertical cross-sectional view, in which the +Z axis is defined as the direction of the visual axis of an observer's eye 102, and the X axis (horizontal direction) and Y axis (vertical direction) relative to the Z axis are defined in right-handed coordinates.

[0011] The imaging device 101 has an imaging optical system 103, an imaging element 104, and a focus drive unit 201, and generates a captured image as image data by capturing an image of an object 105 in the external world (real world) through the imaging optical system 103. The imaging element 104 is a photoelectric conversion element that photoelectrically converts (captures) the external world image formed by the imaging optical system 103, and is configured with a CCD sensor, a CMOS sensor, or the like. Note that the distance L from the pupil of the imaging optical system 103 to the object 105 is ２ is the object distance.

[0012] The display optical system 107 directs light from a display element 106 that displays an image (video) to the eye 102, which is positioned at the pupil (exit pupil) of the display optical system 107, thereby presenting an enlarged virtual image of the displayed image to the viewer. As the display optical system, not only a refractive system but also a reflective system using polarized light can be used. The display optical system 107 in this embodiment has a horizontal angle of view of 80° and a vertical angle of view of 70°. The display element 106 is configured using an organic EL element, a liquid crystal display, or the like. The optical axis of the display optical system 107 and the optical axis of the imaging optical system 103 coincide with each other in both the YZ cross section and the XZ cross section of FIG. 1 . In the following description, the direction in which the optical axis extends (the Z-axis direction) is referred to as the optical axis direction.

[0013] Note that FIG. 1 shows the display element 106 and the display optical system 107 provided for one of the observer's eyes, but in reality, the display element 106 and the display optical system 107 are also provided for the other eye.

[0014] The pupil of the imaging optical system 103 and the pupil of the display optical system 107 where the viewer's eye 102 is located are separated by a distance L in the optical axis direction (Z axis direction). １ In this embodiment, the distance L １ = 35 mm. １ If is too large, the difference from the actual human appearance becomes too large. １ It is desirable that the distance be 60 mm or less.

[0015] In this embodiment, the image pickup device 101 includes the image pickup optical system 103 and the image pickup element 104, and the image pickup device 101 can pick up an image at an image pickup angle θ ２ The captured image corresponding to the angle of view θ ２ and the angle of view θ of the display optical system １ and are consistent with each other.

[0016] The infrared camera 108 serving as a gaze detection sensor captures an image of the eyeball illuminated with infrared light from the infrared light source 109. The controller 205 serving as control means is composed of a CPU and the like, analyzes the eyeball image from the infrared camera 108 to detect the gaze of the eye 102 (i.e., the gaze direction of the observer), and further calculates (acquires) the gaze distance of the observer from the gaze, as will be described later. The infrared camera 108, the infrared light source 109, and the controller 205 constitute an acquisition means.

[0017] Furthermore, the controller 205 controls the driving of the image sensor 104 of the image pickup device 101 and the focus driver 201 .

[0018] The image processing circuit 111, which serves as a generating means, generates a composite image by combining the captured image generated by the imaging device 101 with a virtual image created by CG or the like. At this time, the controller 205 transmits information on the viewer's gaze direction and gaze distance, as well as information on the posture of the HMD 100 (movement of the viewer's head) detected by a posture sensor (not shown), to the image processing circuit 111. The image processing circuit 111 may be built into the HMD 100, or may be provided outside the HMD 100 and connected to the HMD 100 via wired or wireless communication.

[0019] 2 shows a state in which a person is gazing at a gaze point 110 (object 105 in FIG. 1) in the outside world with both eyes (right eye 102R and left eye 102L) as viewed from above. The lines of sight of the right eye 102R and the left eye 102L looking at the gaze point 110 converge at the gaze point 110 with a convergence angle θ ３ The gaze distance L, which is the distance from both eyes to the gaze point 110, ３ , vergence angle θ ３ and an interocular distance I, which is the distance between the right eye 102R and the left eye 102L, have the relationship shown in the following formula (1).

[0020]

[0021] Detects the line of sight and calculates the convergence angle θ ３ By calculating the gaze distance L from Equation (1), ３In the case of the configuration of this embodiment shown in FIG. 1, the observer observes the object 105 through an image captured with the optical axis coincident, so that the convergence angle of the observer is calculated to calculate L ２ In this case, the value of I may be the average human eye distance, or I may be calculated for each observer using the detected line of sight.

[0022] 3 and 4 show focusing of the imaging device 101. The focus driver 201 is an actuator such as a voice coil motor, and moves at least a part of the imaging optical system 103 or the imaging element 104 in the optical axis direction to focus on the object 105. In this embodiment, a case where focusing is performed by moving the entire imaging optical system 103 will be described.

[0023] 3 shows a state in which focusing is performed on an object 105 located at a distance, and in this state, the imaging optical system 103 is moved so as to shorten the distance between it and the image sensor 104. FIG. 4 shows a state in which focusing is performed on an object 105 located at a close distance, and in this state, the imaging optical system 103 is moved so as to lengthen the distance between it and the image sensor 104 compared to the state in FIG. 3. The controller 205 controls the line of sight (i.e., the convergence angle θ) of the right eye 102R and the left eye 102L as described above. ３ ) The gaze distance L calculated from ３ The focus driver 201 is controlled in response to the signal to perform focusing on the object 105 .

[0024] The relationship between the resolution of a real image as a captured image and the pupil diameter of the imaging optical system 103 will be described below. Here, the pupil diameter refers to the entrance pupil diameter of the imaging optical system 103. Here, various characteristics of the imaging device 101 are converted into cycles per degree (CPD), which is the cycle per angle on the object side to be captured, rather than into line pairs per mm, which is the scale on the imaging surface of the image sensor 104. This is because the captured image reproduces the real world as seen by an observer (human). When the focal length of the imaging optical system 103 is f (mm) and the pixel pitch of the image sensor 104 is p (mm), the maximum CPDmax that can be sampled is expressed by the angle that the pixel pitch p forms per object side angle, as shown in the following equation (2):

[0025]

[0026] When converted to an evaluation using the Landolt ring, which is an evaluation index for human visual acuity, it can be said that a 1-minute resolution of 1 pixel is capable of displaying a pattern equivalent to visual acuity of 1.0. For this reason, a CPDmax of 30 is considered to be one target for expressing a natural human appearance. Furthermore, although the MTF of the imaging optical system 103 is usually evaluated in LP / mm units on the image side, it can also be expressed in terms of CPD, and in the paraxial region, it is expressed as in the following equation (3):

[0027]

[0028] From equation (3), when the MTF characteristics of the imaging optical system 103, where the focal length is f (mm) and the pupil diameter is D (mm), i.e., the F-number is f / D, are organized in CPD units, it can be seen that the MTF characteristics of an ideal, aberration-free imaging optical system 103 are determined not by the F-number but by the pupil diameter D. Fig. 5 shows the relationship between the pupil diameter D and the MTF at the best image plane (best focus position), and indicates that the larger the pupil diameter D, the higher the MTF can be secured even in a high CPD region.

[0029] On the other hand, the depth of field becomes shallower as the pupil diameter D increases. When considering using the imaging optical system 103 in a pan-focus mode without focusing, a target MTF of approximately 0.3 must be ensured within a depth range of ±1.5 diopters (1 / m) from the reference object distance. Figure 6 shows the relationship between the pupil diameter D and the maximum CPD at which an MTF of 0.3 can be ensured within a depth range of ±1.5 diopters from the reference object distance. As can be seen from Figure 6, if the pupil diameter D is 0.6 mm or greater but less than 1.3 mm, 10 CPD can be obtained, ensuring a depth of field. Therefore, an imaging optical system with a CPDmax of approximately 10 CPD can utilize the performance of the image sensor 104 even without focusing. On the other hand, in the region where CPDmax exceeds 15 CPD, if the imaging optical system 103 is pan-focused, the resolution of the image sensor 104 cannot be utilized.

[0030] If CPDmax is 15 CPD or more, a real image with relatively high resolution can be obtained. In other words, it is desirable to satisfy the condition πf / (360p)≧15.0. It is more preferable that CPDmax is 20 CPD or more. By setting the pupil diameter D to 1.3 mm or more in the region where CPDmax exceeds 15 CPD and ensuring a high MTF at the best focus position shown in FIG. 5 through focusing, a real image with high resolution close to what the human eye perceives can be obtained. In this embodiment, the pixel pitch of the image sensor 104 is 1.5 μm, the focal length of the imaging optical system 103 is 3.45 mm, and the CPDmax is 20 CPD. The pupil diameter D is 2.0 mm, and the F-number is 1.725.

[0031] 7 shows the through-focus MTF of the on-axis angle of view of the imaging optical system 103 of this embodiment at 20 CPD with a solid line, and for reference, the through-focus MTF at a pupil diameter of 1.0 mm is also shown with a dashed line. As can be seen from FIG. 7, if the pupil diameter D is 2.0 mm as in this embodiment, the gaze distance L ３ By focusing at 1.0 mm, a high MTF can be ensured that makes the most of the performance of CPDmax. On the other hand, when the pupil diameter D is 1.0 mm, the performance of CPDmax cannot be utilized.

[0032] To accommodate a higher CPDmax, as can be seen from Figure 5, it is preferable that the pupil diameter be D1.7 mm or greater, at which the MTF of the aplanatic lens at 30 CPD is 0.3 or greater. The larger the pupil diameter, the higher the resolving power, but it is difficult to achieve an extremely small F-number, which results in a longer focal length of the imaging optical system 103 and is disadvantageous for miniaturizing the HMD 100. For this reason, it is desirable that the pupil diameter D be less than 4.0 mm, which is the same as the diameter of the pupil when a human observes an image.

[0033] 8 and 9 show real images observed by the HMD 100 of this embodiment. Fig. 8 shows a real image in a state where the observer is gazing at a hand 801 as a close-up object. In this real image, the focused hand 801 is observed with high contrast. On the other hand, background objects 802, such as a tree, a house, and the sun, outside the depth of field, are observed blurred.

[0034] 9 shows a real image in which an observer is gazing at a house 901 as a distant object. In this real image, the focused house 901 is observed with high contrast, while the hand 801, which is outside the depth of field, is observed in a blurred state. The appearance at this time is close to what a human actually experiences through their eyes, and therefore, a natural appearance can be achieved in addition to high resolution.

[0035] When providing a sense of mixed reality to an observer, the observer is made to view a composite image in which a CG virtual object (virtual image) is superimposed (combined) on a real image. Fig. 10 shows a composite image in which background objects 1001, such as a tree, a house, and a sun, are superimposed as virtual objects and rendered at high resolution while the observer is gazing at a close-up hand 801. In this case, both the hand 801 and the background object 1001 are rendered at high resolution.

[0036] However, it is desirable to more naturally represent the background object 1001 within the depth of field as being blurred. In this case, the controller 205 causes the image processing circuit 111 to perform blurring using a Gaussian filter or the like on virtual objects outside the predetermined depth of field, i.e., at a distance of ±0.5D or more from the gaze distance in this embodiment, to generate a synthetic image. This makes it possible to reproduce the difference in appearance between inside and outside the depth of field in the synthetic image, as shown in FIG. 8, and to achieve both high resolution and natural appearance even when presenting mixed reality.

[0037] Superimposing a virtual image on a real image requires information on camera parameters, a concept used in computer vision, i.e., external and internal parameters of the imaging device 101. Focusing the imaging device 101 changes the internal parameters of the imaging device 101 (hereinafter referred to as internal camera parameters) required to superimpose a virtual image on a real image, resulting in errors in the relationship between the positions and sizes of the real and virtual images. For this reason, in this embodiment, data indicating the relationship between the adjustment amount and the internal camera parameters is stored in the image processing circuit 111 so that the internal camera parameters can be changed according to the adjustment amount (movement amount of the imaging optical system 103) during focusing of the imaging device 101. By referencing this data, the position and size of the virtual image relative to the real image can be appropriately set even when focusing is performed, making it possible to generate a composite image that can be viewed naturally, with no (or little) error in the positions and sizes of the real and virtual images.

[0038] FIG. 11 shows the relationship between the focal length of the imaging optical system 103, which is one of the camera internal parameters in this embodiment, and the in-focus object distance (1 / m). The focal length is a value normalized by the pixel pitch of the image sensor 104. It can be seen from FIG. 11 that the closer the in-focus object distance, the greater the focal length needs to be. When focusing to the gaze distance, adjusting the focal length, i.e., the imaging magnification, based on this relationship makes it possible to generate a composite image that allows for natural observation with no errors in the position or size of the real image and the virtual image.

[0039] The camera parameters may include the center position of the captured image, distortion correction value, principal point position, etc., and these may be changed in conjunction with the gaze distance for focusing.

[0040] Furthermore, the chief ray angle of the imaging optical system 103 relative to the image sensor 104 is generally a value other than 0. In other words, the magnification of the real image changes due to focusing. In this embodiment, the horizontal angle of view of the imaging optical system 103 is 80°, and the number of pixels of the image sensor 104 corresponding to the horizontal angle of view of 80° in response to focusing, i.e., the magnification, is held in the image processing circuit 111.

[0041] 12 shows the relationship between the in-focus object distance (1 / m) and the number of pixels (pixels) of the image sensor 104 corresponding to a horizontal angle of view of 80° in this embodiment. It can be seen from FIG. 12 that the closer the in-focus object distance, the more pixels of the image sensor 104 must be increased. By adjusting the number of pixels of the image sensor 104 based on this relationship when focusing to the gaze distance, it is possible to generate a composite image that allows for natural observation without changing the magnification of the real image even when focusing.

[0042] An appropriate number of pixels is also set for the vertical angle of view, and the details are the same as for the horizontal angle of view, so a detailed description thereof will be omitted.

[0043] According to the present embodiment described above, it is possible to obtain high-definition captured images and realize an HMD capable of displaying natural, high-quality composite images.

[0044] Next, a description will be given of Example 2. The basic configuration of the HMD in Example 2 is the same as that of the HMD 100 in Example 1, and the same reference numerals are used to designate the common components.

[0045] In the imaging device 101 of this embodiment, the focal length of the imaging optical system 103 is 6.9 mm, the pixel pitch p of the image sensor 104 is 2.0 μm, and the CPDmax is 30 CPD. The pupil diameter D of the imaging optical system 103 is 3.0 mm, and the F-number is 2.3.

[0046] 13 shows the through-focus MTF of the on-axis angle of view at 30 CPD of the imaging optical system 103 of this embodiment with a dashed line. The imaging optical system 103 of this embodiment has a configuration that can ensure a high MTF by focusing even at a frequency of CPDmax.

[0047] Humans have a high ability to distinguish only the area of ​​gaze near the fovea, and do not notice a decrease in resolution outside that area. Taking advantage of this human characteristic, a high-resolution image is generated in the captured image in a region of approximately 10° to 20° that includes the position (coordinates) corresponding to the point of gaze, and the image resolution is reduced to approximately half in the area outside that region (periphery). This reduces the amount of image data processed by the image processing circuit 111. In this embodiment, a high MTF is first ensured by focusing the imaging optical system 103 according to the observer's gaze distance, and an image is generated with a resolution of 2.0 μm pixel pitch in a rectangular region of ±10° centered on the position corresponding to the point of gaze in the captured image. Meanwhile, in the area outside that rectangular region, an image is generated with a resolution equivalent to a pixel pitch of 4.0 μm by binning (smoothing or adding multiple pixels to obtain a single pixel value). Specifically, for example, the number of captured frames per second is set to double the number of displayed frames per second. Then, in even-numbered imaging frames, high-resolution image data is acquired with the rectangular range as a region of interest (ROI), and in odd-numbered imaging frames, image data obtained by binning the entire region is acquired and combined.

[0048] 14 shows a real image (captured image) including a high-resolution region 401 as a region of interest (ROI) and an outer low-resolution region 402. The hand 801 that the observer is gazing at is a real hand. In this embodiment, the controller 205 causes the imaging device 101 to focus on the hand to ensure a high MTF, while causing the image processing circuit 111 to perform processing to reduce the resolution of the low-resolution region 402 to half that of the high-resolution region 401.

[0049] When a virtual image is superimposed on a real image to provide a mixed reality experience, the controller 205 causes the image processing circuit 111 to render the region of interest (ROI) at high resolution and the surrounding area at low resolution, thereby reducing the overall amount of data processing.

[0050] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0051] The embodiments described above are merely representative examples, and various modifications and alterations are possible to each embodiment when implementing the present invention.

Claims

1. An image display device comprising: an imaging device that generates a captured image by capturing an image of the outside world through an imaging optical system; a display element that displays a composite image of the captured image and a virtual image; a display optical system that directs light from the display element to the observer's eyes; an acquisition means that acquires a gaze distance, which is the distance to the observer's gaze point in the outside world; and a control means that controls focusing of the imaging device in accordance with the gaze distance, wherein the pupil diameter of the imaging optical system is 1.3 mm or more and 4.0 mm or less.

2. The image display device according to claim 1, further comprising a generating means for generating the composite image.

3. The image display device according to claim 1 or 2, characterized in that the imaging device has an imaging element that captures an external image formed by the imaging optical system, and satisfies the condition πf / (360p)≧15.0, where f (mm) is the focal length of the imaging optical system and p (mm) is the pixel pitch of the imaging element.

4. An image display device according to any one of claims 1 to 3, characterized in that the control means changes camera parameters in the generation means that generates the composite image in accordance with the amount of adjustment in the focusing.

5. An image display device according to any one of claims 1 to 4, characterized in that the control means changes the imaging magnification of the imaging device.

6. An image display device as described in any one of claims 1 to 5, characterized in that the control means causes the generation means that generates the composite image to perform processing to make the area of ​​the captured image that includes the position corresponding to the point of interest higher in resolution than the area outside of that area.

7. An image display device as described in any one of claims 1 to 6, characterized in that the control means causes the generation means that generates the synthetic image to perform processing to make the area of ​​the virtual image that includes the position corresponding to the gaze point higher in resolution than the area outside of that area.

8. An image display device as described in any one of claims 1 to 7, characterized in that the control means causes the generation means that generates the synthetic image to perform blurring processing on areas of the virtual image that correspond to distances farther than the gaze distance.

9. An image display device according to any one of claims 1 to 8, characterized in that the optical axis of the imaging optical system and the optical axis of the display optical system are aligned with each other, and the pupil of the imaging optical system and the pupil of the display optical system are spaced apart in the optical axis direction by a distance of 60 mm or less.

10. An image display device according to any one of claims 1 to 9, characterized in that the acquisition means detects the line of sight of the viewer and acquires the gaze distance based on the line of sight.

Images