Head-Mounted Displays and Cross-Reality Systems

Abstract

To provide a technique advantageous in eliminating or reducing discomfort and eye strain that an observer often feels when observing a video by using a head-mounted display.SOLUTION: A head-mounted display comprises: a display device that has a display surface; an optical system for adjusting the focus of the eyes of an observer to the display device; a driving mechanism that drives the display device; a line-of-sight detection unit that detects the line of sight of the observer; and a control unit that, based on output from the line-of-sight detection unit, controls the driving mechanism so that the angle formed by the line of sight and the display surface falls within an allowable range set in advance relative to 90 degrees at a point where the line of sight and the display surface intersect with each other.SELECTED DRAWING: Figure 23

Description

[Technical Field]

[0001] The present invention relates to a head-mounted display and a cross reality system (virtual space technology, space expansion technology). [Background technology]

[0002] Patent Document 1 discloses a gaze-tracking head-mounted display that uses information obtained by detecting the gaze of the user to move a display device to an appropriate position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-68670 Summary of the Invention [Problem to be solved by the invention]

[0004] Figure 39 shows an observer looking at a point object 3928 located 250 mm away through their eye 3901. A crystalline lens 3959, which is a lens of the eye 3901, has an optical axis 3929. The optical axis 3929 corresponds to the line of sight of the eye 3901. The thickness of the crystalline lens 3959 is adjusted so that light emitted from a point object 3928 located on an extension of the optical axis 3929 and incident on the pupil 3940 of the eye 3901 forms a focal point 3932 at a point on the retina 3931.

[0005] Here, the fovea 3952 on the retina 3931 has an inclination of 1 to 2 degrees or less with respect to the optical axis 3929, and can exist in a range where light passing through the center of the pupil 3940, such as light 3969 or light 3970, is incident. Photoreceptor cells are concentrated in the fovea 3952, and the observer can precisely recognize objects in a range 3971 around the point object 3928. The focal point 3932 is located approximately at the center of the fovea 3952.

[0006] Photoreceptor cells are also sparsely present in the peripheral region 3964 of the fovea 3952. Because the photoreceptor cells are sparse, it is not possible to precisely distinguish an object that forms a focus in the peripheral region 3964. However, the peripheral region 3964 is a necessary part for determining the color, shape, brightness, and perspective of an object. For example, the focus 3965 in the peripheral region 3964 is the point at which light incident from the point object 3966 onto the pupil 3940 forms a focus. Similarly, the focus 3967 is the point at which light incident from the point object 3968 onto the pupil 3940 forms a focus. These foci 3965 and 3967 are not perfect focuses due to optical aberration, but are tiny regions where light emitted from the point object 3966 and the point object 3968, respectively, is focused.

[0007] A collection of object-side point objects, including point object 3928, point object 3966, and point object 3968, each forming a focus on retina 3931, forms an approximately spherical surface 3972. Spherical surface 3972 can be understood as the object-side focal surface when retina 3931 is considered to be the imaging surface.

[0008] Light emitted from a point object located in front of the focal plane 3972, i.e., close to the eye 3901, forms a focus behind the retina 3931. Therefore, the image of the point object becomes a blurred image on the retina 3931. Conversely, light emitted from a point object located behind the focal plane 3972, i.e., far from the eye 3901, forms a focus in front of the retina 3931. Therefore, the image of the point object becomes a blurred image on the retina 3931.

[0009] The degree of blur of the point object image formed on retina 3931 increases as the distance between focal plane 3972 and the point object increases. Therefore, focal plane 3972 serves as a reference for estimating the distance between the eye and the point object. In other words, if the degree of blur of the point object image on retina 3931 is small, it is determined to be a point object at the same distance as point object 3928 that eye 3901 is looking at, i.e., near focal plane 3972. Conversely, if the degree of blur of the point object image is large, it is determined to be a point object at a different distance from point object 3928, i.e., farther away from focal plane 3972.

[0010] Patent Document 1 discloses a gaze-tracking head-mounted display that moves a display device to an appropriate position using information obtained by detecting the viewer's gaze, as described above. FIG. 40 shows a stereo camera that captures images to be displayed on the head-mounted display of Patent Document 1. Reference numeral 4001 denotes a camera that captures images for the left eye. Reference numeral 4002 denotes a camera that captures images for the right eye. Each camera incorporates a flat image sensor, indicated by the white line in FIG. 40, and the lens is designed so that the image-side focal plane is located above this image sensor. Dashed line 4009 denotes the object-side focal plane when the left-eye camera 4001 focuses on the person indicated by the black circle 4003 and captures the image. Dashed line 4010 denotes the object-side focal plane when the right-eye camera 4002 focuses on the person indicated by the black circle 4003 and captures the image. The in-focus plane 4009 and the in-focus plane 4010 are shown slightly shifted up and down to make them easier to distinguish, but the actual in-focus position is somewhere between them.

[0011] Similarly, dashed line 4011 is the plane of focus when the left-eye camera 4001 focuses on the black X tree 4006 and takes an image, and dashed line 4012 is the plane of focus when the right-eye camera 4002 focuses on the tree 4006 and takes an image. As before, dashed lines 4011 and 4012 are shown slightly offset vertically to make them easier to distinguish, but the actual focus position is somewhere between the two.

[0012] The black circle 4003 indicates the actual location of a person nearby relative to the cameras 4001 and 4002. The gray circle 4004 is the position of the person's image on the focal plane 4011 as seen by the left-eye camera 4001, and the gray circle 4005 is the position of the person's image on the focal plane 4012 as seen by the right-eye camera 4002. The black X 4006 indicates the actual location of a tree that is far away relative to the cameras 4001 and 4002. The gray X 4007 is the position of the tree image on the focal plane 4009 as seen by the left-eye camera 4001, and the gray X 4008 is the position of the tree image on the focal plane 4010 as seen by the right-eye camera 4002.

[0013] FIG. 41 shows an image captured by the stereo camera of FIG. 40. Frame 4109 shows an image captured by the left-eye camera 4001, with the focus set on the person in the black circle 4003. The dashed line 4009 shows frame 4109 at the in-focus position. The person image 4103 to the right of the center of frame 4109 is an image of the person at the position of the black circle 4003. Additionally, the tree image 4107 in the upper left is an image at the position of the in-focus plane 4009 of the tree in the black X 4006. In reality, the image is blurred because it is farther away than the in-focus plane 4009.

[0014] Similarly, frame 4110 shows an image captured by the right-eye camera 4002 with the focus set on the person in the black circle 4003. The dashed line 4010 shows frame 4110 at the in-focus position. The person image 4103 to the right of the center of frame 4110 is an image of the person at the position of the black circle 4003. Additionally, the tree image 4108 in the upper left is an image at the position of the in-focus plane 4010 of the tree in the black X 4006. In reality, the image is blurred because it is farther away than the in-focus plane 4010.

[0015] Furthermore, frames 4111 and 4112 show images captured by the left-eye camera 4001 and the right-eye camera 4002, respectively, with the focus set on the tree marked with a black X 4006. The dashed line 4011 shows frame 4111 at the in-focus position, and the dashed line 4012 shows frame 4112 at the in-focus position. Person images 4104 and 4105 located to the right of the center of frames 4111 and 4112 are images at the in-focus planes 4011 and 4012, respectively, of the person located at the black circle 4003. In reality, the images are blurred because the person is located in front of the in-focus planes 4011 and 4012, respectively. The tree image 4106 in the upper left is an image of the tree located at the black X 4006.

[0016] Figures 42, 43, 44, and 45 show how the image in Figure 41 captured by the stereo camera in Figure 40 is observed using the head-mounted display (hereinafter referred to as HMD) disclosed in Patent Document 1. Figure 42 shows how the observer looks at a person displayed in the image. Similarly, Figure 44 shows how the observer looks at a tree displayed in the image. 4201 and 4401, 4202 and 4402 are the observer's left and right eyes, respectively. 4213 and 4413, 4214 and 4414 are the display device for the left eye and the display device for the right eye, respectively. The left eye lenses 4215, 4415 and the right eye lenses 4216, 4416 are designed so that the focus of the observer's left eyes 4201, 401 and right eyes 4202, 4402 is aligned with the surfaces of the left eye display devices 4213, 4413 and the right eye display devices 4214, 4414, respectively.

[0017] FIG. 43 is a diagram showing an image displayed on the display device of the HMD of FIG. 42. Frame 4313 is the frame of display device 4213. Frame 4309 indicated by a dashed line corresponds to frame 4109 in FIG. 41 and is the range of the image captured by stereo camera 4001. A portion of the image in frame 4109 that is closer to the right is cropped and displayed on display device 4213. Similarly, frame 4314 is the frame of display device 4214. Frame 4310 indicated by a dashed line corresponds to frame 4110 in FIG. 41 and is the range of the image captured by stereo camera 4002. A portion of the image in frame 4110 that is closer to the left is cropped and displayed on display device 4214.

[0018] Frame 4313 and frame 4314 are frames that are located at the same position relative to person 4303, and when the images of frame 4313 and frame 4314 are superimposed, person 4303 appears as one image, as in frame 4321. Because there is parallax between the image of person 4303 in frame 4313 and the image of person 4303 in frame 4314, the image of person 4303 appears three-dimensional to the observer. On the other hand, although tree 4307 and tree 4308 are separated into two, they are observed separately with left eye 4201 and right eye 4202, and so the trees appear to be far away due to the effect of stereoscopic vision.

[0019] 42, black circle 4217 is the position where a person is displayed on display device 4213, and black circle 4218 is the position where the person is displayed on display device 4214. Also, gray X 4219 is the position where a tree is displayed on display device 4213, and gray X 4220 is the position where the tree is displayed on display device 4214. At this time, the viewer perceives the person as being at the position of black circle 4203, which is the point where the line of sight of left eye 4201 and the line of sight of right eye 4202 intersect, and the tree as being further away.

[0020] FIG. 45 is a diagram showing an image displayed on the display device of the HMD of FIG. 44. Frame 4513 is the frame of display device 4413. Frame 4509 indicated by a dashed line corresponds to frame 4111 in FIG. 41, and is the range of the image captured by stereo camera 4001. A portion of the image in frame 4111 that is closer to the left is cropped and displayed on display device 4413. Similarly, frame 4514 is the frame of display device 4414. Frame 4510 indicated by a dashed line corresponds to frame 4112 in FIG. 41, and is the range of the image captured by stereo camera 4002. A portion of the image in frame 4112 that is closer to the right is cropped and displayed on display device 4414.

[0021] Frame 4513 and frame 4514 are both frames that are in the same position relative to tree 4506, and when the images of frame 4513 and frame 4514 are superimposed, the tree 4506 appears to be superimposed as one, as in frame 4521. On the other hand, although person 4504 and person 4505 are separated into two, because they are observed separately by left eye 4401 and right eye 4402, the effect of stereoscopic vision makes it appear as if the people are in the foreground.

[0022] 44, gray circle 4417 is the position where the person is displayed on display device 4413, and gray circle 4418 is the position where the person is displayed on display device 4414. Furthermore, black X 4419 is the position where the tree is displayed on display device 4413, and black X 4420 is the position where the tree is displayed on display device 4414. At this time, the viewer perceives the tree as being at black X 4406, the point where the line of sight of left eye 4401 and the line of sight of right eye 4402 intersect, and the viewer perceives the person as being closer to the tree.

[0023] Here, the angle at which the line of sight of the left eye 4201 intersects with the line of sight of the right eye 4202, or the angle at which the line of sight of the left eye 4401 intersects with the line of sight of the right eye 4402, is detected. Then, from these values, the distance 4222 or the distance 4422 between the eye and the object it is looking at can be calculated. Then, based on the distance 4222 or the distance 4422, in the case of near vision, as shown in FIG. 42, the left eye display device 4213 can be moved to the right and the right eye display device 4214 can be moved to the left to accommodate cross-eye movement. This makes it possible to narrow the gap 4224 between the two. Furthermore, both display devices can be moved downward to accommodate the thickness of the crystalline lens during near vision, thereby reducing the distance 4223 between the eyes and both display devices. Conversely, in the case of far vision as shown in Figure 44, the lines of sight of both eyes become closer to being parallel, so that the display device 4413 for the left eye is moved to the left and the display device 4414 for the right eye is moved to the right, thereby increasing the distance 4423 between both eyes and the display devices.

[0024] The method shown in Figures 42, 43, 44, and 45, in which the image shown in Figure 41 captured by the stereo camera shown in Figure 40 is observed using a head-mounted display (hereinafter referred to as HMD) disclosed in Patent Document 1, differs from the situation in which an observer actually views people and trees on site in the following respects: Therefore, there is a problem in that the observer experiences eye fatigue and discomfort.

[0025] Figures 46, 47, 48, and 49 show how an observer on-site views the scenery photographed by the stereo camera in Figure 40. 4601 and 4801, and 4602 and 4802 are the observer's left and right eyes, respectively. The left eyes 4601 and 4801 are located in the same position as the left-eye camera 4001 in Figure 40. Similarly, the right eyes 4602 and 4802 are located in the same position as the right-eye camera 4002 in Figure 40.

[0026] 46 and 47 show a situation where an observer is looking at a person in the foreground. The line of sight of the left eye 4601 and the line of sight of the right eye 4602 intersect at the position of the black circle 4603 where the person is located. The arc-shaped dashed line 4609 is the arc-shaped focal plane of the eye explained in FIG. 39, and is the focal plane of the left eye 4601. Therefore, the line of sight of the left eye 4601 intersects with the arc-shaped focal plane 4609 at a right angle at the intersection point. Similarly, the arc-shaped dashed line 4610 is the focal plane of the right eye 4602, and the line of sight of the right eye 4602 intersects with the arc-shaped focal plane 4610 at a right angle at this intersection point.

[0027] The first difference between the method in Fig. 42 and the situation in Fig. 46 is that the focal plane 4209 is not perpendicular to the line of sight of the left eye 4201. This causes an imbalance in which it is easier to focus on a near object close to the left eye 4201 on the left side of the line of sight, and easier to focus on an object far from the left eye 4201 on the right side of the line of sight. Similarly, because the focal plane 4210 is not perpendicular to the line of sight of the right eye 4202, it causes an imbalance in which it is easier to focus on a near object close to the right eye 4202 on the left side of the line of sight, and easier to focus on an object far from the right eye 4202 on the right side of the line of sight.

[0028] 46, in comparison, the focal plane 4609 is perpendicular to the line of sight, so the left and right sides of the line of sight tend to be in focus equally relative to the distance from the left eye 4601. This difference is one of the causes of discomfort and eye fatigue for the observer.

[0029] The second difference between the method in Figure 42 and the situation in Figure 46 is due to the difference in the shape of the focal plane. The focal plane 4210 in Figure 42 and the line of sight of the right eye 4202 are approximately perpendicular. However, its shape is approximately flat, which differs from the approximately spherical shape of the focal plane 4610 in Figure 46. As a result, the distance between the subject and the focal plane changes, resulting in a different degree of blur of the subject. This cannot be resolved even if the depth of field of the right-eye camera 4002 and the depth of field of the right eye 4602 in Figure 46 could be matched.

[0030] For example, in Figure 46, the distance between the black X 4606 where a tree is present and the tree image 4608 on the focal plane 4610 is greater than the distance between the black X 4206 where a tree is present and the tree image 4208 on the focal plane 4210 in Figure 42. Therefore, even if the depth of field of the right-eye camera 4002 and the depth of field of the right eye 4202 in Figure 42 are the same, the degree of blurring of the tree image 4608 and the tree image 4208 will be different. This is clearly shown by the difference in thickness between the lines of the tree image 4308 in Figure 43 and the tree image 4708 in Figure 47. This difference in the degree of blurring also contributes to the observer's discomfort and eye strain.

[0031] The third difference between the method in Figure 42 and the situation in Figure 46 is that person 4303 in Figure 43, which corresponds to black circle 4203 at the end of the line of sight, is not at the center of frames 4313 and 4314, which represent the field of view. Person 4703 in Figure 47, which corresponds to black circle 4603 at the end of the line of sight in Figure 46, is at the center of frames 4709 and 4710, which represent the field of view. This is because the human eye is designed to move its field of view in accordance with the movement of the line of sight, so that the end of the line of sight is always at the center of the field of view. Moreover, except in special cases, the focus is adjusted to be always at the end of the line of sight in almost all cases. In contrast, with the system in Figure 42, as shown in Figure 43, the image is in focus outside the center of the field of view, and in this state, it is possible to move the gaze to trees 4307 and 4308, which are out of focus. This causes discomfort to the observer and contributes to eye strain.

[0032] A fourth way in which the method in Fig. 42 differs from the situation in Fig. 46 can be explained by comparing frame 4321 in Fig. 43 with frame 4721 in Fig. 47. As described above, in-focus plane 4209 is different from in-focus plane 4609, and in-focus plane 4210 is different from in-focus plane 4610. This causes the positional relationship between person 4303 and tree 4307, or person 4303 and tree 4308 in frame 4321, to differ from the positional relationship between person 4703 and tree 4707, or person 4703 and tree 4708 in frame 4721. In particular, distance 4325 between tree 4307 and tree 4308 in frame 4321 is different from distance 4725 between tree 4707 and tree 4708 in frame 4721. These distances of 4325 and 4725 are important factors in determining the distance between the person and the tree in the human mind when viewing in 3D, and if these distances are different, the perceived distance between the person and the tree will be different from the actual distance, causing discomfort and eye strain. This is why 3D images can appear to jump out of the screen, contrary to the human sense of sight, or to appear to float when viewed from an angle, creating discomfort.

[0033] The first to fourth differences can be understood in a similar manner by comparing Figures 44 and 45 with Figures 48 and 49. Figures 48 and 49 show a situation in which an observer is looking at a distant tree. The line of sight of the left eye 4801 and the line of sight of the right eye 4802 intersect at the position of the black X 4806 where the tree is located. The arc-shaped dashed line 4811 is the arc-shaped focal plane of the eye described in Figure 39 and is the focal plane of the left eye 4801. Therefore, the line of sight of the left eye 4801 intersects with the arc-shaped focal plane 4811 at a right angle at the intersection. Similarly, the arc-shaped dashed line 4812 is the focal plane of the right eye 4802, and the line of sight of the right eye 4802 intersects with the arc-shaped focal plane 4812 at a right angle at that intersection.

[0034] The first difference between the method in Fig. 44 and the situation in Fig. 48 is that the focal plane 4411 is not perpendicular to the line of sight of the left eye 4401. This causes an imbalance in which the left side of the line of sight tends to focus on objects far from the left eye 4401, and the right side of the line of sight tends to focus on objects in the foreground that are close to the left eye 4401. Similarly, because the focal plane 4411 is not perpendicular to the line of sight of the right eye 4402, this causes an imbalance in which the left side of the line of sight tends to focus on objects far from the right eye 4402, and the right side of the line of sight tends to focus on objects in the foreground that are close to the right eye 4402.

[0035] 48, in comparison, the focal plane 4811 is perpendicular to the line of sight, so the left and right sides of the line of sight tend to be in focus equally relative to the distance from the left eye 4801. This difference is one of the causes of discomfort and eye fatigue for the observer.

[0036] The second difference between the method in FIG. 44 and the situation in FIG. 48 is due to the difference in the shape of the focal plane. The shapes of focal planes 4410 and 4411 in FIG. 44 are approximately flat, which differs from the approximately spherical shapes of focal planes 4811 and 4812 in FIG. 48. Therefore, the distance between the subject and the focal plane changes between the cases in FIG. 44 and FIG. 48, resulting in a different degree of blur of the subject. This cannot be resolved even if the depth of field of left-eye camera 4001 could be matched with the depth of field of left eye 4801 in FIG. 48, and similarly, the depth of field of right-eye camera 4002 could be matched with the depth of field of right eye 4802 in FIG. 48.

[0037] For example, the distance between the position of black circle 4803 where the person in Figure 48 is actually located and the position of gray circle 4805, which is the image of the person on in-focus plane 4812, is greater than the distance between the position of black circle 4403 where the person in Figure 44 is actually located and gray circle 4405, which is the image of the person on in-focus plane 4412. Therefore, the degree of blur of the image of the person in gray circle 4805 is greater than the degree of blur of the image of the person in gray circle 4405. This is clearly shown by the difference in thickness between the lines of person image 4505 in Figure 45 and person image 4905 in Figure 49. This difference in the degree of blur also contributes to the viewer's discomfort and eye fatigue.

[0038] The third difference between the method in Figure 44 and the situation in Figure 48 is that tree 4506 in Figure 45, which corresponds to black X 4406 at the end of the line of sight, is not at the center of the field of view. Tree 4906 in Figure 49, which corresponds to black X 4806 at the end of the line of sight in Figure 48, is at the center of frames 4909 and 4910, which represent the field of view. This is because the human eye is designed to move its field of view in accordance with the movement of the line of sight, so that the end of the line of sight is always at the center of the field of view. Furthermore, except in special cases, the focus is adjusted to be always at the end of the line of sight in almost all cases. In contrast, with the system in Figure 44, as shown in Figure 45, the focus is off-center, and in this state, the viewer can easily move their gaze to out-of-focus figures 4504 and 4505. This can cause discomfort and eye strain for the viewer.

[0039] A fourth way in which the method in Fig. 44 differs from the situation in Fig. 48 can be explained by comparing frame 4521 in Fig. 45 with frame 4921 in Fig. 49. As described above, in-focus plane 4411 and in-focus plane 4811 are different, and in-focus plane 4412 and in-focus plane 4812 are different. As a result, the positional relationship between person 4504 and tree 4506, or person 4505 and tree 4506, in frame 4521 differs from the positional relationship between person 4904 and tree 4906, or person 4904 and tree 4906, in frame 4921. In particular, distance 4525 between person 4504 and person 4505 in frame 4521 differs from distance 4925 between person 4904 and person 4905 in frame 4921. These distances of 4525 and 4925 are important factors in determining the distance between the person and the tree in the human mind when viewing in 3D, and if these are different, the perceived distance between the person and the tree will be different from the actual distance, causing discomfort and eye strain. This is why 3D images can appear to jump out of the screen, contrary to the human sense of sight, or to appear to float when viewed from an angle, creating discomfort.

[0040] Furthermore, conventional HMDs have other causes of eye fatigue that are different from the four different factors mentioned above. In the state shown in Figure 42, the line of sight between the display device 4213 and the left eye 4201 is not perpendicular. It is also not perpendicular to the line of sight between the display device 4214 and the right eye 4202. Therefore, the optical axis of the lens 4215 does not coincide with the line of sight of the left eye 4201, and the optical axis of the lens 4216 does not coincide with the line of sight of the right eye 4202. This means that the image ahead of the observer's line of sight is susceptible to the optical aberration of the lenses 4215 and 4216. This is also true in the state shown in Figure 44. In other words, with conventional HMDs, the optical axis of the lens coincides with the observer's line of sight only when the observer looks directly ahead at a distant object. In most other cases, the optical aberration of the lens easily causes eye fatigue. Another problem is that the optical system becomes larger and more complex in order to suppress the effects of this aberration.

[0041] The present invention provides an advantageous technique for eliminating or reducing the discomfort and eye strain that viewers often experience when viewing using an HMD. [Means for solving the problem]

[0042] A first aspect of the present invention relates to a head-mounted display comprising a display device having a display surface formed by a portion of a sphere, and an optical system for focusing an observer's eyes on the display device, wherein the head-mounted display comprises a drive mechanism for driving the display device, a support unit having a support surface formed by a portion of a sphere and supporting the display device on the support surface, a gaze detection unit for detecting the observer's gaze, and a control unit for controlling the drive mechanism based on the output of the gaze detection unit so that the angle between the gaze and the display surface at the point where the gaze intersects with the display surface falls within a predetermined tolerance range of 90 degrees. A second aspect of the present invention relates to a head-mounted display comprising a display device having a display surface and an optical system for focusing an observer's eyes on the display device, wherein the head-mounted display comprises a drive mechanism for driving the display device, a support unit having a support surface constituted by a part of a sphere and supporting the display device on the support surface, a gaze detection unit for detecting the observer's line of sight, and a control unit for controlling the drive mechanism based on the output of the gaze detection unit so that the angle between the line of sight and the display surface at the point where the line of sight intersects with the display surface falls within a predetermined tolerance range with respect to 90 degrees, and the support surface is configured so that the center of the sphere constituting the support surface approximately coincides with the center of rotation of the movement of the line of sight. [Effects of the Invention]

[0043] The present invention provides an advantageous technique for eliminating or reducing the discomfort and eye strain that viewers often experience when viewing using an HMD. [Brief explanation of the drawings]

[0044] [Figure 1] FIG. 1 is a diagram illustrating a first embodiment. [Figure 2] FIG. 1 is a diagram illustrating a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating a first embodiment. [Figure 4] FIG. 1 is a diagram illustrating a first embodiment. [Figure 5] FIG. 10 is a diagram illustrating a second embodiment. [Figure 6] FIG. 1 is a diagram illustrating a first embodiment. [Figure 7] FIG. 1 is a diagram illustrating a first embodiment. [Figure 8] FIG. 1 is a diagram illustrating a first embodiment. [Figure 9] FIG. 10 is a diagram illustrating a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating a second embodiment. [Figure 11] FIG. 10 is a diagram illustrating a second embodiment. [Figure 12] FIG. 10 is a diagram illustrating a second embodiment. [Figure 13] FIG. 10 is a diagram illustrating a second embodiment. [Figure 14] FIG. 10 is a diagram illustrating a second embodiment. [Figure 15] FIG. 10 is a diagram illustrating a second embodiment. [Figure 16] FIG. 10 is a diagram illustrating a second embodiment. [Figure 17] FIG. 10 is a diagram illustrating a second embodiment. [Figure 18] FIG. 1 is a diagram illustrating a first embodiment. [Figure 19] FIG. 10 is a diagram illustrating a second embodiment. [Figure 20] FIG. 1 is a diagram illustrating a first embodiment. [Figure 21] FIG. 10 is a diagram illustrating a second embodiment. [Figure 22] FIG. 1 is a diagram illustrating a first embodiment. [Figure 23] FIG. 1 is a diagram illustrating a first embodiment. [Figure 24] FIG. 1 is a diagram illustrating a first embodiment. [Figure 25] FIG. 1 is a diagram illustrating a first embodiment. [Figure 26] FIG. 1 is a diagram illustrating a first embodiment. [Figure 27] FIG. 1 is a diagram illustrating a first embodiment. [Figure 28] FIG. 1 is a diagram illustrating a first embodiment. [Figure 29] FIG. 1 is a diagram illustrating a first embodiment. [Figure 30] FIG. 1 is a diagram illustrating a first embodiment. [Figure 31] FIG. 1 is a diagram illustrating a first embodiment. [Figure 32] FIG. 1 is a diagram illustrating a first embodiment. [Figure 33] FIG. 1 is a diagram illustrating a first embodiment. [Figure 34] FIG. 1 is a diagram illustrating a first embodiment. [Figure 35] FIG. 10 is a diagram illustrating a second embodiment. [Figure 36] FIG. 10 is a diagram illustrating a third embodiment. [Figure 37] FIG. 10 is a diagram illustrating a fourth embodiment. [Figure 38] FIG. 10 is a diagram illustrating Example 5. [Figure 39] FIG. 1 is a diagram illustrating a problem. [Figure 40] FIG. 1 is a diagram illustrating a problem. [Figure 41] FIG. 1 is a diagram illustrating a problem. [Figure 42] FIG. 1 is a diagram illustrating a problem. [Figure 43] FIG. 1 is a diagram illustrating a problem. [Figure 44] FIG. 1 is a diagram illustrating a problem. [Figure 45] FIG. 1 is a diagram illustrating a problem. [Figure 46] FIG. 1 is a diagram illustrating a problem. [Figure 47] FIG. 1 is a diagram illustrating a problem. [Figure 48] FIG. 1 is a diagram illustrating a problem. [Figure 49] FIG. 1 is a diagram illustrating a problem. DETAILED DESCRIPTION OF THE INVENTION

[0045] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0046] The first embodiment will be described with reference to Fig. 1 to Fig. 8. Fig. 1 shows one state of a head-mounted display (hereinafter referred to as HMD) of the first embodiment. The HMD has a display device 0113 for the observer's left eye 0101, and a lens (optical system) 0115 for adjusting the focus of the left eye 0101 on the display surface of the display device 0113. Similarly, the HMD has a display device 0114 for the right eye 0102, and a lens (optical system) 0116 for adjusting the focus of the right eye 0102 on the display surface of the display device 0114.

[0047] The lens 0115 and the display device 0113 can be adjusted along five axes so that the optical axis of the lens 0115 coincides with the central normal of the display device 0113. Furthermore, the distance 0126 between the lens 0115 and the display device 0113 can be adjusted along a sixth axis so that it becomes a predetermined distance. The lens 0115 and the display device 0113 can form a single display module for the left eye. The range of the six-axis adjustment described above is a translation distance of ±40 μm, more preferably ±4 μm, and an angle of ±40 arc minutes, more preferably ±4 arc minutes. The left-eye display module can be provided with a mechanism (not shown) that can change the distance 0126 from the predetermined distance within a predetermined movable range. The lens 0116 and the display device 0114 can also form a right-eye display module.

[0048] The line connecting the left eye 0101 and the black circle 0103 is the line of sight of the left eye 0101. The HMD may be provided with an adjustment mechanism (not shown) that adjusts the position of the left eye display module so that the line of sight of the left eye 0101 coincides with the optical axis of the lens 0115. Furthermore, the HMD may also be provided with a mechanism that adjusts the position of the left eye display module so that the distance 0126 between the display device 0113 and the left eye 0101 is a predetermined distance. This makes it possible to adjust the distance 0126 between the lens 0115 in the left eye display module and the display device 0113 accordingly, even if the thickness of the lens of the observer's left eye 0101 changes within a predetermined range. This makes it possible to adjust the focus of the left eye 0101 to the surface of the display device 0113. The accuracy of these position adjustments is the same as the six-axis adjustment range described above.

[0049] The position adjustment mechanism of the left eye display module makes the angle θ between the line of sight of the left eye 0101 and the display surface of the display device 0113 approximately perpendicular at the intersection of the line of sight of the left eye 0101 and the display device 0113. The range of θ is within ±60 arc minutes, more preferably within ±6 arc minutes. The position adjustment mechanism of the right eye display module may also be provided in the HMD, similar to the case of the left eye display module described above. At the intersection of the line of sight of the right eye 0102 and the display device 0114, the angle θ between the line of sight of the right eye 0102 and the display surface of the display device 0114 may also be made approximately perpendicular.

[0050] Fig. 3 shows another state of the HMD of the first embodiment shown in Fig. 1. The direction of the line of sight of the left eye 0301 is different from the direction of the line of sight of the left eye 0101 in Fig. 1. The direction of the line of sight of the right eye 0302 is also different from the direction of the line of sight of the right eye 0102 in Fig. 1.

[0051] The HMD of the first embodiment may be provided with a line-of-sight detection unit (not shown). The line-of-sight detection unit detects the line of sight of the left eye 0301, and a mechanism (not shown) that adjusts the position of a left-eye display module consisting of a lens 0315 and a display device 0313 based on the detection information. This mechanism can maintain the angle θ between the line of sight of the left eye 0301 and the display surface of the display device 0313 at an intersection 0319 between the line of sight of the left eye 0301 and the display device 0313, as shown in FIG. 3 . Here, the range of θ is within ±60 arc minutes, more preferably within ±6 arc minutes.

[0052] Similarly, the line of sight of the right eye 0302 is detected by the line of sight detection unit, and an adjustment unit (not shown) (corresponding to adjustment unit A composed of a drive mechanism D and a control unit C described below) may be provided that adjusts the position of a display module for the right eye, which is composed of a lens 0316 and a display device 0314, based on the detection information. This adjustment unit can also maintain or adjust the angle θ between the line of sight of the right eye 0302 and the display surface of the display device 0314 to be approximately perpendicular at an intersection 0320 between the line of sight of the right eye 0302 and the display surface of the display device 0314. Here, too, the range of θ is within ±60 arc minutes, more preferably within ±6 arc minutes.

[0053] 1 and 3, the angle θ between the line of sight and the display surface of the display device can be controlled by a similar adjustment unit so that it is always approximately perpendicular at the point where the line of sight intersects with the display surface of the display device. By keeping the angle θ between the line of sight and the display surface of the display device perpendicular in this way, two of the causes of eye fatigue that occur with conventional HMDs are eliminated.

[0054] The first factor is eye fatigue caused by the effects of optical aberration, which occurs when the line of sight of the eyes is not perpendicular to the display surface of the display device as shown in the examples of Figures 42 and 44 in conventional HMDs, except when the line of sight is directed straight ahead. This can be eliminated. It also makes it possible to prevent the optical system from becoming complicated and large in order to suppress the effects of this aberration.

[0055] The second factor is the third difference between observation with a conventional HMD, as explained in the comparison between Figures 42 and 46, and Figures 44 and 48, and observation with the eyes in an actual location. In other words, with conventional HMDs, the object in front of the observer's line of sight that the observer wants to see is not always in the center of the field of view, causing eye strain. In the first embodiment, the line of sight is always maintained at the center of the display device, so the image that the observer wants to see is always maintained in the center of the field of view, thereby eliminating another factor causing eye strain.

[0056] 5 is a diagram showing how an image to be displayed on the display device 0113 in FIG. 1 is captured using a left-eye camera module 0501, and how an image to be displayed on the display device 0114 in FIG. 1 is captured using a right-eye camera module 0502. An attitude control unit (not shown) may be provided that receives gaze detection information from the left eye 0101 in FIG. 1 and controls the attitude of the left-eye camera module 0501 so that the optical axis of the camera module 0501 faces the same direction as the gaze of the left eye 0101. Similarly, an attitude control unit (not shown) may be provided that controls the attitude of the right-eye camera module 0502 so that the optical axis of the right-eye camera module 0502 coincides with the gaze of the right eye 0102.

[0057] Camera module 0501 and camera module 0502 may have built-in flat image sensors indicated by white lines. Dashed lines 0509 and 0510 indicate the focal planes on the object side of camera module 0501 and camera module 0502, respectively. Therefore, an image of an object on dashed line 0509 is formed on the image sensor indicated by the white line through the lens of camera module 0501. Similarly, an image of an object on dashed line 0510 is formed on the image sensor indicated by the white line through the lens of camera module 0502.

[0058] In Figure 5, there is a person at the black circle 0503 at the intersection of the optical axis of camera module 0501 and the optical axis of camera module 0502. There is also a tree at the black X 0506 in the distance to the left. The dashed line 0510 and the dot-dash line connecting camera module 0502 and black X 0506 intersect. Therefore, the tree appears in the image captured by camera module 0502. This is indicated by the gray X 0508. If there was actually a tree at the location of the gray X 0508, the image of the tree in the image captured by camera module 0502 would be in focus, but because the tree is actually at the location of the black X 0506, the image will be blurred in proportion to the distance between the black X 0506 and the gray X 0508.

[0059] On the other hand, the dashed-dotted line connecting the camera module 0501 and the black X 0506 does not intersect with the dashed line 0509, but does intersect with the dashed-dotted line that is an extension of the dashed line 0509. This is indicated by the gray dashed X 0507. Therefore, the tree does not appear in the image captured by the camera module 0501.

[0060] Figure 6 shows an image captured by the camera module in Figure 5. Frame 0609 corresponds to dashed line 0509 in Figure 5, and frame 0610 corresponds to dashed line 0510 in Figure 5. Because a person is located at black circle 0503 on the optical axis of camera module 0501 in Figure 5, person image 0603 is captured at the center of frame 0609 in Figure 6. Similarly, black circle 0503 is also located on the optical axis of camera module 0502, and because a person is located there, person image 0603 is captured at the center of frame 0610 in Figure 6. Because both person images 0603 have parallax between camera module 0501 and camera module 0502, although both are in the center of the frame, they are not the same image. A tree image 0608 is captured in the upper left corner of frame 0610, but it is blurred for the reasons mentioned above. This is indicated by the thick gray line. On the other hand, no tree image is captured in frame 0609 for the reasons mentioned above.

[0061] FIG. 2 shows the captured image of FIG. 6 displayed on the display device of FIG. 1. The captured image of frame 0609 of FIG. 6 is displayed on display device 0113 of FIG. 1. The displayed image is frame 0213 of FIG. 2. Similarly, the captured image of frame 0610 of FIG. 6 is displayed on display device 0114 of FIG. 1. The displayed image is frame 0214 of FIG. 2. The observer sees the image of frame 0213 with their left eye 0101 and the image of frame 0214 with their right eye 0102, and the two images are combined in their minds. Frame 0221 shows this state. There is only one central person image 0203, but the person image 0203 of frame 0213 and the person image 0203 of frame 0214, which contain parallax, are superimposed, creating a three-dimensional appearance. Furthermore, the tree image 0208 is perceived only by the right eye 0102, and the sense of spatial distance from the person image 0203 is inferred in the observer's mind from the degree of blurring of the tree image.

[0062] Let us consider a case in which the observer views a person at the black circle 0103, assuming that the display device 0113, the display device 0114, the lens 0115, and the lens 0116 in FIG. 1 are not present. In this case, the lenses of the left eye 0101 and the right eye 0102 are each adjusted to a predetermined thickness in order to focus on the person at the black circle 0103. The image 0117 of the person on the display device 0113, which actually exists, can be formed on the retina of the left eye 0101 while maintaining the predetermined thickness by adjusting the distance 0126 between the display device 0113 and the lens 0115. Similarly, the image 0118 of the person on the display device 0114, which actually exists, can be formed on the retina of the right eye 0102 while maintaining the predetermined thickness by adjusting the distance 0127 between the display device 0114 and the lens 0116. When such adjustments are made, the focal plane perceived by the observer for the left eye 0101 is the dashed line 0109, and the focal plane perceived by the observer for the right eye 0102 is the dashed line 0110, reproducing the focal plane 0509 of the camera module 0501 and the focal plane 0510 of the camera module 0502 in Figure 5.

[0063] Next, as shown in FIG. 3, if the observer's attention shifts from the person to a distant tree 0306, the line of sight of the left eye 0301 shifts to the dash-dotted line connecting the left eye 0301 and the black X 0306. Similarly, the line of sight of the right eye 0302 shifts to the dash-dotted line connecting the right eye 0302 and the black X 0306. This change in line of sight is detected by the line of sight detection unit (not shown) and transmitted to an adjustment unit (not shown) that controls the attitudes of the camera modules 0501 and 0502. As a result, the optical axis of the camera module 0501 coincides with the line of sight of the left eye 0301 in FIG. 3, and the optical axis of the camera module 0502 coincides with the line of sight of the right eye 0302 in FIG. 3. This state is shown in FIG. 7.

[0064] In FIG. 7 , there is a tree at the black X 0706 at the intersection of the optical axis of camera module 0701 and the optical axis of camera module 0702. There is also a person at the black circle 0703 in the foreground on the right. The dashed line connecting camera module 0702 and black circle 0703 intersects with dashed line 0712. Therefore, the person appears in the image captured by camera module 0702. This is indicated by gray circle 0705. If a person were actually located at the gray circle 0705, the image of the person in the image captured by camera module 0702 would be in focus. However, since the person is actually located at the black circle 0703, the image is blurred in proportion to the distance between the black circle 0703 and the gray circle 0705. Meanwhile, the dashed line connecting camera module 0701 and black circle 0703 does not intersect with dashed line 0711, but does intersect with the dashed-dotted line, which is an extension of dashed line 0711. This is indicated by the white grey circle 0704. Therefore, no person appears in the image captured by the camera module 0701.

[0065] Figure 8 shows an image captured by the camera module in Figure 7. Frame 0811 corresponds to dashed line 0711 in Figure 7, and frame 0812 corresponds to dashed line 0712 in Figure 7. Because there is a tree at the black X 0706 on the optical axis of camera module 0701 in Figure 7, an image of the tree 0806 is captured at the center of frame 0811 in Figure 8. Similarly, there is a black X 0706 on the optical axis of camera module 0702, and because there is a tree there, an image of the tree 0806 is captured at the center of frame 0812 in Figure 8. Because both tree images 0806 have parallax between camera module 0701 and camera module 0702, they are not the same image, even though they are both in the center of the frame. A person image 0805 is captured in the lower right corner of frame 0812, but it is blurred for the reasons mentioned above. This is indicated by the large, bold gray line. On the other hand, no person image is captured in frame 0811 for the reasons mentioned above.

[0066] FIG. 4 shows the captured image of FIG. 8 displayed on the display device of FIG. 3. The captured image of frame 0811 of FIG. 8 is displayed on display device 0313 of FIG. 3. The displayed image is frame 0413 of FIG. 4. Similarly, the captured image of frame 0812 of FIG. 8 is displayed on display device 0314 of FIG. 3. The displayed image is frame 0414 of FIG. 4. The observer sees the image of frame 0413 with their left eye 0301 and the image of frame 0414 with their right eye 0302, and the two images are combined in their minds. Frame 0421 shows this state. There is only one wooden figure 0406 in the center, but the wooden figure 0406 in frame 0413 and the wooden figure 0406 in frame 0414 overlap, which includes parallax, creating a three-dimensional appearance. Furthermore, the person's image 0405 is perceived only by the right eye 0302, and the sense of spatial distance from the wooden figure 0406 can be inferred in the observer's mind from the degree of blurring of the person's image.

[0067] Consider the case where an observer views a tree at the black X 0306 without the display device 0313, display device 0314, lens 0315, and lens 0316 in Figure 3. The lenses of the left eye 0301 and the right eye 0302 are each adjusted to a predetermined thickness in order to focus on the tree at the black X 0306. The tree image 0319 on the actual display device 0313 can be formed on the retina of the left eye 0301 while maintaining the predetermined thickness by adjusting the distance 0326 between the display device 0313 and the lens 0315. Similarly, the tree image 0320 on the actual display device 0314 can be formed on the retina of the right eye 0302 while maintaining the predetermined thickness by adjusting the distance 0327 between the display device 0314 and the lens 0316. When such adjustments are made, the focal plane perceived by the observer for the left eye 0301 is the dashed line 0311, and the focal plane perceived by the observer for the right eye 0302 is the dashed line 0312, reproducing the focal plane 0711 of the camera module 0701 and the focal plane 0712 of the camera module 0702 in Figure 7.

[0068] In a cross-reality system equipped with an HMD and a camera module, the optical axis of the camera module can be controlled in real time so that it faces in approximately the same direction as the line of sight of the observer using the HMD. Furthermore, in this cross-reality system, the image captured by the camera module is displayed in real time on the display device of the HMD. This alleviates the problem of eye strain that occurs with conventional HMDs.

[0069] While the example described here uses a camera module in addition to the HMD to exchange gaze information and image data in real time between the HMD and the camera module, a camera module is not required. In that case, the image displayed on the HMD's display device can be computer-generated in real time so that the focal planes 0109 and 0110 in Figure 1 (or the focal planes corresponding to the focal planes 0311 and 0312 in Figure 3, or more preferably the focal planes corresponding to the focal planes 4609 and 4610 in Figure 46 and the focal planes 4811 and 4812 in Figure 48) are reproduced in front of the observer using the HMD in accordance with the observer's gaze direction. This can alleviate the eye strain problem that occurs with conventional HMDs, just as when a camera module is used.

[0070] Furthermore, even when a camera module is used, the camera module may be located far enough from the observer to maintain real-time performance, or may be incorporated into the same HMD. The camera module may capture light of wavelengths invisible to the human eye. Alternatively, text or images may be overlaid on the image captured by the camera module and displayed on the HMD display device.

[0071] The following explains why the first embodiment alleviates the problem of eye fatigue that occurs with conventional HMDs. One of the causes of eye fatigue in conventional HMDs is that the focal planes 4209, 4210, 4411, and 4412 are not perpendicular to the observer's line of sight in the states shown in Figures 42 and 44. This creates an imbalance in the ease of focusing on the left and right sides of the line of sight, causing the observer to experience a sense of discomfort that differs from the cases shown in Figures 46 and 48, which represent actual observations on-site. In the case of the present embodiment shown in Figures 1 and 3, the focal planes 0109, 0110, 0311, and 0312 are perpendicular to the line of sight, as in the cases shown in Figures 46 and 48, eliminating this sense of discomfort. This also applies when the gaze is directed in a direction other than that shown in Figures 1 and 3.

[0072] The third cause of eye fatigue in conventional HMDs was that the gaze point was not at the center of the field of view. In Figures 42 and 44, the subject that the viewer wanted to see at the end of the gaze point was not at the center of the field of view, which differed from the case of actual on-site observation, where the subject that the viewer wanted to see was at the center of the field of view, as in Figures 46 and 48, and this caused discomfort to the viewer, resulting in eye fatigue. In the cases of Figures 1 and 3 of the first embodiment, the gaze point is at the center of the field of view, eliminating this sense of discomfort. The same applies when the gaze point is in a direction other than Figures 1 and 3.

[0073] A fourth cause of eye fatigue in conventional HMDs is the difference in the degree of parallax between objects that are located further back or closer than the object of interest in the line of sight and that are seen by the left and right eyes, compared to when actually observing the object in real-time. For example, in Figure 43, the parallax 4325 between distant trees 4307 and 4308 is small, so both are displayed in frame 4321. However, in Figure 47, which shows actual observation in real-time, the parallax 4725 between distant trees 4707 and 4708 is large, so only one is displayed in frame 4721. In Figure 2 of the first embodiment, the parallax 0225 between distant trees 0207 and 0208 is large, so only one is displayed in frame 0221, as in Figure 47.

[0074] The same can be said for Fig. 45. Because the parallax 4525 between nearby people 4504 and 4505 is small, both are captured in frame 4521. On the other hand, in Fig. 49, which shows actual on-site observation, the parallax 4925 between nearby people 4904 and 4905 is large, so only one of them is captured in frame 4921. In Fig. 4 of the first embodiment, because the parallax 0425 between nearby people 0405 and 0404 is large, only one of them is captured in frame 0421, as in Fig. 49.

[0075] Therefore, in the first embodiment, the sense of distance between the person and the tree felt by the observer is closer to that felt when observing the person in person. This reduces the sense of discomfort and eye strain that occurs with conventional HMDs.

[0076] A supplementary note on the first embodiment. In FIG. 1 , the distance between the left eye 0101 and the display device 0113 is the same as the distance between the right eye 0102 and the display device 0114. The distance 0126 between the display device 0113 and the lens 0115 is determined so that the focus of the left eye 0101 is aligned with the surface of the display device 0113, and the distance 0127 between the display device 0114 and the lens 0116 is determined so that the focus of the right eye 0102 is aligned with the surface of the display device 0114. While this focus adjustment method is easy to implement, the positional relationship between the left eye and the lens is not limited to this. That is, the distance between the eye and the lens may be fixed, and the focus may be adjusted by adjusting the distance between the eye and the display device, or both the distance between the eye and the lens and the distance between the eye and the display device may be adjusted. Furthermore, although FIG. 1 shows a single lens for each of the left and right eyes, multiple lenses may be used to correct optical aberrations, or a flat, light-transmitting member that transmits only certain wavelengths may be inserted between the lenses.

[0077] One state of a head-mounted display (hereinafter, HMD) of the second embodiment is shown in Figure 9. The second embodiment differs from the first embodiment in that the left-eye display device 0913 and the right-eye display device 0914 are curved rather than flat. The use of curved display devices as in the second embodiment makes it possible to reduce the size of the HMD. The distance between the display device 0913 in Figure 9 and the left eye 0901 is shorter than the distance between the display device 0113 in Figure 1 and the left eye 0101, and similarly, the distance between the display device 0914 and the right eye 0902 is shorter than the distance between the display device 0114 and the right eye 0102.

[0078] Figures 17 and 18 compare the cases of a flat display device and a curved display device. In Figure 17, the display surface of the display device 1713 is curved, and this curved surface is a hemisphere or a portion of a sphere. The shape and position of the lens (optical system) 1715 are designed to focus light emitted from the display device 1713 on the retina of the eye 1701. The results of ray tracking calculations show that light emitted from the center of the display device 1713 at angle 1733 enters the eye 1701 without being blocked by the pupil 1740 and forms an image on the fovea centralis of the retina. Similarly, the diagram shows that portions of light emitted from distances 1737 and 1736 from the center of the display device also enter the eye 1701 without being blocked by the pupil 1740 and form images at distances 1739 and 1738, respectively, from the fovea centralis on the retina.

[0079] On the other hand, in Figure 18, the display device 1813 is a flat plate. The shape and position of the lens 1815 are designed so that light emitted from the display device 1813 forms an image on the retina of the eye 1801. The result of ray tracking calculations shows that light emitting from the center of the display device 1813 and entering at angle 1833 enters the eye 1801 without being blocked by the pupil 1840 and forms an image on the fovea centralis of the retina. Similarly, the result shows that portions of light emitted from distances 1837 and 1836 from the center of the display device also enter the eye 1801 without being blocked by the pupil 1840 and form images on the retina at distances 1839 and 1838, respectively.

[0080] Planar size 1730 of display device 1713 in Figure 17 is the same as planar size 1830 of display device 1813 in Figure 18. However, HMD size 1723 in Figure 17 is smaller and more compact than HMD size 1823 in Figure 18. Furthermore, the diameter of lens 1715 is smaller than the diameter of lens 1815. In this way, Figure 17, which uses a curved display device, is advantageous in terms of making the HMD smaller.

[0081] Also, because angle 1733 is wider than angle 1833, even if display device 1713 and display device 1813 are display devices with the same brightness, the observer can observe a brighter image with the HMD in Fig. 17. Furthermore, because angle 1734 is wider than angle 1834, the HMD in Fig. 17 can provide a wider field of view. In this way, Fig. 17, which uses a curved display device, is advantageous in terms of making the HMD brighter and wider in field of view.

[0082] 17 and 18 show examples of simple optical systems consisting of a single lens, but the same effect can be obtained by using a lens system that combines multiple lenses in order to reduce optical aberrations such as chromatic aberration. Furthermore, as shown in FIGS. 19 and 20, using a curved display device is also advantageous when an observer using an HMD observes a nearby object. Consider the case where an observer observes an object at a distance 1941, as shown in FIG. 19. In this case, if the observer observes an object using an HMD consisting of a curved display device and lens 1915 for the left eye 1901, and a curved display device and lens 1916 for the right eye 1902, display device 1913 and display device 1914 will not collide. Therefore, the object can be observed. However, if the observer attempts to observe the object using an HMD consisting of a planar display device 2013 and lens 2015 for the left eye 2001, and a planar display device 2014 and lens 2016 for the right eye 2002, as shown in FIG. 20, display device 2015 and display device 2016 will collide. Therefore, only objects at a distance 2041, which is farther than distance 1941, can be observed.

[0083] In Figure 9, adjustment of the display device and the lens along six axes can be performed in the same manner as in the first embodiment. Also, as in the first embodiment, a line-of-sight detection unit (not shown) that detects the viewer's line of sight may be provided, and a mechanism for adjusting the position of the display module in accordance with the direction of the viewer's line of sight detected thereby may also be provided. Furthermore, as in the first embodiment, a mechanism for adjusting the position of the display module so that the viewer's line of sight coincides with the optical axis of the lens may also be provided. Therefore, as in the first embodiment, the angle θ formed between the line of sight and the display surface of the display device (the angle between the line of sight and the display surface at the intersection of the line of sight and the display surface) is approximately a right angle. The range of θ is within ±60 arc minutes, more preferably within ±6 arc minutes.

[0084] As described above, the second embodiment, like the first embodiment, has the effect of solving or alleviating the problems of the conventional technology. The first effect is to eliminate eye fatigue caused by the influence of optical aberrations, which occurs when the viewer's line of sight and the display device are not perpendicular to each other in conventional HMDs. The second effect is to eliminate eye fatigue caused by the viewer not being able to see the image they want to see in the center of their field of view, which occurs in conventional HMDs.

[0085] FIG. 13 corresponds to FIG. 5 described in the first embodiment. The difference between FIG. 13 and FIG. 5 is that camera module 1301 and camera module 1302 each have a built-in curved image sensor, indicated by the white line. FIG. 21 shows a camera module 2101 incorporating a curved image sensor capturing an image of an object on a focal plane 2109 250 mm away. The imaging surface of the image sensor incorporated in camera module 2101 is curved, and this curved surface is a hemisphere or a portion of a sphere. The shape and position of lens 2115 are designed to focus light emitted from the focal plane 2109 onto the imaging surface of the curved image sensor. Light emitting from the center of focal plane 2109 and entering at angle 2133 enters camera module 2101 without being blocked by the aperture of lens 2115. As a result, an image is formed at the center of the imaging surface of the curved image sensor. Similarly, some of the light emitted from positions at distances 2137 and 2136 from the center of focal plane 2109 also enters camera module 2101 without being blocked by the aperture of lens 2115. As a result, images are formed at distances 2139 and 2138, respectively, from the center of the imaging surface of the curved image sensor.

[0086] The image information received by the curved image sensor in Fig. 21 is then transferred to the display device 1713 in Fig. 17 by a transfer device (not shown) and displayed upside down and left to right. The relationship between display positions is as follows: pixel information at the center point of the image sensor in Fig. 21 is displayed by the center pixel of the display device in Fig. 17. Information about a pixel located at a distance 2139 from the center point of the image sensor in Fig. 21 is displayed by a pixel located at a distance 1737 from the center pixel of the display device in Fig. 17. Similarly, information about a pixel located at a distance 2138 is displayed by a pixel located at a distance 1736.

[0087] As a result, the image formed on the retina of eye 1701 of an observer using the HMD in Fig. 17 is the same as the image formed when the observer is observing in person. In other words, when Fig. 21 is regarded as a diagram showing an observer observing an object 250 mm ahead in person, the image formed on the retina of eye 2101 is the same as the image formed on the retina of eye 1701 of an observer using the HMD in Fig. 17. This gives the observer using the HMD in Fig. 17 the feeling that they are observing an object 250 mm ahead in person, as in Fig. 21.

[0088] While the case where an object 250 mm away is photographed and the image is observed using an HMD has been described, the same applies to distances other than 250 mm. For example, even if the focal plane 2109 is at infinity in Figure 21, by shifting the lens 2115 slightly closer to the image sensor, it is possible to form an image of light emitted from each point on the focal plane 2109 on the curved image sensor. In this state, the image information received by the curved image sensor is inverted and displayed on the curved display device 1713 in Figure 17, and when observing using the HMD, by shifting the lens 1715 closer to the eye 1701, the observer will feel as if they are observing an object at infinity in real time.

[0089] 13 is a diagram showing how a left-eye camera module 1301 is used to capture an image to be displayed on the display device 0913 in FIG. 9, and how a right-eye camera module 1302 is used to capture an image to be displayed on the display device 0914 in FIG. 9. A mechanism (not shown) may be provided that receives gaze detection information of the left eye 0901 in FIG. 9 and controls the attitude of the left-eye camera module 1301 so that the optical axis of the camera module 1301 faces the same direction as the gaze of the left eye 0901. Similarly, a mechanism (not shown) may be provided that controls the attitude of the right-eye camera module 1302 so that the optical axis of the right-eye camera module 1302 coincides with the gaze of the right eye 0902.

[0090] Dashed lines 1309 and 1310 indicate the object-side focal planes of camera module 1301 and camera module 1302, respectively. Therefore, an image of an object on dashed line 1309 is formed on the imaging surface of the image sensor indicated by the white line through the lens of camera module 1301. Similarly, an image of an object on dashed line 1310 is formed on the imaging surface of the image sensor indicated by the white line through the lens of camera module 1302.

[0091] In Figure 13, there is a person at black circle 1303, which is the intersection of the optical axis of camera module 1301 and the optical axis of camera module 1302. There is also a tree at black X 1306 in the distance to the left. The dashed line 1310 connecting camera module 1302 and black X 1306 intersects with dashed line 1308. Therefore, the tree appears in the image captured by camera module 1302. This is indicated by gray X 1308. If there was actually a tree at the position of gray X 1308, the image of the tree in the image captured by camera module 1302 would be in focus, but because the tree is actually at the position of black X 1306, the image is blurred in proportion to the distance between black X 1306 and gray X 1308. On the other hand, the dashed-dotted line connecting camera module 1301 and black X 1306 does not intersect with dashed line 1309, but does intersect with a dashed-dotted line that is an extension of dashed line 1309. This is indicated by gray dotted X 1307. Therefore, the tree does not appear in the image captured by camera module 1301.

[0092] FIG. 14 shows an image captured by the camera module of FIG. 13. Frame 1409 corresponds to dashed line 1309 in FIG. 13, and frame 1410 corresponds to dashed line 1310 in FIG. 13. In FIG. 13, a person is located at black circle 1303 on the optical axis of camera module 1301, so person image 1403 is captured at the center of frame 1409 in FIG. 14. Similarly, black circle 1303 is located on the optical axis of camera module 1302, and a person is located there, so person image 1403 is captured at the center of frame 1410 in FIG. 14. Because both person images 1403 have parallax between camera module 1301 and camera module 1302, they are not the same image, even though they are both in the center of the frame. A tree image 1408 is captured in the upper left corner of frame 1410, but for the reasons mentioned above, it is a blurred image. This is indicated by a thick gray line. On the other hand, frame 1409 does not show the tree statue for the reasons mentioned above.

[0093] FIG. 10 is a diagram showing the captured image of FIG. 14 displayed on the display device of FIG. 9. The captured image of frame 1409 of FIG. 14 is displayed on display device 0913 of FIG. 9. The displayed image is frame 1013 of FIG. 10. Similarly, the captured image of frame 1410 of FIG. 14 is displayed on display device 0914 of FIG. 9. The displayed image is frame 1014 of FIG. 10. The viewer sees the image of frame 1013 with their left eye 0901 and the image of frame 1014 with their right eye 0902, so that the two images are combined in their minds. Frame 1021 shows this state. There is only one central person image 1003, but the person image 1003 of frame 1013, which includes parallax, and the person image 1003 of frame 1014 are superimposed, giving the impression of three-dimensionality. Furthermore, the tree statue 1008 can be perceived only by the right eye 0902, and the degree of blurring of the tree statue allows the sense of spatial distance between the tree statue 1008 and the person statue 1003 to be estimated in the mind.

[0094] 9, let us consider a case in which the observer views a person at the black circle 0903, without the display devices 0913, 0914, lenses 0915, and 0916. In this case, the lenses of the left eye 0901 and the right eye 0902 are each adjusted to a predetermined thickness in order to focus on the person at the black circle 0903. It is possible to form an image of a person 0917 on the actual display device 0913 on the retina of the left eye 0901 while maintaining the predetermined thickness by adjusting the distance 0926 between the display device 0913 and the lens 0915. Similarly, it is possible to form an image of a person 0918 on the actual display device 0914 on the retina of the right eye 0902 while maintaining the predetermined thickness by adjusting the distance 0927 between the display device 0914 and the lens 0916. When such adjustments are made, the focal plane perceived by the observer for the left eye 0901 is the dashed line 0909, and the focal plane perceived by the observer for the right eye 0902 is the dashed line 0910, reproducing the focal plane 1309 of the camera module 1301 and the focal plane 1310 of the camera module 1302 in Figure 13.

[0095] Next, when the observer's attention shifts from the person to a distant tree 1106 as shown in Figure 11, the line of sight of the left eye 1101 shifts to the dashed-dotted line connecting the left eye 1101 and the black X 1106. Similarly, the line of sight of the right eye 1102 shifts to the dashed-dotted line connecting the right eye 1102 and the black X 1106. This change in line of sight can be detected by the line of sight detection unit (not shown) and transmitted to a mechanism (not shown) that controls the attitudes of the camera module 1301 and the camera module 1302. As a result, Figure 15 shows a state in which the optical axis of the camera module 1301 coincides with the line of sight of the left eye 1101 in Figure 11, and the optical axis of the camera module 1302 coincides with the line of sight of the right eye 1102 in Figure 11.

[0096] In FIG. 15 , there is a tree at black X 1506, which is located at the intersection of the optical axis of camera module 1501 and the optical axis of camera module 1502. There is also a person in black circle 1503 in the foreground on the right. The dashed line connecting camera module 1502 and black circle 1503 intersects with dashed line 1512. Therefore, the person appears in the image captured by camera module 1502. This is indicated by gray circle 1505. If a person were actually located at gray circle 1505, the image of the person in the image captured by camera module 1502 would be in focus. However, because the person is actually located at black circle 1503, the image is blurred in proportion to the distance between black circle 1503 and gray circle 1505. Meanwhile, the dashed line connecting camera module 1501 and black circle 1503 does not intersect with dashed line 1511, but it does intersect with the dash-dotted line, which is an extension of dashed line 1511. This is indicated by a white gray circle 1504. Therefore, no person appears in the image captured by the camera module 1501.

[0097] FIG. 16 shows an image captured by the camera module of FIG. 15. Frame 1611 corresponds to dashed line 1511 in FIG. 15, and frame 1612 corresponds to dashed line 1512 in FIG. 15. Because there is a tree at black X 1506 on the optical axis of camera module 1501 in FIG. 15, an image of tree 1606 is captured at the center of frame 1611 in FIG. 16. Similarly, there is a black X 1506 on the optical axis of camera module 1502, and because there is a tree there, an image of tree 1606 is captured at the center of frame 1612 in FIG. 16. Because both tree images 1606 have parallax between camera module 1501 and camera module 1502, they are not the same image, even though they are both in the center of the frame. A person's image 1605 is captured in the lower right corner of frame 1612, but for the reasons mentioned above, it is blurred. This is indicated by the large, bold gray line. On the other hand, frame 1611 does not show any people for the reasons mentioned above.

[0098] FIG. 12 is a diagram showing the captured image of FIG. 16 displayed on the display device of FIG. 11. The captured image of frame 1611 of FIG. 16 is displayed on display device 1113 of FIG. 11. The displayed image is frame 1213 of FIG. 12. Similarly, the captured image of frame 1612 of FIG. 16 is displayed on display device 1114 of FIG. 11. The displayed image is frame 1214 of FIG. 12. The viewer sees the image of frame 1213 with left eye 1101 and the image of frame 1214 with right eye 1102, so the two are combined in the viewer's mind. Frame 1221 shows this state. There is only one wooden figure 1206 in the center, but the wooden figure 1206 of frame 1213 and the wooden figure 1206 of frame 1214, which include parallax, overlap, giving the viewer a three-dimensional appearance. Furthermore, the figure 1205 can only be perceived by the right eye 1102, and the degree of blur of the figure allows the viewer to mentally estimate the spatial distance between the figure and the wooden statue 1206.

[0099] 11, let us consider a case in which a viewer looks at a tree at black X 1106 without display device 1113, display device 1114, lens 1115, and lens 1116. In this case, the lenses of left eye 1101 and right eye 1102 are each adjusted to a predetermined thickness in order to focus on the tree at black X 1106. It is possible to form an image of tree image 1119 on display device 1113, which actually exists, on the retina of left eye 1101 while maintaining the predetermined thickness by adjusting distance 1126 between display device 1113 and lens 1115. Similarly, it is possible to form an image of tree image 1120 on display device 1114, which actually exists, on the retina of right eye 1102 while maintaining the predetermined thickness by adjusting distance 1127 between display device 1114 and lens 1116. When such adjustments are made, the focal plane perceived by the observer for the left eye 1101 is dashed line 1111, and the focal plane perceived by the observer for the right eye 1102 is dashed line 1112, reproducing the focal plane 1311 of camera module 1301 and the focal plane 1312 of camera module 1302 in Figure 13.

[0100] In a cross reality system equipped with an HMD and a camera module, the optical axis of the camera module is controlled in real time so that it faces in approximately the same direction as the line of sight of the observer using the HMD. Furthermore, in this system, images captured by the camera module are displayed in real time on the HMD's display device. This reduces the eye strain problem that occurs with conventional HMDs.

[0101] While an example has been given here in which a camera module is used in addition to the HMD to exchange gaze information and image data in real time between the HMD and the camera module, a camera module is not necessary. In that case, the image displayed on the HMD's display device can be computer-generated in real time so that the focal planes 0909 and 0910 in Figure 9 (or the focal planes corresponding to 1111 and 1112 in Figure 11, or more preferably the focal planes corresponding to 469 and 4610 in Figure 46 and 4811 and 4812 in Figure 48) are reproduced in front of the observer using the HMD in accordance with the observer's gaze direction. This can alleviate the eye strain problem that occurs with conventional HMDs, just as when a camera module is used.

[0102] Furthermore, even when a camera module is used, the camera module may be located far enough from the observer to maintain real-time performance, or may be incorporated into the same HMD. The camera module may capture light of wavelengths invisible to the human eye. Alternatively, text or images may be overlaid on the image captured by the camera module and displayed on the HMD display device.

[0103] The reason why the second embodiment alleviates the problem of eye fatigue that occurs with conventional HMDs will be explained below. One of the causes of eye fatigue in conventional HMDs is that in the states of FIGS. 42 and 44, the focal planes 4209, 4210, 4411, and 4412 are not perpendicular to the observer's line of sight. This creates an imbalance in the ease of focusing on the left and right sides of the line of sight, causing the observer to experience a sense of discomfort that differs from the cases of FIGS. 46 and 48, which show actual observations on-site. In the case of the present embodiment shown in FIGS. 9 and 11, the focal planes 0909, 0910, 1111, and 1112 are perpendicular to the line of sight, as in the cases of FIGS. 46 and 48, so this sense of discomfort is eliminated. This also applies when the line of sight is directed in a direction other than that shown in FIGS. 9 and 11.

[0104] The third cause of eye fatigue in conventional HMDs was that the gaze point was not at the center of the field of view. In Figures 42 and 44, the subject that the viewer wanted to see at the end of the gaze point was not at the center of the field of view, which differed from the case of actual on-site observation, where the subject that the viewer wanted to see was at the center of the field of view, as in Figures 46 and 48, and this caused the viewer to feel uncomfortable and was a cause of eye fatigue. In the cases of Figures 9 and 11 of the second embodiment, the gaze point is at the center of the field of view, so this sense of discomfort is eliminated. The same applies when the gaze point is in a direction other than Figures 9 and 11.

[0105] A fourth cause of eye fatigue in conventional HMDs is the difference in the degree of parallax between objects that are located further back or closer than the object being focused on and that are visible to the left and right eyes, compared to when actually observing the object in real-time. For example, in Figure 43, the parallax 4325 between distant trees 4307 and 4308 is small, so both trees are displayed in frame 4321. However, in Figure 47, which shows actual observations in real-time, the parallax 4725 between distant trees 4707 and 4708 is large, so only one tree is displayed in frame 4721. In Figure 10 of the second embodiment, the parallax 1025 between distant trees 1007 and 1008 is large, so only one tree is displayed in frame 1021, as in Figure 47.

[0106] The same can be said for Fig. 45. Because the parallax 4525 between nearby people 4504 and 4505 is small, both of them are captured in frame 4521. On the other hand, in Fig. 49, which shows actual observation at the site, the parallax 4925 between nearby people 4904 and 4905 is large, so only one of them is captured in frame 4921. In Fig. 12 of the second embodiment, because the parallax 1225 between nearby people 1205 and 1204 is large, only one of them is captured in frame 1221, as in Fig. 49.

[0107] Therefore, in the second embodiment, the sense of distance between the person and the tree perceived by the observer is closer to that felt when observing the person in person. This reduces the sense of discomfort and eye strain that occurs with conventional HMDs.

[0108] Furthermore, the second embodiment also eliminates the second cause of eye fatigue in conventional HMDs. The second cause is the difference in shape between the focal plane when actually observing on-site and the focal plane perceived by the observer when experiencing an HMD. However, in the second embodiment, the situation in Figure 9, which shows observation using an HMD, matches the situation in Figure 46, which shows what the observer actually sees on-site.

[0109] That is, the distance between the black X 0906 where a tree is present and the tree image 0908 on the focal plane 0910 in Fig. 9 is the same as the distance between the black X 4606 where a tree is present and the tree image 4608 on the focal plane 4610 in Fig. 46. Therefore, if the depth of field of the right-eye camera 1302 and the depth of field of the observer's eye 4602 are approximately the same, the tree image 0908 and the tree image 4608 will have the same degree of blur. This is clearly demonstrated by the fact that the line thickness of the tree image 1008 in Fig. 10 and the tree image 4708 in Fig. 47 are the same.

[0110] The same can be said when comparing the case of Fig. 11 with Fig. 48. That is, the distance between the black circle 1103 where the person is located and the person image 1105 on the in-focus plane 1112 in Fig. 11 is the same as the distance between the black circle 4803 where the person is located and the person image 4805 on the in-focus plane 4812 in Fig. 48. Therefore, if the depth of field of the right-eye camera 1502 and the depth of field of the observer's eye 4802 are approximately the same, the degree of blur of the person image 1105 and the person image 4805 will be the same. This is clearly shown by the fact that the line thickness of the person image 1205 in Fig. 12 and the person image 4905 in Fig. 49 are the same.

[0111] As described above, the second embodiment is preferable to the first embodiment in that it also eliminates the second cause of eye fatigue that occurs in conventional HMDs.

[0112] An example embodying the second embodiment will be described below as Example 1. FIG. 22 shows a manufacturing method for a curved display device and a curved image sensor according to Example 1. Pixels 2245 may be arranged at a regular pitch on the surface of a planar device 2244, which is a planar display device or a planar image sensor. In one example, the planar shape of the pixels may be hexagonal, forming a planar tessellation structure like a honeycomb. In one example, the pixel size is 3 μm. Examples of usable planar display devices include liquid crystal displays and organic electroluminescence (EL) displays. Examples of usable planar image sensors include CMOS image sensors, CCD sensors, and SPAD sensors.

[0113] A fiber optical plate (hereinafter, FOP) was prepared, specifically an FOP 2242 with one flat surface and the other spherical surface. The fiber 2243 constituting the FOP may have a diameter of, for example, 3 μm. As shown in the right diagram of Figure 22, the FOP 2242 and a planar device 2244 are bonded together with adhesive 2246 to form a curved device, such as a curved display device or a curved image sensor. A known high-precision die bonder can be used for bonding, for example, to align the center of the pixel 2245 with the center of the fiber 2243. In one example, the bonding accuracy may be ±0.3 μm.

[0114] 23 is a diagram illustrating the HMD of this example, a gaze detection unit that detects the viewer's gaze, and a drive mechanism that drives the optical axis of the display module in accordance with the gaze detected thereby. The HMD of this example is provided with a support 2347 having a support surface formed of a part of a sphere, and the center of the sphere can be positioned so as to approximately coincide with the rotation center of the viewer's eye 2301 (rotation center of gaze movement). Here, "approximately coincident" means within ±1 mm of the rotation center of the eye 2301, more preferably within 0.1 mm.

[0115] The curved display device 2313 may have a back surface opposite to its display surface, which has a spherical shape with a curvature slightly smaller than the curvature of the spherical surface of the support 2347. A Halbach array 2349 of permanent magnets may be fixed to the back surface of the display device 2313. Meanwhile, an electromagnet array 2350 may be provided on the support surface of the support 2347. By controlling the electromagnet array 2350 so that it is always attracted to the Halbach array 2349, the display device 2313 can be always attracted to the support 2347. Meanwhile, a bearing 2348 may be provided to resist the force attracting the display device 2313. The bearing 2348 allows the display device 2313 to move along the support surface of the support 2347. The magnetic field generated by the electromagnet array 2350 is controlled to move the display device 2313. As a result, the display device 2313 is driven up and down and left and right along the spherical surface of the support 2347.

[0116] The Halbach array 2349 and the electromagnet array 2350 may be interchanged, i.e., the Halbach array 2349 may be provided on the display device 2313 and the electromagnet array 2350 may be provided on the support 2347.

[0117] With the optical axis of the display device 2313 and the optical axis of the lens (optical system) 2315 aligned, the lens 2315 can be driven by a drive system (not shown) along the optical axis of the lens 2315. The five-axis adjustment precision for aligning the optical axis of the display device 2313 and the optical axis of the lens 2315 can be, for example, ±4 μm for translation and ±4 arc minutes for angle.

[0118] An infrared mirror (half mirror) 2353 may be provided between the display device 2313 and the lens 2315, intersecting the optical axis at 45°. The infrared mirror 2353 may be configured by providing an optical filter on the surface of a light-transmitting member so as to transmit visible light and reflect infrared light. An infrared LED (irradiation unit) 2351 may be provided to irradiate the pupil of the eye 2301 with infrared light. Of the infrared light irradiated onto the pupil by the infrared LED 2351, light reflected by the pupil toward the lens 2315 may be slightly condensed by the lens 2315 and reflected by the infrared mirror 2353. The infrared light reflected by the infrared mirror 2353 passes through an infrared lens 2354 and may form an image of the pupil of the eye 2301 on the imaging surface of the infrared image sensor 2355. The infrared image sensor 2355 may constitute a gaze detection unit that detects the viewer's gaze. Here, the infrared lens 2354 can be driven by a drive system (not shown) along the optical axis of the infrared image sensor 2355. The drive system can be controlled so that a pupil image is always formed on the infrared image sensor 2355 even when the lens 2315 moves.

[0119] The Halbach array 2349, bearing 2348, display device 2313, lens 2315, infrared LED 2351, infrared mirror 2353, infrared lens 2354, infrared image sensor 2355, etc. may constitute an integrated display module DM. The module M may be driven by a drive mechanism D to rotate around a center point substantially identical to the center of rotation of the eye 2301. Rotation of the eye 2301, i.e., a change in the line of sight, appears as a movement of the pupil image on the imaging surface of the infrared image sensor 233. Therefore, the control unit CNT can detect the movement of the observer's line of sight by detecting the speed and direction of the movement of the pupil image on the imaging surface of the infrared image sensor 233 based on the output of the infrared image sensor 233. The control unit CNT can also control the drive mechanism D so that the module M follows the movement of the observer's line of sight. This allows the angle θ formed between the observer's line of sight and the display surface of the display device 2313 to be maintained substantially perpendicular at the intersection of the line of sight and the display surface of the display device 2313. In other words, the control unit CNT can control the drive mechanism D based on the output of the infrared image sensor 2355 (gaze detection unit) so that the angle between the viewer's gaze and the display surface of the display device 2313 falls within a preset tolerance range with respect to 90 degrees. The tolerance range can be determined arbitrarily according to the required specifications. The tolerance range may be, for example, within ±60 arc minutes of 90 degrees, and is preferably within ±6 arc minutes of 90 degrees.

[0120] 24 is a diagram illustrating a mechanism for quantifying the direction of the viewer's line of sight and obtaining information for controlling the attitude of the camera module. A two-dimensional pattern 2456 having contrast that can be used as a scale in a two-dimensional encoder may be provided on the spherical surface of the support 2447. The encoder sensor 2457 may have an infrared LED and an infrared sensor array (not shown) inside. Infrared light 2458 is irradiated from the infrared LED of the encoder sensor 2457 toward the two-dimensional pattern 2456, and the infrared reflected light from the two-dimensional pattern 2456 can be detected by the infrared sensor array of the encoder sensor 2457.

[0121] The module M, which includes the encoder sensor 2457, display device 2413, lens 2415, infrared mirror 2453, infrared lens 2454, and infrared image sensor 2455, can rotate around a central point that is approximately the same as the center of rotation of the eye 2401, as shown in FIG. 24. The center of rotation of the eye 2401 can also be said to be the center of rotation of the line of sight movement. The control unit CNT can detect the angle of the line of sight by counting changes in the infrared reflected light from the two-dimensional pattern 2456 using the encoder sensor 2457. A real-time cross reality system can be constructed by feeding back the detected line of sight direction to the attitude control of the camera module. Next, the operation of this real-time cross reality system will be described.

[0122] 25 shows a state in which an observer is observing a scene at infinity using the HMD of this example. The line of sight of the left eye 2501 and the line of sight of the right eye 2502 are parallel, and the line of sight of the left eye 2501 coincides with the optical axis of a display module consisting of a display device 2513 and a lens 2515. Similarly, the line of sight also coincides with the optical axis of a display module consisting of a display device 2514 and a lens 2516.

[0123] Figure 26 shows a state in which the observer's attention is directed toward an object located 7° to the right and 250 mm in front. At this time, the lines of sight of the left eye 2601 and the right eye 2602 are each turned 8.6° inward from the parallel state shown in Figure 25, resulting in a cross-eyed state. Changes in the observer's lines of sight are detected using the line-of-sight detection unit described above, and the line of sight of the left eye 2601 is aligned with the optical axis of the module including the display device 2613 and lens 2615 by the drive mechanism D described above. Similarly, the line of sight of the right eye 2602 is aligned with the optical axis of the display module including the display device 2614 and lens 2616. At this time, the 7° rotation angle of the observer's head can be measured by a gyro sensor (not shown). The line of sight angles of the left eye 2601 and the right eye 2602 can be measured by a mechanism using the encoder sensors described above.

[0124] Fig. 27 shows a camera module system that captures images to be displayed on the display device 2513 and display device 2514 that the observer is looking at in Fig. 25. The image captured by the left-eye camera module 2701 is displayed on the display device 2513 in Fig. 25, and the image captured by the right-eye camera module 2702 is displayed on the display device 2514 in Fig. 25. At this time, the direction of the optical axis of the left-eye camera module 2701 coincides with the direction of the line of sight of the left eye 2501 in Fig. 25, and the direction of the optical axis of the right-eye camera module 2702 coincides with the direction of the line of sight of the right eye 2502 in Fig. 25. Furthermore, the direction of the head of the observer in Fig. 25 and the direction of the entire camera module system in Fig. 27 coincide with the horizontal left direction in the figure.

[0125] When the observer's state changes from FIG. 25 to FIG. 26, the rotation angle of the observer's head and the change in the angle of the line of sight of the left and right eyes are transmitted in real time to the camera module system in the state of FIG. 27 by a transmission device (not shown), and the direction of the camera modules can be controlled. Therefore, the state of the camera module system changes from the state of FIG. 27 to the state of FIG. 28. That is, the entire camera module system rotates 7° to the right, and the optical axis directions of the right-eye camera module 2701 and the left-eye camera module 2702 each point 8.6° inward. As a result, the line of sight of the observer's left eye 2601 and the optical axis direction of the left-eye camera module 2801 again coincide, and the line of sight of the right eye 2602 and the optical axis direction of the right-eye camera module 2802 again coincide.

[0126] Next, we will explain the control of the lens system of the real-time cross reality system of this example. Figure 29 is an enlarged view of the camera module in Figure 27 when the focal plane is at infinity. Ray calculations show how parallel light from infinity passes through lens 2915 and focuses on the curved image sensor. Figure 30 is an enlarged view of the camera module in Figure 28 when the focal plane is at 250 mm in the vicinity. Ray calculations show how diffused light emitted from an arc-shaped focal plane (not shown) 250 mm away passes through lens 3015 and focuses on the curved image sensor. Changing from the state in Figure 29 to the state in Figure 30 is achieved by shifting lens 2915 from its position to its position using a known autofocus mechanism. Alternatively, the distance L between the observed object and the observer can be calculated from the angle η between the left eye's line of sight and the right eye's line of sight detected by the HMD, and the amount of shift for lens 2915 can be determined.

[0127] FIG. 31 is an enlarged view of the display module in FIG. 25 when the observer is observing infinity. An image captured by the camera module 2901 in FIG. 29 is displayed in real time on the display device 3113. The image is depicted as a result of ray calculation, depicting diffused light emitted from the surface of the display device 3113 passing through the lens 3115 and the crystalline lens 3159 and forming an image on the retina of the eye 3101. At this time, the position of the lens 3115 is controlled so that the thickness of the crystalline lens 3159 is the same as when the observer is actually observing infinity. The position of the infrared lens 3154 is adjusted to match the position of the lens 3115, and an image of the pupil of the eye 3101 is formed on the infrared image sensor 3155. In other words, a control unit (not shown) can adjust the lens 3115 based on the convergence angle of the observer's eyes so that the thickness of the crystalline lens of the eye looking at the display surface of the display device 3213 matches the thickness of the crystalline lens of the eye when the observer views an object at a convergence distance corresponding to the convergence angle.

[0128] Figure 32 is an enlarged view of the display module when the observer in Figure 26 is observing an object located 250 mm from the camera module. The amount of shift of lens 2915 when the camera module changes from the state in Figure 29 to the state in Figure 30 is transmitted to the HMD system in Figure 31 and fed back in real time to the position control of lens 3115. Alternatively, the distance L between the observed object and the observer may be calculated from the angle η formed by the line of sight of the observer's left eye and right eye detected by the HMD, and fed back to the position control of lens 3115. In accordance with the feedback, the position of lens 3115 shifts from the position of lens 3115 to the position of lens 3215 when changing from the state in Figure 31 to the state in Figure 32.

[0129] As a result, when diffused light emitted from the surface of the display device 3213 passes through the lens 3215 and enters the eye 3201, the thickness of the lens 3159 in the observer's eye 3101 is controlled to be thicker. This is because, if this is not done, the diffused light will not form an image on the retina. The thickness of the lens 3259 in FIG. 32 is the thickness required for the observer to actually see at a distance of approximately 250 mm. Conversely, the amount of shift from the previous position of the lens 3115 to the position of the lens 3215 is calculated and determined so that the thickness of the lens 3259 becomes the thickness shown in FIG. 32. Furthermore, the position of the infrared lens 3154 is shifted to the position of the infrared lens 3254 in accordance with the shift of the lens 3115 so that the focus of the pupil image on the infrared image sensor 3255 does not shift. By carrying out the above-described control, the viewer can view the display without feeling eye strain, just as if he or she were actually viewing the display on-site.

[0130] Image transfer from the camera module in Figure 27 to the display device in Figure 28, or from the camera module in Figure 28 to the display device in Figure 26, is not a problem when the camera module and the viewer are close to each other. However, in remote locations, transfer takes time, which can reduce real-time performance. Therefore, in this example, as shown in Figure 33, playback processing can be performed by thinning pixel information and rendering using AI, as shown in Figure 33. Specifically, Figure 33 (a) shows an image captured by the camera module, and (b) shows an image in which the information on the surrounding pixels has been thinned out by an image signal processor near the camera module. Transferring the image from the camera module to the display device in this state reduces the amount of information, thereby reducing transfer time. In other words, by transferring an image obtained using the front image sensor of the camera module with the peripheral areas of the image made coarser than the center, transfer time can be reduced by transferring the image from the camera module to the display device. Furthermore, the more important portion observed by the fovea of ​​the eye remains intact, thereby minimizing the effects of thinning. (c) shows image information received by an AI processor near the display device, and (d) shows an image displayed on the display device after the AI ​​processor interpolates the pixel information thinned out by AI processing. Building a system that performs such processing is preferable in order to realize a cross reality system that allows users to experience images from remote locations.

[0131] Hereinafter, Example 2 will be described. Differences between Example 2 and Example 1 will be described with reference to Figs. 34 and 35. Fig. 34 is a diagram showing the HMD of Example 1. Halbach arrays 3449 are located in the line of sight of left eye 3401 and right eye 3402, respectively, and are configured so that they cannot protrude from their respective supports 3447. Therefore, in Example 1, the rotation angles of left eye 3401 and right eye 3402 are limited to ±18°.

[0132] On the other hand, FIG. 35 is a diagram showing an HMD according to a second embodiment. The Halbach arrays 3549 are not located in the line of sight of the left eye 3501 and the right eye 3502, but are offset. Therefore, like the left-eye display device 3513 in FIG. 35, it is possible for the display device 3513 to protrude outward from the right end of the left-eye support 3547. From another perspective, the movable range of the display device 3513 includes an area outside the area of ​​the support surface (Halbach array 3549) that supports the display device. Therefore, in the second embodiment, the rotation angles of the left eye 3501 and the right eye 3502 are allowed within a range of ±30°. The allowable range of the rotation angles can be further widened by adjusting the offset amount, and the configuration of the second embodiment has the effect of widening the field of view of the HMD.

[0133] Example 3 will be described below. Example 3 differs from Example 1 in that a support used in a drive system of a display device using a Halbach array is a support 3647 having a support surface formed by a plane, rather than a support 3447 having a support surface formed by a portion of a sphere. The configuration of Example 3 will be described with reference to FIG. 36 . An angle control mechanism 3660 can be provided to keep the angle θ between the viewer's line of sight and the display surface of the display device perpendicular. The angle control mechanism 3660 controls the display device 3613 according to the position of the support 3647 in the Halbach array 3649 so that the angle θ between the line of sight of the eye 3601 and the display surface of the display device 3613 is perpendicular at the intersection of the line of sight of the eye 3601 and the display surface of the display device 3613. At this time, unlike Example 1, since the distance between the eye 3601 and the display surface of the display device 3613 changes, focus adjustment is performed, for example, by moving the lens 3615 in the line of sight direction.

[0134] Example 4 will be described below. Example 4 differs from Example 1 only in the manufacturing method of the curved sensor and curved display device. This method is described with reference to FIG. 37 . A flat wafer process can be used as a wafer process for manufacturing the curved sensor and display device of Example 4. The wafer that has undergone the wafer process is thinned to, for example, 50 μm and singulated into individual sensors and devices. Next, a mold 3761 having a curved surface with a curvature radius of, for example, 400 mm is prepared, and singulated image sensors or display devices 3744 can be placed on the curved surface of the mold 3761 as shown in FIG. 37A. In this state, the closed space formed by the image sensor or display device 3744 and the mold 3761 can be evacuated, thereby bending the image sensor or display device 3744 into a curved shape as shown in FIG. 37B. In this state, the image sensor or display device 3744 can be fixed in its curved shape by adhering the image sensor or display device 3744 to the mold 3761 with an adhesive (not shown).

[0135] Example 5 will be described below. Example 5 differs from Example 4 only in the following respects. In Example 4, the radius of curvature of the curved surface was, for example, 400 mm. In Example 4, the radius of curvature is, for example, 10 mm. Therefore, as shown in FIG. 38 , three square or six trapezoidal planar image sensors or display devices can be bent and connected to form a single curved image sensor or display device 3844. In this case, each image sensor or display device can be composed of surface elements 3862 and spring bridges 3863 connecting the surface elements 3862.

[0136] The thickness of each image sensor or display device can be 10 μm. Thinning can be achieved using a known SOI (Silicon On Insulator) wafer. First, the entire thickness of the Si layer on the oxide film (insulator) can be patterned by deep etching to form surface elements 3862 and spring bridges 3863. Then, the portion above the oxide film can be separated from the Si substrate by etching using XeF2 gas, completing the thinning. The size of each surface element 3862 can be, for example, 100 μm square, and a structure in which, for example, 3 μm square pixels are packed within the surface elements can be adopted. This allows for the production of an image sensor and display device with a smaller radius of curvature than in Example 4.

[0137] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0138] 2313: Display device, 2315: Lens (optical system), D: Drive mechanism, C: Control unit, A: Adjustment unit, DM: Display module, 2301: Eye

Claims

1. A head-mounted display comprising: a display device having a display surface formed of a part of a sphere; and an optical system for focusing an observer's eyes on the display device, a drive mechanism for driving the display device; a support section having a support surface formed of a part of a sphere and supporting the display device on the support surface; a line-of-sight detection section that detects the line of sight of the viewer; a control unit that controls the drive mechanism based on an output of the line-of-sight detection unit so that an angle formed between the line of sight and the display surface at a point where the line of sight intersects with the display surface falls within a predetermined tolerance range with respect to 90 degrees; A head-mounted display comprising:

2. A head-mounted display comprising: a display device having a display surface; and an optical system for focusing an observer's eyes on the display device, a drive mechanism for driving the display device; a support section having a support surface formed of a part of a sphere and supporting the display device on the support surface; a line-of-sight detection section that detects the line of sight of the viewer; a control unit that controls the drive mechanism based on an output of the line-of-sight detection unit so that an angle formed between the line of sight and the display surface at a point where the line of sight intersects with the display surface falls within a predetermined tolerance range with respect to 90 degrees, The support surface is configured such that the center of the spherical surface constituting the support surface substantially coincides with the rotation center of the line of sight movement. A head-mounted display characterized by:

3. the drive mechanism is configured to drive the display device along the support surface; 3. The head-mounted display according to claim 1 or 2.

4. the drive mechanism includes a Halbach array; 4. The head-mounted display according to claim 3.

5. The movable range of the display device includes an area outside the area of ​​the support surface.

5. The head-mounted display according to claim 3 or 4.

6. the control unit adjusts the optical system based on a convergence angle of the eyes of the observer so that a thickness of a lens of the eye looking at the display surface matches a thickness of a lens of the eye when the observer looks at an object that is present at a convergence distance corresponding to the convergence angle.

6. The head-mounted display according to claim 1, wherein the head-mounted display is a display unit.

7. the control unit determines the convergence angle based on a line of sight of the left eye and a line of sight of the right eye of the observer detected based on an output from the line of sight detection unit.

7. The head-mounted display according to claim 6.

8. the line-of-sight detection unit includes: an irradiation unit that irradiates the eye of the viewer with infrared light; a half mirror that is disposed between the display device and the eye and that reflects the infrared light reflected by the eye and transmits visible light; and an image sensor that detects the infrared light reflected by the half mirror.

8. The head-mounted display according to claim 1, wherein the head-mounted display is a display that displays a plurality of images.

9. The tolerance is within ±60 arc minutes of 90 degrees.

9. The head-mounted display according to claim 1, wherein the head-mounted display is a head-mounted display.

10. The tolerance is within ±6 arc minutes of 90 degrees.

9. The head-mounted display according to claim 1, wherein the head-mounted display is a head-mounted display.

11. the control unit controls the drive mechanism based on the output of the line-of-sight detection unit so that the line of sight coincides with a normal to the display surface within ±60 arc minutes at a point where the line of sight intersects with the display surface, on all planes including the line of sight.

11. The head-mounted display according to claim 1.

12. A head-mounted display according to any one of claims 1 to 11; A camera module; an attitude control unit that controls the attitude of the camera module so that the optical axis of the camera module is adjusted according to the direction of the line of sight; A cross reality system further comprising:

13. The camera module includes an image sensor having an imaging surface configured as a curved surface.

13. A cross reality system according to claim 12.

14. an image obtained by using the image sensor of the camera module, the peripheral portion of which is coarser than the central portion of the image, is transferred from the camera module side to the display device side; 14. A cross reality system according to claim 13.

Images