Image observation device, control method for image observation device, and program

Abstract

To provide an image observation device capable of generating an image with less strangeness.SOLUTION: An image observation device (101) includes: an image display element (106); an observation optical system (107) that leads a luminous flux from the image display element to an eyeball of an observer (102); sight line detection means (108, 109) that detect a sight line direction of the observer; imaging means (103, 104) that obtain a first image; and processing means (120) that generates a combined image of the first image and a second image. The processing means changes a magnification of the combined image displayed on the image display element, according to the gaze distance of the observer which is obtained based on the sight line direction.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image observation device, a control method for an image observation device, and a program. [Background technology]

[0002] Mixed reality (MR) technology and augmented reality (AR) technology are known as technologies that fuse the real world and the virtual world in real time. These technologies seamlessly fuse real space with a virtual space created by a computer. This allows the viewer to experience the presence of virtual objects in real space, and is expected to be applied to various fields.

[0003] Patent Documents 1 and 2 disclose a video see-through type image observation device as one of the devices that allows an observer to experience MR (hereinafter, mixed reality technology and augmented reality technology are collectively referred to as MR). This is an image observation device that captures the real world with a video camera, creates a composite image by superimposing CG (Computer Graphics) on the captured image in real time, displays the composite image on an image display element such as a display, and presents it to the observer. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4294093 [Patent Document 2] Patent No. 3604990 Summary of the Invention [Problem to be solved by the invention]

[0005] In MR technology, in order to enhance the sense of unity between the real world and the virtual world, it is necessary to maintain consistency in the size perceived by the observer. Here, consistency in the perceived size means the agreement between the size of the world or object perceived by the observer when actually viewing reality and the size of the world or object perceived through MR. In other words, it is whether the observer can perceive the real world at the same size when using a video see-through type image observation device and when not using it. However, it is difficult to maintain consistency in the perceived size, that is, to generate an image that does not feel unnatural, in a video see-through type image observation device.

[0006] SUMMARY OF THE PRESENT EMBODIMENTS Accordingly, an object of the present invention is to provide an image observation device capable of generating images that give a less unnatural feeling. [Means for solving the problem]

[0007] An image observation device as one aspect of the present invention comprises an image display element, an observation optical system that directs a light beam from the image display element to an observer's eyeball, a gaze detection means that detects the gaze direction of the observer, an imaging means that acquires a first image, and a processing means that generates a composite image of the first image and a second image, wherein the processing means changes the magnification of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the gaze direction.

[0008] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image observation device capable of generating an image that gives a less unnatural feeling. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an image observation device in a first embodiment. [Diagram 2] FIG. 1 is a schematic diagram showing a situation in which an observer is gazing at an object with both eyes. [Diagram 3] FIG. 4 is a diagram showing the relationship between distance and image size ratio in the first embodiment. [Figure 4] FIG. 4 is a diagram showing the relationship between distance and image correction magnification in the first embodiment. [Diagram 5] FIG. 11 is an explanatory diagram of a convergence angle in the second embodiment. [Figure 6] FIG. 11 is a diagram showing the relationship between the distance and the convergence angle difference in the second embodiment. [Figure 7] FIG. 11 is a diagram showing the relationship between the distance and the amount of image shift per eye in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] (First embodiment) First, an image observation device (image display device) 101 in a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram of the image observation device 101, and shows a schematic diagram of a vertical cross section in which an observer 102 observes an image using a video see-through type HMD (Head Mounted Display) as the image observation device 101. As shown in the lower right of Fig. 1, the +Z axis is taken toward the visual axis direction of the observer 102 (left side in Fig. 1), and the X axis (horizontal direction) and Y axis (vertical direction) are defined in right-handed coordinate system relative to the Z axis.

[0013] In Fig. 1, an imaging optical system 103 and an imaging element 104 constitute an imaging means of an image observation device 101, and acquire an image of the real world (an image of the outside world) including an object 105 at a height Y as image data (a first image). The image observation device 101 has a processing means 120 having a function as a composite image forming means constituted by a positioning means (not shown). In Fig. 1, the processing means 120 is disposed outside the image observation device 101, and is connected to the image observation device 101 so as to be able to communicate with it by wire or wirelessly. However, this embodiment is not limited to this, and the processing means 120 may be provided inside the image observation device 101.

[0014] The composite image forming means has a function of superimposing virtual image data (second image) on image data (first image) of the real world (outside world) so that the geometric positional relationship is approximately consistent. The image superimposed by the composite image forming means (superimposed image of the first image and the second image) is called a composite image, and the composite image is displayed on an image display element 106 such as a display. Depending on the situation, only image data of the real world without virtual image data superimposed thereon may be displayed on the image display element 106, and even in such a state, the image is called a composite image in this specification.

[0015] Reference numeral 107 denotes an observation optical system, which presents an enlarged virtual image of the image displayed on the image display element 106 to the observer 102. That is, the observation optical system 107 guides a light beam from the image display element 106 to the eyeball of the observer 102. As the observation optical system 107, not only a refractive system but also a reflective system using polarized light can be used, but is not limited to this. In this embodiment, the optical axis of the observation optical system (ocular optical system) 107 and the optical axis of the imaging optical system 103 coincide with each other, and the optical axes coincide not only in the YZ cross section in FIG. 1 but also in the XZ cross section. In this embodiment, the angle of view θ of the image acquired by the imaging means is 100 nm. 1 and the angle of view θ of the observation optical system 107. 2 are the same as each other. Here, the angle of view θ 1 is a part of the wider angle of view acquired by the imaging optical system 103 and the imaging element 104.

[0016] The position of the imaging optical system 103 and the pupil position of the observer 102 are spaced apart by a distance L 1 The position of the imaging optical system 103 and the position of the object 105 are separated by a distance L 2The position of the imaging optical system 103 is typically, but not limited to, an entrance pupil position or an image-side principal point position. The infrared camera 109 can calculate (acquire) the gaze direction of the eyeball of the observer 102 by capturing an image of the eyeball of the observer 102 illuminated by the infrared light source 108. The infrared light source 108, the infrared camera 109, and a processing means 120 for processing gaze direction information constitute a gaze detection means for detecting the gaze direction of the observer 102. Note that FIG. 1 shows a configuration for one eye (one eye) of the observer 102 as the image observation device 101, but a similar configuration is arranged for the other eye of the observer 102.

[0017] When observing an object 105 without using an image observation device 101, an observer 102 actually senses a distance (L 1 +L 2 ) away. 1 +L 2 ) corresponds to the gaze distance of the observer 102. On the other hand, when observing an object 105 using the image observation device 101, the distance L 2 This corresponds to observing an object 105 at a distance L from the object 105, so the object 105 is perceived as being larger than it actually is. 1 It is possible to reduce the size of the image observation device 101, but this would result in problems such as an increase in size of the image observation device 101. In this embodiment, the processing means 120 obtains information on the gaze distance of the observer 102 by using gaze information (gaze direction) of the observer 102 detected by the gaze detection means. Then, the processing means 120 changes (corrects) the magnification of the composite image displayed on the image display element 106 by using the information on the gaze distance.

[0018] Next, a situation in which the observer 102 is gazing at an object with both eyes will be described with reference to Fig. 2. Fig. 2 is a schematic diagram of a situation in which the observer 102 is gazing at an object 110 with both eyes. The distance (gazing distance) L from the observer 102 to the object 110 is 3 and the convergence angle θ 3and the interocular distance I of the observer 102 have the relationship expressed by the following formula (1).

[0019]

number

[0020] From equation (1), the convergence angle θ 3 By finding the distance L 3 It is possible to use the average human eye width distance as the interocular distance I, but it is also possible to use the interocular distance I of each observer obtained by using a line of sight detection function.

[0021] The processing means 120 having the function of the magnification adjustment means adjusts the distance L 3 The magnification of the image is corrected based on the information about the distance L 1 The ratio of the size of the image captured when the lens is located at the position of the pupil of the observer 102 to the size of the image captured when the lens is located at the position of the pupil of the observer 102 can be calculated. 2 / (L 1 +L 2 )

[0022] FIG. 3 is a diagram showing the ratio of distance to image size. In FIG. 3, the horizontal axis is distance (L 1 +L 2 ) [mm], and the vertical axis indicates the ratio of the size of the captured image. 1 = 40mm, meaning that when the size ratio is 1.0, the image will appear life-size, and when the size ratio is greater than 1.0, the image will appear larger.

[0023] So far, the angle of view θ 1 and angle of view θ 2 Since the distance between the observer 102 and the object 105 (L 1 +L 2 ) is infinite, the ratio of the sizes is 1.0, i.e., the same size. On the other hand, if the distance L 1 +L 2As the distance becomes smaller, the image appears larger. To correct this, the distance information obtained by the line of sight detection function (distance L 3 ), the composite image is reduced overall by using the inverse of the size ratio as the image magnification, and displayed on the image display element 106.

[0024] Next, the image correction magnification (magnification of the image to be corrected) will be described with reference to FIG. 4. FIG. 4 is a diagram showing the relationship between distance and image correction magnification. In FIG. 4, the horizontal axis is distance (L 1 +L 2 ) [mm], and the vertical axis indicates the image correction magnification. In FIG. 4, when the image correction magnification is smaller than 1.0, it means that the image is reduced.

[0025] Specifically, the maximum number of pixels of the image sensor 104 is 4000 pixels horizontally and 4000 pixels vertically, and the angle of view that the image sensor 104 captures in the maximum number of pixels of the image sensor 104 by the image sensor optical system 103 is 80° horizontally and 80° vertically. Also, the number of pixels of the image display element 106 is 2000 pixels horizontally and 2000 pixels vertically, and the angle of view of the observation optical system 107 is 60° horizontally and 60° vertically. In this case, θ 1 =θ 2 In this case, the area on the imaging element 104 corresponding to 60°, i.e., 2752 pixels, is resized to 2000 pixels, and this is displayed on the image display element 106. On the other hand, if it is determined by the line of sight detection means that the observer 102 is looking at an object at a distance of 1000 mm, the reduction magnification is 0.96. Therefore, the 2867 pixels on the imaging element 104 are resized to 2000 pixels, and this is displayed on the image display element 106.

[0026] The processing means 120 having the function of a magnification adjustment means uses the distance information (gazing distance) of the observer 102 to calculate an appropriate magnification that does not cause discomfort to the observer 102, and creates image data (synthetic image data) to be displayed on the image display element 106. For example, the processing means 120 sets the magnification to a first magnification when the gaze distance is a first gaze distance, and sets the magnification to a second magnification lower than the first magnification when the gaze distance is a second gaze distance that is closer than the first gaze distance.

[0027] By correcting (adjusting) the magnification of the composite image displayed on the image display element 106 in this manner, the observer can observe an image with a less sense of discomfort in size without increasing the size of the image observation device. Note that the method of calculating the magnification correction value is not limited to the method using formula (1). For example, when the angle of view θ 1 and angle of view θ 2 The gaze distance obtained by the gaze detection means is, strictly speaking, the distance L 1 Since the gaze distance is calculated while the viewer is viewing an image shifted by only one eye, the correction value of the magnification may be modified taking this into consideration.

[0028] When performing magnification correction, it is preferable that the observer 102 does not notice that the image magnification has changed. Since the human gaze has a high-speed movement called saccade, the detection frame rate is generally high and a lot of information is acquired, but noise components are also included. Therefore, if the distance calculation result (the gaze distance of the observer 102) is used as is, vibration (fluctuation, noise) of the image may be felt. In order to avoid such vibration (to reduce noise related to the gaze distance of the observer 102), it is effective for the processing means 120 to perform a filter process on the gaze detection result to avoid image fluctuation. Here, it is possible to use a known filter such as a time-direction averaging or statistical filter as the filter.

[0029] It is also effective for the processing means 120 to have an upper limit on the amount of change (correction amount) of magnification per unit time (to set an upper limit on the correction amount per unit time) to avoid sudden fluctuations. For example, suppressing the correction amount of magnification to less than 0.1 per second will reduce the sense of incongruity. To further reduce the sense of incongruity, it is preferable to suppress the absolute value of the relative correction amount of magnification to less than 0.01 per second. Here, the relative correction amount of magnification is, for example, 1-(2756 / 2867)=0.04 when the number of pixels of the original image sensor 104 to be displayed on the image display device 106 is changed from 2752 pixels to 2867 pixels.

[0030] It is also effective not to correct the magnification for a slight change in the gaze distance. For example, the processing means 120 does not change the magnification when the change amount (correction amount) of the magnification is less than a predetermined amount. Specifically, it is preferable not to perform correction when the relative correction amount of the magnification is less than 0.03. More preferably, if correction is not performed when the relative correction amount of the magnification is less than 0.01, it is possible to achieve both accurate magnification correction and reduction in discomfort.

[0031] When it is detected that the observer 102 has removed the image observation device 101, or when it becomes impossible to detect the line of sight of the observer 102, it is preferable to return the corrected magnification to a predetermined magnification. For example, when the processing means 120 cannot acquire the gaze distance, it sets the magnification to a predetermined value (predetermined magnification). For example, the predetermined magnification is a magnification of 1.0(θ 1 =θ 2 ) can be selected.

[0032] In this embodiment, the maximum angle of view that can be obtained by the imaging optical system 103 is the angle of view θ 2 However, depending on the amount of correction of the magnification, the maximum angle of view of the imaging optical system 103, and the angle of view θ 1 Depending on the relationship between the magnification and the image area to be displayed on the image display element 106, the image area may be insufficient. Therefore, by setting an upper limit on the magnification correction amount, it is possible to prevent image loss, etc. For example, the processing means 120 has an upper limit on the total amount of change in magnification.

[0033] Second embodiment Next, a second embodiment of the present invention will be described. The image observation device of this embodiment is different from the image observation device of the first embodiment in that the convergence angle of the image is corrected using a processing means having a function as a convergence angle adjustment means, and the image observation device of the first embodiment corrects the magnification of the image using a processing means having a function as a magnification adjustment means. Note that the basic configuration of the image observation device of this embodiment is similar to the image observation device 101 of the first embodiment described with reference to FIG. 1, so a common description will be omitted.

[0034] 5 is an explanatory diagram of the convergence angle, and shows a state in which the observer 102 is observing an object 201. When the observer 102 is not wearing the image observation device (HMD) 101, the observer 102 observes the object 201 at a convergence angle θ5. On the other hand, when the observer 102 observes the object 201 while wearing the image observation device 101, the left eye side image capturing means 202 and the right eye side image capturing means 203 are positioned at a distance L from the observer 102. 1 Since the object 201 is closer to the object 201 by 4 In other words, when observing with the image observation device 101 attached, the convergence angle is larger than when observing with the naked eye, so the observer 102 feels that the object 201 is closer. 4 and convergence angle θ 5 The difference (convergence angle difference) is expressed by the following formula (2).

[0035]

number

[0036] FIG. 6 is a diagram showing the relationship between distance and convergence angle difference. In FIG. 6, the horizontal axis is distance (L 1 +L 2 ) [mm], and the vertical axis indicates the convergence angle difference (°). 16 shows a case where the distance between the left and right cameras of the image observation device 101 (the distance between the left eye side imaging means 202 and the right eye side imaging means 203) is 40 mm, and the interpupillary distance (interpupillary distance I) of the observer 102 is both 63 mm. Under these conditions, the convergence angle difference (deviation from the convergence angle) perceived by the observer 102 when the observer 102 views an object 201 using the image observation device 101 at a certain distance is shown. In FIG. 6, the larger the convergence angle difference, the larger the convergence angle is compared to direct observation, and the closer the object is viewed, the larger the convergence angle difference. It is generally known that the deviation from the convergence angle also affects the perceived size, and this can cause a sense of incongruity compared to direct observation.

[0037] Therefore, in the image observation device of this embodiment, similarly to the first embodiment, the line of sight detection means determines the convergence angle of the object 201 that the observer 102 is gazing at. Then, the processing means 120 having the function of the convergence angle adjustment means shifts the position of the composite image when displayed on the image display element 106 by the shift amount (image shift amount) according to the distance determined from the convergence angle. This makes it possible to correct (change) the convergence angle of the gaze point that the observer 102 is gazing at.

[0038] Next, the image shift amount will be described with reference to FIG. 7. FIG. 7 is a diagram showing the relationship between the distance and the image shift amount per one eye. In FIG. 7, the horizontal axis is the distance (L 1 +L 2 ) [mm], and the vertical axis indicates the image shift amount [pixel]. Fig. 7 shows the image shift amount (pixel) per eye when the focal length of the observation optical system 107 is 20 mm, and the pixel pitch of the image display element 106 is 10 μm. When the image shift amount is positive, it means that the right eye image is shifted to the right, and the left eye image is shifted to the left.

[0039] The processing means 120 having the function of a convergence angle adjustment means calculates (acquires) an appropriate image shift amount that causes little discomfort according to the distance information (gazing distance) of the observer 102, and generates image data (synthetic image data) to be displayed on the image display element 106. That is, the processing means 120 calculates the convergence angle θ of the composite image displayed on the image display element 106 according to the gazing distance of the observer 102 acquired based on the line of sight. 4 Here, the convergence angle θ 4 For example, as shown in FIG. 5, corresponds to the angle between a line connecting an object 201 that the observer 102 is gazing at and the left-eye-side imaging means 202, and a line connecting the object 201 and the right-eye-side imaging means 203.

[0040] For example, the processing means 120 sets the convergence angle to a first convergence angle when the gaze distance is a first gaze distance, and sets the convergence angle to a second convergence angle smaller than the first convergence angle when the gaze distance is a second gaze distance that is closer than the first gaze distance. Preferably, the processing means 120 changes the convergence angle by changing the shift amount of the composite image displayed on the image display element 106. This makes it possible to correct the deviation of the convergence angle. This correction allows the observer to observe an image with less discomfort in size without enlarging the image observation device.

[0041] In this embodiment, the method of calculating the convergence angle correction value (image shift amount) is not limited to the method using formula (2), but can be appropriately selected according to the subjective impression of the observer 102, and other methods may be used as long as they can perform correction that minimizes the sense of discomfort in size. 1 Since the gaze distance is calculated while the viewer is viewing an image shifted by only a certain distance, the convergence angle correction value may be modified taking this into consideration.

[0042] When correcting the convergence angle, it is preferable to prevent the observer 102 from noticing that the convergence angle has changed. Since the human gaze has a high-speed movement called saccade, the detection frame rate is generally high and a lot of information is acquired, but noise components are also included in the information. Therefore, if the distance calculation result (the gaze distance of the observer 102) is used as is, the observer may feel vibration (fluctuation, noise) in the image. In order to avoid such vibration (to reduce noise related to the gaze distance of the observer 102), it is effective for the processing means 120 to avoid image fluctuation by performing a filter process on the gaze detection result. Here, the filter can be a known filter such as a time-direction averaging or statistical filter.

[0043] In addition, the processing means 120 can effectively avoid abrupt fluctuations by having an upper limit value for the change amount (correction amount) of the convergence angle per unit time (setting an upper limit value for the correction amount per unit time). For example, if the relative correction amount of the convergence angle is suppressed to less than 1° per second, the sense of discomfort is reduced. To further suppress the sense of discomfort, it is preferable to suppress the absolute value of the relative correction amount of the convergence angle to less than 0.5° per second. Here, the relative correction amount of the convergence angle is, for example, 0.15-0.62=-0.47° when the convergence angle at the current gaze distance is 0.62° and the convergence angle after correction is 0.15°.

[0044] It is also effective not to correct the convergence angle for a slight change in the convergence angle. For example, the processing means 120 does not change the convergence angle when the change amount (correction amount) of the convergence angle is less than a predetermined amount. Specifically, it is preferable not to perform correction when the absolute value of the relative correction amount of the convergence angle is less than 0.5°. More preferably, if correction is not performed when the absolute value of the relative correction amount of the magnification is less than 0.1°, it is possible to achieve both accurate correction of the convergence angle and reduction of discomfort.

[0045] When it is detected that the observer 102 has removed the image observation device 101, or when it becomes impossible to detect the line of sight of the observer 102, it is preferable to return the corrected convergence angle to a predetermined convergence angle. For example, when the processing means 120 cannot acquire the gaze distance, it sets the convergence angle to a predetermined value (predetermined convergence angle). As the predetermined convergence angle, for example, an angle of 0° (corresponding to gaze at infinity or parallel arrangement) can be selected. In addition, by setting an upper limit on the amount of correction of the convergence angle, it is possible to prevent image loss, etc. For example, the processing means 120 has an upper limit on the total amount of change of the convergence angle.

[0046] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0047] According to each embodiment, it is possible to provide an image observation device capable of generating an image that gives a less unnatural feeling, a control method for an image observation device, and a program.

[0048] The disclosure of each embodiment includes the following configurations and methods. (Configuration 1) An image display element; an observation optical system that guides a light beam from the image display element to an eyeball of an observer; A gaze detection means for detecting a gaze direction of the observer; An imaging means for acquiring a first image; a processing means for generating a composite image of the first image and the second image, The image observation device according to claim 1, wherein the processing means changes a magnification of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight. (Configuration 2) The processing means includes: If the gaze distance is a first gaze distance, the magnification is set to a first magnification; 2. The image observation device according to configuration 1, wherein when the gaze distance is a second gaze distance that is closer than the first gaze distance, the magnification is set to a second magnification that is lower than the first magnification. (Configuration 3) 3. The image observation device according to configuration 1 or 2, wherein the processing means has an upper limit value for the amount of change of the magnification per unit time. (Configuration 4) 4. The image observation device according to any one of configurations 1 to 3, wherein the processing means does not change the magnification when the amount of change in the magnification is less than a predetermined amount. (Configuration 5) 5. An image observation device according to any one of configurations 1 to 4, wherein the processing means has an upper limit value for a total amount of change in the magnification. (Configuration 6) 6. The image observation device according to any one of configurations 1 to 5, wherein the processing means sets the magnification to a predetermined value when the gaze distance cannot be acquired. (Configuration 7) An image display element; an observation optical system that guides a light beam from the image display element to an eyeball of an observer; A gaze detection means for detecting a gaze direction of the observer; An imaging means for acquiring a first image; a processing means for generating a composite image of the first image and the second image, The image observation device according to claim 1, wherein the processing means changes a convergence angle of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight. (Configuration 8) The processing means includes: If the gaze distance is a first gaze distance, the convergence angle is set to a first convergence angle; 8. An image observation device according to configuration 7, wherein when the gaze distance is a second gaze distance that is closer than the first gaze distance, the convergence angle is set to a second convergence angle that is smaller than the first convergence angle. (Configuration 9) 9. The image observation device according to configuration 7 or 8, wherein the processing means changes the convergence angle by changing an amount of shift of the composite image displayed on the image display element. (Configuration 10) 10. An image observation device according to any one of configurations 7 to 9, wherein the processing means has an upper limit value for an amount of change in the convergence angle per unit time. (Configuration 11) 11. An image observation device according to any one of configurations 7 to 10, wherein the processing means does not change the convergence angle when the amount of change in the convergence angle is less than a predetermined amount. (Configuration 12) 12. An image observation device according to any one of configurations 7 to 11, wherein the processing means has an upper limit value for a total amount of change in the convergence angle. (Configuration 13) 13. The image observation device according to any one of configurations 7 to 12, wherein the processing means sets the convergence angle to a predetermined value when the gaze distance cannot be obtained. (Configuration 14) the first image is an image of the outside world; 14. The image observation device according to any one of configurations 1 to 13, wherein the second image is an image superimposed on the first image so as to match a geometric positional relationship in the external world. (Configuration 15) 15. An image observation device according to any one of configurations 1 to 14, wherein the processing means performs a filter process to remove noise relating to the gaze distance of the observer. (Method 1) A step of directing a light beam from the image display element to an eyeball of an observer; detecting a gaze direction of the observer; generating a composite image of a first image and a second image acquired by the imaging means; and changing a magnification of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight. (Method 2) A step of directing a light beam from the image display element to an eyeball of an observer; detecting a gaze direction of the observer; generating a composite image of a first image and a second image acquired by the imaging means; and changing a convergence angle of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight. (Configuration 16) A program for causing a computer to execute the method for controlling an image observation device according to the method 1 or 2.

[0049] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0050] In each embodiment, an HMD using MR technology has been described as an image observation device, but the present invention is not limited to this, and each embodiment can be applied to other image observation devices. [Explanation of symbols]

[0051] 101 Image observation device 102 Observer 103 Imaging optical system (imaging means) 104 Imaging element (imaging means) 106 Image display element 107 Observation Optical System 108 Line of sight detection camera (line of sight detection means) 109 Infrared light source (line of sight detection means) 120 Processing means

Claims

1. An image display element; an observation optical system that guides a light beam from the image display element to an eyeball of an observer; A gaze detection means for detecting a gaze direction of the observer; An imaging means for acquiring a first image; a processing means for generating a composite image of the first image and the second image, The image observation device according to claim 1, wherein the processing means changes a magnification of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight.

2. The processing means includes: If the gaze distance is a first gaze distance, the magnification is set to a first magnification; 2. The image observation device according to claim 1, wherein when the gaze distance is a second gaze distance that is closer than the first gaze distance, the magnification is set to a second magnification that is lower than the first magnification.

3. 2. An image observation device according to claim 1, wherein said processing means has an upper limit value for the amount of change of said magnification per unit time.

4. 2. An image observation device according to claim 1, wherein said processing means does not change said magnification when the amount of change of said magnification is less than a predetermined amount.

5. 2. An image observation apparatus according to claim 1, wherein said processing means has an upper limit value for a total amount of change in said magnification.

6. 2. An image observation device according to claim 1, wherein said processing means sets said magnification to a predetermined value when said gaze distance cannot be obtained.

7. An image display element; an observation optical system that guides a light beam from the image display element to an eyeball of an observer; A gaze detection means for detecting a gaze direction of the observer; An imaging means for acquiring a first image; a processing means for generating a composite image of the first image and the second image, The image observation device according to claim 1, wherein the processing means changes a convergence angle of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight.

8. The processing means includes: If the gaze distance is a first gaze distance, the convergence angle is set to a first convergence angle; 8. An image observation device according to claim 7, wherein when the gaze distance is a second gaze distance that is closer than the first gaze distance, the convergence angle is set to a second convergence angle that is smaller than the first convergence angle.

9. 8. An image observation device according to claim 7, wherein said processing means changes said convergence angle by changing an amount of shift of said composite image displayed on said image display element.

10. 8. An image observation device according to claim 7, wherein said processing means has an upper limit value for the amount of change of said convergence angle per unit time.

11. 8. An image observation device according to claim 7, wherein said processing means does not change said convergence angle when the amount of change in said convergence angle is less than a predetermined amount.

12. 8. An image observation apparatus according to claim 7, wherein said processing means has an upper limit value for a total amount of change in said convergence angle.

13. 8. An image observation device according to claim 7, wherein said processing means sets said convergence angle to a predetermined value when said gaze distance cannot be acquired.

14. the first image is an image of the outside world; 14. The image observation device according to claim 1, wherein the second image is an image superimposed on the first image so as to match a geometric positional relationship in the external world with the first image.

15. 14. An image observation device according to claim 1, wherein the processing means performs a filter process to remove noise relating to the gaze distance of the observer.

16. A step of directing a light beam from the image display element to an eyeball of an observer; detecting a gaze direction of the observer; generating a composite image of a first image and a second image acquired by the imaging means; and changing a magnification of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight.

17. A step of directing a light beam from the image display element to an eyeball of an observer; detecting a gaze direction of the observer; generating a composite image of a first image and a second image acquired by the imaging means; and changing a convergence angle of the composite image displayed on the image display element in accordance with the gaze distance of the observer acquired based on the line of sight.

18. A program for causing a computer to execute the method for controlling an image observation apparatus according to claim 16 or 17.

Images