Image display device and image display system

Abstract

The aim is to achieve miniaturization even when an imaging unit is included. [Solution] The image display device of the present invention is characterized by having an imaging unit 111 and a head-mounted unit 110 held by the user, and a display unit 120 that displays an image that has been converted from an image captured by the imaging unit 111 to an image that appears to have been captured from a position different from the physically positioned position of the imaging unit 111. Since the imaging unit 111 is located on the head-mounted unit 110, the display unit 120 can be made smaller, and thus the entire image display device can be made smaller.

Description

Technical Field

[0001] The present invention relates to an image display device and an image display system.

Background Art

[0002] In recent years, head-mounted displays (HMDs), which are devices worn on the head to observe images, and hand-held displays (HHDs), which are devices held in the hand to observe images, have become widespread. Generally, since an HMD is a device worn on the head, it is preferable for it to be small and lightweight because this leads to an improved wearing experience. Also, in the case of an HHD, it is preferable for it to be small and lightweight because this can reduce physical fatigue.

[0003] Uses of HMDs and HHDs include experiences of virtual reality (VR) and mixed reality (MR). As a specific example of the latter, a video see-through method of photographing the real world with a camera is known. Patent Document 1 discloses a technique (position and orientation conversion type video see-through) for outputting an image from a viewpoint different from the physical arrangement of a camera.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] [[ID=3**]]However, in the HMD disclosed in Patent Document 1, since a camera is arranged inside the housing of the display unit, it is necessary to increase the size of the housing of the display unit, and there is a problem that the entire HMD becomes large. The present invention has been made in view of the above-described problems, and an object thereof is to achieve miniaturization even in the case of having an imaging unit. [Means for solving the problem]

[0006] The present invention is characterized by having an imaging unit and a holding unit that is held by the user, and a display unit that displays an image that has been converted so that the image captured by the imaging unit appears to be an image captured from a position different from the physically positioned position of the imaging unit. [Effects of the Invention]

[0007] According to the present invention, miniaturization can be achieved even when an imaging unit is included. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of the image display system according to the first embodiment. [Figure 2] This is a plan view showing an example of an HMD configuration. [Figure 3] A side view showing an example of an HMD configuration. [Figure 4] This is a block diagram showing an example of the functional configuration of an image processing device. [Figure 5] This diagram illustrates the position of each center of gravity on the HMD. [Figure 6] This figure shows an example of the configuration of the image display system according to the second embodiment. [Figure 7] This is a plan view showing an example of an HHD configuration. [Figure 8] This is a side view showing an example of an HHD configuration. [Figure 9] This diagram illustrates the position and orientation of the imaging units of the HHD and HMD. [Figure 10] This diagram illustrates the position and orientation of the imaging units of the HHD and HMD. [Figure 11] This figure shows an example of the configuration of the image display system according to the third embodiment. [Figure 12] A side view showing an example of an HMD configuration. [Figure 13]This is a perspective view showing an example of an HMD configuration. [Modes for carrying out the invention]

[0009] Preferred embodiments of the present invention will be described below with reference to the attached drawings. <First Embodiment> In the first embodiment, the case where the image display device is an HMD (video see-through type HMD) will be described. Figure 1 is a schematic diagram showing an example of the configuration of the image display system, and includes a perspective view showing an example of the configuration of the HMD 10. Figure 2 is a plan view showing an example of the configuration of the HMD 10. Figure 3 is a side view showing an example of the configuration of the HMD 10. Note that, for the sake of ease of explanation, each figure is shown as front, back, right, and left based on the state in which a user is standing and wearing the HMD 10. The image display system comprises the HMD 10 and the image processing device 140.

[0010] The HMD10 comprises a head-mounted section 110, a non-transparent display section 120, and a connection section 130. The head-mounted part 110 is attached to the user's head. By attaching the head-mounted part 110 to the head, the HMD 10 is held in place by the user. The head-mounted part 110 corresponds to an example of a holding part. The head-mounted unit 110 includes multiple position-and-orientation-changing imaging units 111, a main body unit 112, a head contact unit 113, an operating unit 114, and a retraction mechanism unit 115.

[0011] The imaging unit 111 is a part that captures the real world. The imaging unit 111 is composed of a plurality (at least two). The imaging units 111 are arranged separately left and right at a position in front of and below the main body unit 112. As shown in FIG. 3, the imaging unit 111 of the present embodiment is arranged behind the foremost position Pf of the main body unit 112 in a side view. Also, the imaging unit 111 is arranged above and behind the display unit 120 in a side view. Further, as shown in FIG. 2, in a plan view, the imaging unit 111 is covered from above by a part of the main body unit 112 functioning as a visor. The two imaging units 111 of the present embodiment each capture an image with a parallax. Note that the number of imaging units 111 is not limited to two, and may be three or more.

[0012] The main body unit 112 is a part that serves as the exterior of the head-mounted unit 110. The main body unit 112 is formed in an annular shape in a plan view (when viewed from above). The head contact part 113 is a part that contacts the user's head when the head-mounted unit 110 is worn. The head contact parts 113 are arranged separately front and back at a position inside the main body unit 112 and contact the user's forehead and the back of the head. Note that the head contact parts 113 may be arranged at positions that contact the user's temples.

[0013] The operation part 114 is a part that the user operates when adjusting the size of the main body unit 112 according to the size of the head. The user can mount the head-mounted unit 110 at an appropriate position on the user's head by adjusting the size of the main body unit 112 via the operation part 114, and the user can observe the enlarged virtual image displayed by the display unit 120 from the front. [[ID=1十二]]

[0014] The retraction mechanism part 115 is a mechanism that retracts the display unit 120 from in front of the user's eyes. The retraction mechanism part 115 of the present embodiment is arranged at the front end and the lower end of the main body unit 112, and can be rotated to flip up the display unit 120 forward around a rotation axis O along the left-right direction.

[0015] The display unit 120 is the part that displays the image captured by the imaging unit 111 and after image processing, including positional transformation. The display unit 120 is located below the foremost position Pf of the main unit 112 and is positioned in front of the eyes of the user wearing the head-mounted unit 110. The display unit 120 has a roughly plate-like shape that is long in the left-right direction, with the display surface located at the rear. Specifically, the display unit 120 is composed of a housing as an exterior, a display element, and a display optical system, and guides the magnified virtual image of the display element to the user's eyes. The user can experience MR by observing the image of the real world in front of them, which has been augmented with CG (Computer Graphics), etc., through the display unit 120. For example, the display unit 120 can be an EL panel or an LCD.

[0016] The connecting portion 130 is the part that physically connects the head-mounted portion 110 and the display portion 120. The connecting portion 130 has a one-sided contact portion 131a on the head-mounted portion 110 side, a other-sided contact portion 131b on the display portion 120 side, and a screw 132 as a connecting member. In this embodiment, the one-sided contact portion 131a is formed by a part of the retraction mechanism portion 115, and the other-sided contact portion 131b is formed by a part of the display portion 120. The head-mounted portion 110 and the display portion 120 are connected by bringing the one-sided contact portion 131a and the other-sided contact portion 131b into contact with each other from a detached state and inserting and screwing in the screw 132 from above. Furthermore, the head-mounted portion 110 and the display portion 120 are detachable via the connecting portion 130. Specifically, the user can use a tool to loosen the screw 132, thereby separating the one-sided contact portion 131a and the other-sided contact portion 131b from each other and releasing the connection between the head mounting portion 110 and the display portion 120.

[0017] Furthermore, the connecting portion 130 is not limited to having a screw 132, and can be configured in any way that allows the head mounting portion 110 and the display portion 120 to be detachably connected to each other. For example, one of the contact portion 131a and the other contact portion 131b may be a convex portion and the other a concave portion so that they fit together, or it may be configured to connect and disconnect in a snap-fit ​​manner without using a screw 132 or tools. Also, the connecting portion 130 is not limited to a configuration that allows the head mounting portion 110 and the display portion 120 to be detachably connected, and may be configured to be non-detachable.

[0018] The imaging unit 111 and the display unit 120 may be powered by a wired cable, or by a battery (not shown) provided in the head-mounted unit 110. The battery is preferably positioned on the back of the head-mounted unit 110, taking into consideration the weight balance when worn. Furthermore, the imaging unit 111 and the display unit 120 may communicate with the image processing device 140 using the aforementioned wired cable, or they may communicate wirelessly. The head-mounted unit 110 is provided with a cable holder 116 on the main body 112 for holding the wired cable.

[0019] The image processing device 140 transforms the image captured by the imaging unit 111 so that it appears as if it were captured from a different position than the physically positioned position of the imaging unit 111. Here, with reference to Figure 3, the position and orientation of the imaging unit 111 before and after the transformation will be explained. The imaging unit before the transformation refers to the imaging unit 111 as it is actually positioned. On the other hand, the imaging unit after the transformation refers to the imaging unit as it is positioned at a location corresponding to the user's eye, which is different from the position of the imaging unit 111 as it is actually positioned.

[0020] The position 111Pa of the imaging unit before conversion is shown by a dashed circle, and the orientation (shooting direction) of the imaging unit before conversion is shown by a dashed line as the optical axis 111Ca of the imaging optical system of the imaging unit. In other words, the imaging unit 111 is physically positioned above the display unit 120 and at a downward angle. The downward angle α in this embodiment is approximately 10° to 20°. On the other hand, the position 111Pb of the image sensor after conversion is shown by a solid circle, and the orientation (shooting direction) of the image sensor after conversion is shown by a solid line as the optical axis 111Cb. The position 111Pb of the image sensor after conversion approximately coincides with the exit pupil position of the display optical system (not shown). That is, the position 111Pb of the image sensor after conversion approximately coincides with the position of the user's eye (not shown). Furthermore, the optical axis 111Cb of the image sensor, which represents the orientation after conversion, approximately coincides with the optical axis 120C of the display optical system (not shown) of the display unit 120.

[0021] Figure 4 is a block diagram showing an example of the functional configuration of the image processing device 140. The image processing device 140 can use a PC (personal computer) separate from the HMD 10. However, the image processing device 140 is not limited to a PC; a smartphone, tablet terminal, etc., may also be used. Furthermore, the image processing device 140 is not limited to being separate from the HMD 10; it may be integrated with the HMD 10. In other words, the HMD 10 may have the image processing device 140.

[0022] The image processing device 140 includes an image acquisition unit 141, a storage unit 142, an optical axis parallelization unit 143, a three-dimensional reconstruction unit 144, a viewpoint transformation unit 145, and a synthesis unit 146. The image acquisition unit 141 acquires frame-by-frame images of the video from the imaging unit 111 of the HMD 10.

[0023] The memory unit 142 stores virtual space object data (CG images) generated by the CG generation unit (not shown), captured images acquired by the image acquisition unit 141, and conversion parameters used in each of the processes described later.

[0024] The optical axis parallelization unit 143 uses the image acquired by the image acquisition unit 141 to perform image processing to parallelize the optical axes of the two imaging units 111. Specifically, the image processing involves correcting internal parameters such as the imaging angle of view and distortion of the imaging unit 111, and correcting external parameters such as the physical position and orientation of the imaging unit 111. Each parameter is stored in the storage unit 142.

[0025] The 3D reconstruction unit 144 acquires depth information starting from the imaging unit 111 using the image processed by the optical axis parallelization unit 143. Specifically, depth information is acquired based on the disparity information of each feature point in the image after it has been processed by the optical axis parallelization unit 143. The parameters necessary for acquiring depth information, such as the imaging field of view set in the image processed by the optical axis parallelization unit 143 and the baseline length of the imaging unit 111, are stored in the storage unit 142.

[0026] As shown in Figure 3, the viewpoint transformation unit 145 transforms the position and orientation of the imaging unit 111 into an apparent position and orientation that differs from its physical position and orientation. Specifically, the transformation process is a coordinate transformation process that transforms the imaging unit 111 into a desired position and orientation, based on the correspondence between the image obtained by perspective projection and the depth information in that image. The parameters for transforming into the desired position and orientation are stored in the storage unit 142.

[0027] The synthesis unit 146 synthesizes the image processed by the viewpoint transformation unit 145 by superimposing the CG image input from the storage unit 142 onto it. The display unit 120 displays the image synthesized by the synthesis unit 146.

[0028] By performing image processing in this manner, the image processing device 140 can display an image as if a video see-through camera were placed at the user's eye level, which cannot be physically positioned on the display unit 120. The display unit 120 only needs to be able to display an image that converts the physical position and orientation of the imaging unit 111 into an apparent position and orientation, and the image processing device 140 is not limited to the configuration described above.

[0029] In this embodiment of the HMD10, the imaging unit 111, which is a heat source, is located in the head-mounted unit 110 rather than the display unit 120. Therefore, the heat from the imaging unit 111 can be dissipated using the surface of the head-mounted unit 110 as a heat dissipation surface, enabling effective heat dissipation throughout the entire HMD10. On the other hand, by locating the imaging unit 111 in the head-mounted unit 110, the power consumption within the housing of the display unit 120 can be reduced and the display unit 120 can be made smaller. Furthermore, since the head-mounted unit 110 can secure space for the imaging unit 111 without increasing its size, the HMD10 as a whole can be made smaller and lighter by miniaturizing the display unit 120.

[0030] Furthermore, in this embodiment, the HMD10 is more comfortable to wear because the imaging unit 111 is located on the head-mounted unit 110 rather than the display unit 120. Figure 5 is a diagram illustrating the center of gravity positions of each part of the HMD 10. In Figure 5, the contact point 113P where the head contact portion 113 of the head mounting portion 110 and the user's head come into contact is shown by a dashed line, the center of gravity position 111G of the imaging portion 111 is shown by a solid black circle, and the center of gravity position 120G of the display portion 120 is shown by a solid black circle.

[0031] Since the imaging unit 111 is located on the head-mounted unit 110, the distance from the contact point 113P to the center of gravity position 111G of the imaging unit 111 is shorter than the distance from the contact point 113P to the center of gravity position 120G of the display unit 120. Now, let's assume there is an HMD in which the imaging unit 111 is located on the display unit 120. Even if the total weight of the HMD 10 in this embodiment is the same as the total weight of the assumed HMD, by positioning the imaging unit 111 near the contact point 113P as in the HMD 10 of this embodiment, the wearability of the HMD 10 can be improved from the perspective of moment compared to the assumed HMD.

[0032] The HMD 10 is not limited to the configuration described above and can be modified as appropriate. For example, the head contact portion 113 may be connected to the display unit 120. In this case, the part of the head contact portion 113 that connects to the display unit 120 becomes the connection portion 130. Also, the head mounting portion 110 does not need to be a single integrated unit and may be constructed as a separate component.

[0033] <Second Embodiment> In the second embodiment, the case where the image display device is an HHD (video see-through type HHD) will be described. Figure 6 is a schematic diagram showing an example of the configuration of the image display system, and includes a perspective view showing an example of the configuration of the HHD 20. Figure 7 is a plan view showing an example of the configuration of the HHD 20. Figure 8 is a side view showing an example of the configuration of the HHD 20. Note that in each figure, for the sake of ease of explanation, the front, back, right, and left are shown based on the state in which the user is holding the HHD 20. Note that components similar to those in the first embodiment are denoted by the same reference numerals and their descriptions are omitted as appropriate. The image display system comprises an HHD 20 and an image processing device 140.

[0034] The HHD20 includes a handle 150, a non-transparent display unit 120, and a connection unit 160. The handle portion 150 is a curved bar shape and is formed in a roughly U-shape that opens downwards when viewed from the front. The HHD20 is held by the user when the user holds the handle portion 150 in their hand. The handle portion 150 corresponds to an example of a holding portion. The handheld portion 150 has a gripping portion 151, a central portion 152, a head contact portion 153, and multiple position-and-orientation-changing imaging units 111.

[0035] The gripping portion 151 is the part that the user actually holds with their hands. In this embodiment, the gripping portions 151 are arranged in pairs, separated on the left and right sides, and are bar-shaped and long in the vertical direction. The user can stably hold the HHD20 by holding the gripping portion 151 with each hand. The central portion 152 is located in the center of the handle portion 150 in the left-right direction. The central portion 152 functions as the main body of the handle portion 150. In this embodiment, the central portion 152 is formed continuously with the grip portion 151 and is a long bar-shaped component located between the right grip portion 151 and the left grip portion 151. The head contact portion 153 is the part that comes into contact with the user's forehead. In this embodiment, the head contact portion 153 is formed to protrude from the rear end of the central portion 152 toward the rear. By holding the grip portion 151 with the user's hand and bringing the head contact portion 153 into contact with the forehead, the HHD20 can be held more stably, and the magnified virtual image displayed by the display portion 120 can be observed from the front.

[0036] The imaging unit 111 is the part that captures images of the real world. The imaging unit 111 is composed of multiple units (at least two). The imaging units 111 are positioned on the left and right sides, separated from each other, in front of the central unit 152. As shown in Figure 8, in this embodiment, the imaging unit 111 is positioned above the display unit 120 in a side view. Also, as shown in Figure 7, in a top view, the imaging unit 111 is covered from above by a part of the central unit 152 which functions as an overhang. The two imaging units 111 in this embodiment each capture images with parallax. Note that the imaging unit 111 is not limited to two units, but may be three or more.

[0037] The display unit 120 is the part that displays the image captured by the imaging unit 111 and after image processing, including position and orientation transformation, has been performed. The display unit 120 is located below the central part 152 of the handheld unit 150 and is positioned in front of the eyes of the user holding the handheld unit 150. The display unit 120 in this embodiment has the same configuration as in the first embodiment.

[0038] The connecting portion 160 is the part that physically connects the handle portion 150 and the display portion 120. The connecting portion 160 has a one-sided contact portion 161a on the handle portion 150 side, a other-sided contact portion 161b on the display portion 120 side, and a screw 162 as a connecting member. In this embodiment, the one-sided contact portion 161a is formed by a part of the central portion 152, and the other-sided contact portion 161b is formed by a part of the display portion 120. The handle portion 150 and the display portion 120 are connected by bringing the one-sided contact portion 161a and the other-sided contact portion 161b into contact with each other from a detached state and inserting and screwing in the screw 162 from above. Furthermore, the handle portion 150 and the display portion 120 are detachable via the connecting portion 160. Specifically, the user can loosen the screw 162 using a tool to separate the one-sided contact portion 161a and the other-sided contact portion 161b from each other, thereby releasing the connection between the handheld portion 150 and the display portion 120. The connection portion 160 can be modified as appropriate, similar to the connection portion 130 in the first embodiment.

[0039] Here, at least the other contact portion 161b of the connection portion 160, which is on the display portion 120 side, has the same configuration as the other contact portion 131b of the connection portion 130 in the first embodiment, which is on the display portion 120 side. Therefore, in this embodiment, the display portion 120 removed from the handheld portion 150 can be connected to the head-mounted portion 110 of the first embodiment. In other words, the display portion 120 can be used in common for both the HHD20 and the HMD10. Such a configuration including the HHD20 and the HMD10 is referred to as an image display system.

[0040] The imaging unit 111 and the display unit 120 may be powered by a wired cable, or by a battery (not shown) provided in the handheld unit 150. The battery is preferably placed in the gripping portion 151 of the handheld unit 150, taking weight balance into consideration, and more preferably divided and placed towards the lower end of each pair of gripping portions 151 so that they have approximately the same weight.

[0041] Similar to the first embodiment, the image processing device 140 converts the image captured by the imaging unit 111 so that it appears as if it were captured from a different position than the physically positioned location of the imaging unit 111. In Figure 8, similar to Figure 3 of the first embodiment, the position 111Pa of the imaging unit before conversion is shown by a dashed circle, and the orientation of the imaging unit before conversion is shown as the optical axis 111Ca by a dashed line. On the other hand, the position 111Pb of the image sensor after conversion is shown by a solid circle, and the orientation of the image sensor after conversion is shown by a solid line as the optical axis 111Cb. In addition, the optical axis 120C of the display optical system (not shown) of the display unit 120 is shown by a solid line.

[0042] Figure 9 is a diagram illustrating an example where the physical position and orientation differ between the imaging unit 111 of the HHD20 and the imaging unit 111 of the HMD10 in the first embodiment. As shown in Figure 9, the positions 111Pa (HMD10) and 111Pa (HHD20) of the imaging unit, and the optical axes 111Ca (HMD10) and 111Ca (HHD20) of the imaging unit are different. In such cases, if the common display unit 120 is switched from HMD10 to HHD20, the conversion parameters related to the position and orientation conversion cannot be made to approximately match the converted imaging unit positions 111Pb and optical axes 111Cb of the imaging unit. Therefore, the conversion parameters related to position and orientation conversion may be made changeable by the user depending on the connection destination of the display unit 120. Furthermore, if the imaging unit 111 of the HMD 10 and the imaging unit 111 of the HHD 20 can be positioned in different positions and orientations, an optimal design prioritizing compactness and light weight can be achieved for each configuration.

[0043] Figure 10 is a diagram illustrating an example in which the physical position and orientation of the imaging unit 111 of the HHD20 and the imaging unit 111 of the HMD10 of the first embodiment are substantially the same. As shown in Figure 10, the position 111Pa(HMD10) and position 111Pa(HHD20) of the imaging unit, and the optical axis 111Ca(HMD10) and optical axis 111Ca(HHD20) of the imaging unit are approximately the same. In such a case, even when the common display unit 120 is switched from HMD10 to HHD20, the position 111Pb of the imaging unit and the optical axis 111Cb of the imaging unit after conversion can be made approximately the same without changing the conversion parameters related to the position and orientation conversion. Thus, if the imaging unit 111 of the HMD10 and the imaging unit 111 of the HHD20 can be positioned in substantially the same position and orientation, the image processing device 140 can perform appropriate position and orientation conversion regardless of whether the display unit 120 is connected to the HMD10 or the HHD20. In other words, the conversion parameters related to position and orientation conversion can be kept constant regardless of the connection destination of the display unit 120, and the user can improve the usability of the HMD10 and HHD20.

[0044] In the HHD20 of this embodiment, the imaging unit 111, which is the heat source, is located in the handheld unit 150 rather than the display unit 120. Therefore, similar to the first embodiment, the display unit 120 can be miniaturized, and as a result of miniaturizing the display unit 120, the HHD20 as a whole can be made smaller and lighter. The HHD20 is not limited to the configuration described above and can be modified as appropriate. For example, the head contact portion 153 may be connected to the display portion 120. Also, the HHD20 does not need to be a single integrated unit and may be configured as a separate unit.

[0045] <Third Embodiment> In the third embodiment, a case where the image display device is an HMD (video see-through type HMD) will be described. Figure 11 is a schematic diagram showing an example of the configuration of the image display system, and includes a perspective view showing an example of the configuration of the HMD 30. Figure 12 is a side view showing an example of the configuration of the HMD 30. Figure 13 is a perspective view showing an example of the configuration of the HMD 30. Components similar to those in the first embodiment are denoted by the same reference numerals and their descriptions are omitted as appropriate. The image display system comprises the HMD 30 and the image processing device 140.

[0046] The HMD30 comprises a head-mounted section 210, a non-transparent display section 230, and a connection section 130. The head-mounted unit 210 includes a plurality of position-and-orientation-changing imaging units 111, a main body unit 112, a head contact unit 213, an operation unit 114, a retraction mechanism unit 115, and a function extension unit 220. The imaging units 111, main body unit 112, operation unit 114, and retraction mechanism unit 115 have the same configuration as in the first embodiment.

[0047] The function expansion unit 220 is a part for expanding the functions of the HMD 30. Components for expanding the functions of the HMD 30 are attached to the function expansion unit 220. The function expansion unit 220 is located on the front side of the main body 112. In this embodiment, the function expansion unit 220 is a long block in the left-right direction and is detachable from the main body 112. The function expansion unit 220 has a position and orientation changing type imaging unit 221 and a sensor unit 222. By mechanically connecting the function expansion unit 220 to the main body 112, the imaging unit 221 and the sensor unit 222 are electrically connected to the image processing device 140.

[0048] The imaging unit 221, like the imaging unit 111 described above, is a part that captures images of the real world. The imaging unit 221 corresponds to an example of an imaging unit for functional expansion. The imaging unit 221 consists of multiple units (at least two) and is mounted at separate positions on the left and right sides of the functional expansion unit 220. The imaging unit 221 can be used as a substitute for the imaging unit 111. That is, even if the head-mounted unit 210 does not have an imaging unit 111, by connecting the functional expansion unit 220 to the main unit 112, images captured by the imaging unit 221 of the functional expansion unit 220 and processed can be displayed on the display unit 230. In this way, the head-mounted unit 210 can be equipped with a detachable imaging unit 221.

[0049] The sensor unit 222 detects various detection targets. The sensor unit 222 consists of one unit, but it may be composed of multiple units that detect different targets. For example, the sensor unit 222 can be a depth sensor or an optical sensor.

[0050] If the sensor unit 222 is a depth sensor, the sensor unit 222 can acquire depth information. The image processing device 140 stores the depth information acquired by the sensor unit 222 in the storage unit 142 and can use it in the 3D reconstruction unit 144. If the sensor unit 222 is an optical sensor, the light receiving unit can acquire accurate position and orientation information of the HMD 30. The image processing device 140 stores the position and orientation information acquired by the sensor unit 222 in the storage unit 142 and can use it in the synthesis unit 146.

[0051] The display unit 230 has a display-side imaging unit 231 for functional expansion. The display-side imaging unit 231 is composed of multiple units (at least two). The display-side imaging units 231 are positioned separately to the left and right at the front of the housing of the display unit 230 and capture images of the object side. In this embodiment, the display-side imaging unit 231 is positioned to track the user's hand. By using a small and lightweight type of display-side imaging unit 231 that can be driven with low power consumption, the miniaturization and weight reduction of the display unit 230 can be avoided. Note that the display unit 230 is not limited to the display-side imaging unit 231 for tracking the user's hand, but may also have an imaging unit for detecting the user's gaze. Similarly, a small and lightweight type that can be driven with low power consumption can also be used for the imaging unit that detects the user's gaze. The connecting portion 130 is a part that physically connects the head-mounting portion 210 and the display portion 230, and has the same configuration as in the first embodiment, and is denoted by the same reference numerals, so its description is omitted.

[0052] The retraction mechanism 115 is a mechanism that retracts the display unit 230 from the user's view. Figure 12 is a side view showing the state before and after the display unit 230 is moved out of the user's sight by the retraction mechanism 115. Figure 12(a) shows the state before retraction, and Figure 12(b) shows the state after retraction. The retraction mechanism 115 rotates the display unit 230 around a pivot axis O relative to the head-mounted unit 210, thereby retracting the display unit 230 from the user's view. The retraction mechanism 115 in this embodiment can be fitted with a click hinge that stops rotation at a predetermined position. However, the configuration of the retraction mechanism 115 is not limited as long as it can retract and hold the display unit 230 from the user's view.

[0053] In this way, the retraction mechanism 115 allows the user to easily switch between naked-eye observation and video see-through observation with the HMD 30 attached. When the display unit 230 is retracted from the user's view using the retraction mechanism 115, the relative position and orientation of the display unit 230 and the imaging unit 111 or the imaging unit 221 of the function extension unit 220 changes, but this does not cause any inconvenience because the user does not observe the image on the display unit 230 when it is retracted.

[0054] The head contact portion 213 is the part that comes into contact with the user's head when the head mounting portion 210 is attached to the head. In this embodiment, the head contact portion 213 is adjustable relative to the main body portion 112. As shown in Figure 13, the head contact portion 213 is adjusted by moving relative to the main body portion 112 in the height direction of the user's forehead (arrow A), the direction of the protrusion of the user's forehead (arrow B), and the direction of the tilt of the user's forehead (arrow C).

[0055] An adjustment mechanism such as a gear or rack and pinion is positioned between the main body 112 and the head contact portion 213, and the user can adjust the head contact portion 213 by operating an operating part for driving the adjustment mechanism. However, the method of adjusting the head contact portion 213 is not limited to using an adjustment mechanism. For example, the head contact portion 213 may be configured to move freely in all directions relative to the main body 112. The user may adjust the head contact portion 213 by moving it relative to the main body 112 and fixing the relative position of the head contact portion 213 and the main body 112 with a fixing member such as a screw.

[0056] By making the head contact portion 213 adjustable relative to the main body portion 112, the user can bring their forehead into contact with the head contact portion 213 at a desired position and angle. Therefore, the head mounting portion 210 can be fitted more closely to the user's head, improving the wearability of the HMD 30. Furthermore, even when the head contact portion 213 is adjusted relative to the main body portion 112, the relative position and orientation between the imaging unit 111 or the imaging unit 221 of the function extension unit 220 and the display unit 230 can be maintained. Note that the head contact portion 213 is not limited to being adjustable in the three directions described above relative to the main body portion 112, but may also be configured to be adjustable in any one or two of the three directions.

[0057] Note that the HMD30 is not limited to the configuration described above and can be modified as appropriate. For example, the function expansion unit 220 may be configured to connect to the handheld unit 150 of the HHD20 in the second embodiment. In other words, the function expansion unit 220 may be used in common for both the HHD20 and the HMD30. In this case, as shown in Figure 9 above, if the physical position and orientation of the imaging unit 221 of the function expansion unit 220 differs between the HHD20 and the HMD30, the conversion parameters related to position and orientation conversion may be made changeable by the user according to the connection destination of the function expansion unit 220. On the other hand, as shown in Figure 10 above, if the physical position and orientation of the imaging unit 221 of the function extension unit 220 are approximately the same between the HHD20 and the HMD30, the conversion parameters related to position and orientation conversion can be kept constant regardless of the connection destination of the function extension unit 220.

[0058] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0059] Furthermore, the disclosure of this embodiment includes the following configuration. (Composition 1) A holding unit having an imaging unit and held by the user, A display unit that displays an image that has been transformed from an image captured by the imaging unit to an image captured from a position different from the physically positioned position of the imaging unit, An image display device characterized by having the following features. (Configuration 2) When the holding part is held by the user, The image display device according to configuration 1, characterized in that the imaging unit is positioned above the display unit. (Composition 3) When the holding part is held by the user, The image display device according to configuration 1 or 2, characterized in that the imaging unit is arranged such that it is angled downwards when viewed from the side of the image display device. (Composition 4) The holding portion has a head contact portion that contacts the forehead of the user's head, The image display device according to any one of configurations 1 to 3, characterized in that the distance from the head contact portion to the center of gravity of the imaging portion is shorter than the distance from the head contact portion to the center of gravity of the display portion. (Composition 5) It has a connecting part that detachably connects the holding part and the display part, The image display device according to any one of configurations 1 to 4, characterized in that the connection part allows the holding part and the display part to be separated by operation by the user. (Composition 6) The holding part comprises a main body and a head contact part that contacts the user's head. The image display device according to any one of configurations 1 to 5, characterized in that the head contact portion is adjustable relative to the main body portion without changing the relative position between the imaging portion and the holding portion. (Composition 7) The image display device according to configuration 6, characterized in that the imaging unit is located in the main body. (Composition 8) The image display device according to any one of configurations 1 to 7, characterized in that the imaging unit is detachable from the holding unit. (Composition 9) The image display device according to any one of configurations 1 to 8, characterized in that the holding portion has a function extension portion to which components for extending the functions of the image display device are attached. (Composition 10) The image display device according to configuration 9, characterized in that the imaging unit is attached to the function extension unit. (Composition 11) The image display device according to any one of configurations 1 to 10, characterized in that the display unit has an imaging unit for expanding functionality. (Composition 12) The image display device according to any one of configurations 1 to 11, characterized in that the holding unit has a retraction mechanism for moving the display unit out of the user's sight. (Composition 13) The image display device according to any one of configurations 1 to 12, characterized in that the holding portion is either a head-mounted portion attached to the user's head or a hand-held portion held by the user. (Composition 14) An image display system comprising a plurality of image display devices described in any one of configurations 1 to 12, Of the plurality of image display devices, the first image display device has a holding part which is attached to the user's head. Of the plurality of image display devices, the second image display device has a holding part that is a handheld part that the user holds in their hand, The image display system is characterized in that the display unit is configured to be connectable to either the head-mounted portion of the first image display device or the handheld portion of the second image display device. (Composition 15) The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to configuration 14, characterized in that the conversion parameters used by the image processing means when converting an image remain unchanged regardless of whether the display unit is connected to the head-mounted unit or the handheld unit. (Composition 16) The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to configuration 14, characterized in that the conversion parameters when the image processing means converts an image are changed according to the head-mounted unit or the handheld unit to which the display unit is connected. (Composition 17) An image display system comprising a plurality of image display devices described in any one of configurations 1 to 12, Of the plurality of image display devices, the first image display device has a holding part which is attached to the user's head. Of the plurality of image display devices, the second image display device has a holding part that is a handheld part that the user holds in their hand, The image display system is characterized in that the imaging unit is configured to be connectable to either the head-mounted portion of the first image display device or the handheld portion of the second image display device. (Composition 18) The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to configuration 17, characterized in that the conversion parameters used when the image processing means converts an image remain unchanged regardless of whether the imaging unit is connected to the head-mounted unit or the handheld unit. (Composition 19) The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to configuration 17, characterized in that the conversion parameters when the image processing means converts an image are changed according to the head-mounted unit or the handheld unit to which the imaging unit is connected. [Explanation of Symbols]

[0060] 10: HDD (Image Display Unit) 20: HDD (Image Display Unit) 30: HDD (Image Display Unit) 110, 210: Head-mounted unit (Holding unit) 111, 221: Imaging unit 120, 230: Display unit 150: Handheld unit (Holding unit)

Claims

1. A holding unit having an imaging unit and held by the user, A display unit that displays an image that has been transformed from an image captured by the imaging unit to an image captured from a position different from the physically positioned position of the imaging unit, An image display device characterized by having the following features.

2. When the holding part is held by the user, The image display device according to claim 1, characterized in that the imaging unit is positioned above the display unit.

3. When the holding part is held by the user, The image display device according to claim 1 or 2, characterized in that the imaging unit is arranged such that it is angled downwards when the image display device is viewed from the side.

4. The holding portion has a head contact portion that contacts the forehead of the user's head, The image display device according to claim 1 or 2, characterized in that the distance from the head contact portion to the center of gravity of the imaging portion is shorter than the distance from the head contact portion to the center of gravity of the display portion.

5. It has a connecting part that detachably connects the holding part and the display part, The image display device according to claim 1 or 2, characterized in that the connection portion can be separated from the holding portion and the display portion by operation by the user.

6. The holding part comprises a main body and a head contact part that contacts the user's head. The image display device according to claim 1 or 2, characterized in that the head contact portion is adjustable relative to the main body portion without changing the relative position between the imaging portion and the holding portion.

7. The image display device according to claim 6, characterized in that the imaging unit is located in the main body.

8. The image display device according to claim 1 or 2, characterized in that the imaging unit is detachable from the holding unit.

9. The image display device according to claim 1 or 2, characterized in that the holding portion has a function extension portion to which components for extending the functions of the image display device are attached.

10. The image display device according to claim 9, characterized in that the imaging unit is attached to the function extension unit.

11. The image display device according to claim 1 or 2, characterized in that the display unit has an imaging unit for expanding functionality.

12. The image display device according to claim 1 or 2, characterized in that the holding unit has a retraction mechanism for moving the display unit out of the user's sight.

13. The image display device according to claim 1 or 2, characterized in that the holding portion is a head-mounted portion attached to the user's head or a hand-held portion held by the user.

14. An image display system comprising a plurality of image display devices as described in claim 1, Of the plurality of image display devices, the first image display device has a holding part which is attached to the user's head. Of the plurality of image display devices, the second image display device has a holding part that is held by the user's hand, The image display system is characterized in that the display unit is configured to be connectable to either the head-mounted portion of the first image display device or the handheld portion of the second image display device.

15. The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to claim 14, characterized in that the conversion parameters used by the image processing means when converting an image remain unchanged regardless of whether the display unit is connected to the head-mounted unit or the handheld unit.

16. The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to claim 14, characterized in that the conversion parameters when the image processing means converts an image are changed according to the head-mounted unit or the handheld unit to which the display unit is connected.

17. An image display system comprising a plurality of image display devices as described in claim 1, Of the plurality of image display devices, the first image display device has a holding part which is attached to the user's head. Of the plurality of image display devices, the second image display device has a holding part that is held by the user's hand, The image display system is characterized in that the imaging unit is configured to be connectable to either the head-mounted portion of the first image display device or the handheld portion of the second image display device.

18. The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to claim 17, characterized in that the conversion parameters used by the image processing means when converting an image remain unchanged regardless of whether the imaging unit is connected to the head-mounted unit or the handheld unit.

19. The system includes an image processing means that transforms an image captured by the imaging unit so that it appears as if it were captured from a position different from the physically positioned location of the imaging unit. The image display system according to claim 17, characterized in that the conversion parameters when the image processing means converts an image are changed according to the head-mounted unit or the handheld unit to which the imaging unit is connected.

Images