Display control device, display control method, program

Abstract

This enables more appropriate display in video see-through type display devices. [Solution] A display device to be worn on a user's head, comprising an image acquisition means having an imaging optical system and an image sensor for capturing images of real space, and a display means for displaying images, includes a generation means for generating a second image representing the real space as seen from the user's eye position by transforming a first image, which is an image of the real space acquired by the image acquisition means, based on the distance between the user's eye and the imaging optical system, and a control means for switching the display mode of the display device between a first mode for displaying the first image and a second mode for displaying the second image.

Description

Technical Field

[0005]

[0001] The present invention relates to a display control device, a display control method, and a program.

Background Art

[0002] As a technology for real-time fusion of the real space and the virtual space, Mixed Reality (MR) technology and Augmented Reality (AR) technology are known. These technologies are technologies that seamlessly fuse the real space and the virtual space. These technologies enable an experience where virtual objects seem to exist in the real space, and thus are expected to be applied to various fields.

[0003] As shown in Patent Document 1 and Patent Document 2, as one of the devices for a user to experience MR, there is a video see-through type display device. The video see-through type display device generates a composite image by superimposing CG (Computer Graphics) on an image of the real space captured by a camera in real time. The display device displays the composite image on a display unit such as a display and presents it to the user.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In MR technology, a user desires that an image with high definition or corresponding to the position of the user's eyes (pupils) be displayed according to the situation. However, in a video see-through type display device, it is not easy to perform display with the quality or three-dimensional feeling desired by the user.

[0006] The present invention aims to enable more appropriate display in video see-through type display devices. [Means for solving the problem]

[0007] One aspect of the present invention is, A display device that is worn on the user's head, and which controls a display device having an image capturing means having an imaging optical system and an image sensor for capturing images of real space, A generation means generates a second image representing the real space as seen from the user's eye position by transforming a first image, which is an image of the real space acquired by the image acquisition means, based on the distance between the user's eye and the imaging optical system. Control means for switching the display mode of the display device between a first mode for displaying the first image and a second mode for displaying the second image, This is a display control device characterized by having [a certain feature]. [Effects of the Invention]

[0008] According to the present invention, a more appropriate display can be achieved in a video see-through type display device. [Brief explanation of the drawing]

[0009] [Figure 1] This is a cross-sectional view of the HMD according to Embodiment 1. [Figure 2] This is a diagram illustrating the image perceived by the user according to Embodiment 1. [Figure 3] This figure shows how a user views an object according to Embodiment 2. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the display device of the present invention will be described with reference to the drawings.

[0011] <Embodiment 1> Figure 1 is a schematic diagram showing a vertical cross-section of a video see-through type display device, an HMD (head-mounted display) 101. The HMD 101 is worn on the head of a user 102. In Figure 1, the direction of the user 102's visual axis is defined as the +Z axis, the axis perpendicular to the Z axis and horizontal is defined as the X axis, and the vertical direction is defined as the Y axis. The HMD 101 includes an imaging optical system 103, an image sensor 104, an image display unit 106, an observation optical system 107, an infrared camera 108, and an infrared light source 109.

[0012] The imaging optical system 103 guides light from the real world to the image sensor 104. The image sensor 104 uses the light guided by the imaging optical system 103 to image the real world. The imaging optical system 103 and the image sensor 104 acquire an image of the real world, including the object 105, as image data. The position of the imaging optical system 103 and the position of the user's pupil (eye) are separated by a distance L1 in the Z-axis direction (the optical axis direction of the imaging optical system 103). Hereafter, the imaging optical system 103 and the image sensor 104 will be collectively referred to as the "image acquisition unit 10".

[0013] The image display unit 106 displays the image (captured image) captured by the image acquisition unit 10 of the real world. The image display unit 106 can use, but is not limited to, an organic EL element or a liquid crystal display.

[0014] The observation optical system 107 delivers an enlarged virtual image (light) of the image displayed in the image display unit 106 to the left and right eyes of the user 102. The observation optical system 107 can be a "refractive system" or a "reflective system using polarization," but is not limited to these. In Embodiment 1, the optical axis of the observation optical system 107 and the optical axis of the imaging optical system 103 coincide not only in the YZ cross-section shown in Figure 1, but also in the XZ cross-section.

[0015] Furthermore, the field of view θ2 of the image acquired by the image acquisition unit 10 and the field of view θ1 of the observation optical system 107 for observing the image are the same. Here, the image used is only a portion of the image with a field of view θ2 from the wide-angle image acquired by the imaging optical system 103 and the image sensor 104.

[0016] The infrared camera 108 images the eyes of the user 102. Based on the image of the eyes captured, the infrared camera 108 calculates the line-of-sight direction of the eyes of the user 102. The infrared light source 109 irradiates the eyes of the user 102 with infrared rays (infrared light). Hereinafter, the infrared camera 108, the infrared light source 109, and the configuration for processing the information on the line-of-sight direction (not shown) are collectively referred to as the "line-of-sight detection unit". In FIG. 1, the line-of-sight detection unit is described for one eye of the user, but the same configuration is also arranged for the other eye.

[0017] The image processing unit 111 is a display control device (display control unit) that controls the HMD 101. The image processing unit 111 performs operations on images. The image processing unit 111 performs processing such as estimating the position and orientation of the HMD 101 or superimposing images using the captured image or the image displayed on the image display unit 106 (display image). The image processing unit 111 may be built into the HMD 101 or may be an external computer of the HMD 101.

[0018] FIG. 2A is a schematic diagram of the real space (external world) perceived by a user not wearing the HMD 101 with the naked eye. In FIGS. 2A to 2C, "L image" indicates an image perceived by the left eye, and "R image" indicates an image perceived by the right eye. In the real space shown in FIG. 2A, the user's hand is present at a position close to the user's eyes, and the building and the tree are present at positions far from the user's eyes. The user obtains a stereoscopic feeling by viewing images having a parallax between the right eye and the left eye.

[0019] FIG. 2B is a schematic diagram of an image perceived by a user wearing the HMD 101 at the same position as in FIG. 2A. FIG. 2B shows the captured image displayed on the HMD 101 in the "direct display mode". The "direct display mode" is a display mode in which the captured image acquired by the image capturing unit 10 is displayed as it is. In Embodiment 1, the angle of view of the captured image is adjusted to be approximately the same as the angle of view of the observation optical system 107, and the optical axes of the observation optical system 107 and the imaging optical system 103 coincide. Therefore, the user can observe the image without feeling a significant parallax (change in parallax) from the state of seeing reality with the naked eye.

[0020] Also, by using an image sensor 104 with high resolution for the image capturing unit 10, the user can observe a high-quality image. In the present embodiment, the distance L1, which is the "amount of deviation (distance) between the imaging optical system 103 (image capturing unit 10) and the user's pupil (eye) in the optical axis direction of the imaging optical system 103", is 40 mm (millimeters). The distance L1 is preferably 10 mm to 60 mm as a range that can present an image with less discomfort without causing the HMD 101 (device) to be enlarged. In Embodiment 1, the distance in the X-axis direction of the left and right imaging optical systems 103 and the distance in the X-axis direction of the left and right observation optical systems 107 are both 63 mm. However, the distance in the X-axis direction of the observation optical system 107 or the imaging optical system 103 or both of them may be adjustable according to individual differences in the user's interpupillary distance.

[0021] If the HMD 101 is in the "direct display mode", the user can experience a high-quality image with relatively little discomfort. However, when the HMD 101 is in the "direct display mode", the user sees the captured image taken from a position separated from the object 105 by the distance L2. On the other hand, when the user observes with the naked eye as shown in FIG. 2A, the user sees the object 105 separated by the distance L1 + L2. Therefore, the user perceives the hand in FIG. 2B to be slightly larger than the hand in FIG. 2A.

[0022] On the other hand, the HMD101 has a mode (hereinafter referred to as the "reconstruction display mode") that converts the captured image into an image representing the real space as seen from the position of the human pupil (eye) (hereinafter referred to as the "reconstruction image"). The HMD101 uses a technique called image-based rendering, which reconstructs three-dimensional information, to convert the captured image into a reconstruction image. The reconstruction image is an image that has been converted into an image that is simulated to be seen from the position of the user's pupil. In order to reconstruct the three-dimensional information, distance information indicating the distance between the imaging optical system 103 and the object is necessary.

[0023] Therefore, the HMD101 utilizes stereo distance measurement technology using the left and right image acquisition units 10 to acquire distance information. Specifically, the image processing unit 111 calculates the distance from the imaging optical system 103 to the object in the captured image by searching for corresponding points on the left and right sides of the image from the left and right imaging sensors, and can generate a distance map as distance information. It is also possible to select a method that uses a separate distance sensor, such as a TOF sensor, in conjunction with the HMD101 to acquire distance information. Furthermore, it is also possible to use a separate camera for distance measurement to acquire distance information.

[0024] The distance map can be interpreted as representing the three-dimensional shape of an object. The image processing unit 111 can then generate a two-dimensional image as seen from the user's left and right eye positions by performing a perspective projection transformation on the three-dimensional shape indicated by the distance map. The image processing unit 111 can generate a reconstructed image from a directly acquired image by applying the images acquired by the left and right cameras as a texture to a three-dimensional shape, thereby transforming the image from a new viewpoint. In other words, the image processing unit 111 generates a reconstructed image of "the three-dimensional shape with the texture applied, viewed from the position of an eye located at a distance L1 from the position of the imaging optical system 103." Therefore, it can also be said that the image processing unit 111 transforms the acquired image into a reconstructed image based on distances L1 and L2.

[0025] Furthermore, generating a reconstructed image requires a certain processing time in the image processing unit 111. Therefore, when the image processing unit 111 sends the reconstructed image to the image display unit 106, it is possible to reduce the perceived delay by correcting the reconstructed image based on the latest position and orientation information of the HMD 101. For this reason, the "reconstructed display mode" can reduce the perceived delay from image acquisition to display compared to the "direct display mode".

[0026] Figure 2C shows the reconstructed image. In the reconstructed image, the image shown in Figure 2B, which was taken at a distance L2 from object 105, is transformed into an image seen from a distance L1+L2 from object 105 (i.e., the positions of the user's left and right eyes). Therefore, in Figure 2C, the size of the hand and the parallax are reproduced in the same state as in Figure 2A. However, the reconstructed image has the problem that the image quality (image quality) deteriorates depending on the accuracy of the distance map (depth map). Specifically, in the reconstructed image, significant distortion occurs due to the effect of occlusion, where distance measurement is not possible, and edges tend to be softer than in the image before reconstruction. Therefore, in terms of image detail, the reconstructed image tends to be inferior to the image before reconstruction.

[0027] Thus, the "direct display mode" and the "reconstructed display mode" each have different characteristics (advantages and disadvantages). For this reason, the HMD101 can switch between the "direct display mode" and the "reconstructed display mode" depending on the situation. In Embodiment 1, the image processing unit 111 switches the display mode of the HMD101 to the mode selected by the user from among the "direct display mode" and the "reconstructed display mode". The user can select the display mode of the HMD101 by pressing the operation button installed on the HMD101. The user selects the "direct display mode" when they want to observe a higher-resolution image. On the other hand, the user selects the "reconstructed display mode" when they want to observe an image that shows a more accurate sense of depth or when they want to reduce latency.

[0028] Furthermore, when displaying a CG (virtual object) according to its position and orientation, the image processing unit 111 generates a composite image by combining (superimposing) a virtual image representing the CG onto the captured image or reconstructed image. The image display unit 106 displays the composite image. If the HMD 101 is in "direct display mode," the image processing unit 111 superimposes an image of the CG as seen from the previously determined position of the imaging optical system 103 onto the captured image. If the HMD 101 is in "reconstructed display mode," the image processing unit 111 superimposes an image of the CG as seen from the position of the user's eyes (pupils) onto the reconstructed image. As a result, even if the display mode of the HMD 101 is switched between "direct display mode" and "reconstructed display mode," the user can experience a more natural augmented reality.

[0029] According to Embodiment 1, it is possible to switch between "direct display mode" and "reconstructed display mode". Therefore, the image processing unit 111 can be set to "direct display mode" when prioritizing the detail of the image, and to "reconstructed display mode" when prioritizing the three-dimensional effect of the image. Thus, the user can view the image with the most appropriate display mode depending on the situation.

[0030] <Embodiment 2> In Embodiment 2, the HMD101 uses the gaze detection function to determine what the user is looking at (gazing at) The distance to the object is calculated. Figure 3 is a schematic diagram of a situation in which a person is looking at object 310 with both eyes. The distance L3 from object 310 to the eyes is given by the following equation 1 between the convergence angle θ3 and the interocular distance I.

number

[0031] Therefore, the image processing unit 111 can determine the distance L3 by finding the convergence angle θ3 through gaze detection. The average interpupillary distance I can also be used as the value of the interocular distance. Alternatively, the interocular distance I for each user can be determined using the gaze detection function and this value can be used.

[0032] As shown in Figures 2A and 2B, when the user is looking at an object at close range, the value of L1 / (L1+L2) is large, resulting in differences in how the image appears. Therefore, in Embodiment 2, when the image processing unit 111 determines that the user is looking at an object at close range, it sets the HMD 101 to "reconstructed display mode". When the image processing unit 111 determines that the user is looking at a distant object, it sets the HMD 101 to "direct display mode". This makes it less likely for the user to notice changes in state, allowing for observation with less discomfort.

[0033] In Embodiment 2, for example, the image processing unit 111 determines that the user is looking at a nearby object if the distance between the user's eye and the object (the object the user is looking at) is less than a first distance (for example, less than 70 cm). If the distance between the user's eye and the object is greater than or equal to the first distance (for example, 70 cm or more), the image processing unit 111 determines that the user is looking at a distant object. Alternatively, the image processing unit 111 may determine that the user is looking at a distant object if the convergence angle θ3 is less than a specific angle, and determine that the user is looking at a nearby object if the convergence angle θ3 is greater than or equal to a specific angle.

[0034] In addition, depending on the configuration of the HMD101, if the distance between the user and the object is too close, failures in three-dimensional reconstruction may occur. For this reason, if the distance between the user's eyes and the object (the object the user is looking at) is less than the second distance (for example, less than 30 cm), the image processing unit 111 may be set to "direct display mode" instead of "reconstruction display mode".

[0035] According to Embodiment 2, the user can observe an image displayed in an appropriate display mode according to the position of the object they are viewing, without having to set the display mode themselves.

[0036] <Embodiment 3> In Embodiment 3, the image processing unit 111 sets the display mode of the HMD 101 according to the processing power of the image processing unit 111.

[0037] When the image processing unit 111 is an external PC to the HMD 101, the image processing unit 111 has relatively high computing power. Therefore, the image processing unit 111 can perform computationally intensive three-dimensional reconstruction with high quality and speed. On the other hand, when the image processing unit 111 uses a small PC, mobile terminal, or the computer inside the HMD 101, it is difficult for the image processing unit 111 to perform computationally intensive reconstruction image calculations at high speed. Therefore, in Embodiment 3, the display mode of the HMD 101 is switched depending on the computing power of the image processing unit 111. Specifically, when the image processing unit 111 is a workstation PC with high computing power, the display mode of the HMD 101 is set to "reconstruction display mode". If it is a mobile terminal, set the HMD101 display mode to "Direct Display Mode".

[0038] According to Embodiment 3, the user can observe an image displayed in an appropriate display mode without having to set the display mode themselves.

[0039] <Example 1> The image processing unit 111 compares the image captured for the right eye with the image captured for the left eye, and if the difference between the two images is greater than a certain amount, it may set the display mode of the HMD 101 to "reconstructed display mode". In this case, the difference between the two images gives the user a sense of depth. For this reason, reconstruction is preferable to provide the user with an image that feels less unnatural. The image processing unit 111 compares the image captured for the right eye with the image captured for the left eye, and if the difference between the two images is less than or equal to a certain amount, it may set the display mode of the HMD 101 to "direct display mode". In this case, the degree of depth created by the two images is low, and the user can observe the image with less discomfort even without reconstruction. The "difference between the two images" may be, for example, the difference in the coordinates of a specific subject, the difference in average pixel values, or the difference in average brightness.

[0040] In the embodiments described above, the image processing unit 111 sets the display mode of the HMD 101, but the display mode setting may also be performed by a control unit (processor) or the like included in the HMD 101.

[0041] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0042] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0043] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0044] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0045] The above-disclosed embodiments include the following configurations, methods, and programs. (Composition 1) A display device that is worn on the user's head, and which controls a display device having an image capturing means having an imaging optical system and an image sensor for capturing images of real space, A generation means generates a second image representing the real space as seen from the user's eye position by transforming a first image, which is an image of the real space acquired by the image acquisition means, based on the distance between the user's eye and the imaging optical system. Control means for switching the display mode of the display device between a first mode for displaying the first image and a second mode for displaying the second image, A display control device characterized by having the following features. (Configuration 2) In the optical axis direction of the imaging optical system, the distance between the user's eye and the imaging optical system is a distance between 10 mm and 60 mm. A display control device according to configuration 1, characterized by the above. (Composition 3) In the first mode, the system has a synthesis means that generates an image by superimposing an image of a virtual object as seen from the position of the imaging optical system onto the first image, and in the second mode, it has a synthesis means that generates an image by superimposing an image of the virtual object as seen from the position of the user's eyes onto the second image. The display means displays the image generated by the synthesis means. A display control device according to configuration 1 or 2, characterized by the above. (Composition 4) The control means sets the display mode to either the first mode or the second mode based on the distance between the object the user is looking at and the user's eyes. A display control device according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The control means is If the distance between the object the user is looking at and the user's eyes is greater than or equal to a first distance, the display mode is set to the first mode. If the distance between the object the user is looking at and the user's eyes is less than the first distance, the display mode is set to the second mode. The display control device according to configuration 4, characterized by the above. (Composition 6) The control means sets the display mode to the first mode if the distance between the object the user is looking at and the user's eyes is less than the first distance, but the distance between the object the user is looking at and the user's eyes is less than a second distance which is smaller than the first distance. The display control device according to configuration 5, characterized by the features described herein. (Composition 7) The control means sets the display mode to either the first mode or the second mode based on the computational capability of the generation means. A display control device according to any one of configurations 1 to 4, characterized by the above. (Composition 8) The image sensor acquires an image for the user's right eye and an image for the user's left eye. The control means sets the display mode to either the first mode or the second mode based on the difference between the image for the right eye and the image for the left eye. A display control device according to any one of configurations 1 to 4, characterized by the above. (method) A display control method for controlling a display device that is worn on the user's head, the display device having an image capturing means having an imaging optical system and an image sensor for capturing images of real space, The first image, which is an image of the real space acquired by the image acquisition means, A generation step of generating a second image representing the real space as seen from the user's eye position by converting based on the distance between the user's eye and the imaging optical system, and A control step of switching the display mode of the display device between a first mode for displaying the first image and a second mode for displaying the second image, A display control method characterized by having the following features. (program) A program for causing a computer to function as one of the means of a display control device described in any of configurations 1 to 8. [Explanation of symbols]

[0046] 101: HMD, 10: Image acquisition unit, 103: imaging optical system, 104: image sensor, 106: image display unit, 107: Observation optical system, 111: Image processing unit

Claims

1. A display device that is worn on the user's head, and which controls a display device having an image capturing means having an imaging optical system and an image sensor for capturing images of real space, A generation means generates a second image representing the real space as seen from the user's eye position by transforming a first image, which is an image of the real space acquired by the image acquisition means, based on the distance between the user's eye and the imaging optical system. Control means for switching the display mode of the display device between a first mode for displaying the first image and a second mode for displaying the second image, A display control device characterized by having the following features.

2. In the optical axis direction of the imaging optical system, the distance between the user's eye and the imaging optical system is a distance between 10 mm and 60 mm. The display control device according to feature 1.

3. In the first mode, the system has a synthesis means that generates an image by superimposing an image of a virtual object as seen from the position of the imaging optical system onto the first image, and in the second mode, it has a synthesis means that generates an image by superimposing an image of the virtual object as seen from the position of the user's eyes onto the second image. The display means displays the image generated by the synthesis means. The display control device according to feature 1.

4. The control means sets the display mode to either the first mode or the second mode based on the distance between the object the user is looking at and the user's eyes. The display control device according to feature 1.

5. The control means is If the distance between the object the user is looking at and the user's eyes is greater than or equal to a first distance, the display mode is set to the first mode. If the distance between the object the user is looking at and the user's eyes is less than the first distance, the display mode is set to the second mode. The display control device according to feature 4.

6. The control means sets the display mode to the first mode if the distance between the object the user is looking at and the user's eyes is less than the first distance, but the distance between the object the user is looking at and the user's eyes is less than a second distance which is smaller than the first distance. The display control device according to claim 5.

7. The control means sets the display mode to either the first mode or the second mode based on the computational capability of the generation means. The display control device according to feature 1.

8. The image sensor acquires an image for the user's right eye and an image for the user's left eye. The control means sets the display mode to either the first mode or the second mode based on the difference between the image for the right eye and the image for the left eye. The display control device according to feature 1.

9. A display device worn on the user's head, comprising an image capturing means having an imaging optical system and an image sensor for capturing images of real space, and a display means for displaying images, and a control panel for the display device. A control method, A generation step of generating a second image representing the real space as seen from the user's eye position by transforming a first image, which is an image of the real space acquired by the image acquisition means, based on the distance between the user's eye and the imaging optical system. A control step of switching the display mode of the display device between a first mode for displaying the first image and a second mode for displaying the second image, A display control method characterized by having the following features.

10. A program for causing a computer to function as one of the means of a display control device according to any one of claims 1 to 8.

Images