Control apparatus

Abstract

To provide technique enabling display of a virtual object without applying discomfort to a user.SOLUTION: A control apparatus of the present invention, is a control apparatus which controls a display device to perform such a stereoscopic display that a virtual object is arranged in a three-dimensional space by a display for a right eye and a display for a left eye. The control apparatus includes: object detection means detecting an object existing in a sight of a user out of the three-dimensional space; distance detection means detecting a distance from the display device to the object detected by the object detection means; and control means controlling the display device in such a way that the display device performs the stereoscopic display. On the basis of distance from the display device to a specific object which is present in a display direction of the virtual object and is present at a position nearest to the display device, the control means controls parallax of the virtual object between the display for the right eye and the display for the left eye, thus controls the depth position of the virtual object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a control device that controls a display device such as an HMD (head mounted display). [Background technology]

[0002] With the spread of HMDs, technology for displaying a blended real and virtual worlds has advanced. Such technology is called XR (Cross Reality), which includes AR (Augmented Reality) and MR (Mixed Reality). In XR, it is preferable to appropriately blend the real and virtual worlds to avoid causing discomfort to the user, depending on the situation in the real world and the virtual world. Patent Document 1 discloses that virtual objects are not drawn behind real objects, but are drawn only in front of them. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-62593 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology disclosed in Patent Document 1, a virtual object that is preferentially desired to be displayed in its entirety may not be displayed.

[0005] An object of the present invention is to provide a technology that can display a virtual object without giving a sense of discomfort to the user. [Means for solving the problem]

[0006] A first aspect of the present invention is a control device that controls a display device to perform a stereoscopic display in which a virtual object is arranged in a three-dimensional space using a display for the right eye and a display for the left eye, the control device comprising: an object detection means that detects an object that is present in a user's field of view in the three-dimensional space; a distance detection means that detects the distance from the display device to the object detected by the object detection means; and a control means that controls the display device to perform the stereoscopic display, wherein the control means controls the depth position of the virtual object by controlling the parallax of the virtual object between the display for the right eye and the display for the left eye based on the distance from the display device to a specific object that is the closest object to the display device in the display direction of the virtual object.

[0007] A second aspect of the present invention is a control device having a control means for controlling a display device to perform a stereoscopic display in which a virtual object is placed in a three-dimensional space using a display for the right eye and a display for the left eye, wherein the control means controls the display device to perform the stereoscopic display in which, when placing the virtual object in front of an object in the user's field of view in the three-dimensional space, when the distance from the display device to the object is a first distance, the virtual object is placed closer than when the distance is a second distance longer than the first distance.

[0008] A third aspect of the present invention is a control method for controlling a display device to perform stereoscopic display in which a virtual object is arranged in a three-dimensional space using a display for the right eye and a display for the left eye, the method comprising: an object detection step for detecting an object present in a user's field of view in the three-dimensional space; a distance detection step for detecting a distance from the display device to the object detected by the object detection step; and a control step for controlling the display device to perform the stereoscopic display, This control method is characterized by controlling the depth position of the virtual object by controlling the parallax of the virtual object between the display for the right eye and the display for the left eye based on the distance from the display device to a specific object that is the closest object to the display device and exists in the display direction of the virtual object.

[0009] A fourth aspect of the present invention is a program for causing a computer to function as each means of the control device.A fifth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each means of the control device. [Effects of the Invention]

[0010] According to the present invention, a virtual object can be displayed in a manner that does not give the user a sense of incongruity. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is an external view of a head-mounted display (HMD). [Figure 2] FIG. 2 is a block diagram showing the internal configuration of the HMD. [Figure 3] 10 is a flowchart of a virtual object display process according to the first embodiment. [Figure 4] 3 is a diagram showing the arrangement of virtual objects according to the first embodiment; FIG. [Figure 5] 4A to 4C are diagrams illustrating parallax and the like according to the first embodiment. [Figure 6] FIG. [Figure 7] FIG. [Figure 8] 10 is a flowchart of a virtual object display process according to the second embodiment. [Figure 9] FIG. 10 is a diagram showing the arrangement of virtual objects according to the second embodiment. [Figure 10] 11 is a flowchart of a virtual object display process according to the third embodiment. [Figure 11] FIG. 10 is a diagram illustrating a situation according to a fourth embodiment. [Figure 12] 13 is a flowchart of a virtual object display process according to the fourth embodiment. [Figure 13] FIG. 10 is a diagram illustrating a situation and a process according to a fourth embodiment. [Figure 14] FIG. 10 is a diagram illustrating a situation and a process according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Example 1 A first embodiment of the present invention will be described. In the first embodiment, an example in which the present invention is applied to a video see-through display (video see-through head mounted display (HMD)) will be described. The video see-through display displays a virtual space obtained by capturing an image of real space (the outside world) in approximately real time. A user wearing a video see-through display cannot directly see the real space, but can indirectly see the real space by looking at the displayed virtual space.

[0013] 1(a) and 1(b) are external views of an HMD 100 according to a first embodiment. As shown in FIG. 1(a), the HMD 100 is provided with a headband 200. A user places the HMD 100 in contact with their eyes and secures it to their head with the headband 200. The left image capturing device 108a is a camera that captures an image of real space (the outside world) for display with the left eye, and the right image capturing device 108b is a camera that captures an image of real space (the outside world) for display with the right eye. An image captured by the left image capturing device 108a is displayed on a left display panel (not shown), which is a display panel viewable through the left eyepiece 102a in FIG. 1(b). An image captured by the right image capturing device 108b is displayed on a right display panel (not shown), which is a display panel viewable through the right eyepiece 102b in FIG. 1(b). The user positions his / her left eye on the left eyepiece 102a and looks at the left display panel through the left eyepiece 102a with his / her left eye, and positions his / her right eye on the right eyepiece 102b and looks at the right display panel through the right eyepiece 102b with his / her right eye.

[0014] 2 is a block diagram showing the internal configuration of the HMD 100. The left eye 101a is the left eye of the user, and the right eye 101b is the right eye of the user.

[0015] The left eyepiece 103a is a lens for enlarging the image displayed on the left display panel 107a, and the right eyepiece 103b is a lens for enlarging the image displayed on the right display panel 107b. The user sees these enlarged images.

[0016] Left light splitter 104a transmits light from left display panel 107a (image displayed on left display panel 107a) and guides it to left eyepiece 103a. Left light splitter 104a also reflects light from the left eyepiece 103a and left eyepiece unit 102a side (subject light reflected by left eye 101a and indicating left eye 101a) and guides it to left light receiving lens 105a. Right light splitter 104b transmits light from right display panel 107b (image displayed on right display panel 107b) and guides it to right eyepiece 103b. Right light splitter 104b also reflects light from the right eyepiece 103b and right eyepiece unit 102b side (subject light reflected by right eye 101b and indicating right eye 101b) and guides it to right light receiving lens 105b.

[0017] The left light receiving lens 105a guides the light from the left light splitter 104a to the left imaging element 106a, and the right light receiving lens 105b guides the light from the right light splitter 104b to the right imaging element 106b.

[0018] The left imaging element 106a photoelectrically converts light incident from the left light receiving lens 105a. This results in an image of the left eye 101a. The right imaging element 106b photoelectrically converts light incident from the right light receiving lens 105b. This results in an image of the right eye 101b. Video data for the left eye 101a and video data for the right eye 101b are sent to the necessary blocks via bus 109.

[0019] The left display panel 107a displays an image captured by the left imaging device 108a. The left display panel 107a may also display an image in which an image of a virtual object is superimposed on an image captured by the left imaging device 108a. The right display panel 107b displays an image captured by the right imaging device 108b. The right display panel 107b may also display an image in which an image of a virtual object is superimposed on an image captured by the right imaging device 108b. The HMD 100 can perform stereoscopic display in which a virtual object is arranged in a three-dimensional space (a three-dimensional space centered on the HMD 100 (a user wearing the HMD 100)) using a display for the right eye and a display for the left eye.

[0020] The left and right imaging devices 108a, 108b are arranged at a distance corresponding to the distance between a human's right and left eyes. The left and right imaging devices 108a, 108b capture two images with parallax between the right and left eyes.

[0021] The bus 109 is a bus that enables data to be transmitted and received between blocks, and enables data to be transmitted and received between blocks connected to the bus 109.

[0022] The CPU 110 controls each block of the HMD 100 and performs various processes of the HMD 100. For example, the CPU 110 can detect the direction and position of the user's gaze using images (images of the user's eyes) captured by the left imaging element 106a and the right imaging element 106b (gaze detection). The CPU 110 can also control the object detection unit 113 to detect an object from the video (captured image) captured by the left imaging device 108a and the right imaging device 108b (object detection). The CPU 110 can also control the distance detection unit 114 to detect the distance from the HMD 100 (the user wearing the HMD 100) to an object (distance detection).

[0023] The ROM 111 stores the program for the processing executed by the CPU 110 and information required for the processing. The RAM 112 stores data of images captured by the left imaging element 106a, the right imaging element 106b, the left imaging device 108a, the right imaging device 108b, etc. The RAM 112 is also used as a work memory for temporarily storing data required for processing by the CPU 110.

[0024] The object detection unit 113 detects an object present in an image (the user's field of view in three-dimensional space) captured by the left imaging device 108a or the right imaging device 108b (object detection). Through object detection, the object detection unit 113 can obtain information such as the position, type, and size of the detected object. While an example will be described in which the foreground position of an object (virtual object or real object) is used as the position of the object, the definition of the object position is not particularly limited. For example, the center position or center of gravity position of the object may be used. Similarly, the definition of the position of the HMD 100 is not particularly limited.

[0025] The distance detection unit 114 detects (calculates) the distance from the HMD 100 to the detected object (the object detected by the object detection unit 113) using the difference between the image captured by the left imaging device 108a and the image captured by the right imaging device 108b.

[0026] 3 is a flowchart of a virtual object display process according to the first embodiment. For example, when it is determined that the virtual object display process is to be executed by upper software (program) relative to the software (program) for the virtual object display process, and a command to display a virtual object is issued from the upper software, the virtual object display process is started. The virtual object display process may be started in response to an operation for starting the virtual object display process. The virtual object is, for example, an information display frame that displays information (such as operational precautions or warnings) that should be notified at the time of display.

[0027] In step S301, CPU 110 detects the user's line of sight using images (images of the user's eyes) captured by left imaging element 106a and right imaging element 106b, and determines the line of sight direction as the display direction of the virtual object (direction in which the virtual object will be displayed). Note that the virtual object is not always displayed in the line of sight direction; the line of sight direction when the display of the virtual object begins is determined as the display direction of the virtual object. After the virtual object is displayed, the user can look away from the virtual object.

[0028] In step S302, CPU 110 extracts objects that exist in the display direction of the virtual object from the objects detected by object detection unit 113. As an example, consider a case where a user visually recognizes virtual object 403 at the position and size shown in FIG. 4(a). In this case, virtual object 403 and pillar 401 overlap in the display direction of virtual object 403. Furthermore, virtual object 403 and wall 402 overlap. Therefore, pillar 401 and wall 402 are extracted as objects that exist in the display direction of virtual object 403.

[0029] In step S303, the CPU 110 controls the distance detection unit 114 to detect the distance from the HMD 100 to the object extracted in step S302 (the distance in the display direction). For example, the CPU 110 detects the distance in a direction directly facing the HMD 100 (the front direction of the HMD 100). When the pillar 401 and the wall 402 in FIG. 4(a) are extracted, the distances L1 and L2 shown in FIG. 4(b) are detected. The distance L1 is the distance from the HMD 100 to the pillar 401, and the distance L2 is the distance from the HMD 100 to the wall 402. The CPU 110 may detect the distance in a direction parallel to the optical axis of the left imaging device 108a and the right imaging device 108b, or may detect the distance in a direction from the HMD 100 toward the virtual object.

[0030] In step S304, the CPU 110 determines the shortest distance (the distance closest to the HMD 100 that exists in the display direction of the virtual object from the HMD 100) among the distances detected in step S303. In the example of FIG. 4(b), distance L1 is selected because distance L1 is shorter than distance L2.

[0031] In step S305, the CPU 110 determines the distance from the HMD 100 to the virtual object based on the distance selected in step S304. This process can also be considered as a process of determining the depth position (position in the depth direction) of the virtual object based on the distance selected in step S304. The depth direction of the virtual object may be the same as the display direction of the virtual object, for example, the front direction of the HMD 100, a direction parallel to the optical axis of the left image capture device 108a or the right image capture device 108b, or the direction from the HMD 100 toward the virtual object. The CPU 110 determines, based on the distance selected in step S304, a position in front of the specific object (the side closer to the HMD 100) for which the distance was selected in step S304, as the depth position of the virtual object. For example, the CPU 110 determines a position a predetermined distance in front of the specific object as the depth position of the virtual object. 4(b), distance L1-L0 is determined as the distance from HMD 100 to virtual object 403 so that virtual object 403 is located at a position a predetermined distance L0 in front of pillar 401. In this way, when the distance from HMD 100 to the specific object is a first distance, the virtual object is located closer to HMD 100 than when the distance is a second distance longer than the first distance. Here, the position of the virtual object being close to the position of HMD 100 means, for example, that in the displayed three-dimensional space, the position of the virtual object is closer to the position of HMD 100 (the user wearing HMD 100), that is, closer to the center of the three-dimensional space.

[0032] Furthermore, in step S305, CPU 110 determines the parallax (amount of deviation) of the virtual object between the display for the left eye (display on left display panel 107a) and the display for the right eye (display on right display panel 107b) based on the distance from HMD 100 to the virtual object.

[0033] The relationship between the distance to an object and the parallax of the object will be described with reference to FIG. 5(a). In FIG. 5(a), a left imaging element 507 is an imaging element located inside the left imaging device 108a, and a right imaging element 508 is an imaging element located inside the right imaging device 108b. An image of an object 509 passes through the optical system of the left imaging device 108a and is formed at a position 510 on the left imaging element 507, and passes through the optical system of the right imaging device 108b and is formed at a position 512 on the right imaging element 508. FIG. 5(a) also shows a position 511 on the right imaging element 508. The positional relationship between the right imaging element 508 and the position 511 is the same as the positional relationship between the left imaging element 507 and the position 510. The distance D between the position 511 and the position 512 is the parallax, the distance F is the focal length of the imaging optical system, the distance B is the distance between the optical axis of the left imaging device 108a and the optical axis of the right imaging device 108b (the distance between the optical axes), and the distance Z is the distance from the HMD 100 to the object. In this case, the relational expression Z=(B×F) / D holds.

[0034] For example, CPU 110 calculates the parallax (corresponding to distance Z) of a virtual object between a display for the left eye and a display for the right eye using this relational expression Z=(B×F) / D. The distance from HMD 100 to the virtual object corresponds to distance Z, and the distance based on the center-to-center distance of two imaging systems or the center-to-center distance of two display systems, which is predetermined in the specifications of HMD 100, corresponds to the distance between optical axes B. The focal length of left eyepiece 103a and right eyepiece 103b corresponds to focal length F. Therefore, from this information, the parallax of a virtual object between a display for the left eye and a display for the right eye can be determined.

[0035] In step S306, CPU 110 displays the virtual object on left display panel 107a and right display panel 107b with the parallax determined in step S305 (stereoscopic display in which the virtual object is arranged in a three-dimensional space). On left display panel 107a, an image of the virtual object is displayed superimposed on the image captured by left imaging device 108a, and on right display panel 107b, an image of the virtual object is displayed superimposed on the image captured by right imaging device 108b. In HMD 100, The depth position of a virtual object is controlled by controlling the parallax (amount of displacement) of the object. For example, by displaying a virtual object with a determined parallax reflected, the virtual object can be made to appear to the user as if it were positioned at the determined depth position.

[0036] 5(b) shows an example of a display on the left display panel 107a. A pillar 501, a wall 502, and a virtual object 503 are displayed on the left display panel 107a. The pillar 501 corresponds to the pillar 401 in FIGS. 4(a) and 4(b), the wall 502 corresponds to the wall 402, and the virtual object 503 corresponds to the virtual object 403. Because the distance from the HMD 100 to the pillar 501 is shorter than the distance from the HMD 100 to the wall 502, the pillar 501 is displayed to the right (of the wall 502). Furthermore, the virtual object 503 is displayed to the right of the pillar 501 so that it is positioned in front of the pillar 501.

[0037] 5(c) shows an example of a display on the right display panel 107b. A pillar 504, a wall 505, and a virtual object 506 are displayed on the right display panel 107b. The pillar 504 corresponds to the pillar 401 in FIGS. 4(a) and 4(b), the wall 505 corresponds to the wall 402, and the virtual object 506 corresponds to the virtual object 403. Because the distance from the HMD 100 to the pillar 504 is shorter than the distance from the HMD 100 to the wall 505, the pillar 504 is displayed to the left (relative to the wall 505). Furthermore, the virtual object 506 is displayed to the left of the pillar 504 so that it is positioned in front of the pillar 504.

[0038] The virtual object display process of FIG. 3 makes it possible to display virtual objects in a way that does not give the user a sense of incongruity.

[0039] 6(a) to 6(c) will be used to describe an example (comparison example to Example 1) in which the virtual object display process of FIG. 3 is not performed. Here, as shown in FIG. 6(a), a sufficiently (more than necessary) large value is set as the parallax of the virtual object 603, and the virtual object 603 is placed sufficiently (more than necessary) in front, thereby preventing other objects from being placed in front of the virtual object 603. In FIG. 6(a), a pillar 601 exists in front of a wall 602, and a virtual object 603 is placed in front of the pillar 601. The positional relationship between the pillar 601, the wall 602, and the HMD is the same as the positional relationship between the pillar 401, the wall 402, and the HMD 100 in FIG. 4(b).

[0040] FIG. 6(b) shows an example of a display for the left eye. In the display for the left eye, a pillar 605, a wall 606, and a virtual object 607 are displayed. The pillar 605 corresponds to the pillar 601 in FIG. 6(a) and the pillar 501 in FIG. 5(b), the wall 606 corresponds to the wall 602 in FIG. 6(a) and the wall 502 in FIG. 5(b), and the virtual object 607 corresponds to the virtual object 603 in FIG. 6(a). The pillar 605 is displayed in the same manner as the pillar 501, and the wall 606 is displayed in the same manner as the wall 502. However, in order to place the virtual object 607 sufficiently closer (more than necessary) than the pillar 605, the virtual object 607 is displayed to the right of the virtual object 503 in FIG. 5(b).

[0041] FIG. 6(c) shows an example of a display for the right eye. In the display for the right eye, a pillar 608, a wall 609, and a virtual object 610 are displayed. The pillar 608 corresponds to the pillar 601 in FIG. 6(a) and the pillar 504 in FIG. 5(c), the wall 609 corresponds to the wall 602 in FIG. 6(a) and the wall 505 in FIG. 5(c), and the virtual object 610 corresponds to the virtual object 603 in FIG. 6(a). The pillar 608 is displayed similarly to the pillar 504, and the wall 609 is displayed similarly to the wall 505. However, in order to place the virtual object 610 sufficiently closer (more than necessary) than the pillar 608, the virtual object 610 is displayed more to the left than the virtual object 503 in FIG. 5(c).

[0042] Even with this simple method, it is possible to prevent other objects from being placed in front of a virtual object. However, in order to view a virtual object that is placed in front of the user more than necessary, the user may use their eye muscles to increase the angle of convergence between the lines of sight of the left and right eyes (crossing the eyes). Furthermore, using the eye muscles can cause fatigue in the user. Therefore, the above-described simple method makes the user more likely to get tired than the method according to the first embodiment (the virtual object display process in FIG. 3).

[0043] Although the example in which a position a predetermined distance L0 in front of a specific object is determined as the depth position of a virtual object has been described, this is not limiting. For example, the depth position of a virtual object may be determined based on the distance from HMD 100 to the specific object and the characteristics of the virtual object. The characteristics of the virtual object may be a depth length (length in the depth direction). For example, a position in front of the foreground of the specific object by the depth length of the virtual object may be determined as the depth position of the foreground of the virtual object. FIGS. 7(a) and 7(b) show an example (variation) of the placement of virtual objects. In FIG. 7(a), virtual object 713 is placed in front of pillar 711 and wall 712. Virtual object 713 is a virtual train, and the depth length L3 of virtual object 713 is greater than zero. In this case, if a position a distance shorter than the depth length L3 in front of the foreground of pillar 711 is determined as the foreground position of virtual object 713, virtual object 713 will sink into pillar 711. By determining a position that is a depth length L3 in front of the foreground of pillar 711 as the foreground position of virtual object 713, virtual object 713 can be placed so as not to sink into pillar 711. In FIG. 7(b), virtual object 717 is placed in front of pillar 715 and wall 716. Virtual object 717 is a virtual poster, and the depth length L4 of virtual object 717 is approximately 0. By determining a position that is a depth length L4 in front of the foreground of pillar 715 as the foreground position of virtual object 717, virtual object 717 can be placed so as to be attached to pillar 715.

[0044] Alternatively, a predetermined ratio Ratio1 greater than 0 and less than 1 may be used to determine the depth position of the virtual object as the distance from the HMD 100 to the specific object reduced by the ratio Ratio1, or as the position in front of the specific object.A predetermined ratio Ratio2 (=1-Ratio1) may be used to determine the distance from the HMD 100 to the specific object reduced by the ratio Ratio2, as the distance from the HMD 100 to the virtual object.

[0045] Also, although an example has been described in which the distance from the HMD 100 to the detected object (object detected by the object detection unit 113) is detected using the difference between the image captured by the left imaging device 108a and the image captured by the right imaging device 108b, the method of distance detection is not limited to this. For example, image plane phase difference information may be acquired and the distance may be detected (calculated) using the image plane phase difference information. The distance may also be detected using a method called LiDAR (light detection and ranging). For example, the time from when laser light is emitted until it is reflected by an object and returns may be measured, and the distance to the object may be detected (calculated) using the measured time.

[0046] Although the example has been described in which the line of sight direction at the start of displaying a virtual object is determined as the display direction of the virtual object, the display direction of the virtual object is not limited to this. For example, the display direction of each virtual object may be determined in advance. The display direction of a virtual object may also be determined by other methods.

[0047] Furthermore, although an example in which the present invention is applied to a video see-through display (video see-through HMD) has been described, display devices to which the present invention can be applied are not limited to video see-through displays. For example, the present invention can also be applied to an HMD that displays images of a virtual space unrelated to real space. The present invention can also be applied to an optical see-through display (optical see-through HMD). An optical see-through display has, for example, lenses similar to lenses in ordinary glasses, and projects an image of a virtual object onto the lenses. A user wearing an optical see-through display can directly see the real space through the lenses. Furthermore, the user can also see a virtual object projected onto the lenses. A virtual object is projected onto the lenses by providing parallax between the projection position relative to the right eye lens and the projection position relative to the left eye lens. It is placed in real space.

[0048] Although the present invention has been described as being applied to a display device, it is also applicable to various control devices that control display devices. For example, the present invention is also applicable to a controller or personal computer (PC) that is separate from the display device.

[0049] <Example 2> A second embodiment of the present invention will be described below. Note that the description of the same points (configuration and processing) as those in the first embodiment will be omitted as appropriate.

[0050] 8 is a flowchart of a virtual object display process according to Example 2. For example, similar to Example 1, when a command to display a virtual object is issued from higher-level software, the virtual object display process starts.

[0051] In step S801, CPU 110 determines the display direction of a virtual object. For example, a command for the display direction is issued from higher-level software, and CPU 110 determines the display direction according to the command. The display direction may be a predetermined direction or a direction specified by the user. In the second embodiment, the display direction of a virtual object is a direction that does not depend on the line of sight, and for example, as shown in FIG. 9, a virtual object 900 is placed at the bottom left of the field of view. Steps S802 to S806 are the same as steps S302 to S306 in the first embodiment (FIG. 3).

[0052] By the virtual object display process of FIG. 8, even when the display direction of the virtual object is specified independently of the line of sight, the virtual object can be displayed without giving the user a sense of incongruity.

[0053] Example 3 A description will be given of a third embodiment of the present invention, in which the same points (configuration and processing) as those in the first embodiment will be omitted as appropriate.

[0054] 10 is a flowchart of a virtual object display process according to Example 3. For example, similar to Example 1, when a command to display a virtual object is issued from higher-level software, the virtual object display process starts.

[0055] Steps S1001 to S1006 are the same as steps S301 to S306 in the first embodiment (FIG. 3).

[0056] In step S1007, CPU 110 determines whether a predetermined time has elapsed since performing the processing of step S1006. This predetermined time is a determination cycle for whether or not to update the display of a virtual object. If CPU 110 determines that the predetermined time has elapsed, it proceeds to step S1008, and if it determines that the predetermined time has not elapsed, it proceeds to step S1010.

[0057] In step S1008, CPU 110 determines whether the user's line of sight is directed in the display direction of the virtual object (whether the line of sight is consistent with the display direction). If CPU 110 determines that the user's line of sight is directed in the display direction of the virtual object, it proceeds to step S1009, and if it determines that the line of sight is not directed in the display direction, it proceeds to step S1010.

[0058] In step S1009, the CPU 110 determines whether or not there has been a change in the position of the virtual object in the display direction of the virtual object. If so, the process proceeds to step S1001, and if it is determined that the object position has not changed, the process proceeds to step S1010. For example, in FIG. 4(b), if the user moves to the left without changing the direction of their face (the direction in front of the HMD 100), the pillar 401 moves to the right while the virtual object 403 remains positioned in the center of the field of view, and the virtual object 403 no longer overlaps the pillar 401. In such a case, the process proceeds to step S1001. Then, the position of the virtual object 403 is updated so that the virtual object 403 is positioned a predetermined distance L0 in front of the wall 402.

[0059] In step S1010, CPU 110 determines whether a command to delete a virtual object has been received. For example, when a user performs an operation on the operation unit to delete a virtual object, the command to delete the virtual object is sent from the operation unit to CPU 110. If CPU 110 determines that a command to delete a virtual object has been received, it proceeds to step S1011, and if it determines that a command to delete a virtual object has not been received, it proceeds to step S1007.

[0060] In step S1011, CPU 110 erases the virtual object from the display on left display panel 107a and the display on right display panel 107b.

[0061] 10 allows virtual objects to be displayed without giving the user a sense of incongruity. Furthermore, the position of the virtual object can be updated appropriately when the object placement or the orientation of the HMD 100 changes, or when the user looks at the virtual object (when the user focuses on the virtual object). For example, it is possible to prevent an increase in the processing load caused by performing the process of updating the position of the virtual object more than necessary.

[0062] Note that an example has been described in which the depth position of a virtual object is controlled when both of the following two conditions are satisfied. That is, an example has been described in which the predetermined conditions for controlling the depth position of a virtual object include the following two conditions. However, the predetermined conditions for controlling the depth position of a virtual object are not limited to this. For example, the depth position of a virtual object may be controlled even if one of the following two conditions is not satisfied. That is, the predetermined conditions for controlling the depth position of a virtual object do not have to include one of the following two conditions. The condition that the user's line of sight faces the display direction of the virtual object (step S1008) The condition that there is a change in the position of the virtual object in the display direction (step S1009)

[0063] Example 4 A fourth embodiment of the present invention will be described below. Note that the description of the same points (configuration and processing) as in the first and third embodiments will be omitted as appropriate.

[0064] 11 is a diagram illustrating a situation assumed in Example 4, showing the field of view of a user sitting at a desk in front of which a PC is placed and doing desk work. A left arm 1101, a right arm 1102, and a PC 1103 are depicted as main elements.

[0065] 12 is a flowchart of a virtual object display process according to Example 4. For example, similar to Example 1, when a command to display a virtual object is issued from higher-level software, the virtual object display process starts.

[0066] Steps S1201 to S1203 are the same as steps S301 to S303 in the first embodiment (FIG. 3).

[0067] In step S1204, the CPU 110 calculates the distance detected in step S1203. Among them, the object corresponding to the shortest distance (a specific object that exists in the display direction of the virtual object from the HMD 100 and is the closest object to the HMD 100) is selected.

[0068] In step S1205, CPU 110 determines whether the visual size of the specific object is larger than the visual size of the virtual object (whether the visual size of the virtual object is smaller than the visual size of the specific object). The visual size is the size visually perceived by the user, and in the fourth embodiment, it is the size on the display surface of left display panel 107a or right display panel 107b (display size; number of pixels). If CPU 110 determines that the visual size of the specific object is larger than the visual size of the virtual object (the visual size of the virtual object is smaller than the visual size of the specific object), it proceeds to step S1206; otherwise, it proceeds to step S1213. Note that if CPU 110 determines that the visual size of the specific object is equal to the visual size of the virtual object, it may proceed to step S1206.

[0069] An example of the processing in steps S1204 and S1205 will be described with reference to FIGS. 13(a) to 13(c) and FIGS. 14(a) to 14(c).

[0070] In FIG. 13(a), the user is holding an open newspaper 1303 between his left arm 1301 and right arm 1302. FIG. 13(b) is a schematic diagram of the situation in FIG. 13(a) as seen from the side of the user. The direction of the arrow in FIG. 13(b) is the user's line of sight and also the depth direction from the user (the front direction of the HMD 100). As shown in FIG. 13(b), the newspaper 1303 is located in front of the PC 1304. In this situation, in step S1204, the newspaper 1303 is selected as a specific object. Then, in step S1205, it is determined whether the visible size of the newspaper 1303 is larger than the visible size of the virtual object. The visible size of the newspaper 1303 is the size of the newspaper 1303 shown in FIGS. 13(a) and 13(c), and the visible size of the virtual object is the size of the virtual object 1305 shown in FIG. 13(c). 13(c), the visible size of the newspaper 1303 is larger than the visible size of the virtual object 1305. Therefore, the process proceeds to step S1206.

[0071] In FIG. 14(a), the user is holding nothing in left arm 1401 and is lifting mug 1403 with right arm 1402 to bring it to their mouth. The user can see PC 1404 behind mug 1403. FIG. 14(b) is a schematic diagram of the situation in FIG. 14(a) as seen from the side of the user. The arrow in FIG. 14(b) indicates the user's line of sight and also the depth direction from the user (the front direction of HMD 100). As shown in FIG. 14(b), mug 1403 is positioned in front of PC 1404. In this situation, mug 1403 is selected as the specific object in step S1204. Then, in step S1205, it is determined whether the visible size of mug 1403 is larger than the visible size of the virtual object. The visible size of mug 1403 is the size of mug 1403 shown in Figures 14(a) and 14(c), and the visible size of the virtual object is the size of virtual object 1405 shown in Figure 14(c). As shown in Figure 14(c), the visible size of mug 1403 is smaller than the visible size of virtual object 1405. Therefore, processing proceeds to step S1213.

[0072] Step S1206 is the same as step S305 in the first embodiment (FIG. 3). In step S1206, CPU 110 determines a distance L1-L0 from HMD 100 to a virtual object using a distance L1 from HMD 100 to a specific object and a predetermined distance L0. Furthermore, in step S1206, CPU 110 determines a parallax of the virtual object between a display for the left eye and a display for the right eye based on the distance from HMD 100 to the virtual object.

[0073] Step S1207 is the same as step S306 in the first embodiment (FIG. 3). In step S1207, the CPU 110 applies the parallax determined in step S1206 to the left image. A virtual object is displayed on the display panel 107a and the right display panel 107b. For example, a virtual object 1305 is placed as shown in FIGS. 13(c) and 13(d). FIG. 13(d) is a schematic diagram of the situation in FIG. 13(c) as seen from the side of the user. As shown in FIG. 13(d), the newspaper 1303 is positioned in front of the PC 1304, and the virtual object 1305 is positioned in front of the newspaper 1303. The visible size of the virtual object 1305 is smaller than the visible size of the newspaper 1303, and it is natural to place the virtual object 1305 in front of the newspaper 1303, so this is done in the fourth embodiment.

[0074] In step S1213, CPU 110 determines whether the specified object is an easily movable object (an object that can be easily moved, such as a mug). For example, CPU 110 determines whether the specified object is being held by a user's hand or whether the specified object is the user's hand, using information about the type of detected object obtained by object detection unit 113 performing object detection. CPU 110 then determines that the specified object is an easily movable object when the specified object is being held by a user's hand or is the user's hand. If CPU 110 determines that the specified object is an easily movable object, it proceeds to step S1214. If CPU 110 determines that the specified object is not an easily movable object, it proceeds to step S1206. Therefore, even if the visual size of the virtual object is smaller than the visual size of the specified object, if the specified object is an object that cannot be easily moved, the depth position of the virtual object is controlled to a position in front of the specified object, as in the first embodiment.

[0075] In step S1214, CPU 110 selects the object corresponding to the second shortest distance among the distances detected in step S1203 (the back object that is the object that is second closest to HMD 100 in the display direction of the virtual object from HMD 100). The back object can also be considered as the object that is closest to the specific object among the objects that are behind the specific object in the display direction of the virtual object.

[0076] In step S1215, CPU 110 determines whether a background object was selected in step S1214. For example, if there is no background object, no background object is selected. If CPU 110 determines that a background object has been selected, it proceeds to step S1216, and if it determines that a background object has not been selected, it proceeds to step S1218.

[0077] In step S1216, CPU 110 determines the distance L1'-L0 from HMD 100 to the virtual object using distance L1' from HMD 100 to the background object and a predetermined distance L0. This process can also be considered as a process of determining a position that is the predetermined distance L0 in front of the background object as the depth position of the virtual object. Furthermore, in step S1216, CPU 110 determines the parallax of the virtual object between the left-eye display and the right-eye display based on the distance from HMD 100 to the virtual object. Note that if the difference (L1'-L1) between distance L1 from HMD 100 to the specific object and distance L1' from HMD 100 to the background object is shorter than the predetermined distance L0, the position in front of the specific object will be the depth position of the virtual object. In such a case, a distance shorter than the difference (L1'-L1) may be used instead of the predetermined distance L0. A sufficiently short distance may be used as the predetermined distance L0 so that the difference (L1'-L1) is not shorter than the predetermined distance L0.

[0078] In step S1218, CPU 110 determines the parallax of the virtual object between the display for the left eye and the display for the right eye to be 0. This process can also be considered as a process of determining the position at infinity as the depth position of the virtual object. If there is no background object, even if the virtual object is placed at a position at infinity, no distance inconsistency (a display in which the virtual object hides (covers) an object in front of the virtual object) will occur.

[0079] If the visible size of the virtual object is smaller than the visible size of the specific object and the specific object is an object that can be easily moved, a position behind the specific object is determined as the depth position of the virtual object by the processing of step S1216 or step S1218.

[0080] In step S1217, CPU 110 displays the virtual object on left display panel 107a and right display panel 107b with the parallax determined in step S1216 or step S1218. However, CPU 110 does not render the area of ​​the virtual object that overlaps with a specific object in the display direction of the virtual object. For example, virtual object 1405 is positioned and rendered as shown in FIG. 14(c). FIG. 14(d) is a schematic diagram of the situation in FIG. 14(c) as viewed from the side of the user. As shown in FIG. 14(d), virtual object 1405 is positioned in front of PC 1404, and mug 1403 is positioned in front of virtual object 1405. Then, as shown in FIG. 14(c), virtual object 1405 is not rendered in the area of ​​mug 1403, but is rendered avoiding the area of ​​mug 1403. This makes it possible to prevent a display (inconsistency) in which virtual object 1405 hides mug 1403 that is in front of virtual object 1405. It also makes it possible to prevent the user's view from being blocked by virtual object 1405 while the user is about to take a sip from mug 1403. If the user's view is blocked by virtual object 1405 while the user is about to take a sip from mug 1403, for example, the user may become startled and spill the contents of mug 1403.

[0081] After the process of step S1207 or step S1217 is performed, the process proceeds to step S1208. Steps S1208 to S1211 are the same as steps S1007 to S1011 in the third embodiment (FIG. 10).

[0082] An example of the processing of steps S1208 to S1211 will be described with reference to FIGS. 13(e) and 13(f) and FIGS. 14(e) and 14(f).

[0083] In FIG. 13(e), the user moves newspaper 1303 to a position where it does not overlap with virtual object 1305. In this case, in step S1210, it is determined that there has been a change in the object arrangement in the display direction of virtual object 1305, and the process proceeds to step S1201, where the position of virtual object 1305 is updated. The depth position of virtual object 1305 is updated from the position shown in FIG. 13(d) (slightly in front of newspaper 1303) to the position shown in FIG. 13(f) (slightly in front of PC 1304). By reducing the parallax of virtual object 1305, the depth position of virtual object 1305 is updated so that virtual object 1305 moves away. By reducing the parallax of virtual object 1305, the strain on the user's eyes is reduced. The parallax of the virtual object 1305 may be changed instantaneously to a target value (parallax corresponding to a position slightly in front of the PC 1304) in response to a determination that there has been a change in the position of the object in the display direction of the virtual object 1305, or may be changed gradually to the target value. In other words, when controlling the depth position of the virtual object 1305, the depth position may be changed instantaneously to the target position, or may be changed gradually to the target position. When the parallax of the virtual object 1305 is changed gradually to the target value, the virtual object 1305 gradually retreats to a position slightly in front of the PC 1304.

[0084] In FIG. 14(e), the user has moved mug 1403 to a position where it does not overlap virtual object 1405. In this case, in step S1210, it is determined that there has been a change in the position of virtual object 1405 in the display direction, and processing proceeds to step S1201, where the position of virtual object 1405 is updated. In FIG. 14(c), part of virtual object 1405 was not displayed, but now the entire virtual object 1405 is displayed. This is a natural action in which, after virtual object 1405 appears behind mug 1403, when mug 1403 is moved away, the entire virtual object 1405 can be confirmed. The depth position of virtual object 1405 remains the same as the position shown in FIG. 14(d), as shown in FIG. 14(f). The parallax of the virtual object 1405 is small enough to reduce the strain on the user's eyes.

[0085] The virtual object display process of FIG. 12 makes it possible to display virtual objects in a way that does not give the user a sense of incongruity.

[0086] Although an example has been shown in which the perceived size of a virtual object does not depend on the depth position of the virtual object, like a real object, the virtual object may be perceived as smaller as it moves away. The method for controlling the perceived size of a virtual object may differ depending on the type of virtual object. For example, in the case of a first type, the perceived size of the virtual object may be changed so that the virtual object appears smaller as it moves away, and in the case of a second type, the change in the perceived size of the virtual object may be limited. In the case of the second type, the perceived size of the virtual object may not be changed, or may be changed by a narrower amount than in the case of the first type. In the case of the second type, when the virtual object is moved a predetermined distance in the depth direction, the perceived size of the virtual object may be changed by a smaller amount than in the case of the first type. In the case of the second type, the perceived size of the virtual object may be changed so that the virtual object appears smaller as it moves away until the depth position of the virtual object reaches a predetermined position, and the perceived size of the virtual object may not be changed once the depth position is deeper than the predetermined position. In the case of a virtual object such as a text box containing text, the smaller the visual size of the virtual object, the smaller the text becomes, making it difficult to read. Therefore, by including virtual objects containing text in the second type, it is possible to prevent the text from becoming small and difficult to read.

[0087] Furthermore, when the visual size of the virtual object is smaller than the visual size of the specific object, the depth position of the virtual object may be controlled to a position behind the specific object regardless of whether the specific object is an easy-to-move object.When the specific object is an easy-to-move object, the depth position of the virtual object may be controlled to a position behind the specific object regardless of whether the visual size of the virtual object is smaller than the visual size of the specific object.

[0088] Depending on the type of virtual object, it may be possible to switch whether or not to display the virtual object using the method of the above-described embodiment (a method of displaying the virtual object by controlling the depth position of the virtual object to suppress user fatigue and avoid giving the user a sense of discomfort.) For example, when displaying a virtual object that requires urgent attention, it may be possible to always display the virtual object close (with a large parallax) without using the method of the above-described embodiment.

[0089] The above-described embodiment (including modifications) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the above-described configurations.

[0090] <Other Examples> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0091] 100: Head-mounted display (HMD) 110: CPU

Claims

1. A control device that controls a display device to perform stereoscopic display in which a virtual object is arranged in a three-dimensional space by a display for a right eye and a display for a left eye, an object detection means for detecting an object present in a user's field of view in the three-dimensional space; a distance detection unit that detects a distance from the display device to an object detected by the object detection unit; a control means for controlling the display device to perform the stereoscopic display; having The control means a parallax of the virtual object between a display for the right eye and a display for the left eye is controlled based on a distance from the display device to a specific object that is an object closest to the display device and exists in a display direction of the virtual object, thereby controlling a depth position of the virtual object; When the visual size of the virtual object is larger than the visual size of the specific object, the depth position of the virtual object is controlled to a position behind the specific object, and when the visual size of the virtual object is larger than the visual size of the specific object, the depth position of the virtual object is controlled to a position in front of the specific object. A control device comprising:

2. The control means sets the direction of the user's line of sight when display of the virtual object is started as the display direction of the virtual object. The control device according to claim 1 .

3. The display device is an optical see-through display.

3. The control device according to claim 1 or 2.

4. The display device is a video see-through display.

3. The control device according to claim 1 or 2.

5. The control means controls the depth position of the virtual object when a predetermined condition including a change in the arrangement of the object in the display direction of the virtual object is satisfied. The control device according to any one of claims 1 to 4.

6. The control means controls a depth position of the virtual object when a predetermined condition including a line of sight of the user facing a display direction of the virtual object is satisfied. The control device according to any one of claims 1 to 5.

7. The control means controls a depth position of the virtual object to a position in front of the specific object based on a distance from the display device to the specific object. The control device according to any one of claims 1 to 6.

8. The control means controls a depth position of the virtual object to a position a predetermined distance in front of the specific object. The control device according to any one of claims 1 to 7.

9. The control means controls a depth position of the virtual object to a distance obtained by reducing a distance from the display device to the specific object at a predetermined ratio, or to a position in front of the specific object. The control device according to any one of claims 1 to 7.

10. The control means controls a depth position of the virtual object based on a distance from the display device to the specific object and a depth length of the virtual object. The control device according to any one of claims 1 to 7.

11. The control means controls the depth position of the foreground of the virtual object to a position in front of the foreground of the specific object by a depth length of the virtual object. The control device according to claim 10 .

12. The object detection means detects an object from a captured image. The control device according to any one of claims 1 to 11.

13. When the control means controls the depth position of the virtual object to a position deeper than the specific object, the control means does not render a region of the virtual object that overlaps with the specific object in a display direction of the virtual object. The control device according to any one of claims 1 to 12.

14. When the visual size of the virtual object is larger than the visual size of the specific object, If the specific object is a movable object, the control means controls the depth position of the virtual object to a position deeper than the specific object; If the specific object is an immovable object, the control means controls the depth position of the virtual object to a position in front of the specific object. The control device according to any one of claims 1 to 13.

15. When the specific object is a movable object, the control means controls the depth position of the virtual object to a position deeper than the specific object. The control device according to any one of claims 1 to 13.

16. The control means controls the depth position of the virtual object so as to gradually change to a target position. The control device according to any one of claims 1 to 15.

17. A stereoscopic display is performed by placing virtual objects in a three-dimensional space using a display for the right eye and a display for the left eye. A control method for controlling a display device as described above, an object detection step of detecting an object present in a user's field of view in the three-dimensional space; a distance detection step of detecting a distance from the display device to the object detected by the object detection step; a control step of controlling the display device to perform the stereoscopic display; having In the control step, a parallax of the virtual object between a display for the right eye and a display for the left eye is controlled based on a distance from the display device to a specific object that is an object closest to the display device and exists in a display direction of the virtual object, thereby controlling a depth position of the virtual object; death, When the visual size of the virtual object is larger than the visual size of the specific object, the depth position of the virtual object is controlled to a position behind the specific object, and when the visual size of the virtual object is larger than the visual size of the specific object, the depth position of the virtual object is controlled to a position in front of the specific object. A control method comprising:

18. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 16.

19. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 16.

Images