control device
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- CANON KK
- Filing Date
- 2022-02-08
- Publication Date
- 2026-08-03
Smart Images
Figure 0007898866000001 
Figure 0007898866000002 
Figure 0007898866000003
Abstract
Description
Technical Field
[0001] The present invention relates to a control device for controlling a display device such as an HMD (Head-Mounted Display).
Background Art
[0002] With the spread of HMDs, technologies for fusing and displaying the real world and the virtual world have been advancing. Such technologies are called XR (Cross Reality), and XR includes AR (Augmented Reality), MR (Mixed Reality), etc. In XR, it is preferable to perform appropriate fused display so as not to give the user a sense of discomfort according to the situation of the real world, the virtual world, etc. Patent Document 1 discloses that virtual objects are not drawn behind real objects, but only in front of real objects.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, in the technology disclosed in Patent Document 1, there are cases where virtual objects that the user wants to display preferentially as a whole are not displayed.
[0005] An object of the present invention is to provide a technology capable of displaying virtual objects so as not to give the user a sense of discomfort.
Means for Solving the Problems
[0006] A first aspect of the present invention is a control device for controlling a display device to perform a stereoscopic display in which a virtual object is placed in a three-dimensional space using a display for the right eye and a display for the left eye, comprising: object detection means for detecting an object in the user's field of view in the three-dimensional space; distance detection means for detecting the distance from the display device to the object detected by the object detection means; and control means for controlling the display device to perform the stereoscopic display, wherein the control means controls the depth position of the virtual object by controlling the parallax of the virtual object between the display for the right eye and the display for the left eye based on the distance from the display device to a specific object which is the closest object to the display device and is located in the display direction of the virtual object, and controls the depth position of the virtual object to a position behind the specific object if the visible size of the virtual object is larger than the visible size of the specific object, and controls the depth position of the virtual object to a position behind the specific object if the visible size of the virtual object is smaller than the visible size of the specific object teeth , Even if the virtual object obscures at least a part of the specific object, The control device is characterized by controlling the depth position of the virtual object to be in front of the specified object.
[0008] A second aspect of the present invention is a control method for controlling a display device to perform a stereoscopic display in which a virtual object is placed in a three-dimensional space using a display for the right eye and a display for the left eye, comprising: an object detection step for detecting an object in the user's field of view in the three-dimensional space; a distance detection step for detecting the distance from the display device to the object detected in the object detection step; and a control step for controlling the display device to perform the stereoscopic display, wherein the control step involves detecting the object in the display direction of the virtual object from the display device, and the object closest to the display device. Based on the distance to a specific nearby object, the depth position of the virtual object is controlled by controlling the parallax of the virtual object between the right-eye display and the left-eye display, and if the visible size of the virtual object is larger than the visible size of the specific object, the depth position of the virtual object is controlled to be behind the specific object, and if the visible size of the virtual object is smaller than the visible size of the specific object, teeth , Even if the virtual object obscures at least a part of the specific object,This control method is characterized by controlling the depth position of the virtual object to be in front of the specified object.
[0009] This invention 3 This embodiment is a program for causing a computer to function as each of the means of the control device described above. 4 The embodiment is a computer-readable storage medium that stores a program for causing the computer to function as each of the means of the control device described above. [Effects of the Invention]
[0010] According to the present invention, virtual objects can be displayed in a way that does not cause discomfort to the user. [Brief explanation of the drawing]
[0011] [Figure 1] This is an external view of a head-mounted display (HMD). [Figure 2] This is a block diagram showing the internal structure of an HMD. [Figure 3] This is a flowchart of the virtual object display process according to Example 1. [Figure 4] This diagram shows the arrangement of virtual objects according to Example 1. [Figure 5] This diagram shows the parallax and other aspects related to Example 1. [Figure 6] This is a diagram showing a comparative example. [Figure 7] This is a diagram showing a modified example. [Figure 8] This is a flowchart of the virtual object display process according to Example 2. [Figure 9] This diagram shows the arrangement of virtual objects according to Example 2. [Figure 10] This is a flowchart of the virtual object display process according to Example 3. [Figure 11] This diagram shows the situation related to Example 4. [Figure 12] This is a flowchart of the virtual object display process according to Example 4. [Figure 13]It is a diagram showing the situation and processing according to Example 4. [Figure 14] It is a diagram showing the situation and processing according to Example 4.
Mode for Carrying Out the Invention
[0012] <Example 1> Example 1 of the present invention will be described. In Example 1, an example in which the present invention is applied to a video see-through type display (a video see-through type head-mounted display (HMD)) will be described. The video see-through type display displays a virtual space imaged from the real space (the outside world) almost in real time. A user wearing a video see-through type display cannot directly see the real space, but can indirectly see the real space by viewing the displayed virtual space.
[0013] Figs. 1(a) and 1(b) are external views of the HMD 100 according to Example 1. As shown in Fig. 1(a), the HMD 100 is provided with a headband 200. The user brings the HMD 100 into contact with the eyes and fixes it to the head with the headband 200. The left imaging device 108a is a camera that images the real space (the outside world) for display for the left eye, and the right imaging device 108b is a camera that images the real space (the outside world) for display for the right eye. The video imaged by the left imaging device 108a is displayed on a left display panel (not shown) which is a display panel visible through the left eyepiece 102a in Fig. 1(b). The video imaged by the right imaging device 108b is displayed on a right display panel (not shown) which is a display panel visible through the right eyepiece 102b in Fig. 1(b). The user aligns the left eye with the left eyepiece 102a, peeks at the left display panel with the left eye through the left eyepiece 102a, aligns the right eye with the right eyepiece 102b, and peeks at the right display panel with the right eye through the right eyepiece 102b.
[0014] Fig. 2 is a block diagram showing the internal configuration of the HMD 100. The left eye 101a is the left eye of the user, and the right eye 101b is the right eye of the user.
[0015] The left eyepiece 103a is a lens for magnifying the image displayed on the left display panel 107a, and the right eyepiece 103b is a lens for magnifying the image displayed on the right display panel 107b. The user will see these magnified images.
[0016] The left light splitter 104a transmits light from the left display panel 107a (the image displayed on the left display panel 107a) and directs it to the left eyepiece lens 103a. The left light splitter 104a also reflects light from the left eyepiece lens 103a and the left eyepiece section 102a (the subject light indicating the left eye 101a, reflected by the left eye 101a) and directs it to the left light receiving lens 105a. The right light splitter 104b transmits light from the right display panel 107b (the image displayed on the right display panel 107b) and directs it to the right eyepiece lens 103b. The right light splitter 104b also reflects light from the right eyepiece lens 103b and the right eyepiece section 102b (the subject light indicating the right eye 101b, reflected by the right eye 101b) and directs it to the right light receiving lens 105b.
[0017] The left light-receiving lens 105a guides light from the left light splitter 104a to the left image sensor 106a, and the right light-receiving lens 105b guides light from the right light splitter 104b to the right image sensor 106b.
[0018] The left image sensor 106a converts the light incident from the left light-receiving lens 105a into photoelectric power. This captures an image of the left eye 101a. The right image sensor 106b converts the light incident from the right light-receiving lens 105b into photoelectric power. This captures an image of the right eye 101b. The image data from the left eye 101a and the image data from the right eye 101b are sent to the necessary blocks via the bus 109.
[0019] The left display panel 107a displays the image captured by the left imaging device 108a. The left display panel 107a may also display an image in which the image of a virtual object is superimposed on the image captured by the left imaging device 108a. The right display panel 107b displays the image captured by the right imaging device 108b. The right display panel 107b may also display an image in which the image of a virtual object is superimposed on the image captured by the right imaging device 108b. The HMD 100 can perform a three-dimensional display by placing virtual objects in a three-dimensional space (a three-dimensional space centered on the HMD 100 (the user wearing the HMD 100)) using the display for the right eye and the display for the left eye.
[0020] The left imaging device 108a and the right imaging device 108b are positioned with a distance corresponding to the distance between the right and left eyes of a human being. The left imaging device 108a and the right imaging device 108b capture two images with parallax between the right and left eyes.
[0021] Bus 109 is a bus that enables the transmission and reception of data between blocks, allowing data to be sent and received between blocks connected to bus 109.
[0022] The CPU 110 controls each block of the HMD 100 and performs various processes on the HMD 100. For example, the CPU 110 can detect the user's gaze direction and gaze position using images captured by the left image sensor 106a and the right image sensor 106b (images of the user's eyes) (gaze detection). The CPU 110 can also control the object detection unit 113 to detect objects from images captured by the left imaging device 108a and the right imaging device 108b (object detection). The CPU 110 can also control the distance detection unit 114 to detect the distance from the HMD 100 (the user wearing the HMD 100) to an object (distance detection).
[0023] ROM111 contains the program for the process executed by CPU110 and the information necessary for that process. These are pre-stored. RAM 112 stores video data captured by the left image sensor 106a, the right image sensor 106b, the left imaging device 108a, the right imaging device 108b, etc. RAM 112 is also used as work memory to temporarily store data necessary for processing by the CPU 110.
[0024] The object detection unit 113 detects objects present in an image (the user's field of view in three-dimensional space) captured by the left imaging device 108a and the right imaging device 108b (object detection). Through object detection, the object detection unit 113 can obtain information such as the position, type, and size of the detected object. An example is given in which the position of the foremost object is used as the position of the object (virtual or real object), but the definition of the object's position is not particularly limited. For example, the center position or the center of gravity position of the object may be used. Similarly, the definition of the position of the HMD 100 is not particularly limited.
[0025] The distance detection unit 114 uses the difference between the image captured by the left imaging device 108a and the image captured by the right imaging device 108b to detect (calculate) the distance from the HMD 100 to the detected object (the object detected by the object detection unit 113).
[0026] Figure 3 is a flowchart of the virtual object display process according to Embodiment 1. For example, when a higher-level software (program) determines that it is time to execute the virtual object display process on the virtual object display processing software (program), and the higher-level software issues a command to display the virtual object, the virtual object display process starts. The virtual object display process may also start in response to an operation to start the virtual object display process. A virtual object is, for example, an information display frame that shows information to be notified at the time of display (such as operational precautions or warnings).
[0027] In step S301, the CPU 110 detects the user's gaze direction using the images captured by the left image sensor 106a and the right image sensor 106b (images of the user's eyes), and determines that gaze direction as the display direction of the virtual object (the direction in which the virtual object will be displayed). Note that the virtual object is not always displayed in the gaze direction; the gaze direction at the start of displaying the virtual object is determined as the display direction of the virtual object. After the virtual object is displayed, the user can look away from the virtual object.
[0028] In step S302, the CPU 110 extracts objects from the objects detected by the object detection unit 113 that are located in the display direction of the virtual object. As an example, consider the case where the virtual object 403 is made visible to the user at the position and size shown in Figure 4(a). In this case, the virtual object 403 and the column 401 overlap in the display direction of the virtual object 403. Also, the virtual object 403 and the wall 402 overlap. Therefore, the column 401 and the wall 402 are extracted as objects located in the display direction of the virtual object 403.
[0029] In step S303, the CPU 110 controls the distance detection unit 114 to detect the distance (distance in the display direction) from the HMD 100 to the object extracted in step S302. For example, the CPU 110 detects the distance in the direction directly facing the HMD 100 (the direction in front of the HMD 100). If the pillar 401 and wall 402 in Figure 4(a) are extracted, the distances L1 and L2 shown in Figure 4(b) are detected. Distance L1 is the distance from the HMD 100 to the pillar 401, and distance L2 is the distance from the HMD 100 to the wall 402. The CPU 110 may also detect the distance in a direction parallel to the optical axis of the left imaging device 108a or the right imaging device 108b, or it may detect the distance in the direction from the HMD 100 toward the virtual object.
[0030] In step S304, the CPU 110 determines the shortest distance among the distances detected in step S303 (from HMD 100, in the direction of display of the virtual object, and closest to HMD 100). Select the nearest object (the distance to a specific object). In the example in Figure 4(b), distance L1 is shorter than distance L2, so distance L1 is selected.
[0031] In step S305, the CPU 110 determines the distance from the HMD 100 to the virtual object based on the distance selected in step S304. This process can also be understood as determining the depth position (position in the depth direction) of the virtual object based on the distance selected in step S304. The depth direction for the virtual object may be the same as the display direction of the virtual object, for example, the direction in front of the HMD 100, the direction parallel to the optical axis of the left imaging device 108a or the right imaging device 108b, or the direction from the HMD 100 toward the virtual object. The CPU 110 determines the depth position of the virtual object as a position based on the distance selected in step S304, which is in front of (closer to the HMD 100 than) the specific object whose distance was selected in step S304. For example, the CPU 110 determines the depth position of the virtual object as a position a predetermined distance in front of the specific object. In the example shown in Figure 4(b), the distance L1-L0 is determined as the distance from the HMD100 to the virtual object 403, such that the virtual object 403 is positioned a predetermined distance L0 in front of the column 401. This ensures that when the distance from the HMD100 to a specific object is a first distance, the virtual object is positioned closer to the HMD100 than when the distance is a second distance, which is longer than the first distance. Here, "close to the HMD100" means, for example, that in the displayed 3D space, the virtual object is close to the position of the HMD100 (the user wearing the HMD100), that is, close to the center of the 3D space.
[0032] Furthermore, in step S305, the CPU 110 determines the parallax (amount of shift) of the virtual object between the display for the left eye (display on the left display panel 107a) and the display for the right eye (display on the right display panel 107b) based on the distance from the HMD 100 to the virtual object.
[0033] The relationship between the distance to an object and the object's parallax will be explained using Figure 5(a). In Figure 5(a), the left image sensor 507 is an image sensor located inside the left imaging device 108a, and the right image sensor 508 is an image sensor located inside the right imaging device 108b. The object 509 is imaged at position 510 of the left image sensor 507 via the optical system of the left imaging device 108a, and then imaged at position 512 of the right image sensor 508 via the optical system of the right imaging device 108b. Figure 5(a) also shows the position 511 of the right image sensor 508. The positional relationship between the right image sensor 508 and position 511 is the same as the positional relationship between the left image sensor 507 and position 510. The distance D between positions 511 and 512 is the parallax, the distance F is the focal length of the imaging optical system, the distance B is the distance between the optical axis of the left imaging device 108a and the optical axis of the right imaging device 108b (inter-axial distance), and the distance Z is the distance from the HMD 100 to the object. In this case, the relationship Z = (B × F) / D holds true.
[0034] For example, the CPU 110 uses the relationship Z = (B × F) / D to calculate the parallax of the virtual object between the left-eye display and the right-eye display (corresponding to distance Z). The distance from the HMD 100 to the virtual object corresponds to distance Z, and the distance based on the distance between the centers of the two imaging systems and the distance between the centers of the two display systems, which are predetermined in the specifications of the HMD 100, corresponds to the optical axis distance B. The focal lengths of the left eyepiece 103a and the right eyepiece 103b correspond to focal length F. Therefore, from this information, the parallax of the virtual object between the left-eye display and the right-eye display can be determined.
[0035] In step S306, the CPU 110 displays the virtual object on the left display panel 107a and the right display panel 107b with the parallax determined in step S305 (3D display with the virtual object placed in 3D space). On the left display panel 107a, the image of the virtual object is displayed superimposed on the image captured by the left imaging device 108a, and on the right display panel 107b, the image of the virtual object is displayed superimposed on the image captured by the right imaging device 108b. In the HMD 100, the virtual The depth position of a virtual object is controlled by controlling the parallax (amount of displacement) of the object. For example, by displaying the virtual object while reflecting the determined parallax, the virtual object can be made to appear to the user as if it were positioned at the determined depth position.
[0036] Figure 5(b) shows an example of the display on the left display panel 107a. On the left display panel 107a, a column 501, a wall 502, and a virtual object 503 are displayed. Column 501 corresponds to column 401 in Figures 4(a) and 4(b), wall 502 corresponds to wall 402, and virtual object 503 corresponds to virtual object 403. Since the distance from HMD 100 to column 501 is shorter than the distance from HMD 100 to wall 502, column 501 is displayed to the right (of wall 502). The virtual object 503 is then displayed to the right of column 501 so that it is positioned in front of column 501.
[0037] Figure 5(c) shows an example of the display on the right display panel 107b. On the right display panel 107b, the column 504, wall 505, and virtual object 506 are displayed. Column 504 corresponds to column 401 in Figures 4(a) and 4(b), wall 505 corresponds to wall 402, and virtual object 506 corresponds to virtual object 403. Since the distance from HMD 100 to column 504 is shorter than the distance from HMD 100 to wall 505, column 504 is displayed to the left (of wall 505). The virtual object 506 is then displayed to the left of column 504 so that it is positioned in front of column 504.
[0038] The virtual object display process shown in Figure 3 allows for the display of virtual objects in a way that does not cause discomfort to the user.
[0039] Using Figures 6(a) to 6(c), we will explain an example of a case where the virtual object display processing in Figure 3 is not performed (a comparative example to Example 1). Here, as shown in Figure 6(a), we suppress the placement of other objects in front of the virtual object 603 by setting a sufficiently (unnecessarily) large value as the parallax of the virtual object 603 and positioning the virtual object 603 sufficiently (unnecessarily) in front of it. In Figure 6(a), a column 601 is in front of the wall 602, and the virtual object 603 is positioned in front of the column 601. The positional relationship between the column 601, the wall 602, and the HMD is the same as the positional relationship between the column 401, the wall 402, and the HMD 100 in Figure 4(b).
[0040] Figure 6(b) shows an example of a display for the left eye. In the left-eye display, column 605, wall 606, and virtual object 607 are shown. Column 605 corresponds to column 601 in Figure 6(a) and column 501 in Figure 5(b), wall 606 corresponds to wall 602 in Figure 6(a) and wall 502 in Figure 5(b), and virtual object 607 corresponds to virtual object 603 in Figure 6(a). Column 605 is displayed similarly to column 501, and wall 606 is displayed similarly to wall 502. However, in order to position virtual object 607 sufficiently (more than necessary) in front of column 605, virtual object 607 is displayed further to the right than virtual object 503 in Figure 5(b).
[0041] Figure 6(c) shows an example of a display for the right eye. In the display for the right eye, column 608, wall 609, and virtual object 610 are displayed. Column 608 corresponds to column 601 in Figure 6(a) and column 504 in Figure 5(c), wall 609 corresponds to wall 602 in Figure 6(a) and wall 505 in Figure 5(c), and virtual object 610 corresponds to virtual object 603 in Figure 6(a). Column 608 is displayed similarly to column 504, and wall 609 is displayed similarly to wall 505. However, in order to position virtual object 610 sufficiently (more than necessary) in front of column 608, virtual object 610 is displayed further to the left than virtual object 503 in Figure 5(c).
[0042] Even such a simple method can prevent other objects from being placed in front of a virtual object. However, in order to see the virtual object that is placed in front more than necessary, the user uses their eye muscles to increase the convergence angle between the lines of sight of the left and right eyes (crossing their eyes). It is necessary. Furthermore, the use of eye muscles leads to user fatigue. Therefore, the simple method described above is more likely to cause user fatigue compared to the method according to Example 1 (virtual object display processing in Figure 3).
[0043] The example described here involves determining the depth position of a virtual object at a predetermined distance L0 in front of a specific object, but this is not the only example. For instance, the depth position of a virtual object may be determined based on the distance from the HMD 100 to the specific object and the characteristics of the virtual object. The characteristics of the virtual object include its depth length (length in the depth direction). For example, the position in front of the very front of the specific object by the depth length of the virtual object may be determined as the depth position of the very front of the virtual object. Figures 7(a) and 7(b) show an example (modified) of the arrangement of virtual objects. In Figure 7(a), a virtual object 713 is placed in front of a column 711 and a wall 712. The virtual object 713 is a virtual train, and its depth length L3 is greater than 0. In this case, if the position in front of the very front of the virtual object 713 is determined to be a distance shorter than the depth length L3 from the very front of the column 711, the virtual object 713 will sink into the column 711. By determining the position of the virtual object 713 to be a distance L3 from the frontmost edge of column 711, the virtual object 713 can be positioned so that it does not sink into column 711. In Figure 7(b), a virtual object 717 is positioned in front of column 715 and wall 716. The virtual object 717 is a virtual poster, and its depth L4 is approximately 0. By determining the position of the virtual object 717 to be a distance L4 from the frontmost edge of column 715, the virtual object 717 can be positioned so that it is attached to column 715.
[0044] Alternatively, a predetermined ratio Ratio1, greater than 0 and less than 1, may be used to determine the distance from the HMD100 to a specific object by reducing it by Ratio1, and the position in front of the specific object may be used as the depth position of the virtual object. Alternatively, a predetermined ratio Ratio2 (=1-Ratio1) may be used to determine the distance from the HMD100 to the virtual object by reducing it by Ratio2.
[0045] Furthermore, while an example was described in which the distance from the HMD 100 to the detected object (the object detected by the object detection unit 113) is detected using the difference between the image captured by the left imaging device 108a and the image captured by the right imaging device 108b, the method of distance detection is not limited to this. For example, image plane phase difference information may be acquired and the distance may be detected (calculated) using the image plane phase difference information. Distance may also be detected using a method called LiDAR (light detection and ranging). For example, the time from when a laser beam is emitted until it is reflected back by an object may be measured and the distance to the object may be detected (calculated) using the measured time.
[0046] Furthermore, while we have described an example where the viewing direction at the start of displaying a virtual object is determined as the display direction of the virtual object, the display direction of a virtual object is not limited to this. For example, the display direction of each virtual object may be predetermined. The display direction of a virtual object may also be determined by other methods.
[0047] Furthermore, although an example of applying the present invention to a video see-through display (video see-through HMD) has been described, the display devices to which the present invention can be applied are not limited to video see-through displays. For example, the present invention can also be applied to HMDs that display images of a virtual space unrelated to the real world. The present invention can also be applied to optical see-through displays (optical see-through HMDs). An optical see-through display has, for example, lenses similar to those of ordinary eyeglasses, and projects images of virtual objects onto the lenses. A user wearing an optical see-through display can directly see the real world through the lenses. Furthermore, the user can also see the virtual objects projected onto the lenses. The virtual objects are projected with parallax between the projection position on the right eye lens and the projection position on the left eye lens. It will be placed in real space.
[0048] Furthermore, although an example of applying the present invention to a display device has been described, the present invention is applicable to various control devices that control display devices. For example, the present invention can also be applied to controllers or personal computers (PCs) that are separate from the display device.
[0049] <Example 2> Embodiment 2 of the present invention will now be described. Note that explanations of the same aspects (configuration and processing) as in Embodiment 1 will be omitted as appropriate.
[0050] Figure 8 is a flowchart of the virtual object display process according to Example 2. For example, as in Example 1, when a command to display a virtual object is issued from higher-level software, the virtual object display process starts.
[0051] In step S801, the CPU 110 determines the display direction of the virtual object. For example, a command for the display direction is issued from higher-level software, and the CPU 110 determines the display direction according to that command. The display direction may be a predetermined direction or a direction specified by the user. In Embodiment 2, the display direction of the virtual object is independent of the line of sight direction, and for example, as shown in Figure 9, the virtual object 900 is positioned in the lower left of the field of view. Steps S802 to S806 are the same as steps S302 to S306 in Embodiment 1 (Figure 3).
[0052] The virtual object display process shown in Figure 8 allows virtual objects to be displayed in a way that does not cause discomfort to the user, even when the display direction of the virtual object is specified independently of the user's line of sight.
[0053] <Example 3> Embodiment 3 of the present invention will now be described. Note that explanations of aspects (configuration and processing) that are the same as those in Embodiment 1 will be omitted as appropriate.
[0054] Figure 10 is a flowchart of the virtual object display process according to Example 3. For example, as in Example 1, when a command to display a virtual object is issued from higher-level software, the virtual object display process starts.
[0055] Steps S1001 to S1006 are the same as steps S301 to S306 in Example 1 (Figure 3).
[0056] In step S1007, the CPU 110 determines whether a predetermined time has elapsed since the processing in step S1006. This predetermined time is the period for determining whether or not to update the display of the virtual object. If the CPU 110 determines that the predetermined time has elapsed, it proceeds to step S1008; otherwise, it proceeds to step S1010.
[0057] In step S1008, the CPU 110 determines whether the user's gaze direction is directed towards the display direction of the virtual object (whether the gaze direction matches the display direction). If the CPU 110 determines that the user's gaze direction is directed towards the display direction of the virtual object, it proceeds to step S1009; otherwise, it proceeds to step S1010.
[0058] In step S1009, the CPU 110 determines whether or not there has been a change in the arrangement of the virtual objects in the display direction. The CPU 110 determines that the arrangement of the objects has changed. If the object's position has not changed, the process proceeds to step S1001. For example, in Figure 4(b), if the user moves to the left without changing their head orientation (the direction the HMD 100 is facing forward), the column 401 moves to the right while the virtual object 403 remains in the center of the field of view, and the virtual object 403 no longer overlaps the column 401. In such a case, the process proceeds to step S1001. The position of the virtual object 403 is then updated so that it is positioned at a predetermined distance L0 in front of the wall 402.
[0059] In step S1010, the CPU 110 determines whether or not it has received a command to erase a virtual object. For example, if a user performs an operation on the operation unit to erase a virtual object, a command to erase the virtual object is sent from the operation unit to the CPU 110. If the CPU 110 determines that it has received a command to erase the virtual object, it proceeds to step S1011; otherwise, it proceeds to step S1007.
[0060] In step S1011, the CPU 110 removes the virtual object from the display on the left display panel 107a and the display on the right display panel 107b.
[0061] The virtual object display process shown in Figure 10 allows virtual objects to be displayed in a way that does not cause discomfort to the user. Furthermore, the position of the virtual object can be appropriately updated when the object's placement or the orientation of the HMD 100 changes, or when the user looks at the virtual object (when the user focuses on the virtual object). For example, it is possible to suppress the increase in processing load caused by performing the virtual object position update process more than necessary.
[0062] Furthermore, an example was described in which the depth position of a virtual object is controlled when both of the following two conditions are met. In other words, an example was described in which the predetermined conditions for controlling the depth position of a virtual object include the following two conditions. However, the predetermined conditions for controlling the depth position of a virtual object are not limited to these. For example, the depth position of a virtual object may be controlled even if one of the following two conditions is not met. In other words, the predetermined conditions for controlling the depth position of a virtual object do not have to include one of the following two conditions. • The user's gaze direction is aligned with the direction in which the virtual object is displayed (Step S1008) • The condition that there has been a change in the arrangement of the virtual object in the display direction (Step S1009)
[0063] <Example 4> Embodiment 4 of the present invention will now be described. Note that explanations of the same aspects (configuration and processing) as in Embodiments 1 and 3 will be omitted as appropriate.
[0064] Figure 11 is a diagram illustrating the situation assumed in Example 4, representing the field of view of a user sitting at a desk with a PC on it and performing desk work. The left arm 1101, right arm 1102, and PC 1103 are represented as the main elements.
[0065] Figure 12 is a flowchart of the virtual object display process according to Example 4. For example, as in Example 1, when a command to display a virtual object is issued from higher-level software, the virtual object display process starts.
[0066] Steps S1201 to S1203 are the same as steps S301 to S303 in Example 1 (Figure 3).
[0067] In step S1204, the CPU 110 determines the distance detected in step S1203 The object corresponding to the shortest distance is selected (a specific object that is the closest to the HMD100 and exists in the display direction of the virtual object from the HMD100).
[0068] In step S1205, the CPU 110 determines whether the visible size of the specific object is larger than the visible size of the virtual object (or whether the visible size of the virtual object is smaller than the visible size of the specific object). The visible size is the size that the user sees, and in Embodiment 4, it is the size (display size; number of pixels) on the display surface of the left display panel 107a or the right display panel 107b. If the CPU 110 determines that the visible size of the specific object is larger than the visible size of the virtual object (or that the visible size of the virtual object is smaller than the visible size of the specific object), it proceeds to step S1206; otherwise, it proceeds to step S1213. The CPU 110 may also proceed to step S1206 if it determines that the visible size of the specific object is equal to the visible size of the virtual object.
[0069] Using Figures 13(a) to 13(c) and Figures 14(a) to 14(c), we will explain an example of the process in steps S1204 and S1205.
[0070] In Figure 13(a), the user is holding a newspaper 1303 open with their left arm 1301 and right arm 1302. Figure 13(b) is a schematic diagram of the situation in Figure 13(a) as seen from the side of the user. The direction of the arrow in Figure 13(b) is the user's line of sight and also the depth direction from the user (the direction in front of the HMD 100). As shown in Figure 13(b), the newspaper 1303 is positioned in front of the PC 1304. In this situation, in step S1204, the newspaper 1303 is selected as a specific object. Then, in step S1205, it is determined whether the visible size of the newspaper 1303 is larger than the visible size of the virtual object. The visible size of the newspaper 1303 is the size of the newspaper 1303 shown in Figures 13(a) and 13(c), and the visible size of the virtual object is the size of the virtual object 1305 shown in Figure 13(c). As shown in Figure 13(c), the visible size of the newspaper 1303 is larger than the visible size of the virtual object 1305. Therefore, the process proceeds to step S1206.
[0071] In Figure 14(a), the user is holding nothing in their left arm 1401 and is lifting a mug 1403 with their right arm 1402 to bring it to their mouth. The user can see the PC 1404 behind the mug 1403. Figure 14(b) is a schematic diagram of the situation in Figure 14(a) as seen from the side of the user. The direction of the arrow in Figure 14(b) is the direction of the user's line of sight and also the direction of depth from the user (the direction in front of the HMD 100). As shown in Figure 14(b), the mug 1403 is positioned in front of the PC 1404. In this situation, in step S1204, the mug 1403 is selected as the specific object. Then, in step S1205, it is determined whether the visible size of the mug 1403 is larger than the visible size of the virtual object. The visible size of mug 1403 is the size of mug 1403 shown in Figures 14(a) and 14(c), and the visible size of the virtual object is the size of virtual object 1405 shown in Figure 14(c). As shown in Figure 14(c), the visible size of mug 1403 is smaller than the visible size of virtual object 1405. Therefore, the process proceeds to step S1213.
[0072] Step S1206 is the same as step S305 in Example 1 (Figure 3). In step S1206, the CPU 110 determines the distance L1-L0 from the HMD 100 to the virtual object using the distance L1 from the HMD 100 to the specific object and a predetermined distance L0. Furthermore, in step S1206, the CPU 110 determines the parallax of the virtual object between the left-eye display and the right-eye display based on the distance from the HMD 100 to the virtual object.
[0073] Step S1207 is the same as step S306 in Example 1 (Figure 3). In step S1207, the CPU 110 applies the parallax determined in step S1206 to the left Virtual objects are displayed on display panel 107a and right display panel 107b. For example, virtual object 1305 is positioned as shown in Figures 13(c) and 13(d). Figure 13(d) is a schematic diagram of the situation in Figure 13(c) as seen from the side of the user. As shown in Figure 13(d), the newspaper 1303 is positioned in front of the PC 1304, and the virtual object 1305 is positioned in front of the newspaper 1303. The viewing size of the virtual object 1305 is smaller than the viewing size of the newspaper 1303, and it is natural to position the virtual object 1305 in front of the newspaper 1303, so this is done in Example 4.
[0074] In step S1213, the CPU 110 determines whether the specified object is an easily movable object (an object that can be easily moved, such as a mug). For example, the CPU 110 uses the information on the type of detected object obtained by the object detection unit 113 to determine whether the specified object is being held in the user's hand or whether the specified object is the user's hand. The CPU 110 then determines that the specified object is an easily movable object if it is being held in the user's hand or is the user's hand. If the CPU 110 determines that the specified object is an easily movable object, it proceeds to step S1214; if it determines that the specified object is not an easily movable object, it proceeds to step S1206. Therefore, even if the visible size of the virtual object is smaller than the visible size of the specified object, if the specified object is not easily movable, the depth position of the virtual object is controlled to be in front of the specified object, similar to Example 1.
[0075] In step S1214, the CPU 110 selects the object corresponding to the second shortest distance among the distances detected in step S1203 (the background object, which is the second closest object to the HMD 100, located in the display direction of the virtual object from the HMD 100). The background object can also be considered as the object closest to the specific object among the objects located behind the specific object in the display direction of the virtual object.
[0076] In step S1215, the CPU 110 determines whether or not a background object was selected in step S1214. For example, if no background object exists, no background object is selected. If the CPU 110 determines that a background object was selected, it proceeds to step S1216; if it determines that no background object was selected, it proceeds to step S1218.
[0077] In step S1216, the CPU 110 determines the distance L1'-L0 from the HMD 100 to the virtual object using the distance L1' from the HMD 100 to the object behind it and a predetermined distance L0. This process can also be understood as determining the depth position of the virtual object at a position a predetermined distance L0 in front of the object behind it. Furthermore, in step S1216, the CPU 110 determines the parallax of the virtual object between the left-eye display and the right-eye display based on the distance from the HMD 100 to the virtual object. Note that if the difference (L1'-L1) between the distance L1 from the HMD 100 to a specific object and the distance L1' from the HMD 100 to the object behind it is shorter than the predetermined distance L0, the depth position of the virtual object will be at a position in front of the specific object. In such cases, a distance shorter than the above difference (L1'-L1) can be used instead of the predetermined distance L0. To ensure that the above difference (L1'-L1) is not shorter than the predetermined distance L0, a sufficiently short distance may be used as the predetermined distance L0.
[0078] In step S1218, the CPU 110 determines that the parallax of the virtual object between the left-eye display and the right-eye display is 0. This process can also be understood as determining the position at infinity as the depth position of the virtual object. If there is no object behind it, placing the virtual object at infinity will not result in a distance inconsistency (a display where the virtual object hides (covers) an object closer to it).
[0079] If the visible size of the virtual object is smaller than the visible size of the specific object, and the specific object is an object that can be easily moved, then the processing in step S1216 or step S1218 determines that the position behind the specific object is the depth position of the virtual object.
[0080] In step S1217, the CPU 110 displays virtual objects on the left display panel 107a and the right display panel 107b with the parallax determined in step S1216 or step S1218. However, the CPU 110 does not draw areas of the virtual objects where a specific object overlaps in the display direction of the virtual objects. For example, as shown in Figure 14(c), virtual object 1405 is positioned and drawn. Figure 14(d) is a schematic diagram of the situation in Figure 14(c) as seen from the side of the user. As shown in Figure 14(d), virtual object 1405 is positioned in front of PC 1404, and mug 1403 is positioned in front of virtual object 1405. Then, as shown in Figure 14(c), virtual object 1405 is not drawn in the area of mug 1403, and virtual object 1405 is drawn while avoiding the area of mug 1403. This prevents display inconsistencies (discrepancies) where the virtual object 1405 obscures the mug 1403 which is in front of it. Furthermore, it prevents the user's view from being obstructed by the virtual object 1405 while the user is about to drink from the mug 1403. If the user's view is obstructed by the virtual object 1405 while the user is about to drink from the mug 1403, for example, the user may be startled and spill the contents of the mug 1403.
[0081] After the processing in step S1207 or step S1217 is performed, the process proceeds to step S1208. Steps S1208 to S1211 are the same as steps S1007 to S1011 in Example 3 (Figure 10).
[0082] Using Figures 13(e) and 13(f), and Figures 14(e) and 14(f), an example of the process in steps S1208 to S1211 will be explained.
[0083] In Figure 13(e), the user moves the newspaper 1303 to a position where it does not overlap with the virtual object 1305. In this case, in step S1210, it is determined that there has been a change in the position of the object in the display direction of the virtual object 1305, and the process proceeds to step S1201, where the position of the virtual object 1305 is updated. The depth position of the virtual object 1305 is updated from the position shown in Figure 13(d) (slightly in front of the newspaper 1303) to the position shown in Figure 13(f) (slightly in front of the PC 1304). By reducing the parallax of the virtual object 1305, the depth position of the virtual object 1305 is updated so that the virtual object 1305 appears to be moving away. Reducing the parallax of the virtual object 1305 reduces the strain on the user's eyes. The parallax of the virtual object 1305 may be instantly changed to a target value (the parallax corresponding to being slightly in front of PC 1304) in response to a determination that there has been a change in the position of the object in the display direction of the virtual object 1305, or it may be changed gradually to the target value. In other words, when controlling the depth position of the virtual object 1305, the depth position may be instantly changed to the target position, or it may be changed gradually to the target position. If the parallax of the virtual object 1305 is changed gradually to the target value, the virtual object 1305 will gradually move backward to a position slightly in front of PC 1304.
[0084] Figure 14(e) shows the user moving the mug 1403 to a position where it does not overlap with the virtual object 1405. In this case, in step S1210, it is determined that there has been a change in the position of the object in the display direction of the virtual object 1405, and the process proceeds to step S1201, where the position of the virtual object 1405 is updated. In Figure 14(c), part of the virtual object 1405 was not displayed, but now the entire virtual object 1405 is displayed. This is a natural behavior, as the virtual object 1405 appears behind the mug 1403, and then when the mug 1403 is moved, the entire virtual object 1405 becomes visible. The depth position of the virtual object 1405 does not change from the position shown in Figure 14(d), as shown in Figure 14(f). The parallax of virtual object 1405 is sufficiently small, so it puts less strain on the user's eyes.
[0085] The virtual object display process shown in Figure 12 allows virtual objects to be displayed in a way that does not cause discomfort to the user.
[0086] Although an example was shown where the visible size of a virtual object does not depend on its depth position, virtual objects may appear smaller as they move away, similar to real objects. The method of controlling the visible size of a virtual object may differ depending on the type of virtual object. For example, in the case of the first type, the visible size of the virtual object may be changed so that it appears smaller as it moves away, while in the case of the second type, the change in the visible size of the virtual object may be limited. In the case of the second type, the visible size of the virtual object may not be changed, or it may be changed by a narrower range than in the case of the first type. In the case of the second type, when moving the virtual object a predetermined distance in the depth direction, the visible size of the virtual object may be changed by a smaller amount than in the case of the first type. In the case of the second type, the visible size of the virtual object may be changed so that it appears smaller as it moves away until the depth position of the virtual object is a predetermined position, and the visible size of the virtual object may not be changed once the depth position is beyond the predetermined position. When a virtual object is a text box containing text, the smaller the visible size of the virtual object, the smaller the text becomes, making it difficult to read. Therefore, by including virtual objects containing text in the second category, the problem of the text becoming too small and difficult to read can be suppressed.
[0087] Furthermore, if the visible size of the virtual object is smaller than the visible size of the specific object, the depth position of the virtual object may be controlled to be behind the specific object, regardless of whether the specific object is a movable object or not. If the specific object is a movable object, the depth position of the virtual object may be controlled to be behind the specific object, regardless of whether the visible size of the virtual object is smaller than the visible size of the specific object or not.
[0088] Depending on the type of virtual object, the method described in the above embodiment (a method of displaying virtual objects by controlling the depth position of the virtual object to suppress user fatigue and avoid causing discomfort to the user) may be used to switch whether or not to display the virtual object. For example, when displaying a virtual object that requires urgency, the virtual object may be displayed at close range (with a large parallax) at all times without using the method described in the above embodiment.
[0089] The above-described embodiments (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.
[0090] <Other examples> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]
[0091] 100: Head-mounted display (HMD) 110: CPU
Claims
1. A control device for controlling a display device to perform a three-dimensional display in which virtual objects are placed in three-dimensional space using a display for the right eye and a display for the left eye, Object detection means for detecting objects that exist in the user's field of view within the aforementioned three-dimensional space, Distance detection means for detecting the distance from the display device to the object detected by the object detection means, Control means for controlling the display device to perform the three-dimensional display described above. It has, The control means is The depth position of the virtual object is controlled by controlling the parallax of the virtual object between the right-eye display and the left-eye display based on the distance from the display device to a specific object that is the closest object to the display device and is located in the display direction of the virtual object. If the visible size of the virtual object is larger than the visible size of the specific object, the depth position of the virtual object is controlled to be behind the specific object. If the visible size of the virtual object is smaller than the visible size of the specific object, the depth position of the virtual object is controlled to be in front of the specific object, even if the virtual object hides at least a part of the specific object. A control device characterized by the following features.
2. The control means sets the user's line of sight when the display of the virtual object is started as the display direction of the virtual object. The control device according to feature 1.
3. The display device is an optical see-through type display. The control device according to claim 1 or 2.
4. The display device is a video see-through type display. The control device according to claim 1 or 2.
5. The control means includes the detection of a change in the arrangement of the virtual object in the display direction. When certain conditions are met, the depth position of the virtual object is controlled. The control device according to any one of claims 1 to 4.
6. The control means controls the depth position of the virtual object when predetermined conditions are met, including the user's line of sight being directed toward the display direction of the virtual object. The control device according to any one of claims 1 to 5.
7. The control means controls the depth position of the virtual object based on the distance from the display device to the specific object and the depth length of the virtual object. The control device according to any one of claims 1 to 6.
8. The control means controls the frontmost depth position of the virtual object to be in front of the specific object by the depth length of the virtual object. The control device according to feature 7.
9. The object detection means detects objects from the captured image. The control device according to any one of claims 1 to 8.
10. When the control means controls the depth position of the virtual object to be behind the specific object, it will not draw the region of the virtual object that overlaps with the specific object in the display direction of the virtual object. The control device according to any one of claims 1 to 9.
11. If the visible size of the virtual object is larger than the visible size of the specific object, If the specified object is a movable object, the control means controls the depth position of the virtual object to be located behind the specified object. If the specified object is an immovable object, the control means controls the depth position of the virtual object to be in front of the specified object. The control device according to any one of claims 1 to 10.
12. If the specified object is a movable object, the control means controls the depth position of the virtual object to be behind the specified object. The control device according to any one of claims 1 to 10.
13. The control means controls the depth position of the virtual object to gradually change to the target position. The control device according to any one of claims 1 to 12.
14. The control means controls the depth position of the virtual object to be in front of the specific object if the visible size of the virtual object is equal to the visible size of the specific object. The control device according to any one of claims 1 to 13.
15. A control method for controlling a display device to perform a three-dimensional display in which virtual objects are placed in three-dimensional space using a display for the right eye and a display for the left eye, An object detection step for detecting an object in the three-dimensional space that exists within the user's field of view, A distance detection step in which the distance from the display device to the object detected by the object detection step is detected, A control step of controlling the display device to perform the three-dimensional display described above. It has, In the control step described above, The depth position of the virtual object is controlled by controlling the parallax of the virtual object between the right-eye display and the left-eye display based on the distance from the display device to a specific object that is the closest object to the display device and is located in the display direction of the virtual object. death, If the visible size of the virtual object is larger than the visible size of the specific object, the depth position of the virtual object is controlled to be behind the specific object. If the visible size of the virtual object is smaller than the visible size of the specific object, the depth position of the virtual object is controlled to be in front of the specific object, even if the virtual object hides at least a part of the specific object. A control method characterized by the following:
16. A program for causing a computer to function as one of the means of the control device described in any one of claims 1 to 14.
17. A computer-readable storage medium storing a program for causing a computer to function as one of the means of the control device described in any one of claims 1 to 14.