Information processing device, information processing method, and recording medium

Abstract

The present disclosure relates to an information processing device, an information processing method, and a recording medium that make it possible to appropriately display a virtual object superimposed on a real object. Provided is an information processing device comprising a control unit that acquires information related to the states of a real object as a superimposition object, on which a virtual object is to be superimposed, and a second real object different from the real object as the superimposition object, and controls display of the virtual object on the basis of the acquired information. For example, the present disclosure can be applied to an XR HMD or the like.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium, and more particularly to an information processing device, an information processing method, and a recording medium that are capable of appropriately displaying a virtual object superimposed on a real object.

[0002] Smartphones and XR head-mounted displays (hereafter referred to as XR HMDs) can superimpose three-dimensional virtual objects onto the real world. For example, by displaying a virtual object in front of the user's eyes, users can check the size and appearance of a product before actually manufacturing or purchasing it. However, virtual objects are merely virtual objects displayed on smartphones or XR HMDs, and cannot be grasped or moved in the same way as real objects. If virtual objects could be handled like real objects, it would be possible to check the shape and color in more detail during the manufacturing process or before purchasing, and there is a growing demand for this.

[0003] Known techniques for superimposing a virtual object on a real object include those disclosed in, for example, Patent Documents 1 to 3. Patent Document 1 discloses a technique for blurring the boundary of a virtual object when superimposing the virtual object on the real object. Patent Document 2 discloses a technique for aligning a specified plane of a virtual object so that it is parallel to a plane along the hand or a specified plane of a mockup. Patent Document 3 discloses a technique for superimposing a virtual object of a different size or shape on a real object, and displaying the virtual object semi-transparently when the user's hand is not touching the real object.

[0004] JP 2009-104249 A JP 2016-99839 A JP 2018-92313 A

[0005] When superimposing a virtual object on a real object, there are problems such as the user being unable to grasp the part of the virtual object that they want to grasp due to differences in shape and size between the virtual object and the real object, and there has been a demand for an appropriate display of the virtual object to be superimposed on the real object.

[0006] The present disclosure has been made in light of these circumstances, and makes it possible to appropriately display a virtual object superimposed on a real object.

[0007] An information processing device according to one aspect of the present disclosure is an information processing device that acquires information regarding the status of a real object to be superimposed on a virtual object and a second real object different from the real object to be superimposed, and that controls the display of the virtual object based on the acquired information.

[0008] An information processing method according to one aspect of the present disclosure is an information processing method that includes an information processing device acquiring information regarding the status of a real object to be superimposed on a virtual object and a second real object different from the real object to be superimposed, and controlling the display of the virtual object based on the acquired information.

[0009] A recording medium according to one aspect of the present disclosure is a computer-readable recording medium having recorded thereon a program for causing a computer to function as a control unit that acquires information regarding the status of a real object onto which a virtual object is to be superimposed and a second real object different from the real object onto which the virtual object is to be superimposed, and controls the display of the virtual object based on the acquired information.

[0010] In an information processing device, an information processing method, and a recording medium according to one aspect of the present disclosure, information is acquired regarding the status of a real object to be superimposed on a virtual object, and a second real object different from the real object to be superimposed, and the display of the virtual object is controlled based on the acquired information.

[0011] Note that the information processing device according to one aspect of the present disclosure may be an independent device or an internal block constituting a single device.

[0012] 1 is a diagram illustrating a first example of a conventional method. 2 is a diagram illustrating a second example of a conventional method. 3 is a diagram illustrating a third problem of a conventional method. 4 is a diagram illustrating a fourth problem of a conventional method. 5 is a block diagram illustrating a configuration example of an embodiment of an information processing device to which the present disclosure is applied. 6 is a diagram illustrating a configuration example of an XR HMD to which the present disclosure is applied. 7 is a diagram illustrating a configuration example of a smartphone to which the present disclosure is applied. 8 is a diagram illustrating a configuration example of a stationary 3D display to which the present disclosure is applied. 9 is a diagram illustrating a configuration example of a shooting camera to which the present disclosure is applied. 10 is a block diagram illustrating another configuration example of an embodiment of an information processing device to which the present disclosure is applied. 11 is a flowchart illustrating a process flow for displaying a virtual object relative to a real object to be superimposed. 12 is a diagram illustrating an example of a wireframe display of a virtual object. 13 is a diagram illustrating an example of setting a grasping origin. 14 is a flowchart illustrating a process flow for setting a tracking target. 15 is a diagram illustrating an example of visualization of a region to be grasped of a virtual object. 16 is a diagram illustrating an example of tracking feature points of a real object to be superimposed and a second real object. 17 is a flowchart illustrating a process flow for enlarging, reducing, and rotating a virtual object superimposed on a real object to be superimposed. 18 is a diagram illustrating an example of enlarging a virtual object. 19 is a diagram illustrating an example of rotating a virtual object. FIG. 1 is a flowchart showing a processing flow when a physical object to be superimposed is placed on a plane. FIG. 2 is a diagram showing an example of rotating a virtual object so that it is perpendicular to the plane. FIG. 3 is a flowchart showing a processing flow when a physical object to be superimposed is changed in position. FIG. 4 is a diagram showing an example of resetting the grip origin when a physical object to be superimposed is changed in position. FIG. 5 is a block diagram showing an example of the hardware configuration of a computer.

[0013] <Conventional Methods> Examples of technologies that treat virtual objects displayed on smartphones or XR HMDs in the same way as real objects include the technologies shown in Figures 1 and 2. XR (Cross Reality) is an abbreviation for Extended Reality / Cross Reality, and is a technology that combines real space and virtual space to provide new experiences that cannot be perceived in reality. For example, XR includes VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

[0014] 1 shows a state in which a virtual object 11 is displayed to a user wearing an XR HMD and the user makes a hand gesture 13 with his or her hand 12. As shown in Fig. 1, the hand gesture 13, which is made by moving the hand 12 in the direction of arrow A1, is captured and recognized by a camera mounted on the XR HMD, thereby making it possible to display the virtual object 11 as if it were being virtually grasped and moved in the direction of arrow A2.

[0015] 2 shows a state in which a virtual object 22 is displayed to a user wearing an XR HMD using a physical object such as a marker 21. In this example, the marker 21 is captured and recognized by a camera mounted on the XR HMD, and the virtual object 22 is superimposed according to the position and orientation of the marker 21. In this case, when the user holds the marker 21 in his / her hand 23 and moves it in the direction of arrow A1, the virtual object 22 can be moved indirectly in accordance with the movement of the marker 21.

[0016] A technology has also been proposed that recognizes real objects in three dimensions from images captured by the camera of an XR HMD and estimates their position and size, and by applying this technology to overlay a virtual object on a camera mockup (a life-size model of a camera that looks exactly like the real thing) that has the same shape as the virtual object, the user can grasp the virtual object. However, creating a real object of the same shape every time a virtual object is displayed is time-consuming and costly, so it is preferable to use a real object of similar shape and size so that the user can grasp it without feeling uncomfortable.

[0017] The method of superimposing a virtual object on a real object having a different shape or size from the virtual object and indirectly grasping the virtual object by grasping the real object has the following problems.

[0018] First, when a virtual object is superimposed on a real object of a different shape or size, the user may be unable to grasp the desired location with their hand. For example, as shown in Figure 3, consider a case in which a virtual object 32 is superimposed on a real object 31 having a rectangular parallelepiped shape protruding from a plane. In this case, as indicated by the tip of arrow A1, when the user tries to grasp the virtual object 32 with their hand 33, the real object 31 is not present in the desired location, and the user is unable to grasp the desired location of the virtual object 32.

[0019] Second, when a virtual object superimposed on a real object of a different shape or size can be enlarged, the virtual object may not be enlarged appropriately. For example, as shown in FIG. 4 , consider a case in which a user performs a hand gesture 45 with a hand 44 while a virtual object 43 is superimposed on a real object 41 on a plane 42. In this case, as shown by the tip of arrow A1, the virtual object 43 can be enlarged in the direction of arrow A2 in response to the hand gesture 45, such as separating the index finger and thumb. However, because the virtual object 43 is enlarged around a predefined origin, it may not necessarily be enlarged to fit the part of the real object 41 that the user wants to grasp. Furthermore, as shown in FIG. 4 , it is possible that the virtual object 43 will sink into the plane 42 as a result of the enlargement of the virtual object 43.

[0020] Third, when the shape of the real object to be superimposed does not match the shape of the virtual object, and the virtual object can be placed in a manner that is not perpendicular to the plane, the user needs to be conscious of making the virtual object perpendicular to the plane. For example, as shown in Figure 5, when superimposing a virtual object 53 on a real object 51 on a plane 52, the user needs to be conscious of making the virtual object 53 perpendicular to the plane 52. However, as shown by the tip of arrow A1, there are cases where the virtual object 53 is not perpendicular to the plane 52. In this example, the virtual object 53 is not perpendicular to the plane 52, but is perpendicular to the real object 54 on the plane 52, and is not displayed properly.

[0021] Fourth, when the real object to be superimposed is smaller than the virtual object, the real object to be superimposed is obscured by the hand, particularly when the user holds it in their hand, making it impossible to capture and recognize it with the camera mounted on the XR HMD, resulting in the problem of the virtual object not being able to be displayed. For example, as shown in Figure 6, consider a case in which the user holds a real object 61 with a virtual object 62 superimposed thereon in their hand 63. In this case, if the real object 61 is smaller than the virtual object 62 and is obscured by the user's hand 63 and becomes unrecognizable, as indicated by the tip of arrow A1, it becomes impossible to superimpose and display the virtual object 62 on the real object 61.

[0022] In this disclosure, to address problems such as a user being unable to grasp the part of a virtual object that they want to grasp due to differences in shape or size between a virtual object and a real object that is to be superimposed on it (hereinafter also referred to as the real object to be superimposed), a method is proposed for appropriately displaying a virtual object by changing the display of the virtual object depending on the status of a real object that is different from the real object to be superimposed on it (hereinafter also referred to as the second real object).

[0023] <System Configuration> FIG. 7 is a block diagram showing an example configuration of an embodiment of an information processing device to which the present disclosure is applied.

[0024] In FIG. 7 , the information processing device 70 is composed of a sensor unit 71, an object detection unit 72, a trigger detection unit 73, a superposition target determination unit 74, a grasping area determination unit 75, a data management unit 76, a data storage unit 77, a virtual object position determination unit 78, a drawing unit 79, and a display unit 80.

[0025] The sensor unit 71 is composed of various sensors such as an image sensor, a gyro sensor, an acceleration sensor, etc. The sensor unit 71 performs sensing using the various sensors and outputs the resulting sensor data (image data, etc.) to the object detection unit 72, the virtual object position determination unit 78, and the display unit 80.

[0026] For example, the image sensor can capture the real world in front of the device as a video see-through image. An image sensor for capturing images to be used as input for environmental recognition processing such as object recognition and position and orientation estimation may be provided separately from the image sensor for capturing the video see-through image. An inertial sensor (gyro sensor, acceleration sensor) such as an IMU (Inertial Measurement Unit) may also be provided. When the information processing device 70 is configured as an XR HMD, an image sensor for capturing images of the user's eyeballs may be provided to detect the line of sight of the user wearing the XR HMD.

[0027] The object detection unit 72 estimates the position and orientation of any real object, such as a real object on which a virtual object is to be superimposed or a user's hand holding the real object, based on the image data output from the sensor unit 71. For example, a known 3D object recognition technique can be used to estimate the position and orientation of the real object. The object detection unit 72 outputs the detection result (estimation result) to the trigger detection unit 73, the superimposition target determination unit 74, the holding area determination unit 75, and the virtual object position determination unit 78.

[0028] The trigger detection unit 73 detects a decision action of the user based on the detection result output from the object detection unit 72. For example, the trigger detection unit 73 determines whether the user's hand detected by the object detection unit 72 has a specific shape, and if it determines that the hand has a specific shape, it can consider that the user's decision action has been detected. The trigger detection unit 73 outputs the detection result (decision trigger) to the superimposition target determination unit 74, the gripping area determination unit 75, and the virtual object position determination unit 78.

[0029] The superimposition target determination unit 74 determines a physical object to be superimposed based on the detection result output from the object detection unit 72 and the detection result (decision trigger) output from the trigger detection unit 73. Here, since multiple objects can be detected, for example, the object that is most central on the screen or, if a gaze detection function is installed, the object that the user is gazing at can be set in a selected state, and the object that is in a selected state when the decision trigger is detected can be set as the physical object to be superimposed. The superimposition target determination unit 74 outputs the determination result (physical object to be superimposed) to the gripping area determination unit 75.

[0030] The gripping area determination unit 75 determines whether the distance between the real object to be superimposed and the virtual object drawn by the drawing unit 79 is less than a certain distance, based on the determination result output from the superimposition object determination unit 74. If the gripping area determination unit 75 determines that the distance between the real object to be superimposed and the virtual object is less than the certain distance, the gripping area determination unit 75 outputs a command to the drawing unit 79 to draw the virtual object semi-transparently or in a wireframe. Furthermore, when the gripping area determination unit 75 receives the detection result (grasp area determination trigger) output from the trigger detection unit 73, the gripping area determination unit 75 determines a gripping area and a grip origin for the real object to be superimposed, based on the positional relationship between the real object to be superimposed and the virtual object, and the position of a second real object (e.g., a user's hand) relative to the real object to be superimposed, and outputs (notifies) the determination result to the data management unit 76.

[0031] The data management unit 76 stores the determination results notified by the grasping area determination unit 75 (e.g., the type of real object to be superimposed, the type of linked virtual object, the positional relationship, and the grasping area and grasping origin of the real object to be superimposed) as data. The area in which the data management unit 76 stores data is typically volatile memory, and the data is written to the data storage unit 77 when the device is turned off, the application is terminated, or the device is in standby mode. The data management unit 76 also reads data from the data storage unit 77 when the device is turned on, the application is started, or the like. The data storage unit 77 is configured with nonvolatile memory such as a NAND flash memory, and stores data from the data management unit 76.

[0032] The virtual object position determination unit 78 outputs to the drawing unit 79 a command to draw a virtual object according to the position and setting values ​​specified by the user, based on the sensor data output from the sensor unit 71, the detection results output from the object detection unit 72, the detection results output from the trigger detection unit 73, or data from the data management unit 76.

[0033] For example, the user's designation is determined by receiving a determination trigger output from the trigger detection unit 73 while the user is designating an arbitrary position on the plane detected by the object detection unit 72 with their finger or by directing their gaze. AR applications for XR HMDs and smartphones implement self-position and orientation estimation technologies such as SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry), which can acquire the three-dimensional coordinates of the shooting camera in real space. Using this technology, a virtual object may be displayed at an arbitrary position relative to the camera. If the data management unit 76 holds setting values ​​(setting values ​​related to superimposition) for the real object to be superimposed, the virtual object position determination unit 78 commands the drawing unit 79 to draw the virtual object according to the setting values ​​received from the data management unit 76.

[0034] The rendering unit 79 renders a virtual object using 3DCG (3-dimensional computer graphics) technology based on commands (commands for rendering position, orientation, etc.) output from the virtual object position determination unit 78. The display unit 80 combines an image (a captured image of the real world) based on image data output from the sensor unit 71 with an image (virtual object) based on the rendering result output from the rendering unit 79, and displays the combined image on a display device. The display device may be configured as an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, or the like. When the information processing device 70 is configured as an XR HMD, the display unit 80 also includes an optical engine.

[0035] In the information processing device 70 configured as described above, the object detection unit 72, the trigger detection unit 73, the superimposition target determination unit 74, the gripping region determination unit 75, the virtual object position determination unit 78, and the drawing unit 79 are provided as functions of the control unit 70A. The functions of the control unit 70A are realized by a processor such as a CPU (Central Processing Unit) executing a program. The number of processors is not limited to one, and two or more processors may be installed. For example, the drawing unit 79 may be configured as a GPU (Graphics Processing Unit), and the object detection unit 72, the trigger detection unit 73, the superimposition target determination unit 74, the gripping region determination unit 75, and the virtual object position determination unit 78 may be configured as a CPU.

[0036] The information processing device 70 can be configured as a device such as an XR HMD, a smartphone, a 3D display, or a camera for photography.

[0037] FIG. 8 is a diagram illustrating an example configuration of an XR HMD to which the present disclosure is applied. In FIG. 8, an XR HMD 92 is worn on the head of a user 91, and a sensor unit 93 is provided on the front of the portion covering the head, and a display unit 94 is provided on the back. The XR HMD 92 has a configuration similar to that of the information processing device 70 in FIG. 7, where the sensor unit 93 corresponds to the sensor unit 71 in FIG. 7, and the display unit 94 corresponds to the display unit 80 in FIG. 7. Functions such as the object detection unit 72 in FIG. 7 are realized by a processor such as a CPU installed in the XR HMD 92 executing a program. In FIG. 8, the user 91 is holding a real object 95 to be superimposed in his or her hand (second real object) within the shooting range of the sensor unit 93, and can see a virtual object superimposed on the real object 95 to be superimposed, which is displayed on the display unit 94.

[0038] FIG. 9 is a diagram illustrating an example configuration of a smartphone to which the present disclosure is applied. In FIG. 9 , a smartphone 102 is held by a user 101, and a sensor unit 103 is provided on the back of the housing and a display unit 104 is provided on the front. The smartphone 102 has a configuration similar to that of the information processing device 70 in FIG. 7 , where the sensor unit 103 corresponds to the sensor unit 71 in FIG. 7 and the display unit 104 corresponds to the display unit 80 in FIG. 7 . Functions such as the object detection unit 72 in FIG. 7 are realized by a processor such as a CPU installed in the smartphone 102 executing a program. In FIG. 9 , the user 101 holds the smartphone 102 in one hand and a real object 105 to be superimposed within the imaging range of the sensor unit 103 with the other hand (second real object), and can view a virtual object superimposed on the real object 105 to be superimposed, which is displayed on the display unit 104.

[0039] FIG. 10 is a diagram illustrating an example configuration of a stationary 3D display to which the present disclosure is applied. In FIG. 10 , a 3D display 112 is installed at an arbitrary location, and a sensor unit 113 and a display unit 114 are provided for a user 111 at a predetermined position. The 3D display 112 has a configuration similar to that of the information processing device 70 of FIG. 7 , where the sensor unit 113 corresponds to the sensor unit 71 of FIG. 7 and the display unit 114 corresponds to the display unit 80 of FIG. 7 . Functions of the object detection unit 72 of FIG. 7 and the like are realized by a processor such as a CPU mounted on the 3D display 112 executing a program. In FIG. 10 , the user 111 holds a real object 115 to be superimposed in his or her hand (a second real object) within the imaging range of the sensor unit 113, and can view a virtual object superimposed on the real object 115 to be superimposed, which is displayed in 3D on the display unit 114.

[0040] FIG. 11 is a diagram illustrating an example configuration of a photographing camera to which the present disclosure is applied. In FIG. 11, the photographing camera 122 has a sensor unit 123 and a display unit 124, is installed in an arbitrary location, and is operated by a first user 121. The photographing camera 122 has a configuration similar to that of the information processing device 70 in FIG. 7, where the sensor unit 123 corresponds to the sensor unit 71 in FIG. 7, and the display unit 124 corresponds to the display unit 80 in FIG. 7. Functions such as the object detection unit 72 in FIG. 7 are realized by a processor such as a CPU mounted on the photographing camera 122 executing a program. In FIG. 11, within the photographing range of the sensor unit 123, a second user 125 holds a real object 126 to be superimposed in his hand (second real object), and the first user 121 can see a virtual object superimposed on the real object 126 to be superimposed, which is displayed on the display unit 124.

[0041] Trigger detection is not limited to gesture determination by the trigger detection unit 73 as shown in the configuration of the information processing device 70 in Fig. 7 , but may also be, for example, a controller-type device, an input device such as a keyboard connected to a PC (Personal Computer) connected to the XR HMD, a packet receiving program that receives voice input by a user, or an instruction (command) from the outside via a network, etc. Fig. 12 shows a configuration in which trigger detection is performed using such devices or programs.

[0042] 12 is a block diagram showing another example configuration of an information processing device according to an embodiment of the present disclosure. In FIG. 12, an information processing device 130 includes a sensor unit 131, an object detection unit 132, a trigger detection unit 133, a superimposition target determination unit 134, a gripping region determination unit 135, a data management unit 136, a data storage unit 137, a virtual object position determination unit 138, a drawing unit 139, and a display unit 140.

[0043] The trigger detection unit 133 can use a button input from a controller or an input device as a trigger. Alternatively, the trigger detection unit 133 may use a command received as a packet over a network such as the Internet as a trigger. The trigger detection unit 133 outputs the detection result (determination trigger) to the superimposition target determination unit 134, the grip area determination unit 135, and the virtual object position determination unit 138.

[0044] 12, the configuration other than the trigger detection unit 133 is the same as the configuration shown in Fig. 7, and therefore description thereof will be omitted. Note that in the information processing device 130 in Fig. 12, the object detection unit 132, the trigger detection unit 133, the superimposition target determination unit 134, the gripping area determination unit 135, the virtual object position determination unit 138, and the drawing unit 139 are also provided as functions of the control unit 130A. The functions of the control unit 130A are realized by one or more processors executing programs.

[0045] <Processing Flow> Fig. 13 is a flowchart showing the processing flow for displaying a virtual object on a real object to be superimposed. In Fig. 13, the processing is described as being executed by the control unit 70A of the information processing device 70 in Fig. 7, but the processing may also be executed by the control unit 130A of the information processing device 130 in Fig. 12. The same applies to the processing of the other flowcharts shown in the subsequent figures.

[0046] In step S11, the data management unit 76 reads setting values ​​(data) from the data storage unit 77. In step S12, the virtual object position determination unit 78 commands the drawing unit 79 to draw a virtual object at a location specified by the user, and the drawing result by the drawing unit 79 is displayed on the display unit 80. In step S13, the superimposition target determination unit 74 selects (determines) a real object to be superimposed based on the detection result by the object detection unit 72 and the detection result by the trigger detection unit 73 (user's determination trigger).

[0047] In step S14, the virtual object position determination unit 78 determines whether or not a setting value exists for the real object to be superimposed. If it is determined in step S14 that a setting value exists, the process proceeds to step S15. In step S15, the virtual object position determination unit 78 instructs the rendering unit 79 to superimpose the virtual object on the real object to be superimposed based on the setting value, and the display unit 80 displays the rendering result by the rendering unit 79.

[0048] On the other hand, if it is determined in step S14 that the set value does not exist, the process proceeds to step S16. In step S16, the holding area determination unit 75 checks whether a user decision trigger by the trigger detection unit 73 has been detected. If a decision trigger has not been detected in step S16, the process proceeds to step S17. In step S17, the holding area determination unit 75 measures the distance between the current real object and virtual object to be superimposed, and determines whether the distance between the virtual object and real object is less than a threshold. For example, the distance measurement can be calculated as the distance between the origins of the real object and virtual object to be superimposed.

[0049] If it is determined in step S17 that the distance is less than the threshold, the process proceeds to step S18. In step S18, the gripping area determination unit 75 instructs the drawing unit 79 to display the virtual object in a wireframe, and the display unit 80 displays the drawing result by the drawing unit 79. FIG. 14 is a diagram showing an example in which a virtual object is displayed in a wireframe according to the distance between the virtual object and a real object. FIG. 14 shows a virtual object 151, a real object 152 selected as an object to be superimposed, and a user's hand 153 holding the real object 152 to be superimposed. As indicated by the tip of arrow A1, when the real object 152 to be superimposed is located close to the virtual object 151, the virtual object 151 is displayed in a wireframe. In the wireframe display, a shape represented by vertices and lines is displayed without using surfaces.

[0050] On the other hand, if it is determined in step S17 that the distance is equal to or greater than the threshold, the process proceeds to step S19. In step S19, the gripping area determination unit 75 instructs the rendering unit 79 not to display the virtual object in a wireframe, and the display unit 80 displays the rendering result by the rendering unit 79. For example, as shown at the base (starting point) of arrow A1 in FIG. 14 , when the superimposed real object 152 is located far from the virtual object 151, the wireframe display of the virtual object 151 is terminated. Note that in this example, the virtual object is displayed in a wireframe when the distance between the virtual object and the superimposed real object is less than the threshold before the positional relationship between the virtual object and the superimposed real object is linked. However, the display is not limited to a wireframe display, and other display modes may also be used. For example, the virtual object may be displayed semi-transparently with a predetermined transparency (transparency). When step S18 or S19 is completed, the process returns to step S16, and the subsequent processes are repeated.

[0051] If a determination trigger is detected in step S16, the process proceeds to step S20. In step S20, the grasping region determination unit 75 determines whether or not the user is grasping the physical object to be superimposed. If it is determined in step S20 that the physical object is not being grasped, the process returns to step S16, and the subsequent processes are repeated. If it is determined in step S20 that the physical object is being grasped, the process proceeds to step S21.

[0052] In step S21, the grasping area determination unit 75 determines a grasping origin based on the current positional relationship between the real object and virtual object to be superimposed and the user's grasping position relative to the real object to be superimposed, and notifies the data management unit 76. The data management unit 76 then saves the local origin and grasping origin of the virtual object as setting values. In step S15, the virtual object position determination unit 78 instructs the rendering unit 79 to superimpose the virtual object on the real object to be superimposed based on the saved setting values, and the display unit 80 displays the drawing result by the rendering unit 79.

[0053] 15 is a diagram showing an example of setting a grip origin. A (upper part) of FIG. 15 shows a virtual object 161, a physical object 162 to be superimposed that has a rectangular parallelepiped shape, and a user's hand 163 holding the physical object 162 to be superimposed. The user's hand 163 is a second physical object different from the physical object to be superimposed. In this case, the grip origin 164 is determined based on the positional relationship between the physical object 162 to be superimposed and the user's hand 163 holding the physical object 162 to be superimposed, and is usually set near the center of the palm of the user near the wrist.

[0054] As shown in B (bottom) of FIG. 15 , for example, when the real object 165 to be superimposed is a controller, the virtual object 161 is superimposed on a real object of a different shape. In such a case, as shown on the left side of B of FIG. 15 , the virtual object 161 may be displayed at a location (position) a certain distance away from the real object 165 to be superimposed. In this case, based on the positional relationship between the real object 165 to be superimposed and the user's hand 163 holding the real object 165 to be superimposed, the grip origin 166 is usually set near the center of the palm of the user's hand near the wrist, but it can also be set as follows. That is, when the virtual object 161 and the user's hand 163 are separated by a certain distance and the real object 165 to be superimposed and the virtual object 161 do not intersect, the midpoint thereof may be set as the grip origin 167 (left side of B of FIG. 15 ).

[0055] Alternatively, as shown on the right side of B in Figure 15, if the real object 165 to be superimposed and the virtual object 161 intersect, but the user's hand 163 holding the real object 165 to be superimposed is a certain distance away from the virtual object 161, the grip origin 168 can be set at the center of the intersecting area.

[0056] 13, when the grasping area determination unit 75 detects a reset trigger by the trigger detection unit 73, the positional relationship between the real object to be superimposed and the virtual object and the setting of the grasping origin are erased, and the process returns to step S16. When the process of step S15 ends, the series of processes ends. In this way, in the information processing device 70 of FIG. 7, the control unit 70A determines the positional relationship between the virtual object and the real object to be superimposed, the grasping area, and the grasping origin, and saves them as setting values, and the virtual object is displayed based on the saved setting values.

[0057] <Tracking Target Setting Process> Fig. 16 is a flowchart showing the process of setting a tracking target. This process is performed when a user holds a virtual object with their hand, whose positional relationship between the virtual object and the real object and whose grip origin have already been set by the process shown in the flowchart of Fig. 13.

[0058] In step S41, the virtual object position determination unit 78 determines whether the distance between the user's hand detected by the object detection unit 72 and the real object to be superimposed is less than a threshold (less than a certain distance). If it is determined in step S41 that the distance is less than the threshold, the process proceeds to step S42. In step S42, the virtual object position determination unit 78 commands the drawing unit 79 to draw the gripping region restored from the gripping origin superimposed on the virtual object, making the gripping region visible.

[0059] FIG. 17 is a diagram showing an example of visualization of a region of a virtual object to be grasped. In FIG. 17 , a virtual object 171 is displayed superimposed on a real object 172 to be superimposed, but the real object 172 is not visible to the user. In this case, as indicated by the tip of arrow A1, when a user's hand 173 approaches the virtual object 171, the display of the virtual object is controlled so that the portion of the real object 172 to be superimposed that should be grasped becomes visible. For example, a guide 174 indicating which part of the real object 172 to be superimposed should be grasped may be displayed, or the display of the virtual object 171 may be changed so that the real object 172 to be superimposed becomes visible. For example, when changing the display of the virtual object 171, the grasping region may be displayed transparently or semi-transparently, allowing the grasping region to be transparent with a predetermined transparency (transparency). In other words, when the positional relationship between the virtual object 171 and the real object 172 to be superimposed is linked, if the distance between the real object 172 to be superimposed and a second real object (user's hand 173) attempting to grasp the real object 172 to be superimposed is less than a threshold value, the display of the virtual object 171 or a second virtual object different from the virtual object 171 (e.g., guide 174) is controlled so that the grasped portion of the real object 172 to be superimposed becomes visible.

[0060] In step S43, the object detection unit 72 detects whether the user's hand is gripping the superimposed physical object. If it is detected in step S43 that the user's hand is gripping the superimposed physical object, the process proceeds to step S44. In step S44, the object detection unit 72 treats the superimposed physical object and the hand gripping the superimposed physical object as a single object and tracks them. In other words, not only the superimposed physical object but also the gripping hand as a second physical object are tracked.

[0061] Fig. 18 is a diagram showing an example of tracking feature points of a physical object to be superimposed and a second physical object. In Fig. 18, feature points 182 of a physical object 181 to be superimposed are represented by black squares. Object tracking is a technique for tracking a group of feature points of an object, and the more feature points that are tracked, the less likely the object will be lost (the lower the possibility of it becoming untrackable). As indicated by the tip of arrow A1, when a user holds a physical object 181 to be superimposed with a hand 183, the user's hand 183 is treated as a second physical object, and its feature points 184 are also tracked. In Fig. 18, feature points 184 of the user's hand 183 are represented by white squares.

[0062] As indicated by the tip of arrow A2, tracking is performed by regarding superimposed real object 181 and user's hand 183 as a single object, and tracking a group of feature points including feature point 182 and feature point 184 as tracking targets. In this way, when the positional relationship between the virtual object and the superimposed real object is linked, when the superimposed real object is held by a second real object (user's hand), the feature points of the superimposed real object and the second real object are designated as tracking targets, and the second real object is also included in the tracking targets. In this way, even if at least a portion of the superimposed real object is occluded by the second real object and its feature points become invisible, by tracking the feature points of the second real object, it is possible to reduce the probability of the superimposed real object being lost based on the tracking results of at least the feature points of the second real object.

[0063] The control unit 70A then controls the display of the virtual object to be superimposed on the superimposed real object based on the tracking result. For example, the display of the virtual object can be controlled based on the position and orientation of the second real object and the positional relationship between the second real object and the superimposed real object when grasped. Note that the object compared with the second real object (user's hand) in step S41 is not limited to the superimposed real object, but may also be, for example, a virtual object or a grasping origin, and it may be determined whether the distance from the virtual object or the grasping origin is less than a threshold. In short, it is sufficient to determine an appropriate timing for making visible the grasping region of the superimposed real object grasped by the second real object (user's hand).

[0064] <Flow of Processing for Enlarging, Shrinking, and Rotating a Virtual Object> FIG. 19 is a flowchart showing the flow of processing for enlarging, shrinking, and rotating a virtual object superimposed on a target real object.

[0065] In step S61, the virtual object position determination unit 78 checks whether or not an enlargement trigger has been detected by the trigger detection unit 73. If an enlargement trigger has been detected in step S61, the process proceeds to step S62. In step S62, the virtual object position determination unit 78 commands the drawing unit 79 to enlarge and draw the virtual object in the positive directions of x, y, and z in the coordinate system of the grip origin stored as the setting values.

[0066] 20 is a diagram showing an example of enlarging a virtual object. Shown in FIG. 20 are a virtual object 191, a physical object 193 to be superimposed, and a user's hand 194 holding the physical object 193 to be superimposed. When an enlargement trigger is detected, the virtual object 191 is enlarged from a grip origin 195 determined according to the state of the physical object 193 to be superimposed and the user's hand 194, and a virtual object 192 is displayed. In other words, by enlarging the virtual object 191 before enlargement, the enlarged virtual object 192 is displayed. Note that this example shows a case where the virtual object is displayed as a wireframe.

[0067] If an enlargement trigger has not been detected in step S61, the process proceeds to step S63. In step S63, the virtual object position determination unit 78 checks whether a reduction trigger has been detected by the trigger detection unit 73. If a reduction trigger has been detected in step S63, the process proceeds to step S64. In step S64, the virtual object position determination unit 78 commands the drawing unit 79 to reduce and draw the virtual object in the negative x, y, and z directions of the coordinate system of the grip origin saved as the setting values. For example, in FIG. 20 , by reducing the virtual object 192 before reduction, a reduced virtual object 191 is displayed.

[0068] If a reduction trigger has not been detected in step S63, the process proceeds to step S65. In step S65, the virtual object position determination unit 78 checks whether a rotation trigger has been detected by the trigger detection unit 73. If a rotation trigger has been detected in step S65, the process proceeds to step S66. In step S66, the virtual object position determination unit 78 commands the drawing unit 79 to rotate and draw the virtual object around the grip origin saved as a setting value.

[0069] 21 is a diagram showing an example of virtual object rotation. This diagram shows a virtual object 201, a physical object 202 to be superimposed, and a user's hand 203 holding the physical object 202 to be superimposed. As indicated by the tip of arrow A1, when a rotation trigger is detected, the virtual object 201 is displayed rotated around a grip origin 204 determined based on the state of the physical object 202 to be superimposed and the user's hand 203. In other words, the virtual object 201 before rotation, indicated by the dashed line, is rotated in the direction of arrow A2 (counterclockwise) to display the rotated virtual object 201, indicated by the solid line.

[0070] When steps S62, S64, and S66 are completed, the process returns to step S61, and the above-described process is repeated. If no trigger is detected in steps S61, S63, or S65, the process ends.

[0071] <Processing Flow When Placing a Superimposed Object on a Plane> FIG. 22 is a flowchart showing the processing flow when arranging a physical object to be superimposed on a plane.

[0072] In step S81, the object detection unit 72 detects that a real object to be superimposed has been placed on a plane. In step S82, the virtual object position determination unit 78 commands the drawing unit 79 to rotate the virtual object around the grip origin saved as a setting value so that the virtual object is perpendicular to the plane and draw it.

[0073] 23 is a diagram showing an example in which a virtual object is rotated so as to be perpendicular to a plane. This diagram shows a virtual object 211, a physical object 212 to be superimposed, and a user's hand 213 holding the physical object 212 to be superimposed. As shown in FIG. 23 , when the physical object 212 to be superimposed is placed on a plane 214, the virtual object 211 is displayed after being rotated so as to be perpendicular to the plane 214, starting from a gripping origin 215 determined according to the state of the physical object 212 to be superimposed and the user's hand 213. That is, by correcting the position and orientation of the virtual object 211 before rotation, indicated by the dashed line, so as to rotate in the direction of arrow A1 (clockwise), the virtual object 211 after rotation, indicated by the solid line, is displayed on the plane 214.

[0074] 24 is a flowchart showing the processing flow when a real object to be superimposed is changed. This processing is performed when a user who is holding a real object to be superimposed in one hand changes the real object to be superimposed in another hand, for example, when the real object to be superimposed and the virtual object are similar but not identical in size or shape.

[0075] In step S101, the object detection unit 72 detects that the user has changed the gripping origin of the changed hand from the current gripping origin to the current gripping origin, and stores the changed grip origin as a setting value in the data management unit 76.

[0076] In step S103, the virtual object position determination unit 78 updates the position and orientation of the virtual object in accordance with the stored setting values ​​and commands the rendering unit 79 to superimpose the virtual object on the target real object. In step S104, the object detection unit 72 sets the hand that has been replaced with the target real object as a single object to be tracked, and starts tracking.

[0077] 25 is a diagram showing an example of resetting the grip origin when the superimposed physical object is changed. Shown in FIG. 25 are a superimposed physical object 221, a virtual object 222, and a user's hand holding the superimposed physical object 221. In this example, the superimposed physical object 221 is configured in the shape of a camera and is similar in size and shape to the superimposed virtual object 222. The superimposed physical object 221 is slightly smaller in size than the virtual object 222, and is held by the user's hand.

[0078] As shown by the tip of arrow A1, it is assumed that a user grasps a physical object 221 to be superimposed, which has been held with one hand 223, with the other hand 225, and then, as shown by the tip of arrow A2, releases the one hand 223 from the physical object 221 to be superimposed and switches the hand grasping the physical object 221 from the one hand 223 to the other hand 225. In this case, when the physical object 221 to be superimposed was held with the one hand 223, the virtual object 222 was displayed with a gripping origin 224 determined according to the state of the physical object 221 to be superimposed and the user's hand 223 as the origin. Furthermore, when the physical object 221 to be superimposed is switched to the other hand 225, the virtual object 222 is displayed with a gripping origin 226 determined according to the state of the physical object 221 to be superimposed and the user's hand 225 as the origin. In this way, when the virtual object is switched, the gripping origin can be reset (re-determined) to correct the position and orientation of the virtual object. Furthermore, by tracking the real object to be superimposed and the hand that has changed its grip and is holding it, tracking can be performed reliably as described above.

[0079] As described above, the present disclosure can superimpose a virtual object on a target real object, determine an origin (grasp origin) for displaying the virtual object depending on the status of the target real object (e.g., a controller) and a second real object (e.g., a user's hand), and control the display of the virtual object depending on the determined origin. That is, the present disclosure acquires information regarding the status of the target real object on which the virtual object is to be superimposed and a second real object different from the target real object, and controls the display of the virtual object based on the acquired information. This addresses a problem in which a user cannot grasp a desired portion of a virtual object due to differences in shape or size between the virtual object and the target real object, by changing the display of the virtual object depending on the status of the target real object and the second real object. By appropriately displaying the virtual object superimposed on the target real object, for example, a user can grasp a desired portion of the virtual object superimposed on the target real object.

[0080] The aforementioned Patent Document 1 proposes a technology for blurring the boundary between a virtual object when it is superimposed on a real object. While this reduces the retinal rivalry that occurs when a real object and a virtual object are simultaneously in the field of view, it does not resolve the discomfort that occurs when grasping a virtual object due to different shapes or sizes of the virtual object. Patent Document 2 also proposes a technology for aligning a specified plane of a virtual object so that it is parallel to a plane along the hand or a specified plane of a mockup, but this technology does not suggest which part of the real object on which the virtual object should be grasped, nor does it mention any criteria for enlarging, reducing, or rotating the virtual object. Furthermore, Patent Document 3 proposes a technology for superimposing a virtual object of a different size or shape on a real object, in which the virtual object is displayed semi-transparently when the user's hand is not touching the real object, but it does not mention any criteria for enlarging, reducing, or rotating the virtual object.

[0081] <Modifications> The information processing device 70 of FIG. 7 and the information processing device 130 of FIG. 12 are not limited to the above-described configurations and may employ other configurations. For example, if the information processing device 70 of FIG. 7 is configured as a server connected to a network such as the Internet, it may be configured without the sensor unit 71 and the display unit 80, i.e., with only a control unit 70A and a communication unit. Furthermore, a terminal device (e.g., an XR HMD, a smartphone, etc.) may include the sensor unit 71, the display unit 80, and a communication unit. In this case, the information processing device 70 (server) may receive sensor data transmitted from the terminal device via the network, the control unit 70A may process the received data, and the resulting processed data may be transmitted to the terminal device via the network. Meanwhile, the terminal device may transmit sensor data acquired by the sensor unit 71 to the information processing device 70 (server) via the network. Furthermore, the terminal device may receive processed data from the information processing device 70 (server) via the network, and a virtual object based on the received data may be displayed on the display unit 80. In the information processing device 70 (server), the control unit 70A only needs to have at least some of the blocks among the object detection unit 72, the trigger detection unit 73, the superimposition target determination unit 74, the gripping area determination unit 75, the data management unit 76, the data storage unit 77, the virtual object position determination unit 78, and the drawing unit 79, and the remaining blocks may be provided on the terminal device side. For example, the drawing unit 79 may be provided on the terminal device side instead of the server side.

[0082] In the above example, a camera is displayed as a virtual object to be superimposed on a real object to be superimposed, but other objects (e.g., a product that has not yet been manufactured or purchased) may be displayed as the virtual object. Also, in the above example, a user's hand is used as an example of a second real object that is different from the real object to be superimposed, but other objects such as an object around the user (e.g., a table) may also be displayed.

[0083] <Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, a program constituting the software is installed in a computer. Fig. 26 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0084] In the computer, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are interconnected by a bus 304. An input / output interface 305 is also connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

[0085] The input unit 306 includes a keyboard, a mouse, a microphone, etc. The output unit 307 includes a display, a speaker, etc. The storage unit 308 includes a hard disk, a non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a semiconductor memory, a magnetic disk, an optical disk, or a magneto-optical disk.

[0086] In a computer configured as described above, the CPU 301 loads a program recorded in the ROM 302 or the memory unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executes the program, thereby performing the above-described series of processes.

[0087] The program executed by the computer (CPU 301) can be provided by being recorded on a removable recording medium 311 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0088] In a computer, the program can be installed in the storage unit 308 via the input / output interface 305 by inserting the removable recording medium 311 into the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be installed in advance in the ROM 302 or the storage unit 308.

[0089] The processing performed by a computer according to a program includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing). Furthermore, the program may be processed by a single computer (processor) or may be distributed among multiple computers.

[0090] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. For example, the present disclosure may be configured as a cloud computing system in which a single function is shared and processed jointly by multiple devices via a network.

[0091] Each step described in the above flowchart can be executed by a single device or can be shared and executed by multiple devices. Furthermore, if a single step includes multiple processes, the multiple processes included in that single step can be executed by a single device or can be shared and executed by multiple devices. Note that the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0092] The present disclosure can also be configured as follows.

[0093] (1) An information processing device comprising a control unit that acquires information regarding the status of a real object to be superimposed on a virtual object and a second real object different from the real object to be superimposed, and controls display of the virtual object based on the acquired information. (2) The information processing device described in (1), wherein the control unit determines a starting point for displaying the virtual object depending on the status of the real object to be superimposed and the second real object, and controls display of the virtual object depending on the determined starting point. (3) The information processing device described in (2), wherein the control unit determines a starting point for enlarging or reducing the virtual object depending on the status of the real object to be superimposed and the second real object. (4) The information processing device described in (2) or (3), wherein the control unit determines a starting point for rotating the virtual object depending on the status of the real object to be superimposed and the second real object. (5) The information processing device according to (4), wherein, when placing the physical object to be superimposed on a plane, the control unit rotates the virtual object in accordance with the origin and displays it so as to be perpendicular to the plane. (6) The information processing device according to any of (2) to (5), wherein the physical object to be superimposed is an object held by a user, and the second physical object is the user's hand. (7) The information processing device according to (6), wherein, when the user holds the physical object to be superimposed with one hand, the control unit determines the origin in accordance with the physical object to be superimposed and the one hand holding the real object to be superimposed, and when the user switches the hand holding the real object to be superimposed from one hand to the other, the control unit redetermines the origin in accordance with the physical object to be superimposed and the other hand holding the real object to be superimposed. (8) The information processing device according to (1) or (6), wherein the control unit displays the virtual object semi-transparently or in a wireframe when a distance between the virtual object and the real object to be superimposed is less than a threshold before linking a positional relationship between the virtual object and the real object to be superimposed.(9) The information processing device according to (1) or (6), wherein the control unit controls display of the virtual object or a second virtual object different from the virtual object so that a grasped portion of the real object to be superimposed becomes visible when a distance between the real object to be superimposed and the second real object attempting to grasp the real object to be superimposed is less than a threshold when a positional relationship between the virtual object and the real object to be superimposed is linked. (10) The information processing device according to (1) or (6), wherein the control unit controls display of the virtual object or a second virtual object different from the virtual object so that a grasped portion of the real object to be superimposed becomes visible when a positional relationship between the virtual object and the real object to be superimposed is linked and the real object to be superimposed is grasped by the second real object when the positional relationship between the virtual object and the real object to be superimposed is linked. (11) The information processing device according to (9), wherein the control unit displays a guide indicating a gripped portion of the real object to be superimposed. (12) The information processing device according to (10), wherein the control unit controls display of the virtual object based on the position and orientation of the second real object and the positional relationship between the second real object and the real object to be superimposed when gripped. (13) An information processing method, including an information processing device acquiring information regarding situations of a real object to be superimposed on which a virtual object is to be superimposed and a second real object different from the real object to be superimposed, and controlling display of the virtual object based on the acquired information. (14) The information processing method according to (13), further including determining a starting point for displaying the virtual object according to situations of the real object to be superimposed and the second real object, and controlling display of the virtual object according to the determined starting point. (15) The information processing method according to (14), further comprising determining a starting point for enlarging or reducing the virtual object in accordance with the status of the real object to be superimposed and the second real object. (16) The information processing method according to (14) or (15), further comprising determining a starting point for rotating the virtual object in accordance with the status of the real object to be superimposed and the second real object.(17) The information processing method according to any one of (13) to (16), further comprising: displaying the virtual object semi-transparently or in a wireframe when a distance between the virtual object and the real object to be superimposed is less than a threshold value before the positional relationship between the virtual object and the real object to be superimposed is linked. (18) The information processing method according to any one of (13) to (16), further comprising: controlling display of the virtual object or a second virtual object different from the virtual object so that a portion of the real object to be superimposed that is to be gripped becomes visible when a distance between the real object to be superimposed and the second real object that is attempting to grip the real object to be superimposed is less than a threshold value when the positional relationship between the virtual object and the real object to be superimposed is linked. (19) The information processing method according to any one of (13) to (16), further comprising: when the positional relationship between the virtual object and the real object to be superimposed is linked and the real object to be superimposed is grasped by the second real object, tracking feature points of the real object to be superimposed and the second real object; and when at least a part of the real object to be superimposed is occluded by the second real object, controlling display of the virtual object based on at least a tracking result of the second real object. (20) A computer-readable recording medium having recorded thereon a program for causing a computer to function as a control unit that acquires information on situations of the real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed, and controls display of the virtual object based on the acquired information.

[0094] 70 Information processing device, 70A Control unit, 71 Sensor unit, 72 Object detection unit, 73 Trigger detection unit, 74 Superimposition target determination unit, 75 Grasping area determination unit, 76 Data management unit, 77 Data storage unit, 78 Virtual object position determination unit, 79 Drawing unit, 80 Display unit, 130 Information processing device, 130A Control unit, 131 Sensor unit, 132 Object detection unit, 133 Trigger detection unit, 134 Superimposition target determination unit, 135 Grasping area determination unit, 136 Data management unit, 137 Data storage unit, 138 Virtual object position determination unit, 139 Drawing unit, 140 Display unit

Claims

1. An information processing device comprising: a control unit that acquires information about the status of a real object onto which a virtual object is to be superimposed and a second real object different from the real object onto which the virtual object is to be superimposed; and controls the display of the virtual object based on the acquired information.

2. The information processing device according to claim 1, wherein the control unit determines a starting point for displaying the virtual object depending on the situations of the real object to be superimposed and the second real object, and controls the display of the virtual object depending on the determined starting point.

3. The information processing device according to claim 2, wherein the control unit determines a starting point when enlarging or reducing the virtual object depending on the status of the real object to be superimposed and the second real object.

4. The information processing device according to claim 2, wherein the control unit determines a starting point for rotating the virtual object depending on the status of the real object to be superimposed and the second real object.

5. The information processing device according to claim 4, wherein the control unit, when placing the real object to be superimposed on a plane, rotates the virtual object according to the origin and displays it so as to be perpendicular to the plane.

6. The information processing device according to claim 2, wherein the real object to be superimposed is an object held by a user, and the second real object is the user's hand.

7. The information processing device according to claim 6, wherein the control unit, when the user holds the real object to be superimposed in one hand, determines the starting point in accordance with the real object to be superimposed and the hand holding the real object to be superimposed, and when the user switches the hand holding the real object to be superimposed from one hand to the other, redetermines the starting point in accordance with the real object to be superimposed and the other hand holding the real object to be superimposed.

8. The information processing device according to claim 1, wherein the control unit displays the virtual object semi-transparently or in a wireframe when the distance between the virtual object and the real object to be superimposed is less than a threshold before the positional relationship between the virtual object and the real object to be superimposed is linked.

9. The information processing device according to claim 1, wherein, when the positional relationship between the virtual object and the real object to be superimposed is linked, if the distance between the real object to be superimposed and the second real object attempting to grasp the real object to be superimposed is less than a threshold, the control unit controls the display of the virtual object or a second virtual object different from the virtual object so that the portion of the real object to be superimposed that is to be grasped becomes visible.

10. The information processing device according to claim 1, wherein the control unit, when the positional relationship between the virtual object and the real object to be superimposed is linked, tracks feature points of the real object to be superimposed and the second real object when the real object to be superimposed is grasped by the second real object, and controls the display of the virtual object based on at least the tracking result of the second real object when at least a part of the real object to be superimposed is occluded by the second real object.

11. The information processing device according to claim 9, wherein the control unit displays a guide indicating a portion of the real object to be superimposed that is to be grasped.

12. The information processing device according to claim 10, wherein the control unit controls the display of the virtual object based on the position and orientation of the second physical object and the positional relationship between the second physical object and the physical object to be superimposed when grasped.

13. An information processing method including: an information processing device acquiring information regarding the status of a real object onto which a virtual object is to be superimposed and a second real object different from the real object onto which the virtual object is to be superimposed; and controlling the display of the virtual object based on the acquired information.

14. The information processing method according to claim 13, further comprising: determining a starting point for displaying the virtual object according to the situations of the real object to be superimposed and the second real object; and controlling the display of the virtual object according to the determined starting point.

15. The information processing method according to claim 14, further comprising determining a starting point for enlarging or reducing the virtual object according to the status of the real object to be superimposed and the second real object.

16. The information processing method according to claim 14, further comprising determining a starting point for rotating the virtual object according to the situations of the real object to be superimposed and the second real object.

17. The information processing method according to claim 13, further comprising displaying the virtual object semi-transparently or in a wireframe when the distance between the virtual object and the real object to be superimposed is less than a threshold value before the positional relationship between the virtual object and the real object to be superimposed is linked.

18. The information processing method according to claim 13, further comprising, when the positional relationship between the virtual object and the superimposed real object is linked, controlling the display of the virtual object or a second virtual object different from the virtual object so that the portion of the superimposed real object that is to be grasped becomes visible when the distance between the superimposed real object and the second real object that is attempting to grasp the superimposed real object is less than a threshold value.

19. The information processing method according to claim 13, further comprising: when the positional relationship between the virtual object and the real object to be superimposed is linked, when the real object to be superimposed is grasped by the second real object, tracking feature points of the real object to be superimposed and the second real object; and when at least a part of the real object to be superimposed is occluded by the second real object, controlling the display of the virtual object based on at least the tracking result of the second real object.

20. A computer-readable recording medium having recorded thereon a program for causing a computer to function as a control unit that acquires information regarding the status of a real object onto which a virtual object is to be superimposed and a second real object different from the real object onto which the virtual object is to be superimposed, and controls the display of the virtual object based on the acquired information.

Images