Information processing apparatus, information processing method, and program

Abstract

To provide an information processing device capable of shortening a time until a virtual object is superimposed and displayed at a correct position and posture.SOLUTION: The information processing apparatus includes an acquisition unit configured to acquire information from a sensor, an estimation unit configured to estimate a position of the sensor and an orientation of the sensor based on the information, a calculation unit configured to calculate a first update amount of the position of the sensor based on a relative relationship between the position of the sensor and a position of a virtual object, and calculate a second update amount of the orientation of the sensor based on a relative relationship between the orientation of the sensor and an orientation of the virtual object, and an update unit configured to update the position of the sensor and the orientation of the sensor based on the first update amount and the second update amount.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for correcting the estimation results of the position and orientation of a sensor when a virtual object is displayed superimposed on an image captured by a camera. [Background technology]

[0002] In recent years, a configuration has been proposed in which a virtual object is superimposed on an image captured by a camera, and an image is displayed on a head-mounted display, making it appear as if the virtual object actually exists (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] Patent Publication No. 2021-99383 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the configuration of Patent Document 1, it takes time to complete correction of self-position estimation, and the time for which the virtual object is superimposed and displayed in an inappropriate position and orientation increases.

[0005] The present invention can provide an information processing device that can shorten the time it takes for a virtual object to be superimposed and displayed in the correct position and orientation. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention is characterized by having an acquisition unit that acquires information from a sensor; an estimation unit that estimates the position and orientation of the sensor based on the information; a calculation unit that calculates a first update amount for the position of the sensor based on the relative relationship between the position of the sensor and the position of a virtual object and calculates a second update amount for the orientation of the sensor based on the relative relationship between the orientation of the sensor and the orientation of the virtual object; and an update unit that updates the position and orientation of the sensor based on the first update amount and the second update amount. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an information processing device that can shorten the time it takes for a virtual object to be superimposed and displayed in the correct position and orientation. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing a usage scene of a head-mounted display equipped with an information processing device according to a first embodiment. FIG. [Figure 2] 1 is a hardware configuration diagram of an information processing apparatus according to a first embodiment. [Figure 3] FIG. 2 is a diagram illustrating a logical block configuration of the information processing device and the head-mounted display according to the first embodiment. [Figure 4] 10 is a flowchart showing a process performed when the information processing apparatus of the first embodiment updates the sensor coordinates. [Figure 5] 10 is a diagram illustrating the relationship between each coordinate when a position and orientation update amount calculation process is executed according to the first embodiment. FIG. [Figure 6] 10 is a flowchart showing a position and orientation update amount calculation process according to the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an application scene of an information processing apparatus according to a second embodiment. [Figure 8] 10 is a flowchart showing a position and orientation update amount calculation process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same components, and redundant explanations will be omitted.

[0010] In each embodiment, an example in which the present invention is applied to a head-mounted display will be described. The head-mounted display in each embodiment relates to technologies called augmented reality (AR) or mixed reality (MR).

[0011] To provide a user (wearer) of a head-mounted display with an experience in which a virtually rendered virtual object (virtual object) appears to be real, it is necessary to accurately estimate the coordinates of the sensor (camera) that captures the image displayed on the head-mounted display. Coordinates are information with six degrees of freedom, including three-dimensional position and orientation. While it is necessary to accurately estimate the sensor coordinates, if the sensor coordinates are suddenly changed by optimization calculations or the like, the position of the virtual object will appear to have changed suddenly, degrading the user experience. The present invention improves the user experience by correcting the estimated sensor coordinates as quickly as possible while making the correction less noticeable to the user. [Example]

[0012] 1 is a diagram showing a usage scene of a head-mounted display 200 equipped with an information processing device 100 of this embodiment. The head-mounted display 200 generates an image (mixed reality image) in which a virtual object 300 is superimposed on an image, and displays the generated image. In FIG. 1, the virtual object 300 is a mug-shaped virtual object drawn by the head-mounted display 200, and is displayed as if it is floating in the air in front of the head-mounted display 200.

[0013] 2 is a hardware configuration diagram of the information processing device 100. The information processing device 100 has a CPU 150, a RAM 151, a storage unit 152 such as an HDD or SSD, an I / O 153, and a system bus 154. The CPU 150 uses the RAM 151 as a work memory to execute an operating system (OS) and various computer programs stored in the storage unit 152, and controls each unit via the system bus 154. For example, the programs executed by the CPU 150 include a program for correcting the coordinates of a sensor (described later) and determining the amount of update. The I / O 153 controls the sensor mounted on the head mounted display 200 and acquires image data output by the sensor.

[0014] 3 is a diagram showing the logical block configuration of the information processing device 100 and the head-mounted display 200. The sensor 201 is a camera capable of acquiring full-color stereo color images and outputs the images at regular intervals. The display unit 202 displays a virtual object 300 superimposed on the stereo color image acquired by the sensor information acquisition unit (acquisition unit) 101. The virtual object 300 is rendered as if it actually exists, based on the relationship between the coordinates of the sensor 201 and the coordinates at which the virtual object is superimposed and displayed. Here, the coordinates of the sensor 201 in a reference coordinate system whose origin is the position and orientation of the information processing device 100 at the time of initialization are called sensor coordinates, and the coordinates of the virtual object 300 in the reference coordinate system are called object coordinates.

[0015] The sensor information acquisition unit 101 acquires sensor information including stereo color images (image information) from the sensor 201 via the I / O 153, and transmits the stereo color images to the position and orientation estimation unit (estimation unit) 102. The position and orientation estimation unit 102 grayscales the stereo color images and estimates sensor coordinates by applying an algorithm called Visual Simultaneous Localization and Mapping (Visual SLAM). The position and orientation update amount calculation unit (calculation unit) 103 calculates the amount of update of coordinates related to the information processing device 100. The position and orientation update unit (update unit) 104 updates the sensor coordinates by performing integral calculations based on the amount of update calculated by the position and orientation update amount calculation unit 103.

[0016] The processing procedures shown in the flowcharts in the following explanations are not limited to the examples, and any combination of procedures, combining multiple processes, or subdividing processes is possible as long as the results of the present invention are achieved. Also, each process can be extracted individually and function as a single functional element, and can be used in combination with processes other than those shown.

[0017] FIG. 4 is a flowchart showing the process performed by the information processing device 100 when updating the sensor coordinates.

[0018] In step S101, the information processing device 100 performs an initialization process, thereby making the information processing device 100 ready for operation.

[0019] In step S102, the sensor information acquisition unit 101 acquires sensor information including one frame of stereo color image from the sensor 201 via the I / O 153.

[0020] In step S103, the position and orientation estimation unit 102 estimates sensor coordinates. As described above, the position and orientation estimation unit 102 converts the stereo color image input in step S102 into grayscale, and then estimates the sensor coordinates by applying an algorithm called Visual SLAM.

[0021] In step S104, the position and orientation update amount calculation unit 103 determines whether correction processing is required for the sensor coordinates estimated by the position and orientation estimation unit 102. Correction processing is required for the sensor coordinates when, for example, the estimated sensor coordinates change sharply as a result of correcting the sensor coordinates using an algorithm such as loop closing or loop closure. Other examples include a case where correction processing was performed in the previous frame but is not complete and therefore correction needs to be performed again, or a case where the number of feature points that can be detected from the image acquired by the sensor 201 is small and the estimation accuracy of the sensor coordinates has decreased. If the position and orientation update amount calculation unit 103 determines that correction processing is required for the sensor coordinates, it performs processing in step S105. If it determines that correction processing is not required, it performs processing in step S107.

[0022] In step S105, the position and orientation update amount calculation unit 103 executes a position and orientation update amount calculation process, which will be described later, and calculates the amount of update of coordinates related to the information processing device 100.

[0023] In step S 106 , the position and orientation update unit 104 updates each coordinate based on the update amount calculated by the position and orientation update amount calculation unit 103 .

[0024] In step S107, the position and orientation update amount calculation unit 103 calculates the difference between the sensor coordinates in the previous frame and the newly estimated sensor coordinates as the update amount.

[0025] In step S108, the information processing device 100 determines whether or not to end this flow. If the information processing device 100 determines that an end instruction has been input by an input means (not shown), it ends this flow. If the information processing device 100 determines that an end instruction has not been input, it executes the process of step S102.

[0026] FIG. 5 is a diagram showing the relationship between each coordinate when the position and orientation update amount calculation unit 103 executes the position and orientation update amount calculation process. Sensor coordinates 401 are sensor coordinates estimated by the position and orientation estimation unit 102. Sensor coordinates 402 are coordinates estimated by the position and orientation estimation unit 102 in the frame following the frame in which the sensor coordinates 401 are estimated. Object coordinates 408 are object coordinates of the virtual object 300. Position difference vector 403 is a vector representing the amount of change in position between the sensor coordinates 401 and 402. Direction vector 404 is a vector representing the relative position between the sensor coordinates 401 and the object coordinates 408. Position difference major axis vector 405 is a vector of the position difference vector 403 that has the same directional component as the direction vector 404. Position difference normal vector 406 is a vector obtained by subtracting the position difference major axis vector 405 from the position difference vector 403, and has a directional component perpendicular to the direction vector 404. The position update amount 407 is an update amount related to the position of the sensor coordinates calculated by the position and orientation update amount calculation unit 103.

[0027] FIG. 6 is a flowchart showing the position and orientation update amount calculation process of this embodiment, and shows the details of the process of step S105.

[0028] In step S111, the position and orientation update amount calculation unit 103 calculates the amount of change in the sensor coordinates. In Fig. 5, a position difference vector 403 is calculated from the difference between the sensor coordinates 401 estimated in the previous frame and the sensor coordinates 402 estimated in the current frame.

[0029] In step S112, the position and orientation update amount calculation unit 103 acquires object coordinates. In Fig. 5, object coordinates 408 are acquired.

[0030] In step S113, the position and orientation update amount calculation unit 103 acquires the relative position between the sensor coordinates and the object coordinates. In Fig. 5, a direction vector 404 is acquired from the sensor coordinates 401 and the object coordinates 408.

[0031] In step S114, the position and orientation update amount calculation unit 103 acquires a position update amount. In FIG. 5 , the position difference vector 403 is decomposed into a position difference main axis vector 405 and a position difference normal vector 406. Then, the position difference main axis vector 405 and the position difference normal vector 406 are multiplied by different weighting coefficients and then combined to acquire a position update amount 407. At this time, the weight of the position difference main axis vector 405 in the direction along the line connecting the position of the sensor 200 and the position of the virtual object 300 is made relatively heavier than the weight of the position difference normal vector 406. As a result, the virtual object 300 changes more in size when displayed on the screen of the display unit 202, while the amount of change in its position in the up, down, left, and right directions is reduced. In other words, the position and orientation update amount calculation unit 103 suppresses only the amount of change in the up, down, left, and right directions that are easily perceived by the user of the head-mounted display 200, and calculates a position update amount that will complete correction as quickly as possible.

[0032] In step S115, the position and orientation update amount calculation unit 103 calculates the amount of orientation update. In FIG. 5, an upper limit is set on the relative orientation change amount (the amount of orientation change of the sensor 200) from the sensor coordinate 401 to the sensor coordinate 402, and the amount of orientation change is set to a value within a predetermined value. Specifically, if the relative orientation change amount is smaller than the predetermined value, the amount of orientation update can be calculated based on the relative relationship between the orientation of the sensor 200 and the virtual object 300. If the relative orientation change amount is larger than the predetermined value, the amount of orientation update can be set to the amount when the relative orientation change amount is the predetermined value. As a result, the amount of orientation change in the coordinates of the virtual object 300 in the screen coordinate system of the display unit 202 is reduced. Therefore, the correction is less noticeable to the user of the head-mounted display 200.

[0033] In step S116, the position and orientation update amount calculation unit 103 combines the position update amount and the orientation update amount to calculate the sensor coordinate update amount.

[0034] As described above, in this embodiment, the sensor coordinates are updated based on the relative position and orientation relationship between the sensor coordinates and the object coordinates. This makes it possible to quickly correct the position and orientation at which the virtual object is rendered while preventing the virtual object from appearing to move abruptly. <Variation 1-1> In this embodiment, the position and orientation update amount calculation unit 103 calculates a direction vector 404 from the relationship between the sensor coordinates 401 estimated by the position and orientation estimation unit 102 and the object coordinates 408, and calculates an update amount so as to quickly perform correction in the same direction as the direction vector 404. However, the present invention is not limited to this.

[0035] For example, if the sensor coordinates 402 estimated by the position and orientation estimation unit 102 are sufficiently far from the object coordinates 408 , the update amount calculated by the position and orientation update amount calculation unit 103 may be increased regardless of the direction vector 404 .

[0036] Furthermore, when it is determined that the head mounted display 200 is being shaken horizontally or vertically from a change in the sensor coordinates 402 estimated by the position and orientation estimation unit 102, the position and orientation update amount calculation unit 103 may increase the update amount in that direction compared to other directions. For example, the position and orientation update amount calculation unit 103 may calculate the update amount so that the update amount in the direction of the change detected by the sensor 200 is larger than the update amount in a direction different from the direction of the change.

[0037] Furthermore, when it is determined that the user of the head mounted display 200 is walking based on the change in the position and orientation estimated by the position and orientation estimation unit 102, the position and orientation update amount calculation unit 103 may increase the amount of correction in any direction.

[0038] In this way, the position and orientation update amount calculation unit 103 is more likely to be able to set a large position and orientation update amount, and can complete the correction more quickly. This allows the virtual object 300 to be displayed on the display unit 202 in the correct position and orientation more quickly. <Variation 1-2> In this embodiment, the position and orientation update amount calculation unit 103 calculates a combined position update amount 407 by weighting the position difference main axis vector 405 relatively heavier than the weight of the position difference normal vector 406. Therefore, correction is not completed in one go but is performed multiple times, but it is not necessarily required to perform correction multiple times. For example, if the norm of the position difference normal vector 406 is sufficiently small, the position and orientation update amount calculation unit 103 may calculate the position difference vector 403 as the update amount and complete the correction in one go. <Variation 1-3> In this embodiment, the position and orientation update amount calculation unit 103 calculates the position and orientation update amount based on the relationship between the sensor coordinates 401, 402 and the object coordinates 408. However, the present invention is not limited to this as long as the correction can be completed quickly while suppressing the perception of the correction. For example, information (gaze direction) related to the line of sight of the eyeballs of the user viewing the virtual object 300 on the head-mounted display 200 may be used. The position and orientation update amount calculation unit 103 may calculate the update amount of the sensor coordinates after correcting the orientation of the sensor coordinates estimated by the position and orientation estimation unit 102 based on the gaze information. Furthermore, the position and orientation update amount calculation unit 103 may calculate the update amount such that the update amount in a direction parallel to the gaze direction, which is a direction that is difficult for the user to perceive, is larger than the update amount in a direction different from the gaze direction. [Example]

[0039] In the first embodiment, the position and orientation update amount calculation unit 103 and the position and orientation update unit 104 updated the sensor coordinates of the sensor 201. In this embodiment, a method will be described in which object coordinates are updated in addition to this, so that even when multiple virtual objects exist, the sensor coordinates are corrected as quickly as possible while the user is less likely to notice the correction. The basic configuration of the information processing device 100 in this embodiment is the same as that of the information processing device in the first embodiment, and in this embodiment, only the configuration different from that in the first embodiment will be described, and a description of the same configuration will be omitted.

[0040] FIG. 7 is a diagram showing an application scene of information processing device 100 of this embodiment. Block 301 is a real-world object on which a virtual object is superimposed. A user can hold block 301 in their hand and move it. Virtual mug 302 is an image rendered by head-mounted display 200 and is a virtual object superimposed on block 301. When a user holds and moves block 301 in their hand, virtual mug 302 is superimposed and displayed to follow the user's movement. Virtual desk 303 is a desk-shaped virtual object rendered by head-mounted display 200 and is displayed as if it were placed on the floor in front of head-mounted display 200. In this embodiment, virtual mug 302 is superimposed on block 301 held by the user, and an image is presented to the user as if they were holding a mug.

[0041] FIG. 8 is a flowchart showing the position and orientation update amount calculation process of this embodiment, and shows the details of the process of step S105.

[0042] In step S211, the position and orientation update amount calculation unit 103 selects a reference virtual object that serves as a reference for correcting the sensor coordinates. For example, the virtual object with the largest volume is selected. In FIG. 7, this is the virtual desk 303.

[0043] The processes in steps S212 to S217 are similar to the processes in steps S111 to S116, respectively, and therefore will not be described here.

[0044] In step S218, the position and orientation update amount calculation unit 103 selects a virtual object whose coordinates are to be updated. In this step, a virtual object other than the reference virtual object is selected. In FIG. 7, this is virtual mug 302.

[0045] In step S219, position and orientation update amount calculation unit 103 calculates the amount of update for the coordinates of the virtual object selected in step S218. Position and orientation update amount calculation unit 103 calculates the amount of coordinate update so that the amount of change in the relative position and orientation between the sensor coordinates and the coordinates of the selected virtual object is equal to or less than a predetermined value. For example, the sensor coordinates and the coordinates of virtual mug 302 in the previous frame of FIG. 7 are retained, and the amount of update for the coordinates of virtual mug 302 is calculated so that the relative position and orientation of the coordinates remains unchanged. This makes it possible to prevent abrupt changes in the relative relationship between the sensor coordinates and the object coordinates of virtual mug 302. This makes it possible to prevent abrupt changes in the rendering of not only virtual desk 303 but also virtual mug 302, making the correction less noticeable to the user of head-mounted display 200.

[0046] In step S220, the position and orientation update amount calculation unit 103 determines whether all virtual objects other than the reference virtual object have been selected. If the position and orientation update amount calculation unit 103 determines that all virtual objects have been selected, it ends this flow, and if it determines that this has not been selected, it executes the processing of step S2.

[0047] As described above, in this embodiment, the coordinates of the virtual objects are updated according to the update amounts of the sensor coordinates, which makes it possible to quickly correct the positions and orientations at which the virtual objects are rendered while preventing multiple virtual objects from appearing to move suddenly. <Variation 2-1> In this embodiment, two virtual objects, virtual mug 302 and virtual desk 303, have been described as examples, but the number of target virtual objects is not limited to this. The number of virtual objects to be corrected may be one, or two or more. The position and orientation update amount calculation unit 103 may set the same update amount for all virtual objects to be corrected, or may calculate different update amounts for each. For example, the update amount may be calculated to decrease as the distance from the sensor coordinates increases, or correction may not be performed if a predetermined distance is exceeded. <Variation 2-2> In this embodiment, the update amounts of the coordinates of objects other than the reference virtual object are calculated, but the coordinates of the reference virtual object may also be updated. For example, the same update amount as the update amount of the sensor coordinates may be calculated for the virtual object so that the relative position and orientation between the sensor coordinates and the coordinates of the reference virtual object do not change. <Variation 2-3> In this embodiment, the position and orientation update amount calculation unit 103 selects the virtual desk 303, which has the largest volume, as the reference virtual object, but this is not limited to this. For example, the virtual object closest to the sensor coordinates may be used as the reference virtual object. [Example]

[0048] In a first embodiment, the position and orientation update amount calculation unit 103 calculates an update amount of the sensor coordinates based on the relationship between the sensor coordinates and the object coordinates, and updates the sensor coordinates based on the calculated update amount. In a second embodiment, the position and orientation update amount calculation unit 103 calculates an update amount of the sensor coordinates based on the relationship between the sensor coordinates and the object coordinates, and calculates an update amount of the coordinates of other virtual objects. In this embodiment, a method is described in which whether or not to update the coordinates of the virtual object is switched depending on the state of the virtual object, thereby further reducing the possibility that the user will perceive an inconsistency in the position and orientation or the positional relationship. In this embodiment, the position and orientation update amount calculation unit 103 switches whether or not to update the coordinates of the virtual object depending on the state of the virtual object. For example, the process of step S219 in FIG. 8 is switched depending on whether the block 301 is held by the user of the head-mounted display 200 and whether or not it is placed on an actual desk.

[0049] In step S219, the position and orientation update amount calculation unit 103 first determines whether the selected virtual object is moving in synchronization with the sensor coordinates. The position and orientation update amount calculation unit 103 associates information in which the sensor coordinates for the most recent three seconds are recorded in chronological order with information in which the object coordinates for the most recent three seconds are recorded in chronological order using the Iterative Closest Point (ICP) algorithm. Next, the position and orientation update amount calculation unit 103 calculates the variance of the coordinate error at each associated time. If the calculated variance is less than a predetermined value, the position and orientation update amount calculation unit 103 determines that the virtual object is moving in synchronization with the sensor coordinates. If it is determined that the virtual object is moving in synchronization with the sensor coordinates, the position and orientation update amount calculation unit 103 calculates the update amount of the virtual object so as to suppress the relative change amount between the sensor coordinates and the coordinates of the selected virtual object. If it is determined that the virtual object is not moving in synchronization with the sensor coordinates, the position and orientation update amount calculation unit 103 calculates the update amount of the virtual object as zero (setting the virtual object to be updated without any update).

[0050] As described above, in this embodiment, when a virtual object is held by a user of the head-mounted display 200, the position and orientation update amount calculation unit 103 updates the coordinates of the virtual object to prevent the correction of the sensor coordinates from being perceived. On the other hand, when the virtual object is not held by the user, the occurrence of a mismatch in the positional relationship between a real object and a virtual object is prevented. This not only achieves the effects described in the second embodiment, but also further reduces the possibility that the user will perceive a mismatch in the positional relationship or a correction of the position and orientation depending on the state of the virtual object. [Other Examples] In each embodiment, the information processing device 100 has been described as being built into the head-mounted display 200, but the present invention is not limited to this. For example, the information processing device 100 may be a computer connected to the head-mounted display 200 via wireless communication.

[0051] In each embodiment, the head-mounted display 200 has been described as a wearable head-mounted display, but the present invention is not limited to this. For example, the head-mounted display may be a handheld head-mounted display that is held in the hand, a monitor generally connected to a computer, or a portable device.

[0052] In each embodiment, the sensor that captures the image to be displayed on the display unit 202 and the sensor that the position and orientation estimation unit 102 uses to estimate the sensor coordinates are described as being the same full-color stereo camera. However, the present invention is not limited to this. For example, the sensor used to estimate the sensor coordinates and the sensor that captures the image to be displayed on the display unit 202 may be different sensors. For example, the sensor used to estimate the sensor coordinates may be a monochrome stereo camera, and the sensor that captures the image to be displayed on the display unit 202 may be a full-color stereo camera. In this case, in step S103, the position and orientation estimation unit 102 estimates the coordinates of the monochrome stereo camera and then estimates the coordinates of the full-color stereo camera based on the relative positional relationship between the monochrome stereo camera and the full-color stereo camera. In step S105, the position and orientation update amount calculation unit 103 calculates the update amount of the coordinates of the full-color stereo camera. In step S106, the position and orientation update unit 104 converts the update amount of the coordinates of the full-color stereo camera into the update amount of the coordinates of the monochrome stereo camera based on the relative positional relationship between the monochrome stereo camera and the full-color stereo camera. This updates the coordinates of the monochrome stereo camera.

[0053] Furthermore, the sensor used to estimate the sensor coordinates does not have to be a camera. For example, it may be a 3D LiDAR. In this case, the position and orientation estimation unit 102 estimates the position and orientation of the 3D LiDAR using an algorithm called LiDAR SLAM.

[0054] In the first and second embodiments, the sensor information acquisition unit 101 inputs an image from the sensor 201 mounted on the head-mounted display 200, but the present invention is not limited to this. For example, the movement of a person wearing a motion capture suit may be input using a motion capture system capable of tracking human movement. In this case, the position and orientation estimation unit 102 estimates the position and orientation of the sensor 201 from the movement of the head of the person wearing the motion capture suit.

[0055] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0056] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) an acquisition unit that acquires information from a sensor; an estimation unit that estimates the position and the attitude of the sensor based on the information; a calculation unit that calculates a first update amount for the position of the sensor based on a relative relationship between the position of the sensor and the position of the virtual object, and calculates a second update amount for the orientation of the sensor based on a relative relationship between the orientation of the sensor and the orientation of the virtual object; an update unit that updates the position and the attitude of the sensor based on the first update amount and the second update amount. (Configuration 2) The information processing device according to configuration 1, wherein the calculation unit calculates the first update amount so that the update amount in a first direction is greater than the update amount in a second direction different from the first direction. (Configuration 3) 3. The information processing device according to configuration 2, wherein the first direction is a direction along a straight line connecting the position of the sensor and the position of the virtual object. (Configuration 4) 3. The information processing apparatus according to claim 2, wherein the first direction is a direction of change detected by the sensor. (Configuration 5) the acquisition unit acquires information about a line of sight of an eyeball viewing the virtual object, 3. The information processing device according to configuration 2, wherein the first direction is a direction parallel to the line of sight. (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the calculation unit calculates the second update amount based on a relative relationship between the attitude of the sensor and the attitude of the virtual object when the amount of change in the attitude of the sensor is smaller than a predetermined value, and sets the second update amount to the amount when the amount of change in the attitude of the sensor is the predetermined value when the amount of change in the attitude of the sensor is larger than the predetermined value. (Configuration 7) the calculation unit calculates a third update amount for the position of the virtual object and a fourth update amount for the orientation of the virtual object; The information processing device according to any one of configurations 1 to 6, wherein the update unit updates the position of the virtual object based on the third update amount, and updates the orientation of the virtual object based on the fourth update amount. (Configuration 8) 8. The information processing device according to claim 7, wherein the calculation unit calculates the third update amount and the fourth update amount when it determines that the position of the virtual object is changing in synchronization with the position of the sensor. (Configuration 9) 9. The information processing device according to configuration 7 or 8, wherein the calculation unit calculates the third update amount based on the first update amount, and calculates the fourth update amount based on the second update amount. (Configuration 10) The information processing device according to configuration 9, wherein the calculation unit sets the third update amount to be the same as the first update amount, and sets the fourth update amount to be the same as the second update amount. (Configuration 11) The information processing device according to any one of configurations 7 to 10, wherein the third update amount corresponding to a first virtual object is smaller than the third update amount corresponding to a second virtual object that is closer to the position of the sensor than the first virtual object. (Configuration 12) 12. The information processing device according to any one of configurations 7 to 11, wherein the calculation unit does not calculate the third update amount corresponding to a virtual object whose distance to the sensor is longer than a predetermined value. (Configuration 13) The information processing device according to any one of configurations 1 to 12, wherein the calculation unit selects a virtual object based on at least one of a distance to the position of the sensor, an orientation relative to the orientation of the sensor, and a size. (Configuration 14) the acquisition unit acquires image information, 14. The information processing device according to any one of configurations 1 to 13, wherein the estimation unit estimates the position and the orientation of the sensor using the image information. (Configuration 15) 15. The information processing device according to any one of configurations 1 to 14, wherein the position and orientation of the sensor updated by the update unit are used to generate a mixed reality image. (Method 1) acquiring information from the sensor; estimating the position and orientation of the sensor based on the information; calculating a first update amount for the position of the sensor based on a relative relationship between the position of the sensor and the position of a virtual object; calculating a second update amount for the orientation of the sensor based on a relative relationship between the orientation of the sensor and the orientation of the virtual object; updating the position and the attitude of the sensor based on the first update amount and the second update amount. (Configuration 16) A program for causing a computer to execute the information processing method described in Method 1.

[0057] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0058] 100 Information processing device 101 Sensor information acquisition unit (acquisition unit) 102 Position and orientation estimation unit (estimation unit) 103 Position and orientation update amount calculation unit (calculation unit) 104 Position / orientation update section (update section)

Claims

1. an acquisition unit that acquires information from a sensor; an estimation unit that estimates the position and the attitude of the sensor based on the information; a calculation unit that calculates a first update amount for the position of the sensor based on a relative relationship between the position of the sensor and the position of a virtual object, and calculates a second update amount for the orientation of the sensor based on a relative relationship between the orientation of the sensor and the orientation of the virtual object; an updating unit that updates the position and the orientation of the sensor based on the first update amount and the second update amount.

2. The information processing apparatus according to claim 1 , wherein the calculation unit calculates the first update amount so that the update amount in a first direction is greater than the update amount in a second direction different from the first direction.

3. 3. The information processing apparatus according to claim 2, wherein the first direction is a direction along a straight line connecting the position of the sensor and the position of the virtual object.

4. 3. The information processing apparatus according to claim 2, wherein the first direction is a direction of change detected by the sensor.

5. the acquisition unit acquires information about a line of sight of an eyeball viewing the virtual object, 3. The information processing apparatus according to claim 2, wherein the first direction is a direction parallel to the line of sight.

6. 3. The information processing device according to claim 1, wherein the calculation unit calculates the second update amount based on a relative relationship between the attitude of the sensor and the attitude of the virtual object when the change in the attitude of the sensor is smaller than a predetermined value, and sets the second update amount to the amount when the change in the attitude of the sensor is the predetermined value when the change in the attitude of the sensor is larger than the predetermined value.

7. the calculation unit calculates a third update amount for the position of the virtual object and a fourth update amount for the orientation of the virtual object; 3 . The information processing apparatus according to claim 1 , wherein the update unit updates the position of the virtual object based on the third update amount, and updates the orientation of the virtual object based on the fourth update amount. 4 .

8. 8. The information processing apparatus according to claim 7, wherein the calculation unit calculates the third update amount and the fourth update amount when it is determined that the position of the virtual object is changing in synchronization with the position of the sensor.

9. The information processing apparatus according to claim 7 , wherein the calculation unit calculates the third update amount based on the first update amount, and calculates the fourth update amount based on the second update amount.

10. 10. The information processing apparatus according to claim 9, wherein the calculation unit sets the third update amount to be equal to the first update amount, and sets the fourth update amount to be equal to the second update amount.

11. 8. The information processing apparatus according to claim 7, wherein the third update amount corresponding to a first virtual object is smaller than the third update amount corresponding to a second virtual object that is closer to the position of the sensor than the first virtual object.

12. The information processing apparatus according to claim 7 , wherein the calculation unit does not calculate the third update amount corresponding to a virtual object whose distance to the sensor is longer than a predetermined value.

13. 3 . The information processing apparatus according to claim 1 , wherein the calculation unit selects a virtual object based on at least one of a distance to the position of the sensor, an orientation relative to the orientation of the sensor, and a size.

14. the acquisition unit acquires image information, 3. The information processing apparatus according to claim 1, wherein the estimation unit estimates the position and the orientation of the sensor using the image information.

15. 3. The information processing apparatus according to claim 1, wherein the position and orientation of the sensor updated by the update unit are used to generate a mixed reality image.

16. acquiring information from the sensor; estimating the position and orientation of the sensor based on the information; calculating a first update amount for the position of the sensor based on a relative relationship between the position of the sensor and the position of a virtual object; calculating a second update amount of the orientation of the sensor based on a relative relationship between the orientation of the sensor and the orientation of the virtual object; updating the position of the sensor based on the first update amount; updating the attitude of the sensor based on the second update amount.

17. A program for causing a computer to execute the information processing method according to claim 16.

Images