Information processing device, method for controlling information processing device, system, and program

Abstract

To provide an information processing device capable of distinguishing between a user's position attitude including a position attitude of a moving body and the position attitude in the user's moving body for acquisition without separately preparing a motion sensor to be installed in the moving body when the user gets on the moving body.SOLUTION: An information processing device includes determination means for determining which a feature point corresponds to a first feature point group or to a second feature point group different from the first feature point group on the basis of a detection result of an inertial sensor included in a display device and a feature point detected from a picked-up image acquired from an imaging part included in the display device, and estimation means for estimating a position or an attitude of the display device based on the first feature point group, or estimating a position or an attitude of the display device based on the second feature point group.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In recent years, mixed reality (MR) technology and virtual reality (VR) technology have become known that use a head mounted display (HMD) to allow the user to experience a space different from the real space. In such technology, it is being considered to allow the user to control the HMD in various ways while wearing the HMD. The HMD acquires and controls the position and orientation of the HMD from information from an acceleration sensor and an angular velocity sensor built into the HMD.

[0003] When the above-mentioned HMD is used while riding on a vehicle such as an automobile, the information obtained by the above-mentioned acceleration sensor and angular velocity sensor includes not only the user's motion but also the motion of the vehicle. This can cause a problem that an image different from the amount of movement and direction of movement of the user in the vehicle is displayed on the image display device. In order to solve this problem, Patent Document 1 discloses a technology for determining an image to be displayed on the HMD by subtracting the motion of the vehicle obtained from a motion sensor installed on the vehicle from the motion of the user obtained from a motion sensor installed on the user's body. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-049831 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology disclosed in the above-mentioned patent document requires the trouble of separately preparing a motion sensor to be placed on the vehicle, which is not necessary when using an HMD without boarding the vehicle. Therefore, an object of the present invention is to provide an information processing device that, when the user is boarding a moving body, can distinguish and acquire the position and orientation of the user, including the position and orientation of the moving body, from the position and orientation of the user within the moving body, without separately preparing a motion sensor. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device that communicates with a display device, comprising: a first acquisition means for acquiring detection results of an inertial sensor of the display device; a second acquisition means for acquiring an image from an imaging unit of the display device; a detection means for detecting feature points of a subject from the image acquired by the second acquisition means; a determination means for determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group based on the detection results of the inertial sensor acquired by the first acquisition means and the feature points detected by the detection means; and an estimation means for estimating a position or orientation of the display device based on the first feature point group, or estimating the position or orientation of the display device based on the second feature point group. Effect of the Invention

[0007] According to the present invention, it is possible to provide an information processing device that, when a user is on a moving body, can distinguish and acquire the position and posture of the user, including the position and posture of the moving body, from the position and posture of the user within the moving body, without the need for a separate motion sensor. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram for explaining a first embodiment. [Diagram 3] 1A and 1B are diagrams illustrating a scene in which the HMD 100 is used while riding on a moving object. [Figure 4] 1A and 1B are diagrams illustrating a process of estimating a self-position using a camera image. [Diagram 5] 2A to 2C are diagrams for explaining the process of calculating the amount of movement based on the position and orientation acquired by the orientation sensor unit 204. [Figure 6] 2 is a diagram for explaining the process of estimating the self-position when the moving object moves and the user also moves within the moving object, and calculating the amount of movement based on the position and orientation detected by the orientation sensor unit 204. FIG. [Figure 7] FIG. 4 is a flowchart illustrating a processing procedure of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (Embodiment 1) An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG in a form that corresponds to the posture of the HMD 100.

[0012] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 by a wired connection such as a USB cable, or wirelessly by a connection such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that, although a PC will be described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each component of the PC 110 may be included in the HMD 100.

[0013] The controller 120 performs various controls of the HMD 100. If the PC 110 is in a specific control mode, when a user operates the controller 120, the HMD 100 is controlled in response to the user's operation. As shown in FIG. 1, the controller 120 may be in the shape of a finger ring that can be worn and supported by the user, or in the shape of a handheld type that can be held in the hand. The controller 120 also has physical buttons for performing a decision operation or a selection operation on the display. The controller 120 performs wireless communication with the PC 110 via Bluetooth. The controller is not limited to a system that communicates with the PC 110, and may be a system that communicates with the HMD 100.

[0014] By moving the controller 120, the user can change the indicated position on the display in accordance with the movement of the controller. The indicated position may be expressed as a point, or may be expressed as a virtual ray connecting the point of the indicated position and the controller with a straight line (segment) or dotted line. By pressing a physical button, a menu decision operation or selection operation can be performed. Although the shape of the controller 120 has been described as being ring-shaped or handheld, the shape is not limited to this as long as it can be supported by a finger, hand, or arm. Furthermore, although the buttons have been described as being physical buttons, they may be operated in the same manner as a trackpad, touch panel, wheel, or trackball, and may be operated by sliding, flicking, or touching in addition to pressing a button.

[0015] Note that the controller may be worn on at least one of a finger, a hand, or an arm.

[0016] Note that the controller may be attached to an object held by hand, and used to acquire position information and attitude information of the attached position from a sensor. Examples of such an object include an object imitating a tool.

[0017] <Internal Structure of HMD> With reference to FIG. 2, the internal structure of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, a working memory 206, and an eye imaging unit 207.

[0018] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by synthesizing an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, it displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0019] The imaging unit 202 includes two cameras (imaging devices). The two cameras are for capturing captured images used for compositing with an image of a virtual space and generating position and orientation information, and have an imaging unit for a left eye and an imaging unit for a right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the HMD 100, and an image (captured image) of each frame in the moving image is output from the imaging unit for the left eye. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the HMD 100, and an image (captured image) of each frame in the moving image is output from the imaging unit for the right eye. In other words, the imaging unit 202 acquires captured images as stereo images having parallax that approximately matches the positions of the left eye and right eye of the person wearing the HMD 100. In addition, information on the distance from the two cameras to the subject can be acquired as distance information by distance measurement using the stereo cameras. Note that in an HMD for an MR system, it is preferable to arrange the imaging unit so that the central optical axis of the imaging range of the imaging unit approximately matches the line of sight direction of the person wearing the HMD.

[0020] Each of the left eye imaging section and the right eye imaging section has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the entering light as a captured image. Images of a subject (a range in front of the user) captured by the two cameras are output to the PC 110 and the control unit 201. Note that the imaging unit 202 may capture and output a video instead of a captured image.

[0021] The image display unit 203 displays a composite image. The image display unit 203 has a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, the image display unit 203 is disposed in front of each eye of the user. Note that a device using a semi-transmissive half mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 may display an image by using a technology generally called AR (Augmented Reality) so that the CG is directly superimposed on the real space seen through the half mirror. The image display unit 203 may also display an image of a complete virtual space without using a captured image by using a technology generally called VR (Virtual Reality).

[0022] The attitude sensor unit 204 is a motion sensor that acquires attitude (and position) information of the HMD 100. The attitude sensor unit 204 may acquire attitude information of a user (a user wearing the HMD 100) that corresponds to the attitude (and position) of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU) that includes inertial sensors such as an acceleration sensor and an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 204 is used when acquiring information on the user's attitude (attitude information), and the HMD control unit 201 outputs the detection result of the information on the user's attitude (attitude information) to the PC 110. The attitude information may be acquired from one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0023] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images obtained by the imaging unit 202. The joint points include points that are characteristic of parts such as the joints and fingertips of the fingers, the back of the hand (palm), and the arms. Each joint point indicates a coordinate position, and the posture can be estimated from information on a plurality of joint points. A known object recognition or pose estimation method of machine learning using a convolutional neural network can be used as a method for estimating the position or posture of the hand and each joint point of the hand. In addition, the position information of each joint point of the hand in the depth direction can be obtained by calculating the distance from the imaging unit 202 to each joint point by triangulation using stereo matching using the two camera images obtained by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110.

[0024] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs executed by the control unit 211, which will be described later.

[0025] The working memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a working area for the control unit 201, and the like.

[0026] The gaze imaging unit 207 is a camera that acquires an image for detecting the gaze of the user, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0027] <Internal configuration of the controller> Referring to FIG. 2, the internal configuration of the controller 120 will be described. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0028] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0029] The operation unit 222 includes buttons. The operation unit 222 detects whether a button has been operated and transmits detection information to the PC 110 via the communication unit 223. Note that the operation unit 222 may have multiple input formats.

[0030] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth. If there are multiple controllers, each performs wireless communication with the PC 110 via Bluetooth.

[0031] The controller attitude sensor unit 224 includes an inertial measurement unit (IMU) composed of inertial sensors such as an acceleration sensor and an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 120. The detected position and attitude change information is communicated from the communication unit 223 to the PC 110 via the controller control unit 221.

[0032] The output unit 225 is composed of an LED light source, a speaker, a vibration element, etc.

[0033] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0034] The control unit 211 is a CPU that controls each unit of the PC 110 according to input signals and a program described later. Instead of the control unit 211 controlling the entire device, the entire device may be controlled by having a plurality of hardware share the processing. The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0035] The control unit 211 also obtains the number of controllers included in the captured image. It also executes a process for recognizing the attachment position of each controller using information obtained via the communication unit 214. Then, depending on the recognition result, it performs control to change the operation content for the input information of each controller for each controller.

[0036] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when the control unit 211 places a virtual object represented by the CG near a specific object existing in the real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 is closer. Also, for example, the control unit 211 draws a virtual object (CG) based on a change in the position or orientation of the HMD 100, thereby drawing a CG whose position or orientation changes with the HMD 100, or a CG that is not based on a change in the position or orientation of the HMD 100, and synthesizes it with the real image. By controlling the position, orientation, and size of the CG in this way, the control unit 211 can generate a composite image in which a CG object that is not placed in the real space appears to be placed in the real space.

[0037] In addition, the control unit 211 receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the working memory 213.

[0038] Furthermore, in the control unit 211, the communication unit 214 receives information on a change in the position or attitude of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes on the combined image an indication position corresponding to the change information on the position or attitude of the controller 120. Note that the control unit 211 may also superimpose on the combined image an indication position corresponding to the change information on the position and attitude of the controller 120.

[0039] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG read from the non-volatile memory 212 (i.e., the CG used to generate a composite image).

[0040] The working memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202 and time-series information on the estimated coordinate positions of each joint point of the hand, as an image display memory for the image display unit 203, a working area for the control unit 211, etc.

[0041] The wrist joints may be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100. Note that the control unit 211 may estimate the position and posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100.

[0042] In addition to the above, the MR system also includes other components, such as a control unit of either the HMD 100 or the PC 110, which performs various image processing on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203. However, this is not the focus of the present invention, so a description thereof will be omitted.

[0043] (Example of use of information processing device) With reference to FIG. 3, a scene in which the HMD 100 is used while riding on a moving object such as a car or train will be described. FIG. 3 shows an example in which a user wears the HMD 100 on the head. The outside scenery is visible behind a window 301 installed in the car or train, and while the moving object is moving, the scenery flows according to the moving speed. In FIG. 3, a building 303 is seen as the outside scenery. The flow of the scenery is captured in the camera image captured by the imaging unit 202. For example, the user may ride on a moving object without windows such as an open car, and the scenery may be captured directly instead of through a window. In addition, the camera image captured by the imaging unit 202 captures an interior scene 302, such as walls and doors inside the moving object.

[0044] (Self-location estimation using camera images) SLAM (Simultaneous Localization and Mapping) is known as a method for estimating self-position using camera images. SLAM is disclosed as a method for detecting the positions of feature points in camera images and determining the self-position in three-dimensional space from the amount of movement of the feature points between frames. It is known that various disclosed algorithms of SLAM can be used to estimate the self-position in three-dimensional space from these feature points.

[0045] The process of estimating self-position using camera images will be described with reference to Fig. 4. Fig. 4 shows an example of estimating self-position using camera images as viewed from a third-party viewpoint. Fig. 4 shows that the user has moved by the length of the arrow in the direction indicated by the amount of movement 400 from the state in Fig. 3. From inside the moving object, the building 303 is observed to have moved by the length of the arrow indicated by the amount of movement 404 in the opposite direction to the direction indicated by the amount of movement 400. Fig. 4 describes a case in which the user does not move inside the moving object.

[0046] When a person is on board a moving object, the camera image shows a window 301 installed in the moving object and an interior scene 302 such as walls and doors inside the moving object.

[0047] When the moving body moves and the user also moves within the moving body, the amount of movement linked to the movement of the moving body as well as the movement of the user itself is detected from the external feature point detected in the window 301 installed in the moving body. Since the user does not move here, the building 303 shown by the dotted line appears to have moved relatively to the HMD and is at the position of the building 403. For example, the feature point 405 of the building 403 obtained at the time of FIG. 4 is detected as having moved by the amount of movement 404 from the feature point 305 detected at the time of FIG. 3 between the frames of FIG. 3 and FIG. 4. When self-location estimation is performed using the external feature point detected in the window 301 installed in the moving body, the self-location of the HMD 100 outside the moving body can be estimated.

[0048] Next, for feature points of the internal scene 302 such as walls and doors inside the moving body, the amount of movement unrelated to the movement of the moving body is detected, and when the user moves inside the moving body, the amount of movement of the user himself inside the moving body is detected. For example, feature point 406 of the internal scene 302 is detected from the HMD as a feature point that has not moved because the user does not move. When self-position estimation is performed using the feature points of the internal scene 302, the self-position of the HMD 100 inside the moving body can be estimated.

[0049] (Amount of movement based on the position and orientation detected by the orientation sensor unit 204) With reference to Fig. 5, the process of calculating the amount of movement based on the position and orientation acquired by the orientation sensor unit 204 will be described. Fig. 5 is an example of the amount of movement detected by the orientation sensor unit 204. The orientation sensor unit 204 can acquire the movement acceleration and the facing direction of the HMD 100. The amount of movement 501 of the HMD 100 is calculated by accumulating the information acquired periodically. The amount of movement 501 of the HMD 100 is a three-dimensional amount of movement. The amount of movement may be expressed by dividing it into three components based on the reference coordinates determined by the HMD 100. Furthermore, not only the amount of movement but also the orientation may be expressed by dividing it into three components, namely roll, pitch, and yaw, which are rotations of a three-dimensional vector.

[0050] (Two methods for providing self-location estimation results) A process for providing two types of self-location estimation results will be described with reference to FIGS. 3, 6 and 7. FIG.

[0051] Fig. 6 is a diagram for explaining the process of estimating the self-position when the moving body moves and the user also moves within the moving body, and calculating the amount of movement based on the position and orientation detected by the orientation sensor unit 204. Fig. 6 shows the case where the user moves in the scene shown in Fig. 4. Fig. 6 shows that the user moves within the moving body in the direction indicated by the amount of movement 600 by the length of the arrow, and moves to the position of the HMD 101. In other words, it shows that the HMD 100 moves within the moving body in the direction indicated by the amount of movement 600 by the length of the arrow, and moves to the position of the HMD 101.

[0052] FIG. 7 is a flowchart showing the flow of a process for providing two types of self-location estimation results.

[0053] In step S701, the control unit 211 controls the imaging unit 202 to store an image of the user's surroundings in the work memory 213. The image capturing is performed continuously, and the camera images are stored in the work memory 213 repeatedly.

[0054] In step S702, the control unit 211 detects feature points in the camera image stored in the working memory 213, and stores position information of the feature points detected for each frame in the working memory 213. In addition, the positions of the feature points between the frames are compared to calculate the amount of movement of each feature point between the frames. As shown in Fig. 6, the external feature point 405 detected in the window 301 installed in the moving body is detected as having moved by an amount of movement 602 due to the movement of the moving body and the movement of the user within the moving body. In addition, the feature point 406 of the internal scene 302 is detected as having moved by an amount of movement 603 due to the movement of the user within the moving body.

[0055] In step S703, the control unit 211 acquires a movement amount 601 of the HMD 100 from the orientation sensor unit 204. The movement amount 601 of the HMD 100 is detected as a total movement amount of the moving body and the user's movement within the moving body. The movement amount 601 detected by the orientation sensor unit 204 alone cannot detect how far the moving body has moved and how far the user has moved within the moving body, but detects the amount of movement of the user as viewed from outside the moving body.

[0056] In step S704, the control unit 211 converts the movement amount 501 of the HMD 100 into the movement amount on the camera image. For example, a method of selecting one of the feature points on the camera image from the position information obtained by adding the movement amount 501 of the HMD 100 to the self-position, which is three-dimensional position information, and calculating the movement amount of the feature point can be considered.

[0057] Here, as shown in FIG. 4, it is assumed that the moving body moves in the front direction and the HMD worn by the user is not moved at all inside the moving body, and a positive value is detected as the amount of movement of the orientation sensor unit 204. In FIG. 4, feature point 305 and feature point 405 are feature points detected in area 301 corresponding to the area outside the moving body, and feature point 406 is a feature point detected inside the moving body. In this case, it is assumed that the absolute value of the amount of movement detected by the orientation sensor unit 204 for the feature points on the camera image detected in area 301 corresponding to the area outside the moving body is equal, but the amount of movement is detected as a negative value. In this way, the movement direction of the feature point on the camera image is opposite to the movement direction of the HMD 100 detected by the orientation sensor unit 204. Therefore, the sign of the amount of movement of the detected feature point is reversed to convert the amount of movement 501 of the HMD 100 into the amount of movement on the camera image. Then, the movement amount of the HMD 100 on the camera image obtained by converting the movement amount 501 of the HMD 100 is subtracted from the movement amount of each feature point between frames. The feature point 305 moves to the position of the feature point 405, and moves by the amount of the arrow 404 on the camera image. The movement amount of the arrow 400 is subtracted from the movement amount of the arrow 404, and if the error is not taken into consideration, the absolute value of the movement amount after the subtraction will be 0 in this figure. In this way, the feature point whose absolute value of the movement amount after the subtraction is less than a predetermined threshold is determined to be the outside of the vehicle area, and is determined to be the first feature point and stored in the working memory 213. Here, when a stationary object is used as a feature point in the outside of the vehicle area, it is expected that the movement amount will be 0 as a result of the subtraction, but since an error may occur, the predetermined threshold is set as a value that takes the error into consideration.

[0058] Furthermore, when the user is not moving inside the vehicle, the movement amount of feature point 406 is 0 on the camera image. The movement amount of arrow 400 is subtracted from the feature point with a movement amount of 0, and if the error is not taken into consideration, the absolute value of the movement amount after subtraction in this figure is equal to the absolute value of the movement amount of arrow 400. In this way, a feature point whose absolute value of the movement amount after subtraction is equal to or greater than a predetermined threshold is determined to be in the vehicle interior area and stored in working memory 213 as a second feature point. The predetermined threshold for determining the first feature point and the predetermined threshold for determining the second feature point may be set to the same numerical value or different numerical values.

[0059] Next, in the vehicle exterior area, not only objects that are actually stationary and appear to be moving relatively, such as buildings, but also moving objects such as automobiles are expected to be detected. In this embodiment, among the feature points whose absolute value of the amount of movement after subtraction is equal to or greater than a threshold value, those that correspond to the feature points of the vehicle interior area have an absolute value of the amount of movement after subtraction that corresponds to the amount of movement of the moving body if no error is taken into consideration, so that the absolute value of the amount of movement after subtraction is a similar value. If a large number of feature points having similar values ​​are detected as a result of the subtraction, they are determined to be a second feature point group and stored in the working memory 213. In addition, the determination of whether the values ​​are similar may be made to determine whether the values ​​are close within a certain error range. Among the feature points determined to be the second feature point group, feature points determined to be the second feature point group surrounded by feature points detected as the vehicle exterior area as the first feature points may be determined not to be the second feature point group as feature points that are not the feature points of the vehicle interior area.

[0060] Note that the sign of the amount of movement of the orientation sensor unit 204 may be reversed to convert it to match the amount of movement on the camera image, i.e., the direction of the amount of movement of the feature point detected from the captured image. Note that, in consideration of the fact that the directions are reversed, the amount of movement of the orientation sensor unit 204 and the amount of movement on the camera image may be added with the signs reversed, instead of subtracting.

[0061] In step S705, the control unit 211 determines whether to use the first feature point or the second feature point to perform self-location estimation. When the first feature point, i.e., the feature point in the vehicle exterior area, is used to perform self-location estimation, the process proceeds to step S606. When the second feature point, i.e., the feature point in the vehicle interior area, is used to perform self-location estimation, the process proceeds to step S607. The control unit 211 determines whether to use the first feature point or the second feature point to perform self-location estimation, depending on whether the feature point used in the application selected by the user is the first feature point or the second feature point.

[0062] In step S706, the control unit 211 performs self-location estimation based on the first feature point, that is, the feature point of the vehicle exterior area, and ends the process.

[0063] After performing self-location estimation in step S706, the control unit 211 may draw a virtual object based on the result of the self-location estimation.

[0064] After the self-location estimation is performed in step S706, the result may be transmitted to an application selected by the user.

[0065] In step S707, the control unit 211 performs self-location estimation based on the second feature points, that is, the feature points of the vehicle interior area, and ends the process.

[0066] After performing self-location estimation in step S707, the control unit 211 may draw a virtual object based on the result of the self-location estimation.

[0067] After the self-location estimation is performed in step S707, the result may be transmitted to an application selected by the user.

[0068] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0069] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0070] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0071] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0072] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0073] [Configuration 1] An information processing device that communicates with a display device, A first acquisition means for acquiring a detection result of an inertial sensor included in the display device; A second acquisition means for acquiring a captured image from an imaging unit included in the display device; a detection means for detecting feature points of a subject from the captured image acquired by the second acquisition means; a determination means for determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection result of the inertial sensor acquired by the first acquisition means and the feature points detected by the detection means; an estimation means for estimating a position or orientation of the display device based on the first feature point group, or estimating a position or orientation of the display device based on the second feature point group; 13. An information processing device comprising:

[0074] [Configuration 2] The information processing device is provided in the display device. 2. The information processing device according to configuration 1.

[0075] [Configuration 3] The method further includes a third acquisition means for acquiring a movement amount of the feature point, the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on the detection result of the inertial sensor acquired by the second acquisition means and the movement amount of the feature point acquired by the third acquisition means; 3. The information processing device according to configuration 1 or 2.

[0076] [Configuration 4] the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on a first value obtained by subtracting a movement amount based on the detection result of the inertial sensor from a movement amount of the feature point; 4. The information processing device according to configuration 3.

[0077] [Configuration 5] The determination means determines that the first value corresponds to the first feature point group when the first value is less than a predetermined threshold value. 5. The information processing device according to configuration 4.

[0078] [Configuration 6] the determination means determines that the plurality of feature points belong to a second feature point group when the first value is equal to or greater than a predetermined threshold value and the first values ​​of the plurality of feature points are within a predetermined range. 6. The information processing device according to configuration 4 or 5.

[0079] [Configuration 7] the determination means determines that a feature point does not belong to the second feature point group when the feature point is surrounded by feature points that belong to the first feature point group even if the first value is equal to or greater than the predetermined threshold value. 7. The information processing device according to configuration 6.

[0080] [Configuration 8] When estimating the position or orientation of the display device based on the first feature point group, the estimation means estimates the position or orientation of the display device based on the detection result of the inertial sensor acquired by the first acquisition means. 8. The information processing device according to any one of configurations 1 to 7.

[0081] [Configuration 9] The estimation means estimates a position or an orientation of the display device based on the first feature point group, or estimates a position or an orientation of the display device based on the second feature point group, depending on an application selected by a user. 9. The information processing device according to any one of configurations 1 to 8.

[0082] [Configuration 10] Further comprising a communication means for transmitting an estimation result by the estimation means to the application. 10. The information processing device according to configuration 9.

[0083] [Configuration 11] the estimation means estimates a position or an orientation of the display device based on the first feature point group, and estimates a position or an orientation of the display device based on the second feature point group; the communication means transmits an estimation result based on the first feature point group and an estimation result based on the second feature point group by the estimation means to the application. 11. The information processing device according to configuration 10.

[0084] [Configuration 12] Further comprising a drawing means for drawing a virtual object based on the estimation result by the estimation means. 12. The information processing device according to any one of configurations 1 to 11.

[0085] [Control method] A method for controlling an information processing device that communicates with a display device, comprising: A first acquisition step of acquiring a detection result of an inertial sensor included in the display device; A second acquisition step of acquiring a captured image from an imaging unit included in the display device; a detection step of detecting feature points of a subject from the captured image acquired by the second acquisition step; a determination step of determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection results of the inertial sensor acquired in the first acquisition step and the feature points detected in the detection step; an estimation step of estimating a position or orientation of the display device based on the first feature point group, or estimating a position or orientation of the display device based on the second feature point group; 13. A method for controlling an information processing apparatus comprising the steps of:

[0086] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 12.

[0087] [system] A display device; a first acquisition device that acquires a detection result of an inertial sensor included in the display device; a second acquisition device that acquires an image from an imaging unit included in the display device; a detection device that detects feature points of a subject from the captured image acquired by the second acquisition device; a determination device that determines whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection result of the inertial sensor acquired by the first acquisition device and the feature points detected by the detection device; an estimation device that estimates a position or orientation of the display device based on the first feature point group, or estimates a position or orientation of the display device based on the second feature point group; An information processing system comprising:

Claims

1. An information processing device that communicates with a display device, A first acquisition means for acquiring a detection result of an inertial sensor included in the display device; A second acquisition means for acquiring a captured image from an imaging unit included in the display device; a detection means for detecting feature points of a subject from the captured image acquired by the second acquisition means; a determination means for determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection result of the inertial sensor acquired by the first acquisition means and the feature points detected by the detection means; an estimation means for estimating a position or orientation of the display device based on the first feature point group, or estimating a position or orientation of the display device based on the second feature point group; 13. An information processing device comprising:

2. The information processing device is provided in the display device.

2. The information processing apparatus according to claim 1,

3. The method further includes a third acquisition means for acquiring a movement amount of the feature point, the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on the detection result of the inertial sensor acquired by the second acquisition means and the movement amount of the feature point acquired by the third acquisition means; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on a first value obtained by subtracting a movement amount based on the detection result of the inertial sensor from a movement amount of the feature point; 4. The information processing apparatus according to claim 3.

5. The determination means determines that the first value corresponds to the first feature point group when the first value is less than a predetermined threshold value.

5. The information processing apparatus according to claim 4.

6. the determination means determines that the plurality of feature points belong to a second feature point group when the first value is equal to or greater than a predetermined threshold value and the first values ​​of the plurality of feature points are within a predetermined range.

5. The information processing apparatus according to claim 4.

7. the determination means determines that a feature point does not belong to the second feature point group when the feature point is surrounded by feature points that belong to the first feature point group even if the first value is equal to or greater than the predetermined threshold value.

7. The information processing apparatus according to claim 6,

8. When estimating a position or an orientation of the display device based on the first feature point group, the estimation means estimates the position or the orientation of the display device based on a detection result of the inertial sensor acquired by the first acquisition means.

2. The information processing apparatus according to claim 1,

9. the estimation means estimates a position or an orientation of the display device based on the first feature point group, or estimates a position or an orientation of the display device based on the second feature point group, depending on an application selected by a user.

2. The information processing apparatus according to claim 1,

10. Further comprising a communication means for transmitting an estimation result by the estimation means to the application.

10. The information processing apparatus according to claim 9,

11. the estimation means estimates a position or an orientation of the display device based on the first feature point group, and estimates a position or an orientation of the display device based on the second feature point group; the communication means transmits an estimation result based on the first feature point group and an estimation result based on the second feature point group by the estimation means to the application.

11. The information processing apparatus according to claim 10,

12. Further comprising a drawing means for drawing a virtual object based on the estimation result by the estimation means.

2. The information processing apparatus according to claim 1,

13. A method for controlling an information processing device that communicates with a display device, comprising: A first acquisition step of acquiring a detection result of an inertial sensor included in the display device; A second acquisition step of acquiring a captured image from an imaging unit included in the display device; a detection step of detecting feature points of a subject from the captured image acquired by the second acquisition step; a determination step of determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection result of the inertial sensor acquired in the first acquisition step and the feature points detected in the detection step; an estimation step of estimating a position or orientation of the display device based on the first feature point group, or estimating a position or orientation of the display device based on the second feature point group; 13. A method for controlling an information processing apparatus comprising the steps of:

14. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

15. A display device; A first acquisition device that acquires a detection result of an inertial sensor included in the display device; A second acquisition device that acquires an image from an imaging unit included in the display device; a detection device that detects feature points of a subject from the captured image acquired by the second acquisition device; a determination device that determines whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection result of the inertial sensor acquired by the first acquisition device and the feature points detected by the detection device; an estimation device that estimates a position or an orientation of the display device based on the first feature point group, or estimates a position or an orientation of the display device based on the second feature point group; An information processing system comprising:

Images