Information processing device, information processing device control method, system, and program

Abstract

This information processing device is characterized by comprising: a determination means that, on the basis of a detection result of an inertial sensor of a display device and a feature point detected from a captured image acquired from an imaging unit of the display device, determines to which one of a first feature point group and a second feature point group that is different from the first feature point group the feature point corresponds; and an estimation means that estimates the position or orientation of the display device based on the first feature point group, or estimates the position or orientation of the display device based on the second feature point group.

Description

Information processing device, control method, system, and program for information processing device

[0001] The present invention relates to an information processing device.

[0002] In recent years, mixed reality (MR) technology and virtual reality (VR) technology have become known that use a head-mounted display (HMD) to allow a user to experience a space different from the real world. With these technologies, various controls for the HMD have been considered, allowing a user to perform various functions while wearing the HMD. The HMD controls its position and orientation by acquiring information from an acceleration sensor and an angular velocity sensor built into the HMD.

[0003] When the above-described HMD is used while riding in a vehicle such as an automobile, the information obtained by the acceleration sensor and angular velocity sensor includes not only the user's movements but also the movements of the vehicle. This can cause a problem in that an image displayed on the image display device differs from the amount and direction of the user's movement within the vehicle. To solve this problem, Patent Document 1 discloses a technology for determining an image to be displayed on the HMD by subtracting the movement of the vehicle obtained from a motion sensor installed in the vehicle from the movement of the user obtained from a motion sensor installed on the user's body.

[0004] Japanese Patent Application Laid-Open No. 2019-049831

[0005] The conventional techniques disclosed in the above-mentioned patent documents require the trouble of separately preparing a motion sensor to be placed on a vehicle, which is not necessary when using an HMD without being on a vehicle. Therefore, an object of the present invention is to provide an information processing device that, when a user is on a moving body, can distinguish and acquire the position and orientation of the user, including the position and orientation of the moving body, from the position and orientation of the user within the moving body, without separately preparing a motion sensor.

[0006] One aspect of the present invention is an information processing device that communicates with a display device, comprising: a first acquisition means that acquires detection results from an inertial sensor possessed by the display device; a second acquisition means that acquires an image from an imaging unit possessed by the display device; a detection means that detects feature points of a subject from the image acquired by the second acquisition means; a determination means that determines whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group based on the detection results of the inertial sensor acquired by the first acquisition means and the feature points detected by the detection means; and an estimation means that estimates the position or orientation of the display device based on the first feature point group, or estimates the position or orientation of the display device based on the second feature point group.

[0007] According to the present invention, it is possible to provide an information processing device that, when a user is on a moving body, can distinguish and acquire the position and posture of the user, including the position and posture of the moving body, from the position and posture of the user within the moving body, without providing a separate motion sensor.

[0008] FIG. 1 is a diagram illustrating an information processing system in a first embodiment. FIG. 2 is a diagram illustrating a block diagram in the first embodiment. FIG. 3 is a diagram illustrating a scene in which an HMD 100 is used while riding on a moving body. FIG. 4 is a diagram illustrating the process of self-position estimation using camera images. FIG. 5 is a diagram illustrating the process of movement amount calculation based on the position and orientation acquired by the orientation sensor unit 204. FIG. 6 is a diagram illustrating the process of self-position estimation when a moving body moves and a user also moves within the moving body, and the process of movement amount calculation based on the position and orientation detected by the orientation sensor unit 204. FIG. 7 is a flowchart illustrating the processing procedure of an information processing device.

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] First Embodiment An information processing system 1 according to a first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG in a form that corresponds to the posture of the HMD 100.

[0012] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 via a wired connection such as a USB cable, or wirelessly via a connection such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG and transmits the composite image to the HMD 100. Note that although a PC is described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and the components of the PC 110 may be included in the HMD 100.

[0013] The controller 120 performs various controls of the HMD 100. When the PC 110 is in a specific control mode, a user's operation on the controller 120 controls the HMD 100 in accordance with the user's operation. As shown in FIG. 1 , the controller 120 may be in the shape of a ring that can be worn on the user's finger or a handheld type that can be held in the hand. The controller 120 also has physical buttons for performing confirmation operations and selection operations on the display. The controller 120 communicates wirelessly with the PC 110 via Bluetooth. The controller is not limited to a system that communicates with the PC 110, but may also be a system that communicates with the HMD 100.

[0014] By moving the controller 120, the user can change the pointed position on the display in accordance with the movement of the controller. The pointed position may be represented by a point, or by a virtual ray connecting the pointed position point and the controller with a straight line (line segment) or dotted line. A menu confirmation operation or selection operation can be performed by pressing a physical button. While the controller 120 has been described as being ring-shaped or handheld, this is not a limitation as long as it can be supported by a finger, hand, or arm. Furthermore, although the buttons have been described as being physical buttons, any operation that can be performed, such as a trackpad, touch panel, wheel, or trackball, may be used. In addition to pressing a button, a slide operation, a flick operation, or a touch operation may also be used.

[0015] The controller may be attached to at least one of a finger, a hand, and an arm.

[0016] The controller may be attached to an object held by hand, and the position and orientation information of the attached position may be acquired from the sensor. Such an object may be an object simulating a tool.

[0017] 2, the internal configuration of the HMD 100 will be described. The HMD 100 has an HMD control unit 201, an image capturing unit 202, an image display unit 203, a posture sensor unit 204, a nonvolatile memory 205, a working memory 206, and a gaze image capturing unit 207.

[0018] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0019] The imaging unit 202 includes two cameras (imaging devices). The two cameras are used to capture images to be combined with images of the virtual space and to generate position and orientation information, and include an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures moving images of the real space corresponding to the left eye of the person wearing the HMD 100, and the left eye imaging unit outputs images of each frame of the moving images (captured images). The imaging unit for the right eye captures moving images of the real space corresponding to the right eye of the person wearing the HMD 100, and the right eye imaging unit outputs images of each frame of the moving images (captured images). In other words, the imaging unit 202 acquires captured images as stereo images with parallax that approximately matches the positions of the left and right eyes of the person wearing the HMD 100. Furthermore, distance information can be acquired by measuring distances using the stereo cameras, indicating the distances from the two cameras to the subject. In an HMD for an MR system, it is preferable that the central optical axis of the imaging range of the imaging unit be positioned so as to substantially coincide with the line of sight of the wearer of the HMD.

[0020] The left-eye imaging unit and the right-eye imaging unit each have an optical system and an imaging device. Light entering from the outside enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incoming light as a captured image. The images of the subject (the area in front of the user) captured by the two cameras are output to the PC 110 and the control unit 201. Note that the imaging unit 202 may capture and output video instead of captured images.

[0021] The image display unit 203 displays a composite image. The image display unit 203 includes a liquid crystal panel, an organic EL panel, or the like. When the user is wearing the HMD 100, the image display unit 203 is disposed in front of each of the user's eyes. It is also possible to use a device using a semi-transparent half mirror as the image display unit 203. In this case, for example, the image display unit 203 may use a technology generally called AR (Augmented Reality) to display an image so that CG appears to be directly superimposed on the real space seen through the half mirror. Furthermore, the image display unit 203 may use a technology generally called VR (Virtual Reality) to display an image of a completely virtual space without using captured images.

[0022] The posture sensor unit 204 is a motion sensor that acquires posture (and position) information of the HMD 100. The posture sensor unit 204 may acquire posture information of the user (the user wearing the HMD 100) that corresponds to the posture (and position) of the HMD 100. The posture sensor unit 204 has an inertial measurement unit (IMU) that includes inertial sensors such as an acceleration sensor and an angular acceleration sensor, and a geomagnetic sensor. The posture sensor unit 204 is used to acquire posture information of the user (posture information), and the HMD control unit 201 outputs the detection results of the posture information of the user (posture information) to the PC 110. The posture information may be acquired from one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0023] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images acquired by the imaging unit 202. The joint points include characteristic points of parts such as the finger joints and fingertips, the back of the hand (palm), and the arm. Each joint point indicates a coordinate position, and the posture can be estimated from information on multiple joint points. Methods for estimating the position or posture of the hand and each joint point can include, for example, well-known machine learning object recognition and pose estimation techniques using a convolutional neural network. Depth position information of each joint point of the hand can be obtained by calculating the distance from the imaging unit 202 to each joint point, for example, by triangulation using stereo matching using the two camera images acquired by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110.

[0024] The nonvolatile memory 205 is an electrically erasable and recordable nonvolatile memory, and stores programs to be executed by the control unit 211, which will be described later.

[0025] The work memory 206 is used as a buffer memory for temporarily storing image data captured by the image capturing unit 202, as an image display memory for the image display unit 203, as a work area for the control unit 201, and the like.

[0026] The gaze imaging unit 207 is a camera that acquires an image for detecting the user's gaze, and is attached to the inside of the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (the user's eyes) captured by the camera is output to a control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0027] <Internal Configuration of Controller> The internal configuration of the controller 120 will be described with reference to Fig. 2. The controller 120 has a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0028] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire device, the entire device may be controlled by having a plurality of hardware devices share the processing.

[0029] The operation unit 222 includes buttons. The operation unit 222 detects whether a button has been operated and transmits the detection information to the PC 110 via the communication unit 223. The operation unit 222 may have a plurality of types of input formats.

[0030] The communication unit 223 performs wireless communication via Bluetooth with the PC 110. If there are multiple controllers, each of them performs wireless communication via Bluetooth with the PC 110.

[0031] The controller attitude sensor unit 224 has an inertial measurement unit (IMU) that is configured with inertial sensors such as an acceleration sensor and an angular acceleration sensor, and a geomagnetic sensor. The IMU detects changes in the position or attitude of the controller 120. Information about the detected changes in position and attitude is communicated from the communication unit 223 to the PC 110 via the controller control unit 221.

[0032] The output unit 225 is composed of an LED light source, a speaker, a vibration element, and the like.

[0033] <Internal Configuration of PC> The internal configuration of the PC 110 will be described with reference to Fig. 2. The PC 110 has a control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0034] The control unit 211 is a CPU that controls each unit of the PC 110 in accordance with input signals and a program described below. Instead of the control unit 211 controlling the entire device, the entire device may be controlled by having multiple pieces of hardware share the processing. The control unit 211 receives from the HMD 100 an image (captured image) acquired by the imaging unit 202 and posture information acquired by the posture sensor unit 204. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. The control unit 211 then composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0035] The control unit 211 also finds the number of controllers included in the captured image, and executes processing to recognize the attached position of each controller using information obtained via the communication unit 214. Then, based on the recognition results, it performs control to change the operation content for each controller in response to the input information.

[0036] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger the closer the distance between the specific object and the imaging unit 202. Furthermore, for example, the control unit 211 renders a virtual object (CG) based on changes in the position and orientation of the HMD 100, thereby rendering CG whose position and orientation change along with the HMD 100 and CG that is not based on changes in the position and orientation of the HMD 100, and composites the CG with the real image. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG object not located in real space appears to be located in real space.

[0037] The control unit 211 also receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the working memory 213.

[0038] Furthermore, in the control unit 211, the communication unit 214 receives information about a change in the position or posture of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes on the combined image a designated position according to the information about the change in the position or posture of the controller 120. Note that the control unit 211 may also superimpose on the combined image a designated position according to the information about the change in the position and posture of the controller 120.

[0039] The nonvolatile memory 212 is an electrically erasable and recordable nonvolatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG to be read from the nonvolatile memory 212 (i.e., the CG to be used to generate a composite image).

[0040] The working memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202 and time-series information on the estimated coordinate positions of each joint point of the hand, as image display memory for the image display unit 203, and as a working area for the control unit 211.

[0041] The wrist joints may also be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100. The control unit 211 may also estimate the position and posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100.

[0042] In addition to the above components of the MR system, for example, the control unit of either the HMD 100 or the PC 110 performs various image processing on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203. However, since this is not the main focus of the present invention, a description thereof will be omitted.

[0043] (Example of Use of Information Processing Device) With reference to FIG. 3 , a scene in which the HMD 100 is used while riding in a moving vehicle such as a car or train will be described. FIG. 3 shows an example in which a user wears the HMD 100 on their head. The outside scenery is visible through a window 301 installed in the car or train, and while the moving vehicle is moving, the scenery flows according to the moving speed. In FIG. 3 , a building 303 is seen as the outside scenery. The camera image captured by the imaging unit 202 shows the flow of the scenery. For example, the user may ride in a windowless moving vehicle such as an open-top car, and the scenery may be captured directly rather than through a window. Furthermore, the camera image captured by the imaging unit 202 shows an interior scene 302, such as the walls and doors inside the moving vehicle.

[0044] (Self-position estimation using camera images) SLAM (Simultaneous Localization and Mapping) is known as a method for estimating self-position using camera images. SLAM is disclosed as a method for detecting the positions of feature points in camera images and determining a self-position in three-dimensional space from the amount of movement of the feature point positions between frames. It is known that various disclosed SLAM algorithms can be used to estimate a self-position in three-dimensional space from these feature points.

[0045] The process of estimating self-position using camera images will be described with reference to Fig. 4. Fig. 4 shows an example of self-position estimation using camera images as seen from a third-person perspective. Fig. 4 shows that the user has moved by the length of the arrow in the direction indicated by movement amount 400 from the state in Fig. 3. From inside the moving object, the building 303 appears to have moved by the length of the arrow indicated by movement amount 404 in the opposite direction to the direction indicated by movement amount 400. Fig. 4 describes a case where the user does not move inside the moving object.

[0046] When a person is on board a moving object, the camera image shows a window 301 installed in the moving object and an interior scene 302 such as walls and doors inside the moving object.

[0047] When a mobile object moves and the user also moves within the mobile object, an amount of movement linked to the movement of the mobile object as well as the movement of the user itself is detected from the external feature point detected in the window 301 installed in the mobile object. In this case, since the user does not move, from the HMD, the building 303 indicated by the dotted line appears to have moved relatively and is at the position of building 403. For example, the feature point 405 of the building 403 obtained at the time of Figure 4 is detected as having moved by an amount of movement 404 from the feature point 305 detected at the time of Figure 3 between the frames of Figure 3 and Figure 4. When self-location estimation is performed using the external feature point detected in the window 301 installed in the mobile object, the self-location of the HMD 100 outside the mobile object can be estimated.

[0048] Next, for feature points of the internal scene 302, such as walls and doors inside the moving body, a movement amount unrelated to the movement of the moving body is detected, and when the user moves inside the moving body, the movement amount of the user himself / herself within the moving body is detected. For example, feature point 406 of the internal scene 302 is detected from the HMD as a feature point that has not moved, since the user does not move. When self-position estimation is performed using the feature points of the internal scene 302, the self-position of the HMD 100 within the moving body can be estimated.

[0049] (Movement Amount Based on Position and Orientation Detected by Orientation Sensor Unit 204) Referring to FIG. 5 , the process of calculating the movement amount based on the position and orientation acquired by the orientation sensor unit 204 will be described. FIG. 5 shows an example of the movement amount detected by the orientation sensor unit 204. The orientation sensor unit 204 can acquire the movement acceleration and facing direction of the HMD 100. The movement amount 501 of the HMD 100 is calculated by accumulating this information acquired periodically. The movement amount 501 of the HMD 100 is a three-dimensional movement amount. The movement amount may be expressed by dividing it into three components based on reference coordinates determined by the HMD 100. Furthermore, not only the movement amount but also the orientation may be expressed by dividing it into three components, namely, roll, pitch, and yaw, which are rotations of a three-dimensional vector.

[0050] (Method of Providing Two Types of Self-Location Estimation Results) A process of providing two types of self-location estimation results will be described with reference to FIGS. 3, 6, and 7. FIG.

[0051] Fig. 6 is a diagram for explaining the process of estimating the self-position when the moving body moves and the user also moves within the moving body, and calculating the amount of movement based on the position and orientation detected by the orientation sensor unit 204. Fig. 6 shows a case where the user moves in the scene shown in Fig. 4. Fig. 6 shows that the user moves within the moving body in the direction indicated by the amount of movement 600 by the length of the arrow, and moves to the position of the HMD 101. In other words, it shows that the HMD 100 moves within the moving body in the direction indicated by the amount of movement 600 by the length of the arrow, and moves to the position of the HMD 101.

[0052] FIG. 7 is a flowchart showing the flow of a process for providing two types of self-location estimation results.

[0053] In step S701, the control unit 211 controls the imaging unit 202 to store the captured image of the user's surroundings in the working memory 213. The image capturing is performed continuously, and the camera images are repeatedly stored in the working memory 213.

[0054] In step S702, the control unit 211 detects feature points in the camera images stored in the working memory 213 and stores position information of the feature points detected for each frame in the working memory 213. The control unit 211 also compares the feature point positions between frames and calculates the amount of movement for each feature point between frames. As shown in FIG. 6 , an exterior feature point 405 detected inside a window 301 installed in a moving object is detected as having moved by an amount of movement 602 due to the movement of the moving object and the user's movement within the moving object. Furthermore, a feature point 406 in the internal scene 302 is detected as having moved by an amount of movement 603 due to the user's movement within the moving object.

[0055] In step S703, the control unit 211 acquires a movement amount 601 of the HMD 100 from the orientation sensor unit 204. The movement amount 601 of the HMD 100 is detected as a combination of the movement of the moving body and the movement of the user within the moving body. The movement amount 601 detected by the orientation sensor unit 204 alone cannot detect how far the moving body has moved or how far the user has moved within the moving body, but rather detects the amount of movement of the user as viewed from outside the moving body.

[0056] In step S704, the control unit 211 converts the movement amount 501 of the HMD 100 into the movement amount on the camera image. For example, a method can be considered in which one of the feature points on the camera image is selected from the position information obtained by adding the movement amount 501 of the HMD 100 to the three-dimensional position information of the user's own position, and the movement amount of the selected feature point is calculated.

[0057] As shown in FIG. 4 , assume that the moving object moves forward and the HMD worn by the user does not move at all inside the moving object, and the orientation sensor unit 204 detects a positive value as the amount of movement. In FIG. 4 , feature points 305 and 405 are detected in area 301, which corresponds to the area outside the moving object, and feature point 406 is detected inside the moving object. In this case, it is assumed that the absolute value of the amount of movement detected by the orientation sensor unit 204 for the feature points in the camera image detected in area 301, which corresponds to the area outside the moving object, is the same, but the amount of movement is detected as a negative value. In this way, the direction of movement of the feature points in the camera image is opposite to the direction of movement of the HMD 100 detected by the orientation sensor unit 204. Therefore, the sign of the amount of movement of the detected feature points is reversed, and the amount of movement 501 of the HMD 100 is converted into the amount of movement in the camera image. Then, the movement amount of the HMD 100 on the camera image obtained by converting the movement amount 501 of the HMD 100 is subtracted from the movement amount of each feature point between frames. Feature point 305 moves to the position of feature point 405, and moves by the distance indicated by arrow 404 on the camera image. The movement amount indicated by arrow 400 is subtracted from the movement amount indicated by arrow 404. If error is not taken into consideration, the absolute value of the movement amount after the subtraction will be zero in this figure. In this way, a feature point whose absolute value of the movement amount after subtraction is less than a predetermined threshold is determined to be in the vehicle exterior area, and is determined to be a first feature point and stored in the working memory 213. Here, when a stationary object in the vehicle exterior area is used as a feature point, it is expected that the movement amount will be zero as a result of the subtraction. However, since error may occur, the predetermined threshold is set to a value that takes error into consideration.

[0058] Furthermore, when the user is not moving inside the vehicle, the movement amount of feature point 406 is zero on the camera image. The movement amount of arrow 400 is subtracted from the feature point with a movement amount of zero, and if error is not taken into consideration, the absolute value of the movement amount after subtraction in this figure is equal to the absolute value of the movement amount of arrow 400. In this way, a feature point whose absolute value of the movement amount after subtraction is equal to or greater than a predetermined threshold is determined to be in the vehicle interior area and stored in working memory 213 as a second feature point. The predetermined threshold for determining the first feature point and the predetermined threshold for determining the second feature point may be the same numerical value or may be set to different numerical values.

[0059] Next, it is expected that not only objects that are actually stationary and appear to be moving relative to the vehicle, such as buildings, but also moving objects, such as automobiles, will be detected in the vehicle exterior area. In this embodiment, among feature points whose absolute values ​​of post-subtraction movement amounts are equal to or greater than a threshold, those that correspond to feature points in the vehicle interior area have absolute values ​​of post-subtraction movement amounts that correspond to the movement amounts of moving objects, ignoring error. Therefore, the absolute values ​​of post-subtraction movement amounts are similar. If a large number of feature points having similar values ​​are detected as a result of the subtraction, they are determined to be a second feature point group and stored in the working memory 213. Furthermore, the determination of similar values ​​may be made based on whether the values ​​are close within a certain error range. Among the feature points determined to be part of the second feature point group, feature points determined to be part of the second feature point group and surrounded by feature points detected as the first feature points in the vehicle exterior area may be determined not to be part of the second feature point group because they are not feature points in the vehicle interior area.

[0060] Note that the sign of the movement amount of the orientation sensor unit 204 may be reversed and converted to match the direction of the movement amount on the camera image, i.e., the movement amount of the feature point detected from the captured image. Note that, in consideration of the fact that the directions are reversed, the movement amount of the orientation sensor unit 204 and the movement amount on the camera image may be added with the signs reversed, instead of subtracting.

[0061] In step S705, the control unit 211 determines whether to use the first feature points or the second feature points to estimate the self-location. If the first feature points, i.e., feature points in the vehicle exterior area, are to be used to estimate the self-location, the process proceeds to step S606. If the second feature points, i.e., feature points in the vehicle interior area, are to be used to estimate the self-location, the process proceeds to step S607. The control unit 211 determines whether to use the first feature points or the second feature points to estimate the self-location, depending on whether the feature points used in the application selected by the user are the first feature points or the second feature points.

[0062] In step S706, the control unit 211 estimates the vehicle's own position based on the first feature points, that is, the feature points in the vehicle exterior area, and ends the process.

[0063] After performing self-location estimation in step S706, the control unit 211 may draw a virtual object based on the estimation result of the self-location estimation.

[0064] After the self-location estimation is performed in step S706, the result may be transmitted to an application selected by the user.

[0065] In step S707, the control unit 211 estimates the self-position based on the second feature points, that is, the feature points of the vehicle interior area, and ends the process.

[0066] After performing self-location estimation in step S707, the control unit 211 may draw a virtual object based on the result of self-location estimation.

[0067] After the self-location estimation is performed in step S707, the result may be transmitted to an application selected by the user.

[0068] Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0069] (Other Embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and a computer (or a control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0070] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0071] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0072] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0073] The disclosure of the present embodiments includes the following configurations, methods, programs, and media: [Configuration 1] An information processing device that communicates with a display device, comprising: a first acquisition means that acquires a detection result of an inertial sensor included in the display device, a second acquisition means that acquires a captured image from an imaging unit included in the display device, a detection means that detects feature points of a subject from the captured image acquired by the second acquisition means, a determination means that determines whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection result of the inertial sensor acquired by the first acquisition means and the feature points detected by the detection means, and an estimation means that estimates a position or orientation of the display device based on the first feature point group or the second feature point group.

[0074] [Configuration 2] The information processing device according to Configuration 1, wherein the information processing device is provided in the display device.

[0075] [Configuration 3] The information processing device according to Configuration 1 or 2, further comprising a third acquisition means for acquiring a movement amount of the feature point, wherein the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on the detection result of the inertial sensor acquired by the second acquisition means and the movement amount of the feature point acquired by the third acquisition means.

[0076] [Configuration 4] The information processing device according to Configuration 3, wherein the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on a first value obtained by subtracting a movement amount based on the detection result of the inertial sensor from a movement amount of the feature point.

[0077] [Configuration 5] The information processing device according to Configuration 4, wherein the determining means determines that the first value corresponds to the first feature point group when the first value is less than a predetermined threshold value.

[0078] [Configuration 6] The information processing device according to Configuration 4 or 5, wherein the determination means determines that the plurality of feature points belong to a second feature point group when the first value is equal to or greater than a predetermined threshold value and the first values ​​of the plurality of feature points are numerical values ​​within a predetermined range.

[0079] [Configuration 7] The information processing device according to Configuration 6, wherein the determination means determines that a feature point does not belong to the second feature point group when the feature point is surrounded by feature points that belong to the first feature point group even if the first value is equal to or greater than the predetermined threshold value.

[0080] [Configuration 8] The information processing device according to any one of Configurations 1 to 7, wherein when estimating the position or orientation of the display device based on the first feature point group, the estimation means estimates the position or orientation of the display device based on the detection result of the inertial sensor acquired by the first acquisition means.

[0081] [Configuration 9] The information processing device according to any one of Configurations 1 to 8, wherein the estimation means estimates the position or orientation of the display device based on the first feature point group, or estimates the position or orientation of the display device based on the second feature point group, depending on an application selected by a user.

[0082] [Configuration 10] The information processing device according to Configuration 9, further comprising a communication means for transmitting the estimation result by the estimation means to the application.

[0083] [Configuration 11] The information processing device according to Configuration 10, wherein the estimation means estimates a position or orientation of the display device based on the first feature point group and a position or orientation of the display device based on the second feature point group, and the communication means transmits the estimation results based on the first feature point group and the second feature point group by the estimation means to the application.

[0084] [Configuration 12] The information processing apparatus according to any one of configurations 1 to 11, further comprising a drawing means for drawing a virtual object based on the estimation result by the estimation means.

[0085] [Control Method] A control method for an information processing device that communicates with a display device, comprising: a first acquisition step of acquiring detection results of an inertial sensor included in the display device; a second acquisition step of acquiring a captured image from an imaging unit included in the display device; a detection step of detecting feature points of a subject from the captured image acquired by the second acquisition step; a determination step of determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection results of the inertial sensor acquired by the first acquisition step and the feature points detected by the detection step; and an estimation step of estimating a position or orientation of the display device based on the first feature point group, or estimating a position or orientation of the display device based on the second feature point group.

[0086] [Program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 12.

[0087] [System] An information processing system comprising: a display device; a first acquisition device that acquires detection results of an inertial sensor included in the display device; a second acquisition device that acquires captured images from an imaging unit included in the display device; a detection device that detects feature points of a subject from the captured images acquired by the second acquisition device; a determination device that determines whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group, based on the detection results of the inertial sensor acquired by the first acquisition device and the feature points detected by the detection device; and an estimation device that estimates a position or orientation of the display device based on the first feature point group, or estimates a position or orientation of the display device based on the second feature point group.

[0088] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0089] This application claims priority based on Japanese Patent Application No. 2023-171953, filed on October 3, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device that communicates with a display device, comprising: a first acquisition means for acquiring detection results of an inertial sensor possessed by the display device; a second acquisition means for acquiring an image from an imaging unit possessed by the display device; a detection means for detecting feature points of a subject from the image acquired by the second acquisition means; a determination means for determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group based on the detection results of the inertial sensor acquired by the first acquisition means and the feature points detected by the detection means; and an estimation means for estimating a position or orientation of the display device based on the first feature point group, or estimating a position or orientation of the display device based on the second feature point group.

2. The information processing device according to claim 1, characterized in that said information processing device is provided in said display device.

3. An information processing device as described in claim 1 or 2, further comprising a third acquisition means for acquiring the amount of movement of the feature point, and the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on the detection result of the inertial sensor acquired by the second acquisition means and the amount of movement of the feature point acquired by the third acquisition means.

4. The information processing device according to claim 3, characterized in that the determination means determines whether the feature point belongs to the first feature point group or the second feature point group based on a first value obtained by subtracting the movement amount based on the detection result of the inertial sensor from the movement amount of the feature point.

5. The information processing device according to claim 4, characterized in that the determination means determines that the first value corresponds to the first feature point group when the first value is less than a predetermined threshold value.

6. The information processing device according to claim 4, characterized in that the determination means determines that the multiple feature points belong to a second group of feature points when the first value is equal to or greater than a predetermined threshold value and the first values ​​of the multiple feature points are numerical values ​​within a predetermined range.

7. The information processing device according to claim 6, characterized in that the determination means determines that a feature point does not belong to the second feature point group if the feature point is surrounded by feature points that belong to the first feature point group, even if the first value is equal to or greater than the predetermined threshold value.

8. The information processing device according to claim 1, characterized in that, when estimating the position or orientation of the display device based on the first group of feature points, the estimation means estimates the position or orientation of the display device based on the detection results of the inertial sensor acquired by the first acquisition means.

9. The information processing device according to claim 1, characterized in that the estimation means estimates the position or orientation of the display device based on the first group of feature points, or estimates the position or orientation of the display device based on the second group of feature points, depending on an application selected by a user.

10. The information processing device according to claim 9, further comprising a communication means for transmitting the estimation result by said estimation means to said application.

11. The information processing device according to claim 10, characterized in that the estimation means estimates a position or orientation of the display device based on the first group of feature points, and estimates a position or orientation of the display device based on the second group of feature points, and the communication means transmits the estimation results based on the first group of feature points and the estimation results based on the second group of feature points by the estimation means to the application.

12. The information processing apparatus according to claim 1, further comprising: drawing means for drawing a virtual object based on a result of estimation by said estimation means.

13. A control method for an information processing device that communicates with a display device, comprising: a first acquisition step of acquiring detection results of an inertial sensor of the display device; a second acquisition step of acquiring an image from an imaging unit of the display device; a detection step of detecting feature points of a subject from the image acquired by the second acquisition step; a determination step of determining whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group based on the detection results of the inertial sensor acquired by the first acquisition step and the feature points detected by the detection step; and an estimation step of estimating a position or orientation of the display device based on the first feature point group, or estimating a position or orientation of the display device based on the second feature point group.

14. A program for causing a computer to function as each of the means of the information processing device according to claim 1.

15. An information processing system comprising: a display device; a first acquisition device that acquires detection results of an inertial sensor possessed by the display device; a second acquisition device that acquires an image from an imaging unit possessed by the display device; a detection device that detects feature points of a subject from the image acquired by the second acquisition device; a determination device that determines whether the feature points belong to a first feature point group or a second feature point group different from the first feature point group based on the detection results of the inertial sensor acquired by the first acquisition device and the feature points detected by the detection device; and an estimation device that estimates a position or orientation of the display device based on the first feature point group, or estimates the position or orientation of the display device based on the second feature point group.

Images