Information processing apparatus, device speed estimation method, and device position estimation method

Abstract

To provide a technique necessary to estimate the position of a device even when the device is imaged no longer.SOLUTION: A sensor data acquisition unit 214 acquires sensor data of the acceleration of a device with a vibrator. A second estimation processing unit 250 estimates the rate of the device on the basis of the sensor data. A vibration determination unit 262 determines whether the vibrator is vibrating on the basis of the sensor data. A resting-state determination unit 264 determines whether the device is at rest on the basis of the sensor data. The second estimation processing unit 250 reduces the estimated rate of the device when it is determined that the vibrator is vibrating and the device is at rest.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a technique for estimating the speed and / or position of a device including an oscillator.

Background Art

[0002] Patent Document 1 discloses an information processing apparatus that identifies representative coordinates of a marker image from an image obtained by photographing a device including a plurality of markers, and derives position information and attitude information of the device using the representative coordinates of the marker image. The information processing apparatus disclosed in Patent Document 1 identifies a first boundary box surrounding a region where pixels having a first luminance or higher are continuous in a photographed image, and also identifies a second boundary box surrounding a region where pixels having a second luminance or higher, which is higher than the first luminance, are continuous within the first boundary box, and derives representative coordinates of the marker image based on the pixels within the first boundary box or the second boundary box.

[0003] Patent Document 2 discloses an input device provided with a plurality of light emitting units and a plurality of operation members. The light emitting units of the input device are photographed by a camera provided in a head mounting device, and the position and attitude of the input device are calculated based on the detected positions of the light emitting units.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, information processing technologies for tracking the position and orientation of a device and reflecting them in a 3D model in a VR space have become widespread. By linking the movement of a player character or a game object in a game space to changes in the position and orientation of a device to be tracked, an intuitive operation by the user is realized.

[0006] The tracking process disclosed in Patent Document 1 is performed when a device is photographed by an imaging device and a marker image is included in the photographed image. Therefore, when the device moves outside the angle of view of the imaging device and the marker image is no longer included in the photographed image, the tracking process disclosed in Patent Document 1 cannot be performed.

[0007] Therefore, an object of the present invention is to provide a technique necessary for estimating the device position even when the device is no longer photographed. The device may be an input device having an operation member, or may simply be a device to be tracked without an operation member.

Means for Solving the Problems

[0008] In order to solve the above problems, an information processing apparatus according to an aspect of the present invention is an information processing apparatus that estimates the speed of a device including a vibrator, and includes a sensor data acquisition unit that acquires sensor data indicating the acceleration of the device, an estimation processing unit that estimates the speed of the device based on the sensor data, a vibration determination unit that determines whether the vibrator is vibrating based on the sensor data, and a stationary determination unit that determines whether the device is stationary based on the sensor data. When it is determined that the vibrator is vibrating and the device is stationary, the estimation processing unit reduces the estimated speed of the device.

[0009] An information processing apparatus according to another aspect of the present invention is an information processing apparatus that estimates the position of a device including a vibrator, and includes a captured image acquisition unit that acquires an image of the device, a first estimation processing unit that estimates the position of the device based on the image of the device, a sensor data acquisition unit that acquires sensor data indicating the acceleration of the device, a second estimation processing unit that estimates the speed and position of the device based on the sensor data, a third estimation processing unit that derives the position of the device based on the position of the device estimated by the first estimation processing unit and the position of the device estimated by the second estimation processing unit, a vibration determination unit that determines whether the vibrator is vibrating based on the sensor data, and a stationary determination unit that determines whether the device is stationary based on the sensor data. The second estimation processing unit reduces the estimated speed of the device when it is determined that the vibrator is vibrating and the device is stationary.

[0010] A device speed estimation method according to still another aspect of the present invention is a method for estimating the speed of a device including a vibrator, and includes a step of acquiring sensor data indicating the acceleration of the device, a step of estimating the speed of the device based on the sensor data, a step of determining whether the vibrator is vibrating based on the sensor data, a step of determining whether the device is stationary based on the sensor data, and a step of reducing the estimated speed of the device when it is determined that the vibrator is vibrating and the device is stationary.

[0011] Another aspect of the device position estimation method of the present invention is a method for estimating the position of a device including a vibrator, the method including: obtaining an image captured by an imaging device; a first estimation step of estimating the position of the device based on the image captured by the imaging device; obtaining sensor data indicating the acceleration of the device; a second estimation step of estimating the speed and position of the device based on the sensor data; a third estimation step of estimating the position of the device based on the position of the device estimated in the first estimation step and the position of the device estimated in the second estimation step; a step of determining whether the vibrator is vibrating based on the sensor data; a step of determining whether the device is stationary based on the sensor data; and a step of reducing the speed of the device estimated in the second estimation step when it is determined that the vibrator is vibrating and the device is stationary.

[0012] In addition, any combination of the above components, and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a computer program, a recording medium in which the computer program is readable, a data structure, etc. are also effective as aspects of the present invention.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0014] FIG. 1 shows a configuration example of the information processing system 1 in the embodiment. The information processing system 1 includes an information processing device 10, a recording device 11, a head-mounted display (HMD) 100, an input device 16 that a user holds and operates with a finger, and an output device 15 that outputs images and sounds. The output device 15 may be a television. The information processing device 10 is connected to an external network 2 such as the Internet via an access point (AP) 17. The AP 17 has the functions of a wireless access point and a router, and the information processing device 10 may be connected to the AP 17 by a cable or may be connected by a known wireless communication protocol.

[0015] The recording device 11 records applications such as system software and game software. The information processing device 10 may download game software from the content server to the recording device 11 via the network 2. The information processing device 10 executes the game software and supplies the game image data and audio data to the HMD 100. The information processing device 10 and the HMD 100 may be connected by a known wireless communication protocol or may be connected by a cable.

[0016] The HMD 100 is a display device that displays an image on a display panel located in front of the user's eyes when the user wears it on the head. The HMD 100 separately displays a left-eye image on the left-eye display panel and a right-eye image on the right-eye display panel. These images constitute parallax images viewed from the left and right viewpoints and realize stereoscopic vision. Since the user views the display panel through the optical lens, the information processing device 10 supplies the parallax image data corrected for optical distortion by the lens to the HMD 100.

[0017] The output device 15 is not necessary for the user wearing the HMD 100, but by providing the output device 15, another user can view the display image of the output device 15. The information processing device 10 may display the same image as the image being viewed by the user wearing the HMD 100 on the output device 15, or may display a different image. For example, in a case where a user wearing an HMD and another user play a game together, a game image from the character viewpoint of the other user may be displayed on the output device 15.

[0018] The information processing apparatus 10 and the input device 16 may be connected by a known wireless communication protocol or may be connected by a cable. The input device 16 includes a plurality of operation members such as operation buttons, and the user operates the operation members with fingers while holding the input device 16. When the information processing apparatus 10 executes a game, the input device 16 is used as a game controller. The input device 16 includes an inertial measurement unit (IMU) including a three-axis acceleration sensor and a three-axis angular velocity sensor, and transmits sensor data to the information processing apparatus 10 at a predetermined period (for example, 800 Hz).

[0019] In the game of the embodiment, not only the operation information of the operation members of the input device 16 but also the speed, position, posture, etc. of the input device 16 are treated as operation information and reflected in the movement of the player character in the virtual three-dimensional space. For example, the operation information of the operation members may be used as information for moving the player character, and the operation information such as the speed, position, and posture of the input device 16 may be used as information for moving the arm of the player character. In a battle scene in the game, the movement of the input device 16 is reflected in the movement of the player character with a weapon, so that the user's intuitive operation is realized and the sense of immersion in the game is enhanced.

[0020] In order to track the position and posture of the input device 16, the input device 16 is provided with a plurality of markers (light emitting units) that can be photographed by the imaging device 14. The information processing apparatus 10 has a function (hereinafter, also referred to as "first estimation function") of analyzing an image obtained by photographing the input device 16 and estimating the position and posture of the input device 16 in the real space.

[0021] The HMD 100 is equipped with a plurality of imaging devices 14. The plurality of imaging devices 14 are attached at different positions on the front surface of the HMD 100 in different postures such that the overall imaging range obtained by adding up the respective imaging ranges includes all of the user's field of view. The imaging device 14 includes an image sensor capable of acquiring images of a plurality of markers of the input device 16. For example, when the marker emits visible light, the imaging device 14 has a visible light sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, which is commonly used in general digital video cameras. When the marker emits non-visible light, the imaging device 14 has a non-visible light sensor. The plurality of imaging devices 14 capture the front of the user at a synchronized timing at a predetermined period (for example, 120 frames / second) and transmit the image data of the captured real space to the information processing device 10.

[0022] The information processing device 10 implements a first estimation function to identify the positions of the plurality of marker images of the input device 16 included in the captured image. Although one input device 16 may be captured by a plurality of imaging devices 14 at the same timing, since the mounting positions and mounting postures of the imaging devices 14 are known, the information processing device 10 may synthesize a plurality of captured images to identify the positions of the marker images.

[0023] The three-dimensional shape of the input device 16 and the position coordinates of the plurality of markers arranged on its surface are known, and the information processing device 10 estimates the position and posture of the input device 16 in the real space based on the position coordinates of the plurality of marker images in the captured image. The position of the input device 16 is estimated as world coordinate values in a three-dimensional space with the reference position as the origin, and the reference position may be the position coordinates (latitude, longitude, altitude (elevation)) set before the start of the game.

[0024] The information processing apparatus 10 of the embodiment has a function (hereinafter, also referred to as "second estimation function") of analyzing sensor data transmitted from the input device 16 and estimating the speed, position, and orientation of the input device 16 in the real space. The information processing apparatus 10 derives the position and orientation of the input device 16 using the estimation result by the first estimation function and the estimation result by the second estimation function. The information processing apparatus 10 of the embodiment uses a state estimation technique using a Kalman filter to integrate the estimation result by the first estimation function and the estimation result by the second estimation function, thereby estimating the state of the input device 16 at the current time with high accuracy.

[0025] FIG. 2 shows an example of the external shape of the HMD 100. The HMD 100 includes an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that wraps around the head when worn by the user to fix the HMD 100 to the head. The wearing band 106 has a material or structure whose length can be adjusted according to the user's head circumference.

[0026] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 100, and includes a display panel facing the eyes during wearing inside. The display panel may be a liquid crystal panel, an organic EL panel, or the like. Further inside the housing 108, a pair of left and right optical lenses are provided, which are located between the display panel and the user's eyes and expand the user's viewing angle. The HMD 100 may further include speakers or earphones at positions corresponding to the user's ears, and may be configured to connect external headphones.

[0027] A plurality of imaging devices 14a, 14b, 14c, and 14d are provided on the front outer surface of the housing 108. With reference to the front direction of the user's face, the imaging device 14a is attached to the upper right corner of the front outer surface such that the camera optical axis faces diagonally upward to the right, the imaging device 14b is attached to the upper left corner of the front outer surface such that the camera optical axis faces diagonally upward to the left, the imaging device 14c is attached to the lower right corner of the front outer surface such that the camera optical axis faces diagonally downward to the right, and the imaging device 14d is attached to the lower left corner of the front outer surface such that the camera optical axis faces diagonally downward to the left. By installing the plurality of imaging devices 14 in this way, the overall imaging range obtained by adding up the respective imaging ranges includes all of the user's field of view. This field of view of the user may be the field of view of the user in a three-dimensional virtual space.

[0028] The HMD 100 transmits the sensor data detected by the IMU (inertial measurement unit) and the image data captured by the imaging device 14 to the information processing device 10, and also receives the game image data and game audio data generated by the information processing device 10.

[0029] FIG. 3 shows the functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data such as image data, audio data, and sensor data, as well as instructions. The storage unit 122 temporarily stores the data and instructions processed by the control unit 120. The IMU 124 acquires sensor data related to the movement of the HMD 100. The IMU 124 may include at least a three-axis acceleration sensor and a three-axis angular velocity sensor. The IMU 124 detects the values (sensor data) of the respective axis components at a predetermined period (for example, 800 Hz).

[0030] The communication control unit 128 transmits the data output from the control unit 120 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 128 also receives data from the information processing device 10 and outputs it to the control unit 120.

[0031] When the control unit 120 receives game image data and game audio data from the information processing apparatus 10, it supplies them to the display panel 130 for display and also supplies them to the audio output unit 132 for audio output. The display panel 130 is composed of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images are displayed on each display panel. Further, the control unit 120 causes the communication control unit 128 to transmit sensor data from the IMU 124, audio data from the microphone 126, and photographed image data from the imaging device 14 to the information processing apparatus 10.

[0032] FIG. 4(a) shows the shape of the left-hand input device 16a. The left-hand input device 16a includes a case body 20, a plurality of operation members 22a, 22b, 22c, 22d (hereinafter referred to as "operation members 22" when not particularly distinguished) that a user operates, and a plurality of markers 30 that emit light outside the case body 20. The marker 30 may have a circular cross-sectional emission portion. The operation member 22 may include an analog stick that tilts and a push-button type button. The case body 20 has a grip portion 21 and a curved portion 23 that connects the case body head and the case body bottom. The user puts the left hand into the curved portion 23 and grips the grip portion 21. While gripping the grip portion 21, the user operates the operation members 22a, 22b, 22c, 22d using the thumb of the left hand.

[0033] FIG. 4(b) shows the shape of the right-hand input device 16b. The right-hand input device 16b includes a case body 20, a plurality of operation members 22e, 22f, 22g, 22h (hereinafter referred to as "operation members 22" when not particularly distinguished) that a user operates, and a plurality of markers 30 that emit light outside the case body 20. The operation member 22 may include an analog stick that tilts and a push-button type button. The case body 20 has a grip portion 21 and a curved portion 23 that connects the case body head and the case body bottom. The user puts the right hand into the curved portion 23 and grips the grip portion 21. While gripping the grip portion 21, the user operates the operation members 22e, 22f, 22g, 22h using the thumb of the right hand.

[0034] FIG. 5 shows the shape of the input device 16b for the right hand. In addition to the operation members 22e, 22f, 22g, and 22h shown in FIG. 4(b), the input device 16b has operation members 22i and 22j. While holding the grip portion 21, the user operates the operation member 22i with the index finger of the right hand and operates the operation member 22j with the middle finger. Hereinafter, when the input device 16a and the input device 16b are not particularly distinguished, they are referred to as "input device 16".

[0035] The operation member 22 provided on the input device 16 may be equipped with a touch sense function that recognizes a finger just by touching it without pressing. Regarding the input device 16b for the right hand, the operation members 22f, 22g, and 22j may be provided with capacitive touch sensors. Although the touch sensor may be mounted on other operation members 22, it is preferably mounted on an operation member 22 that does not contact the placement surface when the input device 16 is placed on a placement surface such as a table.

[0036] The marker 30 is a light emitting portion that emits light outside the case body 20, and includes a resin portion that diffusely emits light from a light source such as an LED (Light Emitting Diode) element to the outside on the surface of the case body 20. The marker 30 is photographed by the imaging device 14 and used for the tracking process of the input device 16.

[0037] The information processing device 10 uses the captured image by the imaging device 14 for the tracking process of the input device 16 and the SLAM (Simultaneous Localization and Mapping) process of the HMD 100. In the embodiment, among the images captured by the imaging device 14 at 120 frames per second, the grayscale image captured at 60 frames per second is used for the tracking process of the input device 16, and another full-color image captured at 60 frames per second may be used for the process of simultaneously performing self-position estimation and environmental map creation of the HMD 100.

[0038] FIG. 6 shows an example of a part of an image of the input device 16. This image is an image of the input device 16b held by the right hand and includes images of a plurality of markers 30 that emit light. In the HMD 100, the communication control unit 128 transmits the image data captured by the imaging device 14 to the information processing device 10 in real time.

[0039] FIG. 7 shows a functional block of the input device 16. The control unit 50 receives the operation information input to the operation member 22. The control unit 50 also receives the sensor data detected by the IMU (inertial measurement unit) 32 and the sensor data detected by the touch sensor 24. As described above, the touch sensor 24 is attached to at least a part of the plurality of operation members 22 and detects a state in which the user's finger is in contact with the operation member 22.

[0040] The IMU 32 includes an acceleration sensor 34 that acquires sensor data regarding the movement of the input device 16 and detects at least three-axis acceleration data, and an angular velocity sensor 36 that detects three-axis angular velocity data. The acceleration sensor 34 and the angular velocity sensor 36 detect the values (sensor data) of each axis component at a predetermined period (for example, 800 Hz). The control unit 50 supplies the received operation information and sensor data to the communication control unit 54, and the communication control unit 54 transmits the operation information and sensor data to the information processing device 10 by wired or wireless communication via a network adapter or an antenna.

[0041] The input device 16 includes a plurality of light sources 58 for lighting a plurality of markers 30. The light source 58 may be an LED element that emits light in a predetermined color. When the communication control unit 54 acquires a light emission instruction from the information processing device 10, the control unit 50 causes the light source 58 to emit light based on the light emission instruction and lights the marker 30. In the example shown in FIG. 7, one light source 58 is provided for one marker 30, but one light source 58 may light a plurality of markers 30.

[0042] The vibrator 52 presents a tactile stimulus for game effects to the user. During the user's game play, the information processing device 10 transmits a vibration instruction to the input device 16 according to the game progress. When the communication control unit 54 acquires the vibration instruction from the information processing device 10, the control unit 50 vibrates the vibrator 52 based on the vibration instruction. By presenting the user with a tactile sensation according to the game progress, the vibrator 52 can enhance the user's immersion in the game. The vibrator 52 may be, for example, a voice coil motor.

[0043] FIG. 8 shows the functional blocks of the information processing device 10. The information processing device 10 includes a processing unit 200 and a communication unit 202, and the processing unit 200 includes an acquisition unit 210, a game execution unit 220, an image signal processing unit 222, a marker information holding unit 224, a state holding unit 226, an estimation processing unit 230, an image signal processing unit 268, and a SLAM processing unit 270. The communication unit 202 receives the operation information and sensor data of the operation member 22 transmitted from the input device 16 and supplies them to the acquisition unit 210. The communication unit 202 also receives the captured image data and sensor data transmitted from the HMD 100 and supplies them to the acquisition unit 210. The acquisition unit 210 includes a captured image acquisition unit 212, a sensor data acquisition unit 214, and an operation information acquisition unit 216.

[0044] The information processing device 10 includes a computer, and by the computer executing a program, various functions shown in FIG. 8 are realized. The computer includes, as hardware, a memory for loading the program, one or more processors for executing the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of a plurality of electronic circuits including semiconductor integrated circuits and LSIs. The plurality of electronic circuits may be mounted on one chip or may be mounted on a plurality of chips. It is understood by those skilled in the art that the functional blocks shown in FIG. 8 are realized by the cooperation of hardware and software, and thus these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

[0045] (SLAM function) The captured image acquisition unit 212 acquires a full-color image for SLAM processing of the HMD 100 and supplies it to the image signal processing unit 268. The image signal processing unit 268 performs image signal processing such as noise reduction and optical correction (shading correction) on the image data, and supplies the image data subjected to the image signal processing to the SLAM processing unit 270.

[0046] The sensor data acquisition unit 214 acquires sensor data transmitted from the HMD 100 and supplies it to the SLAM processing unit 270. The SLAM processing unit 270 simultaneously executes self-position estimation and environmental map creation of the HMD 100 based on the image data supplied from the captured image acquisition unit 212 and the sensor data supplied from the sensor data acquisition unit 214.

[0047] (First estimation function using a captured image) The captured image acquisition unit 212 acquires a grayscale image for tracking processing of the input device 16 and supplies it to the image signal processing unit 222. The image signal processing unit 222 performs image signal processing such as noise reduction and optical correction (shading correction) on the image data, and supplies the image data subjected to the image signal processing to the first estimation processing unit 240.

[0048] The first estimation processing unit 240 includes a marker image coordinate specifying unit 242, a position and orientation derivation unit 244, and a noise derivation unit 246, and realizes a first estimation function for estimating the position and orientation of the input device 16 based on an image of the input device 16 captured by the captured image acquisition unit 212. The first estimation processing unit 240 extracts marker images of a plurality of markers 30 of the input device 16 from the captured image, and estimates the position and orientation of the input device 16 from the arrangement of the extracted plurality of marker images. The first estimation processing unit 240 outputs the estimated position and orientation of the input device 16 together with the variance of the noise (error) thereof to the third estimation processing unit 260.

[0049] (Second estimation function using sensor data) The sensor data acquisition unit 214 acquires the sensor data transmitted from the input device 16 and supplies it to the second estimation processing unit 250. The second estimation processing unit 250 realizes a second estimation function for estimating the speed, position, and orientation of the input device 16 based on the sensor data indicating the acceleration and angular velocity of the input device 16. In the embodiment, the second estimation function is a function for performing the state prediction step in the Kalman filter, and the second estimation processing unit 250 adds the change amount of the state vector obtained by integrating the supplied sensor data to the state vector (speed, position, orientation) at the previous time to estimate the state vector at the current time. The second estimation processing unit 250 outputs the estimated state vector, together with the variance of its noise, to the third estimation processing unit 260. Note that since the change amount obtained by the integration operation accumulates noise over time, the state vector (speed, position, orientation) estimated by the second estimation processing unit 250 tends to deviate from the actual state vector (speed, position, orientation).

[0050] (Integration function of estimation results) The third estimation processing unit 260 accurately derives the speed, position, and orientation of the input device 16 from the position and orientation of the input device 16 estimated by the first estimation processing unit 240 and the state vector (speed, position, orientation) of the input device 16 estimated by the second estimation processing unit 250. The third estimation processing unit 260 may perform the filtering step (correction step) of the UKF (Unscented Kalman Filter). The third estimation processing unit 260 acquires the state vector estimated by the second estimation processing unit 250 as a "prior estimate value", acquires the position and orientation estimated by the first estimation processing unit 240 as an "observed value", calculates the Kalman gain, and obtains a "posterior estimate value" obtained by correcting the "prior estimate value" using the Kalman gain. The "posterior estimate value" accurately represents the speed, position, and orientation of the input device 16, is provided to the game execution unit 220, recorded in the state holding unit 226, and used for estimating the state vector at the next time in the second estimation processing unit 250.

[0051] A method of integrating analysis results using a plurality of sensors such as the imaging device 14 and the IMU 32 to improve accuracy is known as sensor fusion. In sensor fusion, it is necessary to represent the time when data is acquired by each sensor on a common time axis. In the information processing system 1, since the imaging cycle of the imaging device 14 and the sampling cycle of the IMU 32 are different and asynchronous, by accurately managing the imaging time of the image and the detection times of the acceleration and angular velocity, the third estimation processing unit 260 can estimate the position and orientation of the input device 16 with high accuracy.

[0052] The operation information acquisition unit 216 acquires operation information transmitted from the input device 16 and supplies it to the game execution unit 220. The game execution unit 220 advances the game based on the operation information and the position and orientation information of the input device 16 estimated by the estimation processing unit 230.

[0053] FIG. 9 is a flowchart showing the position and orientation estimation process by the first estimation processing unit 240. The captured image acquisition unit 212 acquires image data obtained by capturing the input device 16 (S10) and supplies it to the image signal processing unit 222. The image signal processing unit 222 performs image signal processing such as noise reduction and optical correction on the image data (S12) and supplies the image data subjected to the image signal processing to the marker image coordinate specifying unit 242.

[0054] The marker image coordinate specifying unit 242 specifies the representative coordinates of a plurality of marker images included in the captured image (S14). When the luminance of each pixel of the grayscale image is represented by 8 bits and takes a luminance value of 0 to 255, the marker image is captured as an image having a high luminance as shown in FIG. 6. The marker image coordinate specifying unit 242 may specify a region in which pixels having a luminance value equal to or higher than a predetermined value (for example, 128 luminance values) are continuous from the captured image, calculate the centroid coordinates of the continuous pixel region, and specify the representative coordinates of the marker image.

[0055] The captured image includes not only the marker image but also the images of lighting equipment such as electric lights. Therefore, the marker image coordinate specifying unit 242 examines whether a continuous pixel region having a luminance value equal to or greater than a predetermined value corresponds to the marker image in light of several predetermined criteria. For example, when the continuous pixel region is too large or has an elongated shape, it is certain that the continuous pixel region does not correspond to the marker image. Therefore, the marker image coordinate specifying unit 242 may determine that such a continuous pixel region is not the marker image. The marker image coordinate specifying unit 242 calculates the centroid coordinates of the continuous pixel region that satisfies the predetermined criteria, specifies them as the representative coordinates (marker image coordinates) of the marker image, and stores the specified representative coordinates in a memory (not shown).

[0056] The marker information holding unit 224 holds the three-dimensional coordinates of each marker in the three-dimensional model of the input device 16 in the reference position and reference posture. As a method for estimating the position and posture of the imaging device that captured an object whose three-dimensional shape and size are known from the captured image, a method of solving the PNP (Perspective n-Point) problem is known.

[0057] In the embodiment, the position and orientation deriving unit 244 reads N (N is an integer of 3 or more) marker image coordinates from a memory (not shown), and estimates the position and orientation of the input device 16 from the read N marker image coordinates and the three-dimensional coordinates of the N markers in the three-dimensional model of the input device 16. The position and orientation deriving unit 244 estimates the position and orientation of the imaging device 14 using the following (Equation 1), and derives the position and orientation of the input device 16 in the three-dimensional space based on the estimation result.

Equation

[0058] Here, (u, v) are the marker image coordinates in the captured image, and (X, Y, Z) are the position coordinates in the three-dimensional space of the marker 30 when the three-dimensional model of the input device 16 is in the reference position and reference orientation. Note that the three-dimensional model has exactly the same shape and size as the input device 16, and is a model with the marker placed at the same position. The marker information holding unit 224 holds the three-dimensional coordinates of each marker in the three-dimensional model in the reference position and reference orientation. The position and orientation derivation unit 244 reads out the three-dimensional coordinates of each marker from the marker information holding unit 224 and obtains (X, Y, Z).

[0059] (f x 、f y ) is the focal length of the imaging device 14, (c x 、c y ) is the principal point of the image, and both are the internal parameters of the imaging device 14. The matrix with elements r 11 ~r 33 、t1~t3 is a rotation and translation matrix. In Equation (1), (u, v), (f x 、f y ), (c x 、c y ), and (X, Y, Z) are known, and the position and orientation derivation unit 244 obtains a common rotation and translation matrix for them by solving the equations for N markers 30. In the embodiment, the process of estimating the position and orientation of the input device 16 is implemented by solving the P3P problem.

[0060] Specifically, the position and orientation derivation unit 244 extracts any three marker image coordinates from among the plurality of marker image coordinates specified by the marker image coordinate specifying unit 242. The position and orientation derivation unit 244 reads out the three-dimensional coordinates of the markers in the three-dimensional model from the marker information holding unit 224 and solves the P3P problem using Equation (1). When the position and orientation derivation unit 244 specifies a rotation and translation matrix common to the three extracted marker image coordinates, it calculates the reprojection error using the marker image coordinates of the input device 16 other than the three extracted marker image coordinates.

[0061] The position and orientation derivation unit 244 extracts a predetermined number of combinations of the three marker image coordinates. The position and orientation derivation unit 244 identifies a rotation / translation matrix for each of the extracted combinations of the three marker image coordinates, and calculates each reprojection error. Then, the position and orientation derivation unit 244 identifies the rotation / translation matrix that results in the minimum reprojection error among the predetermined number of reprojection errors, and derives the position and orientation of the input device 16 (S16).

[0062] The noise derivation unit 246 derives the variance of the noise (error) of each of the estimated position and orientation (S18). The variance value of the noise corresponds to the reliability of the estimated position and orientation. If the reliability is high, the variance value is small, and if the reliability is low, the variance value is large. The noise derivation unit 246 may derive the variance of the noise based on the distance between the imaging device 14 and the input device 16, and the position of the marker image within the field of view. For example, when the imaging device 14 and the input device 16 are far apart or extremely close, or when the marker image is located at the edge of the captured image, it becomes difficult to derive the accurate centroid coordinates of the marker image, so the noise variance tends to be derived as large.

[0063] The position and orientation estimation process by the first estimation processing unit 240 is performed at the imaging cycle (60 frames / second) of the tracking image of the input device 16 (N in S20). When the game execution unit 220 ends the game, the position and orientation estimation process by the first estimation processing unit 240 ends (Y in S20).

[0064] FIG. 10 shows the internal configuration of the estimation processing unit 230. At time k, the first estimation processing unit 240 outputs the estimated position and orientation as the "observation value n k " and the variances of the position noise and the orientation noise as the "observation noise R k " to the third estimation processing unit 260. · Observation value n k : Observation vector at time k · Observation noise R k : Error covariance matrix of the observation value at time k

[0065] The second estimation processing unit 250 reads the "state vector m" and "estimation error P" at the previous time (time k-1) from the state holding unit 226, and inputs the "state vector m" and "estimation error P" to the prediction unit. The state variable m in the embodiment includes the speed, position, and attitude of the input device 16, but may further include an acceleration bias and an angular velocity bias. k-1|k-1 " and "estimation error P k-1|k-1 " from the state holding unit 226, and inputs the "state vector m k-1|k-1 " and "estimation error P k-1|k-1 " to the prediction unit. The state variable m in the embodiment includes the speed, position, and attitude of the input device 16, but may further include an acceleration bias and an angular velocity bias. · State vector m k-1|k-1 : The state vector at time k-1 estimated based on the information up to time k-1 · Estimation error P k-1|k-1 : The estimation error covariance matrix of the state at time k-1 estimated based on the information up to time k-1

[0066] Also, the second estimation processing unit 250 acquires the acceleration a and angular velocity ω of the input device 16 from the sensor data acquisition unit 214, and inputs the acceleration a and angular velocity ω as the "process input l" to the prediction unit. k and angular velocity ω k from the sensor data acquisition unit 214, and inputs the acceleration a and angular velocity ω as the "process input l" to the prediction unit. k and angular velocity ω k " to the prediction unit. k " as the "process input l · Acceleration a k : The acceleration at time k · Angular velocity ω k : The angular velocity at time k · Process input l k : The process input vector at time k

[0067] The second estimation processing unit 250 calculates the variances of the acceleration noise and the angular velocity noise from the acceleration a, angular velocity ω, and fixed noise parameters (including axis deviation, scale deviation, value deviation, and bias deviation), and inputs them as the "process noise Q" to the prediction unit. k and angular velocity ω k and fixed noise parameters (including axis deviation, scale deviation, value deviation, and bias deviation), and inputs them as the "process noise Q" to the prediction unit. k " to the prediction unit. · Process noise Q k : The error covariance matrix of the process input at time k

[0068] The prediction unit integrates the acceleration a and angular velocity ω respectively to obtain the "state vector m k and angular velocity ω k respectively, and integrates them to obtain the "state vector mk-1|k-1 Calculate the amount of change from "」 (i.e., the amount of change in velocity, the amount of change in position, and the amount of change in attitude), and perform an operation of adding it to "state vector m k-1|k-1 ". Specifically, the prediction unit integrates the acceleration a k to calculate the amount of change in velocity, and adds the calculated amount of change in velocity to the velocity at time k-1 included in "state vector m k-1|k-1 " to estimate the velocity at time k. The prediction unit integrates the estimated velocity at time k to calculate the amount of change in position, and adds the calculated amount of change in position to the position at time k-1 included in "state vector m k-1|k-1 " to estimate the position at time k. Also, the prediction unit integrates the angular velocity ω k to calculate the amount of change in attitude, and adds the calculated amount of change in attitude to the attitude at time k-1 included in "state vector m k-1|k-1 " to estimate the attitude at time k. In this way, the prediction unit calculates "state vector m k|k-1 ". The prediction unit outputs "state vector m k|k-1 " and "estimated error P k|k-1 " to the third estimation processing unit 260. · State vector m k|k-1 : State vector at time k estimated with information up to time k-1 · Estimated error P k|k-1 : Covariance matrix of the estimated error of the state at time k estimated with information up to time k-1

[0069] The third estimation processing unit 260 acquires "observation value n k " and "observation noise R k " from the first estimation processing unit 240, and acquires "state vector m k|k-1 " and "estimated error P k|k-1 " from the second estimation processing unit 250, and calculates a Kalman gain for correcting "state vector m k|k-1 ". The third estimation processing unit 260 corrects "state vector m k|k-1 " using the Kalman gain, and outputs "state vector m k|k " and "estimated error P k|k ". · State vector m k|k: State vector at time k estimated using information up to time k · Estimation error P k|k : Covariance matrix of the estimation error of the state at time k estimated using information up to time k

[0070] "State vector m k|k " indicates the velocity, position, and orientation estimated with high precision, and is provided to the game execution unit 220 and may be used for game operations. "State vector m k|k " and "estimation error P k|k " are temporarily held in the state holding unit 226 and read out during the estimation process at time k + 1 in the second estimation processing unit 250.

[0071] In the estimation processing unit 230, the estimation processing by the first estimation processing unit 240 is performed at a cycle of 60 Hz, while the estimation processing by the second estimation processing unit 250 is performed at a cycle of 800 Hz. Therefore, after the first estimation processing unit 240 outputs an observation value and before the next observation value is output, the second estimation processing unit 250 sequentially updates the state vector, and during this period, the state vector is not corrected. The estimation processing unit 230 in the embodiment performs a correction step based on the state at time k - 1 immediately before the observation time k, that is, uses the observation value to correct the past state.

[0072] As described above, while the tracking process of the input device 16 is being performed, the estimation processing unit 230 estimates the position and orientation of the input device 16 with high precision. However, when the marker 30 of the input device 16 is no longer photographed by the imaging device 14, the first estimation processing unit 240 cannot execute the position and orientation estimation process shown in FIG. 9.

[0073] FIG. 11 schematically shows the photographable range of the imaging device 14 mounted on the HMD 100. Since the imaging device 14 is attached to the front side of the HMD 100, the space on the front side of the HMD 100 can be photographed, but the space on the rear side cannot be photographed. Therefore, when the user moves the input device 16 behind the face, the input device 16 moves out of the viewing angle of the imaging device 14, and the position and orientation estimation process shown in FIG. 9 cannot be executed.

[0074] FIG. 12 shows a state where the input device 16 is out of the photographable range of the imaging device 14. In this example, since the user has moved the input device 16 to the side of the face, the input device 16 is out of the angle of view of the imaging device 14. At this time, the marker image is not included in the photographed image, and the marker image coordinate specifying unit 242 cannot extract the marker image from the photographed image.

[0075] As described above, the second estimation processing unit 250 integrates the acceleration a at time k to calculate the amount of change in velocity, adds the calculated amount of change in velocity to the estimated velocity at time k−1, and estimates the velocity at time k. Then, the second estimation processing unit 250 integrates the estimated velocity at time k to calculate the amount of change in position, adds the calculated amount of change in position to the estimated position at time k−1, and estimates the position at time k. The estimated velocity and position at time k include integration drift, and the amount of drift accumulates each time the integration operation is repeated. Therefore, in the estimation processing unit 230 of the embodiment, the third estimation processing unit 260 uses the highly reliable "observation value n" at the period (60 Hz) of the tracking process in the first estimation processing unit 240. k to correct the state vector (velocity, position, attitude) output by the second estimation processing unit 250. k

[0076] However, when the first estimation processing unit 240 cannot execute the tracking process, the "observation value n" cannot be input to the third estimation processing unit 260, and the third estimation processing unit 260 cannot correct the state vector output by the second estimation processing unit 250. If the period during which the tracking process cannot be executed becomes long, the estimated state vector deviates greatly from the true state vector. However, in normal game play, the user moves the input device 16 forward within a short time and the photographing of the input device 16 is restarted, so it does not become a very big problem. k

[0077] However, when the vibrator 52 is vibrated by the game effect during the period when the input device 16 is not being photographed, the acceleration sensor 34 detects a large acceleration due to the vibration of the vibrator 52. Therefore, even though the user keeps the input device 16 stationary beside the face, when the acceleration sensor 34 detects a large acceleration, the second estimation processing unit 250 calculates a large speed change amount and position change amount.

[0078] FIG. 13(a) shows the time-series data of the acceleration detected during the vibration of the vibrator 52. When measuring this acceleration, the input device 16 is in a stationary state, and the acceleration sensor 34 measures the acceleration due to the vibration of the vibrator 52 at a period of 800 Hz. FIG. 13(b) shows the change in the speed estimated using the acceleration shown in FIG. 13(a), and FIG. 13(c) shows the change in the position estimated using the speed shown in FIG. 13(b).

[0079] Since the input device 16 is stationary, it is ideal that the speed of the input device 16 is estimated to be zero. However, since the second estimation processing unit 250 estimates the speed based on the acceleration, as shown in FIG. 13(b), the estimated speed drifts, and thus, as shown in FIG. 13(c), the estimated position also drifts. Therefore, when the tracking process of the input device 16 by the first estimation processing unit 240 becomes impossible, the second estimation processing unit 250 of the embodiment performs a damping process for suppressing the drift of the estimated speed and position on the condition that the vibrator 52 is vibrating and the input device 16 is in a stationary state.

[0080] FIG. 14(a) shows the time-series data of the acceleration detected when the vibrator 52 vibrates and the input device 16 is stationary. The vibration determination unit 262 determines whether the vibrator 52 is vibrating based on the sensor data indicating the acceleration. As shown in FIG. 14(a), since the acceleration measured during the vibration of the vibrator 52 varies greatly, the variance of the time-series data of the acceleration shows a large value. Therefore, the vibration determination unit 262 calculates the variance from the time-series data of the acceleration and determines whether the vibrator 52 is vibrating based on the calculated variance.

[0081] The vibration determination unit 262 of the embodiment squares the deviation (the difference between each sampling value and the average sampling value) of 100 acceleration values from the current time t to the past time (t - 99), and calculates the average value of the squares of the 100 deviations as the variance. Note that the number of samples for calculating the variance may be other than 100. The vibration determination unit 262 may calculate the variance at the sampling period of the sensor data and constantly determine whether the vibrator 52 is vibrating.

[0082] If the calculated variance is equal to or greater than a predetermined threshold value Vth, the vibration determination unit 262 determines that the vibrator 52 is vibrating. If the calculated variance is less than the threshold value Vth, the vibration determination unit 262 determines that the vibrator 52 is not vibrating. The vibration determination unit 262 determines that the vibrator 52 is vibrating by confirming that the variance calculated from the time-series data of the acceleration shown in FIG. 14(a) is equal to or greater than the threshold value Vth. In order to prevent the determination result from frequently switching, the vibration determination unit 262 may determine that the vibrator 52 is vibrating when the calculated variance is continuously equal to or greater than the threshold value Vth for a predetermined number of times (N1 times). After determining the vibration, when the calculated variance is continuously less than the threshold value Vth for a predetermined number of times (N2 times), the vibration determination unit 262 may determine that the vibrator 52 is not vibrating. N1 and N2 may be the same number of times, but may also be different numbers of times.

[0083] The stationary determination unit 264 determines whether the input device 16 is stationary based on the sensor data indicating acceleration. The stationary determination unit 264 calculates the slope of the acceleration from the time-series data of the acceleration, and determines whether the input device 16 is stationary based on the calculated slope.

[0084] The stationary determination unit 264 of the embodiment applies the least squares method to 100 acceleration values from the current time t to the past time (t - 99) to obtain a regression line and specifies its slope. Note that the number of samples when calculating the slope may be other than 100. The stationary determination unit 264 may calculate the slope at the sampling period of the sensor data and constantly determine whether the input device 16 is stationary.

[0085] When the absolute value of the calculated slope is equal to or greater than a predetermined threshold value Sth, the stationary determination unit 264 determines that the input device 16 is moving. When the absolute value of the calculated slope is less than the threshold value Sth, the stationary determination unit 264 determines that the input device 16 is stationary. The stationary determination unit 264 determines that the input device 16 is stationary by confirming that the slope calculated from the time-series data of the acceleration shown in FIG. 14(a) is less than the threshold value Sth. In order to prevent the determination result from frequently switching, when the absolute value of the slope to be calculated is continuously less than the threshold value Sth for a predetermined number of times (N3 times), the stationary determination unit 264 may determine that the input device 16 is stationary. After determining the stationary state, when the absolute value of the slope to be calculated is continuously equal to or greater than the threshold value Sth for a predetermined number of times (N4 times), the stationary determination unit 264 may determine that the input device 16 is moving. N3 and N4 may be the same number of times, or may be different numbers of times.

[0086] FIG. 14(b) shows the time-series data of the acceleration detected when the vibrator 52 vibrates and the input device 16 moves. The vibration determination unit 262 determines that the vibrator 52 is vibrating by confirming that the variance calculated from the time-series data of the acceleration shown in FIG. 14(b) is equal to or greater than a threshold value Vth.

[0087] Compared with the time series data of acceleration shown in Fig. 14(a), the time series data of acceleration shown in Fig. 14(b) shows a tendency of gradually increasing. The rest-state determination unit 264 determines that the input device 16 is moving by confirming that the absolute value of the slope calculated from the time series data of acceleration shown in Fig. 14(b) is equal to or greater than the threshold value Sth.

[0088] When the first estimation processing unit 240 is unable to perform tracking processing of the input device 16, if the vibration determination unit 262 determines that the vibrator 52 is vibrating and the stillness determination unit 264 determines that the input device 16 is still, the second estimation processing unit 250 performs attenuation processing to reduce the estimated velocity of the input device 16. In this attenuation processing, the stillness determination unit 264 determines that the "state vector m k|k-1 " with a given damping coefficient c(0 <c<1)を乗算して、推定速度を小さくする演算を行う。実施例において減衰係数cは0.99であるが、それ以外の固定値であってもよい。静止判定部264は、推定速度を小さくする減衰処理を実施することで、速度および位置のドリフトを抑制できる。

[0089] 15 is a diagram for explaining the relationship between acceleration time series data, variance change, and slope change. In the graph showing the variance change, the variance values calculated from the most recent 100 acceleration data are arranged on the time axis, and in the graph showing the slope change, the slopes calculated from the most recent 100 acceleration data are arranged on the time axis.

[0090] 15(a) shows time-series data of acceleration detected when the vibrator 52 is not vibrating and the input device 16 is stationary, and the calculated change in variance and change in tilt. The vibration determination unit 262 determines that the vibrator 52 is not vibrating because the calculated variance is less than the threshold value Vth. The rest determination unit 264 also determines that the input device 16 is stationary because the absolute value of the calculated tilt is less than the threshold value Sth.

[0091] FIG. 15(b) shows the time-series data of the acceleration detected when the vibrator 52 vibrates and the input device 16 is stationary, and the calculated changes in variance and slope. Since the calculated variance is equal to or greater than the threshold value Vth, the vibration determination unit 262 determines that the vibrator 52 is vibrating. Note that the vibration determination unit 262 may determine that the vibrator 52 is not vibrating when the calculated variance is less than the threshold value Vth. However, as described above, after determining the vibration, when the calculated variance becomes less than the threshold value Vth for a predetermined number of times (N2 times) in a row, the vibration determination unit 262 may determine that the vibrator 52 is not vibrating. Since the absolute value of the calculated slope is less than the threshold value Sth, the stationary determination unit 264 determines that the input device 16 is stationary.

[0092] FIG. 15(c) shows the time-series data of the acceleration detected when the vibrator 52 vibrates and the input device 16 moves, and the calculated changes in variance and slope. The acceleration shown in FIG. 15(c) is the acceleration detected when the user shakes the input device 16. Since the calculated variance is equal to or greater than the threshold value Vth, the vibration determination unit 262 determines that the vibrator 52 is vibrating. Also, since the absolute value of the calculated slope is equal to or greater than the threshold value Sth, the stationary determination unit 264 determines that the input device 16 is moving. Note that the vibration determination unit 262 may determine that the input device 16 is stationary when the absolute value of the calculated slope is less than the threshold value Sth. However, after determining that the input device 16 is moving, when the absolute value of the calculated slope becomes less than the threshold value Sth for a predetermined number of times in a row, the stationary determination unit 264 may determine that the input device 16 is stationary.

[0093] As shown in FIG. 15(b), when the vibration determination unit 262 determines that the vibrator 52 is vibrating and at the same time the stationary determination unit 264 determines that the input device 16 is stationary, the second estimation processing unit 250 performs a damping process of reducing the estimated speed of the input device 16. If the state where the vibrator 52 vibrates and the input device 16 is stationary continues, the second estimation processing unit 250 continues to multiply the estimated speed by the coefficient c at a period of 800 Hz, so the estimated speed gradually approaches zero.

[0094] FIG. 16(a) shows the time series data of the acceleration detected when the vibrator 52 vibrates and the input device 16 is stationary. FIG. 16(b) shows the change in the estimated velocity after the attenuation process. The estimated velocity decays with the passage of time due to the attenuation process and approaches zero. FIG. 16(c) shows the change in the position estimated based on the estimated velocity. Since the second estimation processing unit 250 estimates the position based on the attenuated estimated velocity, the estimated position does not drift. Thus, when the tracking process by the first estimation processing unit 240 cannot be performed and the vibrator 52 is vibrating and the input device 16 is in a stationary state, the second estimation processing unit 250 can estimate a stable position by attenuating the estimated velocity. Note that the second estimation processing unit 250 performs the attenuation process when the tracking process is not performed, and there is no need to perform the attenuation process if the tracking process is being performed.

[0095] When the input device 16 moves into the viewing angle of the imaging device 14 and the marker image is included in the captured image, the marker image coordinate specifying unit 242 extracts the marker image from the captured image, and the position and orientation derivation unit 244 resumes the position and orientation estimation process based on the marker image. After the resumption of the tracking process by the first estimation processing unit 240, the second estimation processing unit 250 may stop performing the attenuation process.

[0096] As described above, the present invention has been described based on the embodiments. It is understood by those skilled in the art that the above embodiments are examples, and various modifications are possible for each component and each combination of processing processes, and such modifications are also within the scope of the present invention. In the embodiment, the estimation process is performed by the information processing device 10, but the function of the information processing device 10 may be provided in the HMD 100 and the HMD 100 may perform the estimation process. That is, the HMD 100 may be the information processing device 10.

[0097] In the embodiment, the arrangement of the plurality of markers 30 in the input device 16 provided with the operation member 22 has been described. However, the device to be tracked does not necessarily have to be provided with the operation member 22. In the embodiment, the imaging device 14 is attached to the HMD 100. However, the imaging device 14 only needs to be able to capture a marker image and may be attached at a position other than the HMD 100.

Description of Reference Numerals

[0098] 1... Information processing system, 10... Information processing device, 14... Imaging device, 16, 16a, 16b... Input device, 20... Case body, 21... Gripping part, 22... Operation member, 23... Curved part, 24... Touch sensor, 30... Marker, 32... IMU, 34... Acceleration sensor, 36... Angular velocity sensor, 50... Control part, 52... Vibrator, 54... Communication control part, 58... Light source, 100... HMD, 102... Output mechanism part, 104... Mounting mechanism part, 106... Mounting band, 108... Housing, 120... Control part, 122... Storage part, 124... IMU, 126... Microphone, 128... Communication control part, 130... Display panel, 130a... Left-eye display panel, 130b... Right-eye display panel, 132... Audio output part, 200... Processing part, 202... Communication part, 210... Acquisition part, 212... Captured image acquisition part, 214... Sensor data acquisition part, 216... Operation information acquisition part, 220... Game execution part, 222... Image signal processing part, 224... Marker information holding part, 226... State holding part, 230... Estimation processing part, 240... First estimation processing part, 242... Marker image coordinate specifying part, 244... Position and orientation derivation part, 246... Noise derivation part, 250... Second estimation processing part, 260... Third estimation processing part, 262... Vibration determination part, 264... Stationary determination part, 268... Image signal processing part, 270... SLAM processing part.

Claims

1. An information processing apparatus for estimating the speed of a device including an oscillator, comprising: a sensor data acquisition unit that acquires sensor data indicating the acceleration of the device; an estimation processing unit that estimates the speed of the device based on the sensor data; a vibration determination unit that determines whether the oscillator is vibrating based on the sensor data; a stationary determination unit that determines whether the device is stationary based on the sensor data, wherein when it is determined that the oscillator is vibrating and the device is stationary, the estimation processing unit reduces the estimated speed of the device. An information processing apparatus characterized by the above.

2. The vibration determination unit determines whether the oscillator is vibrating based on the variance calculated from the time series data of the acceleration. The information processing apparatus according to claim 1, characterized by the above.

3. The stationary determination unit determines whether the device is stationary based on the slope of the acceleration calculated from the time series data of the acceleration. The information processing apparatus according to claim 1 or 2, characterized by the above.

4. When it is determined that the oscillator is vibrating and the device is stationary, the estimation processing unit multiplies the estimated speed of the device by a predetermined attenuation coefficient c (0 < c < 1). The information processing apparatus according to any one of claims 1 to 3, characterized by the above.

5. The estimation processing unit estimates the speed of the device at the period when the sensor data acquisition unit acquires the sensor data. The information processing apparatus according to any one of claims 1 to 4, characterized by the above.

6. An information processing apparatus for estimating the position of a device including an oscillator, comprising: a captured image acquisition unit that acquires an image of the device; a first estimation processing unit that estimates the position of the device based on the image of the device; a sensor data acquisition unit that acquires sensor data indicating the acceleration of the device; a second estimation processing unit that estimates the speed and position of the device based on the sensor data; a third estimation processing unit that derives the position of the device based on the position of the device estimated by the first estimation processing unit and the position of the device estimated by the second estimation processing unit; a vibration determination unit that determines whether the oscillator is vibrating based on the sensor data. A stationary determination unit that determines whether the device is stationary based on the sensor data. When it is determined that the vibrator is vibrating and the device is stationary, the second estimation processing unit reduces the estimated speed of the device. An information processing apparatus characterized by the above.

7. When the device is included in the captured image, the second estimation processing unit does not reduce the estimated speed of the device, and when the device is no longer included in the captured image, the second estimation processing unit reduces the estimated speed of the device. The information processing apparatus according to claim 6, characterized by the above.

8. A method for estimating the speed of a device including a vibrator, comprising: Obtaining sensor data indicating the acceleration of the device; Estimating the speed of the device based on the sensor data; Determining whether the vibrator is vibrating based on the sensor data; Determining whether the device is stationary based on the sensor data; Reducing the estimated speed of the device when it is determined that the vibrator is vibrating and the device is stationary. A device speed estimation method characterized by the above.

9. A method for estimating the position of a device including a vibrator, comprising: Obtaining an image captured by an imaging device; A first estimation step of estimating the position of the device based on an image captured by the imaging device of the device; Obtaining sensor data indicating the acceleration of the device; A second estimation step of estimating the speed and position of the device based on the sensor data; A third estimation step of estimating the position of the device based on the position of the device estimated in the first estimation step and the position of the device estimated in the second estimation step; Determining whether the vibrator is vibrating based on the sensor data; Determining whether the device is stationary based on the sensor data; Reducing the speed of the device estimated in the second estimation step when it is determined that the vibrator is vibrating and the device is stationary. A device position estimation method characterized by the above.

Images