Information processing device and device position estimation method

Abstract

To provide a technique for estimating a device position when a device is not captured anymore.SOLUTION: A captured image acquisition section 212 acquires an image obtained by capturing a device. A sensor data acquisition section 214 acquires sensor data indicating angular speed of the device. A position posture derivation section 244 derives a position of the device in a three-dimensional space from position coordinates of the device in the captured image when the device is included in the captured image. A portion position estimation section 246 estimates a position of a predetermined portion in a body of a user on the basis of the estimated position of the device. When the device is not included in the captured image, the position posture derivation section 244 derives, as the position of the device, a position that is rotated by a rotation amount corresponding to the sensor data using the position of the portion estimated by the position posture derivation section 244 as a rotation center.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a technology for estimating the location of a device carried by a user. [Background technology]

[0002] Patent Document 1 discloses an information processing device that identifies representative coordinates of marker images from a captured image of a device equipped with multiple markers and derives position information and orientation information of the device using the representative coordinates of the marker images. The information processing device disclosed in Patent Document 1 identifies a first bounding box that encloses an area in the captured image where pixels with a first brightness or higher are consecutive, and identifies a second bounding box that encloses an area within the first bounding box where pixels with a second brightness or higher that is higher than the first brightness are consecutive, and derives representative coordinates of the marker images based on the pixels in the first bounding box or the second bounding box.

[0003] Patent Document 2 discloses an input device equipped with multiple light-emitting units and multiple operating members. The light-emitting units of the input device are photographed by a camera mounted on a head-mounted device, and the position and orientation of the input device are calculated based on the detected positions of the light-emitting units. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-181322 [Patent Document 2] International Publication No. 2021 / 240930 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, information processing technology that tracks the position and orientation of a device and reflects it in a 3D model in a VR space has become widespread. By linking the movement of a player character or game object in a game space to changes in the position and orientation of the device being tracked, intuitive operation by the user is realized.

[0006] The device position estimation process disclosed in Patent Document 1 is performed when a device is photographed by an imaging device and a marker image is included in the photographed image. Therefore, if the device moves out of the field of view of the imaging device and the marker image is no longer included in the photographed image, the device position estimation process disclosed in Patent Document 1 cannot be performed.

[0007] Therefore, an object of the present invention is to provide a technology for estimating the position of a device even when the device is no longer photographed. The device may be an input device having an operation member, or may be a device that does not have an operation member and is simply a target for tracking. [Means for solving the problem]

[0008] In order to solve the above problem, an information processing device according to one aspect of the present invention is an information processing device that estimates the position of a device held by a user, and includes: a captured image acquisition unit that acquires a captured image of the device; an estimation processing unit that estimates the position of the device based on the captured image of the device; and a sensor data acquisition unit that acquires sensor data indicating the angular velocity of the device. The estimation processing unit includes: a derivation unit that, when the device is included in the captured image, derives the position of the device in three-dimensional space from the position coordinates of the device in the captured image; and a body part position estimation unit that estimates the position of a specific body part on the user's body based on the estimated device position. When the device is no longer included in the captured image, the derivation unit derives the position of the device by rotating the position of the body part estimated by the body part position estimation unit by an amount of rotation corresponding to the sensor data as the position of the device.

[0009] Another aspect of the present invention provides an information processing device that estimates the position of a device held by a user, and includes a captured image acquisition unit that acquires a captured image of the device, a first estimation processing unit that estimates the position of the device based on the captured image, a sensor data acquisition unit that acquires sensor data indicating the acceleration and / or angular velocity of the device, a second estimation processing unit that estimates the position of the device based on the sensor data, and a third estimation processing unit that derives the position of the device based on the position of the device estimated by the first estimation processing unit and the position of the device estimated by the second estimation processing unit. The first estimation processing unit includes a derivation unit that, when the device is included in the captured image, derives the position of the device in three-dimensional space from the position coordinates of the device in the captured image, and a body part position estimation unit that estimates the position of a specific body part on the user's body based on the estimated position of the device. When the device is no longer included in the captured image, the derivation unit derives the position of the device by rotating the position of the body part estimated by the body part position estimation unit by an amount of rotation corresponding to the sensor data around the position of the body part estimated by the body part position estimation unit as the rotation center.

[0010] Another aspect of the device position estimation method of the present invention is a method for estimating the position of a device held by a user, and includes the steps of acquiring an image captured by an imaging device, estimating the position of the device based on the image of the device captured by the imaging device, estimating the position of a specified part of the user's body based on the estimated position of the device, acquiring sensor data indicating the angular velocity of the device, and, when the device is no longer included in the image captured by the imaging device, deriving the position of the device as a position rotated by an amount of rotation corresponding to the sensor data around the position of the estimated part as the center of rotation.

[0011] Another corresponding device position estimation method of the present invention is a method for estimating the position of a device held by a user, and includes the steps of: acquiring an image captured by an imaging device; a first estimation step of estimating the position of the device based on the image of the device captured by the imaging device; acquiring sensor data indicating the acceleration and / or angular velocity of the device; a second estimation step of estimating the position of the device based on the sensor data; and a third estimation step of estimating the position of the device based on the position of the device estimated in the first estimation step and the position of the device estimated in the second estimation step. The first estimation step includes the steps of: when the device is included in the captured image, estimating the position of the device in three-dimensional space from the position coordinates of the device in the captured image; estimating the position of a predetermined part of the user's body based on the estimated position of the device; and when the device is no longer included in the captured image, deriving the position of the device by rotating the estimated position of the part of the body by an amount of rotation corresponding to the sensor data around the position of the estimated part as the rotation center.

[0012] In addition, any combination of the above components, and conversions of the present invention between methods, devices, systems, computer programs, recording media on which computer programs are readably recorded, data structures, etc. are also valid aspects of the present invention. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating examples of the external shape of an HMD. [Figure 3] FIG. 2 is a diagram illustrating functional blocks of an HMD. [Figure 4] FIG. 2 is a diagram illustrating the shape of an input device. [Figure 5] FIG. 2 is a diagram illustrating the shape of an input device. [Figure 6] FIG. 10 is a diagram showing an example of a portion of an image captured using an input device. [Figure 7]FIG. 2 is a diagram illustrating functional blocks of an input device. [Figure 8] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Figure 9] 10 is a flowchart illustrating a position and orientation estimation process. [Figure 10] FIG. 2 is a diagram illustrating the internal configuration of an estimation processing unit. [Figure 11] FIG. 2 is a diagram schematically illustrating a photographable range of an imaging device. [Figure 12] FIG. 10 is a diagram illustrating an example of estimated positions of an HMD and an input device. [Figure 13] FIG. 10 is a diagram showing a state in which the input device is out of the captureable range. [Figure 14] FIG. 10 is a diagram for explaining a process of estimating a device position using an elbow position as a base point. DETAILED DESCRIPTION OF THE INVENTION

[0014] 1 shows an example of the configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes an information processing device 10, a recording device 11, a head-mounted display (HMD) 100, an input device 16 that a user holds and operates with their fingers, and an output device 15 that outputs images and sounds. The output device 15 may be a television. The information processing device 10 is connected to an external network 2, such as the Internet, via an access point (AP) 17. The AP 17 has the functions of a wireless access point and a router, and the information processing device 10 may be connected to the AP 17 by a cable or by a known wireless communication protocol.

[0015] The recording device 11 records applications such as system software and game software. The information processing device 10 may download game software to the recording device 11 from a content server via the network 2. The information processing device 10 executes the game software and supplies game image data and audio data to the HMD 100. The information processing device 10 and HMD 100 may be connected using a known wireless communication protocol, or may be connected by cable.

[0016] The HMD 100 is a display device worn by a user on the head, which displays images on display panels positioned in front of the user's eyes. The HMD 100 separately displays an image for the left eye on the left-eye display panel and an image for the right eye on the right-eye display panel. These images form parallax images seen from left and right viewpoints, realizing stereoscopic vision. Because the user views the display panel through optical lenses, the information processing device 10 supplies the HMD 100 with parallax image data that has been corrected for optical distortion caused by the lenses.

[0017] The output device 15 is not necessary for a user wearing the HMD 100, but providing the output device 15 allows another user to view the image displayed on the output device 15. The information processing device 10 may display on the output device 15 the same image as the image viewed by the user wearing the HMD 100, or may display a different image. For example, when a user wearing an HMD and another user play a game together, the output device 15 may display a game image from the viewpoint of the character of the other user.

[0018] The information processing device 10 and the input device 16 may be connected via a known wireless communication protocol or via a cable. The input device 16 has a plurality of operation members such as operation buttons, and a user operates the operation members with their fingers while holding the input device 16. When the information processing device 10 executes a game, the input device 16 is used as a game controller. The input device 16 has an inertial measurement unit (IMU) including a three-axis acceleration sensor and a three-axis angular velocity sensor, and transmits sensor data to the information processing device 10 at a predetermined cycle (for example, 800 Hz).

[0019] In the game of the embodiment, not only operation information of the operation members of the input device 16 but also the position, speed, and attitude of the input device 16 are treated as operation information and reflected in the movement of the player character in the virtual three-dimensional space. For example, operation information of the operation members may be used as information for moving the player character, and operation information such as the position, speed, and attitude of the input device 16 may be used as information for moving the arms of the player character. In battle scenes in the game, the movement of the input device 16 is reflected in the movement of the player character holding a weapon, thereby realizing intuitive operation by the user and increasing the sense of immersion in the game.

[0020] In order to track the position and orientation of the input device 16, the input device 16 is provided with a plurality of markers (light emitting units) that can be photographed by the imaging device 14. The information processing device 10 has a function (hereinafter also referred to as a "first estimation function") of analyzing a photographed image of the input device 16 and estimating the position and orientation of the input device 16 in real space.

[0021] The HMD 100 is equipped with multiple image capture devices 14. The multiple image capture devices 14 are attached to the front of the HMD 100 at different positions and in different orientations so that the combined overall capture range of each captures the entire user's field of view. The image capture devices 14 are equipped with image sensors capable of acquiring images of multiple markers on the input device 16. For example, if the markers emit visible light, the image capture devices 14 have visible light sensors used in general digital video cameras, such as CCD (Charge Coupled Device) sensors or CMOS (Complementary Metal Oxide Semiconductor) sensors. If the markers emit invisible light, the image capture devices 14 have invisible light sensors. The multiple image capture devices 14 capture images of the area in front of the user at a predetermined cycle (e.g., 120 frames per second) in a synchronized manner, and transmit image data of the captured real space to the information processing device 10.

[0022] The information processing device 10 performs the first estimation function to identify the positions of multiple marker images of the input device 16 included in the captured image. Note that although one input device 16 may be captured by multiple image capture devices 14 at the same time, the information processing device 10 may combine the multiple captured images to identify the positions of the marker images because the mounting positions and mounting orientations of the image capture devices 14 are known.

[0023] The three-dimensional shape of the input device 16 and the position coordinates of multiple markers arranged on its surface are known, and the information processing device 10 estimates the position and orientation of the input device 16 in real space based on the position coordinates of multiple marker images in the captured image. The position of the input device 16 is estimated as coordinate values in world coordinates in three-dimensional space with a reference position as the origin, and the reference position may be position coordinates (latitude, longitude, altitude (elevation)) set before the start of the game.

[0024] The information processing device 10 of the embodiment has a function (hereinafter also referred to as a "second estimation function") of analyzing sensor data transmitted from the input device 16 and estimating the position and orientation of the input device 16 in real space. The information processing device 10 derives the position and orientation of the input device 16 using the estimation results from the first estimation function and the second estimation function. The information processing device 10 of the embodiment utilizes a state estimation technique using a Kalman filter to integrate the estimation results from the first estimation function and the estimation results from the second estimation function, thereby estimating the state of the input device 16 at the current time with high accuracy.

[0025] 2 shows an example of the external shape of the HMD 100. The HMD 100 is composed of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that, when worn by the user, goes around the head and secures the HMD 100 to the head. The wearing band 106 is made of a material or has a structure that allows its length to be adjusted to fit the user's head circumference.

[0026] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the HMD 100 is worn by the user, and includes a display panel inside that faces the eyes when worn. The display panel may be a liquid crystal panel, an organic EL panel, or the like. The housing 108 also includes a pair of optical lenses, one on each side, that are positioned between the display panel and the user's eyes and expand the user's field of view. The HMD 100 may also include speakers or earphones at positions corresponding to the user's ears, and may be configured to allow external headphones to be connected.

[0027] The front outer surface of the housing 108 is provided with multiple image capture devices 14a, 14b, 14c, and 14d. With respect to the direction of the user's face, image capture device 14a is attached to the upper right corner of the front outer surface so that its optical axis faces diagonally upward to the right. Image capture device 14b is attached to the upper left corner of the front outer surface so that its optical axis faces diagonally upward to the left. Image capture device 14c is attached to the lower right corner of the front outer surface so that its optical axis faces diagonally downward to the right. Image capture device 14d is attached to the lower left corner of the front outer surface so that its optical axis faces diagonally downward to the left. By installing multiple image capture devices 14 in this manner, the total image capture range obtained by adding up the image capture ranges of each capture device encompasses the entire field of view of the user. This user's field of view may be the user's field of view in a three-dimensional virtual space.

[0028] The HMD 100 transmits sensor data detected by an IMU (Inertial Measurement Unit) and image data captured by an imaging device 14 to the information processing device 10, and also receives game image data and game audio data generated by the information processing device 10.

[0029] FIG. 3 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data such as image data, audio data, and sensor data, as well as commands. The storage unit 122 temporarily stores the data and commands processed by the control unit 120. The IMU 124 acquires sensor data related to the movement of the HMD 100. The IMU 124 may include at least a three-axis acceleration sensor and a three-axis angular velocity sensor. The IMU 124 detects the values of each axial component (sensor data) at a predetermined cycle (e.g., 800 Hz).

[0030] The communication control unit 128 transmits data output from the control unit 120 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 128 also receives data from the information processing device 10 and outputs it to the control unit 120.

[0031] When the control unit 120 receives game image data and game audio data from the information processing device 10, it supplies the data to the display panel 130 for display and to the audio output unit 132 for audio output. The display panel 130 is composed of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images is displayed on each display panel. The control unit 120 also causes the communication control unit 128 to transmit sensor data from the IMU 124, audio data from the microphone 126, and captured image data from the imaging device 14 to the information processing device 10.

[0032] FIG. 4(a) shows the shape of an input device 16a for a left hand. The input device 16a for a left hand includes a case body 20, multiple operation members 22a, 22b, 22c, and 22d (hereinafter referred to as "operation members 22" unless otherwise specified) operated by a user, and multiple markers 30 that emit light to the outside of the case body 20. The markers 30 may have an emission portion with a circular cross section. The operation members 22 may include an analog stick that is operated by tilting, a push-button, or the like. The case body 20 includes a grip portion 21 and a curved portion 23 that connects the top and bottom of the case body. The user inserts their left hand into the curved portion 23 to grip the grip portion 21. While gripping the grip portion 21, the user operates the operation members 22a, 22b, 22c, and 22d using the thumb of their left hand.

[0033] FIG. 4(b) shows the shape of a right-handed input device 16b. The right-handed input device 16b includes a case body 20, multiple operation members 22e, 22f, 22g, and 22h (hereinafter referred to as "operation members 22" unless otherwise specified) operated by the user, and multiple markers 30 that emit light to the outside of the case body 20. The operation members 22 may include an analog stick that is operated by tilting, a push-button, or the like. The case body 20 has a grip portion 21 and a curved portion 23 that connects the top and bottom of the case body. The user inserts their right hand into the curved portion 23 to grip the grip portion 21. While gripping the grip portion 21, the user operates the operation members 22e, 22f, 22g, and 22h with the thumb of their right hand.

[0034] FIG. 5 shows the shape of a right-handed input device 16b. In addition to the operation members 22e, 22f, 22g, and 22h shown in FIG. 4(b), the input device 16b has operation members 22i and 22j. While holding the grip portion 21, the user operates operation member 22i with the index finger of the right hand and operation member 22j with the middle finger. Hereinafter, when there is no particular distinction between the input device 16a and the input device 16b, they will be referred to as "input device 16."

[0035] The operation members 22 provided on the input device 16 may be equipped with a touch sensing function that recognizes a finger simply by touching it, without the need for pressing. With regard to the right-hand input device 16b, the operation members 22f, 22g, and 22j may be equipped with a capacitance-type touch sensor. While the touch sensor may be mounted on other operation members 22, it is preferable that the touch sensor be mounted on an operation member 22 that does not come into contact with the surface on which the input device 16 is placed, such as a table.

[0036] The marker 30 is a light-emitting portion that emits light to the outside of the case body 20, and includes a resin portion that diffuses and emits light from a light source such as an LED (Light Emitting Diode) element to the outside on the surface of the case body 20. The marker 30 is photographed by the imaging device 14 and used for tracking processing of the input device 16.

[0037] The information processing device 10 uses images captured by the imaging device 14 for tracking processing of the input device 16 and for SLAM (Simultaneous Localization and Mapping) processing of the HMD 100. In an embodiment, of the images captured by the imaging device 14 at 120 frames per second, grayscale images captured at 60 frames per second may be used for tracking processing of the input device 16, and other full-color images captured at 60 frames per second may be used for processing of the HMD 100 to simultaneously estimate its own position and create an environmental map.

[0038] 6 shows an example of a portion of an image captured of the input device 16. This image is an image captured of the input device 16b held in the right hand, and includes images of multiple markers 30 that emit light. In the HMD 100, the communication control unit 128 transmits image data captured by the imaging device 14 to the information processing device 10 in real time.

[0039] 7 shows functional blocks of the input device 16. The control unit 50 receives operation information input to the operation members 22. The control unit 50 also receives sensor data detected by the IMU (inertial measurement unit) 32 and sensor data detected by the touch sensor 24. As described above, the touch sensor 24 is attached to at least some of the multiple operation members 22, and detects a state in which the user's finger is in contact with the operation member 22.

[0040] The IMU 32 acquires sensor data related to the movement of the input device 16 and includes an acceleration sensor 34 that detects acceleration data along at least three axes and an angular velocity sensor 36 that detects angular velocity data along three axes. The acceleration sensor 34 and the angular velocity sensor 36 detect the values (sensor data) of each axial component at a predetermined cycle (e.g., 800 Hz). The control unit 50 supplies the received operation information and sensor data to the communication control unit 54, which then transmits the operation information and sensor data to the information processing device 10 via a network adapter or an antenna by wired or wireless communication.

[0041] The input device 16 includes a plurality of light sources 58 for lighting up a plurality of markers 30. The light sources 58 may be LED elements that emit light in a predetermined color. When the communication control unit 54 receives a light-emitting instruction from the information processing device 10, the control unit 50 causes the light sources 58 to emit light based on the light-emitting instruction, thereby lighting up the markers 30. Note that in the example shown in FIG. 7, one light source 58 is provided for one marker 30, but one light source 58 may light up a plurality of markers 30.

[0042] 8 shows functional blocks of the information processing device 10. The information processing device 10 includes a processing unit 200 and a communication unit 202. The processing unit 200 includes an acquisition unit 210, a game execution unit 220, an image signal processing unit 222, a marker information storage unit 224, a state storage unit 226, an estimation processing unit 230, an image signal processing unit 268, and a SLAM processing unit 270. The communication unit 202 receives operation information and sensor data of the operation member 22 transmitted from the input device 16 and supplies the same to the acquisition unit 210. The communication unit 202 also receives captured image data and sensor data transmitted from the HMD 100 and supplies the same to the acquisition unit 210. The acquisition unit 210 includes a captured image acquisition unit 212, a sensor data acquisition unit 214, and an operation information acquisition unit 216.

[0043] The information processing device 10 includes a computer, which executes a program to realize various functions shown in FIG. 8. The computer includes hardware such as a memory into which the program is loaded, one or more processors that execute the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including semiconductor integrated circuits and LSIs, and the multiple electronic circuits may be mounted on a single chip or multiple chips. The functional blocks shown in FIG. 8 are realized by cooperation between hardware and software. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using hardware alone, software alone, or a combination thereof.

[0044] (SLAM function) The captured image acquisition unit 212 acquires a full-color image for SLAM processing of the HMD 100 and supplies it to an image signal processing unit 268. The image signal processing unit 268 performs image signal processing such as noise reduction and optical correction (shading correction) on the image data, and supplies the image data that has undergone image signal processing to a SLAM processing unit 270.

[0045] The sensor data acquisition unit 214 acquires sensor data transmitted from the HMD 100 and supplies the data to the SLAM processing unit 270. The SLAM processing unit 270 simultaneously performs self-position estimation of the HMD 100 and environmental map creation based on the image data supplied from the captured image acquisition unit 212 and the sensor data supplied from the sensor data acquisition unit 214.

[0046] (First estimation function using captured images) The captured image acquisition unit 212 acquires a grayscale image for tracking processing of the input device 16 and supplies it to the image signal processing unit 222. The image signal processing unit 222 performs image signal processing such as noise reduction and optical correction (shading correction) on the image data, and supplies the image data that has undergone the image signal processing to the first estimation processing unit 240.

[0047] The first estimation processing unit 240 includes a marker image coordinate identification unit 242, a position and orientation derivation unit 244, a body part position estimation unit 246, and a noise derivation unit 248, and realizes a first estimation function of estimating the position and orientation of the input device 16 based on a captured image of the input device 16. The first estimation processing unit 240 extracts marker images of multiple markers 30 of the input device 16 from the captured image, and estimates the position and orientation of the input device 16 from the arrangement of the extracted multiple marker images. The first estimation processing unit 240 outputs the estimated position and orientation of the input device 16 together with the variance of its noise (error) to the third estimation processing unit 260.

[0048] (Second estimation function using sensor data) The sensor data acquisition unit 214 acquires sensor data transmitted from the input device 16 and supplies the acquired data to the second estimation processing unit 250. The second estimation processing unit 250 realizes a second estimation function that estimates the position and orientation of the input device 16 based on sensor data indicating the acceleration and angular velocity of the input device 16. In the embodiment, the second estimation function is a function that performs a state prediction step in a Kalman filter. The second estimation processing unit 250 estimates a state vector at a current time by adding a change in the state vector obtained by integrating the supplied sensor data to the state vector (position, velocity, orientation) at the previous time. The second estimation processing unit 250 outputs the estimated state vector together with its noise variance to the third estimation processing unit 260. Note that the change obtained by the integration calculation accumulates noise over time, so the state vector (position, velocity, orientation) estimated by the second estimation processing unit 250 tends to deviate from the actual state vector (position, velocity, orientation).

[0049] (Integration function for estimation results) The third estimation processing unit 260 derives the position and orientation of the input device 16 with high accuracy from the position and orientation of the input device 16 estimated by the first estimation processing unit 240 and the state vector (position, velocity, and orientation) of the input device 16 estimated by the second estimation processing unit 250. The third estimation processing unit 260 may perform a filtering step (correction step) of a UKF (Unscented Kalman Filter). The third estimation processing unit 260 acquires the state vector estimated by the second estimation processing unit 250 as an "a priori estimate" and acquires the position and orientation estimated by the first estimation processing unit 240 as "observed values," calculates a Kalman gain, and uses the Kalman gain to correct the "a priori estimate" to obtain a "posterior estimate." The "posterior estimate" represents the position and orientation of the input device 16 with high accuracy and is provided to the game execution unit 220 and recorded in the state storage unit 226 for use in estimating the state vector at the next time point by the second estimation processing unit 250.

[0050] A technique for improving accuracy by integrating analysis results obtained using multiple sensors, such as the image capture device 14 and the IMU 32, is known as sensor fusion. In sensor fusion, the time at which data is acquired by each sensor must be expressed on a common time axis. In the information processing system 1, the image capture period of the image capture device 14 and the sampling period of the IMU 32 are different and asynchronous. Therefore, by accurately managing the image capture time and the acceleration and angular velocity detection time, the third estimation processing unit 260 can estimate the position and orientation of the input device 16 with high accuracy.

[0051] The operation information acquisition unit 216 acquires operation information transmitted from the input device 16 and supplies it to the game execution unit 220. The game execution unit 220 progresses the game based on the operation information and the position and orientation information of the input device 16 estimated by the estimation processing unit 230.

[0052] 9 is a flowchart showing the position and orientation estimation process by the first estimation processing unit 240. The captured image acquisition unit 212 acquires image data obtained by capturing an image of the input device 16 (S10) and supplies the data to the image signal processing unit 222. The image signal processing unit 222 performs image signal processing such as noise reduction and optical correction on the image data (S12), and supplies the image data that has undergone the image signal processing to the marker image coordinate identification unit 242.

[0053] The marker image coordinate identifying unit 242 identifies the representative coordinates of multiple marker images included in the captured image (S14). When the brightness of each pixel in a grayscale image is expressed in 8 bits and takes brightness values from 0 to 255, the marker image is captured as an image with high brightness, as shown in Fig. 6. The marker image coordinate identifying unit 242 may identify an area of consecutive pixels having a brightness value equal to or greater than a predetermined value (for example, a brightness value of 128) from the captured image, calculate the coordinates of the center of gravity of the consecutive pixel area, and identify the representative coordinates of the marker image.

[0054] The captured image contains not only marker images but also images of lighting devices such as electric lamps. Therefore, the marker image coordinate identification unit 242 checks whether a continuous pixel region having a luminance value equal to or greater than a predetermined value corresponds to a marker image, based on several predetermined criteria. For example, if the continuous pixel region is too large or has an elongated shape, it is certain that the continuous pixel region does not correspond to a marker image. Therefore, the marker image coordinate identification unit 242 may determine that such a continuous pixel region is not a marker image. The marker image coordinate identification unit 242 calculates the coordinates of the center of gravity of the continuous pixel region that meets the predetermined criteria, identifies it as the representative coordinates of the marker image (marker image coordinates), and stores the identified representative coordinates in a memory (not shown).

[0055] The marker information storage unit 224 stores the three-dimensional coordinates of each marker in a three-dimensional model of the input device 16 at a reference position and a reference orientation. A method of solving the PNP (Perspective n-Point) problem is known as a method for estimating the position and orientation of an image capturing device that captured an object whose three-dimensional shape and size are known, based on the captured image of the object.

[0056] In the embodiment, the position and orientation derivation unit 244 reads out N (N is an integer equal to or greater than 3) marker image coordinates from a memory (not shown), and estimates the position and orientation of the input device 16 from the read out N marker image coordinates and the three-dimensional coordinates of the N markers in the three-dimensional model of the input device 16. The position and orientation derivation unit 244 estimates the position and orientation of the image capture device 14 using the following (Equation 1), and derives the position and orientation of the input device 16 in three-dimensional space based on the estimation result.

number

[0057] Here, (u, v) are the marker image coordinates in the captured image, and (X, Y, Z) are the position coordinates of the marker 30 in three-dimensional space when the three-dimensional model of the input device 16 is in the reference position and reference orientation. The three-dimensional model has exactly the same shape and size as the input device 16, and is a model in which the markers are arranged in the same positions, and the marker information storage unit 224 stores the three-dimensional coordinates of each marker in the three-dimensional model in the reference position and reference orientation. The position and orientation derivation unit 244 reads the three-dimensional coordinates of each marker from the marker information storage unit 224 and acquires (X, Y, Z).

[0058] (f x , f y ) is the focal length of the image capture device 14, (c x , c y ) is the image principal point, and both are internal parameters of the image capture device 14. 11 ~r 33 The matrix with elements t1 to t3 is a rotation / translation matrix. x , f y ), (c x , c y ), (X, Y, Z) are known, and the position and orientation derivation unit 244 solves equations for the N markers 30 to find a common rotation and translation matrix for them. In this embodiment, the process of estimating the position and orientation of the input device 16 is performed by solving the P3P problem.

[0059] Specifically, the position and orientation derivation unit 244 extracts any three marker image coordinates from the multiple marker image coordinates identified by the marker image coordinate identification unit 242. The position and orientation derivation unit 244 reads the three-dimensional coordinates of the markers in the three-dimensional model from the marker information storage unit 224, and solves the P3P problem using Equation 1. After identifying a rotation and translation matrix common to the extracted three marker image coordinates, the position and orientation derivation unit 244 calculates a reprojection error using the marker image coordinates of the input device 16 other than the extracted three marker image coordinates.

[0060] The position and orientation derivation unit 244 extracts a predetermined number of combinations of three marker image coordinates. The position and orientation derivation unit 244 identifies a rotation and translation matrix for each combination of the extracted three marker image coordinates and calculates the respective reprojection errors. The position and orientation derivation unit 244 then identifies the rotation and translation matrix that results in the smallest reprojection error from the predetermined number of reprojection errors, and derives the position and orientation of the input device 16 (S16).

[0061] The noise derivation unit 248 derives the variance of the noise (error) of each of the estimated positions and orientations (S18). The noise variance value corresponds to the reliability of the estimated position and orientation; the higher the reliability, the smaller the variance value; and the lower the reliability, the larger the variance value. The noise derivation unit 248 may derive the noise variance based on the distance between the image capture device 14 and the input device 16 and the position of the marker image within the angle of view. For example, when the image capture device 14 and the input device 16 are far apart or extremely close, or when the marker image is located at the edge of the captured image, it is difficult to derive accurate center-of-gravity coordinates of the marker image, and therefore the noise variance tends to be large.

[0062] Note that, because the position and orientation estimated during the tracking process (while the first estimation function is being performed) are reliably highly reliable, the noise derivation unit 248 may set the noise variance of each of the estimated position and orientation to a small, fixed value. For example, the noise derivation unit 248 may set the position noise variance during the tracking process to a fixed value of 0.5 mm and supply it to the third estimation processing unit 260. During the tracking process, the first estimation processing unit 240 may output the variances of the position noise and orientation noise to the third estimation processing unit 260 along with information on the estimated position and orientation. However, if the variances of the position noise and orientation noise are fixed values, the noise variance may be output to the third estimation processing unit 260 once at the start of the tracking process, and the third estimation processing unit 260 may store and use the noise variance.

[0063] The position and orientation estimation process by the first estimation processing unit 240 is performed at the capturing period (60 frames / second) of the tracking images of the input device 16 (N in S20). When the game execution unit 220 ends the game, the position and orientation estimation process by the first estimation processing unit 240 ends (Y in S20).

[0064] 10 shows the internal configuration of the estimation processor 230. At time k, the first estimation processor 240 estimates the position and orientation as “observation value n k ”, and the variance of the position noise and the attitude noise is defined as “observation noise R k " and outputs it to the third estimation processing unit 260. Observation n k : Observation vector at time k Observation noise R k : Error covariance matrix of the observation at time k

[0065] The second estimation processing unit 250 calculates the state vector m k-1|k-1 " and "Estimation error P k-1|k-1 " is read from the state holding unit 226, and "state vector m k-1|k-1 " and "Estimation error P k-1|k-1 " is input to the prediction unit. The state variable m in this embodiment includes the position, velocity, and orientation of the input device 16, but may also include an acceleration bias and an angular velocity bias. State vector m k-1|k-1 : State vector at time k-1 estimated using information up to time k-1 · Estimation error P k-1|k-1 : Estimation error covariance matrix of the state at time k-1 estimated using information up to time k-1

[0066] The second estimation processing unit 250 also receives the acceleration a of the input device 16 from the sensor data acquisition unit 214. k and angular velocity ω k and obtain the acceleration a k and angular velocity ω k The "Process Input k " and input it into the prediction section. · Acceleration a k : Acceleration at time k · Angular velocity ω k : Angular velocity at time k Process Input k : process input vector at time k

[0067] The second estimation processing unit 250 calculates the acceleration a k and angular velocity ω k The variance of the acceleration noise and the variance of the angular velocity noise are calculated from the fixed noise parameters (including axis deviation, scale deviation, value deviation, and bias deviation) to obtain the "process noise Q k " and input it into the prediction section. Process noise Q k : Error covariance matrix of the process input at time k

[0068] The prediction part is acceleration a k and angular velocity ω k By integrating each of these, we obtain the state vector m k-1|k-1 ” (i.e., position change, velocity change, and attitude change) and calculate the “state vector m k-1|k-1 The prediction unit calculates the acceleration a k is integrated to calculate the velocity change, the velocity estimated using the velocity change is integrated to calculate the position change, and the angular velocity ω k The prediction unit calculates the amount of change in posture by integrating the state vector m k|k-1 " and "Estimation error P k|k-1 " to the third estimation processing unit 260. State vector m k|k-1 : State vector at time k estimated using information up to time k-1 · Estimation error P k|k-1 : Estimation error covariance matrix of the state at time k estimated using information up to time k-1

[0069] The third estimation processing unit 260 receives the “observation value n k " and "observation noise R k ” is acquired from the second estimation processing unit 250, and “state vector m k|k-1 " and "Estimation error P k|k-1 " and "state vector m k|k-1The third estimation processing unit 260 calculates a Kalman gain for correcting the state vector m k|k-1 " and "state vector m k|k " and "Estimation error P k|k " is output. State vector m k|k : State vector at time k estimated using information up to time k · Estimation error P k|k : Estimation error covariance matrix of the state at time k estimated using information up to time k

[0070] "State vector m k|k " indicates the highly accurately estimated position, velocity, and attitude, and may be provided to the game execution unit 220 and used in game operation. "State vector m k|k " and "Estimation error P k|k " is temporarily stored in the state storage unit 226 and is read out when the second estimation processing unit 250 performs estimation processing at time k+1.

[0071] In the estimation processing unit 230, the first estimation processing unit 240 performs estimation processing at a cycle of 60 Hz, while the second estimation processing unit 250 performs estimation processing at a cycle of 800 Hz. Therefore, from the time when the first estimation processing unit 240 outputs an observation value until the time when it outputs the next observation value, the second estimation processing unit 250 sequentially updates the state vector, and during this time, the state vector is not corrected. The estimation processing unit 230 of the embodiment performs a correction step based on the state at time k-1, which is immediately before observation time k, i.e., the observation value is used to correct the past state.

[0072] As described above, while the tracking process of the input device 16 is being performed, the estimation processing unit 230 estimates with high accuracy the position and orientation of the input device 16. However, when the marker 30 of the input device 16 is no longer captured by the imaging device 14, the first estimation processing unit 240 cannot perform the position and orientation estimation process shown in FIG.

[0073] 11 schematically shows the range that can be captured by the imaging device 14 mounted on the HMD 100. Because the imaging device 14 is attached to the front side of the HMD 100, it can capture images of the space in front of the HMD 100, but cannot capture images of the space behind the HMD 100. Therefore, if the user moves the input device 16 behind their face, the input device 16 will move out of the angle of view of the imaging device 14, making it impossible to execute the position and orientation estimation process shown in FIG.

[0074] To prepare for such a case, during the tracking process, the body part position estimation unit 246 estimates the position of a predetermined body part on the user's body based on the estimated position of the input device 16. The estimated position of the input device 16 may be the position estimated by the position and orientation derivation unit 244, or may be the estimated position included in the state vector output by the third estimation processing unit 260.

[0075] 12(a) shows an example of estimated positions of the HMD 100 and the input device 16 in a world coordinate system in real space. The information processing device 10 of the embodiment estimates the positions and orientations of the HMD 100 and the input devices 16a and 16b in the world coordinate system by performing SLAM processing on the HMD 100 and tracking processing on the input devices 16a and 16b.

[0076] 12(b) shows a method for estimating the position of a predetermined part of the user's body. The part position estimation unit 246 estimates the position of the elbow, which is a body part, from the positions of the HMD 100 and the input device 16.

[0077] First, the body part position estimation unit 246 estimates the position H1 of the user's right shoulder and the position H2 of the left shoulder from the position and posture of the HMD 100. When the HMD 100 is not tilted, the body part position estimation unit 246 may identify a point I that is a distance d1 below the center position of the HMD 100, and may identify a position H1 that is a distance d2 to the right of point I as the right shoulder position, and a position H2 that is a distance d2 to the left of point I as the left shoulder position. These distances d1 and d2 may be fixed values, or may be set according to the size of the user's body.

[0078] Next, the part position estimation unit 246 estimates the right elbow position J1 based on the right shoulder position H1 and the position of the right-hand input device 16b. At this time, the part position estimation unit 246 may estimate the right elbow position J1 using inverse kinematics from the upper arm length l1 from the right shoulder to the elbow and the forearm length l2 from the elbow to the hand. Note that the upper arm length l1 and the forearm length l2 may be fixed values, or may be set according to the user's body size.

[0079] Since there are an infinite number of candidates for the right elbow position J1 estimated by inverse kinematics, it is preferable that the body part position estimation unit 246 derives the most likely right elbow position J1 based on parameters such as the behavior of the input device 16b up to now and the distance between the input device 16b and the HMD 100. Note that a function or map that uniquely derives the right elbow position J1 according to the relative positional relationship and relative orientation relationship between the HMD 100 and the input device 16b may be prepared in advance, and the body part position estimation unit 246 may derive the right elbow position J1 using this function or map.

[0080] Similarly, the body part position estimation unit 246 estimates the left elbow position J2 based on the left shoulder position H2 and the position of the left-hand input device 16a. Note that if the body part position estimation unit 246 has a function to identify the user's elbow position contained in the full-color captured image by image analysis and derive the coordinates of the elbow position in the world coordinate space, the body part position estimation unit 246 may use the identified elbow position.

[0081] 13 shows a state in which the input device 16 is out of the capture range of the imaging device 14. When the input device 16 is out of the angle of view of the imaging device 14, the marker image is no longer included in the captured image. In this case, the marker image coordinate identification unit 242 determines that the marker image cannot be extracted from the captured image and notifies the position and orientation derivation unit 244 of the determination result. When the position and orientation derivation unit 244 receives the determination result and recognizes that tracking processing cannot be performed, it switches the estimation mode and starts position estimation processing of the input device 16 based on the elbow position estimated by the body part position estimation unit 246.

[0082] The position and orientation derivation unit 244 acquires the elbow position estimated just before the marker image is no longer included in the captured image (i.e., just before tracking is lost) from the part position estimation unit 246. The position and orientation derivation unit 244 derives, as the position of the input device 16, a position obtained by rotating the acquired elbow position as the center of rotation by an amount of rotation corresponding to the sensor data.

[0083] FIG. 14 is a diagram illustrating a process for estimating the device position using the elbow position as a base point. In the figure, the position indicated by an x indicates the position of the input device 16 estimated when the marker 30 was last photographed, and the elbow position J indicates the elbow position estimated at that time. The position and orientation derivation unit 244 derives the position of the input device 16 by rotating a virtual forearm of a predetermined length around the elbow position J as a rotation center by an amount and direction of rotation corresponding to the angular velocity of the input device 16. As shown in FIG. 12, the length of the forearm between the elbow position J and the input device 16 is l2. Therefore, while the tracking-lost state continues, the position and orientation derivation unit 244 derives the position of the input device 16 on a sphere of radius l2 with the elbow position J as the rotation center. While the position derived in this manner is not necessarily accurate, because it is estimated using the elbow position immediately before tracking loss occurs as a base point, it can be said to be sufficiently accurate for continuing the game.

[0084] While the tracking-lost state continues, it is preferable that the part position estimation section 246 moves the elbow position J in accordance with the movement of the HMD 100, and fixes the relative positional relationship between the position of the HMD 100 and the elbow position J.

[0085] The noise derivation unit 248 derives the variance of the position noise during tracking loss. The noise derivation unit 248 sets the variance of the position noise during tracking loss (when the captured image does not contain a marker image) to be larger than the variance of the position noise during tracking processing (when the captured image contains a marker image). In the example described above, the noise derivation unit 248 sets the variance of the position noise during tracking processing to 0.5 mm, but may derive a variance of the position noise of 5 mm or more during tracking loss. Since the reliability of the estimated position decreases as the amount of rotation (rotation angle) around the elbow position J becomes larger, the noise derivation unit 248 may derive a larger variance of the position noise when the amount of rotation becomes larger than when the amount of rotation is small. For example, if the amount of rotation from the device position immediately before tracking loss is less than 20 degrees, the variance of the position noise may be set to 5 mm, and if the amount of rotation is 20 degrees or more, the variance of the position noise may be set to 50 mm. The noise derivation unit 248 may derive the variance of the position noise so that it increases linearly or nonlinearly as the amount of rotation increases.

[0086] When the input device 16 moves within the field of view of the imaging device 14 and the marker image is included in the captured image, the marker image coordinate identification unit 242 extracts the marker image from the captured image, and the position and orientation derivation unit 244 restores the estimation mode and resumes the position and orientation estimation process based on the marker image.

[0087] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and those skilled in the art will understand that various modifications are possible in the combination of the respective components and processing processes, and that such modifications are also within the scope of the present invention. In the embodiments, the estimation process is performed by the information processing device 10, but the functions of the information processing device 10 may be provided in the HMD 100, and the HMD 100 may perform the estimation process. In other words, the HMD 100 may be the information processing device 10.

[0088] In the embodiment, the arrangement of the multiple markers 30 in the input device 16 having the operation member 22 has been described, but the device to be tracked does not necessarily have to have the operation member 22. In the embodiment, the imaging device 14 is attached to the HMD 100, but the imaging device 14 may be attached to a position other than the HMD 100 as long as it can capture marker images. [Explanation of symbols]

[0089] 1 Information processing system, 10 Information processing device, 14 Imaging device, 16, 16a, 16b Input device, 20 Case body, 21 Grip portion, 22 Operation member, 23 Bending portion, 24 Touch sensor, 30 Marker, 32 IMU, 34 Acceleration sensor, 36 Angular velocity sensor, 50 Control unit, 54 Communication control unit, 58 Light source, 100 HMD, 102 Output mechanism unit, 104 Wearing mechanism unit, 106 Wearing band, 108 Housing, 120 Control unit, 122 Memory unit, 124 IMU, 126 Microphone, 128 Communication control unit, 130 Display panel, 130a Left first eye display panel, 130b... right eye display panel, 132... audio output unit, 200... processing unit, 202... communication unit, 210... acquisition unit, 212... captured image acquisition unit, 214... sensor data acquisition unit, 216... operation information acquisition unit, 220... game execution unit, 222... image signal processing unit, 224... marker information holding unit, 226... state holding unit, 230... estimation processing unit, 240... first estimation processing unit, 242... marker image coordinate identification unit, 244... position and orientation derivation unit, 246... body part position estimation unit, 248... noise derivation unit, 250... second estimation processing unit, 260... third estimation processing unit, 268... image signal processing unit, 270... SLAM processing unit.

Claims

1. An information processing device that estimates the location of a device carried by a user, a captured image acquisition unit that acquires an image of the device; an estimation processing unit that estimates a position of the device based on an image of the device; a sensor data acquisition unit that acquires sensor data indicating an angular velocity of the device, The estimation processing unit a derivation unit that derives a position of the device in a three-dimensional space from position coordinates of the device in the captured image when the device is included in the captured image; a body part position estimation unit that estimates the position of a predetermined body part of the user based on the estimated position of the device, When the device is no longer included in the captured image, the derivation unit derives, as the position of the device, a position obtained by rotating the position of the part estimated by the part position estimation unit by an amount of rotation corresponding to the sensor data as a rotation center.

1. An information processing device comprising:

2. the captured image acquisition unit acquires an image captured by an imaging device mounted on a head-mounted display worn by a user, when the device is included in the captured image, the part position estimation unit estimates the position of the part from the position of the head-mounted display and the position of the device using inverse kinematics; 2. The information processing apparatus according to claim 1, wherein:

3. When the device is no longer included in the captured image, the part position estimation unit moves the position of the part in accordance with the movement of the head-mounted display.

3. The information processing apparatus according to claim 2, wherein:

4. When the device is no longer included in the captured image, the derivation unit rotates a virtual arm of a predetermined length by an amount of rotation corresponding to the sensor data around the position of the part as a rotation center, to derive the position of the device.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. An information processing device that estimates the location of a device carried by a user, a captured image acquisition unit that acquires an image of the device; a first estimation processing unit that estimates a position of the device based on an image of the device; a sensor data acquisition unit that acquires sensor data indicating an acceleration and / or an angular velocity of the device; a second estimation processing unit that estimates a position of the device based on the sensor data; a third estimation processing unit that derives a position of the device based on the position of the device estimated by the first estimation processing unit and the position of the device estimated by the second estimation processing unit, The first estimation processing unit a derivation unit that derives a position of the device in a three-dimensional space from position coordinates of the device in the captured image when the device is included in the captured image; a body part position estimation unit that estimates the position of a predetermined body part of the user based on the estimated position of the device, When the device is no longer included in the captured image, the derivation unit derives, as the position of the device, a position obtained by rotating the position of the part estimated by the part position estimation unit by an amount of rotation corresponding to the sensor data as a rotation center.

1. An information processing device comprising:

6. the first estimation processing unit has a noise derivation unit that derives a variance of noise at the device's position, the noise derivation unit derives a first variance when the device is included in the captured image, and derives a second variance greater than the first variance when the device is not included in the captured image; 6. The information processing apparatus according to claim 5,

7. When the device is not included in the captured image, the noise derivation unit derives the second variance that is larger when the amount of rotation is large than when the amount of rotation is small.

7. The information processing apparatus according to claim 6,

8. The part is the user's elbow.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

9. 1. A method for estimating a location of a device carried by a user, comprising: acquiring an image captured by an imaging device; estimating a position of the device based on an image of the device captured by the imaging device; estimating a position of a predetermined part on a user's body based on the estimated position of the device; acquiring sensor data indicative of an angular velocity of the device; When the device is no longer included in the image captured by the imaging device, deriving a position of the device by rotating the estimated position of the part as a rotation center by an amount of rotation corresponding to the sensor data; A device location estimation method comprising:

10. 1. A method for estimating a location of a device carried by a user, comprising: acquiring an image captured by an imaging device; a first estimation step of estimating a position of the device based on an image of the device captured by the imaging device; acquiring sensor data indicative of acceleration and / or angular velocity of the device; a second estimation step of estimating a location of the device based on the sensor data; a third estimation step of estimating a position of the device based on the position of the device estimated in the first estimation step and the position of the device estimated in the second estimation step, The first estimation step When the device is included in the captured image, estimating the position of the device in three-dimensional space from the position coordinates of the device in the captured image; estimating a position of a predetermined part on a user's body based on the estimated position of the device; and when the device is no longer included in the captured image, deriving a position obtained by rotating the device by an amount of rotation corresponding to the sensor data around the estimated position of the part as a rotation center as the position of the device. A device location estimation method comprising:

Images