Information processing apparatus, information processing method, and program

Abstract

To accurately estimate the position attitude of a terminal.SOLUTION: The information processing apparatus includes a first motion information acquisition unit configured to acquire motion information of the terminal from a first sensor included in the terminal, a second motion information acquisition unit configured to acquire motion information of the moving object from a second sensor arranged in the moving object, and a position and orientation estimation unit configured to estimate a position and orientation of the terminal based on a position of the first sensor, a position of the second sensor, the first motion information, and the second motion information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to control of position and orientation estimation of a terminal on a moving body. [Background technology]

[0002] A method is known for estimating the position and orientation of a device based on motion information (angular velocity and acceleration) measured by an IMU (Inertial Measurement Unit) installed in the device.

[0003] When a user holding a device is on board a moving object such as a vehicle or ship, the motion information measured by the IMU includes both the motion of the device on the moving object and the motion of the moving object itself, making it difficult to estimate the position and orientation using only the motion of the device on the moving object.

[0004] Patent Document 1 describes a method for correcting the measurement values ​​of an IMU mounted on a terminal by using an IMU attached to a moving body to measure only the movement of the moving body, separate from the IMU mounted on the terminal. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2023-076069 Summary of the Invention [Problem to be solved by the invention]

[0006] However, if the IMU mounted on the device and the IMU attached to the moving object are located far apart, a discrepancy occurs between the forces applied to both IMUs due to the movement of the moving object, making it impossible to accurately correct the measurements of the IMU mounted on the device.

[0007] The present invention has been made in view of the above problems, and has as its object to accurately estimate the position and orientation of a terminal on a moving body. [Means for solving the problem]

[0008] The present invention is an information processing device that estimates the position and orientation of a terminal on a moving body, characterized by comprising: a first motion information acquisition means that acquires motion information of the terminal from a first sensor equipped on the terminal; a second motion information acquisition means that acquires motion information of the moving body from a second sensor arranged on the moving body; and a position and orientation estimation means that estimates the position and orientation of the terminal based on the position of the first sensor, the position of the second sensor, the first motion information, and the second motion information. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an information processing device that can accurately estimate the position and orientation of a terminal on a moving body. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is an image diagram showing the positional relationship of an IMU in the first embodiment. [Figure 2] 1 is a block diagram showing a hardware configuration of an information processing device according to a first embodiment. [Figure 3] FIG. 2 is a data flow diagram of the information processing device according to the first embodiment. [Figure 4] 4 is a flowchart executed by the information processing device according to the first embodiment. [Figure 5] FIG. 10 is a block diagram showing the positional relationship of an IMU in a second embodiment. [Figure 6] FIG. 10 is a data flow diagram of the information processing device according to the second embodiment. [Figure 7] 10 is a flowchart executed by an information processing device according to the second embodiment. [Figure 8] FIG. 11 is a data flow diagram of an information processing device according to a third embodiment. [Figure 9] 10 is a flowchart executed by an information processing device according to the third embodiment. [Figure 10] FIG. 10 is a conceptual diagram showing a UI according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Example 1 In this embodiment, the position and orientation of a terminal held by a user on board a moving body is estimated.

[0012] FIG. 1 is an image diagram showing the positional relationship of IMUs in this embodiment. Reference numeral 101 denotes a moving body such as a vehicle or a ship, and 111 denotes a terminal carried by a user. Reference numeral 121 denotes a first sensor mounted on the terminal 111, and 122 denotes a second sensor disposed on the moving body 101. The first sensor 121 and the second sensor 122 are sensors that measure motion information (angular velocity and acceleration), and are assumed to be IMUs in this embodiment. The first sensor 121 and the second sensor 122 are assumed to be located at separate positions. The terminal 111 includes an information processing device 131 (not shown).

[0013] FIG. 2 is a block diagram showing the hardware configuration of the information processing device 131.

[0014] Reference numeral 201 denotes a CPU, which controls various devices connected to a system bus 202 .

[0015] 203 is a ROM that stores a BIOS program and a boot program.

[0016] A RAM 204 is used as a main storage device for the CPU 201 and as a memory when the computer executes a computer program.

[0017] An external memory 205 stores programs for operating the information processing device 131 and computer programs.

[0018] An input unit 206 receives input from a keyboard, mouse, or touch panel, and performs processing related to the input of information, etc.

[0019] Reference numeral 207 denotes a sensor receiving unit that acquires the measured values ​​of the first sensor 121 and the second sensor 122 in FIG. 1 by using wired or wireless communication.

[0020] An output unit 208 outputs video information to a monitor or a touch panel.

[0021] FIG. 3 is a data flow diagram of the information processing device 131 used in this embodiment.

[0022] Reference numeral 311 denotes a first movement information acquisition unit that acquires movement information measured by the first sensor 121 .

[0023] A second movement information acquisition unit 312 acquires movement information measured by the second sensor 122 .

[0024] 313 is a motion information conversion unit that converts the motion information of the second sensor 122 acquired by the second motion information acquisition unit into motion information corresponding to the coordinates of the first sensor 121 and calculates the converted second motion information.

[0025] A motion information correcting unit 314 corrects the motion information measured by the first sensor 121 and acquired by the first motion information acquiring unit 311 based on the motion information converted by the motion information converting unit 313 .

[0026] A position and orientation estimation unit 315 estimates the position and orientation of the terminal 111 based on the motion information corrected by the motion information correction unit 314 .

[0027] The terminal 111 includes an information processing device 131 and a first sensor 121 .

[0028] The processing procedures shown in the flowcharts in the following explanations are not limited to the examples, and any combination of procedures, combining multiple processes, or subdividing processes is possible as long as the results of the present invention are achieved. Also, each process can be extracted individually and function as a single functional element, and can be used in combination with processes other than those shown.

[0029] 4 is a flowchart showing the processing procedure in this embodiment. The processing of this flowchart is executed by the information processing device 131.

[0030] In step S410, the information processing device 131 in Fig. 3 is initialized. That is, a program is read from the external memory 205 in Fig. 2, and the information processing device 131 is brought into an operable state.

[0031] In step S411, the first motion information acquisition unit 311 acquires motion information measured by the first sensor 121. In this embodiment, it is assumed that angular velocity ω1 and acceleration a1 are acquired. Here, the first sensor 121 is mounted on the terminal 111 held by a user moving on the moving object 101, and therefore the first sensor 121 measures a mixture of the motion of the terminal 111 and the motion of the moving object 101.

[0032] In step S412, the second movement information acquisition unit 312 acquires movement information measured by the second sensor 122. In this embodiment, the angular velocity ω2 and the acceleration a2 are acquired. Here, since the second sensor 122 is attached to and fixed on the moving body 101, only the movement of the moving body 101 is measured. Note that angular velocity ω1 and angular velocity ω2 are each three-dimensional vectors that hold the rotational velocity for each of the X, Y, and Z axes. Acceleration a1 and acceleration a2 are each three-dimensional vectors that hold the acceleration components for each of the X, Y, and Z axes.

[0033] In step S413, the motion information conversion unit 313 converts the motion information of the second sensor 122 acquired by the second motion information acquisition unit into motion information corresponding to the coordinates of the first sensor 121. This makes it possible to obtain, through conversion, motion information that can be obtained if the second sensor 122 were placed at the position of the first sensor 121. The motion information obtained here is called converted second motion information.

[0034] The movement information conversion unit 313 first acquires the relative position and orientation M2 between the first sensor 121 and the second sensor 122. In this embodiment, the approximate riding position of the user is known, that is, the approximate coordinates of the first sensor 121 are known, and the coordinates of the second sensor 122 on the moving object are also known, and the relative position and orientation M2 between them is assumed to be obtainable. Note that the rotation component of M2 is defined as R2 and the translation component as t2. Here, R2 is a 3x3 rotation matrix, and t2 is a three-dimensional translation vector.

[0035] Next, based on the relative position and orientation M2, the movement information conversion unit 313 converts the angular velocity ω2 and acceleration a2 measured in the coordinates of the second sensor 122 into angular velocity ω'2 and acceleration a'2 in the coordinates of the first sensor 121. The conversion formula is as follows:

number

number

[0036] In step S414 , the motion information correcting unit 314 corrects the motion information of the first sensor 121 acquired by the first motion information acquiring unit 311 based on the motion information converted by the motion information converting unit 313 .

[0037] The corrected angular velocity ω and acceleration a can be calculated using the following formulas.

number

[0038] The formula will be explained below. This calculation makes it possible to subtract the component of the movement of the moving object 101 contained in the first sensor 121 and calculate the movement information that indicates only the movement of the terminal 111. The movement information calculated here is called corrected first movement information.

[0039] In step S415 , the position and orientation estimation unit 315 estimates the position and orientation of the terminal 111 based on the motion information corrected by the motion information correction unit 314 .

[0040] The method of estimating the position and orientation based on the motion information (angular velocity and acceleration) can be realized by a known technique, for example, a method of estimating the position and orientation by combining multiple sensors such as a MEMS sensor, an accelerometer, and a gyroscope.

[0041] As explained above, in this embodiment, by using the motion information converted by the motion information conversion unit 313, the motion information measured by the first sensor 121 can be accurately corrected, and the position and orientation of the terminal 111 can be accurately estimated.

[0042] [Variation 1-1] In the first embodiment, the motion information conversion unit 315 first acquires the relative position and orientation M2 between the first sensor 121 and the second sensor 122. Here, any method for acquiring the position and orientation M2 may be used. A marker whose relative position and orientation is known with respect to the second sensor 122 attached to the moving object 101 may be placed, and the position and orientation of the terminal 111 may be estimated based on an image of the marker captured by a camera provided on the terminal 111. A position and orientation estimated by the position and orientation estimation unit 315 at a previous point in time may also be used. Alternatively, the position and orientation may be input by the user via a touch panel or keyboard provided on the terminal 111.

[0043] [Variation 1-2] In the first embodiment, the position and orientation estimation unit 315 estimates the position and orientation based on the motion information (angular velocity and acceleration). However, the position and orientation estimation unit 315 in this embodiment may use a method of estimating the position and orientation by further using information other than the motion information. For example, the terminal may be equipped with a camera that captures 2D images, a depth camera that measures depth information, a GPS sensor, etc., and the position and orientation may be estimated by combining and using information obtained from these sensors with the motion information.

[0044] The position and orientation estimation method using information from multiple sensors can be realized by a known technique, such as a position and orientation estimation method using information from multiple cameras and IMUs.

[0045] <Example 2> In the first embodiment, the second motion information acquisition unit 312 acquires only motion information measured by one second sensor. In contrast, in the present embodiment, a method is described in which a plurality of second sensors are arranged on the moving object 101, and the motion information of the first sensor 121 is corrected more accurately based on the plurality of second sensor information.

[0046] FIG. 5 is an image diagram showing the positional relationship of the IMU in this embodiment. The description of the components described in the first embodiment will be omitted. Reference numeral 122 denotes a first second sensor A attached to the moving body 101, and is the same as the second sensor 122 described in the first embodiment. Reference numeral 123 denotes a second second sensor B attached to the moving body 101. The second sensor B 123 is a sensor that measures motion information (angular velocity and acceleration), and is an IMU in this embodiment. Note that the second sensor B 123 is located away from the first sensor 1 and the second sensor A 122.

[0047] FIG. 6 is a data flow diagram of the information processing device 131 used in this embodiment.

[0048] Here, the first motion information acquisition unit 321 and the position and orientation estimation unit 325 in FIG. 6 are the same as the motion information acquisition unit 311 and the position and orientation estimation unit 315 in the first embodiment.

[0049] The second movement information acquisition unit 322 acquires movement information measured by each of the second sensor A122 and the second sensor B123.

[0050] The motion information conversion unit 323 performs the same conversion as in the first embodiment for each of the second sensor A 122 and the second sensor B 123. The motion information obtained here is referred to as converted second motion information.

[0051] The motion information correcting unit 324 corrects the motion information measured by the first sensor 121 and acquired by the first motion information acquiring unit 321 based on the plurality of pieces of motion information converted by the motion information converting unit 323 .

[0052] Fig. 7 is a flowchart showing the processing procedure in this embodiment. The processing of this flowchart is executed by the information processing device 131. Here, first steps S420, S421, and S425 in Fig. 7 are the same as steps S410, S411, and S415 in the first embodiment.

[0053] In step S422, the second movement information acquisition unit 322 acquires movement information measured by each of the second sensor A 122 and the second sensor B 123. In this embodiment, it is assumed that the angular velocity ω2 and the acceleration a2 are acquired from the second sensor A 122, and the angular velocity ω3 and the acceleration a3 are acquired from the second sensor B 123.

[0054] In step S423, the motion information conversion unit 323 performs the same conversion as in the first embodiment for each of the second sensor A 122 and the second sensor B 123. This makes it possible to obtain, through conversion, motion information that can be acquired if the second sensor A 122 or the second sensor B 123 were placed at the position of the first sensor 121.

[0055] The conversion formula is as follows:

number

[0056] In step S424, the motion information correction unit 324 corrects the motion information measured by the first sensor 121 and acquired by the first motion information acquisition unit 321 based on the multiple pieces of motion information converted by the motion information conversion unit 323.

[0057] The corrected angular velocity ω and acceleration a can be calculated using the following formulas.

number

[0058] As described above, in this embodiment, by averaging multiple pieces of second motion information converted by the motion information conversion unit 323, the error contained in the converted motion information can be reduced, and the position and orientation of the terminal 111 can be estimated with high accuracy.

[0059] [Variation 2-1] In the second embodiment, the motion information correcting unit 324 simply averages the plurality of pieces of second motion information converted by the motion information converting unit 323. Alternatively, the motion information correcting unit 324 may use the tendency of errors contained in the plurality of pieces of second motion information converted by the motion information converting unit 323 to perform weighted averaging.

[0060] Because the motion information measured by the sensor contains errors, these errors may be amplified by the conversion by the motion information conversion unit 323. For example, in the conversion of acceleration, errors increase depending on the magnitude of the translational component. This is because the conversion of acceleration calculated using Equations 2-2 and 2-4 includes calculations in which the translational component is used as a coefficient in the second and third terms. In other words, the magnitude of the errors contained in the converted a'2 and a'3 changes depending on the magnitude of the translational components t2 and t3.

[0061] In this modification, averaging is performed with weighting according to the magnitude (distance) of the translational component from the first sensor 121. When the distance from the first sensor 121 to the second sensor A 122 is L2 and the distance to the second sensor B 123 is L3, the corrected angular velocity ω and acceleration a can be calculated by the following equations.

number

[0062] Furthermore, if the reliability of the second sensor itself or the magnitude of the error is determined in addition to the distance, weighting may be performed according to that information to perform averaging.

[0063] Furthermore, instead of weighted averaging, the correction calculation may be performed by selecting the second sensor that is closest or has the highest accuracy from among multiple second sensors. In this case, the motion information converted from the second sensors that were not selected does not need to be used.

[0064] Example 3 In the first embodiment, it is assumed that the motion information acquired by the second sensor information acquisition unit 312 is from a predetermined second sensor. In contrast, in the present embodiment, a method will be described in which a list of second sensors from which motion information can be acquired is displayed, and the user selects a second sensor from the list.

[0065] 8 is a data flow diagram of the information processing device 131 used in this embodiment. The processing of this flowchart is executed by the information processing device 131.

[0066] The first motion information acquisition unit 331, motion information conversion unit 333, motion information correction unit 334, and position and orientation estimation unit 335 are the same as the motion information acquisition unit 311, motion information conversion unit 313, motion information correction unit 314, and position and orientation estimation unit 315 in the first embodiment.

[0067] A user interface 141 is used by the user to select a second sensor from a plurality of second sensor candidates. In this embodiment, the user interface 141 is a touch panel mounted on the terminal 111.

[0068] A sensor selection unit 336 selects a second sensor in response to an input from the user interface 141.

[0069] The second movement information acquisition unit 332 acquires movement information measured by the second sensor selected by the sensor selection unit 336 .

[0070] 8 is a flowchart showing the processing procedure in this embodiment, where steps S430, S431, S433, S434, and S435 are the same as steps S410, S411, S413, S414, and S415 in the first embodiment.

[0071] In step S436, the sensor selection unit 336 selects the second sensor in response to an input from the user interface 141.

[0072] First, the sensor selection unit 336 displays a list of sensors that can receive motion information from a database of second sensors registered in advance at the current terminal 111. The information on the second sensors displayed in the list includes information on the distance from the first sensor 121 of the terminal 111 and information on the reliability of the measurement values ​​of the second sensors.

[0073] Next, the sensor selection unit 336 receives the second sensor information selected by the user via the user interface 141. The user selects a desired second sensor from the list of second sensors.

[0074] In step S432, the second movement information acquisition unit 332 acquires movement information using the second sensor selected by the sensor selection unit 336.

[0075] Fig. 10 is an image diagram showing an example of the user interface 141 in this embodiment. Reference numeral 511 in Fig. 10 is a list of second sensors, from which the user selects a desired second sensor. Reference numeral 521 is a map displaying the location information of the second sensors. Reference numeral 522 is the location information of the terminal 111. Reference numeral 523 is the location information of the second sensor that is a candidate for selection.

[0076] Here, the user can check the list of second sensors displayed on the user interface 141 and select, for example, a second sensor with high accuracy or a second sensor that is close to the terminal 111, thereby reducing the occurrence of errors.

[0077] As described above, in this embodiment, correction is performed based on movement information measured from the second sensor selected by the user, and therefore, by selecting an appropriate second sensor, the position and orientation of the terminal 111 can be estimated with high accuracy.

[0078] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. The program is also called a computer program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. [Explanation of symbols]

[0079] 101 Mobile 111 terminal 121 First Sensor 122 Second Sensor

Claims

1. An information processing device that estimates the position and orientation of a terminal on a moving body, a first movement information acquisition means for acquiring movement information of the terminal from a first sensor included in the terminal; a second movement information acquisition means for acquiring movement information of the moving object from a second sensor disposed on the moving object; a position and orientation estimation means for estimating a position and orientation of the terminal based on the position of the first sensor, the position of the second sensor, the first motion information, and the second motion information.

1. An information processing device comprising:

2. a motion information conversion means for converting the second motion information into converted second motion information indicating motion information that can be acquired when the second sensor is placed at the position of the first sensor; a motion information correcting means for correcting the first motion information to the corrected first motion information based on the converted second motion information, the position and orientation estimation means estimates the position and orientation of the terminal based on the corrected first motion information; 2. The information processing apparatus according to claim 1, wherein:

3. The motion information includes angular velocity and acceleration.

3. The information processing apparatus according to claim 2, wherein:

4. the motion information correcting means subtracts the angular velocity and the acceleration included in the converted second motion information from the angular velocity and the acceleration included in the first motion information; 4. The information processing apparatus according to claim 3,

5. the motion information conversion means converts the second motion information into the converted second motion information based on a relative position and orientation between the first sensor and the second sensor; 3. The information processing apparatus according to claim 2, wherein:

6. the second movement information acquisition means acquires movement information of the plurality of moving objects from the plurality of second sensors arranged on the moving object; the motion information conversion means converts each of the plurality of pieces of second motion information into the converted second motion information based on a relative position and orientation between the first sensor and the second sensor; 3. The information processing apparatus according to claim 2, wherein:

7. the motion information correcting means corrects the first motion information in accordance with distances from the first sensor to the plurality of second sensors; 7. The information processing apparatus according to claim 6,

8. the motion information correcting means corrects the first motion information based on the reliabilities of the plurality of second sensors; 7. The information processing apparatus according to claim 6,

9. a sensor selection means for selecting the second sensor that can receive the second movement information in the terminal; the second movement information acquisition means acquires the second movement information from the second sensor selected by the sensor selection means; 7. The information processing apparatus according to claim 6,

10. An information processing method for estimating the position and orientation of a terminal on a moving body, comprising: a first motion information acquisition step of acquiring motion information of the terminal from a first sensor included in the terminal; a second movement information acquisition step of acquiring movement information of the moving object from a second sensor disposed on the moving object; a position and orientation estimation step of estimating a position and orientation of the terminal based on the position of the first sensor, the position of the second sensor, the first movement information, and the second movement information.

10. An information processing method executed by an information processing device.

11. A computer program for causing a computer to execute each step of the information processing method according to claim 10.

Images