Position / pose estimation device, position / pose estimation method, and recording medium recording a program

Abstract

The position / posture estimation device (101) has: a database reading section (15) that reads data of a three-dimensional map from a database; a frame selection section (16) that performs processing of selecting frames for calculating a position / posture from frames of images taken from a plurality of different viewpoints; a relative position / posture acquisition section (17) that performs processing of acquiring a plurality of relative positions / postures with respect to a plurality of frames selected by the frame selection section; an absolute position / posture calculation section (18) that performs processing of acquiring a plurality of absolute positions / postures with respect to a plurality of frames selected by the frame selection section; and an absolute position / posture synthesis section (19) that synthesizes the relative position / posture acquired by the relative position / posture acquisition section (17) and the absolute position / posture acquired by the absolute position / posture calculation section (18) to acquire a final absolute position / posture.

Description

Technical Field

[0001] This disclosure relates to a position / attitude estimation device, a position / attitude estimation method, and a procedure. Background Technology

[0002] In devices employing Augmented Reality (AR) and robots employing Automated Guided Vehicles (AGVs), methods have been proposed to improve the accuracy of position / attitude calculation (i.e., estimation) by combining the calculation results of relative position / attitude with the calculation results of absolute position / attitude. For example, see Patent Document 1.

[0003] Relative position / attitude calculation is the calculation of the amount of movement relative to a given position / attitude, and it is a process that is repeatedly performed according to each fixed period (usually a short period). For example, methods used in relative position / attitude calculation include: Simultaneous Localization and Mapping (SLAM) to calculate movement based on images captured by a camera (camera image) or distance detected by a distance sensor; methods using an Inertial Measurement Unit (IMU) that integrates gyroscope sensors or accelerometer sensors; and autonomous navigation methods that calculate movement based on wheel rotation speed. In these methods, errors are added each time the relative position / attitude is calculated, thus there is a problem that the accumulated error increases when moving long distances. Therefore, the following process is performed: combining the results of the relative position / attitude calculation with the results of the absolute position / attitude calculation, and periodically removing the errors accumulated through the relative position / attitude calculation.

[0004] The calculation of absolute position / pose is performed using a pre-prepared 3D map, for example, in front of an object represented by the 3D map. Alternatively, the calculation can be performed using a 3D map and camera images.

[0005] Figure 1This diagram illustrates an example of how, when a user 112 carrying an AR-enabled terminal 111 moves, the terminal 111 estimates its position / attitude using both relative and absolute position / attitude calculations. The terminal 111 is, for example, a tablet or a terminal using a Head-Mounted Display (HMD). In this case, the terminal 111 moves while performing relative position / attitude calculation (1), removes accumulated errors using the result of absolute position / attitude calculation (2) performed in front of the object 113 represented by the 3D map, and then moves while performing relative position / attitude calculation (3). Thus, even when away from the object 113 represented by the 3D map, the position / attitude can be estimated with high accuracy, resulting in the continuous overlay display of AR content at an appropriate position on the actual image displayed on the terminal's screen.

[0006] Figure 2 This diagram illustrates an example where robot 121, using an AGV, moves and estimates its position / attitude using both relative and absolute position / attitude calculations. In this case, robot 121 moves while performing relative position / attitude calculation (4), removes accumulated errors using the result of absolute position / attitude calculation (5) performed in front of object 123 represented by the 3D map, and then moves while performing relative position / attitude calculation (6). Thus, even when leaving object 123 represented by the 3D map, robot 121 can estimate its position / attitude with high accuracy, resulting in accurately reaching the target location.

[0007] Existing technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Application Publication No. 2019-160147 Summary of the Invention

[0010] The problem the invention aims to solve

[0011] However, previous absolute position / pose estimations were performed using only one frame (i.e., one image frame), resulting in a problem where the computational accuracy (i.e., the accuracy of the estimation) varied significantly depending on the subject contained in that one frame.

[0012] For example, when using camera images, the accuracy of absolute position / attitude calculation sometimes varies depending on the pattern of the subject. That is, when the pattern of the subject is characteristic, the absolute position / attitude can be calculated with high accuracy, but when the pattern of the subject is a repeating pattern such as stripes or borders, or when the subject is an object without a pattern such as a plain white wall, the accuracy of the absolute position / attitude calculation decreases.

[0013] Furthermore, when using distance information from the subject obtained through lasers or infrared light to calculate absolute position / attitude, the accuracy of the calculation depends heavily on significant variations in the subject's shape. That is, when the subject has a distinctive shape, the absolute position / attitude can be calculated with high accuracy, but when the subject's shape is not distinctive, the accuracy of the calculation decreases.

[0014] This disclosure was made to solve the above-mentioned problems, and its purpose is to provide a position / attitude estimation device, position / attitude estimation method and program that can improve the accuracy of position / attitude estimation.

[0015] means for solving problems

[0016] The position / attitude estimation apparatus of this disclosure is characterized by comprising: a database reading unit that reads data of a three-dimensional map from a database; a frame selection unit that selects frames for calculating position / attitude from frames of images captured from multiple different viewpoints; a relative position / attitude acquisition unit that acquires multiple relative positions / attitudes with respect to the multiple frames selected by the frame selection unit; an absolute position / attitude calculation unit that acquires multiple absolute positions / attitudes with respect to the multiple frames selected by the frame selection unit; and an absolute position / attitude synthesis unit that synthesizes the relative position / attitude acquired by the relative position / attitude acquisition unit and the absolute position / attitude acquired by the absolute position / attitude calculation unit to obtain a final absolute position / attitude.

[0017] The position / attitude estimation method disclosed herein is a method performed by a position / attitude estimation device, characterized by comprising the following steps: reading data from a three-dimensional map from a position database; selecting frames for calculating position / attitude from frames of images captured from multiple different viewpoints; obtaining multiple relative positions / attitudes with respect to the selected multiple frames; obtaining multiple absolute positions / attitudes with respect to the selected multiple frames; and combining the obtained relative positions / attitudes and the obtained absolute positions / attitudes to obtain the final absolute position / attitude.

[0018] The effects of the invention

[0019] The apparatus, method, or procedure disclosed herein can improve the accuracy of position / attitude estimation. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating an example of how a terminal using an AR-enabled device estimates its position / attitude by calculating both relative and absolute position / attitude when a user moves around.

[0021] Figure 2 This is a diagram illustrating an example of how a robot using an AGV estimates its position / attitude by calculating both relative and absolute position / attitude.

[0022] Figure 3 This is a diagram illustrating an example of the hardware structure of the position / attitude estimation device of Embodiment 1 and the position / attitude estimation system including the position / attitude estimation device.

[0023] Figure 4 It is shown Figure 3 A diagram illustrating an example of the hardware structure of a position / attitude estimation device.

[0024] Figure 5 This is a functional block diagram that outlines the structure of the three-dimensional map making apparatus according to Embodiment 1.

[0025] Figure 6 This is a diagram illustrating an example of processing that adds random patterns to an image.

[0026] Figure 7 This diagram illustrates the process of aligning and registering a 3D map with a 2D map.

[0027] Figure 8 This is a functional block diagram that outlines the structure of the position / attitude estimation device in Embodiment 1.

[0028] Figure 9 This is a flowchart illustrating an example of a process for creating a three-dimensional map performed by the three-dimensional map-making apparatus of Embodiment 1.

[0029] Figure 10 This is a flowchart illustrating an example of the processing performed by the position / attitude estimation device of Embodiment 1 for estimating position / attitude.

[0030] Figure 11 This is a flowchart illustrating another example of the processing performed by the position / attitude estimation device of Embodiment 1 for estimating position / attitude.

[0031] Figure 12This is a functional block diagram that outlines the structure of the three-dimensional map making apparatus according to Embodiment 2.

[0032] Figure 13 This is a diagram illustrating the method for calculating variance used in the three-dimensional map making apparatus of Embodiment 2.

[0033] Figure 14 This is a functional block diagram that outlines the structure of the position / attitude estimation device in Embodiment 2.

[0034] Figure 15 This is a flowchart illustrating an example of the process for creating a three-dimensional map performed by the three-dimensional map-making apparatus of Embodiment 2.

[0035] Figure 16 This is a flowchart illustrating an example of the processing performed by the position / attitude estimation device of Embodiment 2 for estimating position / attitude.

[0036] Figure 17 This is a functional block diagram that outlines the structure of the position / attitude estimation device in Embodiment 3.

[0037] Figure 18 This is a flowchart illustrating an example of the processing performed by the position / attitude estimation device of Embodiment 3 for estimating position / attitude.

[0038] Figure 19 This is a functional block diagram that outlines the structure of the position / attitude estimation device in Embodiment 4.

[0039] Figure 20 This is a flowchart illustrating another example of the processing performed by the position / attitude estimation device of Embodiment 4 for estimating position / attitude. Detailed Implementation

[0040] Hereinafter, the position / attitude estimation apparatus, position / attitude estimation method, and procedure according to embodiments will be described with reference to the accompanying drawings. The embodiments described below are merely examples, and embodiments can be appropriately combined and modified. Furthermore, in this application, "position / attitude" refers to both position and attitude. "Position" refers, for example, to the position of a terminal or robot equipped with a camera. "Attitude" refers, for example, to the shooting direction of the camera or the measurement direction of the distance sensor.

[0041] Implementation Method 1

[0042] 1-1 Structure

[0043] Summary of "1-1-1"

[0044] In Embodiment 1, the improvement in the calculation accuracy of the absolute position / attitude using images captured by a camera, i.e., camera images (i.e., video images), is described. As a method for estimating position / attitude using camera images, a first estimation method is known. For example, refer to Non-Patent Document 1.

[0045] Non-patent literature 1: Paul-Edouard Sarlin, 3rd book, “From Coarse to Fine: Robust Hierarchical Localization at Large Scale”, 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition.

[0046] The first estimation method uses direct matching. In the first estimation method, the camera's position / pose is directly calculated based on a set of local features of the image.

[0047] Furthermore, a second estimation method is known as a method for estimating position / pose using camera images. In this second estimation method, the position / pose of the camera is estimated from the image using a Convolutional Neural Network (CNN). See, for example, Non-Patent Literature 2.

[0048] Non-patent literature 2: Samarth Brahmbhatt, 4th book, "Geometry-Aware Learning of Maps for Camera Localization", 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition.

[0049] In the second estimation method, a two-stage matching process is used. In this method, multiple images from different viewpoints are prepared. First, the image most similar to the captured camera image is identified from these images. Then, based on the correspondence between local features extracted from the similar image and local features extracted from the camera image, the camera's position / pose is determined. In Implementation 1, the method based on the second estimation method is used.

[0050] Figure 3This is a diagram illustrating an example of the position / attitude estimation device 101 of Embodiment 1 and the hardware structure of the position / attitude estimation system 100 including the position / attitude estimation device 101. The position / attitude estimation device 101 of Embodiment 1 includes a computer that performs calculations for estimating position / attitude. Figure 3 In the example, the position / attitude estimation system 100 of Implementation 1 includes a position / attitude estimation device 101, a three-dimensional map database (3D map DB) stored in a storage device 102, a distance sensor 103, a camera 104 as a camera device, and a display 105 such as a liquid crystal display device. Furthermore, in Figure 3 In this example, the position / attitude estimation system 100 includes a gyroscope sensor 106, an accelerometer sensor 107, and a geomagnetic sensor 108. A device with a gyroscope sensor 106, an accelerometer sensor 107, and a geomagnetic sensor 108 is also referred to as an "IMU". The position / attitude estimation device 101 and... Figure 3 Other structures shown are connected, for example, via a network. The 3D map DB102 can also be part of the position / attitude estimation device 101.

[0051] The 3D map DB102 contains pre-prepared 3D map information used in calculating the absolute position / attitude. The 3D map DB102 need not be part of the position / attitude estimation system 100 of Embodiment 1; it can also be information stored on an external storage device. Furthermore, the 3D map DB102 can also be created by the position / attitude estimation device 101. In this case, the position / attitude estimation device 101 functions as a 3D map creation device. That is, the 3D map creation device of Embodiment 1 is part of the position / attitude estimation device 101. However, the 3D map creation device of Embodiment 1 can also be a different device from the position / attitude estimation device 101.

[0052] Distance sensor 103 is a device that measures distance using infrared light or laser light. Camera 104 is a device that acquires camera images. The position / attitude estimation system 100 may also have only one of the camera 104 and distance sensor 103.

[0053] Display 105 is a necessary display device when AR content is overlaid on a camera image. Position / pose estimation system 100 may also not have display 105.

[0054] The gyroscope sensor 106, accelerometer 107, and magnetometer 108 constitute an IMU, which is a device used to calculate relative position / attitude using an autonomous navigation method. However, if relative position / attitude calculation is not performed using an autonomous navigation method, an IMU may not be required. Furthermore, only one or two of the gyroscope sensor 106, accelerometer 107, and magnetometer 108 may be included. Additionally, the device connected to the position / attitude estimation device 101 may be... Figure 3 Part of the device shown, or may also have Figure 3 Other devices not shown.

[0055] Figure 4 This diagram illustrates an example of the hardware structure of the position / attitude estimation device 101. The position / attitude estimation device 101 includes a CPU (Central Processing Unit) 1011 as an information processing unit, a memory 1012 as a storage device, and an interface 1013. A 3D map DB 102, a distance sensor 103, a camera 104, a display 105, a gyroscope sensor 106, an accelerometer sensor 107, and a geomagnetic sensor 108 are connected to the CPU 1011 via the interface 1013 and a data bus.

[0056] The functions of the position / attitude estimation device 101 are implemented by processing circuitry. Processing circuitry can be dedicated hardware or a CPU 1011 that executes a program (e.g., a position / attitude estimation program) stored in memory 1012. CPU 1011 can also be any of a processing device, a computing device, a microprocessor, a microcomputer, a processor, or a DSP (Digital Signal Processor).

[0057] When the processing circuit is dedicated hardware, the processing circuit may be a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or any combination of these hardware components.

[0058] When the processing circuit is a CPU 1011, the function of the position / attitude estimation device 101 is implemented by software, firmware, or a combination of software and firmware. The software and firmware are described in the form of programs and stored in the memory 1012. The processing circuit implements the functions of each part by reading and executing the programs stored in the memory 1012. That is, when the position / attitude estimation device 101 is processed by the processing circuit, it executes the position / attitude estimation method of Embodiment 1.

[0059] Here, memory 1012 is, for example, any one of non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), or disk, optical disk, high-density disk, DVD (Digital Versatile Disc).

[0060] Alternatively, a portion of the position / attitude estimation device 101 can be implemented by dedicated hardware, while another portion can be implemented by software or firmware. In this way, the processing circuitry can implement its functions through hardware, software, firmware, or any combination thereof.

[0061] 3D Map Production Device (1-1-2)

[0062] Figure 5 This is a functional block diagram that outlines the structure of the three-dimensional map making apparatus according to Embodiment 1. Figure 5 The illustrated 3D map-making apparatus is capable of executing the 3D map-making method of Embodiment 1. Furthermore, in the following description, an example where the 3D map-making apparatus is part of the position / attitude estimation device 101 (i.e., the map-making registration unit) will be described. However, the 3D map-making apparatus may also be a different device from the position / attitude estimation device 101. In this case, the hardware structure of the 3D map-making apparatus is different from... Figure 4 The hardware structures shown are the same.

[0063] like Figure 5 As shown, the 3D map creation apparatus of Embodiment 1 includes a keyframe detection unit 10, a keyframe position / pose calculation unit 11, a position / pose variance calculation unit 12, a correspondence registration unit 13, and a database storage unit (DB storage unit) 14. These structures utilize a camera 104 ( Figure 3Images captured by the camera, and by the distance sensor 103 ( Figure 3 Distance information obtained by IMU ( Figure 3 A 3D map is constructed using sensor values ​​obtained from the camera. The minimum data required in Implementation 1 is camera images. Even without distance information or an IMU, a 3D map can be created.

[0064] The keyframe detection unit 10 performs the following processing: it detects images (e.g., color images) acquired when the position of the camera 104 has moved by more than a predetermined amount of parallel movement or when the pose of the camera 104 has moved (i.e., rotated) by more than a predetermined amount of rotation, as well as the detected distance information, as keyframes.

[0065] The keyframe position / pose calculation unit 11 performs the following processing: it calculates the position / pose of the camera 104 or distance sensor 103 that captured the keyframe detected by the keyframe detection unit 10 using a calculation method that utilizes the relative position / pose of the image, such as SLAM. The keyframe detection unit 10 and the keyframe position / pose calculation unit 11 perform the same processing as in conventional SLAM technology (e.g., the processing described in Non-Patent Document 3).

[0066] Non-patent literature 3: Raul Mur-Artal, “ORB-SLAM2: An Open-Source SLAMSystem for Monocular, Stereo, and RGB-D Cameras”, IEEE Transactions on Robotics, Vol.33, No.5, October 2017.

[0067] The position / attitude variance calculation unit 12 performs processing to calculate the position / attitude variance for each keyframe detected by the keyframe detection unit 10. For example, the first calculation method and the second calculation method shown below are used to calculate the position / attitude variance.

[0068] The first calculation method is to add noise (i.e., random patterns) to the image. Figure 6 This is a diagram illustrating an example of a method for attaching random patterns to an image of a keyframe. In the first calculation method, the process of attaching random patterns to the image of the keyframe multiple times to calculate the position / pose is performed, and the variance of the position / pose is calculated based on the multiple calculation results obtained.

[0069] The second calculation method uses random numbers. In this method, for example, in the processing of absolute position / attitude calculations, Random Sample Consensus (RANSAC) is used to exclude deviation values. The main purpose of RANSAC is to exclude data from the observed data that deviates from the data used to calculate absolute position / attitude. However, in RANSAC, samples of the calculation objects are randomly selected, and the calculation results are different each time. Therefore, RANSAC can be used to calculate the variance of position / attitude.

[0070] [Formula 1]

[0071] and

[0072] These represent the variances of the positions (σ) tk 2 ) and the variance of attitude (σ) Rk 2 The variance is calculated using equations (1) and (2), respectively. In equations (1) and (2), N is a positive integer representing the number of trials used to calculate the variance. Furthermore, k represents a positive integer.

[0073] [Formula 2]

[0074] t n and R n

[0075] Represent the absolute positions (t) obtained in the nth trial. n ) and absolute posture (R) n Additionally, n is an integer greater than 1 and less than N.

[0076] [Formula 3]

[0077] μ t and μ R

[0078] Representing the average position (μ) t ) and the average of the attitude (μ) R ), which are calculated using equations (3) and (4) respectively.

[0079] [Formula 4]

[0080]

[0081]

[0082]

[0083]

[0084] Figure 7 This diagram illustrates the process of registering the alignment of a 3D map with a 2D map. Figure 7 The plan view is a layout depicting the arrangement of equipment. In the plan view, areas where equipment should be placed, i.e., installation areas, are indicated by dashed rectangles. The correspondence registration unit 13 defines the relationships between the equipment and other three-dimensional maps or the overall map. For example... Figure 7 As shown, when a 3D map (represented by solid-lined rectangles) is registered on the layout of a 2D map, the positional relationships between the 3D maps and the correspondence between the 3D map under construction and the already constructed 3D map are obtained. By registering the 3D map in alignment with the 2D map, the matching between the overall map and the 3D map, or the positional relationships between multiple 3D maps, are defined.

[0085] Database storage unit 14 saves the 3D map (i.e., for example, the data of each keyframe obtained by SLAM) produced by the above method in 3D map DB102. Figure 3 In the 3D map DB102, point clouds containing position / pose, camera image, distance information (i.e., distance image), and local features obtained for each keyframe are stored, corresponding to the number of keyframes.

[0086] Position / Attitude Estimation Device (1-1-3)

[0087] Figure 8 This is a functional block diagram that schematically illustrates the structure of the position / attitude estimation device 101 in Embodiment 1. The position / attitude estimation device 101 performs processing to calculate position / attitude based on multiple sensor data captured from different viewpoints. Here, the sensor data includes camera images and detection data from a distance sensor. Figure 8 As shown, the position / attitude estimation device 101 includes a database reading unit 15, a frame selection unit 16, a relative position / attitude acquisition unit 17, an absolute position / attitude calculation unit 18, and an absolute position / attitude synthesis unit 19.

[0088] The database reading unit 15 reads the three-dimensional map (i.e., the pre-prepared three-dimensional map) stored in the database stored in the DB storage unit 14.

[0089] The frame selection unit 16 performs the following process: selecting frames for calculating position / attitude from frames of camera images captured from multiple different viewpoints. For example, frame selection methods include a first selection method, a second selection method, and a third selection method.

[0090] In the first selection method, relative movement is used. In this method, a decision is made regarding whether to select a frame (keyframe) based on two conditions: first, the change in relative position obtained by calculating the relative position / attitude exceeds a predetermined threshold; and second, the change in relative attitude exceeds a predetermined threshold. For example, a keyframe is selected if at least one of the following conditions is met: the change in relative position obtained by calculating the relative position / attitude exceeds a predetermined threshold; and the change in relative attitude exceeds a predetermined threshold.

[0091] In the second selection method, frames that are different in time are used. In the second selection method, frames from temporally adjacent viewpoints are selected, or frames captured at time intervals exceeding a threshold (keyframes).

[0092] In the third selection method, variance is used. In this method, the data to be used is selected based on the variance calculated during the creation of the 3D map. For example, frames (keyframes) of viewpoints with variances smaller than a predetermined variance threshold are selected.

[0093] The relative position / attitude acquisition unit 17 performs processing to acquire the relative position / attitude corresponding to the frame. The relative position / attitude acquisition unit 17 acquires... Figure 1 The result is obtained by calculating the relative position / attitude using any method.

[0094] The absolute position / attitude calculation unit 18 performs processing to calculate the absolute position / attitude using multiple selected frames. At this time, for example, Perspective n Points (PnP: N-point perspective) is used to calculate the position / attitude.

[0095] The absolute position / attitude integration unit 19 performs processing that integrates the calculation results of multiple positions / attitudes to calculate the final absolute position / attitude. Examples of integration methods performed by the absolute position / attitude integration unit 19 include a first integration method, a second integration method, and a third integration method.

[0096] The first synthesis method uses a "winner takes all" approach. That is, in the first synthesis method, the position / pose estimated from the keyframe with the minimum variance is used as the final result.

[0097] The second synthesis method uses a weighted linear sum. That is, in the second synthesis method, weighting is based on variance.

[0098] [Formula 5]

[0099] and

[0100] The final position and attitude are represented by equations (5) and (6), respectively.

[0101] [Formula 6]

[0102] t′ k and R′ k

[0103] These represent the positions (t′) of the k-th frame in the frames obtained by the frame selection unit 16. k ) and the pose (R′) of the kth frame k ).

[0104] [Formula 7]

[0105] and

[0106] These represent the weights (w, w) for the position of the k-th frame. tk ) and the weights (w) for the pose of the k-th frame Rk ), which are calculated using equations (7) and (8) respectively.

[0107] The weight w for the position of the k-th frame tk and the weight w for the pose of the k-th frame Rk It is the variance σ of the location used. tk 2 The variance σ of the attitude Rk 2 The variance σ at that location is calculated. tk 2 The variance σ of the attitude Rk 2 This is the variance of the keyframes used in position / pose calculations. Alternatively, the variance σ, which can be considered as the variance relative to position, can also be used. tk 2 and the variance of attitude σ Rk 2 The standard deviation of the equivalent statistical indicator, i.e., the standard deviation of the location σ tk and the standard deviation of attitude σ Rk To calculate the weights. Using the standard deviation, the weight w for the position of the k-th frame. tk and the weight w for the pose of the k-th frame Rk The calculations are performed using equations (9) and (10) respectively.

[0108] [Formula 8]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115] In equations (5) and (6), the position t′ of the k-th frame is... k and the pose R′ of the kth frame k Instead of directly inputting the position / attitude calculated by the absolute position / attitude calculation unit 18, the position / attitude calculated from the absolute position / attitude calculation unit 18 is used to move to the position / attitude of any frame. For example, when the frame selection unit 16 selects K frames (K is a positive integer) and performs synthesis in conjunction with the Kth frame, the position t′ of the kth frame (k is a positive integer) is used. k and the pose R′ of the kth frame k It is represented by equation (11). In equation (11), the position t of the k-th frame is... k and the pose R of the kth frame k The position / attitude is calculated by the absolute position / attitude calculation unit 18.

[0116] [Formula 9]

[0117] P k→K and t k→K

[0118] These represent the relative attitude movement and relative position movement from the k-th frame to the k-th frame, respectively, derived from the position / attitude obtained by the relative position / attitude acquisition unit 17. If equation (11) is used, the absolute position / attitude of the k-th frame can be obtained.

[0119] [Formula 10]

[0120]

[0121] The third synthesis method is to obtain the absolute position / attitude through nonlinear optimization. For example, as shown in Equation (12), the position t of the k-th frame is obtained in a way that minimizes the reprojection error. k and the pose R of the kth frame k .

[0122] [Formula 11]

[0123] L represents the camera's internal parameters (L).

[0124] [Formula 12]

[0125] p ki and p′ ki

[0126] These represent the 3D location of the matched local features and the points on the image, respectively.

[0127] N k This represents the number of matched local feature pairs in the k-th frame. w k It is the weight corresponding to the k-th frame, using weight w tk and w Rk Any weight in the formula, or a weight obtained by combining them.

[0128] When equation (12) is solved using nonlinear optimization methods such as the steepest descent method, the absolute position / attitude can be obtained.

[0129] [Formula 13]

[0130]

[0131] Actions 1-2

[0132] Generation of 3D Maps (1-2-1)

[0133] Figure 9 This is a flowchart illustrating an example of the process for creating a three-dimensional map performed by the three-dimensional map-making apparatus of Embodiment 1. For example... Figure 9 As shown, the keyframe detection unit 10 and the keyframe position / pose calculation unit 11 perform the creation of a 3D map (step S101). For example, SLAM is used to perform the creation of a 3D map while detecting keyframes.

[0134] The position / attitude variance calculation unit 12 calculates the position / attitude variance for each keyframe (step S102) (step S103). Next, the correspondence registration unit 13... Figure 7 The correspondence is registered as shown (step S104). The correspondence registration unit 13 performs, for example, the process of registering the 3D map on the planar map and defining the positional relationship between it and the overall map or other 3D maps. The database storage unit 14 performs the process of saving the map created by the above process to the 3D map DB102 (step S105).

[0135] Position / Attitude Estimation (1-2-2)

[0136] Figure 10This is a flowchart illustrating an example of the position / attitude estimation process performed by the position / attitude estimation device 101 of Embodiment 1. The database reading unit 15 reads a 3D map as data from the DB storage unit 14 (step S111). The frame selection unit 16 selects a frame (step S112). The selected frame is a frame processed based on predetermined rules.

[0137] The relative position / attitude acquisition unit 17 acquires the relative position / attitude corresponding to each selected frame (step S113) (step S114). The absolute position / attitude calculation unit 18 calculates the absolute position / attitude based on the data of the selected frames (step S115). When the number of selected frames reaches a predetermined number, the processing enters the comprehensive processing stage.

[0138] The absolute position / attitude integration unit 19 integrates the results of absolute position / attitude based on variance (step S116).

[0139] Figure 11 This is a flowchart illustrating another example of the position / attitude estimation process performed by the position / attitude estimation device of Embodiment 1. The database reading unit 15 reads a 3D map as data from the database storage unit 14 (step S121). The frame selection unit 16 determines whether to select a frame (steps S122, S123), and selects a frame if a frame is selected. The selected frame is a frame processed based on predetermined rules.

[0140] The relative position / attitude acquisition unit 17 performs processing to acquire the relative position / attitude corresponding to the selected frame (step S124). The absolute position / attitude calculation unit 18 calculates the absolute position / attitude for each selected data (step S125). The frame selection unit 16 determines whether sufficient frame detection has been completed (step S126). If sufficient frame detection has been completed, comprehensive processing is performed; if insufficient frame detection has not been completed, processing returns to step S122. "Sufficient frame detection has been completed" means, for example, performing a predetermined number of frame detections or obtaining a predetermined number of absolute positions / attitudes.

[0141] After sufficient frame detection has been completed, the absolute position / attitude synthesis unit 19 synthesizes the results of absolute position / attitude based on variance (step S127).

[0142] Effects of 1-3

[0143] As explained above, the position / attitude estimation apparatus or method according to Embodiment 1 estimates position / attitude based on position / attitude data obtained using multiple images, thereby improving the accuracy of position / attitude estimation.

[0144] Furthermore, by not using the position / attitude obtained from images with large variance in the position / attitude calculation results for comprehensive processing, or by reducing the weight of the position / attitude obtained from images with large variance in the position / attitude calculation results for comprehensive processing, it is possible to calculate the absolute position / attitude with high accuracy.

[0145] Furthermore, even when the subject in the image lacks features, it can achieve high-precision estimation of its own position / pose in the absolute coordinate system, and the computational load can be reduced to a level that allows for real-time processing.

[0146] Implementation Method 2

[0147] Structure 2-1

[0148] 2-1-1

[0149] In Embodiment 1, a method for calculating the absolute position / attitude using camera images was described, comprising a first estimation method using direct matching and a second estimation method using two-stage matching. Furthermore, an example using the second estimation method was described in Embodiment 1. In Embodiment 2, the following methods were described: a method for synthesizing the position / attitude obtained using the direct matching method with camera images, and a method for synthesizing the position / attitude using a laser sensor such as LiDAR (Light Detection and Range).

[0150] The hardware structure of the position / attitude estimation device and position / attitude estimation system in Implementation 2 is the same as that in Implementation 1 ( Figure 4 and Figure 5 The hardware structure described in [the previous section] is the same. Therefore, when describing Implementation 2, reference will also be made to [the previous section]. Figure 4 and Figure 5 .

[0151] 2-1-2 3D Map Making Device

[0152] Figure 12 This is a functional block diagram that outlines the structure of the three-dimensional map making apparatus according to Embodiment 2. Figure 12 The 3D map making apparatus shown is an apparatus capable of executing the 3D map making method of Embodiment 2. Furthermore, in the following description, the 3D map making apparatus is described as part of a position / attitude estimation apparatus (i.e., a map making registration unit), but the 3D map making apparatus may also be an apparatus different from a position / attitude estimation apparatus.

[0153] like Figure 12As shown, the three-dimensional map making apparatus of Embodiment 2 includes a three-dimensional map making unit 21, a position / attitude variance calculation unit 22, a correspondence registration unit 23, and a database storage unit (DB storage unit) 24.

[0154] In Implementation 1, the 3D data is managed on a per-keyframe basis, but in Implementation 2, the point cloud is managed as a 3D map. For example, when using an image, the 3D map creation unit 21 creates a 3D map from local features and their locations obtained from the image. When using a laser sensor such as LiDAR, the observed point cloud (location only) is used to create a 3D map. Figure 12 The 3D map production department 21 processes the aforementioned 3D map data to produce the map.

[0155] Figure 13 This is a diagram illustrating the method for calculating variance used in the three-dimensional map making apparatus of Embodiment 2. Figure 13 This illustrates an example where, when robot 131, equipped with an AGV, moves, the robot estimates its position / attitude using both relative and absolute position / attitude calculations. Robot 131 calculates variance; however, the method for calculating variance differs from that in Embodiment 1. Figure 13 The calculation of the variances σ1, σ2, and σ3 for each region (e.g., region #1, #2, and #3) enclosed by an ellipse is explained.

[0156] In the specific variance calculation, an arbitrary number of data points are obtained (sampled) from the data observed during the creation of the 3D map, for each region. For the sampled data, noise is added multiple times using different methods to estimate the absolute position / orientation. The variance to be calculated is the variance of the position / orientation for each specific region. The method of adding noise is the same as in Implementation Method 1; in the case of images, it is the method of adding a random pattern. Similarly, in the case of LiDAR, noise is added by adding a random pattern to local regions. Here, the "random pattern" also includes a pattern used to remove data from that region.

[0157] The correspondence registration unit 23 defines the correlation with the overall map or other three-dimensional maps in the same way as the correspondence registration unit 13 in embodiment 1.

[0158] DB storage unit 24 stores the 3D map and the variance of each region in the database.

[0159] Position / Attitude Estimation Device (2-1-3)

[0160] Figure 14 This is a functional block diagram that outlines the structure of the position / attitude estimation device in Embodiment 2. Figure 14 The position / attitude estimation device shown is an apparatus capable of executing the position / attitude estimation method of Embodiment 2. The position / attitude estimation device includes a database reading unit 25, a frame selection unit 26, a relative position / attitude acquisition unit 27, an absolute position / attitude calculation unit 28, and an absolute position / attitude synthesis unit 29.

[0161] The database reading unit 25 processes the 3D map data stored in the database.

[0162] The frame selection unit 26 selects frames in the same way as the frame selection unit 16 in Embodiment 1, or selects frames from a pre-divided plurality of regions in a non-repeating manner. For example, as Figure 13 The example shown illustrates the calculation and management of variances σ1, σ2, and σ3 for regions #1, #2, and #3. The example further illustrates the calculation of position / attitude, where regions #1, #2, and #3 each contain M frames.

[0163] The relative position / attitude acquisition unit 27 acquires the relative position / attitude using the same method as the relative position / attitude acquisition unit 17 in Embodiment 1.

[0164] In the case of an image, the absolute position / attitude calculation unit 28 calculates the absolute position / attitude using a method that directly matches local features of the image (for example, refer to Non-Patent Document 4). When using data from a LiDAR-based distance sensor, the position / attitude is calculated by matching a 3D map with shape information observed by the distance sensor.

[0165] Non-patent literature 4: Torsten Sattler, two other famous works, “Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.39, No.9, September 2017.

[0166] The absolute position / attitude integration unit 29 integrates multiple positions / attitudes using the same method as the absolute position / attitude integration unit 19 in Embodiment 1. The absolute position / attitude integration unit 29 calculates the final position / attitude based on the variance set according to each region.

[0167] 2-2 Action

[0168] The creation of 3D maps in "2-2-1"

[0169] Figure 15 This is a flowchart illustrating an example of the processing performed by the 3D map-making apparatus of Embodiment 2 for creating a 3D map. The 3D map-making unit 21 and the position / attitude variance calculation unit 22 create a 3D map. When using images, a map representing local features and their locations is created; when using LiDAR, a map representing point clouds (location only) is created (step S201). The correspondence registration unit 23 registers correspondences as processes corresponding to those of the 3D map-making unit 21 (step S202). The database storage unit 24 performs the process of saving the correspondences in a database (step S203).

[0170] Estimation of position / attitude (2-2-2)

[0171] Figure 16 This is a flowchart illustrating an example of the position / attitude estimation process performed by the position / attitude estimation device of Embodiment 2. The database reading unit 25 performs data reading from the database (step S211). The relative position / attitude acquisition unit 27 acquires the relative position / attitude, and the absolute position / attitude calculation unit 28 performs the absolute position / attitude calculation (steps S212 to S214).

[0172] The frame selection unit determines whether frame selection is needed (step S215). If needed, it determines whether sufficient frame detection has been completed (step S216). If sufficient frame detection has been completed, the absolute position / attitude synthesis unit 29 performs absolute position / attitude synthesis (step S217).

[0173] Effects of 2-3

[0174] As explained above, the position / attitude estimation apparatus or method according to Embodiment 2 can improve the accuracy of position / attitude estimation when using images and a direct matching method, or when calculating absolute position / attitude using shapes observed by LiDAR, because multiple frames are used.

[0175] 3. Implementation Method 3

[0176] Structure 3-1

[0177] 3-1-1

[0178] In AR, terminals or robots sometimes manage relative position / pose within the device. In the case of AR, the content is overlaid on the image by converting the position of content defined in absolute position / pose into a coordinate system of relative position / pose managed by the terminal. Similarly, in the case of robots, where the robot's destination is defined in absolute position, that position needs to be converted into a coordinate system of relative position / pose managed by the robot.

[0179] Implementation 3 describes a method for calculating extrinsic parameters with high precision using multiple frames. These extrinsic parameters are matrices representing the transformation from an absolute position / attitude coordinate system to a relative position / attitude coordinate system. Here, the method for calculating extrinsic parameters with high precision using multiple frames is described based on Implementation 1. The structure of Implementation 3 can also be combined with Implementation 2 or 4.

[0180] The hardware structure of the position / attitude estimation device and position / attitude estimation system in Implementation 3 is the same as that in Implementation 1 ( Figure 4 and Figure 5 The hardware structure described in [the document] is the same. Therefore, when describing implementation method 3, reference is also made to [the document]. Figure 4 and Figure 5 .

[0181] 3-1-2 Three-Dimensional Map Making Device

[0182] The three-dimensional map making apparatus of Embodiment 3 is the same as that of Embodiment 1.

[0183] Position / Attitude Estimation Device (3-1-3)

[0184] Figure 17 This is a functional block diagram that outlines the structure of the position / attitude estimation device in Embodiment 3. Figure 17 The position / attitude estimation device shown is an apparatus capable of executing the position / attitude estimation method of Embodiment 3. The position / attitude estimation device includes a database reading unit 35, a frame selection unit 36, a relative position / attitude acquisition unit 37, an absolute position / attitude calculation unit 38, an external parameter calculation unit 38a, and an external parameter synthesis unit 39, which serves as an absolute position / attitude synthesis unit.

[0185] The difference between Embodiment 3 and Embodiment 1 is that Embodiment 3 includes an external parameter calculation unit 38a that calculates external parameters according to each selected frame, and an external parameter synthesis unit 39 that synthesizes multiple external parameters. Other than these, the processing in Embodiment 3 is the same as that in Embodiment 1.

[0186] The extrinsic parameter calculation unit 38a calculates the extrinsic parameters by assuming that the relative position / attitude and the absolute position / attitude are calculated based on the same frame. At this time, the extrinsic parameters are calculated by equation (13).

[0187] [Formula 14]

[0188] and

[0189] This indicates the absolute position and absolute orientation obtained using the k-th frame.

[0190] [Formula 15]

[0191] and

[0192] This represents the relative position and relative orientation of the k-th frame.

[0193] [Formula 16]

[0194] t′ k and R′ k The extrinsic parameter (t′) of the k-th frame is represented. k and R′ k ), which is calculated using equation (13).

[0195] [Formula 17]

[0196]

[0197] The processing of external parameter integration unit 39 and the absolute position / attitude integration unit 19 of implementation method 1 ( Figure 8 The results are roughly the same. Therefore, the external parameter synthesis unit 39 is also called the absolute position / attitude synthesis unit. In Embodiment 1, multiple absolute positions / attitudes are synthesized, but in Embodiment 3, external parameters are synthesized. Specifically, t′ obtained by equation (13) is used. k and R′ k Synthesize. For example, in the case of synthesis by weighted linear sum, t′ of equation (13) is... k 、R′ k Substitute these equations into equations (5) and (6) for synthesis.

[0198] Action 3-2

[0199] The creation of 3D maps in "3-2-1"

[0200] The processing flow for creating 3D maps is the same as in Implementation Method 1, and therefore will be omitted. The processing flow for the method of integrating external parameters will be described.

[0201] Estimation of position / attitude in 3-2-2

[0202] Figure 18 This is a flowchart illustrating an example of the position / attitude estimation process performed by the position / attitude estimation device of Embodiment 3. The database reading unit 35 reads data from the database (step S301). The relative position / attitude acquisition unit 37 acquires the relative position / attitude (steps S302, S303). The absolute position / attitude calculation unit 38 calculates the absolute position / attitude (step S304). The external parameter calculation unit 38a calculates the external parameters (step S305).

[0203] The frame selection unit 36 ​​determines whether frame selection is needed (step S306). If needed, it determines whether sufficient frame detection has been completed (step S307). If sufficient frame detection has been completed, the external parameter synthesis unit 39 performs absolute position / attitude synthesis (step S308).

[0204] Effect of 3-3

[0205] As explained above, the position / attitude estimation device or method according to Embodiment 3 can accurately determine the transformation matrix from the absolute position / attitude coordinate system to the relative position / attitude coordinate system, thus enabling high-precision display of content on AR-enabled terminals. Furthermore, when applied to robots, it can accurately determine the robot's destination.

[0206] Implementation Method 4

[0207] Structure 4-1

[0208] 4-1-1

[0209] In Embodiment 4, an error handling method using multiple frames is described incorporating the position / pose estimation method of Embodiment 1. In the calculation of absolute position / pose using images, when there are few characteristic patterns in the subject, the output position / pose value may contain large errors. To eliminate such results, the position / pose estimation apparatus of Embodiment 4 performs error handling.

[0210] The hardware structure of the position / attitude estimation device and position / attitude estimation system in Implementation 4 is basically the same as that in Implementation 1. Figure 4 and Figure 5 The hardware structure described in [the previous section] is the same. Therefore, when describing implementation method 4, reference is also made to [the previous section]. Figure 4 and Figure 5 .

[0211] 4-1-2 3D Map Production Device

[0212] The structure of the three-dimensional map making device in Embodiment 4 is the same as that in Embodiment 1.

[0213] Position / Attitude Estimation Device (4-1-3)

[0214] Figure 19 This is a functional block diagram that outlines the structure of the position / attitude estimation device in Embodiment 4. Figure 19 The position / attitude estimation device shown is capable of executing the position / attitude estimation method of Embodiment 4. The position / attitude estimation device of Embodiment 4 differs from Embodiment 1 in that it includes an error processing unit 48a.

[0215] like Figure 19 As shown, the position / attitude estimation device of Embodiment 4 includes a database reading unit 45, a frame selection unit 46, a relative position / attitude acquisition unit 47, an absolute position / attitude calculation unit 48, an error processing unit 48a, and an absolute position / attitude synthesis unit 49. The database reading unit 45, frame selection unit 46, relative position / attitude acquisition unit 47, absolute position / attitude calculation unit 48, and absolute position / attitude synthesis unit 49 are... Figure 8 The database reading unit 15, frame selection unit 16, relative position / attitude acquisition unit 17, absolute position / attitude calculation unit 18, and absolute position / attitude synthesis unit 19 shown are the same.

[0216] Error processing unit 48a performs error processing. In error processing, multiple calculation results of absolute position / attitude obtained through multiple frames are compared, thereby excluding calculation results of absolute position / attitude with errors greater than a predetermined threshold from the comprehensive processing. Equations (14) and (15) show an example of error processing. The j-th frame (j is a positive integer less than or equal to K) is the frame that satisfies j ≠ k among the K frames selected by frame selection unit 46. Multiple calculation results of absolute position / attitude obtained through multiple frames should indicate the same position / attitude. Therefore, error processing unit 48a compares the calculation result of absolute position / attitude obtained through a certain frame with the calculation results of absolute position / attitude obtained through other frames. Error processing unit 48a excludes the calculation result of the frame if the difference in calculated position is greater than a predetermined threshold th_t (i.e., if equation (14) is satisfied) or if the difference in calculated attitude is greater than a predetermined threshold th_r (i.e., if equation (15) is satisfied).

[0217] [Formula 18]

[0218] ||t′ k -t′ j ||>th_t (14)

[0219]

[0220] Action 4-2

[0221] The creation of 3D maps in "4-2-1"

[0222] The operation of the three-dimensional map making device in Embodiment 4 is the same as that in Embodiment 1.

[0223] 4-2-2 Position / Attitude Estimation

[0224] Figure 20 This is a flowchart illustrating another example of the processing performed by the position / attitude estimation device of Embodiment 4 for estimating position / attitude. Figure 20 The position / attitude estimation device shown in the diagram adds error handling (step S406a) in this respect. Figure 11 The position / attitude estimation device shown in Implementation 1 operates differently. Figure 20 The processing of steps S401 to S406 and the processing of step S407 shown are... Figure 11 The processes shown in steps S101 to S107 are the same.

[0225] Effect of 4-3

[0226] As explained above, the position / attitude estimation device or method according to Embodiment 4 performs error processing, thus enabling the estimation of absolute position / attitude with higher environmental resistance than that in Embodiment 1 (i.e., it can achieve higher estimation accuracy of absolute position / attitude for various environments).

[0227] Explanation of reference numerals in the attached figures

[0228] 10 Keyframe Detection Unit, 11 Keyframe Position / Attitude Calculation Unit, 12, 22 Position / Attitude Variance Calculation Unit, 13, 23 Correspondence Registration Unit, 14, 24 Database Storage Unit, 15, 25, 35, 45 Database Read-in Unit, 16, 26, 36, 46 Frame Selection Unit, 17, 27, 37, 47 Relative Position / Attitude Acquisition Unit, 18, 28, 38, 48 Absolute Position / Attitude Calculation Unit, 19, 29, 49 Absolute Position / Attitude Integration Unit, 21 3D Map Production Unit, 38a External Parameter Calculation Unit, 39 External Parameter Integration Unit (Absolute Position / Attitude Integration Unit), 48a Error Handling Unit, 100 Position / Attitude Estimation System, 101 Position / Attitude Estimation Device, 102 3D Map Database, 103 Distance Sensor, 104 Camera, 105 Display, 106 Gyroscope Sensor, 107 Accelerometer Sensor, 108 Geomagnetic Sensor.

Claims

1. A position / attitude estimation device, comprising: The database reading section reads 3D map data from the database. The frame selection unit performs the process of selecting frames from images captured from multiple different viewpoints for calculating position / attitude. The relative position / attitude acquisition unit performs processing to acquire multiple relative positions / attitudes with respect to multiple frames selected by the frame selection unit; An absolute position / attitude calculation unit performs processing that uses the 3D map to calculate multiple absolute positions / attitudes for multiple frames selected by the frame selection unit; as well as The absolute position / attitude synthesis unit synthesizes the relative position / attitude obtained by the relative position / attitude acquisition unit and the absolute position / attitude obtained by the absolute position / attitude calculation unit to obtain the final absolute position / attitude.

2. The position / attitude estimation device according to claim 1, wherein, The position / attitude estimation device also includes a map creation and registration unit. The map production and registration department has: The keyframe detection unit detects keyframes from camera images captured by the camera. The keyframe position / pose calculation unit calculates the position and pose of the camera that captured the keyframe; The position / attitude variance calculation unit performs the processing of calculating the position / attitude variance for each keyframe; The Correspondence Registration Department processes the registration data by aligning the 3D map with the 2D map. as well as The database storage unit performs the process of storing the registration data in the database.

3. The position / attitude estimation device according to claim 2, wherein, The absolute position / attitude synthesis unit synthesizes the plurality of absolute positions / attitudes based on the variance of the position / attitude calculated according to each of the keyframes.

4. The position / attitude estimation device according to claim 3, wherein, The absolute position / attitude synthesis unit uses the position / attitude estimated based on the keyframe with the smallest variance among the keyframes as the final absolute position / attitude.

5. The position / attitude estimation device according to claim 3, wherein, The absolute position / attitude synthesis unit calculates weights based on the variance for each keyframe and synthesizes the plurality of absolute positions / attitudes based on a weighted linear sum using the weights.

6. The position / attitude estimation device according to claim 2, wherein, The absolute position / attitude synthesis unit uses nonlinear optimization to synthesize the multiple absolute positions / attitudes.

7. The position / attitude estimation device according to claim 1, wherein, The position / attitude estimation device also includes a map creation and registration unit. The map production and registration department has: The 3D map production department creates 3D maps based on camera images captured by cameras or distance information measured by distance sensors, according to each local area. The position / attitude variance calculation unit performs the processing of calculating the position / attitude variance based on the camera image or the distance information and according to each region; The Correspondence Registration Department processes the registration data by aligning the 3D map with the 2D map. as well as The database storage unit performs the process of storing the registration data in the database.

8. The position / attitude estimation device according to claim 7, wherein, The absolute position / attitude synthesis unit synthesizes the plurality of absolute positions / attitudes based on the variance of the position / attitude of each of the regions calculated according to each region.

9. The position / attitude estimation device according to claim 2, wherein, The position / attitude estimation device also includes an extrinsic parameter calculation unit, which calculates extrinsic parameters for each keyframe. The absolute position / attitude synthesis unit synthesizes multiple external parameters calculated by the external parameter calculation unit, thereby synthesizing the multiple absolute positions / attitudes.

10. The position / attitude estimation device according to claim 9, wherein, The absolute position / attitude synthesis unit synthesizes the plurality of external parameters based on the variance of the position / attitude calculated according to each of the keyframes.

11. The position / attitude estimation device according to any one of claims 2 to 6, 9, and 10, wherein, The position / attitude estimation device also has an error processing unit that, if the error of the absolute position / attitude calculated according to each keyframe is greater than a predetermined threshold, will not use the calculation result of the absolute position / attitude with an error greater than the threshold for comprehensive processing.

12. A position / attitude estimation method, which is performed by a position / attitude estimation device, wherein, The position / attitude estimation method includes the following steps: Import 3D map data from the location database; The process involves selecting frames from images captured from multiple different viewpoints for calculating position / pose. Perform processing to obtain multiple relative positions / poses of the selected multiple frames; The process involves using the 3D map to calculate multiple absolute positions / poses with respect to the selected multiple frames; as well as The obtained relative position / attitude and the obtained absolute position / attitude are combined to obtain the final absolute position / attitude.

13. A recording medium that records a program, wherein, The program causes the computer to perform the following steps: Import 3D map data from the location database; The process involves selecting frames from images captured from multiple different viewpoints for calculating position / pose. Perform processing to obtain multiple relative positions / poses of the selected multiple frames; The process involves using the 3D map to calculate multiple absolute positions / poses with respect to the selected multiple frames; as well as The obtained relative position / attitude and the obtained absolute position / attitude are combined to obtain the final absolute position / attitude.

Images