Program, information processing device, and information processing method

Abstract

Even under conditions involving scale drift, it enables reliable loop closing and prevents the breakdown of the pause graph. [Solution] The information processing device 1 includes an optimization execution unit 113 that performs pose graph optimization by adding loop-closing edges that are candidates for loop closure to the pose graph and calculates the relative pose between nodes (e.g., Sim3), an evaluation unit 114 that projects the map points associated with one node onto the image coordinate system of the other node using the calculated relative pose and evaluates the geometric consistency with the observation data, a determination unit 115 that determines the effectiveness of the loop-closing edges based on the evaluation results, and an update control unit 116 that controls the updating of the pose graph based on the determination results.

Description

[Technical Field]

[0001] This disclosure relates to a technology for estimating the position and orientation of a moving object. In particular, it relates to a loop closure technology in SLAM (Simultaneous Localization and Mapping) using sensors such as cameras. [Background technology]

[0002] In recent years, Visual SLAM (V-SLAM) technology, which uses cameras to simultaneously map the environment and estimate its own position, has been widely used in autonomous vehicles, drones, AR / VR devices, and other applications. V-SLAM constructs a "pose graph" where the trajectory of a moving object is represented by nodes (keyframes) and constraints between nodes are represented by edges. By optimizing this graph, cumulative errors are corrected.

[0003] In particular, "loop closing," which detects when a previously visited location has been revisited, is important for maintaining the accuracy of SLAM because it can eliminate accumulated drift errors. However, SLAM using a monocular camera has the problem of "scale drift," where the actual scale (absolute size) is not fixed. Therefore, when closing loops, optimization that takes into account not only position (rotation and translation) but also differences in scale (for example, similarity transformation such as Sim3 optimization) is necessary.

[0004] Conventionally, loop-closing techniques that take such scales into consideration include, for example, ORB-SLAM described in Non-Patent Document 1. These methods detect loop candidates based on feature point matching in the image and add loop-closing edges to the pose graph after geometric verification. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Raul Mur-Artal, JMM Montiel, and Juan D. Tardos. "ORB-SLAM: A Versatile and Accurate Monocular SLAM System". IEEE Transactions on Robotics, vol. 31, no. 5, pp. 1147-1163, 2015. [Overview of the project] [Problems that the invention aims to solve]

[0006] However, optimizing a 7-degree-of-freedom (Sim3) model, including scaling, is computationally complex and prone to local minima depending on the initial settings. If loop-closing edges are added based on incorrect correspondences (e.g., similar but different locations) or if optimization fails and edges are added while scaling is broken, the overall map of the pose graph becomes significantly distorted, drastically reducing the accuracy of SLAM.

[0007] Furthermore, robust optimization methods (such as Switchable Constraints and Dynamic Covariance Scaling) have been proposed to dynamically reduce the weights of constraints with large errors within the pause graph optimization process. However, these existing methods merely numerically adjust the weights of individual edges during the optimization calculation process and cannot evaluate whether the resulting map maintains the correct shape as a whole (global consistency). Therefore, even if local numerical consistency is ensured, it is sometimes not possible to completely prevent situations where the shape of the map as a whole is distorted (for example, buildings are generated twice, or pathways are twisted).

[0008] This disclosure has been made in view of the above-mentioned problems, and aims to provide a program, an information processing device, and an information processing method that can achieve highly reliable loop closing and prevent the breakdown of the pause graph, even under conditions involving scale drift. [Means for solving the problem]

[0009] In one embodiment, the present invention provides a program for a computer to perform the following steps: acquire a plurality of observational data indicating shape, each observed at a different time by sensors mounted on a moving body; construct a pose graph including a plurality of nodes, each indicating the position and orientation of the moving body at the time of observation of the shape indicated by any of the plurality of observational data, and odometry edges connecting temporally consecutive nodes among the plurality of nodes; perform pose graph optimization by adding loop-closing edges to the pose graph based on the plurality of observational data; evaluate the geometric consistency between the shapes indicated by the observational data of two nodes connected by each of the one or more edges included in the pose graph after the pose graph optimization has been performed; and update the pose graph with content corresponding to the evaluated geometric consistency. [Effects of the Invention]

[0010] According to this disclosure, after calculating the relative orientation (Sim3, etc.) including the scale, the map points of one node are reprojected onto the image of the other node using that relative orientation, and the geometric consistency with the actual observation data is evaluated. In particular, unlike existing methods that are limited to numerical adjustments within the optimization process, the present invention distinguishes between deformation (acceptable distortion) and failure (unacceptable inconsistency) "after" the optimization calculation has been performed, and determines the success or failure of loop closure based on the ratio between them. This allows for a rigorous, comprehensive verification of not only the similarity of how the images appear, but also the three-dimensional positional relationship including the scale and the consistency of the entire map, thereby effectively preventing the failure of the pose graph due to incorrect loop closure. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing the hardware configuration of the information processing device 1. [Figure 2] A diagram showing the data structure of observation data 121. [Figure 3] A diagram showing the data structure of map data 122. [Figure 4] A diagram showing the data structure of graph data 123. [Figure 5] A block diagram showing the functional configuration of the information processing apparatus 1. [Figure 6] A schematic diagram showing the structure of a pose graph. [Figure 7] A flow diagram showing an example of a processing procedure. [Figure 8] A schematic diagram showing the principle of geometric consistency evaluation. [Figure 9] A diagram showing criteria for edge validity determination. [Figure 10] A diagram showing an example of the relationship between cumulative error and a determination threshold value. [Mode for Carrying Out the Invention]

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the same reference numerals are assigned to the same or corresponding parts in each drawing.

[0013] 1. Configuration Figure 1 is a block diagram showing the hardware configuration of the information processing apparatus 1. The information processing apparatus 1 is, for example, a computer, and includes a processor 11, a memory 12, and an interface 13. These components are connected to each other so as to be capable of mutual communication via a bus.

[0014] The processor 11 is an arithmetic unit configured by a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0015] The memory 12 is a storage device such as a read-only memory (ROM) or a random access memory (RAM). The memory 12 stores various types of data required for processing by the processor 11. Specifically, the memory 12 holds observation data 121, map data 122, graph data 123, and an exclusion list 124.

[0016] Interface 13 is an input / output circuit for connecting to external devices. Sensor 2 is connected to the information processing device 1 via interface 13. Sensor 2 is a camera, distance measuring sensor, etc., mounted on a mobile device and used to observe the shapes of objects and other things in the surrounding environment of the mobile device.

[0017] Figure 2 shows the data structure of the observation data 121. The observation data 121 has a table structure that manages information related to each time-series frame (keyframe) acquired by the sensor 2. The observation data 121 stores "image data," "2D feature point list," "descriptor list," and "corresponding map point index" associated with a "frame ID" that uniquely identifies each record.

[0018] "Frame ID" is an identifier assigned according to the order of observation (e.g., F1, F2, etc.). "Image data" is the image data itself captured by sensor 2, or reference information indicating its storage location. "2D feature point list" is the image coordinates (e.g., q) of the feature points extracted from the image data of the frame. 1,1 =(u1,v1) etc.) is a set of. The "descriptor list" is vector data (descriptors, e.g., descriptors) that represent the local image features of each feature point. d 1,1 This is a set of (etc.). This descriptor is used in a matching process to identify identical feature points across different frames. The "corresponding map point index" indicates the identifier (e.g., p1) of the 3D map point associated with each 2D feature point. If a corresponding map point does not exist (e.g., an unobserved point or an outlier), an invalid value (e.g., -1) is set. In other words, the multiple records that make up the observation data 121 are examples of multiple observation data showing shape, each observed by the sensor at a different time.

[0019] Figure 3 shows the data structure of map data 122. Map data 122 is a table that manages information on map points (landmarks) that represent the three-dimensional shape of the environment. Map data 122 has a "map point ID (p iThe "reference frame," "local 3D coordinates," and "descriptor" are stored in association with the "reference frame."

[0020] The "reference frame" is the identifier of the keyframe that serves as the basis for defining the coordinates of the map point (e.g., F1, F2, etc.). The "local 3D coordinates" are relative 3D coordinates in the camera coordinate system of the corresponding reference frame (e.g., (x1, y1, z1)), rather than in the world coordinate system. In this embodiment, by storing the map point as such local coordinates, the absolute pose of the reference frame (T) can be determined by pose graph optimization. k Even if the map data is updated, the geometric relationship between the node and the map point can be maintained without recalculating the coordinate values ​​of the map point. The information processing device 1 of this disclosure may convert the map points in the world coordinate system to the local coordinate system of each keyframe and store them before performing the pause graph optimization. A "descriptor" is vector data of the feature quantities associated with the map point and is used for matching with the observation data 121, etc. A trained neural network (e.g., CNN) may be used to extract the descriptors.

[0021] Figure 4 shows the data structure of graph data 123. Graph data 123 is a table for managing the elements that make up the pause graph, and includes "node list 1231" and "edge list 1232".

[0022] Node list 1231 contains information about each keyframe that will be a node (vertex) in the pose graph. This manages the node list 1231, which stores a node with a unique "node ID," an "absolute orientation" indicating the three-dimensional position and orientation of each node in the reference coordinate system, and a "timestamp" indicating the time the node was created. Here, the node ID (F1, F2, etc.) corresponds to the frame ID in the observation data 121. The absolute orientation (T1, T2, etc.) is updated as needed according to the results of the pose graph optimization. The timestamp is used to identify the generation order between nodes, determine temporally consecutive nodes, or limit the search range for loop closing candidates. Here, "limiting the search range" means, for example, targeting nodes that have been active for a predetermined amount of time or more from the current time.

[0023] The edge list 1232 manages the connection relationships (edges) between nodes. The edge list 1232 stores the "edge ID" that identifies each edge, the "source" which is the starting point of the edge, the "target" which is the ending point, the "type" of the edge, and the "relative orientation" which is the geometric constraint between two nodes, associating them with each other.

[0024] The "type" includes "odometry," which connects temporally consecutive frames, and "loop closing," which is generated when a location is revisited. The illustrated edge ID "E LC This indicates that it is a loop-closing edge from node F10 (source) to node F1 (target).

[0025] The "relative orientation" is the coordinate transformation matrix from the source node to the target node. In particular, the relative orientation of a loop closing edge (S10,1, etc.) includes a scale component in addition to position and orientation. s,t ) is stored.

[0026] Figure 5 is a block diagram showing the functional configuration of the information processing device 1. The processor 11 of the information processing device 1 functions as a data acquisition unit 111, a coordinate transformation unit 112, an optimization execution unit 113, an evaluation unit 114, a determination unit 115, and an update control unit 116 by reading and executing a program stored in the memory 12.

[0027] The data acquisition unit 111 acquires observation data such as image data from the sensor 2 (see Figure 1) via the interface 13 and stores it in the memory 12 as observation data 121. The data acquisition unit 111 also performs processing to extract feature points and descriptors from the image.

[0028] The coordinate transformation unit 112 restores the three-dimensional position of the extracted feature points, converts them into local coordinates based on the camera coordinate system of the corresponding keyframe, and registers them in the map data 122.

[0029] The optimization execution unit 113 performs optimization calculations on the pose graph. In particular, if an edge that is a candidate for loop closure is detected, the optimization execution unit 113 calculates the relative pose between the two nodes (source node and target node) at both ends of the edge as a Sim3 transformation (similarity transformation) including scale.

[0030] The evaluation unit 114 verifies the geometric consistency of the edges of the loop-closing candidates. Specifically, the evaluation unit 114 reprojects the map points associated with the source node onto the image coordinate system of the target node using the calculated Sim3 relative pose. Then, the evaluation unit 114 calculates the reprojection error between the reprojected points and the feature points actually observed at the target node. In order to reduce the computational load, the evaluation unit 114 may evaluate the geometric consistency not by evaluating all edges included in the pose graph, but by selecting edges on the closed-loop path formed by the loop-closing edges, or edges within a predetermined range from the closed-loop path.

[0031] The determination unit 115 determines the effectiveness of the loop closing edge based on the evaluation result from the evaluation unit 114. The determination unit 115 calculates the percentage of map points where the reprojection error exceeds the allowable value for each edge (outlier rate), and classifies the state of each edge (normal, deformed, broken, etc.) based on this outlier rate. The determination unit 115 then calculates the ratio of the number of broken edges to the total number of deformed edges, and determines whether this ratio is below a threshold.

[0032] The update control unit 116 controls the updating of the graph data 123 based on the determination result of the determination unit 115. If the determination is valid, the update control unit 116 adds the edge to the pause graph. If the determination is invalid, the update control unit 116 discards the edge and registers it in the exclusion list 124. Combinations registered in the exclusion list 124 are excluded from the candidates for loop closing in subsequent processing.

[0033] Figure 6 is a schematic diagram illustrating the structure of a pause graph. The pause graph includes nodes F1 to F10, which are generated in chronological order along the trajectory of the moving object, and odometry edges E1 to E9, which connect temporally consecutive nodes. Note that the number of nodes and edges shown in Figure 6 is merely an example for illustrative purposes and is not limited to this. In reality, a large number of nodes (e.g., hundreds to tens of thousands) can be generated depending on the distance and time traveled by the moving object.

[0034] The example shown in Figure 6 illustrates a situation where similarity has been detected between the latest node F10 (target Ft) and the previous node F1 (source Fs). Between node F10 and node F1 is an edge E that is a candidate for loop closure. LC (Dashed line) is set. This edge E LC As a geometric constraint from node F10 to node F1, the Sim3 relative orientation (S10,1) including the scale component is associated.

[0035] 2.Operation Figure 7 is a flowchart showing an example of the processing flow. In step S101, the data acquisition unit 111 acquires the observation data of the current frame from the sensor 2.

[0036] In step S102, the coordinate transformation unit 112 generates three-dimensional map points from the observation data, transforms them into a local coordinate system based on the current frame, and registers them in the map data 122.

[0037] In step S103, the processor 11 searches for past keyframes (loop closing candidates) similar to the current frame using image search with descriptors or the like.

[0038] If no candidates are found in step S104 (NO), the process proceeds to step S113. In step S113, the update control unit 116 updates the pose graph using only the odometry constraints between it and the previous keyframe, and then terminates processing for that frame.

[0039] On the other hand, if a candidate is found in step S104 (YES), the process proceeds to step S105. In step S105, the optimization execution unit 113 performs Sim3 optimization (similarity transformation) on the local pose graph containing the candidate edge and calculates the Sim3 relative pose from the source frame to the target frame. This is because absolute scale cannot be observed in SLAM using a monocular camera, and therefore it is necessary to correct for scale drift associated with loop closure. In the following step S106, the determination unit 115 estimates the magnitude of the cumulative error (or the amount of deformation of the pose graph) estimated by pose graph optimization.

[0040] In step S107, the evaluation unit 114 uses the calculated Sim3 relative pose to perform an evaluation of geometric consistency. In this evaluation, the map points associated with the source frame are reprojected onto the image plane of the target frame, the error with the observation points is calculated, and the outlier rate R for each edge is calculated.

[0041] Figure 8 is a schematic diagram illustrating the principle of geometric consistency evaluation. Figure 8 shows the geometric relationship between the source keyframe (Fs) and the target keyframe (Ft) for loop closing.

[0042] The source frame (Fs) holds 3D map points (Pi) defined in its own local coordinate system. The coordinates of these map points are represented by vector (P local ). The Sim3 relative pose (S s,t [[ID=]]) represents a coordinate transformation including scale change from the source frame (Fs) to the target frame (Ft). The evaluation unit 114 uses this relative pose (S s,t ) to project the map points (Pi) onto the image plane of the target frame (Ft).

[0043] On the image plane, the point (u1) indicated by a white circle is a reprojection point obtained by calculation. On the other hand, the point (u) indicated by a black circle is a feature point actually observed in the target frame (Ft). The evaluation unit 114 calculates the distance between the reprojection point (u1) and the observation point (u) as the reprojection error (e). This "distance" is, for example, the square of the Euclidean distance. Note that in error evaluation, a Mahalanobis distance or the like that takes into account the covariance matrix Σ corresponding to the scale level of each feature point may be used. This error (e) serves as an index for evaluating the validity of the estimated loop closure.

[0044] In step S108, the determination unit 115 classifies the state of each edge based on the calculated outlier rate R.

[0045] FIG. 9 is a diagram illustrating criteria for edge validity determination. As shown in FIG. 9, the determination unit 115 determines an edge as a "normal edge" if the outlier rate R is equal to or less than a first threshold Tl (for example, 0.05), as a "deformed edge" if the outlier rate R exceeds the first threshold Tl, and as a "failed edge" if the outlier rate R exceeds a second threshold Th that is larger than the first threshold Tl. Here, the "deformed edge" indicates a state where the edge is deformed to absorb distortion accompanying loop closure but consistency is maintained, and the "failed edge" indicates a state where geometric consistency is not achieved. The "deformed edge" completely encompasses the "failed edge".

[0046] Next, in step S109, the determination unit 115 makes a final determination on the validity of the loop-closing edge. As shown in Figure 9, the determination unit 115 calculates the ratio of the number of failed edges to the total number of deformed edges (ρ = number of failed edges / number of deformed edges) and compares whether this ratio ρ is less than or equal to the determination threshold Tth (for example, 0.2).

[0047] If the ratio ρ is less than or equal to the judgment threshold Tth (valid), the process proceeds to step S110. In step S110, the update control unit 116 updates the pause graph (e.g., overall optimization) reflecting the optimization results and terminates the process. On the other hand, if the ratio ρ exceeds the judgment threshold Tth (invalid), the process proceeds to step S111. In step S111, the update control unit 116 discards the loop closing edge. Furthermore, in step S112, the update control unit 116 adds the loop closing pair to the exclusion list 124. After that, the process proceeds to step S113, where a normal odometry update is performed and the process terminates.

[0048] As described above, the information processing device 1 in this disclosure evaluates the geometric consistency between map points associated with one node and observation data associated with the other node, between two nodes connected by an odometry edge or a loop closing edge. Based on the evaluated geometric consistency, the information processing device 1 determines whether the loop closing edge is valid and controls the update of the pause graph, thereby achieving reliable loop closing and preventing pause graph failure even under conditions involving scale drift.

[0049] 3. Variant The embodiments described above are merely examples for embodying the technical concept of this disclosure, and this disclosure is not limited to the embodiments described above. Various modifications, substitutions, and variations are possible within the scope of the technical concept of this disclosure. For example, the following modifications are also included in the scope of this disclosure.

[0050] (Variations of the judgment process) In the embodiment described above, effectiveness was determined based on the outlier rate of a single loop-closing edge. However, the determination unit 115 may determine the effectiveness of a loop-closing edge based on the states of multiple edges included in the pause graph.

[0051] Specifically, the evaluation unit 114 calculates the percentage of outlier map points where the reprojection error exceeds the allowable value for each odometry edge or loop closing edge. Here, outlier map points are, for example, points where the reprojection error exceeds the threshold of the chi-squared test. In this case, the determination unit 115 is not limited to a configuration that uses fixed thresholds such as 0.05, 0.95, and 0.2 as described above, but may adjust these parameters according to the environment and sensor characteristics. For example, it is possible to set the threshold looser in an outdoor environment and stricter in an indoor environment.

[0052] Furthermore, the criteria for determining a "broken edge" are not limited to the outlier rate based on the reprojection error described above. For example, the determination unit 115 may calculate geometric indicators such as the rate of change in edge length (stretch rate), the amount of change in relative rotation (angle difference), or the amount of change in scale component before and after pose graph optimization. The determination unit 115 may determine an edge to be a "broken edge" if these geometric changes exceed a predetermined threshold (for example, a deformation that is physically impossible). In this way, by using geometric indicators other than the reprojection error, the state of edges can be robustly classified even in environments with few feature points or environments with drastic changes in lighting conditions.

[0053] Furthermore, the determination unit 115 may dynamically change the criteria used to determine whether or not it is valid (for example, the determination threshold Tth, the first threshold Tl, or the second threshold Th). For example, the determination unit 115 estimates the magnitude of the cumulative error from the covariance matrix estimated by the pause graph optimization. The determination unit 115 then relaxes the threshold as the cumulative error increases (for example, by setting Tth higher).

[0054] Figure 10 is a graph showing an example of the relationship between cumulative error and the judgment threshold. As shown in Figure 10, increasing the threshold as the cumulative error increases prevents the incorrect rejection of necessary deformations when large-scale correction is required.

[0055] (Variable sensor configuration) In the embodiments described above, a monocular camera was used as sensor 2, but sensor 2 is not limited to a monocular camera and may be a stereo camera or an RGB-D camera (depth camera). Stereo cameras and the like can, in principle, observe absolute scale, but errors may accumulate and the pose graph may become distorted depending on long-distance movement or environmental conditions (distribution of feature points, etc.). The evaluation of geometric consistency (checking reprojection errors, etc.) related to this disclosure can also be applied to prevent false detection of loop closure and to improve accuracy in SLAM using such stereo cameras.

[0056] Furthermore, sensor 2 may be a distance measuring sensor such as LiDAR (Light Detection and Ranging). In this case, the observation data 121 includes 3D point cloud data acquired by the distance measuring sensor.

[0057] The evaluation unit 114 calculates the alignment error (e.g., residual of the ICP algorithm) or distance error between the point cloud associated with one node and the point cloud associated with the other node as an evaluation of geometric consistency. Furthermore, when using dense point cloud data, the evaluation unit 114 may perform the evaluation using a portion of the points sampled (decimated) from the point cloud data according to a predetermined criterion in order to reduce computational costs. With this configuration, highly accurate loop closure verification based on shape information becomes possible even in environments where image features are difficult to obtain.

[0058] (Optimization and Coordinate System Variations) In the embodiment described above, Sim3 optimization was used, but if a stereo camera or LiDAR is used as sensor 2 and the absolute scale is reliable, the relative attitude may be calculated using a 6-degree-of-freedom rigid body transformation (SE3) that does not include the scale component. This may further reduce the computational load.

[0059] In the embodiment described above, map points are stored in a local coordinate system, but the invention is not limited to this, and they may also be stored as absolute coordinates in a global coordinate system. However, in that case, the information processing device 1 will need to recalculate the coordinates of the map points each time the orientation of a node is updated by pose graph optimization.

[0060] (A variation of update control) If the update control unit 116 determines that a loop-closing edge is invalid, it may store the combination (pair) of two nodes connected by that loop-closing edge in the exclusion list 124. In subsequent processing, the update control unit 116 excludes the combinations included in the exclusion list 124 from the candidates for loop closing. This configuration prevents similar false detections from being repeated and improves processing efficiency.

[0061] Furthermore, if the update control unit 116 determines that a loop closing edge is invalid, it may remove the loop closing edge from the pose graph and then perform pose graph optimization again. This configuration allows a pose graph that has been temporarily distorted by an incorrect edge to be restored to a normal state.

[0062] (A variation of error) In the embodiments described above, an example was explained in which map points associated with a source frame (Fs) are projected onto the image plane of a target frame (Ft) to calculate the reprojection error, but the direction of reprojection is not limited to this. For example, the evaluation unit 114 may reproject the map points associated with the target frame (Ft) onto the image plane of the source frame (Fs) using the calculated relative orientation (or its inverse transformation) in the reverse direction to calculate the error. Alternatively, the evaluation unit 114 may calculate these bidirectional reprojection errors and evaluate the geometric consistency by combining them (for example, using the average or maximum value).

[0063] Furthermore, even in the case of SLAM using a camera (V-SLAM), the evaluation of geometric consistency is not limited to that based on reprojection errors (two-dimensional errors) on the image plane. For example, the evaluation unit 114 may map a three-dimensional map point reconstructed from the image of one node by triangulation or the like, and a three-dimensional map point reconstructed from the image of the other node, in three-dimensional space using the calculated relative orientation, and evaluate geometric consistency by calculating the alignment error or distance error (three-dimensional error) between them. [Explanation of Symbols]

[0064] 1... Information processing device, 11... Processor, 111... Data acquisition unit, 112... Coordinate transformation unit, 113... Optimization execution unit, 114... Evaluation unit, 115... Judgment unit, 116... Update control unit, 12... Me Mori, 121... Observation data, 122... Map data, 123... Graph data, 1231... Node list, 1232... Edge list, 124... Exclusion list, 13... Interface, 2... Sensor, E1... Odometry edge, F1~F10... Node

Claims

1. On the computer, A data acquisition step involves obtaining multiple observational data showing the shape of objects in the surrounding environment, each observed at a different time by sensors mounted on a mobile device. An optimization execution step of constructing a pose graph that includes a plurality of nodes, each indicating the position and orientation of the moving object at the time of observation of the shape shown by any of the plurality of observation data, and odometry edges connecting the nodes that are temporally consecutive among the plurality of nodes, and adding loop closing edges that connect a pair of temporally non-contiguous nodes included in the pose graph, a source node that is the starting point and a target node that is the ending point, based on the similarity between the shapes shown by the plurality of observation data, and calculating the relative orientation between the source node and the target node using a similarity transformation that includes a scale component in order to correct the cumulative error in the pose graph, An evaluation step to evaluate the geometric consistency of two nodes by comparing the shapes of objects shown in the observation data corresponding to each of the two nodes connected by the edge, with respect to each edge included in the pose graph on which the pose graph optimization has been performed, from among the plurality of observation data; An update step which involves determining the effectiveness of the loop closing edge based on the evaluated geometric consistency and updating the pose graph with content corresponding to the result of the effectiveness determination, A program to execute.

2. The evaluation step is: After performing the pose graph optimization, the geometric consistency between the map points associated with one node and the observational data associated with the other node from the plurality of observational data is evaluated at the two nodes. The program according to claim 1.

3. The evaluation step involves projecting the map points associated with one node onto the keyframe image plane corresponding to the other node using the relative orientation, and calculating the reprojection error as the geometric consistency based on the distance between the projected points and the feature points as observed data associated with the other node. The program according to claim 2.

4. The evaluation step calculates, for each odometry edge or loop closing edge, the percentage of outlier map points where the reprojection error exceeds the allowable value. The update step determines that edges whose proportion exceeds a first threshold are deformed edges, that among the deformed edges, edges whose proportion exceeds a second threshold greater than the first threshold are broken edges, and determines the effectiveness of the loop closing edge based on the ratio of the number of broken edges to the total number of deformed edges. The program according to claim 3.

5. The update step includes determining the effectiveness of the loop closing edge, and dynamically changing the criteria used to determine the effectiveness of the loop closing edge according to the magnitude of the cumulative error estimated by the pause graph optimization. The program according to any one of claims 1 to 4.

6. The update step relaxes the criteria so that when the cumulative error is large, the loop closing edge is more likely to be determined to be valid compared to when the cumulative error is small. The program according to claim 5.

7. The evaluation step is performed by selecting from all edges included in the pause graph edges that are on a closed-loop path formed by the loop-closing edge, or edges that are within a predetermined range from the closed-loop path. The program according to claim 1.

8. The optimization execution step includes calculating the relative pose between the source node and the target node using a similarity transformation that takes scale drift into account. The program according to any one of claims 1 to 4.

9. Each of the aforementioned plurality of observation data includes point cloud data acquired by a ranging sensor. The evaluation step calculates the alignment error or distance error between the point cloud associated with one of the two nodes and the point cloud associated with the other node of the two nodes as the geometric consistency. The program according to claim 1.

10. The evaluation step calculates the alignment error or the distance error using a portion of the points sampled from the point cloud data according to a predetermined criterion. The program according to claim 9.

11. The update step includes determining the validity of the loop-closing edge, and if the loop-closing edge is determined to be invalid, the combination of two nodes connected by the loop-closing edge is stored, and in subsequent processing, the combination is excluded from the list of loop-closing candidates. The program according to claim 1.

12. Prior to the optimization execution step, the process further includes converting the coordinates of the associated map points for each of the plurality of nodes into local coordinates in the camera coordinate system of the keyframe corresponding to that node and storing them therein. The evaluation step uses the local coordinates as the map point associated with one of the two nodes. The program according to claim 1.

13. The update step includes determining the validity of the loop closing edge, and if the loop closing edge is determined to be invalid, the loop closing edge is removed from the pause graph and the pause graph optimization is performed again. The program according to claim 1.

14. An information processing apparatus comprising a memory and a processor coupled to the memory, The aforementioned processor, Sensors mounted on the mobile vehicle acquire multiple observational data showing the shapes of objects in the surrounding environment, each observed at a different time. A pose graph is constructed, each containing a plurality of nodes that indicate the position and orientation of the moving object at the time of observation of the shape shown by any of the plurality of observation data, and odometry edges connecting the nodes that are temporally consecutive among the plurality of nodes. Based on the similarity between the shapes shown by the plurality of observation data, a loop closing edge is added to the pose graph, connecting a pair of temporally non-contiguous nodes, a source node that is the starting point and a target node that is the ending point. Pose graph optimization is performed to calculate the relative orientation between the source node and the target node using a similarity transformation that includes a scale component in order to correct the cumulative error in the pose graph. With respect to each edge included in the pose graph on which the pose graph optimization has been performed, the geometric consistency of the two nodes is evaluated by comparing the shapes of the objects shown in the observation data corresponding to each of the two nodes connected by that edge from among the plurality of observation data. Based on the evaluated geometric consistency, the effectiveness of the loop-closing edge is determined, and the pause graph is updated according to the result of the effectiveness determination. An information processing device configured in such a way.

15. A method of information processing performed by a computer, A data acquisition step involves obtaining multiple observational data showing the shape of objects in the surrounding environment, each observed at a different time by sensors mounted on a mobile device. An optimization execution step of constructing a pose graph that includes a plurality of nodes, each indicating the position and orientation of the moving object at the time of observation of the shape shown by any of the plurality of observation data, and odometry edges connecting the nodes that are temporally consecutive among the plurality of nodes, and adding loop closing edges that connect a pair of temporally non-contiguous nodes included in the pose graph, a source node that is the starting point and a target node that is the ending point, based on the similarity between the shapes shown by the plurality of observation data, and calculating the relative orientation between the source node and the target node using a similarity transformation that includes a scale component in order to correct the cumulative error in the pose graph, An evaluation step to evaluate the geometric consistency of two nodes by comparing the shapes of objects shown in the observation data corresponding to each of the two nodes connected by the edge, with respect to each edge included in the pose graph on which the pose graph optimization has been performed, from among the plurality of observation data; An update step which involves determining the effectiveness of the loop closing edge based on the evaluated geometric consistency and updating the pose graph with content corresponding to the result of the effectiveness determination, Information processing methods including

Images