A method, system, device, and medium for loop closure detection based on panoramic semantic topology graphs.

Abstract

This invention relates to the fields of computer vision and robot localization technology, and provides a method, system, device, and medium for loop closure detection based on a panoramic semantic topology map. The method includes: first, inputting a panoramic image into a semantic segmentation and depth estimation network to obtain a semantic map and a depth map, respectively; removing dynamic objects; and then extracting the centroid coordinates of each static object. The depth value corresponding to each centroid coordinate is calculated based on the depth map, and the three-dimensional spatial coordinates of each static object are obtained through coordinate transformation. When the spatial distance between static objects is less than a threshold, edge connections are established to generate a panoramic semantic topology map. By calculating the similarity between semantic topology maps, it is determined whether the user has returned to a previously visited location, thus completing the loop closure detection task. This method integrates high-level image information such as semantics, depth, and spatial position relationships, reducing perceptual ambiguity and mitigating the impact of changes in viewpoint, lighting, and dynamic objects on loop closure detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robot localization technology, and in particular to a method, system, device and medium for closed-loop detection based on panoramic semantic topology maps. Background Technology

[0002] Loop closure detection (or visual location recognition) is a crucial component of visual SLAM systems. It refers to the robot's ability to determine whether it has returned to its previous position based on image similarity, using this as a basis for map updates. Because errors and uncertainties in robot pose estimation accumulate with robot movement, accurate loop closure detection is essential to eliminate these errors, thereby obtaining a globally consistent map and achieving more precise localization. A key challenge in loop closure detection is accurately describing a specific location under various environmental changes. Most existing loop closure detection methods are easily affected by environmental variations and do not fully utilize image information.

[0003] The paper "Visual SLAM Loop Closure Detection Based on Local Semantic Topology Graph" detects objects in indoor images acquired by an RGB-D sensor, reconstructs the detection results into space, constructs semantic topological relationships between objects after density clustering, calculates the similarity between scenes, and finally determines loop closure by the similarity changes of consecutive keyframes. However, this paper only constructs a local semantic topology graph, which cannot constitute a comprehensive description of the environment. Moreover, the subsequent similarity calculation only considers the rotation, translation, and scale differences between three-dimensional coordinate vectors, resulting in large computational load and complexity. Summary of the Invention

[0004] To address at least one of the problems existing in current technologies, this invention proposes a loop closure detection method based on a panoramic semantic topology map. It constructs a more robust and discriminative location descriptor, and then calculates the similarity between semantic topology maps using a graph neural network to determine whether a previously visited location has been returned, thus completing the loop closure detection task. This invention enables rapid and accurate location identification in complex and changing environments, thereby providing support for robot localization and mapping.

[0005] To address the issues of poor robustness, high computational cost, and low efficiency in traditional loop closure detection methods, this paper proposes a loop closure detection method based on a panoramic semantic topology graph. By combining high-level information such as semantics, depth, and spatial positional relationships, it reduces perceptual ambiguity and mitigates the impact of changes in viewpoint, lighting, and dynamic objects on loop closure detection. This effectively improves the robustness of loop closure detection in complex environments, helping robots to locate objects more accurately and quickly.

[0006] To achieve the objectives of this invention, a loop closure detection method based on a panoramic semantic topology map is provided, the main steps of which include:

[0007] Acquire panoramic images;

[0008] Semantic segmentation is performed on the panoramic image to obtain the predicted semantic map. Dynamic objects in the semantic map are removed based on semantic category information, and the centroid coordinates of the static image are extracted.

[0009] Depth estimation is performed on the panoramic image to obtain the predicted depth map, and the depth value at each centroid coordinate is calculated.

[0010] The static object is projected onto the world coordinate system using the centroid coordinates, depth values, and camera intrinsic and extrinsic parameters to obtain the coordinates of the static object in three-dimensional space. The semantic category information and three-dimensional spatial coordinates of each object are aggregated to generate nodes in the semantic topology graph. The connection between the nodes is established according to the preset distance threshold to generate the semantic topology graph.

[0011] Calculate the similarity between semantic topological graphs to determine whether a closed loop is formed.

[0012] Furthermore, the methods for obtaining panoramic images include: directly acquiring panoramic images using a panoramic camera, or stitching together local images captured by a regular camera to obtain a panoramic image.

[0013] Furthermore, semantic segmentation is performed on the panoramic image. First, a predicted semantic map is obtained through a pre-trained semantic segmentation network. Dynamic objects in the semantic map are removed based on semantic category information. The centroid coordinates of each image region are obtained through grayscale conversion, edge detection, and centroid extraction to represent static objects.

[0014] Furthermore, depth estimation is performed on the panoramic image. A predicted depth map is obtained through a pre-trained depth estimation network, and the depth value at each centroid coordinate is calculated as the depth value of the static object.

[0015] Furthermore, the semantic category, centroid coordinates, and depth values ​​of static objects are combined to generate a semantic topology map of the panoramic image. Using the centroid coordinates, depth values, and camera intrinsic and extrinsic parameters of static objects, they can be projected onto the world coordinate system, i.e., their coordinates in three-dimensional space. Nodes in the semantic topology map are generated by aggregating the semantic category information and three-dimensional spatial coordinates of each object, and a distance threshold is set; edges are only established between nodes when the relative distance between static objects is less than the threshold. This mapping of the image into three-dimensional space allows for the extraction of camera-invariant relationships between objects. Even if lighting and viewpoint change, the relative positional relationships between static objects in the scene remain unchanged. Therefore, this three-dimensional semantic topology map is independent of camera pose and contains the semantic information and spatial positional relationships of objects, making it more concise and efficient.

[0016] Furthermore, the similarity between semantic topological graphs is calculated using graph neural networks to determine whether a closed loop has been formed. By encoding the structural and feature information of the graphs through an end-to-end neural network and introducing an attention mechanism, a pair of graphs is mapped to a similarity score, which can reduce the computational burden while maintaining good performance.

[0017] This invention also provides a loop closure detection system based on a panoramic semantic topology graph, comprising the following modules:

[0018] Panoramic image acquisition module, used to acquire panoramic images;

[0019] The semantic segmentation module is used to perform semantic segmentation on panoramic images to obtain a predicted semantic map. Based on semantic category information, dynamic objects in the semantic map are removed, and the centroid coordinates of the static image are extracted.

[0020] The depth estimation module is used to perform depth estimation on panoramic images, obtain a predicted depth map, and calculate the depth value at each centroid coordinate to obtain the depth value of static objects.

[0021] The semantic topology graph generation module is used to project static objects onto the world coordinate system using the centroid coordinates, depth values, and camera intrinsic and extrinsic parameters of static objects, thereby obtaining the coordinates of static objects in three-dimensional space. It aggregates the semantic category information and three-dimensional spatial coordinates of each object to generate nodes in the semantic topology graph, and establishes edge connections between nodes according to a preset distance threshold to generate the semantic topology graph.

[0022] The similarity calculation module is used to calculate the similarity between semantic topology graphs and determine whether a closed loop is formed.

[0023] Compared with the prior art, the present invention can achieve at least the following beneficial effects:

[0024] This invention first inputs a panoramic image into a semantic segmentation and depth estimation network to obtain a semantic map and a depth map, respectively. Then, dynamic objects such as pedestrians and vehicles are removed. Based on this, the centroid coordinates of each object are extracted to represent all static objects in the image. Next, the depth value corresponding to each centroid coordinate is estimated based on the depth map, and the three-dimensional spatial coordinates of each object are obtained through coordinate transformation. When the spatial distance between objects is less than a preset distance threshold, edge connections are established to generate a panoramic semantic topology map. Finally, the similarity between semantic topology maps is calculated to determine whether a previously visited location has been reached, thus completing the loop closure detection task. This invention's technical solution effectively improves the robustness of loop closure detection in complex environments, obtains more accurate loop closure detection results, and can be used to eliminate accumulated errors in pose estimation, helping robots to perform localization more accurately and quickly. Attached Figure Description

[0025] Figure 1 This is an overall flowchart of a loop closure detection method based on a panoramic semantic topology graph, provided in an embodiment of the present invention.

[0026] Figure 2 This is a diagram of the SimGNN graph neural network model in an embodiment of the present invention.

[0027] Figure 3 This is a module diagram of a loop closure detection based on a panoramic semantic topology graph provided in an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the experimental results for the KITTI 2015 dataset.

[0029] Figure 5 This is a schematic diagram of the experimental results for the City Centre dataset. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This invention provides a loop closure detection method based on a panoramic semantic topology graph. Please refer to the overall flowchart. Figure 1 The specific steps are as follows:

[0032] Step 1: Use a panoramic camera to directly acquire panoramic images, or stitch together partial images captured by a regular camera to obtain a panoramic image.

[0033] Step 2: Perform semantic segmentation and depth estimation on the panoramic image to obtain semantic category information and depth information of objects in the image.

[0034] In this step, a predicted semantic map is first obtained through an existing semantic segmentation network. Dynamic objects in the semantic map are removed based on semantic category information. The centroid coordinates of each image region are obtained through grayscale conversion, edge detection, and centroid extraction to represent static objects. Then, a predicted depth map is obtained through an existing depth estimation network, and the depth value under each centroid coordinate is calculated to serve as the depth value of the static object.

[0035] In some embodiments of the present invention, the semantic segmentation and depth estimation network can use dual-branch multi-task learning networks designed for panoramic images, such as FreDSNet and SegFuse, and the network can be flexibly adjusted according to the type of input image.

[0036] The goal of semantic segmentation is to identify each object and extract its centroid in the semantic graph to estimate its depth and construct a 3D semantic topology graph. After obtaining the semantic segmentation prediction results, dynamic objects such as pedestrians and vehicles are first removed to reduce the impact of dynamic object changes on recognition accuracy. Then, the centroid of each object is extracted, and each object's centroid is... i It is represented as a combination of category and centroid pixel coordinates, as shown in Equation (1).

[0037] o i =(c i ,x i ,y i (1)

[0038] Among them, c i It is a semantic category label, (x i ,y i ) is the pixel coordinate corresponding to the centroid.

[0039] The purpose of depth estimation is to obtain the distance information from each point in the scene to the camera, thereby elevating the 2D image to a 3D semantic topological map. By utilizing the camera-invariant relationships between objects and combining them with the corresponding depth map, the depth value d(x) at each centroid pixel coordinate is obtained. i ,y i This is used as the depth value for each object in the image.

[0040] Step 3: Combine the semantic category, centroid coordinates, and depth values ​​of the objects from Step 2 to generate a semantic topology map of the panoramic image.

[0041] In this step, the centroid coordinates, depth values, and camera intrinsic and extrinsic parameters of objects are used to project them onto the world coordinate system, i.e., their coordinates in three-dimensional space. Nodes in the semantic topology graph are generated by aggregating the semantic category information and three-dimensional spatial coordinates of each object, and a distance threshold is artificially set (in some embodiments of this invention, the distance threshold is 10 for the KITTI 2015 dataset and 16 for the City Centre dataset; both the KITTI 2015 and City Centre datasets are existing datasets). Edges are only established between nodes when the relative distance between objects is less than the distance threshold. This mapping of the image into three-dimensional space allows for the extraction of camera-invariant relationships between objects. Even if lighting and viewpoint change, the relative positional relationships between static objects in the scene remain unchanged. Therefore, this three-dimensional semantic topology graph is independent of camera pose and contains the semantic information and spatial positional relationships of objects, making it more concise and efficient.

[0042] Specifically, in some embodiments of the present invention, this step uses the centroid pixel coordinates and depth values ​​of the objects to map them onto the world coordinate system according to equations (2) and (3). Then, the relative distance between objects in three-dimensional space is calculated according to equation (4). By combining the semantic category information and three-dimensional spatial coordinates of each object to form the representation of nodes in the semantic topology graph, the nodes include the above three types of information, which is an important measure for similarity calculation. Then, a distance threshold is artificially set, and an edge connection is established between nodes only when the relative distance between objects is less than this value.

[0043]

[0044] P W =R WC ·P C +T WC (3)

[0045]

[0046] Among them, P c =(X c ,Y c Z c d(x) represents the position of each object in the camera coordinate system. i ,y i f is the depth value at the centroid pixel coordinates. x ,f y ,c x ,c y R is the camera's internal parameter. wc and T wcThese are the rotation and translation matrices of the world coordinate system relative to the camera coordinate system, D. ij It refers to two three-dimensional coordinates (X, Y, Z) in the world coordinate system. i ,Y i Z i ) and (X j ,Y j Z j The Euclidean distance of ).

[0047] Step 4: Use a graph neural network to calculate the similarity between semantic topological graphs and determine whether a closed loop has been formed.

[0048] The structure and features of the graph are encoded by an end-to-end neural network, and an attention mechanism is introduced to map a pair of graphs to a similarity score.

[0049] Step 4, the similarity calculation based on graph neural networks, mainly includes:

[0050] ① Compared with traditional graph edit distance (GED) and maximum common subgraph (MCS) methods, neural network-based graph similarity calculation can reduce the computational burden while maintaining good performance.

[0051] ② A pair of images is mapped to a similarity score using the SimGNN model. During the training phase, node features and edges of images in the dataset are customized, and model parameters are optimized by minimizing the difference between the predicted similarity score and the ground truth.

[0052] ③ First, node embeddings are generated, converting each node in the graph into a vector that encodes the features and structural attributes surrounding each node. Then, an attention module is introduced to aggregate the node embeddings, assigning more weights to more important nodes to generate a graph-level embedding. Similarity can be calculated based on the interaction between two graph-level embeddings.

[0053] In some embodiments of the present invention, the differences between paired nodes in two graphs are calculated based on node-level embedding to supplement the graph-level embedding, thereby more effectively modeling the similarity between the two graphs. The feature vectors from both levels are concatenated and passed through a fully connected network to output a similarity score.

[0054] In some embodiments of the present invention, a three-dimensional semantic topology graph is constructed for the KITTI 2015 dataset, such as... Figure 4As shown. For a more intuitive representation, a two-dimensional semantic topology graph is used here. These three images correspond to the same location, but there are changes in perspective, forward and backward movement, and interference from dynamic objects between the images. However, the semantic topology graph constructed according to the method of this invention can solve these problems well, accurately describe the location, and is a location descriptor with strong discriminative power and high robustness.

[0055] In some embodiments of the present invention, for the City Centre dataset, image sequences with sequence numbers 880-920 were selected. A three-dimensional semantic topology graph was constructed, and the similarity score between the semantic topology graph and the image was calculated as a loop closure criterion. Figure 5 The calculated PR curve is shown in the figure. The results show that the accuracy and recall are both very high, and the AP value is close to 1, which demonstrates the effectiveness of the method of the present invention.

[0056] The aforementioned embodiment provides a loop closure detection method based on a panoramic semantic topology map. From the perspective of human scene recognition, it utilizes semantic segmentation to obtain object category information in the image, and obtains the depth values ​​of each object through depth estimation. This depth value is then transformed into their relative positional relationships, thereby generating a panoramic semantic topology map to represent the location. By projecting pixel coordinates onto the world coordinate system through coordinate transformation, camera-invariant relationships between objects can be extracted. Even if lighting and viewing angle change, the relative positional relationships between static objects in the scene remain unchanged. Therefore, this panoramic semantic topology map is independent of camera pose and contains semantic information and spatial positional relationships of objects, making it more concise and efficient, and improving the robustness of the loop closure detection algorithm in complex environments.

[0057] In some embodiments of the present invention, a loop closure detection system based on a panoramic semantic topology graph is also provided to implement the method provided in the foregoing embodiments, comprising the following modules:

[0058] Panoramic image acquisition module, used to acquire panoramic images;

[0059] The semantic segmentation module is used to obtain the predicted semantic graph through the semantic segmentation network and extract the centroid coordinates of static objects in the semantic graph;

[0060] The depth estimation module is used to obtain the predicted depth map through the depth estimation network and calculate the depth value at each centroid coordinate to obtain the depth value of the static object.

[0061] The semantic topology graph generation module is used to generate panoramic semantic topology graphs based on the semantic category, centroid coordinates, and depth values ​​of objects.

[0062] The similarity calculation module is used to calculate the similarity between semantic topology graphs and determine whether a closed loop is formed.

[0063] In some embodiments of the present invention, a loop closure detection device based on a panoramic semantic topology graph is also provided, comprising:

[0064] At least one processor;

[0065] At least one memory for storing at least one program;

[0066] When the at least one program is executed by the at least one processor, the at least one processor implements the method provided in the foregoing embodiments.

[0067] In some embodiments of the present invention, a storage medium is also provided, wherein a processor-executable program is stored, which, when executed by a processor, is used to perform the methods provided in the foregoing embodiments.

[0068] The present invention provides a closed-loop detection system, device and medium based on panoramic semantic topology graph, which has the same beneficial effects as the method described above.

[0069] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A loop closure detection method based on a panoramic semantic topology graph, characterized in that, Includes the following steps: Acquire panoramic images; Semantic segmentation is performed on panoramic images to obtain predicted semantic maps. Dynamic objects in the semantic map are removed based on semantic category information, and the centroid coordinates of static images are extracted. Specifically, after removing dynamic objects from the semantic map based on semantic category information, the centroid coordinates of each image region are obtained through grayscale conversion, edge detection, and centroid extraction, thus representing static objects. In semantic segmentation, after removing dynamic objects, the centroid of each static object is extracted, and each static object's coordinates are assigned to a specific coordinate system. i It is represented as a combination of category and centroid pixel coordinates, as shown in Equation (1): o i =(c i ,x i ,y i ) (1) Among them, c i It is a semantic category label, (x i ,y i () represents the pixel coordinates corresponding to the centroid; In depth estimation, the distance information from each point in the scene to the camera is first obtained, thus elevating the 2D image to a 3D semantic topological map. Then, using the camera-invariant relationships between objects, this information is combined with the corresponding depth map to obtain the depth value d(x) at each centroid pixel coordinate. i ,y i This is used as the depth value for each static object in the image; Depth estimation is performed on the panoramic image to obtain the predicted depth map, and the depth value at each centroid coordinate is calculated. The static object is projected onto the world coordinate system using the centroid coordinates, depth values, and camera intrinsic and extrinsic parameters to obtain the coordinates of the static object in three-dimensional space. The semantic category information and three-dimensional spatial coordinates of each object are aggregated to generate nodes in the semantic topology graph. The connection between the nodes is established according to the preset distance threshold to generate the semantic topology graph. Calculate the similarity between semantic topological graphs to determine whether a closed loop is formed.

2. The loop closure detection method based on a panoramic semantic topology graph according to claim 1, characterized in that, The methods for obtaining panoramic images include: directly acquiring panoramic images using a panoramic camera, or stitching together local images captured by a regular camera to obtain a panoramic image.

3. The loop closure detection method based on a panoramic semantic topology graph according to claim 1, characterized in that, Using the centroid pixel coordinates and depth values ​​of the static object, the static object is mapped to the world coordinate system according to equations (2) and (3); Calculate the relative distance between static objects in three-dimensional space according to equation (4); Nodes in the semantic topology graph are generated by aggregating the semantic category information and three-dimensional spatial coordinates of each static object. When the relative distance between static objects is less than the distance threshold, an edge connection is established between the nodes to map the image into three-dimensional space. P W =R WC ·P C +T WC (3) In the formula, P c =(X c ,Y c Z c d(x) represents the position of each object in the camera coordinate system. i ,y i f is the depth value at the centroid pixel coordinates. x ,f y ,c x ,c y R is the camera's internal parameter. wc and T wc These are the rotation and translation matrices of the world coordinate system relative to the camera coordinate system, D. ij It refers to two three-dimensional coordinates (X, Y, Z) in the world coordinate system. i ,Y i Z i ) and (X j ,Y j Z j The Euclidean distance of ).

4. A loop closure detection method based on a panoramic semantic topology graph according to any one of claims 1-3, characterized in that, Graph neural networks are used to calculate the similarity between semantic topological graphs to determine whether a closed loop has been formed.

5. The loop closure detection method based on a panoramic semantic topology graph according to claim 4, characterized in that, The structure and features of the graph are encoded by an end-to-end neural network, and an attention mechanism is introduced to map a pair of graphs to a similarity score. During the training phase, the node features and edges of the images in the dataset are customized, and the model parameters are optimized by minimizing the difference between the predicted and true values ​​of the similarity scores.

6. The loop closure detection method based on a panoramic semantic topology graph according to claim 5, characterized in that, In graph neural networks, node embeddings are first generated, converting each node in the graph into a vector that encodes the features and structural properties around each node. Then, an attention mechanism is introduced to aggregate the node embeddings to assign more weights to more important nodes, generating a graph-level embedding. Similarity is calculated based on the interaction between the two graph-level embeddings.

7. A loop closure detection system based on a panoramic semantic topology graph, characterized in that, The system for implementing the method of any one of claims 1-6 includes the following modules: Panoramic image acquisition module, used to acquire panoramic images; The semantic segmentation module is used to perform semantic segmentation on panoramic images to obtain a predicted semantic map. Based on semantic category information, dynamic objects in the semantic map are removed, and the centroid coordinates of the static image are extracted. The depth estimation module is used to perform depth estimation on panoramic images, obtain a predicted depth map, and calculate the depth value at each centroid coordinate to obtain the depth value of static objects. The semantic topology graph generation module is used to project static objects onto the world coordinate system using the centroid coordinates, depth values, and camera intrinsic and extrinsic parameters of static objects, thereby obtaining the coordinates of static objects in three-dimensional space. It aggregates the semantic category information and three-dimensional spatial coordinates of each object to generate nodes in the semantic topology graph, and establishes edge connections between nodes according to a preset distance threshold to generate the semantic topology graph. The similarity calculation module is used to calculate the similarity between semantic topology graphs and determine whether a closed loop is formed.

8. A loop closure detection device based on a panoramic semantic topology map, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method according to any one of claims 1-6.

9. A storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method described in any one of claims 1-6.

Images