Visual object level SLAM loopback detection method based on graph

Abstract

The invention belongs to the technical field of automatic driving positioning mapping, and particularly relates to a visual object level SLAM loopback detection method based on a graph. The method comprises the following steps: firstly, carrying out object detection on a current frame image, extracting global and local visual features, and converting 3D geometric parameters of an object into natural language description through a text template to generate text features; fusing the three types of features through a multi-layer perceptron to obtain a comprehensive feature vector of each object; then taking the vector as a node, constructing a graph structure in combination with semantic weight and spatial distance weight, and generating a node description vector through weighted aggregation; local map object model parameters subjected to back-end optimization are adopted for calculation of the spatial distance weight of the historical key frame, so that more stable expression is obtained; and finally, calculating cosine similarity of node description vectors and carrying out graph matching to realize loopback judgment. According to the method, through multi-modal deep fusion and refined graph structure modeling, the accuracy of loopback detection in a complex environment is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving localization and mapping, and in particular to a graph-based visual object-level SLAM loop closure detection method. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology in fields such as autonomous driving and mobile robotics, enabling real-time localization and map building in unknown environments. Loop closure detection, a key module in visual SLAM systems, is primarily used to identify whether the robot has revisited a previous scene, thus providing constraints for backend optimization, correcting accumulated drift errors, and improving the global consistency of the map. However, existing loop closure detection methods still face problems such as low feature discrimination and insufficient utilization of scene structure information in complex dynamic environments, specifically as follows: Traditional methods based on hand-designed features, such as ORB and SIFT feature point matching methods, are computationally efficient, but they rely on local texture information and are extremely sensitive to environmental disturbances such as changes in lighting and seasons, which can easily lead to mismatches and missed detections. They are also difficult to maintain stable performance in dynamic or low-texture scenes.

[0003] End-to-end methods based on deep learning, such as using convolutional neural networks to extract global features or train classification networks, can improve feature representation capabilities, but they usually require a large amount of labeled data for training, have high computational complexity and limited generalization ability, often ignore object-level semantic and geometric structural information, and are difficult to deal with the confusion problem of objects with similar appearances.

[0004] Methods based on semantic topology graphs, such as the semantic-assisted visual SLAM loop closure detection method based on topology graph matching disclosed in reference document CN120673090A, construct a semantic graph structure through object detection and use graph matching for loop closure detection, thus incorporating semantic and spatial information to some extent. However, these methods typically rely on single visual or geometric features to construct node descriptions, failing to integrate multimodal information. Furthermore, the node representations lack deep semantic encoding of the object's own attributes, resulting in insufficient discriminative power when objects are occluded, viewing angles change, or appearances are similar, thus affecting matching accuracy.

[0005] In summary, existing loop closure detection methods generally have the following limitations: (1) It is sensitive to changes in appearance such as lighting and viewing angle, and lacks robustness; (2) The object representation is singular and fails to effectively integrate multimodal information such as vision, text and geometry, resulting in limited distinguishability; (3) The lack of fine-grained modeling and utilization of scene-level spatial topology makes it difficult to maintain stable recognition under dynamic interference or local occlusion.

[0006] Therefore, there is an urgent need in this field for a loop closure detection method that can deeply integrate multimodal features, explicitly model object-level spatial topological relationships, and be highly adaptable to complex environmental changes, so as to improve the accuracy of SLAM systems in real-world scenarios. Summary of the Invention

[0007] In view of this, the present invention aims to overcome the technical problems of the existing visual SLAM loop closure detection methods, namely: (1) they are sensitive to changes in environmental appearance and differences in perspective, and lack robustness; (2) they have a single object representation, fail to effectively integrate multimodal information, and have limited distinguishability; (3) they lack effective modeling and utilization of scene-level spatial topology. The present invention proposes a graph-based visual object-level SLAM loop closure detection method, which deeply integrates multimodal information, constructs an object-level spatial topology graph, and uses optimized map information to enhance the stability of feature expression, thereby significantly improving the accuracy of loop closure detection and scene distinguishability in complex dynamic environments.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A graph-based visual object-level SLAM loop closure detection method is proposed. Its core lies in the feature extraction and graph structure modeling of the current scene through multimodal fusion, and the matching with historical scenes based on optimized map information.

[0009] The method specifically includes the following steps: First, object detection is performed on the current frame's RGB image to obtain object masks, 3D bounding boxes, and global visual features. For each detected object, its 3D geometric parameters are transformed to the world coordinate system, converted into a natural language description using a predefined text template, and encoded as text features; its local visual features are then extracted.

[0010] Next, the global visual features, local visual features, and text features are deeply fused through a multilayer perceptron to generate a comprehensive feature vector for each object.

[0011] Then, a scene graph is constructed using the comprehensive feature vectors of objects as nodes. The edges between nodes are defined by both semantic weights and spatial distance weights. By weighted aggregation of the features of a node and its neighboring nodes, a description vector is generated for each node, which simultaneously encodes the object's own attributes and its local contextual relationships.

[0012] For historical keyframes, the same process is used to generate node description vectors, but the key improvement is that when calculating spatial distance weights, the object model parameters in the local map optimized by the backend Bundle Adjustment are used instead of single-frame observations, thus obtaining a more stable and accurate expression of spatial relationships.

[0013] Finally, the cosine similarity of node description vectors between the current frame and historical keyframes is calculated, and the node correspondence is determined through threshold filtering and graph matching algorithms. If the number of successfully matched nodes exceeds a set threshold, a loop closure is detected.

[0014] Preferably, the standardized natural language description format generated by the text template is "center point in world coordinate system". ,long ,Width ,high Rotation angle "objects".

[0015] Preferably, the deep learning models used to generate global visual feature vectors, local visual feature vectors, and text feature vectors are pre-trained before deployment.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: First, to address the issue that traditional methods are sensitive to changes in lighting and viewing angle, the 3D geometric parameters of objects are converted into text descriptions and deeply integrated with visual features. This allows the object's feature representation to include not only appearance information but also stable semantic attributes such as size and position. As a result, it can maintain higher distinguishability and matching stability when faced with changes in lighting, occlusion, or objects with similar appearances, overcoming the limitations of single visual features.

[0017] Secondly, to address the issue of unreliable scene recognition under dynamic interference, this invention constructs an object-level scene map that encodes semantics and spatial relationships, and enhances the representation of historical frames using optimized map information. This makes scene recognition rely on relatively stable spatial topological relationships between objects, rather than isolated objects themselves. Therefore, even if some objects are falsely detected, occluded, or moved, the system can still make reliable judgments based on the scene layout.

[0018] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of the present invention.

[0020] Figure 2 The encoding process for the feature vector of a single node.

[0021] Figure 3 The process of constructing a graph structure. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0024] This embodiment provides a graph-based visual object-level SLAM loop closure detection method, the overall process of which is as follows: Figure 1 As shown, it can be summarized as follows: Object detection and multimodal feature extraction are fused into the current frame image to construct a scene graph with objects as nodes and semantic and spatial relationships as edges, and node description vectors are generated; a similar process is used for historical keyframes, but optimized map model parameters are used when calculating spatial relationships; finally, loop closure detection is achieved through similarity matching and threshold judgment of node description vectors.

[0025] The specific steps are as follows: Step S1: For the current RGB frame image of the input system, use deep learning to obtain the mask of the object in the image and the corresponding 3D detection box, encode the information of the whole image, and generate a global visual feature vector.

[0026] A pre-trained deep learning model is used to process the current RGB frame image, performing semantic segmentation and 3D detection, and simultaneously outputting pixel-level masks of all objects in the image. and the corresponding 3D detection box ,in: : The coordinates of the center point of the 3D detection box in the camera coordinate system; : The length, width, and height of the 3D inspection frame; The 3D detection box revolves in the camera coordinate system. The rotation angle of the axis.

[0027] Simultaneously, a convolutional neural network or Transformer model is used to perform global feature encoding on the entire input image, outputting a global visual feature vector with fixed dimensions. This step provides scene context information for subsequent fusion.

[0028] Step S2: For each object in the image, the parameters of its 3D detection box are transformed into the world coordinate system through the pose of the current frame. The transformed parameter results are converted into natural language descriptions using text templates and encoded into text feature vectors using text encoders. At the same time, individual objects are extracted and encoded to obtain the visual feature vectors of individual objects.

[0029] For each object detection result obtained in step S1 and its mask ,conduct: S201 coordinate system transformation: combining the initial pose transformation matrix of the current frame output from the SLAM front-end odometry. ,in, For rotation matrix, The translation vector transforms the center point of the 3D detection box from the camera coordinate system to the world coordinate system: ,in, The coordinates of the center point in the transformed world coordinate system are denoted as , and the parameters of the transformed 3D bounding box are denoted as . .

[0030] S202 Text Feature Encoding: Parameter-Based Customize text templates and convert them into standardized natural language descriptions. Figure 2 For the encoding process of a single node feature vector, the text template can be "the center point in the world coordinate system". ,long ,Width ,high Rotation angle The sentence is then encoded into a text feature vector using a pre-trained CLIP text encoder. This step primarily semanticizes the geometric parameters, generating textual descriptions with clear physical meanings, thereby enhancing the semantic expressiveness of features and their potential for cross-modal alignment.

[0031] S203 Local Visual Feature Encoding: Based on Object Mask Cropping an image region from the original image that contains only the object ⊙ represents the area where the object mask is preserved. Similarly, a CNN or Transformer model is used to encode this local image region to obtain the local visual feature vector of the object. .

[0032] Step S3: The global visual feature vector obtained in step S1, the visual feature vector of a single object obtained in step S2, and the text feature vector are fused together using a multilayer perceptron to obtain the comprehensive feature vector of a single object.

[0033] global features Local visual features Text features Vertical concatenation is performed to form a combined feature vector. The combined vector is then input into a pre-designed multilayer perceptron (MLP) for deep fusion. This MLP employs a three-stage structure: input layer - hidden layer - output layer. a. Input layer: Input vector is ; b. Hidden layer: Non-linear mapping is achieved through the ReLU activation function. ;in, , Here, 2D represents the learnable parameters of the hidden layer; c. Regularization layer: Embed a BatchNorm layer and a Dropout layer after the hidden layer.

[0034] d. Output layer: Outputs the comprehensive feature vector of a single object: ;in , For the learnable parameters of the output layer, This is the final integrated feature vector.

[0035] This fusion process uses a learnable MLP network to deeply interact with and integrate information from different modalities, generating more discriminative object-level features.

[0036] Step S4: Use the comprehensive feature vector of a single object as a node of the graph, construct the edges between the node and its neighboring nodes based on semantic weights and spatial distance weights, and calculate the description vector of the node through weight aggregation.

[0037] Combine the feature vectors of all objects in the current frame Consider it as the set of nodes in graph G.

[0038] S401 semantic weight calculation: using nodes and Semantic weights between To quantify the semantic similarity between the two, cosine similarity is calculated based on semantic feature vectors:

[0039] S402 Spatial Distance Weight Calculation: Based on the Euclidean distance between the center points of two nodes in the world coordinate system. Calculate spatial distance weights .

[0040] in, For Euclidean distance, The Gaussian kernel width parameter controls the sensitivity of the weights to distance, and its value is adjusted for different datasets and scenarios.

[0041] S403 Node Description Vector Aggregation: For node i, its final description vector By aggregating the features of itself and its neighboring nodes, and then using the aforementioned weights for weighted summation, we obtain:

[0042] in, The total number of neighboring nodes. The value is set to 3 because three objects distributed in space can typically form a stable spatial structure description. The denominator is the weight normalization factor, ensuring... Numerical stability.

[0043] This step constructs a refined object-level scene graph model. Unlike simple graph construction based on distance thresholds, this invention considers both semantic affinity and spatial proximity to define edge weights, resulting in more refined node description vectors. It not only includes the object's own attributes, but also explicitly encodes its semantic and spatial relationships with surrounding objects, forming a robust expression of the local scene structure.

[0044] Step S5: For historical keyframe images, the node description vector is calculated using the same method as in steps S1-S4. The difference is that the spatial distance weight is calculated using the object model parameters in the local map corresponding to the historical keyframe.

[0045] For historical keyframes, the processing flow is basically the same as that of the current frame. The core difference lies in the object center point coordinates used when calculating the spatial distance weight in step S4. The parameters do not originate directly from single-frame detection results, but rather from object model parameters in a local map associated with that historical keyframe, optimized through multi-frame bundle adjustment (BA) iterations. , call Replace the current frame ,Right now:

[0046] This design utilizes more accurate and stable map information from the optimized backend of the SLAM system to calculate spatial relationships, making the graph structure representation of historical frames more reliable, reducing the impact of single-frame observation noise, and improving the accuracy of cross-time slot matching.

[0047] Step S6 Figure 3 In the process of constructing the graph structure, cosine similarity is calculated based on the description vectors of each node in the current frame and the historical keyframes to obtain the similarity measure between the two. If the cosine similarity exceeds a set threshold, a one-to-one correspondence between nodes is obtained. If the number of nodes that can be matched in the current frame meets certain conditions, it is determined that a loopback frame has been found.

[0048] S601 Cosine Similarity Calculation: For the node description vectors of the current frame and historical frames, calculate the cosine similarity between them.

[0049] in, The closer the value is to 1, the more similar the features of the two frames are.

[0050] S602 Threshold Filtering and Node Matching: A similarity threshold is set using the validation set. .like If node i and node j are considered to be a potential matching pair, then bipartite graph matching methods such as the Hungarian Algorithm are typically used to find the optimal one-to-one correspondence among all potential matching pairs, thus obtaining the matching matrix.

[0051] S603 Loop Detection: Count the number of matching nodes A threshold for the number of nodes is set based on the density of objects in the scene. .like If the current frame and the historical keyframe form a loop, the loop constraint is sent to the SLAM backend for optimization.

[0052] In summary, this invention achieves deep fusion of visual appearance, geometric parameter textualization, and global contextual multimodal object features through S1-S3; it achieves refined scene graph construction and robust representation by fusing semantic and spatial information through S4-S5; and finally, it achieves accurate loop closure detection through S6. This method effectively overcomes the shortcomings of traditional methods, such as sensitivity to appearance changes, single feature set, and insufficient utilization of structural information, and significantly improves the accuracy and robustness of loop closure detection in complex dynamic environments.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A graph-based visual object-level SLAM loop closure detection method, characterized in that, Includes the following steps: S1. For the current RGB frame image of the input system, use deep learning to obtain the mask of the object in the image and the corresponding 3D detection box, encode the information of the whole image, and generate a global visual feature vector. S2. For each object in the image, the parameters of the 3D detection box are transformed into the world coordinate system through the pose of the current frame. The transformed parameter results are converted into a natural language description using a text template and encoded into a text feature vector using a text encoder. At the same time, individual objects are extracted and encoded to obtain the visual feature vector of the individual object. S3. The global visual feature vector obtained in step S1, the visual feature vector of a single object obtained in step S2, and the text feature vector are fused together using a multilayer perceptron to obtain the comprehensive feature vector of a single object. S4. Use the comprehensive feature vector of a single object as a node in the graph, construct the edges between the node and its neighboring nodes based on semantic weights and spatial distance weights, and calculate the description vector of the node through weight aggregation. S5. For historical keyframe images, the node description vector is calculated using the same method as steps S1-S4. The difference is that the spatial distance weight is calculated using the object model parameters in the local map corresponding to the historical keyframe. S6. Calculate the cosine similarity between the description vectors of each node in the current frame and the historical keyframes to obtain the similarity measure between the two. If the cosine similarity exceeds the set threshold, a one-to-one correspondence between the nodes is obtained. If the number of nodes that can be matched in the current frame meets certain conditions, it is determined that a loopback frame has been found.

2. The graph-based visual object-level SLAM loop closure detection method according to claim 1, characterized in that, Step S1 includes: A pre-trained deep learning model is used to perform semantic segmentation and 3D detection tasks on the current RGB frame image; pixel-level masks of all objects in the image are output simultaneously. and the corresponding 3D detection box , in: : The coordinates of the center point of the 3D detection box in the camera coordinate system; : The length, width, and height of the 3D inspection frame; The 3D detection box revolves in the camera coordinate system. The rotation angle of the axis; Simultaneously, a global feature encoding of the entire image is performed using a convolutional neural network or a Transformer model, outputting a global visual feature vector with fixed dimensions. .

3. The graph-based visual object-level SLAM loop closure detection method according to claim 1, characterized in that, Step S2 includes: S201 coordinate system transformation: combining the initial pose transformation matrix of the current frame output from the SLAM front-end odometry. ,in, For rotation matrix, The translation vector transforms the center point of the 3D detection box from the camera coordinate system to the world coordinate system. The transformation formula is: ,in, These are the coordinates of the center point in the transformed world coordinate system. These are the parameters of the converted 3D detection box; S202 Text Feature Encoding: Text Template Based on Customization involves transforming parameters into standardized natural language statements, which are then encoded into text feature vectors using a pre-trained text encoder. ; S203 Local Visual Feature Encoding: Based on Object Mask Cropping out the image region of a single object ,in, This represents the masked region of the object; local visual feature vectors are obtained by encoding using a convolutional neural network or Transformer model. .

4. The graph-based visual object-level SLAM loop closure detection method according to claim 1, characterized in that, Step S3 includes: The multilayer perceptron adopts a three-stage structure of "input layer - hidden layer - output layer", where Input layer: Input vector is , () indicates vertical concatenation of vectors; Hidden layer: Achieves non-linear mapping through the ReLU activation function. ,in , Here, 2D represents the learnable parameters of the hidden layer; Regularization layer: Embed a BatchNorm layer and a Dropout layer after the hidden layer. ; Output layer: Outputs the comprehensive feature vector of a single object. ,in , For the learnable parameters of the output layer, This is the final integrated feature vector.

5. The graph-based visual object-level SLAM loop closure detection method according to claim 1, characterized in that, Step S4 includes: S401 Semantic Weight Calculation: Semantic Weight Quantization Node and The semantic similarity is calculated based on the cosine similarity of the semantic feature vectors, using the following formula: ,in, , , For nodes semantic sub-feature vectors, Represents the vector dot product. Represents the L2 norm; S402 Spatial Distance Weight Calculation: Spatial Distance Weight Based on the Euclidean distance mapping between two nodes in the world coordinate system, the formula is: ,in, For Euclidean distance, The Gaussian kernel width parameter; S403 Node Description Vector Aggregation: All node features are aggregated through weighted summation, using the following formula: ,in, This represents the total number of neighboring nodes.

6. The graph-based visual object-level SLAM loop closure detection method according to claim 5, characterized in that, The set of neighboring nodes It consists of the K nodes that are spatially closest to node i, where K is 3.

7. The graph-based visual object-level SLAM loop closure detection method according to claim 1, characterized in that, Step S5 includes: For historical keyframes, when calculating spatial distance weights, the object model parameters in the local map after multi-frame Bundle Adjustment optimization iterations are directly called. Replace the current frame The formula for calculating its spatial distance weight is: 。 8. The graph-based visual object-level SLAM loop closure detection method according to claim 1, characterized in that, Step S6 includes: S601 cosine similarity calculation: The formula for the similarity between node description vectors between the current frame and historical frames is: in, The closer the value is to 1, the more similar the features of the two frames are. S602 Threshold Filtering and Node Matching: Setting Similarity Thresholds ,like Then, the Hungarian algorithm is used to solve the bipartite graph matching problem, and the node correspondence matrix is ​​obtained. , Indicates the current frame node With historical frame nodes If it matches, return 0; otherwise, return 0. S603 Loop Detection: Count the number of matching nodes Set a threshold for the number of nodes. ,like If so, it is determined that a loopback frame has been found.

9. A graph-based visual object-level SLAM loop closure detection method according to any one of claims 1 to 8, characterized in that, The standardized natural language description format generated from the text template in step S2 is: "Center point in the world coordinate system" ,long ,Width ,high Rotation angle "objects".

10. A graph-based visual object-level SLAM loop closure detection method according to any one of claims 1 to 8, characterized in that, The deep learning models used in step S1 to generate the global visual feature vector, and in step S2 to generate the local visual feature vector and text feature vector, are pre-trained before deployment.

Images