A semantic space topology-driven multi-level SLAM loop closure detection method, device, medium, and product

Abstract

This application discloses a semantic space topology-driven multi-level SLAM loop closure detection method, device, medium, and product, relating to the field of computer vision. The method preprocesses the image data corresponding to keyframes using a region growing algorithm and the Laplacian operator; constructs semantic feature vectors based on the preprocessing results; calculates the similarity and similarity threshold between the current keyframe and non-connected keyframes to filter first-level loop closure candidate keyframes; filters second-level loop closure candidate keyframes using continuity conditions among the first-level loop closure candidate keyframes; constructs a BFS tree of the undirected complete graph of the current keyframe using instance abstract spatial information; calculates the similarity with the BFS tree of the undirected complete graph of the second-level loop closure candidate keyframes; and selects the candidate frame with the highest similarity as the loop closure keyframe; and achieves loop closure detection based on the relationship between the similarity of the loop closure keyframes and the threshold. This application solves the problem of inaccurate loop closure detection caused by changes in lighting and viewpoint in the scene.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a semantic space topology-driven multi-level SLAM loop closure detection method, device, medium, and product. Background Technology

[0002] In a complete SLAM (simultaneous localization and mapping) system, loop closure detection is a crucial component, helping to eliminate accumulated errors and optimize location information. Traditional loop closure detection methods rely on extracted, manually generated features and use a bag-of-words model for feature matching. However, this approach is susceptible to changes in lighting and viewpoint within the scene, leading to inaccurate loop closure detection. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, this application provides a semantic space topology-driven multi-level SLAM loop closure detection method, device, medium, and product.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a semantic space topology-driven multi-level SLAM loop closure detection method, including:

[0006] Acquire image data of the scene to be detected; the image data includes RGB images and depth maps of each frame;

[0007] The ORB-SLAM2 system is used to create and remove keyframes based on the image data and keyframes of the scene to be detected, and retains the image data corresponding to the keyframes.

[0008] The image data corresponding to the keyframe is preprocessed using a region growing algorithm and a Laplacian operator to obtain a preprocessing result. The preprocessing result includes instance segmentation results and category information. The instance is an entity object with the same connected region represented by the preprocessed image data corresponding to the keyframe.

[0009] Construct semantic feature vectors based on the preprocessing results;

[0010] Based on the semantic feature vector, the similarity between the current keyframe and the unconnected keyframes is determined; the unconnected keyframes are historical keyframes that are not connected to the current keyframe.

[0011] A similarity threshold is determined based on the maximum similarity between the current keyframe and the non-connected keyframes, and a first-level closed-loop candidate keyframe is obtained by filtering the non-connected keyframes based on the similarity threshold.

[0012] The first-level closed-loop candidate keyframes that satisfy the continuity condition are retained, and the retained first-level closed-loop candidate keyframes are used as second-level closed-loop candidate keyframes.

[0013] Based on the preprocessing results and the depth map, determine the spatial coordinates of the centroid of each instance pixel in the image data corresponding to the current keyframe in the current keyframe coordinate system.

[0014] Transform the spatial coordinates of the centroids of each instance pixel in the current keyframe to the reference coordinate system to obtain the transformed unified spatial coordinate information of the centroids of the instance pixels.

[0015] Using the transformed unified coordinate information of the centroid space of the instance pixels and the category information, a BFS (Breadth-First-Search) tree is constructed for the undirected complete graph of the current keyframe.

[0016] Determine the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-level loop closure candidate keyframe undirected complete graph, and take the second-level loop closure candidate keyframe with the highest similarity as the loop closure keyframe.

[0017] When the similarity between the closed-loop keyframe and the current keyframe exceeds a set threshold, a closed loop is detected and closed-loop correction is performed.

[0018] If the similarity between the closed-loop keyframe and the current keyframe does not exceed a set threshold, then no closed loop is detected and no closed-loop correction is performed.

[0019] Optionally, the image data corresponding to the keyframes are preprocessed using a region growing algorithm and the Laplacian operator to obtain preprocessed results, including:

[0020] Based on the keyframe creation and removal conditions in the ORB-SLAM2 system, a keyframe queue is generated and entered into the closed-loop detection thread, and the first frame in the keyframe queue is taken as the current keyframe.

[0021] The semantic features of the RGB image corresponding to the current keyframe are extracted using a semantic segmentation network to obtain a mask image;

[0022] The Laplacian operator is used to identify the edges of abrupt depth changes in the depth map corresponding to the current keyframe, and the processed depth map is obtained.

[0023] Align the mask image with the processed depth map, assign special values ​​to the edges, and obtain the edges of each instance;

[0024] The aligned image is processed using a region growing algorithm to obtain the preprocessed result of the image data corresponding to the current keyframe.

[0025] Optionally, constructing a semantic feature vector based on the preprocessing result includes:

[0026] A semantic feature dictionary is constructed based on the preprocessing results;

[0027] Different weights are assigned to each semantic category in the semantic feature dictionary to obtain the semantic feature vector.

[0028] Optionally, based on the semantic feature vector, the similarity between the current keyframe and non-connected keyframes is determined, including:

[0029] Based on the semantic feature vector, the similarity between the current keyframe and non-connected keyframes is determined using Euclidean distance.

[0030] Optionally, a similarity threshold is determined based on the maximum similarity between the current keyframe and non-connected keyframes, including:

[0031] The similarity threshold is defined as σ times the maximum similarity between the current keyframe and non-connected keyframes.

[0032] Optionally, the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-order loop-closure candidate keyframe undirected complete graph is determined, including:

[0033] The similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-level closed-loop candidate keyframe undirected complete graph is determined by using the vertex set and edge set of the BFS tree of the current keyframe undirected complete graph. The vertex set of the BFS tree of the current keyframe undirected complete graph is determined based on the semantic category to which the centroid of the current keyframe pixel belongs, the unified coordinates of the pixel centroid space, and the origin information. The edge set of the BFS tree of the current keyframe undirected complete graph is determined based on the magnitude of the vector from the origin to the pixel centroid.

[0034] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the semantic space topology-driven multi-level SLAM loop closure detection method provided above.

[0035] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the semantic space topology-driven multi-level SLAM loop closure detection method provided above.

[0036] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the semantic space topology-driven multi-level SLAM loop closure detection method provided above.

[0037] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0038] This application provides a semantic space topology-driven multi-level SLAM loop closure detection method, device, medium, and product. By constructing semantic feature vectors based on preprocessing results, it utilizes semantic features and spatial topology information to replace the traditional bag-of-words strategy. Compared to the traditional bag-of-words approach, it exhibits higher adaptability to changes in illumination and viewpoint, significantly improving the robustness of loop closure detection. The first-level loop closure candidate keyframes are effectively filtered to remove historical keyframes with significantly different instance types and numbers. Furthermore, the semantic feature vectors are used to further determine the continuity of the first-level loop closure candidate keyframes. Second-level loop closure candidate keyframes are obtained based on the first-level matching, ensuring the efficiency of loop closure detection. The semantic space topology information is fully utilized, ensuring the accuracy of loop closure detection. Therefore, loop closure detection using first-level and second-level matching can balance robustness, efficiency, and accuracy. The BFS tree of the undirected complete graph of the current keyframe is used to associate the spatial topology information of the current keyframe with that of the second-level loop closure candidate keyframes and calculate similarity. Compared to other existing topology matching methods, it has faster processing speed and good accuracy. Furthermore, after completing the first-level and second-level matching, this application only retains the BFS tree and semantic feature vector of the current keyframe undirected complete graph, while its RGB image and depth image are discarded, effectively preventing memory waste. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is an application environment diagram of a semantic space topology-driven multi-level SLAM loop closure detection method in one embodiment of this application;

[0041] Figure 2 A flowchart illustrating a semantic space topology-driven multi-level SLAM loop closure detection method provided in an embodiment of this application;

[0042] Figure 3 A schematic diagram illustrating the result obtained using a region growing algorithm, provided in another embodiment of this application;

[0043] Figure 4 A schematic diagram illustrating the construction of keyframe semantic feature vectors provided in another embodiment of this application;

[0044] Figure 5This is a schematic diagram of a continuity determination process provided in an embodiment of this application;

[0045] Figure 6 A schematic diagram of the centroid spatial coordinate transformation of each keyframe instance pixel provided in another embodiment of this application;

[0046] Figure 7 This is an undirected complete graph of the current keyframe spatial information in the reference coordinate system provided in another embodiment of this application;

[0047] Figure 8 A schematic diagram of the BFS tree of an undirected complete graph in the current keyframe reference coordinate system and a node matching node for a certain node, provided for another embodiment of this application;

[0048] Figure 9 A schematic diagram illustrating the implementation process of a semantic space topology-driven multi-level SLAM loop closure detection method provided in an embodiment of this application;

[0049] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] To improve the robustness of loop closure detection algorithms, combining semantic information for location recognition has become an effective strategy. Based on this, embodiments of this application provide a semantic space topology-driven multi-level SLAM loop closure detection method, which can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send image data of the scene to be detected to server 104. After receiving the image data, server 104 uses the ORB-SLAM2 system to create and remove keyframes based on the image data and keyframe creation and removal conditions of the scene to be detected, obtaining keyframes and their corresponding image data. The image data corresponding to the keyframes is preprocessed using a region growing algorithm and the Laplacian operator to obtain preprocessing results. A semantic feature vector is constructed based on the preprocessing results. The similarity between the current keyframe and non-connected keyframes is determined based on the semantic feature vector. A similarity threshold is determined based on the maximum similarity between the current keyframe and non-connected keyframes, and first-level closed-loop candidate keyframes are selected from the non-connected keyframes based on the similarity threshold. First-level loop closure candidate keyframes that meet the continuity condition are retained, and these retained first-level loop closure candidate keyframes are used as second-level loop closure candidate keyframes. Based on the preprocessing results, the spatial coordinates of the centroids of each instance pixel in the image data corresponding to the current keyframe are determined in the current keyframe coordinate system. The spatial coordinates of the centroids of each instance pixel in the current keyframe are transformed to a reference coordinate system to obtain the transformed coordinate information. A BFS tree for the undirected complete graph of the current keyframe is constructed using the transformed unified spatial coordinate information of the instance pixel centroids and category information. The similarity between the undirected complete graph BFS tree of the current keyframe and the undirected complete graph BFS tree of the second-level loop closure candidate keyframes is determined, and the second-level loop closure candidate keyframe with the highest similarity is used as the loop closure keyframe. Loop closure detection is achieved based on the relationship between the similarity between the loop closure keyframe and the current keyframe and a set threshold. The server 104 can feed back the obtained loop closure detection results to the terminal 102. Furthermore, in some embodiments, the semantic space topology-driven multi-level SLAM loop closure detection method provided in this application can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform loop closure detection on the image data of the scene to be detected, or the server 104 can obtain the image data of the scene to be detected from the data storage system and perform loop closure detection on the image data of the scene to be detected.

[0053] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc.

[0054] In one exemplary embodiment, such as Figure 2As shown, a semantic space topology-driven multi-level SLAM loop closure detection method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 200 to 212. Wherein:

[0055] Step 200: Acquire image data of the scene to be detected. The image data includes RGB images and depth maps of each frame.

[0056] Step 201: Using the ORB-SLAM2 system, keyframes are created and removed based on the image data and keyframes of the scene to be detected, and the image data corresponding to the keyframes is retained.

[0057] Step 202: Preprocess the keyframe-corresponding image data using a region growing algorithm and the Laplacian operator to obtain the preprocessing results. The preprocessing results include instance segmentation results and category information. An instance is an entity object represented by a connected region obtained based on the keyframe-corresponding image data preprocessing (e.g., segmentation).

[0058] Step 203: Construct semantic feature vectors based on the preprocessing results.

[0059] Step 204: Based on the semantic feature vector, determine the similarity between the current keyframe and non-connected keyframes. Non-connected keyframes are historical keyframes that are not connected to the current keyframe.

[0060] Step 205: Determine the similarity threshold based on the maximum similarity between the current keyframe and the non-connected keyframes, and select first-level closed-loop candidate keyframes from the non-connected keyframes based on the similarity threshold.

[0061] Step 206: Retain the first-level closed-loop candidate keyframes that meet the continuity condition, and use the retained first-level closed-loop candidate keyframes as second-level closed-loop candidate keyframes.

[0062] Step 207: Based on the preprocessing results and depth map, determine the spatial coordinates of the centroid of each instance pixel in the image data corresponding to the current keyframe in the coordinate system of the current keyframe.

[0063] Step 208: Transform the spatial coordinates of the centroids of each instance pixel in the current keyframe to the reference coordinate system to obtain the transformed unified spatial coordinate information of the centroids of the instance pixels.

[0064] Step 209: Construct a BFS tree for the undirected complete graph of the current keyframe using the transformed spatial unified coordinate information and the category information.

[0065] Step 210: Determine the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-level loop closure candidate keyframe undirected complete graph, and take the second-level loop closure candidate keyframe with the highest similarity as the loop closure keyframe.

[0066] Step 211: When the similarity between the closed-loop keyframe and the current keyframe exceeds the set threshold, a closed loop is detected and closed-loop correction is performed.

[0067] Step 212: If the similarity between the closed-loop keyframe and the current keyframe does not exceed the set threshold, then no closed loop is detected and no closed-loop correction is performed.

[0068] By implementing steps 200 and 212 above, this application can significantly improve the robustness of closed-loop detection, improve detection accuracy while ensuring closed-loop detection efficiency, and has a faster processing speed and good accuracy and robustness compared to using other spatial topologies.

[0069] In another exemplary embodiment of this application, in steps 200 and 201, an RGB-D camera can be used to acquire RGB images and depth maps of each frame of the environment or scene to be detected. The acquired image data is input into the ORB-SLAM2 system. The inherent function of the ORB-SLAM2 system is to track each frame to create keyframes and estimate poses. Before keyframe creation, the RGB images and depth maps of each frame are not discarded. When the ORB-SLAM2 system reaches the keyframe creation stage, the RGB images and depth maps corresponding to the ordinary frames created as keyframes are retained, while non-keyframe image data are discarded. After the keyframes pass through the local mapping thread, redundant keyframes are removed, and their image data is also removed. The remaining keyframes and their image data are used as input to the loop closure detection thread. The standard for the local mapping thread to propose redundant keyframes is: if more than 90% of the map points in the created keyframe can be observed by more than three other keyframes, they will be removed. Map points are the three-dimensional points in space corresponding to two-dimensional feature points.

[0070] In another exemplary embodiment of this application, the implementation process of step 202 above may be:

[0071] Step 2021: Based on the keyframe creation and culling conditions in the ORB-SLAM2 system, generate a keyframe queue and enter it into the closed-loop detection thread, and take the first frame of the keyframe queue as the current keyframe.

[0072] Step 2022: Extract semantic features from the RGB image corresponding to the current keyframe using a semantic segmentation network to obtain a mask image.

[0073] Step 2023: For the depth map corresponding to the current keyframe, considering the significant differences in depth information among different instances, the Laplacian operator is used to identify edges of abrupt depth changes in the depth map corresponding to the current keyframe, resulting in the processed depth map. For example, the Laplacian operator takes the following form on the image:

[0074]

[0075] In the formula, and This represents the second-order partial derivative of the image at the point (x,y). Let f(*,*) represent the Laplacian operator, f(*,*) represent the image, and (*,*) represent the pixels in the image.

[0076] The Laplacian operator is represented in the image as a convolution kernel. Taking into account all pixels surrounding each pixel in the depth map, the original Laplacian convolution kernel is expanded as follows:

[0077]

[0078] Convolving the depth map with an extended Laplacian kernel K can identify regions with significant depth variations. To distinguish instances, an appropriate threshold Th can be set. border The results are binarized to detect the edges of instances, based on the reasonable assumption that the distance from the camera to the same instance remains relatively stable.

[0079] Step 2024: Align the mask image with the processed depth map, assign special values ​​to the edges, and obtain the edges of each instance.

[0080] Step 2025: Process the aligned image using a region growing algorithm to obtain the preprocessed result of the image data corresponding to the current keyframe. The result obtained using the region growing algorithm is as follows: Figure 3 As shown, connected components with the same characteristics are segmented, thus obtaining the number of connected components, i.e., the number of instances, and determining a threshold Th. pixel_num When the number of pixels in a connected region is less than the threshold Th pixel_num The number of corresponding categories is not increased at any time, and special cases such as ground and walls are also excluded, ultimately achieving the purpose of instance segmentation and counting of the number of each category.

[0081] In another exemplary embodiment of this application, in order to obtain the connected regions of each instance using a region growing algorithm, step 2025 can be implemented as follows:

[0082] (1) Randomly select one or more unlabeled pixels in the aligned image as seed pixels, assign a label value to the seed pixels, and add them to the region set as a region.

[0083] (2) In each iteration, select a region from the region set, traverse the 8 neighborhood pixels of that region, and determine whether the difference between the pixel and the seed pixel is within the specified threshold Th. growth If a pixel is present in the region, add it to that region and mark it as visited. If a new pixel is added to the region, update the region set.

[0084] (3) Repeat step (2) until no new pixels can be added to the region. Return to step (1) until every pixel of the aligned image has been visited or no new pixels can be added to the region, and obtain the processing result of the current keyframe.

[0085] Region growing algorithms can segment connected regions with similar characteristics. From this, the number of connected regions, i.e., the number of instances, can be determined, and a threshold Th can be established. pixel_num When the number of pixels in a connected region is less than the threshold Th pixel_num Without increasing the number of its corresponding categories, the goal of instance segmentation and counting of the number of each category is ultimately achieved.

[0086] In another exemplary embodiment of this application, in order to significantly improve the robustness of loop closure detection and solve the problem that the original technology using bag-of-words features for loop closure detection is greatly affected by changes in illumination and viewpoint, this application uses semantic features and spatial topological information instead of the traditional bag-of-words strategy. Semantic information has a higher adaptability to changes in illumination and viewpoint compared to the traditional bag-of-words approach. Based on this, in this embodiment, the implementation process of step 203 above can be to construct the semantic feature vector of the current keyframe based on the preprocessing results, according to the semantic category and number statistics. Wherein:

[0087] Step 2031: Construct a semantic feature dictionary based on the preprocessing results. For example, construct the semantic feature dictionary of the current keyframe by analyzing the categories and numbers of connected components of each category of instances in the image corresponding to the current keyframe obtained from the above segmentation. The semantic feature dictionary is represented as D. c :

[0088] D c ={A:x cA ,B:x cB ,C:x cC ,…,α:x cα}

[0089] In the formula, A to α represent the semantic category of the current keyframe, and x cA ~x cα This indicates the total number of instances of each category in the current keyframe.

[0090] Step 2032: Assign different weights to each semantic category in the semantic feature dictionary to obtain a semantic feature vector. For example, assign different weights ω based on the specificity of each semantic category in the scene to improve the accuracy of loop closure detection. Multiply each weight by the total number of instances corresponding to it to obtain the following: Figure 4 The semantic feature vector of the current keyframe shown is denoted as V. c :

[0091] V c =[x cA ω A ,x cB ω B ,x cC ω C ,…,x cα ω α ].

[0092] Where, ω A ~ω α This represents the weight of each semantic category.

[0093] In another exemplary embodiment of this application, the similarity with non-connected keyframes is calculated based on semantic feature vectors to filter out first-level closed-loop candidate keyframes, and continuity judgment is performed, such as... Figure 5 As shown, secondary closed-loop candidate keyframes are selected, completing the primary matching. Based on this, the implementation process of steps 205 and 206 above can be as follows:

[0094] Check if a closed loop has just been detected; if it exceeds a certain number of frames from the previous closed loop, then... loop Then, the semantic feature vector of the current keyframe is compared with the semantic feature vectors of the non-connected keyframes (also referred to as non-connected historical keyframes in this application) to calculate similarity. In practical applications, Euclidean distance can be used to calculate the similarity between the current keyframe and the non-connected keyframes. For example, this similarity is denoted as S. V (KF c KF h The calculation formula is as follows:

[0095]

[0096] In the formula, KF c Indicates the current keyframe, KF h Indicates non-connected historical keyframes, x ci Indicates the number of instances of each category in the current keyframe, x hi This indicates the number of instances of each category in non-connected historical keyframes.

[0097] Since historical keyframes connected to the current keyframe share many views with the current keyframe, they are usually more similar to the current keyframe. To eliminate the influence of connected historical keyframes, this application sets a threshold Th. connected Extract the previous Th frame of the current keyframe. connected Connected historical keyframes within a frame are removed during screening.

[0098] Furthermore, similarity is calculated with all non-connected keyframes that meet the criteria to obtain the maximum similarity S. max And using the maximum similarity S max Determine the relative threshold. The relative threshold is determined by the maximum similarity S. max The similarity is σ times the relative threshold, where σ is between 0 and 1. Non-connected historical keyframes with a similarity greater than the relative threshold are first-level closed-loop candidate keyframes. Thus, by using the relative threshold, all first-level closed-loop candidate keyframes that meet the conditions can be found in the historical keyframes.

[0099] Based on the above description, the semantic feature vector of the current keyframe is constructed according to the preprocessing results. The semantic feature vector is used as the basis to judge the similarity between the current keyframe and the non-connected historical keyframes, and the first-level closed-loop candidate keyframes are obtained.

[0100] Based on the above description, according to the SLAM system's observations of map points from keyframes and the observations of keyframes from map points, common-view keyframes of the current keyframe's first-level loop closure candidate keyframes are identified. These first-level loop closure candidate keyframes and their common-view keyframes are grouped into loop closure candidate keyframe groups. For example, assuming there are n loop closure candidate keyframes, there are n loop closure candidate keyframe groups. In practical applications, the loop closure candidate keyframe groups of the current keyframe are retained until the loop closure detection thread processes the next two frames, avoiding memory waste.

[0101] like Figure 5 As shown, the closed-loop candidate keyframe group of the current keyframe is compared with the closed-loop candidate keyframe groups of the previous two frames. If there is a keyframe in the closed-loop candidate keyframe group whose semantic feature vector is the same as a keyframe in the closed-loop candidate keyframe group of the previous two consecutive frames, then the closed-loop candidate keyframe is determined to satisfy the continuity and is retained in the candidate frame queue. Otherwise, it is removed. Through this operation, the second-level closed-loop candidate keyframe can be obtained, and the first-level matching is completed.

[0102] In another exemplary embodiment of this application, based on the connected regions of each instance in the image data corresponding to the current keyframe, the spatial coordinates of the centroid of each instance pixel in the current keyframe coordinate system can be calculated, and then transformed to a unified coordinate system in the reference coordinate system, such as... Figure 6As shown. Based on this, edge sets and vertex sets are constructed using the unified coordinate information and category information of the instance pixel centroid space, thereby further constructing the BFS tree of the current keyframe's undirected complete graph, as follows. Figure 7 and Figure 8 As shown. The similarity is calculated between the BFS tree of the current keyframe's undirected complete graph and the BFS tree of the second-level closed-loop candidate keyframe's undirected complete graph, thus obtaining the closed-loop keyframe and completing the second-level matching. Based on this, the implementation process of steps 207 to 210 of this application can be as follows:

[0103] (1) After obtaining the connected regions of each instance using the region growing algorithm, the centroid of the pixel is used to represent the spatial structure information of the connected region. The value of each pixel in the image is taken as the mass of that pixel. The formula for determining the two-dimensional coordinates of the centroid in a two-dimensional image is expressed as:

[0104]

[0105] Where n is the number of pixels in the connected region, p i Represents pixel value, u i v i Represents pixel coordinates, u c v c This represents the coordinates of the centroid. Further, we can obtain:

[0106]

[0107] From this, we can obtain the two-dimensional coordinates (u) of the pixel centroid of each instance. c ,v c ).

[0108] Next, the two-dimensional coordinates of the centroid are mapped onto the depth map of the current keyframe to obtain the depth Z of the centroid of each instance. c The three-dimensional spatial coordinates of the centroid of each instance under the current keyframe observation are obtained from the camera imaging principle and are expressed as follows:

[0109]

[0110] (2) In order to enable the spatial structure information of the current keyframe and the closed-loop candidate frame to participate in unified matching, it is necessary to use the SLAM system front-end to optimize the pose and transform the three-dimensional spatial coordinates of the pixel centroid in the current keyframe coordinate system to the spatial coordinates in the reference coordinate system. As a result, the spatial coordinates of the instance pixel centroids of the current keyframe and the candidate frame are unified to the same coordinate system, which can be used for matching.

[0111] Through the SLAM system's tracking and local mapping modules, the rotation matrix R and translation vector t estimated for each keyframe relative to the previous keyframe can be obtained. This allows us to describe a coordinate transformation relationship in Euclidean space as follows:

[0112] P w =R w1 P1+t w1 .

[0113] Among them, P w R represents the coordinates of the centroid of the first frame pixel in the world coordinate system. w1 This represents the rotation matrix that rotates the first frame coordinates to the world coordinate system. P1 represents the coordinates of the centroids of each instance pixel in the first frame in the first frame coordinate system. t w1 This represents the translation vector that shifts the coordinates of the first frame to the world coordinate system.

[0114] Since the centroids of pixels in subsequent keyframes need to undergo multiple transformations, a transformation matrix T is introduced for ease of transformation, expressed as:

[0115]

[0116] In the formula, T w1 This represents the transformation matrix that transforms the homogeneous coordinates of the first frame to homogeneous coordinates in the world coordinate system.

[0117] remember For P * Given the homogeneous coordinates of the pixel centroid in the second frame coordinate system, the coordinate transformation of the homogeneous coordinates in the world coordinate system can be expressed as:

[0118]

[0119] In the formula, T 12 This represents the transformation matrix that transforms the coordinates of the second frame to the coordinates of the first frame. This represents the homogeneous coordinates of the centroids of each instance pixel in the second frame in the second frame coordinate system. This represents the homogeneous coordinates of the centroids of each instance pixel in the second frame in the world coordinate system.

[0120] Continuing the recursion, the coordinate transformation from the homogeneous coordinates of the pixel centroid space in the nth frame coordinate system to the world coordinate system is expressed as:

[0121]

[0122] This allows the centroid coordinates of the instance pixels in the current keyframe to be unified to the reference coordinate system, such as... Figure 6 As shown, this is for subsequent secondary matching.

[0123] (3) The origin of the reference coordinate system is taken as the starting node, marked as (none, 0, 0, 0), and used as node information. The instance categories of the current keyframe and their unified spatial coordinate information (class, x, y, z) in the world coordinate system are also recorded as node information. By linking the origin to each instance's centroid coordinate node pairwise, theoretically, an undirected complete graph G = <V, E> of the current keyframe's centroid coordinates can be obtained, such as... Figure 7 As shown. Here, V is a finite, non-empty set of node information, called the vertex set, and E is a set of binary pairs of nodes, called the edge set. The weight of an edge is the Euclidean distance between the coordinates of the two nodes, denoted as E(a,b):

[0124]

[0125] In the formula, (x a ,y a ,z a (x) represents the coordinates of node a. b ,y b ,z b ) represents the coordinates of node b.

[0126] Initialize the BFS tree of the graph using the origin as the starting node, thus obtaining the BFS tree BFST of the undirected complete graph of the current keyframe centroid space coordinates.<V,E> ,like Figure 8 As shown. In practice, the construction process of the undirected complete graph can be omitted. The BFS tree of the current keyframe's undirected complete graph can be directly constructed using the vertex set information V and the edge set E between the starting node and the pixel centroid node. The steps for initializing the BFS tree of the current keyframe's undirected complete graph with the origin as the starting node are as follows:

[0127] (a) Initialize the BFS tree by adding the starting node to the tree.

[0128] (b) Add the starting node to the queue to be visited.

[0129] (c) For each node in the queue to be visited, iterate through all its neighboring nodes in turn.

[0130] (d) For each neighbor node, if the neighbor node has not yet been added to the BFS tree, add it and set the parent node of the neighbor node as the current node.

[0131] (e) Remove visited nodes from the queue and add each neighboring node to the queue to be visited.

[0132] (f) Repeat steps (c) through (e) until the queue is empty.

[0133] Through the above steps, it can be understood that the BFS tree of the current keyframe takes the origin of the world coordinate system as the parent node and the centroid nodes of the remaining pixels as its child nodes.

[0134] (4) Match the BFS tree of the current keyframe undirected complete graph with the BFS tree of the second-level loop closure candidate keyframe undirected complete graph. (Using a certain second-level loop closure candidate keyframe KF...) can For example, traverse the current keyframe KF cur Undirected complete graph BFS tree vertex set V cur Information on the centroid nodes of each instance in the dataset, and search for KF. can Undirected complete graph BFS tree vertex set V can Find the centroid nodes of the same type in the dataset and use Euclidean distance to select the closest one to V. cur The centroid node of the instance is located at its spatial coordinates. Let V be the centroid node. cur The spatial information of the centroid node of a certain instance is A. cur V can The spatial information of the matching centroid nodes is A. can Furthermore, different weights ω can be assigned according to the specificity of semantic categories in the scene, such as:

[0135]

[0136] Among them, S(A cur A can ) represents the similarity between matching centroid nodes, ||A cur ||and|A can || is the magnitude of the vector from the origin to the centroid. Using the index of the centroid node in the vertex set, the magnitude of each centroid vector can be found in the edge set E of the undirected complete graph BFS tree in each frame. and ω represents the coordinates of each dimension in the spatial information of the centroid node of the instance pixel. Euc and ω cos For similarity weights, ω Euc +ω cos =1, generally take ω Euc >ω cos If a centroid node in the current keyframe does not have a corresponding centroid node in the candidate closed-loop keyframe, then the similarity is 0.

[0137] Furthermore, the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-level loop-closure candidate keyframe undirected complete graph is calculated using the similarity of the centroid nodes of each instance pixel in the current keyframe, and is expressed as:

[0138]

[0139] Where N is the number of centroid nodes of the current keyframe instance pixels.BFS This represents the similarity between the current keyframe and the undirected complete graph BFS tree of the second-level closed-loop candidate keyframe.

[0140] Finally, the candidate keyframe with the highest similarity in the second-level closed loop was selected as the closed loop keyframe.

[0141] Based on the above description, in this embodiment, the process of determining the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-order loop-closing candidate keyframe undirected complete graph can be described as follows:

[0142] The similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-level closed-loop candidate keyframe undirected complete graph is determined by using the vertex set and edge set of the BFS tree of the current keyframe undirected complete graph. The vertex set of the current keyframe undirected complete graph BFS tree is determined based on the semantic category of the centroid of each instance pixel in the current keyframe and the unified spatial coordinates of the centroid of each instance pixel. The edge set of the current keyframe undirected complete graph BFS tree is determined based on the magnitude of the vector from the origin to the pixel centroid.

[0143] In another exemplary embodiment of this application, if the similarity between the closed-loop keyframe and the current keyframe does not exceed the threshold Th sim If no loop closure is detected, no loop closure correction is performed, and the current keyframe is added to the history keyframes. If the threshold Th is exceeded... sim If a loop is detected, loop closure correction is performed, and the current keyframe is added to the history keyframes.

[0144] In another exemplary embodiment of this application, in order to effectively prevent memory waste, after completing the first and second level matching, only the spatial topology information and semantic feature vector of the current keyframe are retained, while its RGB image and depth image are discarded. That is, the BFS tree and semantic feature vector of the undirected complete graph of the current keyframe are retained, while the RGB image and depth image are discarded.

[0145] In summary, in specific implementation, the implementation process of the semantic space topology-driven multi-level SLAM loop closure detection method provided in this application can be as follows: Figure 9 As shown. Furthermore, based on the above description, this application has the following advantages compared to the prior art:

[0146] 1. Compared with the original technology that uses bag-of-words features for loop closure detection, which is greatly affected by changes in lighting and viewing angle, this application uses semantic features and spatial topological information to replace the traditional bag-of-words strategy. Compared with the traditional bag-of-words strategy, it has higher adaptability to changes in lighting and viewing angle. Using semantic features can significantly improve the robustness of loop closure detection.

[0147] 2. Compared to existing technologies, using first-level and second-level matching can effectively and accurately filter out loop-closing keyframes. First-level matching effectively filters historical keyframes with significantly different instance types and numbers, and further uses semantic feature vectors to determine the continuity of first-level loop-closing candidate keyframes, ensuring the efficiency of loop detection. Second-level matching, building upon first-level matching, fully utilizes semantic space topological information, ensuring the accuracy of loop detection. Therefore, it can be seen that this application's multi-level matching-based loop detection achieves a good balance between efficiency and accuracy.

[0148] 3. Compared to existing technologies, the semantic space topological information constructed in this application adds an additional dimension of information compared to the traditional two-dimensional bag-of-words features. Furthermore, calculating the similarity between the current keyframe and secondary loop-closing candidate keyframes using a BFS tree of the undirected complete graph of keyframes has a faster processing speed than using other spatial topologies, while also exhibiting good accuracy and robustness.

[0149] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores loop closure detection data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a semantic space topology-driven multi-level SLAM loop closure detection method.

[0150] Those skilled in the art will understand that Figure 10 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0151] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0152] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0155] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A semantic space topology-driven multi-level SLAM loop closure detection method, characterized in that, The semantic space topology-driven multi-level SLAM loop closure detection method includes: Acquire image data of the scene to be detected; the image data includes RGB images and depth maps of each frame; The ORB-SLAM2 system is used to create and remove keyframes based on the image data and keyframes of the scene to be detected, and retains the image data corresponding to the keyframes. The image data corresponding to the keyframes are preprocessed using a region growing algorithm and the Laplacian operator to obtain preprocessing results. The preprocessing results include instance segmentation results and category information. The instance is an entity object represented by a connected region obtained from the preprocessing of the image data corresponding to the keyframes. Construct semantic feature vectors based on the preprocessing results; Based on the semantic feature vector, the similarity between the current keyframe and the unconnected keyframes is determined; the unconnected keyframes are historical keyframes that are not connected to the current keyframe. A similarity threshold is determined based on the maximum similarity between the current keyframe and the non-connected keyframes, and a first-level closed-loop candidate keyframe is obtained by filtering the non-connected keyframes based on the similarity threshold. The first-level closed-loop candidate keyframes that satisfy the continuity condition are retained, and the retained first-level closed-loop candidate keyframes are used as second-level closed-loop candidate keyframes. Based on the preprocessing results and the depth map, determine the spatial coordinates of the centroid of each instance pixel in the image data corresponding to the current keyframe in the current keyframe coordinate system. Transform the spatial coordinates of the centroids of each instance pixel in the current keyframe to the reference coordinate system to obtain the transformed unified spatial coordinate information of the centroids of the instance pixels. A BFS tree for the undirected complete graph of the current keyframe is constructed using the transformed unified coordinate information of the centroid space of the instance pixels and the category information. Determine the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-level loop closure candidate keyframe undirected complete graph, and take the second-level loop closure candidate keyframe with the highest similarity as the loop closure keyframe. When the similarity between the closed-loop keyframe and the current keyframe exceeds a set threshold, a closed loop is detected and closed-loop correction is performed. If the similarity between the closed-loop keyframe and the current keyframe does not exceed a set threshold, then no closed loop is detected and no closed-loop correction is performed.

2. The semantic space topology-driven multi-level SLAM loop closure detection method according to claim 1, characterized in that, The image data corresponding to the keyframes are preprocessed using a region growing algorithm and the Laplacian operator to obtain the preprocessed results, including: Based on the keyframe creation and removal conditions in the ORB-SLAM2 system, a keyframe queue is generated and entered into the closed-loop detection thread, and the first frame in the keyframe queue is taken as the current keyframe. The semantic features of the RGB image corresponding to the current keyframe are extracted using a semantic segmentation network to obtain a mask image; The Laplacian operator is used to identify the edges of abrupt depth changes in the depth map corresponding to the current keyframe, and the processed depth map is obtained. Align the mask image with the processed depth map, assign special values ​​to the edges, and obtain the edges of each instance; The aligned image is processed using a region growing algorithm to obtain the preprocessed result of the image data corresponding to the current keyframe.

3. The semantic space topology-driven multi-level SLAM loop closure detection method according to claim 1, characterized in that, Based on the preprocessing results, a semantic feature vector is constructed, including: A semantic feature dictionary is constructed based on the preprocessing results; Different weights are assigned to each semantic category in the semantic feature dictionary to obtain the semantic feature vector.

4. The semantic space topology-driven multi-level SLAM loop closure detection method according to claim 1, characterized in that, Based on the semantic feature vector, the similarity between the current keyframe and non-connected keyframes is determined, including: Based on the semantic feature vector, the similarity between the current keyframe and non-connected keyframes is determined using Euclidean distance.

5. The semantic space topology-driven multi-level SLAM loop closure detection method according to claim 1, characterized in that, The similarity threshold is determined based on the maximum similarity between the current keyframe and non-connected keyframes, including: The similarity threshold is defined as σ times the maximum similarity between the current keyframe and non-connected keyframes.

6. The semantic space topology-driven multi-level SLAM loop closure detection method according to claim 1, characterized in that, Determine the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-order loop-closure candidate keyframe undirected complete graph, including: The similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the second-level closed-loop candidate keyframe undirected complete graph is determined by using the vertex set and edge set of the BFS tree of the current keyframe undirected complete graph. The vertex set of the BFS tree of the current keyframe undirected complete graph is determined based on the semantic category to which the centroid of the current keyframe pixel belongs, the unified coordinates of the pixel centroid space, and the origin information. The edge set of the BFS tree of the current keyframe undirected complete graph is determined based on the magnitude of the vector from the origin to the pixel centroid.

7. A computer device, comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the semantic space topology-driven multi-level SLAM loop closure detection method according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the semantic space topology-driven multi-level SLAM loop closure detection method as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the semantic space topology-driven multi-level SLAM loop closure detection method as described in any one of claims 1-6.

Images