Semantic space topology driven multistage SLAM closed-loop detection method and device, medium and product

Abstract

The invention discloses a semantic space topology driven multistage SLAM closed-loop detection method and device, a medium and a product, and relates to the field of computer vision. The method comprises the following steps: preprocessing image data corresponding to a key frame by using a region growing algorithm and a Laplacian operator; constructing a semantic feature vector based on a preprocessing result, calculating the similarity and a similarity threshold value of the current key frame and the non-connected key frame, and screening a first-level closed-loop candidate key frame; screening second-level closed-loop candidate key frames from the first-level closed-loop candidate key frames by utilizing a continuity condition; constructing a BFS tree of an undirected complete graph of the current key frame by using instance abstract space information, calculating the similarity with the BFS tree of the undirected complete graph of the secondary closed-loop candidate key frame, and taking the candidate frame with the highest similarity as a closed-loop key frame; and realizing closed-loop detection according to the relationship between the closed-loop key frame similarity and a threshold value. The problem that closed-loop detection cannot be accurately realized due to the influence of illumination and view angle change in a scene is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a semantic space topology driven multi-level SLAM closed-loop detection method, equipment, medium and product. Background Art

[0002] In a complete SLAM (simultaneous localization and mapping) system, closed loop detection is a key component that can help eliminate accumulated errors and optimize location information. Traditional closed loop detection methods are based on extracted artificial features and use bag-of-words models for feature matching. However, this method is easily affected by changes in lighting and perspective in the scene, resulting in inaccurate closed loop detection. Summary of the invention

[0003] In order to solve the above-mentioned problems existing in the prior art, the present application provides a semantic space topology driven multi-level SLAM closed-loop detection method, equipment, medium and product.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a multi-level SLAM closed-loop detection method driven by semantic space topology, comprising:

[0006] Acquire image data of the scene to be detected; the image data includes RGB images and depth maps of each frame;

[0007] The ORB-SLAM2 system is used to obtain key frames based on the image data of the scene to be detected and the key frame creation and elimination conditions, and the corresponding image data of the key frames are retained;

[0008] Preprocessing the image data corresponding to the key frame using a region growing algorithm and a Laplace operator to obtain a preprocessing result; the preprocessing result includes an instance segmentation result and category information; the instance is an entity object represented by the same connected region obtained based on the preprocessing of the image data corresponding to the key frame;

[0009] Constructing a semantic feature vector based on the preprocessing result;

[0010] Based on the semantic feature vector, determining the similarity between the current key frame and the non-connected key frame; the non-connected key frame is a historical key frame that is not connected to the current key frame;

[0011] Determine a similarity threshold based on the maximum similarity between the current key frame and the non-connected key frames, and select the first-level closed-loop candidate key frames from the non-connected key frames based on the similarity threshold;

[0012] Retaining the first-level closed-loop candidate key frames that meet the continuity condition, and using the retained first-level closed-loop candidate key frames as second-level closed-loop candidate key frames;

[0013] Determine the spatial coordinates of the centroid of each instance pixel in the image data corresponding to the current key frame in the coordinate system of the current key frame based on the preprocessing result and the depth map;

[0014] The spatial coordinates of the centroid of each instance pixel of the current key frame are transformed into the reference coordinate system to obtain the uniform spatial coordinate information of the centroid of each instance pixel after the transformation;

[0015] Constructing a BFS (Breadth-First-Search) tree of the undirected complete graph of the current key frame using the transformed instance pixel centroid space unified coordinate information and the category information;

[0016] Determine the similarity between the BFS tree of the undirected complete graph of the current key frame and the BFS tree of the undirected complete graph of the secondary closed-loop candidate key frame, and use the secondary closed-loop candidate key frame with the highest similarity as the closed-loop key frame;

[0017] When the similarity between the closed-loop key frame and the current key frame exceeds a set threshold, a closed loop is detected and closed-loop correction is performed;

[0018] When the similarity between the closed-loop key frame and the current key frame does not exceed the set threshold, the closed loop is not detected and no closed-loop correction is performed.

[0019] Optionally, the image data corresponding to the key frame is preprocessed using a region growing algorithm and a Laplace operator to obtain a preprocessing result, including:

[0020] Based on the key frame creation and elimination conditions in the ORB-SLAM2 system, a key frame queue is generated to enter the closed-loop detection thread, and the head frame in the key frame queue is taken out as the current key frame;

[0021] The RGB image corresponding to the current key frame is used to extract semantic features using a semantic segmentation network to obtain a mask image;

[0022] The Laplacian operator is used to identify the edge of the depth mutation in the depth map corresponding to the current key frame, and the processed depth map is obtained;

[0023] Aligning the mask image with the processed depth map, performing special assignment on the edge, and obtaining the edge of each instance;

[0024] The aligned images are processed using a region growing algorithm to obtain the preprocessing result of the image data corresponding to the current key frame.

[0025] Optionally, constructing a semantic feature vector based on the preprocessing result includes:

[0026] Building a semantic feature dictionary based on the preprocessing results;

[0027] Different weights are assigned to each semantic category in the semantic feature dictionary to obtain the semantic feature vector.

[0028] Optionally, determining the similarity between the current key frame and the non-connected key frame based on the semantic feature vector includes:

[0029] Based on the semantic feature vector, the similarity between the current key frame and the non-connected key frames is determined using the Euclidean distance.

[0030] Optionally, determining a similarity threshold based on a maximum similarity between the current key frame and the non-connected key frames includes:

[0031] The similarity threshold is taken as σ times the maximum similarity between the current key frame and the non-connected key frames.

[0032] Optionally, determining the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the secondary closed-loop candidate keyframe undirected complete graph includes:

[0033] The similarity between the BFS tree of the current key frame undirected complete graph and the BFS tree of the secondary closed-loop candidate key frame undirected complete graph is determined by using the vertex set and edge set of the BFS tree of the key frame undirected complete graph; the vertex set of the BFS tree of the current key frame undirected complete graph is determined based on the semantic category of the pixel centroid of the current key frame and the unified coordinates of the pixel centroid space and the origin information; the edge set of the BFS tree of the current key frame undirected complete graph is determined based on the modulus length from the origin to the pixel centroid vector.

[0034] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the semantic space topology driven multi-level SLAM closed-loop detection method provided above.

[0035] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the semantic space topology-driven multi-level SLAM closed-loop detection method provided above.

[0036] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the semantic space topology driven multi-level SLAM closed-loop detection method provided above.

[0037] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0038] The present application provides a semantic space topology driven multi-level SLAM closed-loop detection method, equipment, medium and product, by constructing a semantic feature vector based on the preprocessing result, so as to replace the traditional word bag strategy with semantic features and spatial topology information, which has higher adaptability to illumination and perspective changes than the traditional word bag, and can significantly improve the robustness of closed-loop detection. The first-level closed-loop candidate key frame obtained by screening can effectively filter the historical key frames with large differences in instance types and numbers, and further judge the continuity of the first-level closed-loop candidate key frame by using the semantic feature vector, and obtain the second-level closed-loop candidate key frame on the basis of the first-level matching, so as to ensure the efficiency of closed-loop detection. The semantic space topology information is fully utilized to ensure the accuracy of closed-loop detection. Therefore, the closed-loop detection performed by using the first-level matching and the second-level matching can take into account robustness, efficiency and accuracy. The BFS tree of the undirected complete graph of the current key frame is used to associate the spatial topology information of the current key frame with the second-level closed-loop candidate key frame and calculate the similarity, which has a faster processing speed and good accuracy compared with the other existing topology matching methods. In addition, after completing the first-level matching and the second-level matching, the present application only retains the BFS tree and semantic feature vector of the undirected complete graph of the current key frame, and its RGB image and depth image are discarded, effectively preventing memory waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is an application environment diagram of a semantic space topology driven multi-level SLAM closed-loop detection method in one embodiment of the present application;

[0041] Figure 2 A flowchart of a multi-level SLAM closed-loop detection method driven by semantic space topology provided in one embodiment of the present application;

[0042] Figure 3 A schematic diagram of a result obtained by using a region growing algorithm provided in another embodiment of the present application;

[0043] Figure 4 A schematic diagram of constructing a key frame semantic feature vector provided by another embodiment of the present application;

[0044] Figure 5A schematic diagram of a continuity determination process provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the spatial coordinate transformation of the pixel centroid of each key frame instance provided in another embodiment of the present application;

[0046] Figure 7 An undirected complete graph of the current key frame spatial information in a reference coordinate system provided by another embodiment of the present application;

[0047] Figure 8 A schematic diagram of a BFS tree of an undirected complete graph in a current key frame reference coordinate system and a node matching node of a certain node provided by another embodiment of the present application;

[0048] Fig. 9 A schematic diagram of an implementation flow of a multi-level SLAM closed-loop detection method driven by semantic space topology provided in one embodiment of the present application;

[0049] Fig.10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0052] In order to improve the robustness of the closed-loop detection algorithm, combining semantic information for location recognition becomes an effective strategy. Based on this, the embodiment of the present application provides a multi-level SLAM closed-loop detection method driven by semantic space topology. This method can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the image data of the scene to be detected to the server 104. After the server 104 receives the image data of the scene to be detected, for the image data of the scene to be detected, the server 104 uses the ORB-SLAM2 system to create and eliminate conditions based on the image data of the scene to be detected and the key frame and the key frame The corresponding image data is obtained. The key frame corresponding image data is preprocessed using the regional growing algorithm and the Laplace operator to obtain a preprocessing result. A semantic feature vector is constructed based on the preprocessing result. Based on the semantic feature vector, the similarity between the current key frame and the non-connected key frame is determined. The similarity threshold is determined based on the maximum similarity between the current key frame and the non-connected key frame, and the first-level closed-loop candidate key frame is screened in the non-connected key frame based on the similarity threshold. The first-level closed-loop candidate key frames that meet the continuity condition are retained, and the retained first-level closed-loop candidate key frames are used as the second-level closed-loop candidate key frames. Based on the preprocessing results, the spatial coordinates of the centroids of each instance pixel in the image data corresponding to the current key frame in the current key frame coordinate system are determined. The spatial coordinates of the centroids of each instance pixel of the current key frame are transformed to the reference coordinate system to obtain the transformed coordinate information. The BFS tree of the undirected complete graph of the current key frame is constructed using the transformed instance pixel centroid space unified coordinate information and category information. Determine the similarity between the undirected complete graph BFS tree of the current key frame and the undirected complete graph BFS tree of the second-level closed-loop candidate key frame, and use the second-level closed-loop candidate key frame with the highest similarity as the closed-loop key frame. Based on the relationship between the similarity between the closed-loop key frame and the current key frame and the set threshold, closed-loop detection is achieved. The server 104 can feedback the obtained closed-loop detection result to the terminal 102. In addition, in some embodiments, the semantic space topology driven multi-level SLAM closed-loop detection method provided in the present application can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform closed-loop detection on the image data of the scene to be detected, or the server 104 can obtain the image data of the scene to be detected from the data storage system and perform closed-loop detection on the image data of the scene to be detected.

[0053] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0054] In an exemplary embodiment, Figure 2As shown, a semantic space topology driven multi-level SLAM closed-loop detection method is provided, which is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, including the following steps 200 to 212. Among them:

[0055] Step 200: Obtain image data of the scene to be detected. The image data includes RGB images and depth maps of each frame.

[0056] Step 201: Using the ORB-SLAM2 system to obtain key frames based on the image data of the scene to be detected and the key frame creation and elimination conditions, and retaining the image data corresponding to the key frames.

[0057] Step 202: Preprocess the image data corresponding to the key frame using a region growing algorithm and a Laplace operator to obtain a preprocessing result. The preprocessing result includes an instance segmentation result and category information. An instance is a physical object represented by a connected region obtained based on the preprocessing (e.g., segmentation) of the image data corresponding to the key frame.

[0058] Step 203: constructing a semantic feature vector based on the preprocessing result.

[0059] Step 204: Based on the semantic feature vector, determine the similarity between the current key frame and the non-connected key frame. The non-connected key frame is a historical key frame that is not connected to the current key frame.

[0060] Step 205: determining a similarity threshold based on the maximum similarity between the current key frame and the non-connected key frames, and screening the non-connected key frames to obtain first-level closed-loop candidate key frames based on the similarity threshold.

[0061] Step 206: retaining the first-level closed-loop candidate key frames that meet the continuity condition, and using the retained first-level closed-loop candidate key frames as the second-level closed-loop candidate key frames.

[0062] Step 207: Determine the spatial coordinates of the centroid of each instance pixel in the image data corresponding to the current key frame in the coordinate system of the current key frame based on the preprocessing result and the depth map.

[0063] Step 208: transform the spatial coordinates of the centroid of each instance pixel of the current key frame into the reference coordinate system to obtain the transformed unified spatial coordinate information of the centroid of the instance pixel.

[0064] Step 209: construct a BFS tree of the undirected complete graph of the current key frame using the transformed spatial unified coordinate information and the category information.

[0065] Step 210: Determine the similarity between the BFS tree of the undirected complete graph of the current key frame and the BFS tree of the undirected complete graph of the secondary closed-loop candidate key frame, and use the secondary closed-loop candidate key frame with the highest similarity as the closed-loop key frame.

[0066] Step 211: When the similarity between the closed-loop key frame and the current key frame exceeds a set threshold, a closed loop is detected and closed-loop correction is performed.

[0067] Step 212: When the similarity between the closed-loop key frame and the current key frame does not exceed the set threshold, the closed loop is not detected and no closed-loop correction is performed.

[0068] By implementing the above steps 200 and 212, the present application can significantly improve the robustness of closed-loop detection, improve detection accuracy while ensuring closed-loop detection efficiency, and has a faster processing speed than using other spatial topologies while also having excellent accuracy and robustness.

[0069] In another exemplary embodiment of the present application, in step 200 and step 201, an RGB-D camera can be used to collect RGB images and depth maps of each frame of the environment or scene to be detected, and the collected image data can be input into the ORB-SLAM2 system. The inherent function of the ORB-SLAM2 system tracks each frame to create a key frame and estimate the pose. The RGB image and depth image of each frame are not discarded before the key frame is created. When the ORB-SLAM2 system runs to the creation of the key frame, the RGB image and depth map corresponding to the ordinary frame created as the key frame are retained, and the non-key frame image data is discarded. After the key frame passes through the local mapping thread, the redundant key frames are removed, and their image data are also removed, and the remaining key frames and their image data are used as inputs to the closed-loop detection thread. Among them, the local mapping thread proposes the standard for redundant key frames: if more than 90% of the map points in the created key frame can be observed by more than 3 other key frames, they will be removed. Map points are three-dimensional points in space corresponding to two-dimensional feature points.

[0070] In another exemplary embodiment of the present application, the implementation process of the above step 202 may be:

[0071] Step 221: Generate a key frame queue based on the key frame creation and elimination conditions in the ORB-SLAM2 system to enter the closed-loop detection thread, and take out the head frame in the key frame queue as the current key frame.

[0072] Step 2022: Using a semantic segmentation network to extract semantic features from the RGB image corresponding to the current key frame to obtain a mask image.

[0073] Step 2023: For the depth map corresponding to the current key frame, considering that the depth information of different instances in the depth map is significantly different, the Laplacian operator is used to identify the edge of the depth mutation in the depth map corresponding to the current key frame to obtain the processed depth map. For example, the form of the Laplacian operator on the image is:

[0074]

[0075] In the formula, and Represents the second-order partial derivative of the image at the point (x,y). represents the Laplace operator, f(*,*) represents the image, and (*,*) represents the pixel points of the image.

[0076] The Laplacian operator is expressed in the form of a convolution kernel on the image. Considering all the pixels around each pixel in the depth map, the original Laplacian convolution kernel is expanded as follows:

[0077]

[0078] Using the extended Laplacian convolution kernel K to convolve the depth map can find areas with large depth changes. In order to distinguish instances, set an appropriate threshold Th border The results are binarized to detect the edges of instances, which is based on the reasonable assumption that the distance from the camera to the same instance remains relatively stable.

[0079] Step 2024: Align the mask image with the processed depth map, perform special assignment on the edges, and obtain the edges of each instance.

[0080] Step 2025: Process the aligned images using a region growing algorithm to obtain a preprocessing result of the image data corresponding to the current key frame. Figure 3 As shown, the connected regions with the same characteristics are segmented out, and the number of connected regions, that is, the number of instances, can be obtained, and a threshold Th is determined. pixel_num , when the number of pixels in the connected area is less than the threshold Th pixel_num The number of corresponding categories is not increased, and special cases such as the ground and the wall are also excluded, ultimately achieving the purpose of instance segmentation and counting the number of categories.

[0081] In another exemplary embodiment of the present application, in order to obtain the connected regions of each instance using a region growing algorithm, the implementation process of step 2025 may be:

[0082] (1) Randomly select one or more unlabeled pixels in the aligned image as seed pixels, assign a label value to the seed pixels, and add them to the region set as a region.

[0083] (2) In each iteration, a region is selected from the region set, the 8-domain pixels in the region are traversed, and the difference between the pixel point and the seed pixel is determined to be within the specified threshold Th. growth If it is, then add the pixel to the region and mark it as visited. If a new pixel is added to the region, update the region set.

[0084] (3) Repeat step (2) until no new pixels can be added to the region. Return to step (1) until every pixel of the aligned image has been visited or no new pixels can be added to the region, and obtain the processing result of the current key frame.

[0085] The region growing algorithm can segment connected regions with the same characteristics. From this, the number of connected regions, that is, the number of instances, can be obtained, and a threshold Th can be determined. pixel_num , when the number of pixels in the connected area is less than the threshold Th pixel_num The number of corresponding categories does not increase, and the purpose of instance segmentation and statistics of the number of categories is finally achieved.

[0086] In another exemplary embodiment of the present application, in order to significantly improve the robustness of closed-loop detection and solve the problem that the existing technology using bag-of-words features for closed-loop detection is greatly affected by changes in illumination and perspective, the present application uses semantic features and spatial topology information to replace the traditional bag-of-words strategy. Among them, semantic information has higher adaptability to changes in illumination and perspective than traditional bag-of-words. Based on this, in this embodiment, the implementation process of the above step 203 can be to use the preprocessing results to construct a semantic feature vector of the current key frame according to semantic categories and number statistics. Among them:

[0087] Step 2031: Construct a semantic feature dictionary based on the preprocessing results. For example, the semantic feature dictionary of the current key frame is constructed based on the categories and numbers of connected areas of each category instance in the current key frame corresponding image obtained by the above segmentation. The semantic feature dictionary is represented by D c :

[0088] D c ={A:x cA ,B:x cB ,C:x cC ,…,α:x cα}.

[0089] In the formula, A~α represents the semantic category of the current key frame, x cA ~x cα Indicates the total number of instances of each category in the current keyframe.

[0090] Step 2032: assign different weights to each semantic category in the semantic feature dictionary to obtain a semantic feature vector. For example, assign different weights ω to each semantic category according to its particularity in the scene to improve the accuracy of closed loop detection, and multiply them by the total number of instances corresponding to each category to obtain Figure 4 The semantic feature vector of the current key frame is shown as V c :

[0091] V c =[x cA ω A ,x cB ω B ,x cC ω C ,…,x cα ω α ].

[0092] Among them, ω A ~ω α Represents the weight of each semantic category.

[0093] In another exemplary embodiment of the present application, the similarity with the non-connected key frames is calculated based on the semantic feature vector, and the first-level closed-loop candidate key frames are screened out to perform continuity judgment, such as Figure 5 As shown, the secondary closed-loop candidate key frames are screened out to complete the primary matching. Based on this, the implementation process of the above steps 205 and 206 can be:

[0094] Check whether a closed loop has just been detected. If it exceeds a certain number of frames Th of the previous closed loop loop , then the semantic feature vector of the current key frame is matched with the semantic feature vector of the non-connected key frame (also referred to as non-connected historical key frame in this application) for similarity matching calculation. In actual application, the Euclidean distance can be used to calculate the similarity between the current key frame and the non-connected key frame. For example, this similarity is recorded as S V (KF c ,KF h ), and its calculation formula is:

[0095]

[0096] Where KF c Indicates the current key frame, KF h represents non-connected historical keyframes, x ci Indicates the number of instances of each category in the current key frame, x hi Indicates the number of instances of each category of non-connected historical key frames.

[0097] Since the historical key frames connected to the current key frame have more common views with the current key frame, they are usually more similar to the current key frame. In order to exclude the influence of the connected historical key frames, this application sets the threshold Th connected , take out the Th before the current key frame connected The connected historical key frames within a frame are removed during screening.

[0098] Further, similarity calculation is performed with all non-connected key frames that meet the conditions to obtain the maximum similarity S max , and use the maximum similarity S max Determine the relative threshold. The relative threshold is determined as the maximum similarity S max σ times, σ is between 0 and 1. The non-connected historical key frames corresponding to the similarity greater than the relative threshold are the first-level closed-loop candidate key frames, so all qualified first-level closed-loop candidate key frames can be found in the historical key frames through the relative threshold.

[0099] Based on the above description, a semantic feature vector of the current key frame is constructed according to the preprocessing result, and the semantic feature vector is used as a basis to judge the similarity between the current key frame and the non-connected historical key frames to obtain a first-level closed-loop candidate key frame.

[0100] Based on the above description, according to the observation of the SLAM system keyframe on the map point and the observation of the map point on the keyframe, the common view keyframe of the first-level closed-loop candidate keyframe of the current keyframe is found. The first-level closed-loop candidate keyframe and its common view keyframe are combined into a closed-loop candidate keyframe group. For example, if there are n closed-loop candidate keyframes, there are n closed-loop candidate keyframe groups. In actual application, the closed-loop candidate keyframe group of the current keyframe is retained until the closed-loop detection thread processes two frames after the current frame to avoid memory waste.

[0101] like Figure 5 As shown, the closed-loop candidate key frame group of the current key frame is compared with the closed-loop candidate key frame group of the previous two frames. If there is a key frame in the closed-loop candidate key frame group, and its semantic feature vector is the same as a key frame in the closed-loop candidate key frame group of the previous two connected frames, then it is determined that the closed-loop candidate key frame meets the continuity, and the closed-loop candidate key frame is retained in the candidate frame queue, otherwise it is removed. Through this operation, the secondary closed-loop candidate key frame can be obtained, and then the primary matching is completed.

[0102] In another exemplary embodiment of the present application, based on the connected areas of each instance in the image data corresponding to the current key frame, the spatial coordinates of the pixel centroid of each instance in the current key frame coordinate system can be calculated and transformed into a unified coordinate in the reference coordinate system, such as Figure 6Based on this, the edge set and vertex set are constructed with the unified coordinate information and category information of the instance pixel centroid space to further construct the BFS tree of the undirected complete graph of the current key frame, as shown in Figure 7 and Figure 8 As shown. By calculating the similarity between the BFS tree of the current key frame undirected complete graph and the BFS tree of the secondary closed-loop candidate key frame undirected complete graph, a closed-loop key frame is obtained to complete the secondary matching. Based on this, the implementation process of the above steps 207 to 210 of the present application can be:

[0103] (1) After the connected regions of each instance are obtained by the region growing algorithm, the pixel centroid is used to represent the spatial structure information of the connected region. The value of each pixel in the image is taken as the quality of the pixel. The formula for determining the two-dimensional coordinates of the centroid in the two-dimensional image is expressed as:

[0104]

[0105] Among them, n is the number of all pixels in the connected area, p i represents the pixel value, u i 、v i Represents pixel coordinates, u c 、v c represents the coordinates of the center of mass. Further, we can get:

[0106]

[0107] From this, we can get the two-dimensional coordinates of the pixel centroid of each instance (u c ,v c ).

[0108] Next, the two-dimensional coordinates of the centroid are mapped to the depth map of the current key frame to obtain the depth Z of the centroid of each instance. c , the three-dimensional spatial coordinates of the centroid of each instance under the observation of the current key frame are obtained according to the camera imaging principle, which is expressed as:

[0109]

[0110] (2) In order to make the spatial structure information of the current key frame and the spatial structure information of the closed-loop candidate frame participate in unified matching, it is necessary to use the SLAM system front-end to optimize the pose and transform the three-dimensional spatial coordinates of the pixel centroid in the current key frame coordinate system to the spatial coordinates in the reference coordinate system. In this way, the spatial coordinates of the pixel centroid of the current key frame and the candidate frame are unified in the same coordinate system and can be matched.

[0111] Through the SLAM system tracking and local mapping module, the estimated rotation matrix R and translation vector t of each key frame relative to the previous key frame can be obtained, which can describe the coordinate transformation relationship in a Euclidean space as follows:

[0112] P w =R w1 P1+t w1 .

[0113] Among them, P w Represents the coordinates of the pixel centroid of the first frame in the world coordinate system. w1 Represents the rotation matrix that rotates the first frame coordinates to the world coordinate system. P1 represents the coordinates of the centroid of each instance pixel in the first frame in the first frame coordinate system. w1 Represents the translation vector that translates the first frame coordinates to the world coordinate system.

[0114] Since the subsequent key frame pixel centroids need to undergo multiple transformations, the transformation matrix T is introduced for the convenience of transformation, which is expressed as:

[0115]

[0116] Where, T w1 Represents the transformation matrix that transforms the homogeneous coordinates of the first frame into homogeneous coordinates in the world coordinate system.

[0117] remember P * The coordinate transformation of the homogeneous coordinates of the pixel centroid space in the second frame coordinate system in the world coordinate system can be expressed as:

[0118]

[0119] Where, T 12 represents the transformation matrix that transforms the coordinates of the second frame to the coordinates of the first frame, Represents the homogeneous coordinates of the pixel centroid of each instance in the second frame in the coordinate system of the second frame, Represents the homogeneous coordinates of the pixel centroid of each instance in the second frame in the world coordinate system.

[0120] Continuing the recursion, the coordinate transformation from the pixel centroid space homogeneous coordinates in the nth frame coordinate system to the world coordinate system is expressed as:

[0121]

[0122] This allows the centroid space coordinates of the instance pixels of the current keyframe to be unified to the reference coordinate system, such as Figure 6 As shown, so as to facilitate the subsequent secondary matching.

[0123] (3) The origin of the reference coordinate system is taken as the starting node, marked as (none, 0, 0, 0), and used as node information. The class of each instance of the current key frame and its spatial unified coordinate information in the world coordinate system (class, x, y, z) are also recorded as node information. By connecting the origin with each instance centroid coordinate node, the undirected complete graph G = 〈V, E〉 of the centroid coordinates of the current key frame can be obtained in theory, as follows: Figure 7 As shown. Among them, V is a finite non-empty set of node information called the vertex set, and E is a set of two-tuples of nodes called the edge set. The weight of the edge is the Euclidean distance between the coordinates of the two nodes, denoted by E(a,b):

[0124]

[0125] In the formula, (x a ,y a ,z a ) represents the coordinates of node a, (x b ,y b ,z b ) represents the coordinates of node b.

[0126] Initialize the BFS tree of the graph with the origin as the starting node, and then you can get the BFS tree BFST of the undirected complete graph of the center of mass space coordinates of the current key frame.<V,E> ,like Figure 8 As shown. In actual operation, the construction process of the undirected complete graph can be omitted, and the BFS tree of the current key frame undirected complete graph can be directly constructed through the vertex set information V and the edge set E between the starting node and the pixel centroid node. Among them, the implementation steps of initializing the BFS tree of the current key frame undirected complete graph with the origin as the starting node are as follows:

[0127] (a) Initialize the BFS tree and add the start node to the tree.

[0128] (b) Add the starting node to the queue to be visited.

[0129] (c) For each node in the queue to be visited, traverse all its neighboring nodes in turn.

[0130] (d) For each neighbor node, if the neighbor node has not yet been added to the BFS tree, add it and set the parent node of the neighbor node as the current node.

[0131] (e) Remove the visited nodes from the queue and add each neighbor node to the queue to be visited.

[0132] (f) Repeat steps (c) to (e) until the queue is empty.

[0133] Through the above steps, it can be understood that the BFS tree of the current key frame takes the origin of the world coordinate system as the parent node and the remaining pixel centroid nodes as its child nodes.

[0134] (4) Match the BFS tree of the current keyframe undirected complete graph with the BFS tree of the secondary closed-loop candidate keyframe undirected complete graph. can Take the current key frame KF as an example. cur Undirected complete graph BFS tree vertex set V cur Find the centroid node information of each instance in KF can Undirected complete graph BFS tree vertex set V can The same centroid nodes are selected and the closest V is selected by using the Euclidean distance. cur The node of the instance mass center node space coordinates. Let V cur The spatial information of the centroid node of an instance is A cur , V can The spatial information of the matching centroid node is A can At the same time, different weights ω can be assigned according to the particularity of the semantic category in the scene, as follows:

[0135]

[0136] Among them, S(A cur ,A can ) represents the similarity of matching centroid nodes, ||A cur || and |A can || is the modulus length from the origin to the centroid vector. Using the index of the centroid node in the vertex set, the modulus length of each centroid vector can be found in the edge set E of the undirected complete graph BFS tree of each frame. and is the coordinate value of each dimension in the spatial information of the instance pixel centroid node. Euc and ω cos is the similarity weight, ω Euc +ω cos =1, generally take ω Euc >ω cos If a centroid node of the current keyframe does not find a corresponding centroid node in the candidate closed-loop keyframe, the similarity is 0.

[0137] Furthermore, the similarity between the BFS tree of the undirected complete graph of the current key frame and the BFS tree of the undirected complete graph of the secondary closed-loop candidate key frame is calculated using the similarity of the pixel centroid nodes of each instance of the current key frame, which is expressed as:

[0138]

[0139] Where N is the number of pixel centroid nodes of the current keyframe instance.BFS Represents the similarity between the current keyframe and the undirected complete graph BFS tree of the secondary closed-loop candidate keyframes.

[0140] Finally, the secondary closed-loop candidate keyframe with the highest similarity is selected as the closed-loop keyframe.

[0141] Based on the above description, in this embodiment, the process of determining the similarity between the BFS tree of the current key frame undirected complete graph and the BFS tree of the secondary closed-loop candidate key frame undirected complete graph can be described as:

[0142] The similarity between the BFS tree of the current key frame undirected complete graph and the BFS tree of the secondary closed-loop candidate key frame undirected complete graph is determined by using the vertex set and edge set of the key frame undirected complete graph BFS tree; the vertex set of the current key frame undirected complete graph BFS tree is determined based on the semantic category of the pixel centroid of each instance in the current key frame and the unified spatial coordinates of the pixel centroid of each instance; the edge set of the current key frame undirected complete graph BFS tree is determined based on the modulus length of the vector from the origin to the pixel centroid.

[0143] In another exemplary embodiment of the present application, if the similarity between the closed-loop key frame and the current key frame does not exceed the threshold value Th sim , no closed loop is detected, no closed loop correction is performed, and the current key frame is added to the historical key frame. sim , a closed loop is detected, closed loop correction is performed, and the current key frame is added to the historical key frames.

[0144] In another exemplary embodiment of the present application, in order to effectively prevent memory waste, after completing the first-level and second-level matching, only the spatial topology information and semantic feature vector of the current key frame are retained, and its RGB image and depth image are discarded. That is, the BFS tree and semantic feature vector of the undirected complete graph of the current key frame are retained, and the RGB image and depth image are discarded.

[0145] In summary, in the specific implementation process, the implementation process of the semantic space topology driven multi-level SLAM closed-loop detection method provided in the present application can be as follows: Fig. 9 Moreover, based on the above description, the present application has the following advantages compared with the prior art:

[0146] 1. Compared with the original technology that uses bag-of-words features for closed-loop detection, which is greatly affected by changes in lighting and perspective, this application uses semantic features and spatial topology information to replace the traditional bag-of-words strategy. Compared with the traditional bag-of-words, it has higher adaptability to changes in lighting and perspective. The use of semantic features can significantly improve the robustness of closed-loop detection.

[0147] 2. Compared with the original technology, the use of primary matching and secondary matching can effectively and accurately screen out closed-loop key frames. Among them, the primary matching can effectively filter out historical key frames with large differences in instance types and numbers, and at the same time use the semantic feature vector to further judge the continuity of the primary closed-loop candidate key frames, thereby ensuring the efficiency of closed-loop detection. The secondary matching makes full use of the semantic space topology information on the basis of the primary matching to ensure the accuracy of closed-loop detection. It can be seen from this that the closed-loop detection performed by multi-level matching in this application can achieve an effective balance between efficiency and accuracy.

[0148] 3. Compared with the existing technology, the semantic space topology information constructed by this application adds one dimension of information compared to the bag-of-words feature of the traditional two-dimensional plane. At the same time, the BFS tree of the keyframe undirected complete graph is used to calculate the similarity between the current keyframe and the secondary closed-loop candidate keyframe, which has a faster processing speed than using other spatial topologies, and also has good accuracy and robustness.

[0149] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Wherein, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store closed-loop detection data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-level SLAM closed-loop detection method driven by a semantic space topology is implemented.

[0150] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0151] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0152] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0154] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0155] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0156] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-level SLAM closed-loop detection method driven by semantic space topology, characterized in that: The multi-level SLAM closed-loop detection method driven by semantic space topology includes: Acquire image data of the scene to be detected; the image data includes RGB images and depth maps of each frame; The ORB-SLAM2 system is used to obtain key frames based on the image data of the scene to be detected and the key frame creation and elimination conditions, and the corresponding image data of the key frames are retained; Preprocessing the image data corresponding to the key frame using a region growing algorithm and a Laplace operator to obtain a preprocessing result; the preprocessing result includes an instance segmentation result and category information; the instance is an entity object represented by a connected region obtained based on the preprocessing of the image data corresponding to the key frame; Constructing a semantic feature vector based on the preprocessing result; Based on the semantic feature vector, determining the similarity between the current key frame and the non-connected key frame; the non-connected key frame is a historical key frame that is not connected to the current key frame; Determine a similarity threshold based on the maximum similarity between the current key frame and the non-connected key frames, and select the first-level closed-loop candidate key frames from the non-connected key frames based on the similarity threshold; Retaining the first-level closed-loop candidate key frames that meet the continuity condition, and using the retained first-level closed-loop candidate key frames as second-level closed-loop candidate key frames; Determine the spatial coordinates of the centroid of each instance pixel in the image data corresponding to the current key frame in the coordinate system of the current key frame based on the preprocessing result and the depth map; The spatial coordinates of the centroid of each instance pixel of the current key frame are transformed into the reference coordinate system to obtain the uniform spatial coordinate information of the centroid of each instance pixel after the transformation; Constructing a BFS tree of the undirected complete graph of the current key frame using the transformed instance pixel centroid space unified coordinate information and the category information; Determine the similarity between the BFS tree of the undirected complete graph of the current key frame and the BFS tree of the undirected complete graph of the secondary closed-loop candidate key frame, and use the secondary closed-loop candidate key frame with the highest similarity as the closed-loop key frame; When the similarity between the closed-loop key frame and the current key frame exceeds a set threshold, a closed loop is detected and closed-loop correction is performed; When the similarity between the closed-loop key frame and the current key frame does not exceed the set threshold, the closed loop is not detected and no closed-loop correction is performed.

2. The multi-level SLAM closed-loop detection method driven by semantic space topology according to claim 1, characterized in that, The region growing algorithm and Laplace operator are used to preprocess the image data corresponding to the key frame to obtain the preprocessing results, including: Based on the key frame creation and elimination conditions in the ORB-SLAM2 system, a key frame queue is generated to enter the closed-loop detection thread, and the head frame in the key frame queue is taken out as the current key frame; The RGB image corresponding to the current key frame is used to extract semantic features using a semantic segmentation network to obtain a mask image; The Laplacian operator is used to identify the edge of the depth mutation in the depth map corresponding to the current key frame, and the processed depth map is obtained; Aligning the mask image with the processed depth map, performing special assignment on the edge, and obtaining the edge of each instance; The aligned images are processed using a region growing algorithm to obtain the preprocessing result of the image data corresponding to the current key frame.

3. The multi-level SLAM closed-loop detection method driven by semantic space topology according to claim 1, characterized in that, Constructing a semantic feature vector based on the preprocessing result includes: Building a semantic feature dictionary based on the preprocessing results; Different weights are assigned to each semantic category in the semantic feature dictionary to obtain the semantic feature vector.

4. The multi-level SLAM closed-loop detection method driven by semantic space topology according to claim 1, characterized in that, Based on the semantic feature vector, determining the similarity between the current key frame and the non-connected key frame includes: Based on the semantic feature vector, the similarity between the current key frame and the non-connected key frames is determined using the Euclidean distance.

5. The multi-level SLAM closed-loop detection method driven by semantic space topology according to claim 1, characterized in that, The similarity threshold is determined based on the maximum similarity between the current key frame and the non-connected key frames, including: The similarity threshold is taken as σ times the maximum similarity between the current key frame and the non-connected key frames.

6. The multi-level SLAM closed-loop detection method driven by semantic space topology according to claim 1, characterized in that: Determine the similarity between the BFS tree of the current keyframe undirected complete graph and the BFS tree of the secondary closed-loop candidate keyframe undirected complete graph, including: The similarity between the BFS tree of the current key frame undirected complete graph and the BFS tree of the secondary closed-loop candidate key frame undirected complete graph is determined by using the vertex set and edge set of the BFS tree of the key frame undirected complete graph; the vertex set of the BFS tree of the current key frame undirected complete graph is determined based on the semantic category of the pixel centroid of the current key frame and the unified coordinates of the pixel centroid space and the origin information; the edge set of the BFS tree of the current key frame undirected complete graph is determined based on the modulus length from the origin to the pixel centroid vector.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-level SLAM closed-loop detection method driven by the semantic space topology according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-level SLAM closed-loop detection method driven by the semantic space topology described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-level SLAM closed-loop detection method driven by the semantic space topology described in any one of claims 1 to 6 is implemented.

Images