A loop detection method based on the geometric structure between planes

Abstract

The present invention relates to a loop detection method based on the geometric structure between planes. An RGB-D camera is used to obtain the corresponding RGB image and depth map. The preprocessed RGB image is subjected to superpixel segmentation to obtain a number of pixel regions. The point cloud of the depth map is divided corresponding to the pixel regions, and the normal vector of the point cloud block corresponding to each pixel region after division is calculated. The point cloud blocks are combined based on the normal vector relationship to obtain the corresponding planes and edges, and an undirected graph is constructed to search for loops. A common subgraph of the undirected graphs of the current frame and the frames in the list is searched. If there is an effective common subgraph, the frame is listed as a candidate frame and geometric verification is performed according to the overlap degree of the point cloud blocks, and the loop frame is determined based on the verification result. The present invention uses an RGB-D camera as a sensor and combines the point cloud geometric structure information for loop detection, which can perform well in complex environments and enhance the performance of RGB-D SLAM in complex actual scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image or video recognition or understanding, and particularly to a loop detection method based on the geometric structure between planes. Background Art

[0002] In recent years, with the rapid development of technology, mobile robot technology has been widely applied in various industries, such as palletizing robots, logistics robots, and sweeping robots. For these intelligent robot systems, the Simultaneous Localization and Mapping (SLAM) technology is crucial because it enables them to achieve autonomous navigation in unknown environments, provide accurate environmental perception capabilities, perform effective path planning, and avoid obstacles. Therefore, the SLAM technology has always been a hot research field. According to the different types of sensors used, the current mainstream SLAM systems are mainly divided into visual SLAM and lidar SLAM. Visual SLAM mainly uses monocular cameras, binocular cameras, and RGB-D cameras. Since RGB-D cameras can not only provide color images but also directly measure the depth information of pixel points, compared with monocular cameras and binocular cameras, they have higher accuracy, smaller computational complexity, and lower costs than lidar, so RGB-D cameras are widely used.

[0003] Since errors are likely to occur during the SLAM mapping process, and the gradually accumulated errors will cause drift phenomena, to solve this problem, loop detection is added to the SLAM system as an important module.

[0004] Currently, in RGB-D SLAM, most loop detections are image-based methods. By extracting point features or line features, and making descriptors manually according to the pixels around the features for feature matching. However, in the actual work of the robot, the environment it faces is complex and changeable. For example, scenes that are not friendly to image features, such as dark, weak texture, and photometric changes, widely exist in the actual environment, resulting in the insufficient performance of the loop detection algorithm in the actual environment, and the existing technology has not made good use of the relatively accurate point cloud geometric structure information of RGB-D cameras. Summary of the Invention

[0005] The present invention solves the problems existing in the prior art and provides a loop detection method based on the geometric structure between planes.

[0006] The technical solution adopted by the present invention is a loop detection method based on the geometric structure between planes, and the method includes the following steps:

[0007] S1 Obtain the corresponding RGB image and depth image with an RGB-D camera and perform preprocessing;

[0008] S2 performs superpixel segmentation on the preprocessed RGB image to obtain a number of pixel regions, divides the point cloud of the depth map corresponding to the pixel regions, calculates the normal vectors of the point cloud blocks corresponding to each pixel region after division, combines the point cloud blocks based on the normal vector relationship to obtain the corresponding planes and edges, and constructs an undirected graph;

[0009] S3 Based on the undirected graph, find the basic loop;

[0010] S4 Find the optimal common subgraph of the undirected graphs of the current frame and the query frame in the key frame list. If there is an effective optimal common subgraph, list this frame as a candidate frame and perform geometric verification according to the overlap degree of the point cloud blocks, and determine the loop closure frame based on the verification result.

[0011] Preferably, in S1, the preprocessing includes performing bilateral Gaussian filtering on the RGB image and the depth map.

[0012] Preferably, in S2, a set of points is obtained by dividing the point cloud of the depth map corresponding to each pixel region except the boundary , and , calculate the covariance matrix of this set of points and perform eigenvalue decomposition, using the eigenvector corresponding to the largest eigenvalue as the normal vector of the current point cloud block , and the fitting plane of the current point cloud block is denoted as .

[0013] Preferably, verifying the effectiveness of the fitting plane includes the following steps:

[0014] S2.1 Define a dynamic threshold , if the mean square error of the fitting plane , then the fitting plane is invalid, otherwise proceed to the next step;

[0015] S2.2 Check the pixel points corresponding to the 3D points , if it satisfies max(|depth(p m,n ) - depth(p m+1,n )|, |depth(p m,n ) - depth(p m,n+1 )|) < T ds , then the fitting plane is effective, where calculate the depth value of the pixel point , and are respectively the adjacent points in the horizontal and vertical directions of , is the depth value, is the user-defined ratio, for tolerance;

[0016] S2.3 Store the valid plane into the plane list ; Get the adjacent planes of the planes in the list.

[0017] Preferably, based on the similarity of normal vectors between adjacent point cloud blocks, a depth-first search algorithm is used to merge the valid planes fitted by all searched point cloud blocks that meet the conditions into one plane, wherein the conditions are evaluated based on the normal vectors and distances between planes; the plane list is updated, and the center point and normal vector of the new plane are calculated.

[0018] Preferably, a comparison matrix is ​​constructed ,

[0019]

[0020] in, Is a plane The mean square error of the merged plane is If it is unchanged, otherwise it is zero, and the planes corresponding to the non-zero minimum terms are merged and Remove the corresponding rows and columns, add the new plane to the matrix until the matrix is ​​a zero matrix; update the plane list and add the planes with an area less than the threshold Plane culling.

[0021] Preferably, the undirected graph constructed in S2 In the vertex set , edge set ;

[0022] In S3, each vertex is used as a seed, the search depth is four, and rings with three or four vertices are found. These two types of rings are uniformly defined as basic rings, and a basic ring list is established. and a hash table with vertices as hash values .

[0023] Preferably, in S4, according to the established undirected graph and the found basic rings, the basic rings in the current frame and the query frame are compared, and the front one with the highest similarity and satisfying the same number of ring vertices, the same adjacency matrix of the rings and the area difference of the plane represented by the vertex is less than the threshold is selected. The rings are taken as candidate ring pairs, added to the candidate ring pair list, and the common subgraph is expanded. Based on the common subgraph, the plane matching between the two frames is obtained, and the difference value is used. Verify the validity of the common sub-image; take the common sub-image with the smallest difference value as the optimal common sub-image, and take the difference value as the only difference value between the two frames;

[0024] Finally, among all the query frames with the existing optimal common subgraphs, the top K query frames with the smallest difference values are used as candidate frames.

[0025] Preferably, according to the candidate loop pair list, for each pair of candidate loop pairs, the base loop is used to construct a subgraph according to their respective graphs. Find the loops adjacent to the base loop in each graph. The definition of adjacent is that two loops have the same vertices, and then make a comparison. Add the pair of loops with the largest similarity to the subgraph. If the adjacency matrices of the two new subgraphs are equal and this pair of new loops meet the valid conditions in the process of finding the base loop, then this pair of loops is considered valid. Then, find adjacent loops on the basis of the new subgraph and repeat the process to expand the subgraph until there is nothing to add, so as to obtain the common subgraph of the two graphs.

[0026] Preferably, the maximum overlap degree between the current frame and the candidate frames The obtained transformation matrix is used as the rough registration between the current frame and the candidate frames. If it exceeds the preset threshold, the loop is valid. Finally, the candidate frame with the largest overlap degree is used as the loop closure frame.

[0027] The present invention relates to a loop closure detection method based on the geometric structure between planes. An RGB-D camera is used to obtain the corresponding RGB image and depth image. The preprocessed RGB image is subjected to superpixel segmentation to obtain a number of pixel regions. The point cloud of the depth image is divided corresponding to the pixel regions, and the normal vectors of the point cloud blocks corresponding to each pixel region after division are calculated. The point cloud blocks are combined based on the normal vector relationship to obtain the corresponding planes and edges, and an undirected graph is constructed to find loops; find the common subgraph of the undirected graphs of the current frame and the frames in the list. If there is an effective common subgraph, then this frame is listed as a candidate frame and geometric verification is performed according to the overlap degree of the point cloud blocks, and the loop closure frame is determined based on the verification result.

[0028] The beneficial effect of the present invention is that by using an RGB-D camera as a sensor and combining the point cloud geometric structure information for loop closure detection, it can perform well in complex environments and enhance the performance of RGB-D SLAM in complex actual scenarios. Description of the Drawings

[0029] Figure 1 is the flowchart of the method of the present invention;

[0030] Figure 2 is the flowchart of the loop closure detection method of the present invention;

[0031] Figure 3 are the blocks obtained by dividing the point cloud of the current frame according to the image superpixel segmentation;

[0032] Figure 4 is the plane fitted by the point cloud of the current frame;

[0033] Figure 5Among them, (a) is the plane with a vertical relationship in the current frame, and (b) is the undirected graph established for the current frame according to the vertical relationship between planes;

[0034] Figure 6 Among them, (a) is the plane with a vertical relationship in the candidate frame, and (b) is the undirected graph established for the candidate frame according to the vertical relationship between planes;

[0035] Figure 7 Among them, (a) is the plane matched by the current frame, and (b) is the plane matched by the candidate frame;

[0036] Figure 8 Among them, (a) is the block overlap situation of the current frame, (b) is the block overlap situation of the candidate frame, (c) is the point cloud overlap situation after rough registration of the current frame and the candidate frame, and the green blocks in (a), (b), and (c) are the overlapping areas, while the red ones are not. Detailed implementation manners

[0037] The following further describes the present invention in detail in conjunction with embodiments, but the protection scope of the present invention is not limited thereto.

[0038] The present invention relates to a loop detection method based on the geometric structure between planes. As Figure 1 shown, the method includes the following steps:

[0039] (1) Obtain the corresponding RGB image and depth image by an RGB-D camera and perform preprocessing;

[0040] (2) Perform superpixel segmentation on the preprocessed RGB image to obtain a number of pixel regions, divide the point cloud of the depth image corresponding to the pixel regions, calculate the normal vectors of the point cloud blocks corresponding to each pixel region after division, combine the point cloud blocks based on the normal vector relationship, obtain the corresponding planes and edges, and construct an undirected graph;

[0041] (3) Based on the undirected graph, search for basic loops;

[0042] (4) Search for the optimal common subgraph of the undirected graphs of the current frame and the query frame in the key frame list. If there is an effective optimal common subgraph, list this frame as a candidate frame and perform geometric verification according to the overlap degree of the point cloud blocks, and determine the loop frame based on the verification result.

[0043] The following is described in conjunction with specific embodiments. As Figure 2 shown, when the embodiment is applied to the task of simultaneous localization and mapping in a room by an agent, the agent can identify the repeatedly visited scenes, so that the agent can calibrate and update the position of the agent and update the map. Considering that the depth camera may be distorted at a long distance, 5m is defined as the effective value, and the effective scene is without reflection and direct light source.

[0044] (1) Obtain the corresponding RGB image and depth image using an RGB-D camera and perform preprocessing;

[0045] The preprocessing includes performing bilateral Gaussian filtering on the RGB image and the depth image.

[0046] (2) Perform superpixel segmentation on the preprocessed RGB image to obtain a number of pixel regions, partition the point cloud of the depth image corresponding to the pixel regions, calculate the normal vector of the point cloud block corresponding to each pixel region after partitioning, and combine the point cloud blocks based on the normal vector relationship to obtain the corresponding plane and edges, and construct an undirected graph;

[0047] Specifically, first, according to the partitioned pixel regions, partition the point cloud of the depth image corresponding to each pixel region except the boundary, and combine the internal parameters of the camera and the calculation of the depth image to obtain a set of points , and (if the number of points is too small, it is invalid), calculate the covariance matrix of this set of points and perform eigenvalue decomposition,

[0048] (1)

[0049] (2)

[0050] where is the center point of the point cloud block, and the eigenvalues , , are obtained. Take the eigenvector corresponding to the largest eigenvalue as the normal vector of the current point cloud block , and the fitting plane of the current point cloud block is denoted as .

[0051] It is necessary to verify the effectiveness of the fitting plane, including the following steps:

[0052] (2-1) Define a dynamic threshold . If the mean square error of the fitting plane , then the fitting plane is invalid, otherwise proceed to the next step;

[0053] Here, the mean square error of the fitting plane satisfies

[0054] (3)

[0055] where is 's projection point on ;

[0056] The dynamic threshold then satisfies

[0057] (4)

[0058] Among them, is the depth standard deviation, DS is the depth scaling factor of the depth camera, is the tolerance, is the average depth value;

[0059] In this embodiment, .

[0060] The result is as Figure 3 shown.

[0061] (2-2) Verify the validity of the plane by calculating the abnormality of each three-dimensional point except the boundary;

[0062] Check the pixel point corresponding to the three-dimensional point , if it satisfies max(|depth(p m,n )-depth(p m+1,n )|,|depth(p m,n )-depth(p m,n+1 )|) < T ds , then the fitted plane is valid, where Calculate the depth value of the pixel point , and are respectively the adjacent points in the horizontal and vertical directions, is a dynamic threshold, satisfying,

[0063]

[0064] Among them, is a custom ratio, is the tolerance, is the depth value;

[0065] In this embodiment, .

[0066] (2-3) Store the valid plane into the plane list and the block list ; obtain the adjacent planes of the planes in the list.

[0067] Based on the similarity of the normal vectors between adjacent point cloud blocks, use the depth-first search algorithm to merge the valid planes fitted by all the qualified point cloud blocks found into one plane, and the condition is based on the evaluation of the normal vectors and distances between planes; update the plane list and calculate the center point and normal vector of the new plane.

[0068] Specifically, sort in ascending order according to the mean square error of the planes, traverse from the beginning. For the planes that are not referenced, perform a depth-first search based on the adjacent planes until the search depth exceeds the threshold or there are no planes to be merged. Group the searched planes into a cluster, and finally merge the planes in each cluster; assume there are two adjacent planes , and the corresponding center points and normal vectors are respectively and . The conditions for merging the two planes are:

[0069] (7)

[0070] (8)

[0071] Here and are two thresholds. Substitute the points included in the planes of each cluster into equations (1) and (2) to obtain the center point and normal vector of the merged plane; update the plane list , add the obtained new plane to the list, and remove the merged planes;

[0072] In a specific implementation, if the included angle of the normal vectors is less than , and the maximum projection distance is less than , then merge the planes.

[0073] After completing the basic merging of adjacent planes, further merge the planes according to the idea of hierarchical clustering;

[0074] Construct a comparison matrix ,

[0075]

[0076] wherein, is the mean square error of the merged plane of plane . If it meets the merging conditions and , then it remains unchanged (not processed), otherwise it is zero. Merge the planes corresponding to the non-zero minimum term and remove the corresponding rows and columns in , add the new plane to the matrix until the matrix becomes a zero matrix; update the plane list, and remove the planes with an area less than the threshold ;

[0077] In the specific implementation process, if the included angle of the normal vectors of two planes is less than , and the maximum projection distance is less than , then merge the planes, and then consider the planes with an area less than as invalid and update the list Remove the invalid planes.

[0078] The result is as shown in Figure 4 .

[0079] Based on the obtained planar list above, considering that there are generally perpendicular planes in the indoor environment, such as Figure 5 , Figure 6 as shown, construct an undirected graph of the perpendicular relationship between the planes in the current frame and the candidate frame , with the vertex set , and the edge set ;

[0080] Generally, planes with a normal vector angle greater than are regarded as perpendicular planes.

[0081] (3) Based on the undirected graph, find the basic loops;

[0082] Take each vertex as a seed, search with a depth of four, and find loops with three or four vertices. These are indoor planar combinations with common characteristics. Define these two types of loops as basic loops and establish a basic loop list , and a hash table with the vertex as the hash value , .

[0083] (4) Find the optimal common subgraph of the undirected graphs of the current frame and the query frames in the key frame list. If there is an effective optimal common subgraph, list this frame as a candidate frame and perform geometric verification based on the overlap degree of the point cloud blocks. Determine the loop closure frame based on the verification result;

[0084] (4-1) According to the established undirected graph and the found basic loops, compare the basic loops in the current frame and the query frame. Select the top pairs of loops with the highest similarity, satisfying the same number of vertices in the loop, the same adjacency matrix of the loop, and the area difference degree of the planes represented by the vertices being less than the threshold as candidate loop pairs. Generally, N is taken as 5 and added to the candidate loop pair list;

[0085] Assume:

[0086] The graph of the current frame is , and its basic loop list ;

[0087] The graph of the query frame is , and its basic loop list ;

[0088] Retrieve a pair of loops from and respectively:

[0089]

[0090]

[0091] and One-to-one correspondence is used to match this pair of rings, and there will be 4 matching results. The adjacency matrix of the rings is constructed for each result The valid condition for this matching result of this pair of rings is that the number of vertices of the rings is the same, the adjacency matrices of the rings are the same, and the area difference degree of the planes represented by the vertices is less than the threshold, that is,[[]]

[0092]

[0093]

[0094]

[0095] Among them,[[]] is a preset threshold, and 0.8 is taken here,[[]] represents the vertex represents the area of the plane, and the method for constructing the adjacency matrix is The similarity s calculation method for two rings is

[0096]

[0097]

[0098]

[0099] Match this pair of rings according to the valid and most similar matching result, and finally add this pair of rings with the greatest similarity to the candidate ring pair list Satisfy

[0100]

[0101] (4-2) Based on the candidate ring pair list Expand to obtain the common subgraph;

[0102] According to the candidate ring pair list, construct subgraphs for the base rings of each pair of candidate ring pairs according to their respective graphs. Find the rings adjacent to the base rings in each graph and make comparisons. Add the pair of rings with the greatest similarity that is valid to the subgraph. If the adjacency matrices of the two new subgraphs are equal and this pair of new rings meets the valid conditions in the process of finding the base rings, then this pair of rings is considered valid. Then, find adjacent rings on the basis of the new subgraph and repeat the process until no more items can be added, obtaining the common subgraph of the two graphs.[[]]

[0103] Define the relationship between two rings that belong to the same undirected graph and have the same vertices as adjacent. For each candidate ring pair the base ring and Add subgraphs constructed according to their respective graphs In it, find the rings adjacent to the base ring in their respective graphs and having at least one vertex not in the subgraph, and select the pair of rings with the largest similarity and effectiveness and add them to the subgraph. Here, the effective condition is that the effective condition for the previous ring matching is equal to the adjacency matrix established based on the new subgraph;

[0104] After adding a new base ring, continue to search for eligible base rings adjacent to the subgraph and add them to the subgraph until no match can be found. The vertices of the final subgraph correspond one by one and the edges are equal, obtaining two subgraphs with the same structure, which are the common subgraphs of the two graphs , and at most N pairs of common subgraphs can be found.

[0105] The result is as Figure 7 shown.

[0106] (4-3) Obtain the planar matching between two frames based on the common subgraph, and use the difference value to verify the effectiveness of the common subgraph; take the common subgraph with the smallest difference value as the optimal common subgraph, and use this difference value as the only difference value between the two frames;

[0107] (9)

[0108] (10)

[0109] (11)

[0110] When , the common subgraph is effective. Take the common subgraph with the smallest difference value as the optimal common subgraph, where is a preset threshold, and here it is taken as 0.3;

[0111] Finally, among the query frames of all existing optimal common subgraphs, take the K query frames with the smallest difference values as candidate frames.

[0112] (4-4) Use the transformation matrix obtained from the maximum overlap degree between the current frame and the candidate frames as the rough registration between the current frame and the candidate frames. If exceeds the preset threshold, the loop is effective. Finally, take the candidate frame with the largest overlap degree as the loop frame.

[0113] Specifically, during rough registration, in the current frame and the candidate frames, according to the common subgraph, traverse the edges to find two pairs of perpendicular planes , , where , , calculate the pose transformation matrix from the current frame to the candidate frame,

[0114] (12)

[0115] (13)

[0116] (14)

[0117] (15)

[0118] Among them, the functions and are to obtain the normal vector and the center point of the plane represented by the vertex ; in this process, we construct a coordinate system to calculate the rotation matrix through the normal vectors of two pairs of planes. In order to correctly correspond the coordinate systems of the current frame and the candidate frame, it is necessary to unify the directions of the normal vectors of the planes, that is, the directions are inward, .

[0119] To verify that the candidate frame is the loop closure frame of the current frame, it is necessary to calculate the overlap degree between the two frames :

[0120] (16)

[0121] Among them, is the number of overlapping blocks, is the number of blocks in the candidate frame. The calculation process of the number of overlapping blocks is to establish a new KD-tree based on the center points of the blocks in the candidate frame, traverse the blocks in the current frame, and use the center coordinates of the block after pose transformation as the query value to find the nearest neighbor in the KD-tree (the item with the nearest query coordinate). The judgment basis for whether the blocks overlap is

[0122] (17)

[0123] (18)

[0124] Generally speaking, if the included angle between the normal vectors of two blocks is less than , and the maximum projection distance is less than , then these two blocks are regarded as overlapping, and finally the overlap degree corresponding to this edge can be obtained.

[0125] If there are n edges in the common subgraph, then n overlap degrees can be obtained. The maximum value of the overlap degrees is used as the only overlap degree between the current frame and the candidate frame, and the corresponding pose transformation matrix is its rough registration;

[0126] If Exceed the preset threshold value , generally 0.3 is adopted here, then the loop is valid, and the candidate frame with the largest overlap degree is used as the loop frame.

[0127] The result is as Figure 8 shown.

[0128] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0130] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in one Figure 1 process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0132] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0133] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A loop detection method based on geometric structure between planes, characterized by: The method comprises the following steps: S1 uses an RGB-D camera to obtain the corresponding RGB image and depth image and perform preprocessing; S2 performs superpixel segmentation on the preprocessed RGB image to obtain several pixel regions, divides the point cloud of the depth map corresponding to the pixel regions, and calculates the normal vector of the point cloud block corresponding to each pixel region after the division, combines the point cloud blocks based on the normal vector relationship, obtains the corresponding plane and edge, and constructs an undirected graph. In the vertex set , edge set ; S3, based on the undirected graph, takes each vertex as a seed, searches for a ring with three or four vertices at a depth of four, and defines these two types of rings as basic rings; S4 searches for the optimal common subgraph of the undirected graph of the current frame and the query frame in the key frame list. According to the established undirected graph and the found basic ring, the basic rings in the current frame and the query frame are compared. The top subgraphs with the highest similarity and satisfying the same number of ring vertices, the same adjacency matrix of the rings and the difference in the area of ​​the planes represented by the vertices is less than the threshold are selected. The rings are taken as candidate ring pairs, added to the candidate ring pair list, and the common subgraph is expanded. Based on the common subgraph, the plane matching between the two frames is obtained, and the difference value is used. Verify the validity of the common subgraph; take the common subgraph with the smallest difference value as the optimal common subgraph, and take the difference value as the only difference value between the two frames; finally, among all the query frames of the optimal common subgraph, take the query frames with the smallest difference values ​​as candidate frames, and perform geometric verification according to the overlap of the point cloud blocks, and determine the loop frame based on the verification results.

2. The loop closure detection method based on inter-plane geometric structure according to claim 1, characterized in that: In S1, preprocessing includes bilateral Gaussian filtering of RGB images and depth images.

3. The loop closure detection method based on inter-plane geometric structure according to claim 1, characterized in that: In S2, the point cloud of the depth map is divided into a set of points corresponding to each pixel area except the boundary. ,and , calculate the covariance matrix of this set of points And perform eigenvalue decomposition, ， ， in is the center point of the point cloud block, and the eigenvalue is obtained , , , take the largest eigenvalue The corresponding eigenvector is used as the normal vector of the current point cloud block , the fitting plane of the current point cloud block is recorded as .

4. The loop closure detection method based on inter-plane geometric structure according to claim 3, characterized in that: Verifying the validity of the fitting plane includes the following steps: S2.1 Defining dynamic thresholds , if the mean square error of the fitted plane is , then the fitting plane is invalid, otherwise proceed to the next step; S2.2 Check the pixel points corresponding to the 3D points , if max(|depth(p m,n )-depth(p m+1,n )|,|depth(p m,n )-depth(p m,n+1 )|)<T ds , then the fitting plane is valid, where Counting pixels The depth value of and They are At adjacent points in the horizontal and vertical directions, is the dynamic threshold, , is the depth value, For custom ratio, for tolerance; S2.3 Store the valid plane into the plane list ; Get the adjacent planes of the planes in the list.

5. The method for loop closure detection based on inter-plane geometric structure according to claim 3 or 4, characterized in that: Based on the similarity of normal vectors between adjacent point cloud blocks, a depth-first search algorithm is used to merge the valid planes fitted by all the searched point cloud blocks that meet the conditions into one plane, where the conditions are based on the normal vectors and distance evaluation between the planes; Update the plane list and calculate the center point and normal vector of the new plane.

6. The loop closure detection method based on inter-plane geometric structure according to claim 5, characterized in that: Constructing a comparison matrix , ， in, Is a plane The mean square error of the merged plane is If it is unchanged, otherwise it is zero, and the planes corresponding to the non-zero minimum terms are merged and Remove the corresponding rows and columns, add the new plane to the matrix until the matrix is ​​a zero matrix; update the plane list and add the planes with an area less than the threshold Plane culling.

7. The loop closure detection method based on inter-plane geometric structure according to claim 1, characterized in that: In S3, create a basic ring list and a hash table with vertices as hash values .

8. The loop closure detection method based on inter-plane geometric structure according to claim 1, characterized in that: According to the list of candidate ring pairs, the basic rings of each pair of candidate ring pairs are used to construct subgraphs according to their respective graphs. The rings adjacent to the basic rings are found in each graph and compared. The pair of valid rings with the greatest similarity is added to the subgraph. If the adjacency matrices of the two new subgraphs are equal and the pair of new rings meet the valid conditions in the process of searching for basic rings, the pair of rings is considered valid. Then, adjacent rings are searched on the basis of the new subgraph. Repeat this process and expand the subgraph until there are no more items to add, thus obtaining a common subgraph of the two graphs.

9. The loop closure detection method based on inter-plane geometric structure according to claim 1, characterized in that: The maximum overlap between the current frame and the candidate frame The obtained transformation matrix is ​​used as the rough registration of the current frame and the candidate frame. If the preset threshold is exceeded, the loop is valid, and finally the candidate frame with the largest overlap is taken as the loop frame.

Images