Visual SLAM Loop Detection Method, Device and System Based on Semantic Topology

Abstract

The present invention relates to the field of machine vision technology, and specifically discloses a loop detection method, device and system for visual SLAM based on semantic topology, including: acquiring real-time environmental image information during the movement of a mobile robot, and obtaining image semantic information and the movement information of the mobile robot according to the real-time environmental image information; constructing object road signs according to the image semantic information, and constructing map point road signs according to the movement information of the mobile robot; constructing a hierarchical map according to the image semantic information, object road signs and map point road signs; extracting the semantic map information of the current frame, and performing layer-by-layer matching of the current frame semantic map information with the hierarchical map; performing object constraint pose optimization and map point constraint pose optimization in sequence according to the semantic object road sign matching result to obtain the current pose information of the mobile robot. The loop detection method for visual SLAM based on semantic topology provided by the present invention can achieve robust and accurate loop detection.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular, to a visual SLAM loop detection method based on semantic topology, a visual SLAM loop detection device based on semantic topology, and a visual SLAM loop detection system based on semantic topology. Background Art

[0002] For a mobile robot, perceiving the environment through visual image data and obtaining its own positioning are the keys to realizing autonomous movement of the robot. Simultaneous Localization and Mapping (SLAM) is one of the key technologies for realizing environmental perception and autonomous positioning. Visual SLAM usually consists of three modules: a visual odometer, a backend, and a loop detection module. The visual odometer can obtain a locally high-precision trajectory. However, as time goes by, the system will inevitably accumulate errors. The loop detection module mainly solves the above problems. Its main function is to determine whether the environment where the camera is currently located has been reached before. After obtaining a loop, it provides more accurate pose constraints for backend optimization, eliminates the accumulated drift error, and obtains a globally consistent map.

[0003] Currently, the mainstream loop detection methods are mainly based on image appearance. The representative method is BoW. It encodes images using artificially designed pixel features, interprets the loop detection problem as an image retrieval problem, and determines whether it is the same scene by calculating the similarity between the current image and historical images. However, the artificially designed features (point and line features) are less robust to changes in illumination and viewing angles. When the above conditions change greatly, the performance of the loop detection method based on image appearance will be greatly reduced.

[0004] Therefore, how to achieve robust and accurate loop detection has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0005] The present invention provides a visual SLAM loop detection method based on semantic topology, a visual SLAM loop detection device based on semantic topology, and a visual SLAM loop detection system based on semantic topology, which solve the problem in the related art that robust and accurate loop detection cannot be achieved.

[0006] As a first aspect of the present invention, a loop detection method for visual SLAM based on semantic topology is provided, which includes: obtaining real-time environmental image information during the movement of a mobile robot, and obtaining image semantic information and the movement information of the mobile robot according to the real-time environmental image information; constructing object road signs according to the image semantic information, and constructing map point road signs according to the movement information of the mobile robot; constructing a hierarchical map according to the image semantic information, object road signs and map point road signs, wherein the hierarchical map includes a semantic layer based on image semantic information, an object layer based on object road signs, and a map point layer based on map point road signs; extracting current frame semantic map information according to the real-time environmental image information of the current frame, and performing layer-by-layer matching between the current frame semantic map information and the hierarchical map to obtain a semantic object road sign matching result; performing object constraint pose optimization and map point constraint pose optimization in sequence according to the semantic object road sign matching result to obtain the current pose information of the mobile robot.

[0007] Further, constructing object road signs according to the image semantic information includes: determining that an ellipsoid model represents an object, and optimizing the ellipsoid model according to the image semantic information to construct object road signs; extracting the object road signs as nodes, and extracting edges according to the local topological relationship of the semantic map to construct a semantic topology graph.

[0008] Further, extracting the object road signs as nodes and extracting edges according to the local topological relationship of the semantic map to construct a semantic topology graph includes: extracting the object road signs as nodes, where each node includes the geometric parameter attribute information of the object road sign itself; judging whether the same key frames of the current object road sign and the observed object road signs exceed a preset number according to the co-visibility information of the current key frame; if it exceeds the preset number, adding the current object road sign to the local object map representing the adjacent area of the current node; creating edges according to the Euclidean distance sorting of the local object map in the adjacent area of the current node and the current node to construct a semantic topology graph.

[0009] Further, constructing map point road signs according to the movement information of the mobile robot includes: obtaining feature points associated with the object road signs according to the movement information of the mobile robot; constructing map point road signs according to the feature points associated with the object road signs.

[0010] Further, extract the current frame semantic map information based on the real-time environmental image information of the current frame, and perform layer-by-layer matching between the current frame semantic map information and the hierarchical map to obtain the semantic object landmark matching result, including: extracting the current frame image semantic information based on the real-time environmental image information of the current frame; performing semantic map extraction on the current frame image semantic information to obtain the current frame semantic map; performing matching in the semantic layer of the hierarchical map according to the node label information in the current frame semantic map to obtain the set of object landmarks matching the current frame scene; calculating the topological descriptor similarity between the set of object landmarks and the semantic landmarks in the object layer to obtain the preliminary semantic object landmark matching result; evaluating the spatial distribution similarity of the preliminary semantic object landmark matching result to eliminate the mis-matching results and obtain the final semantic object landmark matching result.

[0011] Further, calculate the topological descriptor similarity between the set of object landmarks and the semantic landmarks in the object layer to obtain the preliminary semantic object landmark matching result, including: calculating the similarity between the node and the object matching according to the local semantic topological descriptor; determining the matching with the highest similarity score as the true matching to obtain the preliminary semantic object landmark matching result.

[0012] Further, evaluate the spatial distribution similarity of the preliminary semantic object landmark matching result to eliminate the mis-matching results and obtain the final semantic object landmark matching result, including: converting the preliminary semantic object landmark matching result into the same coordinate system; evaluating the spatial distribution similarity of the preliminary semantic object landmark matching result in the same coordinate system to eliminate the mis-matching results and obtain the final semantic object landmark matching result.

[0013] Further, perform object constraint pose optimization and map point constraint pose optimization in sequence according to the semantic object landmark matching result to obtain the current pose information of the mobile robot, including: performing rough pose estimation between the semantic object landmark matching result and the target detection result according to the reprojection error to obtain the object constraint pose optimization result; performing pose refinement on the object constraint pose optimization result according to the map point layer based on the map point landmark to obtain the current pose information of the mobile robot.

[0014] As another aspect of the present invention, there is provided a loop detection device for visual SLAM based on semantic topology, which is used to implement the loop detection method for visual SLAM based on semantic topology described above. The device includes: an acquisition module, configured to acquire real-time environmental image information during the movement of the mobile robot, and obtain image semantic information and the movement information of the mobile robot according to the real-time environmental image information; a landmark construction module, configured to construct object landmarks according to the image semantic information, and construct map point landmarks according to the movement information of the mobile robot; a map construction module, configured to construct a hierarchical map according to the image semantic information, the object landmarks, and the map point landmarks, where the hierarchical map includes a semantic layer based on the image semantic information, an object layer based on the object landmarks, and a map point layer based on the map point landmarks; a matching module, configured to extract the current frame semantic map information according to the real-time environmental image information of the current frame, and perform layer-by-layer matching with the hierarchical map according to the current frame semantic map information to obtain a semantic object landmark matching result; and an optimization module, configured to perform object constraint pose optimization and map point constraint pose optimization in sequence according to the semantic object landmark matching result to obtain the current pose information of the mobile robot.

[0015] As another aspect of the present invention, there is provided a loop detection system for visual SLAM based on semantic topology, which includes: a mobile robot, an image acquisition device, and the loop detection device for visual SLAM based on semantic topology described above. The image acquisition device is disposed on the mobile robot, and the image acquisition device is communicatively connected to the loop detection device for visual SLAM based on semantic topology; the image acquisition device is capable of acquiring real-time environmental image information in real time during the movement of the mobile robot to obtain real-time environmental image information; the loop detection device for visual SLAM based on semantic topology is capable of constructing object landmarks, map point landmarks, and a hierarchical map according to the real-time environmental image information, extracting the current frame semantic map information according to the real-time environmental image information of the current frame, performing layer-by-layer matching with the hierarchical map, and performing object constraint pose optimization and map point constraint pose optimization on the semantic object landmark matching result after layer-by-layer matching to obtain the current pose information of the mobile robot.

[0016] The loop detection method for visual SLAM based on semantic topology provided by the present invention constructs an object-level semantic map based on image semantic information, uses a topological descriptor to realize the association between the current frame and the object landmarks of the global semantic map, and combines high-level object-level landmarks and low-level map point landmarks to achieve coarse-to-fine loop pose estimation; robust and accurate loop detection can be realized by performing layer-by-layer matching between the current frame semantic map information and the hierarchical map; after obtaining an accurate semantic landmark matching result, object reprojection constraints and map point reprojection constraints are respectively used to achieve coarse-to-fine pose estimation. Description of the Drawings

[0017] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation to the present invention.

[0018] Figure 1 It is a flowchart of the visual SLAM loop detection method based on semantic topology provided by the present invention.

[0019] Figure 2 It is a flowchart of the specific implementation of the visual SLAM loop detection method based on semantic topology provided by the present invention.

[0020] Figure 3 It is a flowchart of the specific implementation for obtaining object road signs provided by the present invention.

[0021] Figure 4 It is a flowchart of constructing a semantic topology map provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the extraction of local topology descriptors provided by the present invention.

[0023] Figure 6 It is a flowchart of constructing map point road signs provided by the present invention.

[0024] Figure 7 It is a schematic diagram of hierarchical road signs provided by the present invention.

[0025] Figure 8 It is a flowchart of obtaining the matching result of semantic object road signs provided by the present invention.

[0026] Figure 9 It is a flowchart of obtaining the preliminary matching result of semantic object road signs provided by the present invention.

[0027] Figure 10 It is a flowchart of obtaining the final matching result of semantic object road signs provided by the present invention.

[0028] Figure 11 It is a schematic diagram of the node polar coordinate system provided by the present invention.

[0029] Figure 12 It is a flowchart of obtaining the current pose information of the mobile robot provided by the present invention.

[0030] Figure 13 It is a structural block diagram of the visual SLAM loop detection device based on semantic topology provided by the present invention. Specific Embodiments

[0031] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0032] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances for the embodiments of the present invention described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] The environment contains rich semantic information. The semantic road signs have long-term invariability and are robust to changes in light and viewing angles. There are methods in the prior art for loop detection by fusing semantic information. However, the loop detection of fusing semantic information in the prior art usually only utilizes image semantic information and does not fully consider the topological information of the environment. When there are occlusions in the image and the observations are incomplete, the semantic feature vectors change significantly, ultimately resulting in missed detections in loop detection; in addition, it is also possible that the observation conditions of different images are different, resulting in inaccurate positions of semantic nodes, and ultimately leading to failed loop judgments for images of the same scene.

[0035] Based on this, in the present embodiment, a visual SLAM loop detection method based on semantic topology is provided. Figure 1 It is a flowchart of the visual SLAM loop detection method based on semantic topology provided according to the embodiment of the present invention, as Figure 1 shown, including: S100, obtaining real-time environmental image information during the movement of the mobile robot, and obtaining image semantic information and the movement information of the mobile robot according to the real-time environmental image information.

[0036] In an embodiment of the present invention, during the movement of the mobile robot, the image acquisition device installed on the mobile robot can collect the environmental image information around the mobile robot in real time to obtain real-time environmental image information.

[0037] It should be noted that the real-time environmental image information here can specifically be an RGB-D image, that is, image information including depth information. For this real-time environmental image information, the semantic information in the RGB-D image can be extracted through a semantic information extraction algorithm, such as the YOLOv8 algorithm, to obtain image semantic information; in addition, the motion information of the mobile robot can also be detected and evaluated through a visual odometer to obtain the motion information of the mobile robot.

[0038] It should be understood that when extracting image semantic information, an open-source object detection framework YOLOv8 is used. In order to obtain the semantic information of the image, other object detection models and semantic segmentation models can also achieve this purpose.

[0039] S200. Construct object road signs according to the image semantic information, and construct map point road signs according to the motion information of the mobile robot.

[0040] In an embodiment of the present invention, object road signs are constructed according to the image semantic information in combination with the depth information, and map point road signs are constructed according to the motion information of the mobile robot and by using the feature points associated with the object road signs.

[0041] S300. Construct a hierarchical map according to the image semantic information, object road signs, and map point road signs, where the hierarchical map includes a semantic layer based on the image semantic information, an object layer based on the object road signs, and a map point layer based on the map point road signs.

[0042] In an embodiment of the present invention, a hierarchical map is constructed based on the image semantic information, object road signs, and map point road signs. As Figure 2 shown, the topological information of the high-level road signs and the geometric information of the low-level point road signs are fused according to the hierarchical strategy to maintain the hierarchical map.

[0043] S400. Extract the current frame semantic map information according to the real-time environmental image information of the current frame, and perform layer-by-layer matching between the current frame semantic map information and the hierarchical map to obtain a semantic object road sign matching result.

[0044] In the embodiments of the present invention, after extracting the semantic map through the semantic information of the current frame RGB-D image, layer-by-layer matching is performed in the hierarchical map. Matching is searched in the semantic layer according to the node label information in the current frame semantic map to obtain the set of object road signs that match in the current scene. By calculating the similarity of the topological descriptors between the semantic road signs in the object layer, semantic-object road sign matching is obtained. The semantic-object road signs are unified in the same coordinate system, and false matches are eliminated by evaluating the similarity of the spatial distribution to obtain an accurate semantic object road sign matching result.

[0045] S500. According to the semantic object road sign matching result, object constraint pose optimization and map point constraint pose optimization are sequentially performed to obtain the current pose information of the mobile robot.

[0046] In the embodiments of the present invention, based on the semantic association result, rough initial pose estimation is performed using the object reprojection constraint represented by an ellipsoid. The current frame feature points are guided by semantic association and the initial pose to match in the map point layer, and more accurate pose estimation is achieved using precise geometric information.

[0047] In summary, the visual SLAM loop detection method based on semantic topology provided by the present invention constructs an object-level semantic map based on image semantic information, uses topological descriptors to associate the current frame with the object road signs of the global semantic map, and combines high-level object-level road signs and low-level map point road signs to achieve loop pose estimation from rough to fine; robust and accurate loop detection can be achieved by layer-by-layer matching of the current frame semantic map information with the hierarchical map; after obtaining an accurate semantic road sign matching result, object reprojection constraint and map point reprojection constraint are respectively used to achieve pose estimation from rough to fine.

[0048] In the embodiments of the present invention, object road signs are constructed according to the image semantic information, such as Figure 3 shown, including: S210. Determine that the object is represented by an ellipsoid model, and optimize the ellipsoid model according to the image semantic information to construct object road signs.

[0049] The object can be represented as a quadratic surface with constraints , which projects onto the image plane of the th frame as a quadratic curve . When the image semantic information is obtained, a semantic mask is used to fit an ellipse as the observation result of the object in the current frame. Through the observation results of multiple frames, the ellipsoid model is optimized by minimizing the reprojection error of the ellipsoid to obtain accurate object road signs: , where the residual is the Wasserstein distance between the ellipse and the ellipsoid.

[0050] S220. Extract the object road signs as nodes, and extract edges according to the local topological relationship of the semantic map to construct a semantic topological graph.

[0051] It should be understood that in order to more accurately obtain the object road sign information in the image, the object road signs are extracted as nodes, edges are extracted according to the local topological relationship of the semantic map, and a semantic topological graph is constructed.

[0052] Specifically, the object road signs are extracted as nodes, and edges are extracted according to the local topological relationship of the semantic map to construct a semantic topological graph, as Figure 4 shown, including: S221. Extract the object road signs as nodes, where each node includes the geometric parameter attribute information of the object road sign itself; S222. Determine whether the number of common key frames of the current object road sign and the observed object road signs exceeds a preset number according to the co-visibility information of the current key frame; S223. If it exceeds the preset number, add the current object road sign to the local object map representing the adjacent area of the current node; S224. Create edges according to the Euclidean distance between the local object map in the adjacent area of the current node and the current node to construct a semantic topological graph.

[0053] It should be understood that for the object map target semantic map the extraction specifically includes: extracting the object-level road signs as nodes, and each node contains the object own geometric parameter attribute information , where is the dual quadric surface representation of the object, , and respectively represent the vectors of the object position, orientation and scale.

[0054] After extracting the nodes, according to the co-visibility information of the key frames, if a certain object and the same key frames of the observed current object exceed 3 frames, then add the object to the local object map representing the adjacent area of the current node , sort according to the Euclidean distance from the current node, and connect each node to its neighbor nodes within a certain threshold to create edges. The Euclidean distance between nodes is defined as the weight of the edge .

[0055] The above process of graph extraction can be represented as a topological descriptor of the object, and the extraction process is as Figure 5 shown. This descriptor describes the information around each node and enhances the recognizability of a single node. Therefore, it can be added to the object attributes, , and the newly added A description vector for local semantic topological information.

[0056] In an embodiment of the present invention, a map point landmark is constructed according to the motion information of the mobile robot, as Figure 6 shown, including: S230. Obtaining feature points associated with an object landmark according to the motion information of the mobile robot; S240. Constructing a map point landmark according to the feature points associated with the object landmark.

[0057] It should be understood that, based on the motion information of the mobile robot, feature points associated with an object landmark can be obtained, and based on the associated feature points, a map point landmark can be constructed, and thus the construction of a hierarchical map can be realized.

[0058] In an embodiment of the present invention, a hierarchical map is constructed according to the image semantic information, object landmarks, and map point landmarks. Specifically, as Figure 7 shown, after constructing semantic landmarks and extracting a semantic map, a hierarchical landmark database is maintained according to a hierarchical strategy. The database is divided into three layers: a semantic layer, an object layer, and a map point layer. Semantic layer: The semantic layer covers all semantic labels of the current object map, and each label is associated with an object instance corresponding to the label. This layer can be expressed as a set of labels in the object map . When inputting the current frame semantic map, all possible matching objects in the target map can be quickly queried through the object labels in the current frame semantic map. Object layer: It contains object instances corresponding to each semantic label. To enhance the recognizability of objects, the local co-visibility relationship between objects is used to construct a local topological descriptor of the object. Through the local topological descriptor, matching object landmarks are screened again to find the two most likely matching object landmarks, a polar coordinate system is constructed, and the current frame semantic map and the object landmark semantic map are unified in the same polar coordinate system to evaluate the spatial distribution similarity, eliminate false matches, and obtain the final matching result. Based on the semantic matching result, a rough corrected pose is obtained. Map point layer: This layer contains map points associated with a single object instance. Under the result of object association, source frame feature points are matched, and the pose is refined using the feature point projection constraint to obtain the final corrected pose.

[0059] In an embodiment of the present invention, the current frame semantic map information is extracted according to the real-time environmental image information of the current frame, and the current frame semantic map information is matched layer by layer with the hierarchical map to obtain a semantic object landmark matching result, as Figure 8 shown, including: S410. Extracting the current frame image semantic information according to the real-time environmental image information of the current frame; S420. Performing semantic map extraction on the current frame image semantic information to obtain the current frame semantic map.

[0060] For the current frame source semantic map The extraction specifically may include: a source topological map Constructed based on semantic RGB-D images, obtain semantic information through YOLOv8, and after filtering by confidence, obtain the object detection results of the current frame, and at the same time associate it with feature points. Then, initialize the two-dimensional objects in the source frame as three-dimensional object road signs , where is the semantic label information of the object, is the image semantic mask information, is the initialized centroid, which is obtained by randomly sampling the depth map of the corresponding semantic mask.

[0061] After initializing the object map from the source frame, in the same way as extracting the target topology map, each object is extracted as a node, and at the same time, the nearest neighbor nodes are connected according to the distance threshold to create edges to complete the extraction of the source topology map The above process can also be expressed as a descriptor of the object, and the descriptor form is the same as , and add it to the object attribute .

[0062] S430. Match in the semantic layer of the hierarchical map according to the node label information in the semantic map of the current frame to obtain a set of object road signs that match the current frame scene.

[0063] When the semantic map of the current frame is given , find matching nodes in the semantic map of the object map. First, according to the node label information in , match the labels in the label layer of the maintained hierarchical road sign database . In the hierarchical road sign database, each label in the label layer indexes a set of objects corresponding to the label. Therefore , each node in matches the corresponding label in , and all possible associated object sets can be obtained .

[0064] S440. Calculate the topological descriptor similarity between the set of object road signs and the semantic road signs in the object layer to obtain a preliminary matching result of the semantic object road signs;

[0065] In the embodiment of the present invention, quickly search in the semantic label layer to obtain the set of possible matching object road signs of the nodes in . At this time, the matching between a single node and the object set is one-to-many, and the real matching object needs to be found from the one-to-many .

[0066] Specifically, calculate the topological descriptor similarity between the set of object road signs and the semantic road signs in the object layer to obtain a preliminary matching result of the semantic object road signs, such asFigure 9 As shown in Figure 9 , it includes: S441. Calculate the similarity between a node and an object match according to a local semantic topology descriptor.

[0067] In an embodiment of the present invention, based on a local semantic topology descriptor, the similarity between a node-object match is calculated, and the match with the highest similarity score is considered the true match. For a pair of node-object matches , their similarity score calculation method is as follows: , is the number of same edges between descriptor and . It is determined whether they match according to the label and distance weight of the edge, represents the number of edges of the descriptor.

[0068] S442. Determine the match with the highest similarity score as the true match to obtain a preliminary match result of semantic object road signs.

[0069] In an embodiment of the present invention, the node-object match with the highest similarity score is considered the true match, and finally a node-object match set can be obtained.

[0070] S450. Evaluate the spatial distribution similarity of the preliminary match result of the semantic object road signs to eliminate mis-match results and obtain a final match result of the semantic object road signs.

[0071] Since the descriptor only implicitly encodes the topological relationship within the local range of the node, using the descriptor similarity evaluation to obtain node-object match pairs may have mis-matches. To further eliminate outliers, the spatial distribution similarity is directly evaluated by unifying the node and the object in the same polar coordinate system.

[0072] Specifically, evaluating the spatial distribution similarity of the preliminary match result of the semantic object road signs to eliminate mis-match results and obtain a final match result of the semantic object road signs, as Figure 10 shown, includes:

[0073] S451. Convert the preliminary match result of the semantic object road signs into the same coordinate system;

[0074] S452. Evaluate the spatial distribution similarity of the preliminary match result of the semantic object road signs in the same coordinate system to eliminate mis-match results and obtain a final match result of the semantic object road signs.

[0075] In , select two pairs of node-object match pairs with the highest descriptor similarity to construct a source graph and a target graph The vector in the z-axis direction. At the same time, select the midpoint of the two endpoints as the origin. Source image The origin of the source image , Z-axis and the target image The origin of the target image , Z-axis are represented as follows: , , then select the plane passing through the origin and perpendicular to the Z-axis as the base plane (x-y plane), select any unit vector passing through the origin under this plane as the X-axis, and obtain the Y-axis through cross product: , calculate the azimuth angle between the projection line of the connection line from other nodes and objects to the origin of their respective coordinate systems on the base plane and the positive x-axis , the distance from the node to the base plane (i.e., the Z-axis coordinate), the distance from the node to the Z-axis . Use to describe the coordinates of the node in the current coordinate system. As Figure 11 shown, the coordinate calculation method for any point is as follows: , , , source image and target image The remaining coordinates of can be obtained through the above formula. Compare the coordinates of the node-object matching pairs. If the difference between the corresponding coordinates exceeds a certain threshold , then this match is considered an outlier. After eliminating the mis-matches through polar coordinate verification, the final node-object matching result is obtained.

[0076] In the embodiment of the present invention, according to the semantic object landmark matching result, object constraint pose optimization and map point constraint pose optimization are sequentially performed to obtain the current pose information of the mobile robot. As Figure 12 shown, it includes: S510. Rough pose estimation is performed between the semantic object landmark matching result and the target detection result according to the reprojection error to obtain the object constraint pose optimization result.

[0077] After the node-object matching pairs with outliers removed are obtained through graph matching , the pose is roughly estimated by minimizing the reprojection error between the 3D object ellipsoid landmark and the ellipse fitted by the 2D target detection box.

[0078] , where The residual is determined by the Wasserstein distance between the 2D ellipse and the 3D ellipsoid, which can effectively avoid the deterioration of pose estimation caused by object occlusion and inaccurate observation.

[0079] S520: Refine the pose optimization result of the object constraint pose according to the map point layer based on the map point landmarks to obtain current pose information of the mobile robot.

[0080] After the object reprojection constraint is given an initial value, based on the hierarchical landmarks, the map points associated with the objects in the map point layer and the feature points associated with the matching nodes in the current frame are used for further pose refinement to obtain a more accurate pose. The reprojection constraints of the map points are as follows: ,in, represents the observation value of the feature point, Indicates the pixel coordinates of the map point after being projected onto the image according to the current pose.

[0081] In summary, the present invention provides a visual SLAM loop detection method based on semantic topology. The robot acquires an image of the surrounding environment during movement, extracts image semantic information to construct object-level landmarks, and constructs a hierarchical map by combining object-associated map point landmarks. The current frame image extracts semantic information to construct a semantic landmark, calculates semantic topology descriptors based on local topological relationships, and calculates similarity with high-level object landmarks in the global semantic map to obtain semantic landmark-object landmark matching. The obtained semantic landmark-object landmark matching is uniformly verified under the same polar coordinates, spatial similarity is evaluated and outliers are filtered out to obtain the final semantic association result. Object-level reprojection constraints are constructed based on the semantic landmark-object landmark association result, and the initial pose is calculated. The low-level map point landmarks obtain a more refined pose under the guidance of semantic association and initial pose. Therefore, the visual SLAM loop detection method based on semantic topology utilizes the characteristics of different levels of landmarks: the long-term invariance of high-level semantic landmarks is robust to changes in illumination and perspective; the geometric information of low-level map point landmarks is accurate and can be used for high-precision pose estimation. A hierarchical landmark maintenance strategy integrates the topological information of high-level object-level landmarks with the geometric information of low-level point landmarks, achieving more robust coarse-to-fine loop detection. During semantic landmark matching, efficient local semantic descriptors are used to encode the local context of object landmarks, enabling rapid semantic landmark matching. By unifying semantically matched landmarks in the same polar coordinate system, the spatial distribution similarity of the landmarks is evaluated to eliminate false matches. A comprehensive evaluation of these similarity-based loops enables robust and accurate loop detection. After obtaining accurate semantic landmark matching results, object reprojection constraints and map point reprojection constraints are used to achieve coarse-to-fine pose estimation.

[0082] As another embodiment of the present invention, a visual SLAM loop detection device 100 based on semantic topology is provided, which is used to implement the visual SLAM loop detection method based on semantic topology as described above, wherein, Figure 13As shown in the figure, it includes: an acquisition module 110, configured to acquire real-time environmental image information during the movement of the mobile robot, and obtain image semantic information and the movement information of the mobile robot according to the real-time environmental image information; a landmark construction module 120, configured to construct object landmarks according to the image semantic information, and construct map point landmarks according to the movement information of the mobile robot; a map construction module 130, configured to construct a hierarchical map according to the image semantic information, object landmarks, and map point landmarks, where the hierarchical map includes a semantic layer based on the image semantic information, an object layer based on the object landmarks, and a map point layer based on the map point landmarks; a matching module 140, configured to extract the current frame semantic map information according to the real-time environmental image information of the current frame, and perform layer-by-layer matching with the hierarchical map according to the current frame semantic map information to obtain a semantic object landmark matching result; an optimization module 150, configured to perform object constraint pose optimization and map point constraint pose optimization in sequence according to the semantic object landmark matching result to obtain the current pose information of the mobile robot.

[0083] The visual SLAM loop detection device based on semantic topology provided by the present invention constructs an object-level semantic map based on image semantic information, uses a topological descriptor to associate the current frame with the object landmarks of the global semantic map, and combines high-level object landmarks and low-level map point landmarks to achieve coarse-to-fine loop pose estimation; robust and accurate loop detection can be achieved through layer-by-layer matching of the current frame semantic map information with the hierarchical map; after obtaining an accurate semantic landmark matching result, object reprojection constraints and map point reprojection constraints are respectively used to achieve coarse-to-fine pose estimation.

[0084] For the specific working principle of the visual SLAM loop detection device based on semantic topology of the present invention, reference can be made to the description of the visual SLAM loop detection method based on semantic topology in the foregoing text, and details are not described herein again.

[0085] As another embodiment of the present invention, a visual SLAM loop detection system based on semantic topology is provided, which includes: a mobile robot, an image acquisition device, and the aforementioned visual SLAM loop detection device based on semantic topology. The image acquisition device is arranged on the mobile robot and is communicatively connected to the visual SLAM loop detection device based on semantic topology. The image acquisition device can collect environmental image information in real time during the movement of the mobile robot to obtain real-time environmental image information. The visual SLAM loop detection device based on semantic topology can construct object road signs, map point road signs, and a hierarchical map according to the real-time environmental image information, extract the semantic map information of the current frame and perform layer-by-layer matching with the hierarchical map, and optimize the object constraint pose and the map point constraint pose for the semantic object road sign matching result after layer-by-layer matching to obtain the current pose information of the mobile robot.

[0086] In the embodiment of the present invention, the visual SLAM loop detection device based on semantic topology mainly includes hierarchical map construction, semantic loop detection, and pose optimization. The hierarchical map construction process mainly completes three aspects of work: semantic road sign construction, semantic map extraction, and hierarchical road sign maintenance. After obtaining the image semantic information, an ellipsoid model is used to represent the object, and the observation results of multiple frames are used to optimize the object ellipsoid model to construct accurate semantic road signs. The object road signs are extracted as nodes, and the edges are extracted according to the local topological relationship of the semantic map to construct a semantic topology graph. The topological information of the high-level object-level road signs and the geometric information of the low-level point road signs are fused according to the hierarchical strategy to maintain the hierarchical map.

[0087] The semantic loop detection mainly completes three aspects of work: semantic label matching, topological descriptor matching, and polar coordinate verification. After extracting the semantic map from the semantic information of the current frame RGB-D image, layer-by-layer matching is performed from the hierarchical map. According to the node label information in the current frame semantic map, matching is searched in the semantic layer to obtain the set of object road signs matched in the current scene. By calculating the similarity of the topological descriptors between the semantic road signs in the object layer, the semantic-object road sign matching is obtained. The semantic-object road signs are unified in the same coordinate system, and the false matches are eliminated by evaluating the spatial distribution similarity to obtain the accurate semantic-object road sign matching result.

[0088] The pose optimization process mainly completes two aspects of work: object constraint pose optimization and map point constraint pose optimization. Based on the semantic association result, a rough initial pose estimation is performed using the object reprojection constraint represented by an ellipsoid. The current frame feature points are guided by the semantic association and the initial pose to match in the map point layer, and a more accurate pose estimation is achieved using the accurate geometric information.

[0089] The visual SLAM loop detection system based on semantic topology provided by the present invention constructs an object-level semantic map based on image semantic information, uses a topological descriptor to associate the current frame with the object landmarks in the global semantic map, and combines high-level object-level landmarks and low-level map point landmarks to achieve coarse-to-fine loop pose estimation; robust and accurate loop detection can be achieved by layer-by-layer matching of the current frame semantic map information with the hierarchical map; after obtaining an accurate semantic landmark matching result, object reprojection constraints and map point reprojection constraints are respectively used to achieve coarse-to-fine pose estimation.

[0090] Regarding the specific working principle of the visual SLAM loop detection system based on semantic topology of the present invention, reference can be made to the description of the visual SLAM loop detection method based on semantic topology in the foregoing text, and details will not be elaborated here.

[0091] It can be understood that the above embodiments are merely exemplary embodiments adopted to illustrate the principle of the present invention, and the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also regarded as the protection scope of the present invention.

Claims

1. A visual SLAM loop detection method based on semantic topology, characterized in that Including: Obtain real-time environmental image information during the movement of the mobile robot, and obtain image semantic information and the movement information of the mobile robot according to the real-time environmental image information; Construct object road signs according to the image semantic information, and construct map point road signs according to the movement information of the mobile robot; Construct a hierarchical map according to the image semantic information, object road signs and map point road signs, wherein the hierarchical map includes a semantic layer based on image semantic information, an object layer based on object road signs, and a map point layer based on map point road signs; Extract the current frame semantic map information according to the real-time environmental image information of the current frame, and perform layer-by-layer matching with the hierarchical map according to the current frame semantic map information to obtain a semantic object road sign matching result; Perform object constraint pose optimization and map point constraint pose optimization in sequence according to the semantic object road sign matching result to obtain the current pose information of the mobile robot; Among them, constructing object road signs according to the image semantic information includes: Determine that an ellipsoid model represents an object, and optimize the ellipsoid model according to the image semantic information to construct object road signs; Extract the object road signs as nodes, and extract edges according to the local topological relationship of the semantic map to construct a semantic topological graph; Among them, extracting the object road signs as nodes and extracting edges according to the local topological relationship of the semantic map to construct a semantic topological graph includes: Extract the object road signs as nodes, and each node includes the geometric parameter attribute information of the object road sign itself; Judge whether the number of the same key frames of the current object road sign and the observed object road sign exceeds a preset number according to the co-visibility information of the current key frame; If it exceeds the preset number, add the current object road sign to the local object map representing the adjacent area of the current node; Create edges according to the Euclidean distance between the local object map in the adjacent area of the current node and the current node to construct a semantic topological graph; Among them, constructing map point road signs according to the movement information of the mobile robot includes: Obtain the feature points associated with the object road signs according to the movement information of the mobile robot; Construct map point road signs according to the feature points associated with the object road signs.

2. The method for loop detection of visual SLAM based on semantic topology according to claim 1, wherein Extract the current frame semantic map information according to the real-time environmental image information of the current frame, and perform layer-by-layer matching with the hierarchical map according to the current frame semantic map information to obtain a semantic object road sign matching result, including: Extract the current frame image semantic information according to the real-time environmental image information of the current frame; Perform semantic map extraction on the current frame image semantic information to obtain the current frame semantic map; Match according to the node label information in the current frame semantic map in the semantic layer of the hierarchical map to obtain a set of object road signs matching the current frame scene; Calculate the similarity of the topological descriptors between the set of object road signs and the semantic road signs in the object layer to obtain a preliminary semantic object road sign matching result; Evaluate the spatial distribution similarity of the preliminary semantic object road sign matching result to eliminate mis-matching results and obtain the final semantic object road sign matching result.

3. The visual SLAM loop detection method based on semantic topology according to claim 2, characterized in that, Calculate the similarity of the topological descriptors between the set of object road signs and the semantic road signs in the object layer to obtain a preliminary semantic object road sign matching result, including: Calculate the similarity between the node and the object match according to the local semantic topology descriptor; Determine the match with the highest similarity score as the true match to obtain the preliminary matching result of the semantic object landmark.

4. The visual SLAM loop detection method based on semantic topology according to claim 2, characterized in that Evaluate the spatial distribution similarity of the preliminary matching result of the semantic object landmark to eliminate the false matching result and obtain the final matching result of the semantic object landmark, including: Convert the preliminary matching result of the semantic object landmark into the same coordinate system; Evaluate the spatial distribution similarity of the preliminary matching result of the semantic object landmark in the same coordinate system to eliminate the false matching result and obtain the final matching result of the semantic object landmark.

5. The method for loop detection of visual SLAM based on semantic topology according to claim 1, characterized in that Successively perform object-constrained pose optimization and map-point-constrained pose optimization according to the semantic object landmark matching result to obtain the current pose information of the mobile robot, including: Perform rough pose estimation between the semantic object landmark matching result and the target detection result according to the reprojection error to obtain the object-constrained pose optimization result; Refine the pose of the object-constrained pose optimization result according to the map point layer based on the map point landmark to obtain the current pose information of the mobile robot.

6. A visual SLAM loop detection device based on semantic topology, which is used to implement the visual SLAM loop detection method based on semantic topology according to any one of claims 1 to 5, and is characterized in that, Including: An acquisition module, configured to acquire real-time environmental image information during the movement of the mobile robot, and obtain image semantic information and the movement information of the mobile robot according to the real-time environmental image information; A landmark construction module, configured to construct an object landmark according to the image semantic information, and construct a map point landmark according to the movement information of the mobile robot; A map construction module, configured to construct a hierarchical map according to the image semantic information, the object landmark, and the map point landmark, where the hierarchical map includes a semantic layer based on the image semantic information, an object layer based on the object landmark, and a map point layer based on the map point landmark; A matching module, configured to extract the current frame semantic map information according to the real-time environmental image information of the current frame, and perform layer-by-layer matching with the hierarchical map according to the current frame semantic map information to obtain the semantic object landmark matching result; An optimization module, configured to successively perform object-constrained pose optimization and map-point-constrained pose optimization according to the semantic object landmark matching result to obtain the current pose information of the mobile robot.

7. A visual SLAM loop detection system based on semantic topology, characterized in that Including: A mobile robot, an image acquisition device, and the semantic topology-based visual SLAM loop detection device according to claim 6, where the image acquisition device is disposed on the mobile robot, and the image acquisition device is communicatively connected to the semantic topology-based visual SLAM loop detection device; The image acquisition device can acquire real-time environmental image information in real time during the movement of the mobile robot to obtain real-time environmental image information; The semantic topology-based visual SLAM loop detection device can construct an object landmark, a map point landmark, and a hierarchical map according to the real-time environmental image information, extract the current frame semantic map information according to the real-time environmental image information of the current frame, perform layer-by-layer matching with the hierarchical map, and perform object-constrained pose optimization and map-point-constrained pose optimization on the semantic object landmark matching result after layer-by-layer matching to obtain the current pose information of the mobile robot.

Images