Method, system, and storage medium for detecting multi-robot loops based on dual-mode verification

Abstract

This invention provides a multi-robot loop detection method, system, and storage medium based on dual-mode verification that reliably and accurately identifies loops in dynamic environments and eliminates redundant loops. [Solution] The multi-robot loop detection method includes the steps of: constructing a multi-task deep semantic augmentation network that extracts semantic and geometric features of an image based on deep learning; distinguishing dynamic / static regions in the image based on the extracted semantic features, assigning a low weight to the dynamic region, calculating similarity based on the weighted region, and performing initial fusion accordingly; selecting robot-to-robot loop candidates that satisfy geometric consistency, semantic similarity, and spatiotemporal continuity from the initial fused image; and constructing a collision graph based on the robot-to-robot loop candidates selected in step S3, selecting the largest independent set, and removing redundant loops.

Description

[Technical Field]

[0001] The present invention relates to synchronous position estimation and map construction for robots, and more particularly to a multi-robot loop detection method, system, and storage medium based on dual-mode verification. [Background technology]

[0002] Simultaneous Localization and Mapping (SLAM), a collaborative multi-robot approach, can improve work efficiency and is applicable to large-scale and complex scenarios such as urban canyons, underground spaces, and large indoor environments. In multi-robot collaborative SLAM, the accuracy and robustness of loop detection directly affect the map fusion quality and trajectory optimization effect. However, conventional techniques have the following significant drawbacks.

[0003] Existing methods rely on extracted geometric features for feature fusion and matching, but these are often ineffective in dynamic scenarios because geometric features are sensitive to changes in lighting and viewpoints. This leads to a decrease in the success rate of inter-viewpoint matching in dynamic scenarios, resulting in reduced robustness of subsequent loop detection. Existing loop detection methods are insufficient in accuracy because they over-rely on image similarity and ignore spatiotemporal constraints. Furthermore, in multi-robot collaborative SLAM, the number of loop candidates increases significantly compared to single-robot SLAM, resulting in a large number of redundant loop candidates. Without optimization, this significantly increases the overhead of backend graph optimization. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The object of the present invention is to provide a multi-robot loop detection method, system, and storage medium based on dual-mode verification that can reliably and accurately identify loops in a dynamic environment and eliminate redundant loops.

Means for Solving the Problem

[0005] A multi-robot loop detection method based on dual-mode verification described in the present invention, Step S1 of constructing a multi-task deep semantic extension network for extracting semantic features and geometric features of an image based on deep learning; Step S2 of distinguishing dynamic and static regions in the image based on the extracted semantic features, assigning a lower weight to the dynamic region, calculating the similarity for the multi-robot image based on the weighted region, and performing initial fusion; Step S3 of selecting inter-robot loop candidates that satisfy all of geometric consistency, semantic similarity, and spatio-temporal continuity from the initial fusion image of the multi-robot; Characterized by including step S4 of constructing a collision graph based on the inter-robot loop candidates selected in step S3, selecting a maximum independent set, and removing redundant loops.

[0006] Based on the above technical solution, extract the semantic features of the image via the constructed multi-task deep semantic extension network, distinguish the dynamic and static regions in the image, and assign a lower weight to the dynamic region, thereby avoiding interference by the dynamic region when performing viewpoint matching and improving the accuracy of matching. Also, on this basis, select loop candidates based on geometric consistency, semantic similarity, and spatio-temporal continuity. By considering not only the image similarity, i.e., semantic similarity, but also geometric consistency and spatio-temporal continuity, the accuracy of loop detection can be improved. Finally, in the selected loop candidates, construct a collision graph, select a maximum independent set, and use it as the finally detected loop, thereby removing a large number of redundant loops and reducing the overhead of back-end graph optimization.

[0007] Preferably, the step S4 is Substep S4.1 involves constructing a collision graph of loop candidate nodes based on the nodes within the robot-to-robot loop candidates selected in step S3, and selecting the largest set of independent nodes. The substep S4.2 includes constructing loop candidate edges based on the loop candidate nodes selected in step S4.1, constructing a collision graph of the loop candidate edges accordingly, selecting the largest independent set, and finally setting the loop candidate edges in the largest independent set as the detected inter-robot loop edges.

[0008] The collision graph is constructed twice, and selection is performed in stages. In the first stage, a collision graph of loop candidate nodes is constructed, and loop candidate nodes without collisions are selected to the maximum extent possible, i.e., redundant loop candidate nodes are removed. Based on this foundation, the collision graph of loop candidate edges constructed in the second stage is used to select loop edges without collisions to the maximum extent possible, which are the final detected loop edges, and redundant loop edges are further removed. Through these two processing steps, redundant loops can be significantly reduced, and the efficiency of backend graph optimization can be improved.

[0009] A multi-robot loop detection system based on dual-mode verification as described in the present invention, A network construction module for building a multitask deep semantic augmentation network to extract semantic and geometric features of images based on deep learning, An initial fusion module for performing initial fusion, which distinguishes dynamic and static regions within an image based on extracted semantic features, assigns lower weights to dynamic regions, calculates similarity based on weighted regions for multi-robot images, and performs initial fusion. An initial selection module for selecting candidate inter-robot loops that satisfy geometric consistency, semantic similarity, and spatiotemporal continuity from initial fused images of multiple robots, The system comprises a final selection module that constructs a collision graph based on candidate robot loops selected by an initial selection module, selects the largest set of independent loops, and removes redundant loops.

[0010] A computer-readable storage medium storing one or more programs described in the present invention, comprising one or more program instructions, wherein when the instructions are executed by a computer, the computer is instructed to perform the method described in any one of the above items. [Effects of the Invention]

[0011] By collaboratively learning semantic and geometric features via a multi-task deep semantic augmentation network and assigning lower weights to dynamic regions based on extracted semantic features, interference from dynamic regions is avoided, improving the robustness and success rate of inter-viewpoint matching. By selecting loop candidates based on geometric consistency, semantic similarity, and spatiotemporal continuity, the bottleneck of error loops based on conventional image similarity is overcome, significantly improving the accuracy of loop detection. By constructing a system that selects the largest independent set based on collision graphs, numerous redundant loops can be eliminated, ensuring detection accuracy while reducing the overhead of backend graph optimization. [Brief explanation of the drawing]

[0012] [Figure 1] This is a flowchart of the method. [Figure 2] This is a schematic diagram of the structure of a multitask deep semantic augmentation network. [Figure 3] This is a flowchart of the initial fusion module according to the present invention. [Figure 4] This is a flowchart of the initial sorting module according to the present invention. [Figure 5] This is a flowchart of the final sorting module according to the present invention. [Modes for carrying out the invention]

[0013] As shown in the figure, the multi-robot loop detection method based on dual-mode verification described in the present invention includes the following steps.

[0014] S1. Construct a multi-task deep semantic augmentation network based on deep learning to extract semantic and geometric features from images.

[0015] The aforementioned multitask deep semantic extension network employs a shared encoder (shared coding network) and a dual-branch structure, where the dual-branch structure consists of a semantic decoding network and a geometric feature decoding network, respectively. The shared coding network consists of three sequentially connected convolutional layers, each layer comprising a 3x3 convolutional kernel, a stride of 2, padding of 1, batch normalization (BatchNorm), and a ReLU activation function. During the coding process, downsampling is achieved by the convolutional stride, and the backbone coded output (256x60x80) is shared between the semantic decoding network and the geometric feature decoding network.

[0016] The semantic decoding network consists of three sequentially connected transposed convolutional layers, with a structure comprising a 3x3 convolutional kernel, a stride of 2, padding of 1, batch normalization (BatchNorm), and a ReLU activation function. The final output implicitly implements Softmax classification through a cross-entropy loss function.

[0017] The geometric feature decoding network consists of an adaptive mean pooling layer and a fully connected layer. The adaptive mean pooling layer compresses a 256×60×80 feature map to 256×1×1, and the fully connected layer transforms the 256-dimensional input into a 128-dimensional output before outputting a geometric descriptor via the ReLU activation function.

[0018] After inputting RGB images into a multitask deep semantic augmentation network, deep visual features are extracted using a shared encoder and input into a semantic decoding network and a geometric feature decoding network, respectively. These are then used to generate semantic predictions and 128-dimensional global descriptors, which are used for loop matching and graph optimization.

[0019] To achieve co-optimization of two tasks—a semantic task and a geometric task—a combined loss function is introduced to train a multi-task deep semantic extension network. Specifically, this is done as follows:

[0020] In a semantic decoding network, pixel-level monitoring is performed using standard cross-entropy loss, L sem The following calculation formula was adopted:

number

number

[0021] In a geometric feature decoding network, feature optimization is performed using triplet margin loss, and descriptor discriminability is optimized by constructing (anchor, positive, negative) triplets. geom The following calculation formula was adopted:

number

[0022] The final combined loss function of the multi-task deep semantic extension network is expressed by the following expression, and is used to optimize the accuracy of semantic prediction and the discriminability of geometric descriptors simultaneously.

Number

[0023] S2. Based on the extracted semantic features, distinguish the dynamic and static regions in the image, assign lower weights to the dynamic regions, calculate the similarity for the multi-robot images based on the weighted regions, and perform initial fusion.

[0024] Conventional feature matching methods are prone to a large number of false matches under the interference from moving objects in dynamic scenarios. The semantically induced local matching mechanism distinguishes static and dynamic regions by introducing semantic information, assigns different attention weights to different regions, thereby suppressing false matches in dynamic regions.

[0025] The core idea of this mechanism is as follows. Use the prediction results of the semantic decoding network to identify moving objects, attenuate the feature matching weights of dynamic regions, that is, generate semantic attention weights, assign lower matching weights to dynamic regions, maintain high weights in static regions, and generate an attention mask.

[0026] As the semantic attention weight, the following calculation formula is adopted.

number

[0027] When calculating feature similarity, semantic attention weights are considered to mitigate the influence of dynamic regions on the matching results. When calculating cosine similarity between different images, semantic attention weights are also considered, specifically using the following formula:

number

[0028] By performing initial fusion of multiple robot images based on cosine similarity that takes semantic attention weights into account, interference from dynamic regions can be avoided, improving the robustness and success rate of inter-viewpoint matching. Furthermore, by performing subsequent loop detection based on these results, the robustness of loop detection can also be improved.

[0029] S3. From the initial fusion images of the multi-robot system, we select candidate inter-robot loops that satisfy all of the following criteria: geometric consistency, semantic similarity, and spatiotemporal continuity.

[0030] In loop detection, mismatching of loop edges can cause divergence in graph optimization. Conventional techniques are overly reliant on image similarity, so it is necessary to select valid loop candidates using multidimensional constraints to reduce loop misselection. Specifically, selection is performed in the following three dimensions.

[0031] (1) The consistency of the spatial distribution of feature points is evaluated using the Mahalanobis distance, and geometric outliers caused by noise are excluded. The Mahalanobis distance between loop frames of different robots is calculated using the following formula.

number

[0032] D M (x,y)≦τ g In the case of τ g This is the geometric threshold corresponding to the Mahalanobis distance, that is, D M If (x,y) is below the corresponding threshold, the spatial distribution consistency is satisfied, i.e., the geometric consistency is satisfied.

[0033] (2) The cosine similarity of the deep features is used to verify the consistency of the scenarios from the perspective of the robots, i.e., semantic similarity. The following formula is used as the cosine similarity:

number

[0034] CosSim(f a ,f b )≧0.6τ s In the case of τ s This is the semantic similarity threshold, that is, CosSim(f a ,f b ) is above the corresponding threshold (0.6τ s τ in s If the coefficient before it can be adjusted according to the actual situation, then it is considered to satisfy semantic similarity.

[0035] (3) If the following determination formula is satisfied, it is determined that the loop frame of a different robot or each k frame before and after it (k is a constant and can be set to a different value depending on the actual situation) satisfies spatiotemporal continuity.

number

number

[0036] N a and N b These are the set of loop frame a and the k frames before and after it, and the set of loop frame b and the k frames before and after it, respectively. Specifically, the following expression is adopted:

number

[0037] Only loops that simultaneously satisfy the constraints of the three dimensions mentioned above are considered loop candidates.

[0038] The pairing process based on the above determination formula primarily involves first extracting the global feature f of each frame within the window, then calculating the similarity of different frame pairs, selecting candidate frame pairs that satisfy the conditions, and from these, selecting a frame pair that satisfies one strong match and the overall matching quality to constitute a set of valid pairs. Furthermore, if the number of elements in this set is ≥ 2, loop frames of two different robots that satisfy these conditions are considered to satisfy spatiotemporal continuity.

[0039] S4. Based on the robot-to-robot loop candidates selected in step S3, construct a collision graph, select the largest set of independent loops, and eliminate redundant loops.

[0040] S4.1. Based on the nodes within the robot-to-robot loop candidates selected in step S3, a collision graph of loop candidate nodes is constructed, and the largest set of independent nodes is selected.

[0041] The following expression is used as the collision graph for loop candidate nodes: G=(V,E) Here, G is the collision graph of loop candidate nodes, V is the set of vertices in the collision graph, and its expression is

number

number

[0042] S4.2. Based on the loop candidate nodes selected in step S4.1, loop candidate edges are constructed, a collision graph of the loop candidate edges is constructed accordingly, the largest independent set is selected, and the loop candidate edges in that largest independent set are ultimately the detected inter-robot loop edges.

[0043] The following expression is used as the collision graph for candidate loop edges: G c =(V c ,E c ) Here is the G c This is the collision graph of loop candidate edges, V c is the set of vertices in the collision graph, and its expression is

number

number

[0044] The purpose of selecting the Largest Independent Set (MIS) is to find the collision-free subset with the most vertices in a collision graph. At its core, this involves avoiding collisions between loop candidates (e.g., spatial overlap, temporal redundancy, geometric contradictions, etc.) while preserving as many effective loops as possible. In sections S4.1 and S4.2, the MIS is selected using a greedy algorithm, but it is of course possible to select the MIS using other algorithms as well.

[0045] A multi-robot loop detection system based on dual-mode verification as described in the present invention, A network construction module for building a multitask deep semantic augmentation network to extract semantic and geometric features of images based on deep learning, An initial fusion module for performing initial fusion, which distinguishes dynamic and static regions within an image based on extracted semantic features, assigns lower weights to dynamic regions, calculates similarity based on weighted regions for multi-robot images, and performs initial fusion. An initial selection module for selecting candidate inter-robot loops that satisfy geometric consistency, semantic similarity, and spatiotemporal continuity from initial fused images of multiple robots, The system comprises a final selection module that constructs a collision graph based on candidate robot loops selected by an initial selection module, selects the largest set of independent loops, and removes redundant loops.

[0046] A computer-readable storage medium storing one or more programs described in the present invention, comprising one or more program instructions, wherein when the instructions are executed by a computer, the computer is instructed to perform the method described in any one of the above items.

Claims

1. A multi-robot loop detection method based on dual-mode verification, Step S1 involves constructing a multitask deep semantic augmentation network for extracting semantic and geometric features of an image based on deep learning, Step S2 involves distinguishing between dynamic and static regions in the image based on extracted semantic features, assigning lower weights to dynamic regions, calculating similarity based on the weighted regions for the multi-robot image, and performing initial fusion. Step S3 involves selecting candidate inter-robot loops from initial fused images of multiple robots that satisfy geometric consistency, semantic similarity, and spatiotemporal continuity. A multi-robot loop detection method based on dual-mode verification, characterized by comprising: step S4: constructing a collision graph based on the inter-robot loop candidates selected in step S3, selecting the largest set of independent loops, and removing redundant loops.

2. The multi-robot loop detection method based on dual-mode verification according to claim 1, characterized in that, in step S1, the multi-task deep semantic augmentation network employs a shared encoder and a dual-branch structure, the shared encoder includes three sequentially connected convolutional layers, a batch normalization layer and a ReLU activation function, the dual-branch structure is a semantic decoding network and a geometric feature decoding network, the structure of the semantic decoding network is the same as that of the shared encoder and the geometric feature decoding network includes sequentially connected adaptive mean pooling layers and fully connected layers.

3. In step S1, the following formula is adopted as the loss function for the multi-task deep semantic augmentation network: [Number 16] Here, L ｔｏｔａｌ λ is the loss function of a multitask deep semantic augmentation network, where λ is a hyperparameter, and L ｓｅｍ and L ｇｅｏｍ These are the loss functions for the semantic decoding network and the geometric feature decoding network, respectively. L ｓｅｍ The following calculation formula was adopted: [Number 17] Here, N is the total number of pixels, and C is the number of categories. [Number 18] These are the true label and predicted value of the i-th pixel, which belongs to category c, respectively. L ｇｅｏｍ The following calculation formula was adopted: [Number 19] Here, T is the number of triplets, and f ａ ｉ represents the feature of the i-th set of anchor images, and f ｐ ｉ represents the feature of the i-th set of positive samples, and f ｎ ｉ represents the feature of the i-th set of negative samples, and α is an interval constant. The multi-robot loop detection method based on dual-mode verification according to claim 2 is characterized by this.

4. The multi-robot loop detection method based on dual-mode verification according to claim 1, characterized in that in step S3, suitability to the loop conditions is determined on the condition that the Mahalanobis distance between loop frames of different robots is less than or equal to a corresponding threshold, the cosine similarity between loop frames of different robots is greater than or equal to a corresponding threshold, and the cosine similarity between a loop frame of a different robot or multiple frames before and after it exceeds a corresponding threshold.

5. The following formula was adopted as the Mahalanobis distance: [Number 20] Here, D Ｍ (x, y) is the Mahalanobis distance, and x and y are the spatial coordinate vectors of feature points in loop frames of different robots, Σ －１ This is an information matrix, The following formula was adopted for calculating cosine similarity: [Number 21] Here, CosSim(f ａ , f ｂ ) is the cosine similarity, and f ａ and f ｂ These are the global geometric feature vectors of loop frames a and b, respectively, and loop frames a and b belong to different robots. The following determination formula is adopted for spatiotemporal continuity: [Number 22] Here, τ ｓ is the semantic similarity threshold, N ａ and N ｂ These are loop frame a and a set of multiple frames before and after it, and loop frame N. ｂ This is a set of multiple frames before and after it, and specifically, the following expression is adopted: [Number 23] Here, v ｉ i is the loop frame, and v ｉ－ｋ and v ｉ＋ｋ The multi-robot loop detection method based on dual-mode verification according to claim 4, characterized in that the kth frames are the kth frames before and after loop frame i, respectively, and k is a set constant.

6. Step S4 is, Substep S4.1 involves constructing a collision graph of loop candidate nodes based on the nodes within the robot-to-robot loop candidates selected in step S3, and selecting the largest set of independent nodes. A multi-robot loop detection method based on dual-mode verification according to claim 1, comprising: substep S4.2: constructing loop candidate edges based on the loop candidate nodes selected in step S4.1, constructing a collision graph of loop candidate edges accordingly, selecting the largest independent set, and finally setting the loop candidate edges in the largest independent set as the detected inter-robot loop edges.

7. In step S4.1, the following expression is adopted as the collision graph of loop candidate nodes: G = (V, E) Here, G is the collision graph of loop candidate nodes, V is the set of vertices in the collision graph, and its expression is [Number 24] and l ｎ is the nth loop candidate node, where n is a constant, E is the set of edges in the collision graph, and its expression is [Number 25] And here, Overlap(l ｉ ,l ｊ ) is loop l ｉ and l ｊ The ratio of shared feature points, τ ｃｏｎｆｌｉｃｔ The multi-robot loop detection method based on dual-mode verification according to claim 6, wherein is a collision threshold.

8. In step S4.2, the following expression is adopted as the collision graph of the loop candidate edges: G ｃ =(@ ｃ ,E ｃ ) Here, G ｃ This is the collision graph of loop candidate edges, V ｃ is the set of vertices of the collision graph, and its expression is [Number 26] and e ｎ is the nth loop candidate edge, where n is a constant, and E ｃ is the set of edges of the collision graph, and its expression is [Number 27] And here, T(e ｉ， e ｊ ), S(e ｉ， e ｊ ), and G(e ｉ， e ｊ ) are each loop candidate edge e ｉ and e ｊ A multi-robot loop detection method based on dual-mode verification according to claim 6, characterized in that it shows temporal collisions, spatial collisions, and geometric inconsistencies between them.

9. A multi-robot loop detection system based on dual-mode verification, A network construction module for building a multitask deep semantic augmentation network to extract semantic and geometric features of images based on deep learning, An initial fusion module for performing initial fusion, which distinguishes dynamic and static regions within an image based on extracted semantic features, assigns lower weights to dynamic regions, calculates similarity based on weighted regions for multi-robot images, and performs initial fusion. An initial selection module for selecting candidate inter-robot loops that satisfy geometric consistency, semantic similarity, and spatiotemporal continuity from initial fused images of multiple robots, A multi-robot loop detection system based on dual-mode verification, comprising: a final selection module for constructing a collision graph based on inter-robot loop candidates selected by an initial selection module, selecting the largest set of independent loops, and eliminating redundant loops.

10. A computer-readable storage medium in which one or more programs are stored, A computer-readable storage medium comprising one or more program instructions, wherein when the instructions are executed by a computer, the computer is instructed to perform the method described in any one of claims 1 to 8.

Images