A biomimetic SLAM loop closure detection method incorporating semantic information

Abstract

This invention discloses a biomimetic SLAM loop closure detection method that integrates semantic information. The invention uses the YOLOv5 model to identify semantic tags in images, expanding semantic information into additional information for the visual template. During the template matching stage, it comprehensively utilizes pixel information from the image and information such as the type and location of the semantic tags, increasing the amount of information for loop closure detection and effectively improving its accuracy. The use of YOLOv5 to extract semantic tags from images better simulates the human brain's recognition process of its surroundings, facilitating the practical application of biomimetic SLAM. It increases the use of semantic information in image loop closure detection, reduces the impact of perceptual confusion, and improves the stability and environmental adaptability of the biomimetic SLAM algorithm.

Description

Technical Field

[0001] This invention belongs to the field of visual inspection methods, specifically relating to a biomimetic SLAM loop closure detection method that integrates semantic information. Background Technology

[0002] In recent years, the rapid development of brain-perception models has provided improvements for robot simultaneous localization and mapping (SLAM). Loop closure detection is a key component of SLAM systems. Existing brain-perception-based SLAM systems suffer from low loop closure detection accuracy in complex environments. Similar scene appearances in different locations lead to severe loop closure matching errors, hindering accurate relocalization and cumulative error correction for the robot.

[0003] Spatial environmental perception is one of the most important cognitive abilities in animals, enabling them to recognize and navigate in unfamiliar environments. Experimental studies have found that the hippocampus is a key brain region for environmental cognition in animals, containing a series of cells related to spatial environmental cognition. For example, place cells in the CA area (hippocampal horn) generate electricity when the animal is in a specific spatial location, expressing that specific location point and forming the basic mechanism for cognitive maps; head orientation cells in the posterior inferior fossa fire only in a specific head direction, representing the animal's head orientation angle; and grid cells in the entorhinal cortex periodically fire in hexagonal spatial regions, updating the firing fields of place cells by integrating self-motor signals, providing spatial measurements for the formation of cognitive maps. The population activity of these neurons provides a coding form for cognitive space. These early studies on spatial cognition not only contribute to a deeper understanding of how the brain perceives the environment but also provide researchers with new directions for the development of robotic environmental cognition.

[0004] Current detection methods achieve path integration and construct large-scale maps through the collaborative work of connecting HD-by-velocity units and connecting grid-by-velocity units. A brain-derived SLAM model with high biofidelity is proposed, which integrates grid cell models, placement cell models, self-motion signals, and visual information to construct a cognitive map on a mobile robot.

[0005] However, several issues need to be addressed before these biomimetic SLAM models can be applied to complex environments, particularly in loop closure detection. Traditional RatSLAM algorithms use SAD (Sum of Absolute Interpolation) for local scene matching, without geometric processing or feature extraction. The presence of similar scene features in different locations can significantly interfere with visual template recognition and matching, resulting in low reliability and poor environmental adaptability. Research shows that when the human eye observes objects, it sees them as a series of objects, rather than extracting feature points. Therefore, annotating objects in images with semantic information can simulate human eye recognition of various objects in a scene, effectively reducing perceptual confusion and improving matching accuracy. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides a biomimetic SLAM loop closure detection method that integrates semantic information, solving the problems of low accuracy and poor stability in current loop closure detection methods.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a biomimetic SLAM loop closure detection method incorporating semantic information, comprising the following steps:

[0008] S1. Use the Yolov5 network to perform semantic annotation on the image to be matched to obtain the semantic information of the image to be matched;

[0009] S2. Perform coarse matching between the semantic information of the image to be matched and the matching template to obtain a preliminary set of similar matching templates;

[0010] S3. Calculate the comprehensive similarity between the image to be matched and the matching template set based on the semantic and pixel information of the image to be matched and the matching template set.

[0011] S4. Based on the comprehensive similarity between the image to be matched and the matching template set, complete the biomimetic SLAM loop closure detection.

[0012] Further: In S1, the semantic information of the image to be matched specifically refers to all class semantic tags in the image to be matched;

[0013] Each semantic tag includes the semantic tag category, the horizontal coordinate of the center point, the vertical coordinate of the center point, the length value, and the width value.

[0014] The beneficial effects of the above-mentioned further solutions are: the Yolov5 network can find all discriminative markers in an image and has a more stable and accurate matching effect for complex scenes.

[0015] Furthermore: S2 specifically refers to:

[0016] A similarity threshold is set, and the RatSLAM algorithm is used to perform coarse matching with the matching template based on semantic information. Templates with coarse matching similarity values ​​lower than the similarity threshold are deleted, and the set of the remaining templates in the matching template is used as the initial similar matching template set.

[0017] The beneficial effects of the above-mentioned further solutions are as follows: the coarse matching of the present invention can quickly filter out templates with low similarity to the image to be matched, which can reduce matching errors caused by the similarity of two scenes and reduce the workload of matching calculation.

[0018] Further: S3 includes the following steps:

[0019] S31. Obtain the scene description information of the image to be matched based on the semantic information of the image to be matched;

[0020] S32. Delete 100 matching templates that are adjacent to the image to be matched in the matching template set to obtain a sub-template set. Calculate the semantic similarity between the image to be matched and the sub-template set based on the scene description information of the image to be matched.

[0021] S33. Based on the semantic similarity and pixel similarity between the image to be matched and the sub-template set, calculate the comprehensive similarity between the image to be matched and the sub-template set, and use it as the comprehensive similarity between the image to be matched and the matching template set.

[0022] Furthermore, S31 specifically refers to:

[0023] All semantic tags of the image to be matched are converted into scene description information, which includes four scene description sub-information of the top left, bottom left, top right and bottom right of the image.

[0024] Furthermore: in S32, the matching template set includes several templates to be matched;

[0025] S32 specifically involves: deleting 100 matching templates adjacent to the image to be matched from the matching template set to obtain a sub-template set; calculating the semantic similarity between the image to be matched and all matching templates in the sub-template set; wherein, calculating the semantic similarity Vt between the scene description information of the image to be matched a and the semantic similarity Vt between the image to be matched a and the image to be matched b. S The expression for (a, b) is as follows:

[0026]

[0027] In the formula, Let be the semantic similarity between the i-th scene of the image to be matched 'a' and the template to be matched 'b', and its specific expression is:

[0028]

[0029] In the formula, Zi (a) Z is the scene descriptor information of the i-th image a to be matched. i (b) w is the i-th scene description sub-information of the template b to be matched. j Let be the similarity weight of the semantic labels of the j-th class in the image to be matched.

[0030] Furthermore: the similarity weight w of the j-th semantic tag j The specific expression is:

[0031]

[0032] In the formula, c j is the total number of semantic labels of the j-th class in the image to be matched.

[0033] Further: S33 specifically involves: calculating the comprehensive similarity between the image to be matched and the templates to be matched in the sub-template set, obtaining the comprehensive similarity between the image to be matched and the sub-template set, and using it as the comprehensive similarity between the image to be matched and the matching template set;

[0034] The specific method for calculating the comprehensive similarity between the image to be matched a and the template to be matched b is as follows:

[0035] The pixel similarity between the image to be matched and the sub-template set is calculated using the SAD algorithm, and then the comprehensive similarity Vt between the image to be matched and the template b to be matched is calculated using the following formula. error :

[0036] Vt error =α*Vt S (a, b) + β*Vt err

[0037] In the formula, Vt err Let α be the pixel similarity between the image to be matched and the template b to be matched, β be the weight of the semantic similarity, and β be the weight parameter of the pixel similarity.

[0038] Furthermore: S4 specifically refers to:

[0039] Set a matching threshold to determine whether the overall similarity between the image to be matched and the matching template set is less than the matching threshold. If yes, the matching result between the image to be matched and the matching template set is a closed loop; otherwise, the matching result between the image to be matched and the matching template set is not a closed loop. Based on the matching result between the image to be matched and the matching template set, complete the biomimetic SLAM loop closure detection.

[0040] Furthermore: when the matching result between the image to be matched and the matching template set is a closed loop, the matching template with the highest comprehensive similarity between the image to be matched and the matching template is selected as the closed loop template.

[0041] The beneficial effects of this invention are as follows:

[0042] (1) This invention expands semantic information into additional information of visual templates. By using the semantic information of images to perform loop closure detection, the types and positions in semantic labels are compared in the template matching stage, which increases the amount of information for loop closure detection. This can effectively reduce the impact of perceptual confusion, improve the accuracy of loop closure detection, and make the improved RatSLAM algorithm have better stability and environmental adaptability. This has important theoretical and practical significance.

[0043] (2) This invention provides a general method to reduce the impact of perceptual confusion. Using YOLOv5 to extract semantic labels from images can better simulate the human brain’s recognition process of the scene, which is helpful for the application of biomimetic SLAM in practice. It has been applied in a hippocampus-entorhinolaryngology-based SLAM system. Experimental results show that the proposed method can effectively improve the accuracy and stability of loop closure detection. Attached Figure Description

[0044] Figure 1 This is a flowchart of a biomimetic SLAM loop closure detection method that integrates semantic information according to the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0046] like Figure 1 As shown, in one embodiment of the present invention, a biomimetic SLAM loop closure detection method incorporating semantic information includes the following steps:

[0047] S1. Use the Yolov5 network to perform semantic annotation on the image to be matched to obtain the semantic information of the image to be matched;

[0048] S2. Perform coarse matching between the semantic information of the image to be matched and the matching template to obtain a preliminary set of similar matching templates;

[0049] S3. Calculate the comprehensive similarity between the image to be matched and the matching template set based on the semantic and pixel information of the image to be matched and the matching template set.

[0050] S4. Based on the comprehensive similarity between the image to be matched and the matching template set, complete the biomimetic SLAM loop closure detection.

[0051] In S1, the semantic information of the image to be matched specifically refers to all class semantic tags in the image to be matched;

[0052] Each semantic tag includes the semantic tag category, the horizontal coordinate of the center point, the vertical coordinate of the center point, the length value, and the width value.

[0053] In this embodiment, the YOLOv5 network can identify all discriminative markers in an image, achieving more stable and accurate matching results for complex scenes. The image to be matched is input into the YOLOv5 network, resulting in a semantic region S = {s1, ... s...} with pixel labels. n}, where s n For semantic labels, n is the total number of categories; s n =[type n x n y n w n h n ], where type, x, y, w, and h are the category, x-coordinate of the center point, y-coordinate of the center point, length, and width of the semantic tag, respectively.

[0054] To facilitate the use of semantic information in loop closure detection, the semantic label S n With visual template V n and pose cells P n Related, to obtain experience e n ;

[0055] e n ={S n V n P n p n}

[0056] In the formula, p n For experience e n Experience the location on the map.

[0057] Specifically, S2 is:

[0058] A similarity threshold is set, and the RatSLAM algorithm is used to perform coarse matching with the matching template based on semantic information. Templates with coarse matching similarity values ​​lower than the similarity threshold are deleted, and the set of the remaining templates in the matching template is used as the initial similar matching template set.

[0059] Since the original RatSLAM algorithm has a large computational workload when using SAD to match the template with the image to be matched pairwise, this invention sets a similarity threshold to quickly filter out templates with low similarity to the image to be matched, and sets it not to match with the nearest 100 templates, so as to reduce matching errors caused by the similarity of the two scenes.

[0060] S3 includes the following steps:

[0061] S31. Obtain the scene description information of the image to be matched based on the semantic information of the image to be matched;

[0062] S32. Delete 100 matching templates that are adjacent to the image to be matched in the matching template set to obtain a sub-template set, so as to avoid the image to be matched and the neighboring templates forming a closed loop. Based on the scene description information of the image to be matched, calculate the semantic similarity between the image to be matched and the sub-template set.

[0063] S33. Based on the semantic similarity and pixel similarity between the image to be matched and the sub-template set, calculate the comprehensive similarity between the image to be matched and the sub-template set, and use it as the comprehensive similarity between the image to be matched and the matching template set.

[0064] Specifically, S31 is:

[0065] All semantic tags of the image to be matched are converted into scene description information, which includes four scene description sub-information of the top left, bottom left, top right and bottom right of the image.

[0066] In this embodiment, the image to be matched is divided into upper left, lower left, upper right, and lower right regions, with the center point of the image as the origin. Each region is used as a scene descriptor sub-information. For any semantic tag s in the image to be matched... i Based on its central coordinates, it is classified into one of the four scene description sub-information.

[0067] Scene description information U = [Z 1 Z 2 Z 3 Z 4 ], where Z 1 Z 2 Z 3 Z 4 These are the top-left scene description sub-information, bottom-left scene description sub-information, top-right scene description sub-information, and bottom-right scene description sub-information, respectively. The l-th scene description sub-information Z l =[z1z2...z q ], where l = 1, ..., 4, z q Let U be the total number of semantic tags of type q appearing in scene l. Each column of U corresponds to a semantic tag, and each row corresponds to the number of semantic tags of each type in the corresponding area.

[0068] In step S32, the matching template set includes several templates to be matched;

[0069] S32 specifically involves: deleting 100 neighboring templates from the matching template set to obtain a sub-template set, thus avoiding loop detection between the image to be matched and its neighboring templates; calculating the semantic similarity between the image to be matched and all templates in the sub-template set, wherein the semantic similarity Vt between the scene description information of image a and template b is calculated. S The expression for (a, b) is as follows:

[0070]

[0071] In the formula, Let be the semantic similarity between the i-th scene of the image to be matched 'a' and the template to be matched 'b', and its specific expression is:

[0072]

[0073] In the formula, Z i (a) Z is the scene descriptor information of the i-th image a to be matched. i (b) w is the i-th scene description sub-information of the template b to be matched. j Let be the similarity weight of the semantic labels of the j-th class in the image to be matched.

[0074] The similarity weight w of the j-th semantic tag j The specific expression is:

[0075]

[0076] In the formula, c j is the total number of semantic labels of the j-th class in the image to be matched.

[0077] In this embodiment, to avoid the very few semantic tags obtained from target detection from affecting scene matching, the similarity weight of such semantic tags is set to 0; to avoid the large number of semantic tags obtained from target detection from interfering with the closed-loop judgment, the similarity weight of such semantic tags is suppressed, so that the more numerous the tags, the lower their similarity weight.

[0078] Specifically, S33 involves: calculating the comprehensive similarity between the image to be matched and the templates to be matched in the sub-template set, obtaining the comprehensive similarity between the image to be matched and the sub-template set, and using it as the comprehensive similarity between the image to be matched and the matching template set;

[0079] The specific method for calculating the comprehensive similarity between the image to be matched a and the template to be matched b is as follows:

[0080] The pixel similarity between the image to be matched and the sub-template set is calculated using the SAD algorithm, and then the comprehensive similarity Vt between the image to be matched and the template b to be matched is calculated using the following formula. error :

[0081] Vt error =α*Vt S (a, b) + β*Vt err

[0082] In the formula, Vt err Let α be the pixel similarity between the image to be matched and the template b to be matched, β be the weight of the semantic similarity, and β be the weight parameter of the pixel similarity.

[0083] Specifically, S4 is:

[0084] Set a matching threshold to determine whether the overall similarity between the image to be matched and the matching template set is less than the matching threshold. If yes, the matching result between the image to be matched and the matching template set is a closed loop; otherwise, the matching result between the image to be matched and the matching template set is not a closed loop. Based on the matching result between the image to be matched and the matching template set, complete the biomimetic SLAM loop closure detection.

[0085] When the matching result between the image to be matched and the matching template set is a closed loop, the matching template with the highest comprehensive similarity between the image to be matched and the matching template is selected as the closed loop template.

[0086] The beneficial effects of this invention are as follows: This invention expands semantic information into additional information for visual templates. By using the semantic information of images for loop closure detection, and comparing the types and positions in semantic labels during the template matching stage, the amount of information for loop closure detection is increased. This can effectively reduce the impact of perceptual confusion, improve the accuracy of loop closure detection, and make the improved RatSLAM algorithm have better stability and environmental adaptability. This has important theoretical and practical significance.

[0087] This invention provides a general method to reduce the impact of perceptual confusion. Using YOLOv5 to extract semantic labels from images can better simulate the human brain's recognition process of the scene, which is helpful for the practical application of biomimetic SLAM. It has been applied in a hippocampus-entorhinal SLAM system. Experimental results show that the proposed method can effectively improve the accuracy and stability of loop closure detection.

[0088] In the description of this invention, it should be understood that the terms "center," "thickness," "upper," "lower," "horizontal," "top," "bottom," "inner," "outer," and "radial," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include one or more of that feature.

Claims

1. A biomimetic SLAM loop closure detection method incorporating semantic information, characterized in that, Includes the following steps: S1. Use the Yolov5 network to perform semantic annotation on the image to be matched to obtain the semantic information of the image to be matched; S2. Perform coarse matching between the semantic information of the image to be matched and the matching template to obtain a preliminary set of similar matching templates; S3. Calculate the comprehensive similarity between the image to be matched and the matching template set based on the semantic and pixel information of the image to be matched and the matching template set. S3 includes the following steps: S31. Obtain the scene description information of the image to be matched based on the semantic information of the image to be matched; S32. Delete 100 matching templates that are adjacent to the image to be matched in the matching template set to obtain a sub-template set. Calculate the semantic similarity between the image to be matched and the sub-template set based on the scene description information of the image to be matched. In step S32, the matching template set includes several templates to be matched; S32 specifically involves: deleting 100 matching templates from the matching template set that are adjacent to the image to be matched to obtain a sub-template set; calculating the semantic similarity between the image to be matched and all matching templates in the sub-template set; wherein, calculating the semantic similarity between the image to be matched and all matching templates in the sub-template set. a Scene description information and template to be matched b semantic similarity The specific expression is: In the formula, Image to be matched a Matching template b The i The semantic similarity of a scene is expressed as follows: In the formula, Image to be matched a The i Each scene description sub-information Template to be matched b The i Each scene description sub-information For the first image to be matched Similarity weights for semantic tags; S33. Based on the semantic similarity and pixel similarity between the image to be matched and the sub-template set, calculate the comprehensive similarity between the image to be matched and the sub-template set, and use it as the comprehensive similarity between the image to be matched and the matching template set. Specifically, S33 involves: calculating the comprehensive similarity between the image to be matched and the templates to be matched in the sub-template set, obtaining the comprehensive similarity between the image to be matched and the sub-template set, and using it as the comprehensive similarity between the image to be matched and the matching template set; Among them, the image to be matched is calculated. a Matching template b The comprehensive similarity method is as follows: The pixel similarity between the image to be matched and the sub-template set is calculated using the SAD algorithm, and then the similarity between the image to be matched and the sub-template set is calculated using the following formula. b Overall similarity : In the formula, Image to be matched and template to be matched b Pixel similarity, Weights for semantic similarity These are the weighting parameters for pixel similarity; S4. Based on the comprehensive similarity between the image to be matched and the matching template set, complete the biomimetic SLAM loop closure detection.

2. The biomimetic SLAM loop closure detection method incorporating semantic information according to claim 1, characterized in that, In S1, the semantic information of the image to be matched specifically refers to all class semantic tags in the image to be matched; Each semantic tag includes the semantic tag category, the horizontal coordinate of the center point, the vertical coordinate of the center point, the length value, and the width value.

3. The biomimetic SLAM loop closure detection method incorporating semantic information according to claim 1, characterized in that, Specifically, S2 is: A similarity threshold is set, and the RatSLAM algorithm is used to perform coarse matching with the matching template based on semantic information. Templates with coarse matching similarity values ​​lower than the similarity threshold are deleted, and the set of the remaining templates in the matching template is used as the initial similar matching template set.

4. The biomimetic SLAM loop closure detection method incorporating semantic information according to claim 1, characterized in that, Specifically, S31 is: All semantic tags of the image to be matched are converted into scene description information, which includes four scene description sub-information of the top left, bottom left, top right and bottom right of the image.

5. The biomimetic SLAM loop closure detection method incorporating semantic information according to claim 1, characterized in that, The first Similarity weight of semantic tags The specific expression is: In the formula, For the first image to be matched The total number of semantic tags.

6. The biomimetic SLAM loop closure detection method incorporating semantic information according to claim 1, characterized in that, Specifically, S4 is: Set a matching threshold to determine whether the overall similarity between the image to be matched and the matching template set is less than the matching threshold. If yes, the matching result between the image to be matched and the matching template set is a closed loop; otherwise, the matching result between the image to be matched and the matching template set is not a closed loop. Based on the matching result between the image to be matched and the matching template set, complete the biomimetic SLAM loop closure detection.

7. The biomimetic SLAM loop closure detection method incorporating semantic information according to claim 6, characterized in that, When the matching result between the image to be matched and the matching template set is a closed loop, the matching template with the highest comprehensive similarity between the image to be matched and the matching template is selected as the closed loop template.

Images