Loopback detection method for visual SLAM (Simultaneous Localization and Mapping)

Through the combination of deep learning network and bag-of-word model, the problems of dynamic object influence and semantic information neglect in visual SLAM loop detection are solved, the detection accuracy and robustness are improved, and it is suitable for fields such as autonomous driving.

CN119206378AInactive Publication Date: 2024-12-27安徽中科星驰自动驾驶技术有限公司
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411696809.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-26
Publication Date
2024-12-27
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing SLAM loopback detection based on visual information is susceptible to dynamic objects, neglecting semantic information, resulting in insufficient detection accuracy.

Method used

Use deep neural network yolov9 to classify each frame of pictures, combine semantic model and geometric model to establish a constraint model, eliminate dynamic objects through weighted feature matching, integrate the bag of words model to calculate semantic similarity score, and combine it with geometric feature similarity to calculate the final loop Test score.

Benefits of technology

It improves the reliability and accuracy of feature matching accuracy and loop detection, enhances the stability of graph construction, and has a wide range of application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119206378A_ABST
    Figure CN119206378A_ABST
Patent Text Reader

Abstract

The invention is applicable to the technical field of loopback detection of visual SLAM, and provides a loopback detection method for visual SLAM, which comprises the following steps: S1, classifying objects in each frame of picture by using a deep neural network yolov9, and allocating semantic categories on the objects to features on the objects; s2, a constraint model F is established by combining the semantic model and the geometric model, and a normal feature vector of the constraint model F comprises semantic tags of poses and object classes; according to the method, semantic information is added through a deep learning network on the basis of an existing method at the present stage, the accuracy of feature matching is improved to obtain better geometric feature loopback similarity, a semantic weight can be provided during loopback detection, the robustness and accuracy of mapping are improved, and in addition, the method can be applied to the field of mapping. The semantic information adding design provided by the invention is very ingenious, is not restricted by a specific environment, and can be easily transplanted to other systems, so that the application range is very wide.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of loop closure detection of visual SLAM, and in particular relates to a loop closure detection method for visual SLAM. Background Art

[0002] SLAM is widely used in many fields such as autonomous driving and smart home. SLAM mapping based on visual information is a typical working condition. Although visual information is easily disturbed by external environment such as rain, snow, dust, etc., it is low-cost and relatively mature in technology.

[0003] In the prior art, SLAM loop detection based on visual information is often affected by dynamic objects and ignores the semantic information of objects. Generally speaking, the fewer dynamic objects, the more accurate the loop effect. However, many dynamic objects often appear in loop detection and the semantic information of objects is ignored, which will cause the failure of loop detection or insufficient accuracy. In this regard, the present invention provides a loop detection method for visual SLAM to solve the above technical problems. Summary of the invention

[0004] The object of the present invention is to provide a loop detection method for visual SLAM, aiming to solve the problems raised in the background technology.

[0005] The present invention is achieved by a loop detection method for visual SLAM, the method comprising: S1: Use the deep neural network yolov9 to classify the objects in each frame and assign the semantic categories of these objects to the features of these objects; S2: Combine the semantic model and geometric model to establish a constraint model F, whose normal feature vector contains the semantic label of the pose and object class:

[0006] Where k is the geometric weight, (1-k) is the semantic weight, and in feature matching, the matching of feature points on objects of static semantic categories is weighted and the matching of feature points in dynamic objects is eliminated; G refers to the geometric model, and ,in is the registration information of the extracted features, T is the total number of feature point pairs extracted; S refers to the semantic model, and ,in Represents the semantic information of the label of the object detected by yolov9, and T is the total number of objects detected; S3: Incorporating semantic category information into the common method based on feature descriptors and bag-of-words models in visual SLAM loop closures. First, generate a bag-of-words vector. Since different words have different weights in similarity descriptions, the TF-IDF method is used to calculate the weights. The semantic description v is designed, and the dimension is the number of semantic segmentation categories M. The i-th item is defined as: , in Indicates whether the i-th category appears in the current picture, the weight vector Defined as:

[0007] in Indicates the number of objects recognized in the current frame, Indicates the number of times the i-th category appears in the current frame, Indicates the total number of frames in the data, Indicates the number of frames containing the i-th category in the data; Indicates the proportion of the number of the i-th feature word in the total number of words in the current observation. It indicates that the inverse of the ratio of the number of pictures in which word i appears to the number of all pictures is positively correlated; S4: The matching similarity score based on semantic categories can be obtained by The norm is solved to get: ; Among them, s 1 refers to the matching similarity score, d is the distance function, p and q represent different frames, where refers to the semantic description of the pth frame, refers to the semantic description of the qth frame, is the dimension of the vector, i.e. the number of categories for semantic segmentation; S5: Calculate the final loop detection score by combining it with the geometric feature similarity s: , where R refers to the final loop detection score, is the weight of semantic feature similarity.

[0008] As a further limitation of the technical solution of the embodiment of the present invention, a deep learning network is used to classify the semantic categories in each frame of the picture, and the matching of feature points on objects of static semantic categories is weighted in feature matching and the matching of feature points in dynamic objects is eliminated to improve the accuracy of feature matching, and then the semantic category information is integrated into the bag-of-words model in the loop detection, and the weight is calculated by the TF-IDF method to calculate the weight of the initial frame. The similarity score is obtained by the norm and finally combined with the geometric feature similarity to obtain the final loop detection score.

[0009] Compared with the prior art, the present invention has the following beneficial effects: semantic information is added through a deep learning network on the basis of the existing methods at this stage, which not only improves the accuracy of feature matching to obtain better geometric feature loop similarity, but also provides a semantic weight during loop detection, thereby improving the robustness and accuracy of mapping. In addition, the design of adding semantic information proposed in the present invention is very clever, is not constrained by the specific environment, and can be easily transplanted to other systems, so it has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0012] Figure 1 A flow chart of a method provided by an embodiment of the present invention is shown.

[0013] Specifically, a loop detection method for visual SLAM, specifically a method for integrating semantic information into SLAM loop detection to obtain more accurate loop results, the present invention can be widely used in multiple fields such as autonomous driving, smart home, etc.; The method specifically comprises the following steps: S1: object detection to obtain semantic information; Use the deep neural network yolov9 to classify the objects in each frame and assign the semantic categories of these objects to the features of these objects; S2: Adjust the weight of dynamic and static semantic category matching; The constraint model F is established by combining the semantic model and the geometric model, whose normal feature vector contains the semantic labels of the pose and object class:

[0014] Where k is the geometric weight, (1-k) is the semantic weight, and in feature matching, the matching of feature points on objects of static semantic categories is weighted and the matching of feature points in dynamic objects is eliminated; G refers to the geometric model, and ,in is the registration information of the extracted features, T is the total number of feature point pairs extracted; S refers to the semantic model, and ,in Represents the semantic information of the label of the object detected by yolov9, and T is the total number of objects detected; S3: Calculate semantic weight; The method based on feature descriptors and bag-of-words model, which is common in visual SLAM loop closure, is to first generate bag-of-words vectors. Since different words have different weights in similarity descriptions, the TF-IDF method is used to calculate the weights. The semantic description v is designed, and the dimension is the number of semantic segmentation categories M. The i-th item is defined as: , in Indicates whether the i-th category appears in the current picture, the weight vector Defined as:

[0015] in Indicates the number of objects recognized in the current frame, Indicates the number of times the i-th category appears in the current frame, Indicates the total number of frames in the data, Indicates the number of frames containing the i-th category in the data; Indicates the proportion of the number of the i-th feature word in the total number of words in the current observation. It indicates that the inverse of the ratio of the number of pictures in which word i appears to the number of all pictures is positively correlated; S4: Calculate similarity score by semantic weight; The matching similarity score based on semantic category can be obtained by The norm is solved to get: ; Among them, s 1 refers to the matching similarity score, d is the distance function, p and q represent different frames, where refers to the semantic description of the pth frame, refers to the semantic description of the qth frame, is the dimension of the vector, i.e. the number of categories for semantic segmentation; S5: Get the final score of loop detection; The final loop detection score is calculated by combining it with the geometric feature similarity s: , where R refers to the final loop detection score, is the weight of semantic feature similarity.

[0016] In the embodiment of the present invention, a deep learning network is used to classify the semantic categories in each frame of the picture. In the feature matching, the matching of feature points on objects of static semantic categories is weighted and the matching of feature points in dynamic objects is eliminated to improve the accuracy of feature matching. Then, the semantic category information is integrated into the bag-of-words model in the loop detection, and the weight is calculated by the TF-IDF method to calculate the weights with the initial frame. The similarity score is obtained by the norm, and finally the final loop detection score is obtained by combining it with the geometric feature similarity result.

[0017] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0018] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0019] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0020] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0021] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A loop detection method for visual SLAM, characterized in that: The method comprises: S1: Use the deep neural network yolov9 to classify the objects in each frame and assign the semantic categories of these objects to the features of these objects; S2: Combine the semantic model and geometric model to establish a constraint model F, whose normal feature vector contains the semantic label of the pose and object class: , Where k is the geometric weight, (1-k) is the semantic weight, and in feature matching, the matching of feature points on objects of static semantic categories is weighted and the matching of feature points in dynamic objects is eliminated; G refers to the geometric model, and ,in is the registration information of the extracted features, T is the total number of feature point pairs extracted; S refers to the semantic model, and ,in Represents the semantic information of the label of the object detected by yolov9, and T is the total number of objects detected; S3: Incorporating semantic category information into the common method based on feature descriptors and bag-of-words models in visual SLAM loop closures. First, generate a bag-of-words vector. Since different words have different weights in similarity descriptions, the TF-IDF method is used to calculate the weights. The semantic description v is designed, and the dimension is the number of semantic segmentation categories M. The i-th item is defined as: , in Indicates whether the i-th category appears in the current picture, the weight vector Defined as: , in Indicates the number of objects recognized in the current frame, Indicates the number of times the i-th category appears in the current frame, Indicates the total number of frames in the data, Indicates the number of frames containing the i-th category in the data; Indicates the proportion of the number of the i-th feature word in the total number of words in the current observation. It indicates that the inverse of the ratio of the number of pictures in which word i appears to the number of all pictures is positively correlated; S4: The matching similarity score based on semantic categories can be obtained by Solving the norm yields: ; Among them, s1 refers to the matching similarity score, d is the distance function, p and q represent different frames, refers to the semantic description of the pth frame, refers to the semantic description of the qth frame, is the dimension of the vector, i.e. the number of categories for semantic segmentation; S5: Calculate the final loop detection score by combining it with the geometric feature similarity s: , where R refers to the final loop detection score, is the weight of semantic feature similarity.

2. The loop detection method for visual SLAM according to claim 1, characterized in that Use a deep learning network to classify the semantic categories in each frame of the picture. In feature matching, weight the matching of feature points on objects of static semantic categories and eliminate the matching of feature points in dynamic objects to improve the accuracy of feature matching. Then, integrate the semantic category information into the bag-of-words model in loop detection, calculate the weights using the TF-IDF method, and calculate the weights compared with the initial frame. The similarity score is obtained by the norm and finally combined with the geometric feature similarity to obtain the final loop detection score.

Citation Information

Patent Citations

  • Multi-sensor SLAM (Simultaneous Localization and Mapping) method based on dynamic feature point elimination and loopback detection

    CN118225096A

  • Visual slam method applicable to indoor dynamic environment

    WO2022041596A1

  • Loop closure detection method and apparatus, terminal device, and readable storage medium

    WO2022142855A1