Visual SLAM method based on semantic information fusion in indoor dynamic scene

By introducing feature extraction and semantic segmentation modules in ORB-SLAM2, identifying and eliminating feature points of dynamic objects, the problem of traditional visual SLAM degradation in dynamic environments is solved, and higher robustness and accuracy are achieved. It is suitable for Application requirements for complex scenarios.

CN118896599APending Publication Date: 2024-11-05BEIJING UNIV OF CIVIL ENG & ARCHITECTURE

Patent Information

Application Number
CN202410931407.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Traditional visual SLAM methods are difficult to distinguish between static and dynamic objects in dynamic environments, resulting in mismatch and incorrect pose estimation, affecting the stability and accuracy of the system. The single sensor data cannot fully utilize the advantages of multiple sensor fusion, resulting in complexity. Poor performance in the environment.

Method used

On the basis of ORB-SLAM2, feature extraction module and semantic segmentation module are introduced. High dynamic and low dynamic objects are identified through GCNv2 feature extraction algorithm and YOLOv5 algorithm, feature points of dynamic objects are eliminated, local map construction and optimization are carried out, and global optimization is carried out , improve the robustness and accuracy of the system.

Benefits of technology

It significantly improves the system's positioning and mapping accuracy in dynamic environments, reduces error accumulation, enhances the real-time and robustness of the system, and maintains stable performance in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118896599A_ABST
    Figure CN118896599A_ABST
Patent Text Reader

Abstract

The invention discloses a visual SLAM method and system based on semantic information fusion in an indoor dynamic scene, and belongs to the technical field of visual SLAM, and the method comprises the steps: introducing a feature extraction module and a semantic segmentation module for improvement based on an ORB-SLAM2 model, and constructing a visual SLAM model; the feature point recognition module is used for recognizing a high-dynamic object and a low-dynamic object in a dynamic scene, removing feature points located in a bounding box of the high-dynamic object but not in a bounding box of the low-dynamic object, obtaining static feature points, and performing construction and optimization of a local map; and performing loopback detection and global optimization according to the data processed by the local map thread. According to the method, semantic segmentation is performed on the dynamic objects in the scene by using the deep learning model, and dynamic targets such as people and the like are identified, so that adverse effects of the dynamic objects on the SLAM system are eliminated, and the environmental perception capability and the anti-interference capability of the SLAM system are further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of visual SLAM technology, and in particular to a visual SLAM method based on fusing semantic information in an indoor dynamic scene. Background Art

[0002] At present, many visual SLAM-based methods have achieved success in static environments, such as ORB-SLAM, PTAM, etc. These methods rely on extracting and matching visual feature points to achieve positioning and mapping, and can provide high accuracy in static environments. In dynamic environments, due to the presence of a large number of moving objects, the performance of traditional visual SLAM methods has significantly decreased. These dynamic objects can cause mismatching and incorrect pose estimation, which in turn affects the stability and accuracy of the entire SLAM system.

[0003] Traditional visual SLAM methods have difficulty distinguishing between static and dynamic objects, resulting in interference from dynamic objects affecting the accuracy of positioning and mapping. In dynamic environments, SLAM systems are easily affected by external changes and lack sufficient robustness to cope with complex and changing scenes. Processing a large amount of irrelevant information in dynamic scenes increases the computational burden and reduces the operating efficiency of the system. Relying on data from a single sensor and being unable to fully utilize the advantages of multiple sensor fusion leads to poor performance in complex environments.

[0004] In traditional visual SLAM (Simultaneous Localization and Mapping, simultaneous positioning and map construction) systems, dynamic objects in dynamic scenes can seriously affect the positioning accuracy and map construction quality of the system. ORB-SLAM2 is a classic SLAM framework, but it cannot effectively deal with the interference of dynamic objects. Therefore, the present invention is intended to improve on the basis of the ORB-SLAM2 framework to solve the problems existing in traditional visual SLAM for dynamic object processing. Summary of the invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide an improved visual SLAM method, which adds a feature extraction module and a semantic segmentation module on the basis of ORB-SLAM2 to improve the robustness and accuracy of the algorithm in dynamic scenes.

[0006] In order to achieve the above technical objectives, the present application provides a visual SLAM method based on the fusion of semantic information in indoor dynamic scenes, comprising the following steps:

[0007] Based on the ORB-SLAM2 model, feature extraction module and semantic segmentation module are introduced to improve and build a visual SLAM model;

[0008] Through the visual SLAM model, high-dynamic objects and low-dynamic objects in dynamic scenes are identified, and feature points that are within the bounding box of high-dynamic objects but not within the bounding box of low-dynamic objects are eliminated, static feature points are obtained, and local maps are constructed and optimized, including generating new map points, eliminating redundant keyframes, and performing local BA optimization;

[0009] Based on the data processed by the local map thread, loop detection and global optimization are performed, including calculation of the Sim3 matrix of pose transformation, loop fusion, pose graph optimization and global BA optimization.

[0010] Preferably, in the process of introducing the feature extraction module, a feature extraction module is generated according to the GCNv2 feature extraction algorithm to extract GCNv2 feature points in the image and generate feature points and descriptors consistent with the ORB structure.

[0011] Preferably, in the process of introducing the semantic segmentation module, a semantic segmentation module is generated according to the YOLOv5 algorithm, which is used to perform semantic segmentation on the image, detect dynamic and static objects in the image frame, and generate a semantic label for each pixel.

[0012] Preferably, in the process of constructing the visual SLAM model, the visual SLAM model is obtained by training with the TUMRGB-D data set based on the improved ORB-SLAM2 model.

[0013] Preferably, in the process of acquiring high-dynamic objects, objects whose positions change greatly between each frame are regarded as high-dynamic objects.

[0014] Preferably, in the process of acquiring low-dynamic objects, objects that remain stationary or change very little in the scene and have little effect on the positions of key points are regarded as low-dynamic objects.

[0015] Preferably, in constructing the local map, based on the static feature points, initial pose estimation is performed, GCNv2 feature points in the image are extracted, feature points of the current frame and the previous frame are matched, and a suitable frame is selected as a key frame and inserted into the local map.

[0016] The present invention also discloses a visual SLAM system based on fusion of semantic information in indoor dynamic scenes, comprising:

[0017] Model building module, which is used to build a visual SLAM model based on the ORB-SLAM2 model by introducing feature extraction module and semantic segmentation module for improvement;

[0018] The feature point extraction module is used to identify high-dynamic objects and low-dynamic objects in dynamic scenes through the visual SLAM model, and remove the feature points that are within the bounding box of the high-dynamic object but not within the bounding box of the low-dynamic object to obtain static feature points;

[0019] The local map generation module is used to construct and optimize the local map based on static feature points, including generating new map points, removing redundant keyframes, and performing local BA optimization;

[0020] The global map generation module performs loop detection and global optimization based on the data processed by the local map thread, including calculating the Sim3 matrix of pose transformation, loop fusion, pose graph optimization and global BA optimization.

[0021] The present invention discloses the following technical effects:

[0022] The present invention uses the GCNv2 feature extraction algorithm instead of the ORB feature extraction algorithm to extract GCNv2 feature points in the image in the tracking thread. Compared with the traditional ORB feature extraction algorithm, the GCNv2 network extracts features through a convolutional neural network, which has higher computational efficiency and greatly improves the real-time performance of the VSLAM system.

[0023] The present invention uses the YOLOv5 algorithm to perform semantic segmentation on the image, detect dynamic and static objects in the image frame, and generate semantic labels for each pixel. By introducing semantic information fusion technology, the method can identify and eliminate the interference of dynamic objects on the SLAM system, thereby improving the accuracy of positioning and mapping. Multi-sensor data fusion technology provides richer and more accurate environmental information by integrating multiple sensor data, thereby improving the overall accuracy of the system. Semantic information fusion technology enables the system to distinguish between static and dynamic objects and reduce error accumulation in dynamic scenes. The adaptive dynamic object filtering algorithm dynamically adjusts the parameters of the SLAM system according to the dynamic object information detected in real time, so that the system can still maintain stable performance in a complex and changing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0026] Figure 2 It is a schematic diagram of the GCNv2 network structure described in the present invention. DETAILED DESCRIPTION

[0027] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0028] like Figure 1-2 As shown, the present invention provides an improved visual SLAM method, which adds a feature extraction module and a semantic segmentation module on the basis of ORB-SLAM2, and specifically includes the following processes:

[0029] Step 1: Input image data from TUM dataset:

[0030] Image data of indoor scenes are obtained from the TUM RGB-D dataset (Technical University of Munich RGB-DDataset). The TUM RGB-D dataset is a dataset widely used for SLAM and visual odometer evaluation, which contains RGB and depth image sequences in various indoor scenes. Download the required image sequences by visiting the official website of the dataset. Each frame of image data in the dataset includes an RGB image and a corresponding depth image. These image sequences can provide rich indoor scene information and contain a variety of dynamic objects, which are suitable for the experiments and evaluations of the present invention.

[0031] The input data processing steps are as follows:

[0032] 1. Download the required image sequence from the TUM dataset official website.

[0033] 2. Unzip the downloaded image sequence file.

[0034] 3. Pair the RGB image and depth image according to the timestamp.

[0035] 4. Input the paired image sequence into the SLAM system as raw data for algorithm processing.

[0036] Step 2: Feature Extraction:

[0037] In the tracking thread, the GCNv2 feature extraction algorithm is used to extract GCNv2 feature points in the image. Compared with the traditional ORB feature extraction algorithm, the GCNv2 network has significant advantages. First, the GCNv2 network uses a convolutional neural network to extract features, which is more efficient and overcomes the shortcomings of the traditional ORB feature extraction algorithm, which has a large amount of computation, thereby greatly improving the real-time performance of the VSLAM system. Secondly, the GCNv2 network can calculate feature points and descriptors similar to the ORB structure, and the generated feature point data structure is consistent with the format of the ORB feature, which makes it easy to apply the GCNv2 feature points in the VSLAM system based on the ORB feature for seamless integration. In addition, the GCNv2 network makes full use of the rich information in the image, mainly through multi-layer convolutional layers to calculate each pixel unit of the original image, so that feature points can be extracted more accurately. This in-depth use of image information makes the GCNv2 network more robust in dynamic scenes, improving the overall positioning accuracy and map construction quality of the system.

[0038] The GCNv2 network processing steps are as follows:

[0039] Input image: Input the preprocessed image into the GCNv2 network.

[0040] Convolutional layer calculation: The GCNv2 network calculates the image layer by layer through multiple convolutional layers to extract feature information in the image.

[0041] Feature point extraction: Through the output layer of the network, feature points and descriptors similar to the ORB structure are generated. The data structure of each feature point is consistent with the format of the ORB feature, where the size of each pixel unit is 16.

[0042] Step 3: Semantic Segmentation:

[0043] The YOLOv5 algorithm is used to perform semantic segmentation on the image, detect dynamic and static objects in the image frame, and generate semantic labels for each pixel. Specifically, the YOLOv5s algorithm has a high intersection over union (IoU), which provides significant advantages in speed, efficiency, and applicability. IoU is a commonly used indicator in target detection to measure the detection effect of target detection algorithms. Bounding box is a rectangular box used to mark the position and size of objects in images or videos. IoU is used to measure the degree of overlap between the bounding box predicted by the detection algorithm and the actual bounding box. The calculation formula is as follows:

[0044]

[0045] In formula (1), each box is p i , p bestTo accurately predict the box, S pi represents the area of ​​each bounding box, S pbest Represents the area of ​​the accurate prediction box, and IoU is the ratio of the intersection area of ​​two bounding boxes to the union area.

[0046] Since the target object may be detected by multiple bounding boxes, non-maximum suppression (NMS) is used to filter duplicate bounding boxes and retain the best bounding box. The filtering formula is as follows:

[0047]

[0048] In formula (2), α is the IoU threshold between two bounding boxes. The threshold value is used to determine whether to retain the bounding box and then select the best bounding box.

[0049] In step 2, the tracking thread uses the GCNv2 feature extraction algorithm instead of the ORB feature extraction algorithm to extract GCNv2 feature points in the image. Compared with the traditional ORB feature extraction algorithm, the GCNv2 network uses a convolutional neural network to extract features, which has higher computational efficiency and greatly improves the real-time performance of the VSLAM system.

[0050] In step 3, the YOLOv5 algorithm is used to perform semantic segmentation on the image, detect dynamic and static objects in the image frame, and generate semantic labels for each pixel. By introducing semantic information fusion technology, this method can identify and eliminate the interference of dynamic objects on the SLAM system, thereby improving the accuracy of positioning and mapping. Multi-sensor data fusion technology provides richer and more accurate environmental information by integrating multiple sensor data, thereby improving the overall accuracy of the system. Semantic information fusion technology enables the system to distinguish between static and dynamic objects and reduce error accumulation in dynamic scenes. The adaptive dynamic object filtering algorithm dynamically adjusts the parameters of the SLAM system according to the real-time detected dynamic object information, so that the system can still maintain stable performance in complex and changing environments.

[0051] First, the present invention classifies high-dynamic objects and static objects. High-dynamic objects include people walking or moving, and the positions of these objects change greatly between each frame; while low-dynamic objects such as tables, chairs, books, computers, etc. usually remain stationary or change very little in the scene, and have little effect on the position of the key points. Next, the feature points are classified. Static points include those points outside the dynamic detection frame, and feature points that are located in the dynamic detection frame but can be extracted from the static background; dynamic points refer to those feature points that are directly located on the moving object and move with it. In terms of processing strategy, the YOLOv5s algorithm is used to detect targets in the image and identify high-dynamic objects and low-dynamic objects. Only when the feature point is located in the bounding box of the high-dynamic object but not in the bounding box of the low-dynamic object, the feature point will be eliminated. This method ensures that static objects can be reliably detected and tracked in a dynamic environment, while separating and tracking objects in motion, improving the robustness and accuracy of the system in complex scenes.

[0052] The visual SLAM method based on fusion of semantic information mentioned in the present invention includes four main operating modules: semantic segmentation module, tracking thread, local map thread and loop detection thread. Its overall architecture is as follows Figure 1 As shown. During the use of this method, the tracking thread receives data from the RGB image frame and the Depth image frame, performs initial pose estimation and GCNv2 feature extraction, then matches feature points and generates new key frames to insert into the local map. The semantic segmentation module uses the YOLOv5 algorithm to perform semantic segmentation on the image, detects high-dynamic objects and low-dynamic objects, removes dynamic feature points from the image, and ensures that only static feature points are retained. These processed data are fed back to the tracking thread. The local map thread receives the key frames generated by the tracking thread and is responsible for the construction and optimization of the local map, including generating new map points, removing redundant key frames, and performing local BA optimization, so as to maintain the accuracy and integrity of the map. The loop detection thread receives the data processed by the local map thread, performs loop detection and global optimization, including calculating the Sim3 matrix of pose transformation, loop fusion, pose graph optimization, and global BA optimization, to ensure that the system can maintain high robustness and accuracy in the presence of cumulative errors.

[0053] The specific functions of the tracking thread include initial pose estimation, extracting GCNv2 feature points in the image, matching feature points of the current frame with the previous frame, selecting appropriate frames as key frames and inserting them into the local map. The semantic segmentation module uses the YOLOv5 algorithm to perform target detection and semantic association, remove feature points on dynamic objects, and retain static feature points, thereby ensuring that static objects can be reliably detected and tracked in dynamic environments. The local map thread is mainly responsible for tracking feature points in the local map, selecting key frames according to specific strategies, generating new map points, performing local BA optimization, eliminating map points with large errors and redundant key frames, and simplifying the map structure. The loop detection thread detects whether the camera has returned to its previous position, calculates the Sim3 matrix of the camera pose transformation, integrates the loop detection information into the map, and performs pose graph and global BA optimization, thereby reducing cumulative errors and improving the consistency and accuracy of the map.

[0054] Through the collaborative work of the above modules, the present invention can achieve high-precision positioning and map construction in dynamic environments, effectively deal with the interference of dynamic objects, significantly improve the real-time and robustness of the SLAM system, and is suitable for application needs in complex scenarios.

[0055] The present invention is applicable to various indoor robot navigation, indoor UAV flight, augmented reality (AR) devices and other application scenarios that require high-precision indoor positioning and mapping. By improving the performance of the SLAM system in a dynamic environment, the method has broad market application prospects and commercial value.

[0056] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0058] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A visual SLAM method based on the fusion of semantic information in indoor dynamic scenes, characterized in that: The following steps are involved: Based on the ORB-SLAM2 model, feature extraction module and semantic segmentation module are introduced to improve and build a visual SLAM model; Through the visual SLAM model, high-dynamic objects and low-dynamic objects in dynamic scenes are identified, and feature points that are within the bounding box of high-dynamic objects but not within the bounding box of low-dynamic objects are removed, static feature points are obtained, and local maps are constructed and optimized, including generating new map points, removing redundant key frames, and performing local BA optimization; Based on the data processed by the local map thread, loop detection and global optimization are performed, including calculation of the Sim3 matrix of pose transformation, loop fusion, pose graph optimization and global BA optimization.

2. The visual SLAM method based on fusion of semantic information in indoor dynamic scenes according to claim 1, characterized in that: In the process of introducing the feature extraction module, the feature extraction module is generated according to the GCNv2 feature extraction algorithm, which is used to extract GCNv2 feature points in the image and generate feature points and descriptors consistent with the ORB structure.

3. The visual SLAM method based on fusion of semantic information in indoor dynamic scenes according to claim 2, characterized in that: In the process of introducing the semantic segmentation module, the semantic segmentation module is generated according to the YOLOv5 algorithm, which is used to perform semantic segmentation on the image, detect dynamic and static objects in the image frame, and generate a semantic label for each pixel.

4. The visual SLAM method based on fusion of semantic information in indoor dynamic scenes according to claim 3, characterized in that: In the process of constructing the visual SLAM model, the visual SLAM model is obtained by training the improved ORB-SLAM2 model through the TUM RGB-D data set.

5. The visual SLAM method based on fusion of semantic information in indoor dynamic scenes according to claim 4, characterized in that: In the process of acquiring high-dynamic objects, objects whose positions change greatly between each frame are regarded as the high-dynamic objects.

6. The visual SLAM method based on fusion of semantic information in indoor dynamic scenes according to claim 5, characterized in that: In the process of acquiring low-dynamic objects, objects that remain stationary or change very little in the scene and have little effect on the positions of key points are regarded as the low-dynamic objects.

7. The visual SLAM method based on fusion of semantic information in indoor dynamic scenes according to claim 6, characterized in that: In the construction of the local map, based on the static feature points, initial pose estimation is performed, GCNv2 feature points in the image are extracted, feature points of the current frame and the previous frame are matched, and a suitable frame is selected as a key frame and inserted into the local map.

8. A visual SLAM system based on fusion of semantic information in indoor dynamic scenes, characterized in that: include: Model building module, which is used to build a visual SLAM model based on the ORB-SLAM2 model by introducing feature extraction module and semantic segmentation module for improvement; A feature point extraction module is used to identify high-dynamic objects and low-dynamic objects in a dynamic scene through the visual SLAM model, and remove feature points that are within the bounding box of the high-dynamic object but not within the bounding box of the low-dynamic object to obtain static feature points; A local map generation module, used to construct and optimize a local map based on the static feature points, including generating new map points, removing redundant key frames, and performing local BA optimization; The global map generation module performs loop detection and global optimization based on the data processed by the local map thread, including calculating the Sim3 matrix of pose transformation, loop fusion, pose graph optimization and global BA optimization.

Citation Information

Patent Citations

  • Visual SLAM method and system for indoor dynamic scene based on deep learning

    CN115393538A

  • Robustness dynamic vision synchronous positioning and mapping method and system

    CN117557748A

  • Visual SLAM (Simultaneous Localization and Mapping) system and method for weak textures and dynamic scenes

    CN118229774A

Cited By

  • Map determination method and device, vehicle and medium

    CN119984298A

  • Map determination method, device, vehicle and medium

    CN119984298B

  • Indoor SLAM map construction method and system

    CN121577019A

  • Indoor slam map construction method and system

    CN121577019B