A dynamic SLAM method based on multi-sensor and polarization attention

Abstract

The application relates to a dynamic SLAM method based on a multi-sensor and polarization attention, and belongs to the technical field of computer vision. The method adopts a PSA-Det segmentation module, utilizes a polarization self-attention mechanism to perform instance segmentation on image data, efficiently detects and removes dynamic objects, simultaneously utilizes a multi-sensor fusion technology for data complementation, improves accuracy, enhances robustness and realizes real-time performance, and thus significantly improves the performance and adaptability of a SLAM system in various environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a de-dynamic SLAM method based on multi-sensor and polarization attention. Background Technology

[0002] In the removal of dynamic objects, early works mainly used optical flow and geometric features to identify and remove dynamic objects. However, these methods perform poorly in complex environments and are prone to misclassification. In recent years, deep learning-based dynamic object removal methods have been gradually applied, typically using object detection algorithms to label dynamic objects. However, when object detection labels dynamic objects with bounding boxes, it may also remove some static points within the bounding box, affecting the accuracy of subsequent localization and mapping.

[0003] Currently, many SLAM (Simultaneous Localization and Mapping) systems suffer from inaccurate localization and "ghosting" in mapping under dynamic environments. Single-vision solutions are limited by their strong dependence on ambient light, failing to obtain clear and sufficient environmental information in low-light and dimly lit scenes. Individual sensors often have limitations; for example, single-LiDAR solutions are easily affected by environmental noise, such as rain, fog, strong light, and low-reflectivity environments. They are also susceptible to mapping errors caused by moving objects, resulting in insufficient accuracy, poor robustness, and poor performance under complex conditions.

[0004] Therefore, there is an urgent need for a SLAM method that can remove the interference of dynamic objects in a dynamic environment and fuse multi-sensor data. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a de-dynamic SLAM method based on multi-sensor and polarization attention, focusing on removing dynamic object interference and fusing multi-sensor data in dynamic environments. This method effectively identifies and removes dynamic points by employing a neural network model, and significantly reduces the impact of dynamic objects on static map construction by deeply coupling LiDAR and camera data. Furthermore, by leveraging a multi-sensor fusion strategy, it further improves positioning accuracy and increases the efficiency of static map construction.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A de-dynamic SLAM method based on multi-sensor and polarization attention is proposed. It uses a corner-based prediction head to estimate the bounding boxes of target objects and utilizes semantic information from image data obtained through instance segmentation to optimize the semantic subgraph and track trajectories. A dense map is constructed using semantic information, and a semantic subgraph is generated to assist in trajectory estimation for tracking dynamic targets. For dynamic objects that need to be removed, semantic information is used to assist in removal, and the semantic subgraph is updated. Subgraph optimization is performed again after trajectory tracking, and loop closure detection is performed simultaneously. Finally, IMU data is added to the global optimization, and the optimized subgraphs are combined to obtain the global map.

[0008] The method specifically includes the following steps:

[0009] S1: Data Input: The system acquires image data from the camera, point cloud information from the LiDAR, and combines it with IMU data to provide multi-dimensional sensor information input;

[0010] S2: Segmentation and Fusion: The PSA-Det segmentation module is used to perform semantic segmentation on the image, identify and generate segmentation results for dynamic objects; then, the system projects and fuses the segmentation results with point cloud data, and clusters the point cloud data with semantic information to ensure that the features of dynamic objects are correctly identified and removed; where PSA-Det represents a real-time segmentation model that introduces the polarization self-attention mechanism (PSA).

[0011] S3: Pose Estimation and Optimization: Use IMU for pose initialization and integration, and combine the segmented semantic information to generate a semantic subgraph; use a corner prediction head to estimate the bounding box of the target object and track dynamic objects;

[0012] During the trajectory tracking process, the system continuously optimizes the subgraph to ensure the accuracy and robustness of the mapping; semantic information assists in removing dynamic objects and updating the semantic subgraph during this process.

[0013] S4: Loop closure detection and global optimization: The system introduces loop closure detection and loop closure correction mechanisms to ensure that the map maintains global consistency during long-term motion; through global pose optimization of IMU data, the sub-map after loop closure correction is jointly optimized to finally generate a globally dense map.

[0014] Furthermore, in step S2, the PSA-Det segmentation module specifically includes a backbone network, a PAFPN, and a head part; wherein, PAFPN represents an enhanced feature pyramid network.

[0015] Furthermore, in step S2, the backbone network is a CSPNEXtPSA backbone network with large kernel depth convolution, used to extract multi-layer features at different scales; wherein, CSPNEXtPSA represents a backbone block integrating a polarization self-attention mechanism.

[0016] Furthermore, in step S2, the CSPNEXtPSA is composed of a 3x3 convolution BN SiLU, a 5x5 depthwise convolution BN SiLU, and a PSA.

[0017] Furthermore, in step S2, the PAFPN is used to fuse feature maps from different scales of the backbone network. Specifically, the fusion process between feature maps is represented by multiple concatenation and resizing operations. By upsampling and concatenating features at different levels, the expressive power of multi-scale features can be enhanced. The convolution operation further extracts information from the fused features.

[0018] Furthermore, in step S2, the Head part is used to simultaneously perform target detection and instance segmentation, generating a bounding box and a corresponding segmentation mask for each target.

[0019] The beneficial effects of this invention are as follows: Through multi-sensor fusion and polarization self-attention mechanism, this invention achieves a comprehensive improvement in the accuracy, robustness, and real-time performance of SLAM systems in dynamic environments, possessing broad application prospects. Specifically, this is reflected in the following aspects:

[0020] (1) This invention has higher accuracy in removing dynamic objects: Through the polarization self-attention mechanism, the PSA-Det segmentation model can accurately mark the contours of dynamic objects, avoiding the problem of erroneous removal of static points when using rectangular boxes for marking in object detection. This not only improves the recognition accuracy of dynamic objects, but also greatly reduces the misjudgment and removal of static objects, ensuring the accuracy of localization and mapping.

[0021] (2) Enhanced robustness of multi-sensor fusion in this invention: By fusing data from multiple sensors (LiDAR, camera, and IMU), this invention effectively addresses sensor errors and data loss under different environmental conditions. This fusion technology enables the system to maintain stable perception capabilities in complex dynamic environments, enhancing the robustness and adaptability of the SLAM system.

[0022] (3) This invention improves mapping efficiency: The polarization self-attention mechanism not only improves the accuracy of instance segmentation, but also reduces computational complexity by optimizing the network structure, thereby improving the real-time performance and efficiency of the SLAM system in dynamic environments. Due to accurate dynamic object segmentation and efficient masking, the system can process dynamic environments more quickly and generate higher-quality maps.

[0023] (4) The present invention can adapt to complex dynamic environments: The solution of the present invention can efficiently remove dynamic objects, such as fast-moving or non-rigid objects, significantly improving the performance of the SLAM system in complex dynamic environments. The system can adapt to a variety of real-world application scenarios, ensuring long-term stability and accuracy in dynamic environments.

[0024] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0026] Figure 1 This is a flowchart of the de-dynamic SLAM method based on multi-sensor and polarization attention of the present invention;

[0027] Figure 2 PSA-Det framework diagram;

[0028] Figure 3 This is a diagram of the CSPNEXtPSA Block framework. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] Please see Figures 1-3 This paper presents a de-dynamic SLAM method based on multi-sensor and polarization attention. It effectively identifies and removes dynamic points by employing a neural network model, and significantly reduces the impact of dynamic objects on static map construction by deeply coupling LiDAR and camera data. Furthermore, a multi-sensor fusion strategy further improves positioning accuracy and increases the efficiency of static map construction.

[0031] Specifically, in the construction of the PSA-Det segmentation model, large-kernel convolutions are used in the basic building blocks of the neck region to improve model efficiency. Furthermore, a polarization self-attention (PSA) mechanism is employed in the neck region. PSA is a lightweight self-attention mechanism used to capture global dependencies in feature maps. By enhancing feature representation, it improves the model's ability to detect targets. Using PSA on deep feature maps enhances the fusion of global contextual information, compensating for the shortcomings of convolutional networks in modeling long-range dependencies. Simultaneously, PSA dynamically adjusts the weights of each channel based on the correlation between channels and suppresses irrelevant or useless feature channels, thereby improving segmentation accuracy.

[0032] In the neck block, the CSP layer retains the model's feature learning ability while reducing computation. The Spatial Pyramid Pooling (SPP) layer uses different convolutional kernel sizes to process feature maps, thereby better capturing multi-scale information. By integrating PSA into the last feature layer, the correlation between features can be improved, forming a complementary structure.

[0033] This method processes image data from a camera sensor and uses a pre-trained PSA-Det segmentation model to segment dynamic objects in the image. Subsequently, the segmented image data is fused with point cloud data to obtain comprehensive point cloud information. By projecting the point cloud onto the image, the image pixel position corresponding to each point in the point cloud can be determined. After clustering the point cloud data containing semantic information, the value of the pixel and the clustering result are queried to determine whether the corresponding point belongs to a dynamic object, thereby achieving the removal of dynamic objects and the efficient construction of static maps.

[0034] The PSA-Det segmentation model can more accurately mark the contours of dynamic objects and remove them by processing the segmented masks, thereby effectively avoiding the problem of erroneously removing static points and improving the robustness of the SLAM system in dynamic environments.

[0035] like Figure 1As shown, the multi-sensor fusion SLAM framework proposed in this invention integrates information processing from cameras, LiDAR, point cloud data, and IMU (Inertial Measurement Unit), aiming to address the limitations of traditional SLAM systems in dynamic environments, such as mapping errors caused by dynamic object interference. The core of this framework employs a PSA-Det segmentation module, utilizing a polarization self-attention mechanism to segment image data into instances, efficiently detecting and removing dynamic objects. The advantages of multi-sensor fusion lie in data complementarity, improved accuracy, enhanced robustness, and real-time performance, thereby significantly improving the performance and adaptability of the SLAM system in various environments. A corner-based prediction head is used to estimate the bounding boxes of target objects, and semantic information from the image data obtained through instance segmentation is used to optimize the semantic subgraph and track trajectories. A dense map is constructed using semantic information, and a semantic subgraph is generated to assist in trajectory estimation for tracking dynamic targets. For dynamic objects that need to be removed, semantic information and geometric clustering are used to assist in removal, and the semantic subgraph is updated. Subgraph optimization is also performed after trajectory tracking, along with loop closure detection. Finally, IMU data is added to the global optimization, and the optimized subgraphs are combined to obtain the global map. Figure 1 A detailed framework diagram is shown. The overall system flow is as follows:

[0036] (1) Data input: The system acquires image data from the camera, point cloud information from the lidar, and combines IMU data to provide multi-dimensional sensor information input.

[0037] (2) Segmentation and Fusion: The PSA-Det segmentation module is used to perform semantic segmentation on the image, identify and generate segmentation masks for dynamic objects. Subsequently, the system projects and fuses the segmentation results with point cloud data, and clusters the point cloud data with semantic information to ensure that the features of dynamic objects are correctly identified and removed.

[0038] (3) Pose estimation and optimization: IMU is used for pose initialization and integration, and semantic subgraphs are generated by combining the segmented semantic information. Boundary boxes of target objects are estimated by corner prediction heads, and dynamic objects are tracked.

[0039] During trajectory tracking, the system continuously optimizes the subgraph to ensure the accuracy and robustness of the mapping. Semantic information assists in removing dynamic objects and updating the semantic subgraph during this process.

[0040] (4) Loop closure detection and global optimization: The system introduces loop closure detection and loop closure correction mechanisms to ensure that the map maintains global consistency during long-term motion. Through global pose optimization of IMU data, the sub-map after loop closure correction is jointly optimized to finally generate a globally dense map.

[0041] like Figure 2As shown, the PSA-Det segmentation model is a segmentation model that introduces a polarization self-attention (PSA) mechanism into the backbone network of the model structure. It combines multi-scale feature fusion PAFPN with an enhanced instance segmentation head, possessing stronger global modeling capabilities and being particularly suitable for instance segmentation in complex dynamic environments, improving the performance of dynamic object detection and removal. The framework structure is as follows: Figure 2 As shown.

[0042] (1) Backbone: C3, C4, C5 and other multi-layer features are extracted from the backbone network of CSPNEXtPSA (backbone block with integrated polarization self-attention mechanism) with large kernel depth convolution.

[0043] like Figure 3 As shown, the CSPNEXtPSA Block consists of a 3x3 convolutional BN SiLU, a 5x5 depthwise convolutional BN SiLU, and a PSA.

[0044] Batch Normalization (BN): Batch normalization is added after convolution, which helps accelerate training and stabilize gradients, thus allowing the model to converge faster. The formula for calculating batch normalization is:

[0045]

[0046] Where x is the input value, μ and σ 2 These are the mean and variance of the small batch of data, respectively. ∈ is a very small value used to prevent division by zero errors.

[0047] SiLU (Sigmoid Linear Unit) activation function: The SiLU activation function is a smooth, non-linear activation function, and its formula is:

[0048] SiLU(x)=x·σ(x)

[0049] Here, σ(x) is the standard sigmoid function. Compared to ReLU, SiLU achieves better performance while preserving negative values.

[0050] Depthwise Convolution: This step uses a 5x5 depthwise convolution. Unlike standard convolution, depthwise convolution performs a separate convolution operation on each input channel, reducing computational complexity. It can be decomposed from standard convolution into depthwise convolution and pointwise convolution. The mathematical expression for depthwise convolution is:

[0051]

[0052] Where X is the input feature map, W is the convolution kernel, i,j are the spatial dimensions, k is the number of channels, p corresponds to the vertical offset (height) of the convolution kernel, and q corresponds to the horizontal offset (width) of the convolution kernel.

[0053] PSA (Polarized Self-Attention): Unlike standard self-attention, PSA reduces computational cost by polarizing channels and space while maintaining good global information modeling capabilities. Its mathematical representation is:

[0054]

[0055] Where Q, K, and V are the matrices of query, key, and value, respectively, and d k It is a scaling factor. PSA polarizes information in different dimensions in the attention mechanism, improving computational efficiency.

[0056] Finally, residual connections are performed to concatenate the input and output, reducing the impact of the vanishing gradient problem and thus improving training efficiency.

[0057] (2) PAFPN (Enhanced Feature Pyramid Network): Improves detection accuracy by fusing feature maps from different scales, especially for multi-scale target detection. Figure 2 In this diagram, multiple Concat and Resize operations represent the fusion process between feature maps. Low-level feature maps typically have high resolution but less semantic information, while high-level feature maps have the opposite. By upsampling and concatenating features from different levels, the expressive power of multi-scale features can be enhanced. The Conv (convolution) operation further extracts information from the fused features.

[0058] (3) In the Head part, the network simultaneously performs target detection and instance segmentation, generating bounding boxes and corresponding segmentation masks for each target.

[0059] In summary, this invention presents a dynamic object removal method for instance segmentation based on a polarization self-attention mechanism. This technique enhances the accuracy of the PSA-Det segmentation model in labeling the contours of dynamic objects by introducing the polarization self-attention (PSA) mechanism. This mechanism can accurately identify and remove dynamic objects in complex environments while preserving surrounding static points, avoiding the problem of incorrect static point removal caused by the use of bounding boxes in object detection methods, thereby improving the accuracy and robustness of mapping.

[0060] This invention utilizes multi-sensor fusion technology. This technology fuses data from multiple sensors, such as LiDAR, cameras, and IMUs, enhancing the SLAM system's perception capabilities and data integrity in dynamic environments. By complementing multi-sensor data, it can address data loss or errors caused by changes in the environment from a single sensor, improving the system's robustness and adaptability in complex scenarios.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A de-dynamic SLAM method based on multi-sensor and polarization attention, characterized in that, The method specifically includes the following steps: S1: Data Input: The system acquires image data from the camera, point cloud information from the LiDAR, and combines it with IMU data to provide multi-dimensional sensor information input; S2: Segmentation and Fusion: The PSA-Det segmentation module is used to perform semantic segmentation on the image, identify and generate segmentation results for dynamic objects. Subsequently, the system projects and fuses the segmentation results with point cloud data, and clusters the point cloud data with semantic information to ensure that the features of dynamic objects are correctly identified and removed. The PSA-Det segmentation module specifically includes a backbone network, PAPPN, and a Head part. PSA-Det represents a real-time segmentation model that introduces the polarization self-attention mechanism PSA, and PAPPN represents an enhanced feature pyramid network. The backbone network is a CSPNEXtPSA backbone network with large kernel depth convolutions, used to extract multi-layer features at different scales. CSPNEXtPSA represents a backbone block integrating the polarization self-attention mechanism, composed of 3x3 convolutional BN SiLU, 5x5 depth convolutional BN SiLU, and PSA. S3: Pose Estimation and Optimization: Use IMU for pose initialization and integration, and combine the segmented semantic information to generate a semantic subgraph; use a corner prediction head to estimate the bounding box of the target object and track dynamic objects; During the trajectory tracking process, the system continuously optimizes the subgraph to ensure the accuracy and robustness of the mapping; semantic information assists in removing dynamic objects and updating the semantic subgraph during this process. S4: Loop closure detection and global optimization: The system introduces loop closure detection and loop closure correction mechanisms to ensure that the map maintains global consistency during long-term motion; through global pose optimization of IMU data, the sub-map after loop closure correction is jointly optimized to finally generate a globally dense map.

2. The de-dynamic SLAM method based on multi-sensor and polarization attention according to claim 1, characterized in that, In step S2, the PAFPN is used to fuse feature maps from different scales of the backbone network. Specifically, it represents the fusion process between feature maps through multiple concatenation and resizing operations; it enhances the expressive power of multi-scale features by upsampling and concatenating features at different levels; and it further extracts information from the fused features through convolution operations.

3. The de-dynamic SLAM method based on multi-sensor and polarization attention according to claim 1, characterized in that, In step S2, the Head part is used to simultaneously perform target detection and instance segmentation, generating the bounding box and corresponding segmentation mask for each target.

Images