Dynamic SLAM removing method based on multiple sensors and polarized attention

Abstract

The invention relates to a dynamic SLAM removing method based on multiple sensors and polarized attention, and belongs to the technical field of computer vision. According to the method, a PSA-Det segmentation module is adopted, instance segmentation is carried out on image data by using a polarization self-attention mechanism, and dynamic objects are efficiently detected and removed; and meanwhile, a multi-sensor fusion technology is used for data complementation, precision improvement, robustness enhancement and real-time performance realization, so that the performance and adaptability of the SLAM system in various environments are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a de-dynamic SLAM method based on multi-sensors and polarized attention. Background Art

[0002] In terms of dynamic object removal, early work mainly used optical flow and geometric features to judge and remove dynamic objects, but these methods perform poorly in complex environments and are prone to misjudgment. In recent years, dynamic object removal methods based on deep learning have gradually been applied, and target detection algorithms are usually used to mark dynamic objects. However, when target detection marks dynamic objects with rectangular boxes, some static points in the marked box may be removed together, affecting the subsequent positioning and mapping accuracy.

[0003] Currently, many SLAM (Simultaneous Localization and Mapping) systems have problems such as inaccurate positioning and "ghosting" in mapping in dynamic environments. Single vision solutions are limited by their strong dependence on ambient light and cannot obtain clear and sufficient environmental information in low-light and dim scenes. Single sensors often have limitations. For example, single lidar solutions are easily affected by environmental noise, such as rainy and foggy weather, strong light and low reflectivity environments; such as mapping errors caused by interference from dynamic objects, insufficient accuracy, poor robustness and poor performance under complex conditions.

[0004] Therefore, there is an urgent need for a SLAM method that can remove dynamic object interference and fuse multi-sensor data in a dynamic environment. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a de-dynamic SLAM method based on multi-sensor and polarized attention, focusing on removing dynamic object interference and fusing multi-sensor data in a dynamic environment. This method effectively identifies and removes dynamic points by adopting a neural network model, and significantly reduces the impact of dynamic objects on static map construction by deeply coupling lidar and camera data. In addition, with the help of a multi-sensor fusion strategy, the positioning accuracy is further improved and the efficiency of static map construction is improved.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A de-dynamic SLAM method based on multi-sensor and polarized attention uses a corner-based prediction head to estimate the bounding box of the target object, and uses the semantic information of the image data obtained by instance segmentation to optimize the semantic subgraph and track the trajectory. Using the semantic information, a dense map is constructed, and a semantic subgraph is generated to assist in tracking the trajectory estimation of dynamic targets. For dynamic objects that need to be eliminated, the semantic information is used to assist in eliminating them, and then the semantic subgraph is updated. After trajectory tracking, subgraph optimization is also performed, and closed-loop detection is performed at the same time. Finally, the IMU data is added to the global optimization, and the optimized subgraphs are combined to obtain a global map.

[0008] The method specifically comprises the following steps:

[0009] S1: Data input: The system obtains image data from the camera, point cloud information from the lidar, and combines it with IMU data to provide multi-dimensional sensor information input;

[0010] S2: Segmentation and fusion: The PSA-Det segmentation module is used to perform semantic segmentation on the image, identify and generate segmentation results of dynamic objects; then, the system projects and fuses the segmentation results with the point cloud data, and clusters the point cloud data with semantic information to ensure that the features of dynamic objects are correctly identified and removed; PSA-Det represents a real-time segmentation model that introduces the polarized self-attention mechanism (PSA);

[0011] S3: Pose estimation and optimization: Use IMU to initialize and integrate the pose, and generate semantic subgraphs based on the segmented semantic information; use the corner prediction head to estimate the bounding box of the target object and track dynamic objects;

[0012] In the process of tracking trajectories, the system continuously optimizes the subgraph to ensure the accuracy and robustness of the map. In this process, semantic information helps to remove dynamic objects and update the semantic subgraph.

[0013] S4: Loop detection and global optimization: The system introduces loop detection and closed-loop correction mechanisms to ensure that the map maintains global consistency during long-term movement. Through global pose optimization of IMU data, the sub-map after closed-loop correction is jointly optimized to finally generate a global dense map.

[0014] Further, in step S2, the PSA-Det segmentation module specifically includes a backbone network (Backbone), PAFPN and Head parts; wherein PAFPN represents an enhanced feature pyramid network.

[0015] Further, in step S2, the backbone network is a CSPNeXtPSA backbone network with large kernel depth convolution, which is used to extract multi-layer features of different scales; wherein CSPNeXtPSA represents a backbone block integrating a polarized self-attention mechanism.

[0016] Further, in step S2, the CSPNeXtPSA is composed of 3x3 convolution BN SiLU, 5x5 depth convolution BN SiLU and PSA.

[0017] Further, in step S2, the PAFPN is used to fuse feature maps of different scales from the backbone network, specifically: a plurality of concatenation (Concat) and resize (Resize) operations are used to represent the fusion process between feature maps; by up-sampling and down-sampling and concatenating features at different levels, the expression capability of multi-scale features can be enhanced; the convolution (Conv) operation further extracts information on the fused features.

[0018] Furthermore, in step S2, the Head part is used to simultaneously complete target detection and instance segmentation, and generate a bounding box and a corresponding segmentation mask for each target.

[0019] The beneficial effects of the present invention are: through multi-sensor fusion and polarization self-attention mechanism, the present invention realizes comprehensive improvement in the accuracy, robustness and real-time performance of the SLAM system in a dynamic environment, and has broad application prospects. Specifically, it is reflected in the following aspects:

[0020] (1) The present invention has higher accuracy in removing dynamic objects: Through the polarized self-attention mechanism, the PSA-Det segmentation model can accurately mark the contours of dynamic objects, avoiding the problem of incorrectly removing static points when using rectangular box markings in target detection. This not only improves the recognition accuracy of dynamic objects, but also greatly reduces the misjudgment and removal of static objects, ensuring the accuracy of positioning and mapping.

[0021] (2) Enhanced robustness of multi-sensor fusion of the present invention: The present invention effectively addresses sensor errors and loss problems under different environmental conditions by fusing data from multiple sensors (lidar, camera, and IMU). This fusion technology enables the system to maintain stable perception capabilities in complex dynamic environments, enhancing the robustness and adaptability of the SLAM system.

[0022] (3) The present invention improves the efficiency of mapping: the polarized self-attention mechanism not only improves the accuracy of instance segmentation, but also reduces the computational complexity by optimizing the network structure, thereby improving the real-time performance and efficiency of the SLAM system in dynamic environments. Due to the accurate segmentation of dynamic objects and efficient mask processing, the system can process dynamic environments more quickly and generate higher quality maps.

[0023] (4) The present invention can adapt to complex dynamic environments: The present invention can efficiently remove dynamic objects for fast-moving or non-rigid objects, significantly improving the performance of the SLAM system in complex dynamic environments. The system can adapt to a variety of real-world application scenarios and ensure long-term stability and accuracy in dynamic environments.

[0024] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0026] Figure 1 It is a flowchart of the de-dynamic SLAM method based on multi-sensor and polarized attention of the present invention;

[0027] Figure 2 This is the PSA-Det framework diagram;

[0028] Figure 3 This is the CSPNeXtPSA Block framework diagram. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] See also Figure 1 to Figure 3 , a de-dynamic SLAM method based on multi-sensor and polarized attention, which effectively identifies and removes dynamic points by using a neural network model, and significantly reduces the impact of dynamic objects on static map construction by deeply coupling lidar and camera data. In addition, with the help of multi-sensor fusion strategy, the positioning accuracy is further improved and the efficiency of static map construction is improved.

[0031] Specifically, in the construction of the PSA-Det segmentation model, large kernel convolution is used in the basic building block of the trunk neck to improve the model efficiency, and the polarized self-attention mechanism (PSA) is used in the neck. The polarized self-attention mechanism is a lightweight self-attention mechanism used to capture global dependencies in feature maps. By enhancing feature expression, it can improve the model's ability to detect targets. Using PSA on deep feature maps can enhance the fusion of global context information and make up for the lack of long-distance dependency modeling capabilities of convolutional networks. At the same time, PSA will dynamically adjust the weight of each channel according to the correlation between channels and suppress irrelevant or useless feature channels, thereby achieving the effect of improving segmentation accuracy.

[0032] In the neck block, the CSP layer retains the feature learning ability of the model while reducing the amount of computation. The spatial pyramid pooling (SPP) layer uses different convolution kernel sizes to process feature maps to better capture multi-scale information. By integrating PSA into the last feature layer, the correlation between features can be improved to form a complementary structure.

[0033] This method processes image data from the camera sensor and uses the pre-trained PSA-Det segmentation model to segment dynamic objects in the image. Subsequently, the segmented image data is fused with the point cloud data to obtain comprehensive point cloud information. By projecting the point cloud onto the image, the image pixel position corresponding to each point in the point cloud can be determined. After clustering the point cloud data containing semantic information, the corresponding point can be judged whether it belongs to a dynamic object by querying the value of the pixel point and the clustering result, thereby achieving the removal of dynamic objects and the efficient construction of static maps.

[0034] The PSA-Det segmentation model can mark the contours of dynamic objects more accurately and remove dynamic objects by processing the segmented masks, thereby effectively avoiding the problem of mistakenly removing static points and improving the robustness of the SLAM system in dynamic environments.

[0035] like Figure 1As shown, the multi-sensor fusion SLAM framework proposed in the present invention integrates the information processing of cameras, lidars, point cloud data and IMU (inertial measurement unit), aiming to solve the limitations of traditional SLAM systems in dynamic environments, such as mapping errors caused by interference from dynamic objects. The core of the framework adopts the PSA-Det segmentation module, and uses the polarization self-attention mechanism to perform instance segmentation on image data, efficiently detecting and removing dynamic objects. The advantages of multi-sensor fusion are data complementarity, improved accuracy, enhanced robustness and real-time performance, thereby significantly improving the performance and adaptability of SLAM systems in various environments. The bounding box of the target object is estimated using a prediction head based on corner points, and the semantic information of the image data obtained by instance segmentation is used to optimize the semantic subgraph and track the trajectory. Using semantic information, a dense map is constructed, and a semantic subgraph is generated to assist in tracking the trajectory estimation of dynamic targets. For dynamic objects that need to be eliminated, the semantic subgraph is updated after being eliminated by semantic information and geometric clustering. Subgraph optimization is also performed after trajectory tracking, and closed-loop detection is performed at the same time. Finally, the IMU data is added to the global optimization, and the optimized subgraph is combined to obtain a global map. Figure 1 The specific framework diagram is shown. The entire system process is as follows:

[0036] (1) Data input: The system obtains image data from the camera, point cloud information from the lidar, and combines it with IMU data to provide multi-dimensional sensor information input.

[0037] (2) Segmentation and fusion: The PSA-Det segmentation module is used to perform semantic segmentation on the image, identify and generate segmentation masks for dynamic objects. Subsequently, the system projects and fuses the segmentation results with the point cloud data, and clusters the point cloud data with semantic information to ensure that the features of dynamic objects are correctly identified and removed.

[0038] (3) Pose estimation and optimization: Use the IMU to initialize and integrate the pose, and generate semantic subgraphs based on the semantic information after segmentation. Use the corner prediction head to estimate the bounding box of the target object and track dynamic objects.

[0039] In the process of tracking the trajectory, the system continuously optimizes the sub-graph to ensure the accuracy and robustness of the map. The semantic information helps to remove dynamic objects and update the semantic sub-graph in this process.

[0040] (4) Loop detection and global optimization: The system introduces loop detection and closed-loop correction mechanisms to ensure that the map maintains global consistency during long-term movement. Through global pose optimization of IMU data, the sub-map after closed-loop correction is jointly optimized to finally generate a global dense map.

[0041] like Figure 2As shown in the figure, the PSA-Det segmentation model is a segmentation model that introduces the polarized self-attention mechanism (PSA) into the backbone network of the model structure. It combines the PAFPN with multi-scale feature fusion and the enhanced instance segmentation head, has stronger global modeling capabilities, and is particularly suitable for instance segmentation in complex dynamic environments, improving the effect of dynamic object detection and removal. The framework structure is shown in the figure. Figure 2 shown.

[0042] (1) Backbone: Multi-layer features such as C3, C4, and C5 are extracted through the backbone network of CSPNeXtPSA (backbone block with integrated polarized self-attention mechanism) with large kernel deep convolution.

[0043] like Figure 3 As shown, the CSPNeXtPSA Block consists of 3x3 convolution BN SiLU, 5x5 depth convolution BN SiLU and PSA.

[0044] BN (Batch Normalization): Batch normalization is added after convolution, which helps to speed up training and stabilize the gradient, so that the model converges faster. The calculation formula of batch normalization is:

[0045]

[0046] Where x is the input value, μ and σ 2 are the mean and variance of the small batch data respectively, and ∈ is a very small value to prevent division by zero errors.

[0047] SiLU (Sigmoid Linear Unit) activation function: The SiLU activation function is a smooth nonlinear activation function with the formula:

[0048] SiLU(x)=x·σ(x)

[0049] Among them, σ(x) is the standard sigmoid function. Compared with ReLU, SiLU can achieve better performance while retaining negative values.

[0050] Depthwise Convolution: This step uses a 5x5 depthwise convolution. Depthwise convolution is different from standard convolution. It performs a separate convolution operation on each input channel, reducing the computational complexity. It can decompose standard convolution into depthwise convolution and pointwise convolution. The mathematical expression of depthwise convolution is:

[0051]

[0052] Among them, X is the input feature map, W is the convolution kernel, i, j are the spatial dimensions, k is the number of channels, p corresponds to the offset of the convolution kernel in the vertical direction (height), and q corresponds to the offset of the convolution kernel in the horizontal direction (width).

[0053] PSA (Polarized Self-Attention): Different from standard self-attention, PSA reduces the amount of computation by polarizing channels and spaces while maintaining good global information modeling capabilities. The specific mathematical representation is:

[0054]

[0055] Among them, Q, K, and V are the matrices of query, key, and value respectively, and d k is the scaling factor. PSA polarizes information of different dimensions in the attention mechanism and improves computational efficiency.

[0056] Finally, a residual connection is performed to concatenate the input and output to reduce the impact of the gradient vanishing problem, thereby achieving the goal of improving training efficiency.

[0057] (2) PAFPN (Enhanced Feature Pyramid Network): Improves detection accuracy by fusing feature maps from different scales, especially for the detection of multi-scale objects. Figure 2 In the example, multiple Concat and Resize operations represent the fusion process between feature maps. Low-level feature maps usually have higher resolution but less semantic information, while high-level feature maps are the opposite. By upsampling and downsampling and concatenating features at different levels, the expressive power of multi-scale features can be enhanced. Conv (convolution) The convolution operation further extracts information on the fused features.

[0058] (3) In the Head part, the network simultaneously completes target detection and instance segmentation, generating the bounding box and corresponding segmentation mask of each target.

[0059] In summary, the present invention removes dynamic objects based on instance segmentation of polarized self-attention mechanism. This technology improves the ability of the PSA-Det segmentation model to accurately mark the contours of dynamic objects by introducing the polarized self-attention mechanism (PSA). This mechanism can accurately identify and remove dynamic objects in complex environments while retaining the surrounding static points, avoiding the problem of incorrect removal of static points caused by the use of rectangular boxes in target detection methods, thereby improving the accuracy and robustness of mapping.

[0060] The present invention applies multi-sensor fusion technology. This technology fuses the data of multiple sensors such as laser radar, camera, IMU, etc., and enhances the perception ability and data integrity of the SLAM system in a dynamic environment. Through the complementarity of multi-sensor data, it is possible to deal with data loss or error caused by environmental changes in a single sensor, and improve the robustness and adaptability of the system in complex scenes.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A de-dynamic SLAM method based on multi-sensor and polarized attention, characterized in that: The method specifically comprises the following steps: S1: Data input: The system obtains image data from the camera, point cloud information from the lidar, and combines it with IMU data to provide multi-dimensional sensor information input; S2: Segmentation and fusion: The PSA-Det segmentation module is used to perform semantic segmentation on the image, identify and generate segmentation results of dynamic objects; then, the system projects and fuses the segmentation results with the point cloud data, and clusters the point cloud data with semantic information to ensure that the features of dynamic objects are correctly identified and removed; PSA-Det represents a real-time segmentation model that introduces the polarized self-attention mechanism PSA; S3: Pose estimation and optimization: Use IMU to initialize and integrate the pose, and generate semantic subgraphs based on the segmented semantic information; use the corner prediction head to estimate the bounding box of the target object and track dynamic objects; In the process of tracking trajectories, the system continuously optimizes the subgraph to ensure the accuracy and robustness of the map. In this process, semantic information helps to remove dynamic objects and update the semantic subgraph. S4: Loop detection and global optimization: The system introduces loop detection and closed-loop correction mechanisms to ensure that the map maintains global consistency during long-term movement. Through global pose optimization of IMU data, the sub-map after closed-loop correction is jointly optimized to finally generate a global dense map.

2. The de-dynamic SLAM method based on multi-sensor and polarized attention according to claim 1, characterized in that, In step S2, the PSA-Det segmentation module specifically includes a backbone network, PAFPN and Head parts; wherein PAFPN represents an enhanced feature pyramid network.

3. The de-dynamic SLAM method based on multi-sensor and polarized attention according to claim 2, characterized in that, In step S2, the backbone network is a CSPNeXtPSA backbone network with large kernel depth convolution, which is used to extract multi-layer features of different scales; wherein CSPNeXtPSA represents a backbone block integrating a polarized self-attention mechanism.

4. The de-dynamic SLAM method based on multi-sensor and polarized attention according to claim 3, characterized in that, In step S2, the CSPNeXtPSA consists of 3x3 convolution BN SiLU, 5x5 depth convolution BN SiLU and PSA.

5. The de-dynamic SLAM method based on multi-sensor and polarized attention according to claim 2, characterized in that, In step S2, the PAFPN is used to fuse feature maps of different scales from the backbone network, specifically: the fusion process between feature maps is represented by multiple splicing and resizing operations; the expression ability of multi-scale features is enhanced by up-sampling and down-sampling and splicing features of different levels; and the convolution operation further extracts information on the fused features.

6. The de-dynamic SLAM method based on multi-sensor and polarized attention according to claim 2, characterized in that, In step S2, the Head part is used to simultaneously complete target detection and instance segmentation, and generate a bounding box and a corresponding segmentation mask for each target.

Images