A dynamic environment visual slam positioning method, system, device, medium and product
By combining a lightweight environment perception model with the LiftFeat feature extraction network, the problem of insufficient localization accuracy and robustness of visual SLAM algorithms in dynamic environments is solved, and efficient dynamic feature point selection is achieved on resource-constrained platforms, which is applicable to fields such as service robots and autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA AGRI UNIV
- Filing Date
- 2026-04-23
- Publication Date
- 2026-07-21
AI Technical Summary
Existing visual SLAM algorithms suffer from decreased localization accuracy and insufficient robustness in dynamic environments. Existing methods rely on predefined dynamic category labels, which can lead to the accidental deletion of feature points or the retention of undetected dynamic feature points. Furthermore, they have high computational complexity and are difficult to deploy on resource-constrained platforms.
The backbone network, feature extraction module, and context aggregation module are reconstructed using the lightweight environment-aware model YOLOv11 framework. Combined with the LiftFeat deep learning feature extraction network, dynamic feature points are accurately removed through semantic recognition and multi-constrained region similarity measurement. Lightweight large-kernel separable convolution and spatial pyramid pooling modules are used to improve detection performance.
While ensuring real-time performance, it significantly improves positioning accuracy and robustness in dynamic environments, making it suitable for resource-constrained robots and embedded devices, as well as service robots, autonomous driving, and other fields.
Smart Images
Figure CN122435002A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer vision and robotics, and in particular to a dynamic environment visual SLAM localization method, system, device, medium and product. Background Technology
[0002] Visual Simultaneous Localization and Mapping (VSLAM), a key technology in mobile robotics, augmented reality, and autonomous driving, utilizes image information acquired by cameras to estimate its own pose and construct an environmental map. Current mainstream visual SLAM algorithms are typically based on the assumption of a static scene, meaning that most feature points in the environment originate from rigid structures that do not move. Under this assumption, relatively stable localization and mapping results can be achieved through feature extraction, feature matching, and geometric constraint optimization.
[0003] However, in real-world indoor environments, dynamic targets such as pedestrians and mobile devices are common, and their movement introduces a large number of dynamic feature points that do not satisfy the static scene assumptions. These dynamic feature points participate in the pose estimation process, which can easily violate geometric consistency constraints, leading to decreased positioning accuracy, increased trajectory drift, and even system tracking failure. This seriously affects the robustness and reliability of visual SLAM systems in complex real-world scenarios.
[0004] To address interference from dynamic environments, existing research has proposed incorporating semantic information into visual SLAM systems. This involves identifying dynamic target regions within the scene using object detection or semantic segmentation methods, and then removing feature points from these dynamic regions to reduce the impact of dynamic factors on the system. However, these methods typically rely on predefined dynamic category labels and employ an indiscriminate overall removal strategy for feature points within dynamic target regions during feature selection. This approach can easily lead to the accidental deletion of valid feature points located within dynamic target regions but actually remaining stationary, while also potentially retaining some inaccurately detected dynamic feature points. This can weaken the accuracy of pose estimation and the overall stability of the system to some extent.
[0005] In addition, some dynamic SLAM methods introduce complex deep learning models, which improve robustness in dynamic scenarios to some extent, but often lead to high computational overhead, reduced system real-time performance, and are not conducive to deployment and application on resource-constrained platforms.
[0006] Therefore, how to achieve refined discrimination and effective screening of feature points in dynamic environments while ensuring the real-time performance of the system remains a technical problem that urgently needs to be solved in the field of visual SLAM. Summary of the Invention
[0007] The purpose of this application is to provide a dynamic environment visual SLAM localization method, system, device, medium and product, which can significantly improve the robustness and real-time performance in dynamic and complex environments while ensuring SLAM localization accuracy.
[0008] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a dynamic environment visual SLAM localization method, including: Based on the environmental color image of the target area, a lightweight environmental perception model is used for target semantic recognition, and the target area is divided into candidate dynamic areas and static areas. The lightweight environmental perception model is based on the YOLOv11 framework and is obtained by systematically reconstructing the backbone network, feature extraction module and context aggregation module. The LiftFeat deep learning feature extraction network is used to extract features from the environmental color image to obtain a set of feature points in the environmental color image. The feature points in the candidate dynamic region are marked as candidate dynamic feature points, and the feature points in the static region are marked as static feature points. Based on the environmental color image and environmental depth image of the target area, the candidate dynamic regions are subjected to multi-constraint region similarity measurement to determine the dynamic regions; The motion trajectory of candidate dynamic feature points within the dynamic region is modeled to obtain dynamic feature points, and the dynamic feature points in the feature point set are removed to obtain a static feature point set. SLAM localization is performed based on the set of static feature points to determine the robot's pose.
[0009] Secondly, this application provides a dynamic environment visual SLAM localization system, comprising: The environment perception module is used to perform target semantic recognition based on the environmental color image of the target area using a lightweight environment perception model, and to divide the target area into candidate dynamic regions and static regions. The lightweight environment perception model is based on the YOLOv11 framework and is obtained by systematically reconstructing the backbone network, feature extraction module and context aggregation module. The feature point extraction module is used to extract features from the environmental color image using the LiftFeat deep learning feature extraction network, obtain a set of feature points in the environmental color image, and mark the feature points in the candidate dynamic region as candidate dynamic feature points and the feature points in the static region as static feature points. The dynamic region determination module is used to perform multi-constraint region similarity measurement on the candidate dynamic regions based on the environmental color image and environmental depth image of the target region, and determine the dynamic region. The static feature point determination module is used to model the motion trajectory of candidate dynamic feature points in the dynamic region to obtain dynamic feature points, and to remove the dynamic feature points from the feature point set to obtain a static feature point set. The pose estimation module is used to perform SLAM localization based on the set of static feature points to determine the robot's pose.
[0010] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described dynamic environment visual SLAM localization method.
[0011] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described dynamic environment visual SLAM localization method.
[0012] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described dynamic environment visual SLAM localization method.
[0013] According to the specific embodiments provided in this application, this application achieves the following technical effects: By accurately eliminating dynamic feature points through semantic recognition and multi-constraint similarity measurement, the interference of dynamic targets on localization is avoided at the source, solving the key problems of pose estimation drift and accuracy degradation in dynamic environments, and ensuring the stability and reliability of robot localization results. A lightweight perception model is reconstructed based on YOLOv11 and paired with the LiftFeat feature extraction network, significantly reducing model computation and memory usage while maintaining the accuracy of feature extraction and semantic recognition. This method is suitable for terminal scenarios with limited computing power, such as robots and embedded devices, improving the practicality of the technology. This method does not rely on a specific hardware platform, and its refined processing logic for dynamic regions is compatible with dynamic environments of varying lighting and scene complexity. It is applicable to various fields requiring high-precision localization, such as service robots, autonomous driving, and intelligent inspection, and has broad application prospects. Attached Figure Description
[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0015] Figure 1 This is a schematic diagram of the overall process of a dynamic environment visual SLAM localization method provided in an embodiment of this application.
[0016] Figure 2 This is a framework diagram of a dynamic environment visual SLAM localization method provided in an embodiment of this application.
[0017] Figure 3 This is a detailed flowchart illustrating a dynamic environment visual SLAM localization method provided in an embodiment of this application.
[0018] Figure 4 This is a schematic diagram of the network structure of a lightweight environment perception model in one embodiment of this application.
[0019] Figure 5 This is a schematic diagram of a lightweight, large-kernel separable convolutional network structure in one embodiment of this application.
[0020] Figure 6 This is a schematic diagram of the network structure of the lightweight C3 feature extraction module in one embodiment of this application.
[0021] Figure 7 This is a schematic diagram of the network structure of a lightweight spatial pyramid pooling module in one embodiment of this application.
[0022] Figure 8 This is a schematic diagram of the network structure of the LiftFeat deep learning feature extraction network in one embodiment of this application.
[0023] Figure 9 This is a schematic diagram of the functional modules of a dynamic environment visual SLAM positioning system provided in an embodiment of this application.
[0024] Figure 10 This is a schematic diagram of a dynamic environment visual SLAM positioning device provided in an embodiment of this application. Detailed Implementation
[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0026] To address the issues of traditional visual SLAM algorithms, which typically assume a static environment and are prone to decreased localization accuracy and robustness in real-world indoor scenes with dynamic targets such as pedestrians, and existing dynamic SLAM methods, which rely on predefined dynamic category semantic information and indiscriminately remove feature points within dynamic target regions, thus erroneously deleting valid static feature points, this application provides a dynamic environment visual SLAM localization method, system, device, medium, and product. Compared to existing methods that rely solely on semantic information for feature removal, this application combines lightweight environmental perception with motion geometric constraints to achieve adaptive selection of feature points within dynamic regions, avoiding the loss of localization information caused by blind removal strategies. This application can significantly improve the system's localization accuracy and robustness in dynamic environments while maintaining good real-time performance.
[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0028] In one exemplary embodiment, such as Figures 1 to 3 As shown, a dynamic environment visual SLAM localization method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method includes steps 101 to 105.
[0029] Step 101: Based on the environmental color image of the target region, a lightweight environmental perception model is used to perform target semantic recognition, and the target region is divided into candidate dynamic regions and static regions. The lightweight environmental perception model is based on the YOLOv11 framework, and is obtained by systematically reconstructing the backbone network, feature extraction module, and context aggregation module.
[0030] Traditional dynamic SLAM algorithms directly employ deep learning networks for moving object detection. Deep learning networks have a massive number of parameters and require enormous computational resources, making them difficult to deploy on resource-constrained hardware. Furthermore, existing lightweight detection methods typically reduce computational complexity by decreasing the number of channels, reducing network depth, or using small-scale convolutional kernels. While these strategies reduce model parameters and computational complexity to some extent, they also significantly weaken the network's ability to model large receptive fields and long-range spatial dependencies, thus limiting detection performance. This is particularly true in lightweight YOLO models, where over-reliance on convolutions can lead to slow receptive field growth, limiting the utilization of global contextual information by high-level features. SLAM is used in the real world, and YOLOv11 has a mature and stable training and inference process, with wide applications in many real-world scenarios. Moreover, YOLOv11 maintains a good balance between detection accuracy, parameter size, and computational complexity, making it a commonly used and representative baseline model in lightweight object detection research. Optimizing the structure based on this model can more realistically reflect the impact of lightweight design on performance, avoiding the over-complexity or oversimplification of baseline models that might obscure the method's own contributions. Therefore, this application selects YOLOv11 as the basic framework and systematically improves its lightweight structure. Specifically, it proposes a lightweight environment perception model from three levels: convolution operators, feature extraction, and spatial pyramid pooling. This method achieves efficient large receptive field modeling and multi-scale contextual information fusion without significantly increasing computational overhead. The network structure of the lightweight environment perception model is as follows: Figure 4 As shown.
[0031] In a specific application example, the lightweight environment perception model replaces the CBS module in YOLOv11 with a lightweight large kernel separable convolution, replaces the C3K2 module with a lightweight C3 feature extraction module, and replaces the spatial pyramid pooling module with a lightweight spatial pyramid pooling module.
[0032] The lightweight environment perception model is based on the YOLOv11 framework, maintaining its mature multi-scale detection paradigm, feature fusion structure, and position-sensitive attention mechanism (C2 Position-Sensitive Attention, C2PSA) to ensure the network's engineering stability and reproducibility. Building upon this, this application systematically reconstructs the backbone network, feature extraction module, and context aggregation module, forming a unified lightweight operator system. Specifically, the backbone network mainly consists of lightweight large-kernel separable convolution (LiteLsConv) and a lightweight C3 feature extraction module (LiteC3LS) for efficient spatial feature modeling; a lightweight spatial pyramid pooling module (LiteSPPF) is introduced at high-level semantic features to enhance global context perception capabilities. Through this design, a lightweight environment perception model that balances efficiency and expressive power is constructed. This lightweight environment perception model can quickly perceive potential dynamic targets in images and obtain semantic information about targets in the environment without introducing additional engineering dependencies. Based on the obtained semantic information, the target region is divided into candidate dynamic regions and static regions.
[0033] The lightweight large-kernel separable convolution decomposes the traditional large-kernel convolution into an axial-depth separable convolution and introduces adaptive grouped pointwise convolution and an efficient channel attention mechanism.
[0034] To overcome the problem of limited receptive field caused by small convolutional kernels in lightweight networks, this application, inspired by LSNet, proposes a lightweight large-kernel separable convolution (LiteLsConv). By decomposing traditional large-kernel convolutions into axially and depthwise separable convolutions and introducing adaptive grouped pointwise convolutions, computational complexity and parameter count are effectively reduced, while enhancing the network's ability to model long-range spatial dependencies. The network structure of LiteLsConv is as follows: Figure 5 As shown, LiteLsConv does not simply stack convolutional layers, but rather efficiently reconstructs traditional large-kernel convolutions at the operator level. It employs an axially depthwise separable convolution (DSC) structure, transforming the traditional... The convolution is decomposed into two consecutive depthwise convolutions (DCs). The kernel sizes are respectively... and Do this part first. Horizontal axis convolution, then perform... Vertical axis convolution. This axial decomposition method maintains the ability to model large-scale spaces while reducing computational complexity from... Significantly reduced to This makes large receptive field modeling a feasible option in lightweight networks. To further reduce computational overhead, this application employs a grouping approach to implement pointwise convolution (PC). The number of groups is adaptively determined by the greatest common divisor of the input and output channels. The kernel sizes are as follows: This strategy effectively reduces the number of floating-point operations while maintaining sufficient inter-channel interaction. Furthermore, LiteLsConv integrates an efficient channel attention (ECA) mechanism. It consists of one-dimensional convolutions and has no fully connected layers. This attention mechanism adaptively emphasizes key channel features without significantly impacting the overall computational complexity.
[0035] ECA is not a single one-dimensional convolutional layer, but rather an end-to-end channel attention architecture centered on one-dimensional convolution, combined with global pooling and activation functions. This one-dimensional convolutional layer differs fundamentally from ordinary convolution: its input is global statistical features in the channel dimension, acting only on the channel sequence to capture local dependencies between adjacent channels, rather than acting on the spatial / temporal dimensions as in ordinary convolution; furthermore, its input and output channels are both 1, with no bias, and padding ensures the channel number remains constant. Its design goal is to generate lightweight channel attention weights, rather than extracting spatial features. In contrast, ordinary convolutions (two-dimensional / one-dimensional) typically act on the spatial / temporal dimensions, have a larger number of parameters, and their core goal is feature extraction rather than attention weight generation.
[0036] In summary, the lightweight large-kernel separable convolution provided in this application includes sequentially connected axial depth separable convolutions, batch normalization layers, activation functions, and an efficient channel attention mechanism. The axial depth separable convolutions include sequentially connected horizontal axial depth convolutions, vertical axial depth convolutions, and pointwise convolutions.
[0037] The lightweight C3 feature extraction module embeds the lightweight large kernel separable convolution in the main branch of the C3 module.
[0038] Based on LiteLsConv, this application further constructs a lightweight C3 feature extraction module (LiteC3LS). While retaining the dual-path characteristics and gradient stability of the C3 structure, LiteLsConv is embedded as the core operator in the main branch, achieving efficient feature reuse and lightweight representation, suitable for deep feature extraction. The network structure of LiteC3LS is as follows: Figure 6As shown, LiteLsConv is a lightweight variant of the C3 module. LiteC3LS retains the dual-path structure of the original C3 module. By dividing the feature mapping into a main branch and a sub-branch, it improves feature reuse efficiency while ensuring stable gradient propagation. The main branch uses LiteLsConv as the core operator, allowing the network to continuously expand its effective receptive field as it deepens, thereby enhancing its spatial modeling ability. The sub-branch uses a simple 1×1 convolution to preserve the original feature information and avoid excessive information loss. The outputs of the two branches are connected and fused through pointwise convolutions. LiteLsConv significantly enhances the spatial representation ability of deep features with almost no additional computational burden.
[0039] Specifically, the lightweight C3 feature extraction module includes a main branch, a sub-branch, and a splicing and fusion module. Both the main branch and the sub-branch are connected to the splicing and fusion module. The main branch includes two sequentially connected lightweight large-kernel separable convolutions. The sub-branch includes sequentially connected two-dimensional convolutions, batch normalization layers, and activation functions. The splicing and fusion module includes sequentially connected connection layers, two-dimensional convolutions, batch normalization layers, and activation functions.
[0040] The lightweight spatial pyramid pooling module, based on the spatial pyramid pooling module, compresses the channels of the input features using a controllable compression ratio through the first convolutional layer, and uses parallel multi-scale pooling and depth-separable fusion strategies to achieve global context aggregation with low computational cost.
[0041] Traditional SPPF modules are widely used to enhance global context awareness, but their channel expansion and computational overhead in high-dimensional feature maps limit their application in lightweight models. To address this issue, this application designs a lightweight spatial pyramid pooling module (LiteSPPF), a lightweight redesign of the traditional SPPF structure. Through parallel multi-scale pooling, controllable compression ratio, and a depth-separable fusion strategy, low-computational-cost global context aggregation is achieved. The network structure of LiteSPPF is as follows: Figure 7As shown. The original SPPF relies on a serial design. Unlike the original SPPF, LiteSPPF first uses a controllable compression ratio to compress feature channels, thereby significantly limiting channel expansion during multi-scale aggregation. The controllable compression ratio is achieved as follows: a configurable parameter `reduction` is introduced during module initialization. The number of intermediate channels after compression is the quotient of the number of input channels `c1` and `reduction`. The first convolutional layer maps the number of input feature channels from `c1` to `c2`, completing channel compression. By adjusting the value of `reduction` (e.g., 2, 3, 4), the channel compression ratio (e.g., 1 / 2, 1 / 3, 1 / 4) can be flexibly controlled. Compared to the original SPPF's fixed compression ratio of 1 / 2, this design achieves configurable compression, thus controllably reducing the channel dimension after subsequent multi-scale pooling feature concatenation (the number of channels after concatenation is `c2×4`), ultimately achieving lightweight design.
[0042] Subsequently, multiple max-pooling operations of different scales are employed in parallel to extract contextual information from different receptive fields. This avoids the inference latency introduced by sequential pooling structures. After concatenating the multi-scale features, efficient fusion is achieved using depthwise separable pointwise convolutions, significantly reducing computational complexity. Finally, a ShuffleAttention mechanism is introduced to recalibrate the fused features, compensating for the loss of local discriminative information caused by pooling operations, thereby improving feature representation quality with low computational cost.
[0043] Specifically, the lightweight spatial pyramid pooling module includes a first convolutional layer, a first max pooling layer, a second max pooling layer, a third max pooling layer, a connection layer, a second convolutional layer, and a permutation attention mechanism. The first convolutional layer is connected to the first max pooling layer, the second max pooling layer, the third max pooling layer, and the connection layer. The first max pooling layer, the second max pooling layer, and the third max pooling layer are all connected to the connection layer. The connection layer, the second convolutional layer, and the permutation attention mechanism are connected sequentially.
[0044] The three detection heads of the lightweight environment perception model are a typical multi-scale object detection architecture, each corresponding to a different feature map scale: (1) 80×80 detector head: used to predict small targets and output the bounding box and category of small targets.
[0045] (2) 40×40 detector head: used to predict medium-sized targets and output the bounding box and category of medium-sized targets.
[0046] (3) 20×20 detector head: used to predict large targets and output the bounding box and category of large targets.
[0047] For all prediction results from the three detection heads: first, low-confidence candidate boxes are filtered out by a confidence threshold, then non-maximum suppression is used to eliminate duplicate detections of the same target, and finally each bounding box is retained with its optimal semantic category label and confidence score, thus completing the final determination of the target semantic information.
[0048] The semantic information of the target mainly includes bounding box information and category information. Bounding box regions categorized as vehicles, people, etc., are considered candidate dynamic regions, while other bounding box regions are considered static regions.
[0049] It should be noted that the lightweight environment perception model provided in this application is not a simple aggregation of single modules, but rather a collaborative optimization of multiple lightweight strategies from an overall architectural perspective. LiteLsConv provides the network with efficient large receptive field modeling capabilities, LiteC3LS builds a stable and efficient feature extraction structure on this basis, and LiteSPPF further enhances the utilization of global contextual information. The three work together to enable the lightweight environment perception model to maintain its lightweight characteristics while possessing strong detection performance. This application provides a feasible and practical solution for efficient object detection in resource-constrained scenarios.
[0050] Step 102: Use the LiftFeat deep learning feature extraction network to extract features from the environmental color image to obtain a set of feature points in the environmental color image. Mark the feature points in the candidate dynamic region as candidate dynamic feature points and the feature points in the static region as static feature points.
[0051] To improve the stability and discriminative ability of feature point extraction in complex environments, this application employs the LiftFeat deep learning feature extraction network to extract feature points from environmental images. Unlike traditional SLAM systems that rely on manually designed feature operators, this application uses a trained feature extraction network to automatically learn multi-level discriminative features from the input image, thereby enhancing the robustness of feature points under conditions of illumination variation, viewpoint variation, and dynamic disturbance. The structure of the LiftFeat deep learning feature extraction network is as follows: Figure 8 As shown.
[0052] The LiftFeat deep learning feature extraction network takes the current frame image as input and encodes the spatial structure and semantic information of the image layer by layer through 5 convolutional layers, outputting a feature representation containing feature point locations, normals, and corresponding feature descriptors. Each of the 5 convolutional layers is composed of... It consists of convolutional layers and max-pooling layers with a stride of 2. The feature map output by convolutional module 5 has a spatial resolution of [missing information]. ,in and These represent the width and height of the input image, respectively. The depth of the feature maps output by each convolutional module increases progressively, with the output depths of the five convolutional modules being respectively... Subsequently, the fusion module uses Convolution and bilinear interpolation are aligned and summed using the low-level features of convolution modules 3, 4, and 5 to obtain a size of [size missing]. The fusion feature map.
[0053] Feature nodding use Convolution generates a size of The feature point map is then processed, and a channel-level softmax operation is performed to obtain the feature point score distribution at the original image resolution. The descriptor head is obtained using bilinear interpolation and L2 normalization to achieve a size of [missing value]. The descriptor map is obtained. Similarly, the normal head uses bilinear interpolation to obtain a 3-channel map with the same resolution as the original image.
[0054] In actual operation, the LiftFeat deep learning feature extraction network, as part of the front-end of the visual SLAM system, works in conjunction with subsequent feature point motion trajectory analysis and pose estimation. The feature points extracted in this way can provide more stable and reliable input for subsequent motion consistency judgment and pose estimation, thereby further improving the localization accuracy and overall robustness in dynamic environments.
[0055] Step 103: Based on the environmental color image and environmental depth image of the target region, perform multi-constraint region similarity measurement on the candidate dynamic regions to determine the dynamic regions. To determine the true motion state of potential dynamic targets in the current frame, this application performs similarity detection on candidate dynamic regions to complete target region-level motion state discrimination and obtain the dynamic regions. The multi-constraint region similarity measurement is mainly used to evaluate the comprehensive consistency between the template region and the candidate dynamic regions in terms of appearance, structure, and pixel depth.
[0056] In a specific application example, step 103 includes steps 31 to 37.
[0057] Step 31: Use the candidate dynamic region as a template region, and search for the set of candidate regions that are most similar to the template region in the environmental color image.
[0058] First, at any time Acquire environmental color images and environmental depth images The lightweight environment perception model quickly perceives potential dynamic targets in environmental color images without introducing additional engineering dependencies, and outputs a set of candidate dynamic regions: ;in, Indicates the first A template region for each potential dynamic target. Then, for a given template region... and environmental color images Search for the set of candidate regions in the environmental color image that are most similar to the template region. This application models region similarity as a multi-constraint joint metric function, which is used to evaluate the overall consistency between template regions and candidate regions in terms of appearance, structure, and geometry.
[0059] Step 32: For any candidate region in the candidate region set, extract the feature descriptors of the candidate region and the template region respectively, and calculate the cosine similarity between the two feature descriptors to obtain the appearance similarity between the candidate region and the template region. This metric is robust to changes in illumination and can effectively measure region-level semantic consistency. To assess the visual similarity between candidate regions and template regions, For the feature descriptor of the candidate region, This is a feature descriptor for the template region.
[0060] Step 33: Calculate the structural similarity between the candidate region and the template region based on the grayscale values of the candidate region and the template region. Structural similarity measures the similarity between the template region and the candidate region at the local structural level. It comprehensively evaluates the structural consistency of local regions by considering brightness, contrast, and structural information, thereby suppressing mismatches caused by local texture similarity.
[0061] To further constrain the structural consistency of local regions, this application introduces a structural similarity index to measure the template region. With candidate regions The formula for calculating similarity at the local structural level is as follows: ;in, To assess the structural similarity between candidate regions and template regions, The average grayscale value of the template area. The average gray value of the candidate region. The grayscale variance of the template region. The gray-level variance of the candidate region. The gray-level covariance between the candidate region and the template region. and It is the stability constant.
[0062] Step 34: Determine the average pixel depth within the candidate region based on the environmental depth image. and the average pixel depth within the template region And based on the average pixel depth in the candidate region and the average pixel depth in the template region, the depth consistency between the candidate region and the template region is calculated: .in, To ensure depth consistency between candidate regions and template regions. This is the depth scale factor. By using depth consistency, regions with consistent positions in 3D space are classified as similar regions, effectively suppressing mismatched regions with inconsistent depth space.
[0063] Step 35: Calculate the region motion similarity between the candidate region and the template region based on their appearance similarity, structural similarity, and depth consistency. .in, For the first n Regional motion similarity between candidate regions and template regions Weights for appearance similarity Weights for structural similarity. For the weights of depth consistency, .
[0064] Step 36: Calculate the region motion similarity threshold based on the region motion similarity between the template region and each candidate region in the candidate region set. For all template region and candidate region pairs in the current frame, calculate their joint similarity score; between adjacent frames, assume that the region motion similarity follows an approximately Gaussian distribution; based on the statistical significance criterion, obtain the region motion similarity threshold.
[0065] In this application, the regional motion similarity threshold is... Instead of using a fixed threshold, the threshold is adaptively determined based on the statistical distribution of similarity scores within each frame. Therefore, this application uniformly models region similarity determination as an adaptive decision problem based on statistical distribution. For all template region and candidate region pairs in the current frame, their joint similarity score set is calculated. : ;in, N Let be the number of candidate regions. Assume that the motion similarity of the regions follows an approximately Gaussian distribution. ,in, , Based on the statistical significance criterion, the adaptive dynamic region discrimination condition is defined as follows: ;in, This is a scaling factor used to control the confidence level of dynamic region detection. This strategy adaptively adjusts the discrimination boundary based on the overall similarity distribution of the current scene.
[0066] Step 37: Based on the region motion similarity between the template region and each candidate region in the candidate region set, and the region motion similarity threshold, dynamic regions are selected from the candidate region set. Specifically, when the region motion similarity is greater than the region motion similarity threshold, the candidate region is determined to be a dynamic region. This determination does not depend on the category label, but is based on the temporal consistency between regions.
[0067] Step 104: Model the motion trajectory of the candidate dynamic feature points in the dynamic region to obtain dynamic feature points, and remove the dynamic feature points from the feature point set to obtain a static feature point set.
[0068] After determining the motion state at the target region level, further analysis of the true motion attributes of feature points within the target region is still necessary. Since dynamic targets may simultaneously contain static structures or short-term stable features, simply removing them based on semantics or single-frame motion information can easily lead to the loss of effective geometric constraints. Therefore, this application models the motion trajectories of feature points and uses a motion consistency discrimination method based on the similarity of feature point motion trajectories to perform refined motion state discrimination of feature points within the target region, thus obtaining dynamic feature points.
[0069] Specifically, for dynamic regions, motion trajectory modeling of feature points is used to model the motion trajectory of candidate dynamic feature points, thereby refining the motion state of feature points within the target region and obtaining dynamic feature points; all feature points determined to be dynamic will be removed from the feature set of the current frame, and only static feature points will be retained for pose estimation to realize SLAM localization function, thereby effectively suppressing the interference of dynamic objects on SLAM localization accuracy.
[0070] In a specific application example, the motion trajectory modeling of candidate dynamic feature points in the dynamic region to obtain dynamic feature points includes the following steps 41 to 46.
[0071] Step 41: For any candidate dynamic feature point in the dynamic region, track the pixel position of the candidate dynamic feature point in multiple consecutive frames of environmental color images to obtain a two-dimensional motion trajectory sequence of the candidate dynamic feature point.
[0072] At any moment Using a length of Sliding time window The motion trajectories of feature points are constructed, a definition that ensures both the causality of online SLAM and the consistency of trajectory sequence length. For candidate dynamic feature points located within a dynamic region... In continuous Track the pixel positions within the frame to construct a two-dimensional motion trajectory sequence. : ;in, Representing candidate dynamic feature points In the Pixel coordinates in a frame.
[0073] Step 42: Based on the camera pose sequence and the depth information of the candidate dynamic feature points, back-project the two-dimensional motion trajectory sequence to three-dimensional space, and predict the reference motion trajectory of the candidate dynamic feature points based on the camera motion.
[0074] Under the assumption of a static scene, the motion of feature points is mainly caused by the camera's own motion, and its change trend in the time dimension should be consistent with the camera motion model; however, for feature points on dynamic targets, their trajectories usually exhibit non-rigid changes that are inconsistent with the camera motion.
[0075] This paper assumes that static feature points in the scene satisfy a rigid body motion model determined by changes in camera pose. To characterize the ideal motion trajectory under the static assumption, this application utilizes the camera pose sequence estimated by the front end. The feature points are back-projected in 3D using depth information, and their pixel positions in subsequent frames are predicted based on camera pose changes. Therefore, the predicted positions of the feature points under the static assumption are: ;in, This indicates a projection operation. This indicates a back projection operation. The depth is the feature point depth.
[0076] Subsequently, the camera pose transformation is used to project it back onto the image plane frame by frame to obtain the reference motion trajectory: This trajectory reflects the expected motion behavior of the feature points under the static world assumption.
[0077] Step 43: The similarity between the two-dimensional motion trajectory sequence and the reference motion trajectory is calculated using the Dynamic Time Warping (DTW) method.
[0078] Due to issues such as short-term occlusion, interruption of feature tracking, and inconsistent time sampling in actual observed trajectories, this application employs dynamic time warping to measure the similarity between trajectories. The local distance between trajectory points is defined as: .in, Candidate dynamic feature points In the Pixel coordinates in a frame Candidate dynamic feature points In the Pixel coordinates in a frame.
[0079] Step 44: Determine the motion consistency score of the candidate dynamic feature points based on the similarity between the two-dimensional motion trajectory sequence and the reference motion trajectory.
[0080] Let the DTW cumulative cost matrix be... ,but: .
[0081] The initialization conditions are: .
[0082] Finally, candidate dynamic feature points are obtained. Motion consistency score : ;in, The optimal alignment path for DTW.
[0083] Step 45: Construct an adaptive discrimination criterion based on the motion consistency scores of each candidate dynamic feature point within the dynamic region. Specifically, first, calculate the mean and variance of the consistency scores of the candidate dynamic feature points within the dynamic region; then, construct the adaptive discrimination criterion based on the obtained mean and variance.
[0084] To avoid the sensitivity of a fixed threshold to different scenarios, an online statistical model is used for dynamic discrimination. At time... For the set of feature point consistency scores across all potential dynamic regions: Calculate its mean and variance: ; Where M is the number of candidate dynamic feature points.
[0085] Define the adaptive discrimination criterion: ;in, This is the stability coefficient.
[0086] Step 46: Based on the motion consistency score of each candidate dynamic feature point within the dynamic region, the adaptive discrimination criterion is used to determine whether each candidate dynamic feature point is a dynamic feature point. When the motion consistency score is less than the adaptive discrimination criterion, the candidate dynamic feature point is determined to be a dynamic feature point.
[0087] Step 105: Perform SLAM localization based on the set of static feature points to determine the robot's pose.
[0088] Compared with existing technologies, this application introduces a lightweight environment perception model to quickly detect potential dynamic targets in a scene. Based on this, it combines feature motion trajectory analysis and a camera motion model to perform feature motion consistency judgment on feature points within the dynamic target area. This enables adaptive filtering of dynamic and static feature points, avoiding the problem of erroneous deletion of effective static feature points caused by relying solely on semantic information to remove all feature points. It effectively suppresses the violation of geometric constraints by dynamic feature points participating in pose estimation, significantly improving the localization accuracy and operational stability of the visual SLAM system in dynamic environments. Simultaneously, this application employs a lightweight network structure for environmental semantic perception, effectively reducing computational complexity while ensuring dynamic target detection performance. It can balance real-time performance and robustness without increasing additional engineering dependencies, demonstrating good engineering feasibility and application promotion value. This application can significantly improve the localization accuracy and robustness of visual SLAM in dynamic environments while maintaining good real-time performance.
[0089] Based on the same inventive concept, this application also provides a dynamic environment visual SLAM localization system for implementing the dynamic environment visual SLAM localization method described above. The solution provided by this system is similar to the implementation described in the above method; therefore, the specific limitations of one or more dynamic environment visual SLAM localization system embodiments provided below can be found in the limitations of the dynamic environment visual SLAM localization method described above, and will not be repeated here.
[0090] In one exemplary embodiment, such as Figure 9 As shown, a dynamic environment visual SLAM localization system is provided, including: an environment perception module 901, a feature point extraction module 902, a dynamic region determination module 903, a static feature point determination module 904, and a pose estimation module 905.
[0091] The environment perception module 901 is used to perform target semantic recognition based on the environmental color image of the target area using a lightweight environment perception model, and to divide the target area into candidate dynamic regions and static regions. The lightweight environment perception model is based on the YOLOv11 framework, and is obtained by systematically reconstructing the backbone network, feature extraction module, and context aggregation module. Specifically, the environment perception module 901 efficiently detects potential dynamic targets in the scene while ensuring real-time performance, and provides reliable semantic priors for subsequent feature selection.
[0092] The feature point extraction module 902 is used to extract features from the environmental color image using the LiftFeat deep learning feature extraction network, obtain a set of feature points in the environmental color image, and mark the feature points in the candidate dynamic region as candidate dynamic feature points and the feature points in the static region as static feature points.
[0093] The dynamic region determination module 903 is used to determine the dynamic region by performing a multi-constraint region similarity measurement on the candidate dynamic regions based on the environmental color image and environmental depth image of the target region.
[0094] The static feature point determination module 904 is used to model the motion trajectory of candidate dynamic feature points in the dynamic region to obtain dynamic feature points, and to remove dynamic feature points from the feature point set to obtain a static feature point set.
[0095] Specifically, the dynamic region determination module 903 refines the judgment of candidate dynamic regions through multi-constraint region similarity measurement to obtain dynamic regions. The static feature point determination module 904 analyzes the motion trajectory of feature points in continuous frames and combines it with the camera motion model to mark feature points that do not meet the motion trajectory consistency constraint as dynamic feature points, thereby effectively distinguishing static feature points from dynamic feature points within the dynamic target region.
[0096] The pose estimation module 905 is used to perform SLAM localization based on the set of static feature points to determine the robot's pose.
[0097] In one exemplary embodiment, a dynamic environment visual SLAM localization device is provided, such as... Figure 10 As shown, the device primarily uses an NVIDIA Xavier NX as its core processor, to which an RGB-D camera and a display are connected. The aforementioned dynamic environment visual SLAM localization system is deployed within the processor. The RGB-D camera is mainly used to acquire information about the surrounding environment. The display is primarily used to show the pose state estimated by the dynamic environment visual SLAM localization system.
[0098] Specifically, the processor's operating system is Ubuntu 18.04; it includes a processor and memory; the processor primarily implements the aforementioned dynamic environment visual SLAM localization method. It also includes an image processor and a central processing unit (CPU). The image processor, deployed within the core processor of the SLAM system, is primarily responsible for parallel computation of tasks, accelerating system execution speed and the inference speed of the dynamic target recognition model. The CPU is mainly responsible for tasks such as basic instruction transmission and reception, overall process control, and communication with external devices. The memory primarily stores executable program code and saves temporary data, configuration files, map data, and feature data required by the processor when executing the executable code.
[0099] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0100] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0101] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0103] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.
[0104] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0105] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.
[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0107] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A dynamic environment visual SLAM localization method, characterized in that, The dynamic environment visual SLAM localization method includes: Based on the environmental color image of the target area, a lightweight environmental perception model is used for target semantic recognition, and the target area is divided into candidate dynamic areas and static areas. The lightweight environmental perception model is based on the YOLOv11 framework and is obtained by systematically reconstructing the backbone network, feature extraction module and context aggregation module. The LiftFeat deep learning feature extraction network is used to extract features from the environmental color image to obtain a set of feature points in the environmental color image. The feature points in the candidate dynamic region are marked as candidate dynamic feature points, and the feature points in the static region are marked as static feature points. Based on the environmental color image and environmental depth image of the target area, the candidate dynamic regions are subjected to multi-constraint region similarity measurement to determine the dynamic regions; The motion trajectory of candidate dynamic feature points within the dynamic region is modeled to obtain dynamic feature points, and the dynamic feature points in the feature point set are removed to obtain a static feature point set. SLAM localization is performed based on the set of static feature points to determine the robot's pose.
2. The dynamic environment visual SLAM localization method according to claim 1, characterized in that, The lightweight environment perception model replaces the CBS module in YOLOv11 with a lightweight large kernel separable convolution, replaces the C3K2 module with a lightweight C3 feature extraction module, and replaces the spatial pyramid pooling module with a lightweight spatial pyramid pooling module. The lightweight large kernel separable convolution decomposes the traditional large kernel convolution into axial depth separable convolution and introduces adaptive grouped pointwise convolution and efficient channel attention mechanism. The lightweight C3 feature extraction module embeds the lightweight large-kernel separable convolution in the main branch of the C3 module; The lightweight spatial pyramid pooling module, based on the spatial pyramid pooling module, compresses the channels of the input features using a controllable compression ratio through the first convolutional layer, and uses parallel multi-scale pooling and depth-separable fusion strategies to achieve global context aggregation with low computational cost.
3. The dynamic environment visual SLAM localization method according to claim 2, characterized in that, The lightweight large-kernel separable convolution includes sequentially connected axial depth separable convolutions, batch normalization layers, activation functions, and efficient channel attention mechanisms; the axial depth separable convolution includes sequentially connected horizontal axial depth convolutions, vertical axial depth convolutions, and pointwise convolutions.
4. The dynamic environment visual SLAM localization method according to claim 2, characterized in that, The lightweight C3 feature extraction module includes a main branch, a sub-branch, and a splicing and fusion module; both the main branch and the sub-branch are connected to the splicing and fusion module; the main branch includes two sequentially connected lightweight large-kernel separable convolutions; the sub-branch includes sequentially connected two-dimensional convolutions, batch normalization layers, and activation functions; the splicing and fusion module includes sequentially connected connection layers, two-dimensional convolutions, batch normalization layers, and activation functions.
5. The dynamic environment visual SLAM localization method according to claim 1, characterized in that, Based on the environmental color image and environmental depth image of the target region, the candidate dynamic regions are subjected to multi-constraint region similarity measurement to determine the dynamic regions, including: The candidate dynamic region is used as a template region, and a set of candidate regions most similar to the template region is searched in the environmental color image; For any candidate region in the candidate region set, the feature descriptors of the candidate region and the template region are extracted respectively, and the cosine similarity between the two feature descriptors is calculated to obtain the appearance similarity between the candidate region and the template region. Calculate the structural similarity between the candidate region and the template region based on the gray values of the candidate region and the template region; Based on the environmental depth image, the average pixel depth in the candidate region and the average pixel depth in the template region are determined, and the depth consistency between the candidate region and the template region is calculated based on the average pixel depth in the candidate region and the average pixel depth in the template region. Based on the appearance similarity, structural similarity, and depth consistency between the candidate region and the template region, the regional motion similarity between the candidate region and the template region is calculated. Calculate the region motion similarity threshold based on the region motion similarity between the template region and each candidate region in the candidate region set; Based on the region motion similarity between the template region and each candidate region in the candidate region set, and the region motion similarity threshold, dynamic regions are selected from the candidate region set.
6. The dynamic environment visual SLAM localization method according to claim 1, characterized in that, Motion trajectory modeling is performed on candidate dynamic feature points within the dynamic region to obtain dynamic feature points, including: For any candidate dynamic feature point within the dynamic region, the pixel position of the candidate dynamic feature point is tracked in multiple consecutive frames of environmental color images to obtain a two-dimensional motion trajectory sequence of the candidate dynamic feature point. Based on the camera pose sequence and the depth information of the candidate dynamic feature points, the two-dimensional motion trajectory sequence is back-projected into three-dimensional space, and the reference motion trajectory of the candidate dynamic feature points is predicted based on the camera motion. The similarity between the two-dimensional motion trajectory sequence and the reference motion trajectory is calculated using a dynamic time warping method; Based on the similarity between the two-dimensional motion trajectory sequence and the reference motion trajectory, the motion consistency score of the candidate dynamic feature points is determined; An adaptive discrimination criterion is constructed based on the motion consistency score of each candidate dynamic feature point in the dynamic region. Based on the motion consistency score of each candidate dynamic feature point within the dynamic region, the adaptive discrimination criterion is used to determine whether each candidate dynamic feature point is a dynamic feature point.
7. A dynamic environment visual SLAM localization system, applied to the dynamic environment visual SLAM localization method according to any one of claims 1-6, characterized in that, The dynamic environment visual SLAM localization system includes: The environment perception module is used to perform target semantic recognition based on the environmental color image of the target area using a lightweight environment perception model, and to divide the target area into candidate dynamic regions and static regions. The lightweight environment perception model is based on the YOLOv11 framework and is obtained by systematically reconstructing the backbone network, feature extraction module and context aggregation module. The feature point extraction module is used to extract features from the environmental color image using the LiftFeat deep learning feature extraction network, obtain a set of feature points in the environmental color image, and mark the feature points in the candidate dynamic region as candidate dynamic feature points and the feature points in the static region as static feature points. The dynamic region determination module is used to perform multi-constraint region similarity measurement on the candidate dynamic regions based on the environmental color image and environmental depth image of the target region, and determine the dynamic region. The static feature point determination module is used to model the motion trajectory of candidate dynamic feature points in the dynamic region to obtain dynamic feature points, and to remove the dynamic feature points from the feature point set to obtain a static feature point set. The pose estimation module is used to perform SLAM localization based on the set of static feature points to determine the robot's pose.
8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the dynamic environment visual SLAM localization method according to any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dynamic environment visual SLAM localization method as described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the dynamic environment visual SLAM localization method as described in any one of claims 1-6.