Dynamic semantic vision SLAM (Simultaneous Localization and Mapping) method based on point and line feature adaptive weighting

Abstract

The invention discloses a dynamic semantic vision SLAM (Simultaneous Localization and Mapping) method based on point and line feature adaptive weighting. The method comprises the following steps: firstly, obtaining potential dynamic region prior by combining target detection and image segmentation; then adaptive weighting is carried out on point and line features in two stages: in the first stage, initial weighting is carried out based on motion consistency check and dynamic levels, and feature matching and pose preliminary estimation are realized; in the second stage, weighting is further carried out on the key frame level through multi-view consistency verification, and weighted optimization and map maintenance are combined; and finally, loopback verification and global optimization are completed based on weighted features, and cumulative drift is effectively eliminated. According to the method, interference of dynamic and low-credibility features is suppressed through differential feature weighting, the adaptability to an intermittent moving object is improved through a double-stage weighting mechanism, and the stability and observability of a low-texture scene are enhanced through point-line combined modeling.

Description

Technical Field

[0001] This invention relates to the field of semantic mining technology, specifically to a dynamic semantic visual SLAM method based on adaptive weighting of point and line features. Background Technology

[0002] In dynamic indoor environments, the uneven distribution of moving objects and structural features poses a significant challenge to the stability and accuracy of Visual Simultaneous Localization and Mapping (Visual SLAM). Most existing SLAM systems still treat point and line features from different objects and regions as equivalent and reliable observations, which can easily lead to pose estimation errors and inaccuracies under dynamic disturbances or low-texture conditions. Figure 1 Consistency degradation. To address dynamic interference or low-texture issues, existing technologies primarily employ the following improvement strategies:

[0003] One type of method removes anomalous feature points by using geometric consistency constraints or reprojection error thresholds to reduce the impact of dynamic objects on pose estimation. However, this type of method is highly dependent on the quality of initialization and has difficulty distinguishing between real dynamic objects and pseudo-dynamic features caused by factors such as changes in viewpoint and illumination, which can easily lead to false deletions or false misses.

[0004] Another approach incorporates object detection or semantic segmentation techniques to directly remove features considered to be on dynamic objects based on object category information. While semantic information can improve the recall rate of dynamic object detection, existing solutions typically perform hard deletion on the entire object category, ignoring the possibility that the same object may be in a static or moving state at different times. Furthermore, semantic false detections or inaccurate segmentation boundaries can easily lead to the erroneous removal of a large number of reliable static features.

[0005] Furthermore, some methods attempt to introduce line features or other geometric features to enhance the system's stability in low-texture environments. However, in practical applications, point features and line features are often treated equally, failing to fully consider the reliability differences of different objects, regions, and feature types in dynamic scenes. Existing technologies generally lack an effective mechanism that can comprehensively consider semantic information, motion state, and feature quality to finely adjust the contribution of features in pose estimation and map optimization processes.

[0006] Therefore, how to accurately identify potential dynamic regions in indoor dynamic environments, reasonably distinguish objects in different motion states, and dynamically adjust their impact on pose estimation and mapping processes based on the temporal stability and geometric reliability of features remains a technical problem that urgently needs to be solved in existing visual SLAM technology. Summary of the Invention

[0007] Purpose of the invention: The purpose of this invention is to provide a dynamic semantic visual SLAM method based on adaptive weighting of point and line features, which solves the problems of insufficient feature points, poor matching quality, low positioning accuracy, and low mapping quality in highly dynamic human-computer collaborative environments and scenes containing a large number of structured but textureless objects (such as walls, ceilings, etc.).

[0008] Technical solution: The dynamic semantic visual SLAM method based on adaptive weighting of point and line features described in this invention includes the following steps:

[0009] (1) The object detection model and the image segmentation model are used together to perform object detection and pixel-level segmentation on the input image sequence, and the pixel-level dynamic mask and mask metadata are output, including dynamic mask information such as category identifier, pixel location set and confidence parameter, which provides potential dynamic region priors for subsequent modules;

[0010] (2) Receive dynamic mask and mask metadata, perform motion consistency check and dynamic level classification based on optical flow residual and depth change rate, and perform first adaptive weighting of point and line features based on dynamic level and feature quality (reflecting the quality of features from the inherent attributes of features), and then complete feature matching and preliminary pose estimation.

[0011] (3) Perform multi-view consistency verification on point and line features at the key frame level, perform a second adaptive weighting based on reprojection error, geometric consistency, depth consistency and matching consistency, and perform weighted local optimization and map maintenance.

[0012] (4) Based on weighted appearance description and geometric consistency verification, generate and confirm closure constraints to suppress the interference of dynamic features on closure determination;

[0013] (5) Integrate loop constraints and weighted point and line feature errors, and perform weighted global optimization to eliminate cumulative drift and maintain global map consistency.

[0014] Furthermore, in step (1), an open environment target detection model and an image segmentation model are cascaded. First, a candidate region is generated through the target detection model, then the candidate region is segmented at the pixel level, and the overlapping region is fused through semantic priority rules to output a pixel-level dynamic mask and semantic metadata.

[0015] Furthermore, in step (2), the motion consistency check includes fusion optical flow consistency, reprojection error and depth change rate, motion verification of potential dynamic regions, and dynamic level classification based on motion intensity.

[0016] Furthermore, in step (2), the first adaptive weighting of point and line features is as follows: for point features, weighting is performed based on corner degree, neighborhood repeatability and depth confidence; for line features, weighting is performed based on line segment length, gradient consistency, depth smoothness and reprojection angle; the weighting results are used to adjust the feature matching priority and contribution in pose estimation.

[0017] Furthermore, in step (3), the second adaptive weighting is comprehensively evaluated based on the reprojection error, geometric direction consistency, depth consistency and descriptor matching consistency under multi-view observation, and the first weighting result is fused to obtain the final feature weight, which is used for weighted local optimization.

[0018] Furthermore, in step (4), feature weight modulation is introduced when constructing the appearance description, high-weight features are used preferentially, and the interference of dynamic features is reduced by weighted constraints in the geometric verification stage, thereby improving the robustness of loop closure detection.

[0019] Furthermore, in step (5), the loop closure constraint and the weighted point-line feature error are modeled uniformly in the global graph, and dynamic feature suppression and global optimization are achieved through weighted global optimization. Figure 1 Consistent maintenance.

[0020] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: ① By differentiating the contribution of features in modeling, the negative impact of dynamic and low-confidence features on pose estimation is effectively suppressed; ② The two-stage adaptive weighting mechanism enables the system to handle intermittent moving objects and avoids the long-term accumulation of misjudgments; ③ The joint modeling and weighting strategy of point and line features significantly enhances the observability and structural stability in low-texture scenes. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, this embodiment of the invention provides a dynamic semantic visual SLAM method based on adaptive weighting of point and line features, including the following steps:

[0024] (1) The object detection model and the image segmentation model are used together to perform object detection and pixel-level segmentation on the input image sequence, and output pixel-level dynamic mask and mask metadata to provide prior information on potential dynamic regions for subsequent modules; the open environment object detection model and the image segmentation model are cascaded. First, the object detection model generates candidate regions, then the candidate regions are segmented at the pixel level, and overlapping regions are fused by semantic priority rules to output pixel-level dynamic mask and semantic metadata; the specific process is as follows: the model first receives RGB images Then, after conventional image preprocessing (such as resizing and normalizing), the image is fed into the visual backbone network (usually a Swin Transformer) to extract multi-scale feature representations. Next, the multi-scale features are uniformly mapped to the token form required by the Transformer and spatial location encoding is added. The text prompts are mapped to a set of text embedding vectors through a language encoder (a Transformer-based text encoding structure): Each text token corresponds to a sentence or a semantic category. In cross-modal Transformer fusion, image feature tokens serve as Key / Value pairs, and text tokens as Queries. Through a multi-layered cross-modal attention mechanism, explicit alignment between semantics and spatial regions is achieved. This mechanism enables the model to directly output "text-region" level correspondences, rather than fixed class probabilities as in traditional detection. Then, predictions are made for each potential target region, outputting a set of candidate bounding boxes. and text-related confidence score ( ).

[0025] The segmentation model receives the same RGB images as the detection model. Then, high-resolution but lightweight visual feature representations are extracted using the ESAM image encoder. Secondly, it will receive the candidate box set output by GroundingDINO. And transform it into its acceptable spatial cue form. Next, image features With prompt features The results are fused in the mask decoder to output pixel-level segmentation results for the corresponding detection boxes. To address the issue that overlapping candidate regions can lead to the same pixel being covered by multiple semantic hypotheses, preventing the backend from obtaining a definitive pixel-level judgment, the fusion strategy is as follows:

[0026] Candidate mask set: for each candidate region ESAM outputs a binary or probability mask. At the same time, each candidate region is associated with a semantic category. .

[0027] Pixel-level coverage determination: for any pixel The set of masks covered by candidate masks is defined as follows: ;like Then the pixel is determined to be a static background; if Then it enters the conflict resolution phase;

[0028] Deterministic resolution rules for overlapping regions: When a pixel is covered by multiple candidate masks, a semantic priority rule is used for judgment: a set of semantic priorities is predefined (e.g., person > vehicle > object), and the semantic category with the highest priority is selected. ;

[0029] The final output is a pixel-level dynamic mask and the corresponding mask metadata.

[0030] (2) Receive dynamic mask and mask metadata, perform motion consistency checks and dynamic level classification, and perform initial adaptive weighting of point and line features based on dynamic level and feature quality to complete feature matching and preliminary pose estimation; motion consistency checks include fusion of optical flow consistency, reprojection error and depth change rate, motion verification of potential dynamic regions, and classification of dynamic levels based on motion intensity. The initial adaptive weighting of point and line features is as follows: for point features, weighting is performed based on corner degree, neighborhood repeatability and depth confidence; for line features, weighting is performed based on line segment length, gradient consistency, depth smoothness and reprojection angle; the weighting results are used to adjust the feature matching priority and contribution in pose estimation. The specific process is as follows: first, receive the output from the detection and segmentation module, including: pixel-level dynamic mask. : Mark pixel regions in an image that may belong to dynamic objects; structured object masks The module labels structured objects in the image that may be static and usable for geometric reconstruction; it also includes mask metadata (JSON): the semantic category and confidence information for each mask; this information provides spatial and semantic priors for subsequent tracking, allowing for a clear distinction between dynamic and static regions before feature extraction. To prevent false detections or temporarily static dynamic objects from affecting tracking, the module uses RGB frames... and depth map The system performs motion consistency analysis on dynamic object regions: optical flow and depth variation analysis are performed on the image features of dynamic mask regions. Based on pixel motion consistency and depth variation, dynamic objects are classified into different dynamic levels, ensuring the weight of truly static regions, suppressing the weight ratio of weakly dynamic regions, and eliminating features from strongly dynamic regions. Through this stage, the system can effectively reduce the interference of dynamic objects on feature matching and pose estimation, while retaining the ability to perceive highly moving objects. After dynamic consistency checking and hierarchical processing, feature extraction is performed from the following regions (point features are extracted using the ORB feature method, and line features are extracted using the gradient consistency-based line segment detection method LSD. First, line segments with stable direction and length are detected in the image, and then a corresponding line feature descriptor is constructed for each line segment for subsequent matching and pose estimation): the truly dynamic object part (after consistency screening) and the structured object part (static usable area). The extracted feature types include point features and line features, taking into account both local texture information and geometric structure information. During feature extraction, dynamic and structured regions are processed separately to ensure the quality and reliability of the features. In the initial weighting stage, the extracted point and line features undergo adaptive processing based on the following factors: corner degree, neighborhood repeatability, and depth reliability. Line features are adaptively weighted based on length, gradient consistency, depth smoothness, and reprojection angle. Through weighting, the influence of highly dynamic object features on subsequent matching and pose optimization is appropriately reduced, while structured object features and reliable dynamic features retain their full weights, providing a reliable basis for pose estimation. Simultaneously, the point and line weights are assigned according to the quality of the object's own point and line features. After feature extraction, feature matching is performed. The system matches point features and line features separately. The matching process prioritizes high-weight features based on the initial weighting results, while narrowing the matching radius and strictly matching low-weight features. For point features, descriptor matching is performed between point features in the current frame and those in the reference frame (or previous frame). For line features, a matching method combining endpoints and geometric attributes is used. After feature matching is completed, the system enters the pose estimation stage. Unlike traditional methods that treat all features as equivalent observations, this method explicitly introduces initial point and line weights calculated in the front end into pose estimation to adjust the contribution of different features in the optimization process. Specifically, each pair of matched point or line features is assigned a corresponding initial weight before entering pose estimation. This weight is jointly determined by the semantic prior of the front end, the results of dynamic analysis, and the quality of the feature itself. Based on this, pose estimation is uniformly modeled as a weighted least squares optimization problem, in which features from static regions and structurally stable objects naturally have higher weights and thus play a dominant role in the optimization, while the influence of potentially dynamic or low-reliability features is effectively suppressed.

[0031] For objects that may be moving, the following operations are performed: Calculate optical flow residuals: Purpose: to aggregate the displacement of points within the region into a single-value metric that represents the magnitude of visual motion. The steps are as follows: a. Point sampling: In each region, select points (preferred corner points) that fall on the bounding box (detection bounding box) and have a gray value of 255 from the existing features (ORB) of the previous frame. Then call LK optical flow to output the tracked point coordinates (trackedPoints) and status (status).

[0032] b. For each successfully tracked point i, calculate the optical flow residual:

[0033]

[0034] in, Indicates the pixel coordinates of the previous frame; This represents the pixels actually observed in the current frame; Indicates the depth of the previous frame; express , That is, pixel coordinates, d represents depth, K' represents camera intrinsic parameters, and the general meaning is to back-project the pixel coordinates u and its depth d onto a 3D point in the camera coordinate system; This represents the currently estimated camera pose; π(∙) represents projecting the 3D point onto pixel coordinates.

[0035] c. Take the median value of the pixel distances within the region:

[0036] d. If the number of success points is less than the threshold, directly set flowResidual = 0 or mark it as invalid; for weakly dynamic objects in the specific weight mapping, the formula should be... Replace with , Replace with The final results are shown in Table 1 below:

[0037] Table 1 Weight Mapping Results

[0038]

[0039] (3) At the keyframe level, point and line features are verified for consistency across multiple perspectives. A second adaptive weighting is performed based on reprojection error, geometric consistency, depth consistency, and matching consistency, followed by weighted local optimization and map maintenance. The second adaptive weighting is a comprehensive evaluation based on reprojection error, geometric consistency, depth consistency, and descriptor matching consistency under multi-view observations, and the final feature weights are obtained by fusing the first weighting results. These weights are then used for weighted local optimization. The specific process is as follows: First, point and line features are adaptively weighted a second time based on reprojection error, geometric consistency, depth consistency, and matching consistency to adjust their influence on map construction and optimization. Then, the system determines whether a new keyframe needs to be inserted based on the tracking quality of the current frame, perspective changes, and redundancy with existing keyframes. If the insertion conditions are met, the current frame is marked as a new keyframe, and its corresponding point and line features and weight information are added to the local keyframe set. Static and dynamic maps are constructed based on the keyframes, with the static map used for geometric constraints and optimization, and the dynamic map used to describe movable objects. During map maintenance, the system removes old map elements with excessive reprojection errors, those that have not been observed for a long time, or those with poor stability, and adds newly observed point and line features with high consistency to the map to maintain its timeliness. Next, the system performs weighted beamforming (BA) on keyframes and map points within a local area to further optimize pose and map structure. The secondary weighting result is used to adjust the influence of different features on the optimization objective. Finally, based on the optimization results of the local BA, redundant or less contributing keyframes are removed to maintain the compactness of the local map.

[0040] The second adaptive weighting formula is as follows: Calculate the reprojection error weights: (v=3~6, smaller v is recommended in high dynamic environments); among which, Calculate the geometric consistency weight (to ensure consistency in direction / angle): ;in, θ represents the direction or angle. The direction of a point feature is a one-dimensional scalar, while the direction of a line feature is a vector / angle. A vector is essentially a two-dimensional representation of an angle. The standard deviation of the reprojection error is represented; the depth consistency weight is calculated. , The standard deviation of the depth noise is represented by the square of the standard deviation; for point features, For line features, ; Calculate the matching consistency weight:

[0041] ;

[0042] in, ;in, This represents the Hamming distance of a candidate match. The weights after the second weighting represent the minimum Hamming distance for a given descriptor in the candidate pool. The final weights used for BA are: ;

[0043] (4) Based on weighted appearance description and geometric consistency verification, loop closure constraints are generated and confirmed to suppress the interference of dynamic features on loop closure judgment. Feature weight modulation is introduced when constructing appearance description, and high-weight features are used first. In the geometric verification stage, weighted constraints are used to reduce the interference of dynamic features and improve the robustness of loop closure detection. The specific process is as follows: loop closure candidates are generated by analyzing the appearance similarity between the current keyframe and the historical keyframe, and stable features are selected by weighting the reliability of point and line features to reduce the interference of dynamic objects. Subsequently, geometric consistency verification (loop closure geometric verification) is performed on the candidate frames, and only loop closure relationships that meet the constraints under multiple views are retained (loop closure constraint generation). The confirmed loop closure constraints and feature credibility information are passed to the global optimization module (loop closure constraint transmission) to assist in global graph optimization and maintain the consistency of map and pose. This module ensures that the system can robustly identify loops and effectively correct accumulated errors in dynamic and complex indoor environments.

[0044] (5) Integrate loop closure constraints and weighted point-line feature errors, and perform weighted global optimization to eliminate cumulative drift and maintain global map consistency. Loop closure constraints and weighted point-line feature errors are modeled uniformly in the global map, and dynamic feature suppression and global map consistency are achieved through weighted global optimization. Figure 1 Consistent maintenance. The specific process is as follows: First, based on the existing keyframe relationships and map elements in the system, a global graph structure model is constructed, unifying the keyframe poses, point and line map elements, and their mutual constraints within the same global optimization framework. During the optimization process, the module adopts a weighted error model, assigning different levels of importance to geometric constraints from different sources. Among them, constraints from static structures and high-confidence features play a dominant role in the optimization results, while the influence of dynamic or unstable features is effectively weakened through a dynamic feature suppression mechanism, thereby avoiding the impact of dynamic objects on the global graph. Figure 1 To prevent interference with consistency, when the loop closure detection module provides new loop closure information, the global BA integrates the loop closure constraints into the existing global graph structure. Through joint optimization, it eliminates long-term accumulated pose drift and restores the map's closure consistency over a large scale. This process not only corrects local errors but also coordinates observations from different time periods at the global level. After optimization, the module maintains and updates the global map for consistency, including updating keyframe poses, map point and line structures, and related feature states. This ensures the entire system maintains a consistent and stable global map representation, providing a reliable foundation for subsequent localization, mapping, and navigation tasks.

[0045] For each point feature observation, the reprojection error term is defined as:

[0046]

[0047] in: This represents the pose of the k-th keyframe; This represents the i-th map point; The observed pixel coordinates of the point; These are the feature weights for the second stage.

[0048] For line features, a weighted error term is defined by combining endpoint reprojection error and direction consistency constraint:

[0049]

[0050] in: This represents the pose of the k-th keyframe; This represents an endpoint of the j-th map line; This represents the other endpoint of the j-th map line; , These are the observed pixel coordinates of the two endpoints of the line; The second-stage line feature weights.

Claims

1. A dynamic semantic visual SLAM method based on adaptive weighting of point and line features, characterized in that, Includes the following steps: (1) The object detection model and the image segmentation model are used together to perform object detection and pixel-level segmentation on the input image sequence, and the pixel-level dynamic mask and mask metadata are output, including dynamic mask information such as category identifier, pixel location set and confidence parameter, which provides potential dynamic region priors for subsequent modules; (2) Receive dynamic mask and mask metadata, perform motion consistency check and dynamic level classification based on optical flow residual and depth change rate, and perform first adaptive weighting of point and line features based on dynamic level and feature quality, thereby completing feature matching and preliminary pose estimation; (3) Perform multi-view consistency verification on point and line features at the key frame level, perform a second adaptive weighting based on reprojection error, geometric consistency, depth consistency and matching consistency, and perform weighted local optimization and map maintenance. (4) Based on weighted appearance description and geometric consistency verification, generate and confirm closure constraints to suppress the interference of dynamic features on closure determination; (5) Integrate loop constraints and weighted point and line feature errors, and perform weighted global optimization to eliminate cumulative drift and maintain global map consistency.

2. The dynamic semantic visual SLAM method based on adaptive weighting of point and line features according to claim 1, characterized in that, In step (1), an open environment target detection model and an image segmentation model are cascaded. First, a candidate region is generated through the target detection model, then the candidate region is segmented at the pixel level, and the overlapping region is fused through semantic priority rules to output a pixel-level dynamic mask and semantic metadata.

3. The dynamic semantic visual SLAM method based on adaptive weighting of point and line features according to claim 1, characterized in that, In step (2), motion consistency check includes fusion optical flow consistency, reprojection error and depth change rate, motion verification of potential dynamic regions, and dynamic level classification according to motion intensity.

4. The dynamic semantic visual SLAM method based on adaptive weighting of point and line features according to claim 1, characterized in that, In step (2), the first adaptive weighting of point and line features is as follows: for point features, weighting is performed based on corner degree, neighborhood repeatability and depth confidence; for line features, weighting is performed based on line segment length, gradient consistency, depth smoothness and reprojection angle; the weighting results are used to adjust the feature matching priority and contribution in pose estimation.

5. The dynamic semantic visual SLAM method based on adaptive weighting of point and line features according to claim 1, characterized in that, In step (3), the second adaptive weighting is based on a comprehensive evaluation of reprojection error, geometric direction consistency, depth consistency and descriptor matching consistency under multi-view observation, and the first weighting result is fused to obtain the final feature weights, which are used for weighted local optimization.

6. The dynamic semantic visual SLAM method based on adaptive weighting of point and line features according to claim 1, characterized in that, In step (4), feature weight modulation is introduced when constructing the appearance description, high-weight features are used first, and the interference of dynamic features is reduced by weighted constraints in the geometric verification stage, thereby improving the robustness of loop closure detection.

7. The dynamic semantic visual SLAM method based on adaptive weighting of point and line features according to claim 1, characterized in that, In step (5), the loop closure constraint and the weighted point-line feature error are modeled in the global graph, and dynamic feature suppression and global map consistency maintenance are achieved through weighted global optimization.

Images