Progressive dynamic feature point elimination method based on geometric semantic probabilistic reasoning

Abstract

The invention discloses a progressive dynamic feature point elimination method based on geometric semantic probabilistic reasoning, and relates to the technical field of computer vision and robot perception. In order to solve the defects that in the prior art, real-time performance and accuracy cannot be considered in a dynamic scene, and a dynamic feature point recognition and elimination mechanism fusing geometric and semantic information and carrying out progressive probabilistic reasoning is lacked, geometric dynamic prior is obtained through optical flow matching and basic matrix calculation, and then a semantic detection result is obtained; according to the method, geometric prior and semantic weight are fused to realize preliminary recognition of feature points, and on the basis, a conditional random field model is constructed to carry out global probabilistic reasoning and dynamic point refined elimination, so that progressive elimination of the dynamic feature points is realized. According to the method, the high real-time performance is kept, and meanwhile, the positioning precision and robustness in a dynamic scene are remarkably improved; and the method is suitable for application scenes, such as mobile robots, unmanned driving and augmented reality, which need to realize high-precision visual positioning and mapping in a complex dynamic environment.

Description

Technical Field

[0001] This involves the fields of computer vision and robot perception technology, specifically progressive dynamic feature point removal based on geometric semantic probabilistic reasoning. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) has become a core research area in robot perception and autonomous navigation in recent years. Traditional visual SLAM systems are typically based on the assumption of a static environment, extracting, matching, and optimizing features from consecutive image frames captured by a camera to simultaneously estimate the camera's trajectory and the density or sparseness of the environment's map. In open-source systems such as ORB-SLAM, LSD-SLAM, and DSO, the algorithms largely rely on sufficient image texture and stable lighting conditions, enabling high-precision pose estimation and mapping performance in static scenes.

[0003] However, in practical applications such as service robots, mobile inspection, and autonomous driving, the environment often contains a large number of dynamic targets, such as pedestrians, vehicles, or other moving objects. In dynamic scenes, these moving objects generate significant pixel displacements between image frames, resulting in a large number of dynamic feature points in the feature points extracted by traditional SLAM algorithms. This can lead to matching errors, pose drift, and even system failure. To address this, scholars both domestically and internationally have conducted extensive research on visual SLAM in dynamic scenes, proposing various methods for dynamic feature point detection and removal.

[0004] A typical approach is based on geometric constraints, such as optical flow consistency detection, epipolar constraints, and fundamental matrix estimation, to determine the dynamic nature of a point by calculating its offset distance relative to the epipolar line. This type of method has low computational cost and can achieve real-time performance without relying on deep learning models. However, its accuracy is significantly affected by threshold settings and noise interference, especially when the direction of movement of a dynamic object is close to the epipolar line direction, making it prone to misjudgment.

[0005] Another type of method is based on semantic information, using deep learning models such as convolutional neural networks (CNNs), single-shot multi-box detectors (SSDs), and Mask-RCNNs to detect and segment dynamic objects in images, thereby removing semantically dynamic regions from the feature point set. This type of method has high recognition accuracy, but its complex network structure and large number of parameters make it difficult to implement in real-time on resource-constrained platforms such as mobile terminals and embedded systems. Furthermore, semantic detection results typically only provide object category information and cannot directly reflect the motion state of feature points.

[0006] In recent years, some studies have attempted to combine geometric constraints with semantic detection. For example, dynamic candidate regions are first obtained through object detection, and then geometric constraints are used to verify the dynamics of feature points to improve detection accuracy. However, these methods still have two main shortcomings: First, the fusion of geometric and semantic information is mostly limited to heuristic weighting and has failed to establish a unified probabilistic reasoning framework; second, the recognition of dynamic feature points relies heavily on short-term information from adjacent frames and cannot fully utilize global constraints over long time spans, resulting in some dynamic points remaining.

[0007] In summary, existing technologies have shortcomings such as being unable to balance real-time performance and accuracy in dynamic scenarios, and lacking a dynamic feature point recognition and elimination mechanism that integrates geometric and semantic information and performs progressive probabilistic reasoning. Summary of the Invention

[0008] To address the shortcomings of existing technologies in achieving both real-time performance and accuracy in dynamic scenarios, and in lacking a dynamic feature point recognition and elimination mechanism that integrates geometric and semantic information for progressive probabilistic reasoning, the technical solution provided by this invention is as follows: A progressive dynamic feature point removal method based on geometric semantic probabilistic reasoning includes: The steps involved in extracting feature points with scale-invariant and rotation-invariant properties from consecutive image frames captured by a camera to obtain a set of feature points. The steps include: matching feature points between adjacent frames, obtaining matching relationships based on the pyramid iterative optical flow method, calculating the fundamental matrix between two frames using the graph cut random sampling consensus algorithm, determining the dynamic prior probability of feature points based on the offset distance between feature points and corresponding epipolar lines, and outputting a geometric dynamic prior set containing geometric constraint information. The process involves inputting the current frame image into a lightweight MobileNetV3-SSD object detection network, extracting multi-scale features through depthwise separable convolution and inverse residual structures, and outputting a semantic information map containing bounding boxes of dynamic and static regions. The steps involve fusing the geometric dynamic prior set with the semantic information graph at the pixel coordinate level, assigning higher dynamic weights to feature points located within the dynamic detection box and lower dynamic weights to feature points in static regions, calculating the comprehensive dynamic probability, performing preliminary classification based on a threshold, and outputting a preliminary set of dynamic feature points. The steps involve constructing a conditional random field model, establishing a fully connected graph structure with feature points as nodes, calculating the unit potential energy based on photometric reprojection error and the number of observations, calculating the pairwise potential energy based on spatial proximity and observation similarity, obtaining the optimal static or dynamic label distribution of feature points through energy minimization inference, and outputting the dynamic feature point set after refined inference. Based on the refined reasoning results, feature points that are judged to be dynamic are removed, and only static feature points are retained for camera pose estimation and map optimization, thus realizing the progressive removal of dynamic feature points.

[0009] Furthermore, in a preferred embodiment, in the feature point extraction step, the ORB operator with scale-invariant and rotation-invariant properties is used to detect feature points, and key points and descriptors are extracted under a multi-layer pyramid structure.

[0010] Furthermore, in a preferred embodiment, in the step of generating the geometric dynamic prior set, the inter-frame feature point matching relationship is established by using the pyramid iterative Lucas-Kanade optical flow method, and the fundamental matrix is ​​solved by using the graph cut random sampling consensus algorithm.

[0011] Furthermore, in a preferred embodiment, in the semantic information map generation step, the MobileNetV3-SSD object detection network constructs a lightweight backbone network through depthwise separable convolution and inverse residual structure, extracts multi-scale feature maps from layers bneck14, Conv17, Conv19, Conv21, Conv23, and Conv25, and outputs semantic detection results containing dynamic and static regions after non-maximum suppression.

[0012] Furthermore, in a preferred embodiment, in the fusion calculation step, feature points located within the dynamic detection box are assigned a dynamic region weight three, and feature points located in the static region are assigned a static region weight one. The geometric dynamic probability corresponding to the epipolar offset is then weighted and fused with the semantic weight.

[0013] Furthermore, in a preferred embodiment, in the refined inference step of the conditional random field model, the unit potential energy calculates the static probability based on the average photometric reprojection error of the feature point and the number of observations, and the pairwise potential energy constructs a Gaussian kernel function model based on the spatial location of the feature point and the observation difference, which includes an observation kernel and a location kernel respectively, and obtains the optimal label distribution of each feature point by minimizing the global energy.

[0014] Based on the same inventive concept, this invention also provides a progressive dynamic feature point removal device based on geometric semantic probabilistic reasoning, comprising: This module extracts feature points with scale-invariant and rotation-invariant properties from continuous image frames captured by the camera to obtain a set of feature points. The module performs feature point matching on adjacent frames, obtains the matching relationship based on the pyramid iterative optical flow method, calculates the fundamental matrix between two frames using the graph cut random sampling consensus algorithm, determines the dynamic prior probability of the feature point based on the offset distance between the feature point and the corresponding epipolar line, and outputs a geometric dynamic prior set containing geometric constraint information. The current frame image is input into the lightweight MobileNetV3-SSD object detection network, which extracts multi-scale features through depthwise separable convolution and inverse residual structure, and outputs a module containing semantic information maps containing bounding boxes of dynamic and static regions. This module fuses the geometric dynamic prior set with the semantic information graph at the pixel coordinate level, assigns higher dynamic weights to feature points located within the dynamic detection box and lower dynamic weights to feature points in static regions, calculates the comprehensive dynamic probability, performs preliminary classification based on a threshold, and outputs a preliminary set of dynamic feature points. A conditional random field model is constructed, a fully connected graph structure is established with feature points as nodes, the unit potential energy is calculated based on photometric reprojection error and the number of observations, the pairwise potential energy is calculated based on spatial proximity and observation similarity, the optimal label distribution of feature points in static or dynamic form is obtained through energy minimization reasoning, and the module outputs the dynamic feature point set after refined reasoning. Based on the refined reasoning results, feature points that are judged to be dynamic are removed, and only static feature points are retained for camera pose estimation and map optimization, thus realizing a module for progressive removal of dynamic feature points.

[0015] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, wherein when the computer program is read by a computer, the computer executes the method described thereon.

[0016] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, wherein when the processor reads a computer program stored in the storage medium, the computer executes the method described thereon.

[0017] Based on the same inventive concept, the present invention also provides a computer program product, which, when executed, implements the method described.

[0018] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows: This scheme effectively improves the robustness and localization accuracy of SLAM systems in dynamic environments by combining geometric constraints with semantic information in a dynamic feature point recognition strategy. Specifically, the epipolar constraint-based geometric information acquisition part utilizes the fundamental matrix relationship between adjacent frames to calculate the dynamic probability through the offset distance of feature points to their corresponding epipolar lines, thereby quickly identifying potential dynamic feature points without relying on deep learning models. Compared to traditional geometric methods that rely solely on optical flow consistency or fixed thresholds, this approach, by introducing the GC-RANSAC algorithm, significantly reduces mismatches in fundamental matrix estimation, improves the accuracy of feature point dynamics assessment, and provides reliable geometric priors for subsequent semantic information fusion.

[0019] This solution employs the MobileNetV3-SSD object detection module, replacing the traditional VGG16 backbone with a lightweight network to achieve real-time semantic detection on mobile platforms. Through the design of depthwise separable convolutions and inverse residual structures, this module significantly reduces the number of parameters and computational load, enabling the system to maintain high detection speed even in resource-constrained embedded environments. Compared to the heavy networks widely used in existing research, such as Mask-RCNN and YOLOv5, this MobileNetV3-SSD solution reduces inference latency by approximately 30% to 40% while maintaining comparable detection accuracy, effectively improving the overall real-time performance and stability of the SLAM system.

[0020] In the initial dynamic feature point identification stage, the scheme jointly calculates the geometric dynamic probability of epipolar constraints with the weights of semantic detection regions, and distinguishes the motion attributes of feature points in different scenarios through a dynamic region weight amplification mechanism. This method assigns higher weights to dynamic regions and lower weights to static regions, realizing probabilistic identification of dynamic points and effectively alleviating the problems of difficulty in setting epipolar offset thresholds and high false detection rates. Compared with traditional methods based on single feature distribution, this fusion strategy has higher sensitivity to slightly moving targets in complex scenes, enabling the system to maintain stable tracking performance under low-texture or low-light conditions.

[0021] In the refinement stage, this scheme introduces a dynamic feature point probabilistic inference model based on Conditional Random Fields (CRF). This model comprehensively considers the spatial relationships between feature points, photometric reprojection errors, and observation counts, achieving globally consistent inference through a fully connected graph model. This part fully utilizes long-term dependency information in the time series, overcoming the limitations of traditional methods based on the geometric relationships between adjacent frames. This allows the algorithm to distinguish dynamic feature points whose motion direction is close to the epipolar direction. Compared with existing methods that only rely on short-term inter-frame differences for removal, the CRF model significantly improves the accuracy of dynamic feature point recognition and the retention rate of static points in the refinement inference stage, reducing false deletions and missed deletions.

[0022] Finally, the scheme organically combines the three stages of "epidural constraint—semantic detection—conditional random field" through a progressive dynamic point elimination process, forming a bottom-up, multi-layered dynamic point recognition mechanism. This mechanism retains the efficiency of geometric constraints, incorporates the high accuracy of semantic recognition, and achieves globally consistent processing of dynamic information through probabilistic reasoning. Compared with existing systems that only use geometric or semantic methods, this scheme reduces pose estimation error by approximately 20% in highly dynamic scenarios, significantly reduces trajectory drift, and comprehensively improves the robustness, accuracy, and generalization ability of the overall SLAM system.

[0023] It is suitable for applications such as mobile robots, autonomous driving, and augmented reality that require high-precision visual positioning and mapping in dynamic environments. Attached Figure Description

[0024] Figure 1 A schematic diagram of the dynamic feature point removal algorithm; Figure 2 This is a schematic diagram of the epipolar geometry constraint; Figure 3 A schematic diagram illustrating the determination of dynamic feature points using epipolar constraints; Figure 4 This is a schematic diagram of the SSD model network structure; Figure 5 This is a schematic diagram of the MobileNetV3-SSD network architecture; Figure 6 This is a schematic diagram illustrating the principle of the CRF probability model. Detailed Implementation

[0025] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically: Implementation Method 1: This implementation method provides a progressive dynamic feature point removal method based on geometric semantic probabilistic reasoning, including: The steps involved in extracting feature points with scale-invariant and rotation-invariant properties from consecutive image frames captured by a camera to obtain a set of feature points. The steps include: matching feature points between adjacent frames, obtaining matching relationships based on the pyramid iterative optical flow method, calculating the fundamental matrix between two frames using the graph cut random sampling consensus algorithm, determining the dynamic prior probability of feature points based on the offset distance between feature points and corresponding epipolar lines, and outputting a geometric dynamic prior set containing geometric constraint information. The process involves inputting the current frame image into a lightweight MobileNetV3-SSD object detection network, extracting multi-scale features through depthwise separable convolution and inverse residual structures, and outputting a semantic information map containing bounding boxes of dynamic and static regions. The steps involve fusing the geometric dynamic prior set with the semantic information graph at the pixel coordinate level, assigning higher dynamic weights to feature points located within the dynamic detection box and lower dynamic weights to feature points in static regions, calculating the comprehensive dynamic probability, performing preliminary classification based on a threshold, and outputting a preliminary set of dynamic feature points. The steps involve constructing a conditional random field model, establishing a fully connected graph structure with feature points as nodes, calculating the unit potential energy based on photometric reprojection error and the number of observations, calculating the pairwise potential energy based on spatial proximity and observation similarity, obtaining the optimal static or dynamic label distribution of feature points through energy minimization inference, and outputting the dynamic feature point set after refined inference. Based on the refined reasoning results, feature points that are judged to be dynamic are removed, and only static feature points are retained for camera pose estimation and map optimization, thus realizing the progressive removal of dynamic feature points.

[0026] In the feature point extraction step, the ORB operator, which has scale-invariant and rotation-invariant properties, is used to detect feature points, and key points and descriptors are extracted under a multi-layer pyramid structure.

[0027] In the generation step of the geometric dynamic prior set, the Lucas-Kanade optical flow method with pyramid iteration is used to establish the inter-frame feature point matching relationship, and the graph cut random sampling consensus algorithm is used to solve the fundamental matrix.

[0028] In the semantic information graph generation step, the MobileNetV3-SSD object detection network constructs a lightweight backbone network through depthwise separable convolution and inverse residual structure, extracts multi-scale feature maps from layers bneck14, Conv17, Conv19, Conv21, Conv23, and Conv25, and outputs semantic detection results containing dynamic and static regions after non-maximum suppression.

[0029] In the fusion calculation step, feature points located within the dynamic detection box are assigned dynamic region weight three, and feature points located in the static region are assigned static region weight one. The geometric dynamic probability corresponding to the epipolar offset is then weighted and fused with the semantic weight.

[0030] Furthermore, in a preferred embodiment, in the refined inference step of the conditional random field model, the unit potential energy calculates the static probability based on the average photometric reprojection error of the feature point and the number of observations, and the pairwise potential energy constructs a Gaussian kernel function model based on the spatial location of the feature point and the observation difference, which includes an observation kernel and a location kernel respectively, and obtains the optimal label distribution of each feature point by minimizing the global energy.

[0031] A progressive dynamic feature point removal device based on geometric semantic probabilistic reasoning is also provided, comprising: This module extracts feature points with scale-invariant and rotation-invariant properties from continuous image frames captured by the camera to obtain a set of feature points. The module performs feature point matching on adjacent frames, obtains the matching relationship based on the pyramid iterative optical flow method, calculates the fundamental matrix between two frames using the graph cut random sampling consensus algorithm, determines the dynamic prior probability of the feature point based on the offset distance between the feature point and the corresponding epipolar line, and outputs a geometric dynamic prior set containing geometric constraint information. The current frame image is input into the lightweight MobileNetV3-SSD object detection network, which extracts multi-scale features through depthwise separable convolution and inverse residual structure, and outputs a module containing semantic information maps containing bounding boxes of dynamic and static regions. This module fuses the geometric dynamic prior set with the semantic information graph at the pixel coordinate level, assigns higher dynamic weights to feature points located within the dynamic detection box and lower dynamic weights to feature points in static regions, calculates the comprehensive dynamic probability, performs preliminary classification based on a threshold, and outputs a preliminary set of dynamic feature points. A conditional random field model is constructed, a fully connected graph structure is established with feature points as nodes, the unit potential energy is calculated based on photometric reprojection error and the number of observations, the pairwise potential energy is calculated based on spatial proximity and observation similarity, the optimal label distribution of feature points in static or dynamic form is obtained through energy minimization reasoning, and the module outputs the dynamic feature point set after refined reasoning. Based on the refined reasoning results, feature points that are judged to be dynamic are removed, and only static feature points are retained for camera pose estimation and map optimization, thus realizing a module for progressive removal of dynamic feature points.

[0032] A computer storage medium is also provided for storing a computer program, which, when read by the computer, executes the method.

[0033] A computer is also provided, including a processor and a storage medium, wherein the computer executes the method when the processor reads a computer program stored in the storage medium.

[0034] A computer program product is also provided, which, when executed, implements the method described.

[0035] Implementation Method Two: This implementation method is a further detailed description of the technical solution provided in Implementation Method One, specifically: This paper presents a progressive dynamic feature point removal method based on geometric-semantic probabilistic reasoning, applied to visual SLAM systems. This method enables accurate feature point recognition and removal in dynamic environments, thereby improving the system's localization accuracy and robustness in dynamic scenes. The overall process includes six stages: feature point extraction, geometric information acquisition, semantic information acquisition, preliminary dynamic feature point recognition, conditional random field refinement reasoning, and dynamic feature point removal.

[0036] The first step is feature point extraction.

[0037] Key feature points are extracted from consecutive image frames captured by the camera. Feature operators with scale-invariant and rotation-invariant properties, such as the ORB operator, are used to obtain a set of image feature points with stable matching properties. The extracted feature points serve as input for subsequent geometric constraint calculations and semantic information fusion, providing the basic visual feature data for the entire dynamic recognition process.

[0038] The second step is to acquire geometric information based on epipolar constraints.

[0039] Feature point matching is performed between adjacent image frames, and the correspondence between the current frame and the previous frame is calculated using the Lucas-Kanade optical flow method with pyramid iteration. Based on this, the graph cut random sampling consensus algorithm (GC-RANSAC) is used to solve for the fundamental matrix between the two image frames, which describes the geometric constraints of the camera between the two frames. According to the pinhole camera model, the distance between each feature point and its corresponding epipolar line is calculated; a larger distance indicates a higher probability that it is a dynamic point. Under ideal static conditions, the offset distance should be close to zero, but in the presence of noise, it is usually less than an empirical threshold. Thus, a geometric dynamic prior probability is generated for each feature point. This set of geometric information serves as the input for the next stage of semantic recognition and fusion.

[0040] The third step is to obtain semantic information based on MobileNetV3-SSD.

[0041] The current frame image is input into a lightweight MobileNetV3-SSD object detection network. This network uses MobileNetV3 as its feature extraction backbone and combines it with the SSD multi-scale detection head to achieve efficient object detection. MobileNetV3 reduces the number of model parameters and computational complexity through depthwise separable convolutions and inverse residual structures, thus enabling real-time detection on embedded devices. This network detects dynamic objects (such as pedestrians and vehicles) in the image and outputs a semantic information map containing object bounding boxes and category labels. This semantic information map spatially corresponds to the geometric information set output in the previous step at the pixel coordinate level, providing a semantic reference for the next step of dynamic feature point probabilistic fusion.

[0042] The fourth step is the preliminary identification of dynamic feature points.

[0043] The geometric dynamic probabilities from the second step are fused with the semantic region weights from the third step. Feature points located within dynamic detection boxes are assigned higher dynamic weights, while those located in static regions are assigned lower dynamic weights. Through weighted fusion, the comprehensive dynamic probability of each feature point is obtained, and preliminary classification is performed based on a preset threshold. Feature points with dynamic probabilities higher than the threshold are classified as dynamic feature points, while those lower are classified as static feature points. This preliminary set of dynamic feature points serves as the input for the next step of conditional random field inference.

[0044] The fifth step is to refine the reasoning based on the dynamic feature points of the conditional random field.

[0045] Based on the initial identification, a conditional random field model is constructed to perform global probabilistic inference on the dynamic attributes of all feature points. The model uses each feature point as a node, establishing spatial and observational relationships between feature points through a fully connected structure. The unit potential is determined based on the average photometric reprojection error of the feature point across multiple frames and the number of observations; feature points with higher observational consistency have higher static probabilities. Paired potentials are constructed based on the spatial proximity and observational similarity of feature points, ensuring that adjacent points tend to have consistent dynamic attributes. A Gaussian kernel function is used to measure the difference between reprojection error and spatial distance, and an iterative inference algorithm is employed to minimize the global energy function, obtaining the optimal label distribution for each feature point, i.e., its confidence probability of being a static or dynamic point. After this inference process, a refined set of dynamic feature points is output.

[0046] Step 6: Dynamic feature point removal.

[0047] Based on the labeling results obtained in step five, feature points identified as dynamic are removed from the feature matching set, retaining only static feature points for subsequent camera pose estimation and map optimization. This progressive removal mechanism effectively eliminates the interference of dynamic objects on pose estimation, thereby significantly improving the stability and accuracy of the SLAM system in dynamic environments.

[0048] In the above implementation, the steps are interconnected, with the output data of the previous stage serving as the input parameters for the next stage. Geometric constraints provide motion priors for semantic detection, semantic information provides spatial constraints for probabilistic reasoning, and the conditional random field model further integrates multi-frame features and neighborhood features to achieve progressive dynamic feature point identification and elimination from local geometric constraints to global probabilistic reasoning.

[0049] By organically combining the above steps, this invention can effectively suppress feature drift and mismatches caused by moving objects in dynamic environments, significantly improving the robustness, localization accuracy, and generalization performance of visual SLAM systems. Compared with existing technologies, this invention not only ensures the real-time performance of the algorithm but also improves the system's adaptability to complex motion scenes in dynamic environments, demonstrating promising prospects for engineering applications.

[0050] Implementation Method 3: Combination Figure 1-6 This embodiment describes the technical solution provided above in further detail through specific examples. Specifically: Many mature visual SLAM solutions are available on the market, but most of these solutions are based on static environments and require sufficient texture and lighting conditions to function properly. However, in reality, robot motion scenarios are often in dynamic environments. For example, service robots in service scenarios often need to deal with constantly moving crowds. In such cases, some visual SLAM solutions will fail. Dynamic environments are relatively unfavorable for SLAM. If the proportion of dynamic points in the feature points extracted by the algorithm is too high, it will cause a decrease in trajectory accuracy, and may even lead to drifting away or tracking loss. The main work of this section is to add a dynamic point removal module based on Chapter 3, and propose a progressive dynamic feature point detection and removal method that combines geometric and semantic information and is based on probabilistic reasoning. This chapter first introduces the use of geometric constraints and deep learning-based object detection to obtain semantic information for the identification and removal of dynamic feature points. Then, it further introduces a probabilistic reasoning model based on conditional random fields to effectively identify and remove residual dynamic feature points. The process is as follows: Figure 1 As shown.

[0051] Geometric Information Acquisition Based on Epipolar Constraints Epipolar geometric constraints describe the specific geometric relationships between imaging information at different camera positions. Figure 2 This geometric relationship is vividly illustrated. The two parallelograms on the left and right represent the imaging planes of the camera at two different positions. and These represent the optical centers of the camera at two different locations, connected... and The straight line is called the baseline. The intersection of the baseline and the two imaging planes is... and These are called poles. Three-dimensional points in space. The projections on the previous frame and the current frame are respectively and From point , and The resulting plane is called the polar plane. The line of intersection between the polar plane and the two imaging planes... and These are called polar lines.

[0052] Judging from the camera position in the previous frame, if there is a lack of The depth information of the point is then obtained from... The ray that sets off All points on it could be The true location of these points. Clearly, the projections of these points onto the current frame all lie on the epipolar line. superior.

[0053] Based on these geometric relationships, the following algebraic identities can be derived: (2-30) The fundamental matrix between two image frames is represented by the epipolar geometric constraint. The epipolar geometric constraint describes the positional relationship between matching feature points, can be used to solve the pose transformation between image frames, and can also determine the correctness of feature point matching given the fundamental matrix.

[0054] The dynamic prior probability of feature points in the current frame is calculated using the distance between a feature point and its corresponding epipolar line in epipolar geometry constraints. The specific method is as follows: First, establish the matching relationship between feature points in two adjacent color image frames; second, solve for the fundamental matrix based on the epipolar geometry constraints; finally, calculate the distance between each feature point in the current frame and its corresponding epipolar line. The larger the distance, the higher the probability that the feature point is a dynamic point.

[0055] To obtain an accurate fundamental matrix between two image frames, correct feature point matching is crucial. However, the calculation of the fundamental matrix itself is intended to verify the accuracy of these matches, creating a classic "chicken or egg" problem. In the ORB-SLAM2 system, feature point matching typically employs the bag-of-words method or projection region search. However, these methods cannot determine whether the object containing the feature point is moving, thus failing to effectively eliminate the influence of the dynamic environment on the system.

[0056] To address these issues, this implementation system employs an improved method. First, the pyramid iterative Lucas-Kanade optical flow method is used to calculate the corresponding matching points of the extracted feature points in the current frame in the previous frame. Next, if potential dynamic objects are detected in the previous frame, these potential dynamic features located within the dynamic object detection box are removed. Then, the graph-cut random sample consensus algorithm (GC-RANSAC) is used to calculate the fundamental matrix between the two frames. GC-RANSAC considers both the matching degree of point-to-point models and the consistency of spatial structure, and its results in estimating the fundamental matrix are more accurate than those of the traditional RANSAC algorithm.

[0057] The estimated fundamental matrix is ​​used to solve for the offset distance, which is then used to calculate the static probability prior of the feature points. Based on the pinhole camera model, such as... Figure 3 As shown, the camera observes the same 3D map point from two different positions during its movement. . and The homogeneous coordinate form is shown below:

[0058] in, and Represents the coordinates of feature points in the image pixel coordinate system. The epipolar line in the current frame. Based on the fundamental matrix Feature points from previous frames The coordinates are obtained as follows:

[0059] in, , and Represents a line vector. Polar constraints are expressed in the following form:

[0060] Next, the distance between the feature point and the corresponding epipolar line is defined as the offset distance. , using symbols The offset distance can be calculated using the following formula:

[0061] If point For a static spatial point, combining equations (3-4) and (3-5), then The offset distance of the point is:

[0062] Ideally, the feature points in the current frame The point falls exactly on the polar line However, in reality, due to various factors such as noise, the offset distance is generally greater than zero but less than an empirical threshold.

[0063] If point Not a static spatial point, such as a point that moves as the camera moves from a previous frame to the current frame. The 3D map points they represent have also moved. In this situation, with The points that match are those made by Mapped to the current frame Feature points. If 3D map points If the movement does not degenerate, then under normal circumstances... The offset distance is greater than the threshold Therefore, the offset distance can be compared with an empirical threshold. This is used to determine whether a feature point is a dynamic feature point. However, empirical thresholds... Choosing the right threshold is very difficult. If the threshold is set too high, a large number of actual dynamic features may go undetected; if the threshold is set too low, many static points may be misclassified as dynamic features, leading to false detections.

[0064] Semantic information acquisition based on MobileNetV3-SSD On resource-constrained platforms, considering limitations in computing power, memory space, and battery capacity, the goal is to provide sufficient computing power to the device while minimizing energy consumption and extending operating time. In this context, NCNN, a high-performance neural network inference framework optimized for mobile platforms, is an ideal choice for integration into the object detection thread of a SLAM system due to its pure C++ implementation and zero third-party dependencies. Its lightweight design is not only suitable for operation in environments with limited computing resources but also enables seamless integration with SLAM systems, providing powerful visual processing capabilities for mobile robots.

[0065] Currently, many SLAM systems operate slowly and consume enormous amounts of computing power due to complex semantic segmentation networks or unreasonable system architectures, making it difficult to run in real time on resource-constrained platforms. Considering that SLAM is a core component of state estimation, its real-time performance directly affects the robot's navigation and decision-making capabilities. Therefore, the goal of this implementation is to improve the speed and efficiency of object detection and achieve real-time performance on resource-limited devices.

[0066] Therefore, this embodiment uses a single-shot multi-box detector (SSD) as the detection head. For example... Figure 4 As shown, the SSD network structure includes a front-end VGG-16 backbone network and a back-end multi-scale feature detection network. The algorithm performs forward propagation based on the VGG-16 network and utilizes multiple convolutional layers to determine the bounding box locations and classification information of targets of different sizes in the image, facilitating subsequent multi-scale feature extraction. The SSD algorithm primarily extracts multi-scale features from the feature maps output by six convolutional layers in the network; these feature maps are called prediction layers. The six prediction layers are Conv4_3, Conv7, Conv8_2, Conv9_2, Conv10_2, and Conv11_2, with output feature map sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1, respectively. Each prediction layer processes the multi-scale branch regression network (Loc_layers) and classification network (Conf_layers) through convolutional operations, followed by post-processing steps such as non-maximum suppression (NMS) to finally obtain the corresponding output results.

[0067] The loss function of the SSD model consists of localization loss and classification loss function, and its expression is as follows:

[0068] in, The classification loss function is represented as a multi-class softmax loss function, which includes both positive and negative samples. c For category predictions; The localization loss function is represented by the predicted bounding box ( l ) and real frame ( g The loss between () is calculated using the SmoothL1 loss function. x The deviation between the position and the actual bounding box. x The result is the match between the candidate box and the ground truth box; if there is a match, the value is 1, otherwise it is 0. N The number of candidate boxes that match the ground truth boxes. N If it is 0, then .

[0069]

[0070]

[0071] in, The classification probability after softmax normalization. This is the offset of the actual bounding box. It is used for smoothing regression. L 1. Loss function.

[0072] The SSD backbone feature extraction network, VGG16, consists of 13 convolutional layers and 3 fully connected layers, which directly leads to an excessive number of parameters. Although the SSD network exhibits good detection performance, in SLAM systems, a large number of parameters can cause detection latency and stuttering, affecting the real-time performance of the system. Therefore, this implementation uses the lightweight MobileNetV3 network as the backbone network of the SSD model. MobileNetV3 significantly reduces the number of parameters and optimizes the network structure through depthwise separable convolutions, inverse residual structures, and network simplification algorithms.

[0073] In the initial implementation, this method uses MobileNetV3 as the backbone network of SSD, thereby reducing the number of model parameters and improving lightweight design to meet the requirements of efficient and accurate real-time data processing. By replacing the backbone network in the SSD+VGG16 network model with MobileNetV3, a MobileNetV3-SSD network model is formed. This model has a total of 29 layers, including 17 layers of the MobileNetV3 feature extraction network and 12 layers derived from the 17th feature extraction layer. These 12 layers include 8 Conv2d convolutional layers and 4 depthwise convolutional layers (DW), which aim to provide feature maps of different scales for the SSD model. To provide the SSD module with feature maps of six different scales, the MobileNetV3-SSD network is connected to the MobileNetV3 backbone network via VGG feature extraction to a fully connected layer. Feature maps are extracted from layers bneck14, Conv17, Conv19, Conv21, Conv23, and Conv25, respectively. After prior bounding box processing, these maps are then fed into the fully connected layer for classification and regression tasks. The network structure is as follows: Figure 5 As shown.

[0074] MobileNetV3+SSD not only effectively reduces the computational and storage burden on the model, but also meets the accuracy requirements of SLAM systems. Through this design, this implementation significantly improves the target detection speed of the SLAM system while maintaining high accuracy, ensuring the entire system can run smoothly on resource-constrained platforms.

[0075] Preliminary identification strategy for dynamic feature points While epipolar constraints provide an effective method for identifying dynamic feature points, they can face accuracy challenges in certain situations. Specifically, when the displacement of dynamic feature points is small, or when static feature points exhibit large offsets due to computational errors, epipolar constraints may lead to misclassification. In such cases, relying solely on the offset distance of a feature point relative to the epipolar line to determine whether it is a dynamic feature point is not entirely reliable. In practice, feature points that should be identified as dynamic may be misclassified as static, or vice versa.

[0076] To address this issue, this implementation proposes a dynamic feature point recognition strategy combining epipolar constraints and target detection. First, this implementation calculates the dynamic probability of feature points using epipolar constraints. This method relies on the change in distance between a feature point and its corresponding epipolar line, thus assigning an initial dynamic probability to each feature point. Next, to further improve recognition accuracy, this implementation introduces semantic information provided by the target detection thread. Based on different regions and scene features in the image, different dynamic probability weights are assigned to feature points within different regions, with a weight of 3 for dynamic regions and 1 for static regions. By combining epipolar constraints and semantic information, this implementation can more accurately determine the dynamic characteristics of feature points and preliminarily classify them as dynamic or static based on their dynamic probabilities. Algorithm 1 summarizes the steps of the preliminary dynamic feature point recognition strategy.

[0077]

[0078] Dynamic Feature Point Refinement Inference Based on Conditional Random Fields Relying on epipolar constraints to identify dynamic feature points often only allows for judgment based on short-term geometric relationships between adjacent consecutive frames. However, when the trajectory of a dynamic object is close to the epipolar direction, or when the dynamic object occupies a large proportion of the image, the epipolar constraint-based method easily fails and struggles to effectively identify all dynamic feature points. To further improve the robustness and accuracy of dynamic feature point identification, this section proposes a dynamic feature point refinement inference strategy based on conditional random fields.

[0079] The introduction of the Conditional Random Field (CRF) model effectively utilizes information from different frames over a longer time span, enabling a more accurate distinction between the dynamic and static attributes of feature points. Practical observations show that dynamic feature points typically exhibit significant spatial displacement due to changes in camera position, orientation, and distance over time, while static feature points tend to remain stable. Furthermore, the photometric reprojection error of dynamic feature points is usually significantly higher than that of static feature points, resulting in poorer visual consistency; simultaneously, the spatial neighborhood of a feature point often possesses similar dynamic or static characteristics. Based on these observations, this implementation uses a CRF probabilistic model to establish a probabilistic graphical model of feature points, combining local geometric constraints with global spatial relationships to achieve refined judgment of dynamic feature points.

[0080] Conditional Random Fields This implementation constructs a conditional random field on the currently detected landmarks, connecting each pair of landmarks to each other through a fully connected graph model. Each landmark They were all given a label This belongs to the set {0,1} (where 0 represents static properties and 1 represents dynamic properties). This implementation aims to minimize the Gibbs energy defined under the CRF model. This is the ideal label configuration for identifying all landmarks.

[0081]

[0082] For a single potential energy In this embodiment, during the SLAM processing, observations of each landmark are recorded across multiple keyframes. This embodiment captures these landmarks in each keyframe. Two-dimensional observation position By measuring Its corresponding observation Photometric reprojection error between This embodiment calculates the average reprojection error. , here Indicates landmark The number of times it is observed. Furthermore, this implementation defines a first static probability based on all observations:

[0083] And a second static probability based on the average reprojection error:

[0084] in, and These represent the mean and standard deviation of their respective indicators. This framework allows this implementation to comprehensively assess the static nature of each landmark and provide input data for the CRF model to infer the most probable state.

[0085] For each detected landmark, this implementation yields two independent probability estimates, which indicate the landmark's... Whether it remains stationary. These probabilities are respectively... and This implementation calculates the landmark by taking a weighted average of these probabilities. The total probability of remaining stationary: The weighting coefficient , , The sum is 1. If Exceeding a specific threshold Landmark It has been preliminarily identified as static and has a static confidence level. If the value is below this threshold, it is considered dynamic and has a static confidence level. In practice, this embodiment will use all two... The values ​​are set equally. Following the definition in reference

[40] , the unit potential energy is set in the following form in this embodiment:

[0086] here Represents an indicator function.

[0087] For paired potential energy This implementation constructs a mechanism to promote the consistency of labels between landmarks and their neighbors, as shown below:

[0088] here, This represents a simplified Potts model. and Representative node and eigenvectors, and It is a Gaussian function. This implementation uses two types of Gaussian functions: the observation kernel and the position kernel.

[0089] The observation kernel is based on the idea that those kernels exhibit similar average reprojection errors. and number of observations Landmarks are likely to be of the same category. Dynamic landmarks appear in the same location in very few keyframes, while static landmarks appear consistently across multiple keyframes.

[0090] Within the Conditional Random Field (CRF) model, landmarks are assigned static or dynamic labels, a process facilitated primarily by two types of kernel functions: the observation kernel and the location kernel. These kernel functions crucially influence the calculation of pairwise potentials, thereby affecting the landmark labeling results.

[0091] The first kernel function, the observation kernel. The average reprojection error of landmarks in different frames was taken into account. and number of observations The difference, expressed as:

[0092] here, and These represent the average reprojection error and the number of observations for the landmark, respectively. and It represents the standard deviation of the corresponding statistic.

[0093] The second kernel function, the position kernel. Based on the assumption that neighboring 3D landmarks typically belong to the same object, regardless of whether the object is static or dynamic, the position kernel function is described as follows:

[0094] Here, and This refers to the location of a landmark in world coordinates. and This refers to the position of a landmark in the image coordinate system. and It is the standard deviation parameter that controls the width of the kernel function.

[0095] These two kernel functions work together to help the probabilistic model distinguish the states of landmarks. The observation kernel focuses on observation consistency, while the location kernel emphasizes spatial proximity. In this way, this implementation can effectively distinguish between static and dynamic landmarks, thereby improving the accuracy of camera pose estimation in dynamic scenes.

[0096] Dynamic feature point refinement inference strategy After initial feature point identification based on geometric and semantic information, this implementation obtains the dynamic probability prior of the feature points and initially classifies feature points with higher dynamic probabilities as dynamic points. However, a small number of misclassifications still occur. For example, Figure 5 As shown, a moving human body may have a small number of static feature points, while static areas may also have some dynamic feature points. To further distinguish between static and dynamic feature points more accurately, this implementation introduces a Conditional Random Field (CRF) model to construct a more accurate probabilistic reasoning mechanism, which refines the determination of dynamic feature points by combining preliminary dynamic probabilities.

[0097] During the initial identification process, most dynamic feature points have been eliminated, which significantly improves the accuracy of the initial pose estimation. Accurate initial pose estimation provides a reliable foundation for subsequent dynamic feature point refinement inference, enabling the conditional random field model to identify dynamic feature points more efficiently and accurately, especially in highly dynamic scenarios, effectively overcoming the limitations of epipolar constraints and probabilistic models in complex dynamic environments.

[0098] When using conditional random fields to identify dynamic feature points, this implementation further incorporates an object detection module. By assigning different threshold weights to feature points in the dynamic region (i.e., dynamic bounding box) and the static background region identified by object detection, it is ensured that feature points within the dynamic bounding box are more likely to be identified as dynamic points, while feature points located in the static background region are more likely to be identified as static points. In this way, this implementation effectively solves the problem that traditional methods struggle to accurately identify dynamic and static points in almost static scenes due to the small probability difference between them. Algorithm 2 demonstrates the process of the dynamic feature point refinement inference algorithm.

[0099] Through this strategy, this implementation method achieves more accurate dynamic feature point recognition in highly dynamic scenarios, optimizes the performance of the SLAM system in dynamic environments, enhances the robustness and accuracy of the system, and provides more reliable support for the application of visual SLAM systems in complex dynamic environments.

[0100]

[0101] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A progressive dynamic feature point elimination method based on geometric semantic probabilistic reasoning, characterized in that, The method comprises the following steps: a step of extracting feature points with scale invariance and rotation invariance from continuous image frames collected by a camera to obtain a feature point set; a step of performing feature point matching on adjacent frame images, obtaining a matching relationship based on a pyramid iterative optical flow method, calculating a fundamental matrix between two frames by using a graph cut random sample consensus algorithm, determining a dynamic prior probability of the feature points according to a displacement distance of the feature points and corresponding epipolar lines, and outputting a geometric dynamic prior set containing geometric constraint information; a step of inputting a current frame image into a lightweight MobileNetV3-SSD target detection network, extracting multi-scale features through a depth separable convolution and an inverse residual structure, and outputting a semantic information graph containing a dynamic region and a static region bounding box; a step of fusing the geometric dynamic prior set and the semantic information graph at a pixel coordinate level, assigning a higher dynamic weight to feature points located in a dynamic detection box, assigning a lower dynamic weight to static region feature points, calculating a comprehensive dynamic probability and performing preliminary classification according to a threshold, and outputting a preliminary dynamic feature point set; a step of constructing a conditional random field model, establishing a full connection graph structure taking the feature points as nodes, calculating unit potential energy according to a photometric re-projection error and a number of observations, calculating pair potential energy according to spatial proximity and observation similarity, and obtaining an optimal label distribution of the feature points being static or dynamic through energy minimization inference, and outputting a dynamic feature point set refined through inference; a step of removing feature points determined to be dynamic according to the refined inference result, and only retaining static feature points for camera pose estimation and map optimization, so as to realize progressive removal of dynamic feature points. 2.The progressive dynamic feature point elimination method based on geometric semantic probabilistic reasoning according to claim 1, wherein, In the step of feature point extraction, an ORB operator with scale invariance and rotation invariance is used for feature point detection, and key points and descriptors are extracted under a multi-layer pyramid structure. 3.The progressive dynamic feature point elimination method based on geometric semantic probabilistic reasoning according to claim 1, wherein, In the step of generating the geometric dynamic prior set, a pyramid iterative Lucas-Kanade optical flow method is used to establish an inter-frame feature point matching relationship, and a graph cut random sample consensus algorithm is used to solve the fundamental matrix.

4. The method of claim 1, wherein, In the step of generating the semantic information graph, the MobileNetV3-SSD target detection network constructs a lightweight backbone network through a depth separable convolution and an inverse residual structure, extracts multi-scale feature maps from bneck14, Conv17, Conv19, Conv21, Conv23 and Conv25 layers, and outputs semantic detection results containing dynamic and static regions through non-maximum suppression.

5. The method of claim 1, wherein, In the step of fusion calculation, dynamic region weight three is assigned to feature points located in a dynamic detection box, and static region weight one is assigned to feature points located in a static region, and the geometric dynamic probability corresponding to the epipolar line displacement and the semantic weight are fused by weighting.

6. The method of claim 1, wherein, In the step of refining inference of the conditional random field model, the unit potential energy is calculated based on the average photometric re-projection error and the number of observations of the feature points to obtain a static probability, the pair potential energy is based on a Gaussian kernel function model constructed based on the spatial position and observation difference of the feature points, including an observation kernel and a position kernel, and the optimal label distribution of each feature point is obtained through global energy minimization.

7. A device for progressive dynamic feature point rejection based on geometric semantic probabilistic reasoning, characterized in that, The method comprises the following steps: A module for extracting feature points with scale and rotation invariance from consecutive image frames captured by a camera to obtain a feature point set; A module for performing feature point matching on adjacent frame images, obtaining a matching relationship based on a pyramid iterative optical flow method, calculating an essential matrix between two frames using a graph cut random sample consensus algorithm, determining a dynamic prior probability of a feature point based on an offset distance of the feature point and a corresponding epipolar line, and outputting a geometric dynamic prior set containing geometric constraint information; A module for inputting a current frame image into a lightweight MobileNetV3-SSD target detection network, extracting multi-scale features through depth separable convolution and inverse residual structure, and outputting a semantic information graph containing dynamic region and static region bounding boxes; A module for fusing the geometric dynamic prior set and the semantic information graph at the pixel coordinate level, assigning a higher dynamic weight to feature points located in a dynamic detection box and a lower dynamic weight to static region feature points, calculating a comprehensive dynamic probability, and performing preliminary classification based on a threshold to output a preliminary dynamic feature point set; A module for constructing a conditional random field model, establishing a fully connected graph structure with feature points as nodes, calculating unit potential based on photometric re-projection error and observed times, calculating pairwise potential based on spatial proximity and observation similarity, and obtaining an optimal label distribution of feature points being static or dynamic through energy minimization inference to output a dynamic feature point set refined through inference; A module for removing feature points determined to be dynamic based on the refined inference result and retaining only static feature points for camera pose estimation and map optimization to achieve progressive removal of dynamic feature points.

8. Computer storage medium for storing a computer program, characterized in that When the computer program is read by the computer, the computer executes the method of claim 1.

9. A computer comprising a processor and a storage medium, characterized in that When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.

10. Computer program product as computer program, characterized in that When the computer program is executed, the method of claim 1 is implemented.

Images