A dynamic scene low-resource visual inertial odometer method based on VINS-RGB-D

Abstract

The application relates to the technical field of autonomous positioning of intelligent sanitation vehicles in dynamic scenes, and discloses a dynamic scene low-resource visual inertial odometer method based on VINS-RGB-D, which comprises the following steps: firstly, in order to obtain more uniform and stable feature distribution in images, a feature detection method based on image texture and interframe information is adopted to realize stable feature extraction; secondly, in order to realize fine-grained dynamic and static point differentiation, dynamic objects in the image are combined with target detection and recognition, depth distribution of feature points is constructed by using image depth information, and adaptive segmentation of foreground and background is realized by using clustering; dynamic state discrimination is carried out on the feature points in the target frame area. Meanwhile, dynamic feature recognition based on multi-frame consistency is carried out for the missed dynamic objects, and finally, residual dynamic points are further removed through re-projection error distribution, so that the positioning precision of the method in the dynamic scene is improved.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a low-resource visual inertial odometry method for dynamic scenes based on VINS-RGB-D. Background Technology

[0002] Visual-inertial simultaneous localization and mapping (VIO / VI-SLAM) technology, by fusing data from visual sensors and inertial measurement units, enables real-time pose estimation of mobile devices in unknown environments and has been widely applied in mobile robots, drones, augmented reality, and autonomous driving. However, most existing visual-inertial SLAM methods are based on static environmental assumptions and are easily affected by moving objects in dynamic scenes. The introduction of dynamic features can cause observation constraints to fail, resulting in decreased pose estimation accuracy or even system divergence.

[0003] To address the aforementioned issues, existing technologies have proposed various improved solutions for dynamic scenes. One type of method filters out anomalous features through geometric consistency constraints or reprojection error elimination mechanisms with fixed thresholds. However, these methods typically rely on empirical thresholds, making them difficult to adapt to different motion states and complex environmental changes, and prone to misjudgments in the presence of significant camera movement or noise interference. Another type of method introduces visual perception techniques such as semantic segmentation or object detection to mask or eliminate regions that may contain dynamic objects, thereby reducing the impact of dynamic features. However, these methods usually require highly complex deep learning models, incurring significant computational overhead and making them unsuitable for real-time operation on low-resource computing platforms. Furthermore, the detection results are easily affected by factors such as missed detections and jitter in practical applications, leading to temporal discontinuities in dynamic region constraints and thus residual dynamic features.

[0004] Existing semantic or detection-based methods often employ coarse-grained removal strategies at the target or region level. They typically treat all features within the target detection bounding box as dynamic features and delete them, ignoring potential static background structures within the target region. This can easily lead to over-removal of effective features, reducing the sufficiency of geometric constraints. Furthermore, in cases of short-term missed detections of dynamic targets, existing methods often lack effective compensation mechanisms, causing interruptions in dynamic region constraints. This allows dynamic features from the missed detection period to re-enter the system, further affecting the stability of the localization results. Current visual-inertial SLAM technology still suffers from the following shortcomings in dynamic scenes: dynamic feature removal often relies on fixed rules or high-computational-power models, making it difficult to balance robustness with low-resource real-time performance; the granularity of dynamic region constraints is coarse, easily leading to the accidental deletion of static background features; there is a lack of effective temporal compensation methods for missed target detections; and there is a lack of adaptive fallback removal mechanisms for residual dynamic features and mismatched outliers. Summary of the Invention

[0005] This invention proposes a low-resource visual inertial odometry method for dynamic scenes based on VINS-RGB-D. Unlike existing methods that either completely remove features within the target detection region or rely primarily on fixed rule filtering, the key to this invention lies in: under the constraint of the target detection bounding box, combining RGB-D depth information to perform point-level subdivision of feature points within the box into foreground and background, and further incorporating multi-frame consistency verification, dynamic target omission compensation, and adaptive reprojection error removal mechanisms to achieve fine-grained dynamic feature suppression in dynamic scenes. This method, while ensuring real-time performance, reduces the false deletion of static background features and minimizes the impact of residual dynamic features on pose estimation, thereby improving the robustness and positioning accuracy of the method in dynamic environments.

[0006] This invention provides a low-resource visual inertial odometry method for dynamic scenes based on VINS-RGB-D, comprising the following steps:

[0007] 1. Acquire RGB-D image sequences and IMU data, and perform time synchronization and preprocessing;

[0008] 2. Extract and track feature points in the image to form cross-frame feature trajectories;

[0009] 3. Perform object detection on the current frame image and obtain the target bounding boxes (ROIs) that may contain moving objects;

[0010] 4. Within the target detection bounding box, feature points are divided into fine-grained point-level segments based on depth information to obtain dynamic candidate features;

[0011] 5. Based on the temporal consistency of multiple frames, dynamic candidate features are confirmed, and only dynamic features that appear continuously are eliminated;

[0012] 6. When a target detection miss occurs, the target detection bounding box is extrapolated and predicted based on the historical motion state to achieve miss compensation;

[0013] 7. Statistically analyze the reprojection error distribution of feature points within the sliding window and adaptively remove residual outliers;

[0014] 8. Use the static feature points after removing dynamic features and IMU data to perform visual inertial optimization and output the camera pose.

[0015] It should be noted that the improvement of this invention does not lie in using any single technique among target detection, depth information utilization, or reprojection error removal, but rather in combining target detection region constraints, in-frame depth point subdivision, multi-frame temporal consistency verification, missed detection compensation, and adaptive removal of residual outliers, and applying these combinations to the visual-inertial odometry process in dynamic scenes. Through this combination, hierarchical suppression of dynamic features can be achieved with lower computational resources, enhancing the stability and adaptability of the method in complex dynamic environments.

[0016] To address the issue that the bounding box of an object detection may simultaneously contain both dynamic foreground and static background, this invention does not remove all features within the detection box, but instead performs point-level subdivision by combining RGB-D depth information.

[0017] For the Target detection bounding box Construct a depth set of feature points with effective depth observations within this region:

[0018]

[0019] In one implementation, the depth set Perform one-dimensional clustering

[0020] Two cluster centers were obtained and The threshold for separating foreground and background can be determined based on the clustering results.

[0021]

[0022] When the depth of the feature points satisfies When a feature point is located in the foreground of the target area, it is identified as a dynamic candidate feature point; otherwise, it is considered a static background feature point and retained. This method enables fine-grained differentiation of feature points within the target area, avoiding the accidental deletion of static background features.

[0023] To reduce misjudgments caused by single-frame noise, depth errors, or detection jitter, this invention introduces a multi-frame temporal consistency constraint, which performs final elimination only on features that have been judged as dynamic candidates in multiple consecutive frames.

[0024] In one implementation, consistency determination between two adjacent frames can be used, expressed as:

[0025]

[0026] in, Representing feature points In the Whether a frame is identified as a dynamic candidate. This indicates that the feature has been identified as a dynamic feature.

[0027] This multi-frame consistency mechanism effectively suppresses erroneous deletions caused by transient noise and improves the stability of dynamic feature determination.

[0028] To address the issues of missed detections or short-term failures in target detection in practical applications, this invention introduces a missed detection compensation mechanism based on historical motion states when target detection results are missing, in order to maintain the temporal continuity of dynamic region constraints.

[0029] When no dynamic target present in the previous frame is detected in the current frame, the target detection bounding box is extrapolated and predicted based on the target's motion state in historical frames. In one implementation, the target detection bounding box is predicted as follows:

[0030]

[0031] in, Indicates the location of the historical bounding box. This represents the target's velocity estimated from historical frames.

[0032] In another embodiment, the motion velocity can be estimated using an exponential moving average or a motion model incorporating an acceleration term to enhance the smoothness and robustness of the extrapolation prediction. Through missed detection compensation, the reintroduction of dynamic targets into the system approach during short-term missed detection periods can be avoided, thereby reducing residual dynamic features.

[0033] Even after dynamic feature removal based on detection and depth, residual outliers may still exist due to missed detections, mismatches, or extreme motion. Therefore, this invention further performs statistical analysis on the reprojection error of feature points within a sliding window.

[0034] In one implementation, the reprojection error of all feature points within the sliding window is statistically analyzed, and its mean is calculated. with standard deviation Based on this, the rejection threshold is adaptively determined:

[0035]

[0036] When the reprojection error of feature points is greater than the threshold When this occurs, it is identified as an outlier and removed. This method can serve as a fallback mechanism for dynamic feature suppression, further purifying the geometric constraints entering the backend optimization.

[0037] This invention performs depth subdivision, multi-frame consistency verification, and residual outlier removal only within the area corresponding to the target detection bounding box, avoiding high-complexity dynamic feature screening of the entire image, thereby reducing the overall computational load and making it suitable for low-resource computing platforms.

[0038] The beneficial effects of this invention are as follows:

[0039] (1) Under the constraint of the target detection bounding box, the present invention combines the depth information of the RGB-D image to perform point-level subdivision of the feature points in the box into foreground and background, which can avoid the overall deletion of static background features in the target area and improve the retention rate of effective visual geometric constraints.

[0040] (2) By introducing a multi-frame temporal consistency confirmation mechanism and a bounding box extrapolation prediction mechanism based on historical motion states, this invention can reduce the impact of instantaneous noise, depth error, detection jitter and short-term missed detection on the dynamic feature recognition results, and improve the robustness and stability of the system in complex dynamic scenarios.

[0041] (3) By adaptively determining the rejection threshold based on the reprojection error distribution, and performing processes such as depth subdivision, time sequence confirmation and outlier rejection only within the candidate dynamic region, this invention can further purify the back-end optimization constraints while reducing the overall computational overhead, thereby improving the positioning accuracy and real-time performance of the visual inertial odometry. Attached Figure Description

[0042] Figure 1 This is a diagram showing the overall system framework of the present invention;

[0043] Figure 2 This is a flowchart of the method of the present invention;

[0044] Figure 3 This is a schematic diagram illustrating the multi-frame consistency verification of dynamic candidate features in this invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. In this embodiment, an RGB-D camera is used to acquire image and depth information in a dynamic scene, an inertial measurement unit (IMU) is used to acquire acceleration and angular velocity information, and a processor executes the corresponding program to complete visual inertial odometry processing. It should be noted that the following embodiments are only used to illustrate the technical solution of the present invention, and are not intended to limit the scope of protection of the present invention; equivalent substitutions or modifications made by those skilled in the art without departing from the technical concept of the present invention should all fall within the scope of protection of the present invention.

[0046] The method of this invention does not completely delete features within the target detection bounding box. Instead, based on the target detection results, it combines depth information to divide the feature points within the box into foreground and background, and further combines multi-frame consistency verification, missed detection compensation, and adaptive removal of reprojection errors to suppress dynamic features layer by layer. The method specifically includes the following steps:

[0047] 1: Data Acquisition and Synchronization

[0048] In this embodiment, a continuous image sequence is acquired using an RGB-D camera, while acceleration and angular velocity data are acquired simultaneously using an inertial measurement unit (IMU). The RGB-D image data and IMU data are synchronized in time, and preprocessing operations such as distortion correction and scale normalization are performed on the images to ensure consistency of multi-sensor data along the time axis.

[0049] 2: Image Feature Extraction and Tracking

[0050] Image feature points are extracted from each frame of RGB image, and cross-frame association of feature points is performed through optical flow tracing or feature matching to form a continuous feature trajectory.

[0051] In one implementation, the spatial distribution of feature points can be constrained to ensure the uniformity of feature points on the image plane, thereby reducing the computational overhead caused by redundant features and improving the operating efficiency on low-resource platforms.

[0052] 3: Dynamic Target Detection

[0053] An object detection operation is performed on the current frame image to obtain at least one object detection bounding box. This bounding box serves as a candidate dynamic region of interest (ROI), indicating an image region that may contain dynamic objects. The object detection output is in the form of a rectangular bounding box, whose category information can be used to identify potential dynamic targets. In one implementation, the object detection module may prioritize pedestrians, vehicles, cyclists, or other target categories with the potential for movement, and output the position and category information of the corresponding bounding boxes to provide region constraints for subsequent dynamic discrimination of feature points within the bounding boxes.

[0054] 4: Depth-based point-level fine-grained dynamic candidate feature determination

[0055] For each object detection bounding box, feature points located within the bounding box are collected in the RGB-D image, and a depth set of feature points with effective depth observations is constructed:

[0056]

[0057] in, Indicates the first Each target detection bounding box region This represents the depth value of the corresponding feature point. In one implementation, the depth set... One-dimensional clustering was performed with a cluster number of 2, resulting in two cluster centers. and Based on the clustering results, determine the threshold for separating foreground and background, and take the median of the cluster centers:

[0058]

[0059] When the feature point depth satisfies If the desired feature point is selected, it is marked as a dynamic candidate feature point; otherwise, it is considered a static background feature point and retained. This method enables fine-grained differentiation of feature points within the target detection bounding box, avoiding the accidental deletion of static background features. When the number of effective depth feature points within the target detection bounding box is insufficient, or when there are missing depth points or abnormal depth values, the final deletion is not based directly on the clustering results of that frame. Instead, the corresponding features are retained for further evaluation in subsequent multi-frame consistency confirmation and reprojection error removal steps.

[0060] 5: Confirmation of dynamic features for multi-frame consistency

[0061] To reduce misjudgments caused by single-frame noise or detection jitter, this implementation introduces multi-frame temporal consistency constraints. A feature point is only ultimately confirmed as a dynamic feature and removed if it is identified as a dynamic candidate feature point in multiple consecutive frames. In one implementation, consistency judgment between adjacent frames is used, and the judgment logic is expressed as follows:

[0062]

[0063] in, Representing feature points In the Dynamic candidate states within a frame. In other implementations, a consistency determination method based on three or more consecutive frames can be used to further reduce the risk of misjudgment caused by single-frame false detection, depth noise, or detection jitter.

[0064] 6: Dynamic target missed detection compensation

[0065] When no dynamic target present in the previous frame is detected in the current frame, this embodiment extrapolates and predicts the target detection bounding box based on the target's motion state in historical frames to maintain the temporal continuity of dynamic region constraints. In one embodiment, the target detection bounding box prediction method is as follows:

[0066]

[0067] in, This represents the target motion velocity estimated from historical frames. In another embodiment, the motion velocity can be estimated using an exponential moving average of historical velocities or a motion model including an acceleration term to enhance the stability of the prediction results.

[0068] The predicted bounding boxes are used for dynamic feature determination in the current frame, thereby reducing the risk of dynamic features re-entering the system during missed target detection. When the same target is not detected again for several consecutive frames, bounding box extrapolation prediction can be stopped to avoid misjudgments caused by continuous shift in the predicted region.

[0069] 7: Adaptive Residual Outline Removal Based on Reprojection Error Distribution

[0070] Even after completing the dynamic feature removal process described above, residual outliers may still exist due to missed detections, mismatches, or complex motions. Therefore, the reprojection error distribution of feature points is statistically analyzed within a sliding window, and the removal threshold is adaptively determined based on this distribution.

[0071] In one implementation, the mean of the reprojection error is calculated. and standard deviation And based on this, determine the rejection threshold:

[0072]

[0073] When the reprojection error of feature points is greater than the threshold When that happens, it is identified as an outside point and removed.

[0074] 8: Visual-inertial pose estimation

[0075] After removing dynamic features and residual outliers, a visual inertial constraint model is constructed using the retained static feature points and IMU data. A sliding window optimization method is then employed for visual inertial state estimation, ultimately outputting the camera pose. Since the aforementioned depth subdivision, multi-frame consistency verification, and missed detection compensation processes primarily revolve around the region corresponding to the target detection bounding box, they can reduce overall computational overhead while maintaining dynamic feature suppression effectiveness, making them more suitable for real-time operation on low-resource computing platforms.

Claims

1. A low-resource visual inertial odometry method for dynamic scenes based on VINS-RGB-D, characterized in that, Includes the following steps: S1. Acquire RGB-D camera image sequences and inertial measurement unit (IMU) data, and perform time synchronization and preprocessing on the image data and IMU data; S2. Extract and track image feature points in each frame to obtain cross-frame associated feature trajectories; S3. Perform object detection on the current image and obtain at least one object detection bounding box to indicate the image region that may contain dynamic objects; S4. For each target detection bounding box, the feature points located within the bounding box are grouped using the depth information in the RGB-D image, and a feature point depth set is constructed using the depth information provided by the RGB-D image: in, Indicates the first Each target detection bounding box region This represents the depth value of the corresponding feature point. Based on the clustering results, a threshold for distinguishing between foreground and background points can be determined, and the median of the cluster centers is taken. When the feature point depth satisfies When this happens, mark it as a dynamic candidate feature point. S5. Based on the determination results of feature points in multiple consecutive frames, perform temporal consistency judgment on the dynamic candidate feature points. Only when a feature point continuously meets the dynamic candidate condition in multiple frames is it confirmed and removed; the consistency judgment is adopted between adjacent frames. in, Representing feature points In the Dynamic candidate states in frames S6. When the target detection does not detect the dynamic target that existed in the previous frame in the current frame, extrapolate the corresponding target detection bounding box according to the motion state of the dynamic target in the historical frame, and use the predicted bounding box for the dynamic feature determination of the current frame. S7. Statistically determine the reprojection error distribution of feature points within the sliding window, adaptively determine the removal threshold based on the error distribution, and further remove residual out-of-point features; S8. Use the static feature points after removing dynamic features and IMU data to perform visual inertial optimization and output the camera pose estimation results.

2. The method for low-resource visual inertial odometry in dynamic scenes based on VINS-RGB-D as described in claim 1, characterized in that: The grouping process in step S4 is a one-dimensional clustering process with 2 clusters. The threshold for separating foreground and background points is determined by the median of the cluster centers obtained from the one-dimensional clustering. The multi-frame consistency judgment in step S5 is based at least on the judgment results of two adjacent frames. The target detection bounding box extrapolation prediction in step S6 is based on the target's motion velocity in historical frames. The motion velocity is obtained through an exponential moving average of historical velocities or a motion model including an acceleration term. The adaptive culling threshold in step S7 satisfies the following: in, and These represent the mean and standard deviation of the reprojection error, respectively. The statistical distribution includes the mean and standard deviation of the reprojection error. The calculations in steps S4 to S7 are performed only within the image region corresponding to the target detection bounding box to reduce computational overhead. In step S8, a sliding window optimization method is used for visual inertial state estimation.

Images