A dynamic SLAM method and system based on feature point screening
By using the improved RT-DETR algorithm and geometric optical flow method in visual SLAM, the problem of too few feature points in the dynamic environment is solved, the dynamic scene performance and stability of the SLAM algorithm are improved, and more accurate Pose optimization and map construction.
Patent Information
- Application Number
- CN202410091027.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-01-23
AI Technical Summary
The existing visual SLAM algorithm is difficult to accurately build maps in dynamic environments, especially when there are many dynamic objects. Directly removing feature points on dynamic objects will lead to too few feature points, affecting the accuracy and stability of position optimization. .
Using a dynamic SLAM method based on feature point filtering, the improved RT-DETR algorithm is used to divide the dynamic areas, and the feature points are screened in combination with geometry and optical flow methods, retaining the feature points on the stationary dynamic object to participate in the position optimization, which improves the algorithm Performance and stability in dynamic scenarios.
It improves the positioning accuracy and stability of the SLAM algorithm in dynamic scenarios, avoids the problem of inaccurate map construction caused by too few feature points, and takes into account the situation of stationary dynamic objects, which enhances the applicability of the system.
Smart Images

Figure CN117911461B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a dynamic SLAM method and system based on feature point screening. Background Art
[0002] SLAM refers to a subject equipped with specific sensors, which builds a model of the environment and estimates its own movement during movement without prior information about the environment. Common sensors include LiDAR, camera, IMU, etc. Visual SLAM uses camera as the main sensor and image as the main source of environmental information. Compared with LiDAR and inertial devices, cameras are cheaper, smaller, lighter, and the information returned contains environmental semantic information, such as the category and attributes of objects. Visual SLAM can achieve high-precision positioning and navigation in a variety of indoor and outdoor scenarios, and has a wide range of applications in autonomous driving, drones, robots, AR / VR, smart homes and other fields. At present, the more classic visual SLAM algorithms include ORB-SLAM2, VINS-Fusion, etc. Visual SLAM processes continuous camera frames, extracts pixels with certain features from the camera frames, and estimates the position of the camera when taking each frame by tracking the position of these feature points in the camera frame, and then builds an environmental map based on the camera position. The existing visual SLAM algorithm mainly includes five modules: sensor data reading: reading and preprocessing of camera image information; visual odometer: extracting feature points in the image, and estimating the movement of the camera between adjacent images through matching pairs of feature points between two frames, and building a local map; back-end optimization: dealing with the noise problem in the SLAM process; mapping: identifying scenes that have been reached, and then correcting the deviation of the camera motion trajectory; loop detection: establishing the corresponding surrounding environment map based on the estimated camera motion trajectory.
[0003] Traditional visual SLAM algorithms usually assume that the environment is static when estimating the movement of the camera between adjacent images, extract feature points from all objects in the camera frame and track them. However, when there are dynamic objects in the environment, the dynamic feature points located on the dynamic objects will contain huge errors due to the movement of the dynamic objects. Using these dynamic feature points for pose estimation will lead to inaccurate map construction, which often leads to SLAM failure when there are many dynamic objects in the field of view. Existing dynamic SLAM algorithms generally use geometry, optical flow or deep learning methods to eliminate the influence of dynamic objects. SLAM algorithms that use geometry and optical flow methods to eliminate dynamic objects often find it difficult to accurately eliminate dynamic feature points and complete positioning and map construction tasks due to the lack of prior accurate semantic information. SLAM algorithms that use deep learning methods to remove dynamic objects remove feature points on dynamic objects through the semantic information of the objects. This method can often accurately remove dynamic feature points in the camera frame while retaining static feature points. It has shown good performance in most dynamic scenes. However, most methods remove all feature points on dynamic objects without verification, and do not take into account the situation of stationary dynamic objects. When there are many dynamic objects in the field of view, directly removing all feature points on all dynamic objects will result in too few feature points participating in pose optimization, making it impossible to accurately complete map construction. Summary of the invention
[0004] One of the purposes of the present invention is to provide a dynamic SLAM method based on feature point screening, first using an improved RT-DETR algorithm for dynamic area division, then using geometry and optical flow methods to screen feature points, retaining as many feature points as possible to participate in pose optimization. Compared with the original ORB-SLAM3 algorithm, the method of this application improves the performance of the algorithm in dynamic scenes, and compared with the SLAM method of fusion deep learning that directly removes feature points on all dynamic objects, the method of this application takes into account the situation of static dynamic objects, further improving the stability of SLAM.
[0005] An embodiment of the present invention provides a dynamic SLAM method based on feature point screening, comprising:
[0006] Get the current frame image;
[0007] Extracting ORB feature points from the current frame image to obtain ORB feature points;
[0008] Based on the improved RT-DETR algorithm, identifying the dynamic area in the current frame image;
[0009] Performing epipolar constraint verification and optical flow verification on the ORB feature points falling into the dynamic area respectively;
[0010] It is used to involve the ORB feature points that pass the epipolar constraint check and the optical flow check and the ORB feature points that do not fall into the dynamic area in camera tracking, and then perform local mapping, loop detection and map merging.
[0011] Preferably, the step of improving the RT-DETR algorithm comprises:
[0012] Use the PP-LCNet network to replace the backbone network in the original RT-DETR algorithm;
[0013] The improved RT-DETR algorithm is used to identify the dynamic area in the current frame image.
[0014] Preferably, the step of checking the epipolar constraint comprises:
[0015] Based on the epipolar constraint method, epipolar constraint verification is performed on the ORB feature points falling into the dynamic area.
[0016] Preferably, the optical flow verification step includes:
[0017] Based on the optical flow method, an optical flow check is performed on the ORB feature points falling into the dynamic area.
[0018] Preferably, the dynamic SLAM method based on feature point screening also includes:
[0019] Get the first historical map within the most recent preset time;
[0020] Determining a transition probability based on the first historical map;
[0021] When the transition probability is greater than or equal to a preset probability threshold, obtaining a second historical map within a recent preset second time;
[0022] Based on the second historical mapping and the preset standard mapping library, determining the transition type and the probability table of dynamic object appearance after the transition;
[0023] Referring to the transition type and the probability table of dynamic object appearance after the transition, the dynamic area in the future frame image is identified based on the improved RT-DETR algorithm.
[0024] Preferably, the determining the transition probability based on the first historical map building includes:
[0025] Determining whether the first historical mapping meets a first image condition;
[0026] When the condition is met, the transition probability is calculated based on a preset first threshold; the first threshold is greater than or equal to the probability threshold;
[0027] When it does not meet the requirements, determining whether the first historical mapping meets the second image condition;
[0028] When there is a match, the transition probability is the sum of the preset condition representation degrees corresponding to the second image condition that the first historical mapping meets;
[0029] When neither condition is met, the transition probability is calculated based on a preset second threshold; the second threshold is smaller than the probability threshold;
[0030] Wherein, the first image condition includes:
[0031] The mapping completeness of the first historical mapping is greater than or equal to a preset first completeness threshold;
[0032] A partial image representing a transition indicator object in a preset transition indicator object library appears in the first historical mapping;
[0033] The matching degree between the local direction sequence after the local image is established in the mapping direction sequence of the first historical mapping and the preset standard direction sequence is greater than or equal to a preset matching degree threshold;
[0034] Wherein, the second image condition includes:
[0035] The mapping completeness of the first historical mapping is less than the first completeness threshold and greater than or equal to a preset second completeness threshold;
[0036] The proportion of return routes in the mapping movement trajectory of the first historical mapping is greater than or equal to a preset proportion threshold.
[0037] Preferably, the determining of the transition type and the probability table of dynamic object appearance after the transition based on the second historical mapping and the preset standard mapping library includes:
[0038] Performing feature extraction on the second historical map to obtain multiple image features;
[0039] generating an image search condition based on the plurality of image features;
[0040] Searching for a plurality of standard images that meet the image search condition from the standard image library;
[0041] Classifying the plurality of standard images to obtain a set of dynamic objects appearing in a plurality of image scene types;
[0042] Using the image scene type as a transition type;
[0043] Counting the appearance frequencies of different dynamic objects in the dynamic object set;
[0044] Based on the occurrence frequencies of different dynamic objects, a probability table of occurrence of dynamic objects after the transition is generated.
[0045] An embodiment of the present invention provides a dynamic SLAM system based on feature point screening, comprising:
[0046] An acquisition module is used to acquire the current frame image;
[0047] An extraction module, used for extracting ORB feature points from the current frame image to obtain ORB feature points;
[0048] An identification module, used for identifying a dynamic area in the current frame image based on an improved RT-DETR algorithm;
[0049] A verification module, used for performing epipolar constraint verification and optical flow verification on the ORB feature points falling into the dynamic area;
[0050] The subsequent processing module is used to involve the ORB feature points that have passed the epipolar constraint check and the optical flow check and the ORB feature points that do not fall into the dynamic area in camera tracking, and then perform local mapping, loop detection and map merging.
[0051] Preferably, the step of improving the RT-DETR algorithm comprises:
[0052] Use the PP-LCNet network to replace the backbone network in the original RT-DETR algorithm;
[0053] The improved RT-DETR algorithm is used to identify the dynamic area in the current frame image.
[0054] Preferably, the step of checking the epipolar constraint comprises:
[0055] Based on the epipolar constraint method, epipolar constraint verification is performed on the ORB feature points falling into the dynamic area.
[0056] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.
[0057] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0059] Figure 1 Schematic diagram of the architecture of a dynamic SLAM method based on feature point screening in an embodiment of the present invention;
[0060] Figure 2 A schematic diagram of the PPLCNet-RT-DETR network structure in an embodiment of the present invention;
[0061] Figure 3 is a schematic diagram of epipolar constraint in an embodiment of the present invention;
[0062] Figure 4 A schematic diagram of the epipolar constraint screening results in an embodiment of the present invention;
[0063] Figure 5 A schematic diagram of the screening results of the optical flow method in an embodiment of the present invention;
[0064] Figure 6 A schematic diagram showing comparison of feature point extraction results in an embodiment of the present invention;
[0065] Figure 7 The figure is a schematic diagram comparing the absolute trajectory error and the relative trajectory error between the embodiment of the present invention and ORB-SLAM3. DETAILED DESCRIPTION
[0066] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0067] First, to facilitate the understanding of the embodiments, the applicant explains the following professional terms:
[0068] IMU: Inertial motion unit, a combination of accelerometer and gyroscope sensors. Used to detect acceleration and angular velocity to indicate motion and motion intensity.
[0069] Pose: The 3D pose of a camera includes the spatial position of the camera and the orientation of the camera.
[0070] Feature points: The main problem of visual odometer in visual SLAM is how to estimate camera motion based on images. However, the image itself is a matrix composed of brightness and color. It will be very difficult to consider motion estimation directly at the matrix level. Therefore, it is generally customary to adopt such an approach: first, select more representative points from the image. These points will remain unchanged after a small change in the camera's perspective, so the same points will be found in each image. Then, based on these points, the camera pose estimation problem and the positioning problem of these points are discussed. In the classic SLAM model, they are called landmarks.
[0071] Absolute trajectory error (ATE): directly calculates the difference between the true value of the camera pose and the estimated value of the SLAM system. The program first aligns the true value and the estimated value according to the timestamp of the pose, then calculates the difference between each pair of poses, and finally outputs it in the form of a chart. This criterion is very suitable for evaluating the performance of visual SLAM systems.
[0072] Relative trajectory error (RPE): used to calculate the difference in the pose change at the same two timestamps. Similarly, after aligning the timestamps, the true pose and the estimated pose are calculated at the same interval of time, and then the change is subtracted to obtain the relative pose error. This standard is suitable for estimating the drift of the system.
[0073] Instance segmentation: A combination of object detection and semantic segmentation, where objects are detected in an image (object detection) and then each pixel is labeled (semantic segmentation).
[0074] The embodiment of the present invention provides a dynamic SLAM method based on feature point screening, such as Figure 1 As shown, including:
[0075] Get the current frame image;
[0076] Extracting ORB feature points from the current frame image to obtain ORB feature points;
[0077] Based on the improved RT-DETR algorithm, identifying the dynamic area in the current frame image;
[0078] Performing epipolar constraint verification and optical flow verification on the ORB feature points falling into the dynamic area respectively;
[0079] It is used to involve the ORB feature points that pass the epipolar constraint check and the optical flow check and the ORB feature points that do not fall into the dynamic area in camera tracking, and then perform local mapping, loop detection and map merging.
[0080] The steps of improving the RT-DETR algorithm include:
[0081] Use the PP-LCNet network to replace the backbone network in the original RT-DETR algorithm;
[0082] The improved RT-DETR algorithm is used to identify the dynamic area in the current frame image.
[0083] The step of checking the epipolar constraint comprises:
[0084] Based on the epipolar constraint method, epipolar constraint verification is performed on the ORB feature points falling into the dynamic area.
[0085] The steps of optical flow verification include:
[0086] Based on the optical flow method, an optical flow check is performed on the ORB feature points falling into the dynamic area.
[0087] like Figure 1 As shown, this application uses ORB-SLAM3 as the basic framework. ORB-SLAM3 has three threads: Tracking, Local Mapping, Loop & Map Merging. Based on ORB-SLAM3, a dynamic area division module and a feature point screening module are added. In the tracking thread, the dynamic area division module is first used to divide the dynamic area. Then, in the feature point screening module, the feature points falling in the dynamic area are double-screened using the epipolar constraint and the optical flow method. Finally, the feature points that pass the double screening are regarded as valid static feature points and retained to participate in the subsequent pose optimization to improve the positioning accuracy and stability of the SLAM algorithm.
[0088] Improved RT-DETR algorithm: RT-DETR (Real-Time DEtection TRansformer) is the first real-time end-to-end object detector. This model designs an efficient hybrid encoder. Different from the general DETR series algorithms, this encoder improves the running speed by decoupling the intra-scale interaction and cross-scale fusion. RT-DETR provides pre-trained models that use networks such as ResNet and HGNet as backbones. These models have deeper network layers and wider network bandwidth, and have higher requirements on the computing power of the running platform. In order to reduce network parameters and increase the running rate to adapt to the limited computing power environment of the robot platform, the method of this application uses the PP-LCNet network to replace the original backbone network. PP-LCNet is a lightweight, high-performance CPU network based on the MKLDNN acceleration strategy. The network is based on deep separable convolution as the basic module, uses HSwish instead of ReLU as the activation function, and adds the SE attention mechanism to enhance significant features while suppressing unimportant features. It shows strong performance on the CPU side and can better adapt to the general application environment of the SLAM algorithm. The structural diagram of the improved RT-DETR algorithm of this application is as follows Figure 2 shown.
[0089] In application scenarios, semantically dynamic objects may actually be in a static state, such as sleeping cats and dogs, cars parked on the roadside, etc. At the same time, the target detection frame often covers part of the static background. In the case of many dynamic objects, blindly excluding all feature points in the dynamic object detection frame may result in too few feature points for pose estimation, and the accuracy of the SLAM algorithm will decrease instead of increase. Therefore, the dynamic area divided by PPLCNet-RT-DETR is screened again using the optical flow method and epipolar constraints, and those feature points that pass the optical flow check and epipolar constraint check, that is, those that are likely to be static dynamic objects or belong to the background, are retained to participate in the subsequent pose optimization to improve the performance of the SLAM algorithm.
[0090] The epipolar constraint utilizes the matching relationship of feature points in camera images collected at different times or locations, and determines the relative position of the cameras by calculating the essential matrix or basic matrix between the two camera views. It is applied to matching image feature points, 3D reconstruction, target tracking, camera positioning, image stitching, feature matching and 3D object recognition to improve accuracy and performance.
[0091] like Figure 3 As shown, O1 and O2 are the camera centers of the corresponding cameras that took I1 and the two frames of images respectively. Consider that there is a feature point p1 in I1 and the matching feature point p2 in I2. p1 and p2 are the projections of the spatial point P in the real world on the two imaging planes. Ideally, the extension line of the line O1p1 and the extension line of the line O2p2 will intersect at point P. This plane formed by PO1O2 is called the polar plane. The intersection points I2e1 and e2 of the line O1O2 and the image planes I1 and I2 are called poles, and the line O1O2 is called the baseline. The intersection lines of l1 and l2 as the polar plane and the image plane are called polar lines.
[0092] Given two consecutive frames I1 and I2, the basic matrix can transform the projection point of the spatial point P in the first frame into the corresponding epipolar line of the second frame. Ideally, the projection point of the spatial point P in the second frame should fall on the corresponding epipolar line of the second frame. Let p1 and p2 be the projection of the spatial point P in the I1 and I2 frames respectively, and the coordinates in the corresponding camera coordinate system are denoted as p1 = [x1, y1, 1] T 、p2=[x2,y2,1] T ;
[0093] l2 is the polar line corresponding to the spatial point P in the I2 frame, and l2 = Fp1,
[0094]
[0095] Where A, B, and C represent line vectors, and the basic matrix F can be estimated by using the RANSAC algorithm on the ORB feature point matching pairs of the previous and next frames. When the matching relationship between p1 and p2 between the two frames I1 and I2 is known, the distance between the actual position of p2 and the ideal epipolar line l2 is calculated by the following formula:
[0096]
[0097] After obtaining the distance, the moving points are extracted from the outlier point set by setting the threshold. If
[0098] Distanse>thresholds;
[0099] Then the point is considered to be part of the moving object. The thresholds is the distance between two pixels. The epipolar constraint screening results are as follows Figure 4 As shown in the figure, through the epipolar constraint check, most of the feature points on the moving dynamic objects can be screened out, while the feature points on the static dynamic objects can be retained. Figure 4 (a) represents all feature points, (b) represents points that do not satisfy the epipolar constraint, and (c) represents the result of removing abnormal points.
[0100] Optical flow is a two-dimensional vector that provides information about the relative displacement of each pixel in the current frame relative to the previous frame. Optical flow has a wide range of applications, such as motion-based segmentation, global motion compensation, and video compression. In indoor environments, all points in the background tend to have similar optical flows when the camera moves, because the projected position of the object on the imaging plane moves with the movement of the camera, and the magnitude of the optical flow of dynamic objects is different due to their independent movement from the background.
[0101] Generally speaking, the optical flow problem can be solved as an optimization problem. However, the Horn-schunck and Lucas-Kanade optical flow methods are only applicable to small motions. If the object moves quickly and the brightness changes rapidly, a small mask (such as 2*2 or 3*3 size) will not be able to estimate the spatiotemporal derivatives. In this case, a pyramid can be used to calculate a large optical flow vector. When calculating the optical flow, the brightness constant assumption is usually used, so: I(x+dx,y+dy,t+dt)=I(x,y,t);
[0102] Expand the left side using Taylor's formula, retaining the first-order term, and obtain
[0103]
[0104] thereby
[0105]
[0106] At the same time, dividing by dt gives:
[0107]
[0108] in is the speed of the pixel in the X-axis direction, are the speeds on the Y-axis, denoted as u and v respectively. In LK optical flow, considering that the pixels in a small window have the same motion, an overdetermined linear equation about u and v can be obtained. By solving this equation, the position of a block of pixels in several images can be estimated. In SLAM, LK optical flow is often used to track the movement of feature points. This method uses the iterative Lucas-Kanade optical flow algorithm with pyramid to track the corresponding feature points of the feature points on the current frame in the previous frame. After obtaining the correspondence between the feature points, this method calculates the optical flow amplitude mag of all feature points in the current frame relative to the previous frame i :
[0109] mag i =(x1-x2,y1-y2)×scale i , i∈L;
[0110] Where L = L 动态区域 +L 非动态区域 , represents the set of all feature points extracted, scale i Indicates the scaling scale of the pyramid layer where feature point i is located. ORB-SLAM3 uses 8 layers of pyramids when extracting feature points, and the corresponding scaling scales are 1, 1.2, 1.44, 1.728, 2.074, 2.488, 2.986, and 3.583.
[0111] This method also calculates the mean of the optical flow amplitude in the non-dynamic area And the standard deviation magstd:
[0112]
[0113]
[0114] get After magstd, this method uses the following formula to filter all feature points in the dynamic area one by one:
[0115]
[0116]
[0117] All optical flows outside the 1.2 standard deviation boundary within the dynamic region are removed, as they are likely to affect the accuracy of SLAM. In addition, if the standard deviation is below the 1.8 threshold, the requirement to be an outlier is reduced to a simple difference from the mean. Figure 5 As shown, Figure 5 (a) in the figure represents all optical flow vectors, (b) represents abnormal optical flow vectors, and (c) represents the remaining optical flow vectors after removing the abnormal optical flow vectors.
[0118] In order to verify the effectiveness of the improved RT-DETR algorithm, the improved network model was retrained on the coco2017 dataset. The coco2017 dataset contains 80 categories. The training set uses train2017 and the test set uses val2017.
[0119] After training, the performance test results of the improved PPLCNet-RT-DETR and RT-DETR series are shown in Table 1. As shown in Table 1, the network parameters of PPLCNet-RT-DETR are reduced by 50% compared with RT-DETR-R18, and the number of frames is increased by 37%; in terms of accuracy, since the advantages of the PP-LCNet network are small model and fast speed, but the detection performance is insufficient, the improved target detection network is 4.3% lower than RT-DETR-R18, sacrificing some accuracy and improving the running speed of the algorithm.
[0120] Table 1 Target detection network performance test
[0121]
[0122]
[0123] The SLAM algorithm performance verification dataset uses the dynamic-object category in the TUMRGB dataset, which includes scenes of two people sitting and talking (sitting) and two people walking left and right (walking). There are four types of camera motions in each scene: (1) halfsphere: the camera moves along a hemispherical trajectory with a radius of 1 meter, (2) xyz: the camera moves along the x, y, and z axes, (3) rpy: the camera rotates around the roll, pitch, and yaw axes, and (4) static: the camera remains basically stationary.
[0124] This method first divides the dynamic object area based on the dynamic object classification label and the corresponding detection frame of the target detection result of the PPLCNet-RT-DETR algorithm, and then uses the epipolar constraint and optical flow method to filter the feature points extracted from the dynamic area, remove the feature points on the moving dynamic objects, and retain the feature points of the static dynamic objects and the background part of the target detection frame to participate in the pose optimization. Figure 6 As shown in the figure, (a) represents the feature points extracted by the original ORB-SLAM3 algorithm, and (b) represents the feature points extracted by the algorithm of this method. The feature points in the dynamic area are screened using the epipolar constraint and optical flow (the feature points left by the screening are yellow dots in the figure). It can be seen that the improved method of this method effectively retains the feature points on the stationary person and the stationary objects around the human body, while eliminating the feature points on the moving human body, so that as many feature points as possible are involved in the camera pose optimization to improve the SLAM positioning accuracy and stability.
[0125] Since the proposed method is improved from ORB-SLAM3, the performance of the proposed method is compared with that of ORB-SLAM3 on the walking_halfsphere, walking_xyz, walking_static, and sitting_static sequences in the TUMRGB dataset. Tables 2 and 3 show the experimental results of ORB-SLAM3 and the proposed method in terms of absolute trajectory error (ATE) and relative trajectory error (RPE). The relative improvement rate α of the algorithm performance is calculated using the following formula:
[0126]
[0127] Table 2 Comparison of absolute trajectory error (ATE) results (unit: m)
[0128]
[0129] Table 3 Comparison of relative trajectory error (RPE) results (unit: m)
[0130]
[0131] Figure 7 Comparison of absolute trajectory error and relative trajectory error, where (a) is the walking_halfsphere sequence, (b) is the walking_xyz sequence, (c) is the sitting_static sequence, and (d) is the sitting_halfsphere sequence;
[0132] Where m is the running result of the original ORB-SLAM3, and n is the running result of the method of this application.
[0133] From the data analysis in the table, it can be seen that the method of this application can achieve an average improvement of more than 90% in the RMSE values of absolute trajectory error and relative trajectory error compared to the original ORB-SLAM3 in high-dynamic walking sequences. In low-dynamic environments, the method of this application can retain the feature points of the static part of the dynamic object while removing the feature points of the moving part. Therefore, the improved algorithm can also achieve an average improvement of more than 20% in the RMSE value of absolute trajectory error compared to ORB-SLAM3 in low-dynamic sitting scenes. In order to more intuitively reflect the improvement of the algorithm, Figure 7 The absolute trajectory error of ape and the relative trajectory error of rpe of the proposed method and ORB-SLAM3 are plotted. Figure 7 As shown, the method of the present application shows better performance than the original ORB-SLAM3 in both low-dynamic scenes and high-dynamic scenes.
[0134] Further comparison with the performance of other current classic SLAM methods for dynamic environments, DynaSLAM, DS-SLAM and RDS-SLAM on the TUM dataset, as shown in Table 4, shows that the method of this application can perform better than most SLAM methods for processing dynamic scenes in both high-dynamic and low-dynamic scenes.
[0135] Table 4 Comparison of absolute trajectory error (ATE) results of some advanced SLAM methods (unit: m)
[0136]
[0137] Table 5 Single frame image tracking time comparison (GPU acceleration)
[0138]
[0139] Table 6 Single frame image tracking time comparison (without GPU acceleration)
[0140]
[0141] It can be seen that when using GPU accelerated reasoning, the tracking speed of the present application method is three times faster than DynaSLAM. This is because DynaSLAM uses the MaskR-CNN instance segmentation algorithm, which is more time-consuming in image processing than the RT-DETR target detection algorithm used in the present application method. Due to the characteristics of PP-LCNet based on CPU acceleration, when GPU acceleration is not used, the tracking speed of the present application method can be improved by four times compared to DynaSLAM. Compared with the RT-DETR-R18 network model, the improved PPLCNet-RT-DETR network model used in the present application method can be further improved in real-time performance after being integrated into the SLAM algorithm regardless of whether GPU acceleration is used.
[0142] The method of this application first uses the improved RT-DETR algorithm to divide the dynamic area, and then uses the geometry and optical flow methods to screen the feature points, retaining as many feature points as possible to participate in the pose optimization. Compared with the original ORB-SLAM3 algorithm, the method of this application improves the performance of the algorithm in dynamic scenes. At the same time, compared with the SLAM method that directly removes the feature points on all dynamic objects, the method of this application takes into account the situation of static dynamic objects, further improving the stability of SLAM.
[0143] In one embodiment, the dynamic SLAM method based on feature point screening is characterized by further comprising:
[0144] Get the first historical map within the most recent preset time;
[0145] Determining a transition probability based on the first historical map;
[0146] When the transition probability is greater than or equal to a preset probability threshold, obtaining a second historical map within a recent preset second time;
[0147] Based on the second historical mapping and the preset standard mapping library, determining the transition type and the probability table of dynamic object appearance after the transition;
[0148] Referring to the transition type and the probability table of dynamic object appearance after the transition, the dynamic area in the future frame image is identified based on the improved RT-DETR algorithm.
[0149] The first time can be, for example, 20 seconds; the first historical map is the environment map constructed by the SLAM robot in the most recent SLAM map; the transition probability is the probability that the SLAM robot is about to enter other SLAM environments from the current SLAM environment, for example, if the SLAM robot is a sweeping robot, the transition probability is the probability that it is about to enter the bedroom from the living room; the probability threshold can be, for example, 75%; the second time can be, for example, 5 minutes; the second historical map is the environment map constructed by the SLAM robot in the most recent second time; the transition type is the type of SLAM environment that the SLAM robot is about to enter, for example, bedroom; the probability table of dynamic object appearance after transition contains dynamic objects and their probability of appearance in scenes with transition types, for example, shaking fan, probability 35%, etc.; the future frame image is the next frame image obtained and waiting for feature point screening; Referring to the transition type and the probability table of dynamic objects appearing after the transition, based on the improved RT-DETR algorithm, when identifying the dynamic area in the future frame image, the dynamic object with a greater probability of appearing in the probability table of dynamic objects appearing after the transition is given priority in identifying whether it appears in the future frame image based on the RT-DETR algorithm, thereby improving the efficiency of dynamic area recognition; generally, when a SLAM robot enters a new SLAM environment, it may encounter multiple dynamic objects, and it is necessary to identify the dynamic area of each dynamic object in turn based on the RT-DETR algorithm. If the recognition is not performed in a targeted manner, the recognition workload is huge and the recognition efficiency is low. However, after the SLAM robot enters a new SLAM environment, it generally moves continuously. For example, a sweeping robot enters the bedroom from the living room, and still needs to continue cleaning after entering the bedroom. If the dynamic areas of multiple newly appearing dynamic objects cannot be quickly identified, it may affect the working state of the SLAM robot. The present invention can solve the problem in this special situation: first, determine whether the SLAM robot is about to transition. When it is determined that the transition is about to take place, determine the transition type and the probability table of dynamic objects appearing after the transition. With reference to the transition type and the probability table of dynamic objects appearing after the transition, based on the improved RT-DETR algorithm, identify the dynamic area in the future frame image, and perform targeted identification, which greatly improves the recognition efficiency and the applicability of the system.
[0150] In one embodiment, determining the transition probability based on the first historical map building includes:
[0151] Determining whether the first historical mapping meets a first image condition;
[0152] When the condition is met, the transition probability is calculated based on a preset first threshold; the first threshold is greater than or equal to the probability threshold;
[0153] When it does not meet the requirements, determining whether the first historical mapping meets the second image condition;
[0154] When there is a match, the transition probability is the sum of the preset condition representation degrees corresponding to the second image condition that the first historical mapping meets;
[0155] When neither condition is met, the transition probability is calculated based on a preset second threshold; the second threshold is smaller than the probability threshold;
[0156] Wherein, the first image condition includes:
[0157] The mapping completeness of the first historical mapping is greater than or equal to a preset first completeness threshold;
[0158] A partial image representing a transition indicator object in a preset transition indicator object library appears in the first historical mapping;
[0159] The matching degree between the local direction sequence after the local image is established in the mapping direction sequence of the first historical mapping and the preset standard direction sequence is greater than or equal to a preset matching degree threshold;
[0160] Wherein, the second image condition includes:
[0161] The mapping completeness of the first historical mapping is less than the first completeness threshold and greater than or equal to a preset second completeness threshold;
[0162] The proportion of return routes in the mapping movement trajectory of the first historical mapping is greater than or equal to a preset proportion threshold.
[0163] When the first historical map meets the first image condition, it means that the probability that the SLAM robot is about to enter other SLAM environments from the current SLAM environment is very high, and the transition probability is calculated by a first threshold that is greater than or equal to the probability threshold; the first threshold can be, for example: 90%; when the second image condition is met, the preset conditional characterization degrees corresponding to the second image condition that the first historical map meets are summed as the transition probability; the conditional characterization degree represents the probability that the SLAM robot is about to enter other SLAM environments from the current SLAM environment when the second image condition is met, for example: the conditional characterization degree is 20%, etc.; when any second image condition is not met, it means that the SLAM robot is about to enter other SLAM environments from the current SLAM environment. The probability is very small, and the transition probability is calculated by a second threshold value that is less than the probability threshold value; the second threshold value can be, for example, 20%; the first completeness threshold value can be, for example, 85%; under the first time limit, the first historical map is generated by the SLAM robot's SLAM map construction of its surrounding environment in the most recent period of time. Therefore, if the map construction completeness is greater than or equal to the preset first completeness threshold value, it means that the SLAM robot is about to move its position and the transition probability is relatively high; the map construction completeness refers to the ratio of the area where the SLAM map construction has been completed to the total area where the SLAM map construction needs to be performed (generally, the SLAM robot will automatically calculate it when working, for example: the sweeping robot will push the environmental scanning progress to the user in real time); the transition indicator object is Objects for the SLAM robot to complete the transition, such as: doors, etc.; when a local image representing the transition indication object appears in the first historical map, it means that the SLAM robot is about to move and the probability of transition is high; the mapping direction sequence is the SLAM robot scanning and building the first historical map, scanning and building direction changes, such as: starting from the southwest direction, scanning to the south direction, and building the first historical map; the local direction sequence refers to the SLAM robot scanning and building the first historical map. After the scanning and construction of the local image is completed, the scanning and construction direction changes continue; the standard direction sequence is the scanning and construction direction change that represents the SLAM robot is about to approach the transition prompt object, such as: continuously scanning and building the SLAM of the transition prompt object AM image, the direction of scanning and construction changes from southwest to south (indicating that the SLAM robot is approaching the transition prompt object); the matching degree threshold can be, for example, 60%; when the matching degree is greater than or equal to the matching degree threshold, it means that the SLAM robot is approaching the transition prompt object, indicating that the probability of the SLAM robot transition is relatively high; the second integrity threshold is less than the first integrity threshold, for example, it can be 45%; the mapping movement trajectory of the first historical mapping is the trajectory of the SLAM robot scanning and constructing the first historical mapping, and the return route on it is a repeated route of the SLAM robot going back and forth and turning back. The larger the proportion, the greater the probability that the SLAM robot is about to transfer; the proportion threshold can be, for example, 30%;
[0164] According to different situations, the embodiment of the present invention adaptively determines the transition probability based on the first historical map, thereby improving the applicability of the system and further improving the accuracy and efficiency of determining the transition probability.
[0165] In one embodiment, the determining of the transition type and the probability table of dynamic object appearance after the transition based on the second historical mapping and the preset standard mapping library includes:
[0166] Perform feature extraction on the second historical map to obtain multiple image features; the image features include: objects that appear successively in the second historical map and the order in which they appear, etc.;
[0167] Based on the multiple image features, an image search condition is generated; the image search condition makes the standard image similar to the second historical image, for example, if the image feature is that a refrigerator and a television appear in sequence, then the image search condition is that a refrigerator and a television appear in sequence in the standard image;
[0168] Searching for multiple standard images that meet the image search condition from the standard image library; the standard image library can be constructed in advance by a technician according to actual needs;
[0169] Classifying the plurality of standard images to obtain a plurality of sets of dynamic objects appearing in the image scene types; the image scene type is the scene type represented by the standard image, and the set of dynamic objects appearing is the dynamic objects appearing in the standard image, for example, the scene type is a living room, and the dynamic object is a domestic cat, etc.;
[0170] Using the image scene type as a transition type;
[0171] Counting the appearance frequencies of different dynamic objects in the dynamic object set;
[0172] Based on the occurrence frequencies of different dynamic objects, the occurrence probability table of the dynamic objects after the transition is generated. The occurrence frequencies can be directly used as the occurrence probabilities of the dynamic objects for tabulation.
[0173] Based on the second historical mapping and the preset standard mapping library, when determining the transition type and the probability table of dynamic object appearance after the transition, the standard mapping library is introduced to improve the accuracy and efficiency of determining the transition type and the probability table of dynamic object appearance after the transition.
[0174] The embodiment of the present invention provides a dynamic SLAM system based on feature point screening, comprising:
[0175] An acquisition module is used to acquire the current frame image;
[0176] An extraction module, used for extracting ORB feature points from the current frame image to obtain ORB feature points;
[0177] An identification module, used for identifying a dynamic area in the current frame image based on an improved RT-DETR algorithm;
[0178] A verification module, used for performing epipolar constraint verification and optical flow verification on the ORB feature points falling into the dynamic area;
[0179] The subsequent processing module is used to involve the ORB feature points that have passed the epipolar constraint check and the optical flow check and the ORB feature points that do not fall into the dynamic area in camera tracking, and then perform local mapping, loop detection and map merging.
[0180] The steps of improving the RT-DETR algorithm include:
[0181] Use the PP-LCNet network to replace the backbone network in the original RT-DETR algorithm;
[0182] The improved RT-DETR algorithm is used to identify the dynamic area in the current frame image.
[0183] The step of checking the epipolar constraint comprises:
[0184] Based on the epipolar constraint method, epipolar constraint verification is performed on the ORB feature points falling into the dynamic area.
[0185] The steps of optical flow verification include:
[0186] Based on the optical flow method, an optical flow check is performed on the ORB feature points falling into the dynamic area.
[0187] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A dynamic SLAM method based on feature point screening, characterized in that: include: Get the current frame image; Extracting ORB feature points from the current frame image to obtain ORB feature points; Based on the improved RT-DETR algorithm, identifying the dynamic area in the current frame image; Performing epipolar constraint verification and optical flow verification on the ORB feature points falling into the dynamic area respectively; The ORB feature points that have passed the epipolar constraint check and the optical flow check and the ORB feature points that do not fall into the dynamic area are involved in camera tracking, and then local mapping, loop closure detection and map merging are performed; The improved RT-DETR algorithm refers to: The PP-LCNet network is used to replace the backbone network in the original RT-DETR algorithm.
2. A dynamic SLAM method based on feature point screening as claimed in claim 1, characterized in that, The step of checking the epipolar constraint comprises: Based on the epipolar constraint method, epipolar constraint verification is performed on the ORB feature points falling into the dynamic area.
3. A dynamic SLAM method based on feature point screening as claimed in claim 1, characterized in that, The steps of optical flow verification include: Based on the optical flow method, an optical flow check is performed on the ORB feature points falling into the dynamic area.
4. A dynamic SLAM method based on feature point screening as claimed in claim 1, characterized in that, Also includes: Get the first historical map within the most recent preset time; Determining a transition probability based on the first historical map; When the transition probability is greater than or equal to a preset probability threshold, obtaining a second historical map within a recent preset second time; Based on the second historical mapping and the preset standard mapping library, determining the transition type and the probability table of dynamic object appearance after the transition; Referring to the transition type and the probability table of dynamic object appearance after the transition, the dynamic area in the future frame image is identified based on the improved RT-DETR algorithm.
5. A dynamic SLAM method based on feature point screening as claimed in claim 4, characterized in that: The determining the transition probability based on the first historical map building includes: Determining whether the first historical mapping meets a first image condition; When the conditions are met, the transition probability is calculated based on a preset first threshold; the first threshold is greater than or equal to the probability threshold; When it does not meet the requirements, determining whether the first historical mapping meets the second image condition; When there is a match, the transition probability is the sum of the preset condition representation degrees corresponding to the second image condition that the first historical mapping meets; When neither condition is met, the transition probability is calculated based on a preset second threshold; the second threshold is smaller than the probability threshold; Wherein, the first image condition includes: The mapping completeness of the first historical mapping is greater than or equal to a preset first completeness threshold; A partial image representing a transition indicator object in a preset transition indicator object library appears in the first historical map; The matching degree between the local direction sequence after the local image is established in the mapping direction sequence of the first historical mapping and the preset standard direction sequence is greater than or equal to a preset matching degree threshold; Wherein, the second image condition includes: The mapping completeness of the first historical mapping is less than the first completeness threshold and greater than or equal to a preset second completeness threshold; The proportion of return routes in the mapping movement trajectory of the first historical mapping is greater than or equal to a preset proportion threshold.
6. A dynamic SLAM method based on feature point screening as claimed in claim 4, characterized in that: The determining of the transition type and the probability table of dynamic object appearance after the transition based on the second historical mapping and the preset standard mapping library includes: Performing feature extraction on the second historical map to obtain multiple image features; generating an image search condition based on the plurality of image features; Searching for a plurality of standard images that meet the image search condition from the standard image library; Classifying the plurality of standard images to obtain a set of dynamic objects appearing in a plurality of image scene types; Using the image scene type as a transition type; Counting the appearance frequencies of different dynamic objects in the dynamic object set; Based on the occurrence frequencies of different dynamic objects, a probability table of occurrence of dynamic objects after the transition is generated.
7. A dynamic SLAM system based on feature point screening, characterized in that: include: An acquisition module is used to acquire the current frame image; An extraction module, used for extracting ORB feature points from the current frame image to obtain ORB feature points; An identification module, used for identifying a dynamic area in the current frame image based on an improved RT-DETR algorithm; A verification module, used for performing epipolar constraint verification and optical flow verification on the ORB feature points falling into the dynamic area; A subsequent processing module, used for participating the ORB feature points that have passed the epipolar constraint check and the optical flow check and the ORB feature points that do not fall into the dynamic area in camera tracking, and then performing local mapping, loop closure detection and map merging; The improved RT-DETR algorithm refers to using the PP-LCNet network to replace the backbone network in the original RT-DETR algorithm.
8. A dynamic SLAM system based on feature point screening as claimed in claim 7, characterized in that: The epipolar constraint check includes: Based on the epipolar constraint method, epipolar constraint verification is performed on the ORB feature points falling into the dynamic area.
Citation Information
Patent Citations
Heart valve operation assisting method and system based on combination of neural network and augmented reality
CN117116456A
Rapid detection algorithm for mine aerial passenger device
CN117197582A
Supermarket intelligent service robot
CN117301088A
Target detection method based on transformer architecture
CN117315223A
Method for merging multiple images and post-processing of panorama
US20210295467A1
