Positioning and mapping method and system based on semantic and scene flow geometric information

Abstract

The present invention provides a positioning and mapping method and system based on semantic and scene flow geometric information. The present invention uses the semantic information of the environment to assist in the detection of dynamic targets, tightly coupling the acquired semantic information with multi-view geometry to achieve rapid and accurate recognition of dynamic targets. Combined with depth- and density-based clustering of scene flow geometric information, it effectively reduces the limitations of commonly used geometric constraints in dynamic environments. After removing prior dynamic objects, it can quickly and accurately calculate geometric dynamic areas. In complex dynamic scenes, mapping is less time-consuming, has higher accuracy and robustness, and can create intuitive semantic metric maps. The present invention not only significantly improves the positioning accuracy of the system, but also has excellent computational efficiency, is applicable to complex environments, and significantly improves the real-time performance, accuracy, and robustness in complex dynamic environments.

Description

Technical Field

[0001] The present invention belongs to the field of positioning and mapping, and specifically relates to a positioning and mapping method and system based on semantic and scene flow geometric information. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Simultaneous Localization and Mapping (SLAM) has long been a key research area in mobile robot control. SLAM aims to enable autonomous navigation by simultaneously sensing the environment and acquiring the robot's own position. Visual SLAM, a camera-based SLAM method, has garnered widespread attention due to its low cost, readily available equipment, high-resolution image capture, and rich environmental information processing capabilities.

[0004] The development of visual SLAM has gone through several stages, with the emergence of many classic visual SLAM frameworks. Among these classic visual SLAM frameworks, ORB-SLAM2 has become a highly favored choice due to its excellent performance and robustness. However, in dynamic environments, ORB-SLAM2 struggles to handle dynamic objects, often mistakenly identifying them as part of the environment during mapping. Therefore, research on dynamic SLAM has become a current hot topic and a challenge.

[0005] Most advanced dynamic SLAM methods rely on the assumption of a static world. However, the real world is full of dynamic objects, making it difficult to achieve good segmentation results in highly dynamic environments. Misidentification of dynamic objects can lead to mismatched feature points, reducing the overall positioning and mapping accuracy of the system. Therefore, accurate identification of dynamic feature points is a prerequisite for achieving precise positioning in dynamic environments.

[0006] In visual SLAM systems, dynamic feature recognition is primarily accomplished in two ways: distinguishing between dynamic and static points through geometric consistency, and combining semantic information with geometric information. The core idea behind the geometric consistency approach is to identify dynamic feature points by analyzing their geometric relationships. These methods typically require minimal computation and do not rely on pre-training data, but are prone to false detection when dealing with complex scenarios. The semantic-integrated geometric approach, by incorporating the results of target detection or semantic segmentation, enhances the system's understanding of the environment through semantic information, thereby improving the system's robustness and accuracy. Semantic annotation can provide prior semantic information for the selection of dynamic feature points, facilitating their accurate identification in complex scenarios.

[0007] However, while object detection systems offer better real-time performance than semantic segmentation methods, they cannot accurately identify the masks of dynamic objects, leading to inaccurate dynamic point recognition. Even with reduced processing frames, SLAM methods using pixel-level semantic segmentation still require significant computation, making real-time operation difficult. Summary of the Invention

[0008] In order to solve the above problems, the present invention proposes a positioning and mapping method and system based on semantics and scene flow geometric information. The present invention clusters the scene flow geometric information, reduces the influence of incorrect feature matching and motion approximation on the judgment of the motion attributes of feature points; and integrates semantics and scene flow geometric information to suppress dynamic features, which not only significantly improves the positioning accuracy of the system, but also has excellent computational efficiency, is suitable for complex environments, significantly improves the real-time performance, accuracy and robustness in complex dynamic environments, and is conducive to further research on mobile robots.

[0009] According to some embodiments, the present invention adopts the following technical solutions:

[0010] A positioning and mapping method based on semantic and scene flow geometric information includes the following steps:

[0011] Get RGB image and depth image;

[0012] Performing target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects;

[0013] Matching feature points of two consecutive frames in the RGB image, and using the dynamic semantic information of the previous frame, matching the points corresponding to the current frame as prior dynamic points and removing them;

[0014] Calculate the basic matrix between the two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. According to the essential matrix, obtain the camera extrinsic parameters, determine the camera pose, and calculate the scene flow of the feature points in combination with the depth information, and perform scene flow clustering;

[0015] Combine the dynamic points obtained by the current frame target detection and the points in the geometric dynamic area obtained by scene flow clustering to select the final dynamic feature points and obtain the static feature points;

[0016] Select a new key frame and perform camera pose prediction based on the static feature points;

[0017] Based on the depth information, the predicted camera pose and the feature points of the key frames, the point cloud information is determined, the semantic information is converted into three-dimensional semantic information, and the similarity matching between the point cloud information and the three-dimensional semantic information is performed to obtain the final semantic map.

[0018] As an optional implementation, the process of matching the feature points of two consecutive frames in the RGB image is to match the feature points of the current frame with the previous frame through the LK optical flow method, and then combine the depth information to obtain the points of the previous frame that are within the prior dynamic frame and whose depth information is similar to the depth information of the midpoint of the prior dynamic frame, and match these points to the current frame as the prior dynamic points of the current frame for elimination.

[0019] As an optional implementation, the process of calculating the basic matrix between two frames after removing the prior dynamic points and calculating the essential matrix based on the camera intrinsic parameters includes removing the dynamic points of the previous frame and the points corresponding to the matched current frame based on the dynamic semantic information of the previous frame to obtain the basic matrix, and calculating the essential matrix based on the basic matrix:

[0020] E=K T ·F·K

[0021] Among them, K is the camera intrinsic parameter matrix and F is the basic matrix.

[0022] As an optional implementation, the process of obtaining camera extrinsic parameters based on the essential matrix includes: performing singular value decomposition on the essential matrix E to obtain 4 sets of camera extrinsic parameters R, t, comparing the 4 sets of R, t to restore the legal number of spatial points, and selecting R, t with the largest legal number as the camera extrinsic parameters.

[0023] As an optional implementation method, the process of scene flow clustering includes: first performing deep clustering, calculating the distance between points based on the deep clustering, establishing connecting lines based on the distances, comparing the similarities of the scene flows at both ends of the connecting lines, and further clustering them; and grouping points with similar distances and consistent movement characteristics to determine their motion status.

[0024] As a further step, the process of performing depth clustering includes: obtaining the dynamic points of the current frame through matching based on the prior dynamic points detected in the previous frame, and removing them to obtain a point set, traversing all the points in the point set, and judging whether the depth of the current feature point is less than the average depth of the existing different depth categories through the depth threshold to decide whether a new cluster needs to be established.

[0025] As a further step, a connecting line is established based on the distance, and the similarity of the scene flows at both ends of the connecting line is compared. The process of further clustering includes calculating the similarity of the scene flows between the feature points based on the distance between the feature points, and establishing a connecting line based on the similarity of the scene flows, and the distance between the feature points uses the Euclidean distance;

[0026] Based on connecting line clustering, the points at both ends of the connecting line are classified into one category, and highly correlated feature point clusters are detected according to the distance of feature points and scene flow geometric information.

[0027] As an optional implementation method, the process of scene flow clustering to obtain geometric dynamic areas includes: multiplying the scene flow by the inverse of the depth; after eliminating a portion of dynamic points based on the prior dynamic information of the previous frame, calculating the average scene flow of all points, using the average scene flow to represent the scene flow of the static features of the current frame, and judging whether the category is a dynamic category based on the similarity between the average scene flow of each category and the scene flow of the static features of the current frame; if there is a dynamic scene flow category, it is considered that there are points on objects with passive motion or objects with active motion that have not been completely eliminated, and the convex hull algorithm is used to calculate the area containing all points of the category, and it is considered that the points in the area are all dynamic feature points.

[0028] A localization and mapping system based on semantic and scene flow geometry information, including:

[0029] An image acquisition module is configured to acquire an RGB image and a depth image;

[0030] An object detection module is configured to perform object detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects; and obtain a specific semantic mask in combination with the depth image;

[0031] An optical flow matching module is configured to match feature points of two consecutive frames in the RGB image, and based on the dynamic semantic information of the previous frame, the points corresponding to the current frame obtained by matching are used as prior dynamic points and are eliminated;

[0032] The scene flow clustering module is configured to calculate the basic matrix between two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. Based on the essential matrix, the camera extrinsic parameters are obtained to determine the camera pose, and the scene flow of the feature points is calculated in combination with the depth information to perform scene flow clustering.

[0033] The dynamic selection module is configured to select the final dynamic feature points by combining the dynamic points of the current frame and the points in the geometric dynamic area obtained by scene flow clustering to obtain static feature points;

[0034] A pose prediction module is configured to select a new key frame and perform camera pose prediction based on the static feature points;

[0035] The semantic mapping module is configured to determine the point cloud information based on the depth information, the predicted camera pose, and the feature points of the key frames, convert the semantic information into three-dimensional semantic information, perform similarity matching between the point cloud information and the three-dimensional semantic information, and obtain the final semantic map.

[0036] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0037] The electronic device can be mounted on a robot.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention is based on a visual SLAM system and adds a target detection thread and a semantic mapping thread. The multiple threads run in parallel, effectively improving the real-time performance of the SLAM system. The target detection thread provides a priori semantic information through the target detection algorithm, which can provide target detection results in real time. The semantic mapping thread uses the semantic information provided by the target detection thread to remove point clouds located in dynamically marked areas, generate a three-dimensional semantic object database, and ultimately obtain a static semantic scene reconstruction.

[0040] Before scene flow clustering, the present invention uses the prior dynamic information of the previous frame to eliminate the prior dynamic points, and combines the prior dynamic points of the current frame and the points in the dynamic area obtained by scene flow clustering to perform final dynamic feature point elimination. This not only reduces the impact of the static world assumption on high dynamic situations, but also enables the system to run in real time.

[0041] The present invention uses the semantic information of the environment to assist in the detection of dynamic targets, tightly coupling the acquired semantic information with multi-view geometry to achieve rapid and accurate recognition of dynamic targets. Combined with depth- and density-based clustering of scene flow geometric information, it effectively reduces the limitations of commonly used geometric constraints in dynamic environments. After removing prior dynamic objects, it can quickly and accurately calculate the geometric dynamic area. In complex dynamic scenes, mapping takes less time, has higher accuracy and robustness, and can create intuitive semantic metric maps.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0044] Figure 1 This is a flow chart of a positioning and mapping method based on semantics and scene flow geometric information according to an embodiment;

[0045] Figure 2 This is a schematic diagram of a positioning and mapping method based on semantics and scene flow geometric information according to an embodiment;

[0046] Figure 3 is a schematic diagram of a scene flow clustering process according to an embodiment;

[0047] Figure 4 is a schematic diagram of a process for calculating a geometric dynamic area based on scene flow information according to an embodiment;

[0048] Figure 5 This is a schematic diagram of semantic mapping results using different methods according to an embodiment. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0052] In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0053] Example 1

[0054] A localization and mapping method based on semantic and scene flow geometric information, such as Figure 1 As shown, the following steps are included:

[0055] Get RGB image and depth image;

[0056] Performing target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects;

[0057] Matching feature points of two consecutive frames in the RGB image, and using the dynamic semantic information of the previous frame, matching the points corresponding to the current frame as prior dynamic points and removing them;

[0058] Calculate the basic matrix between the two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. According to the essential matrix, obtain the camera extrinsic parameters, determine the camera pose, and calculate the scene flow of the feature points in combination with the depth information, and perform scene flow clustering;

[0059] Combine the dynamic points obtained by the current frame target detection and the points in the geometric dynamic area obtained by scene flow clustering to select the final dynamic feature points and obtain the static feature points;

[0060] Select a new key frame and perform camera pose prediction based on the static feature points;

[0061] Based on the depth information, the predicted camera pose and the feature points of the key frames, the point cloud information is determined, the semantic information is converted into three-dimensional semantic information, and the similarity matching between the point cloud information and the three-dimensional semantic information is performed to obtain the final semantic map.

[0062] The process of the above method is divided into multiple threads, and at least some of the threads can be operated in parallel.

[0063] First, for the convenience of description, a positioning and mapping method based on semantics and scene flow geometric information in this embodiment is referred to as SSF-SLAM.

[0064] The SSF-SLAM provided in this embodiment is based on the ORB-SLAM2 framework. ORB-SLAM2 contains three main threads: tracking thread, local mapping thread, and loop detection thread. In order to fully utilize the advantages of ORB-SLAM2, this embodiment uses ORB-SLAM2 as the framework. The principle of SSF-SLAM is as follows: Figure 2 As shown in the figure, the blue box is the original structure of ORB-SLAM2, and the red box is the added or modified content.

[0065] Most advanced dynamic SLAM methods cannot guarantee the real-time operation of the system while using semantic information to improve robustness. Therefore, in SSF-SLAM, this embodiment adds two threads: a target detection thread and a semantic mapping thread.

[0066] Multi-threaded parallel operation effectively improves the real-time performance of the SLAM system. The object detection thread provides prior semantic information through the object detection algorithm. Since the object detection algorithm does not require the calculation of detailed semantic masks, it can provide object detection results in real time. The semantic mapping thread uses this semantic information to remove point clouds located in dynamically marked areas, generating a 3D semantic object database and ultimately reconstructing a static semantic scene.

[0067] Specifically, such as Figure 2 As shown in Figure 2, when SSF-SLAM is executed, the image frames captured from the RGB-D camera are first fed to the tracking thread and the target detection thread.

[0068] While the tracking thread extracts ORB features, the object detection thread performs object detection and labels dynamic objects a priori. Object detection in images captured by the RGB-D camera not only detects dynamic semantic information but also semantic information about other objects. Here, points on the target object, a person, are used as a priori dynamic points, specifically those on the person in the previous frame.

[0069] After the tracking thread obtains the dynamic semantic information of the previous frame, it uses the LK optical flow method to match the feature points of the current frame with the previous frame. Then, combined with the depth information, it obtains points in the previous frame that are within the prior dynamic frame and have a depth similar to the midpoints in the prior dynamic frame. These points are matched to the current frame and removed as the prior dynamic points of the current frame. This method can roughly obtain static points in the current frame that match each other, and can minimize the removal of points on non-dynamic objects within the dynamic frame.

[0070] After removing the dynamic points of the previous frame matched by the current frame, this embodiment uses the RANSAC-based seven-point method to calculate the fundamental matrix between the two frames and calculates the essential matrix based on the camera intrinsic parameters. This can reduce the adverse effects caused by incorrect data association in dynamic areas.

[0071] Afterwards, the obtained essential matrix is ​​decomposed by singular value SVD and triangulated to obtain the camera extrinsic parameters (R, t), and the scene flow of the point is calculated in combination with the depth information, and scene flow clustering is performed.

[0072] Compared with feature extraction, feature matching, essential matrix calculation, and scene flow calculation, the target detection process is more time-consuming. If the prior dynamic information of the current frame is directly used to calculate the essential matrix and cluster the scene flow, it is necessary to wait for the completion of target detection. Although this embodiment has adopted a target detection method with less computational effort, in order to minimize the time spent waiting for target detection, the prior dynamic information of the previous frame is used to eliminate the prior dynamic points before scene flow clustering.

[0073] Finally, the final dynamic feature points are removed by combining the prior dynamic points of the current frame and the points in the dynamic area obtained by scene flow clustering. This not only reduces the impact of the static world assumption on high dynamic situations, but also enables the system to run in real time.

[0074] The following is a detailed introduction to the content of each thread.

[0075] First, the object detection process in this embodiment utilizes NCNN. NCNN is a lightweight neural network computing framework optimized specifically for mobile and embedded devices. Since mobile robots generally use ARM architecture processors with high performance per watt, NCNN is implemented in pure C++, without any third-party libraries, supporting cross-platform operation, with minimal accuracy loss and memory usage, and fast computation speed, making it easily integrated into SLAM systems. Therefore, it was chosen as the basic framework for object detection.

[0076] NCNN supports most commonly used CNN networks, such as VGG, AlexNet, GoogleNet Inception, SqueezeNet, MobileNetV1 / V2 / V3, and MTCNN in the detection field. To achieve real-time operation of the system, this example uses MobileNetV3 as a direct replacement for the backbone feature extraction in SSDLite. This avoids the slow operation problem of traditional SLAM systems due to the complexity of semantic segmentation. Finally, the network is trained using the PASCAL VOC 2007 dataset.

[0077] Given the rich capabilities of NCNN, in practical applications, other deep learning methods can be flexibly used according to the performance of the hardware to strike a balance between accuracy and speed.

[0078] This embodiment uses geometric information obtained through scene flow clustering to determine whether a feature point is dynamic. In a static scene, points move in a similar direction within the camera coordinate system, while dynamic points do not. Therefore, the scene flow information of the feature point can be used to determine whether the point is dynamic. This involves two issues: scene flow calculation and scene flow clustering.

[0079] The scene flow information of the feature points is calculated by multi-view geometry. For binocular and RGBD cameras, the stereo information of the feature points can be obtained, including the right eye feature point information and the depth information of the feature points. In this way, the three-dimensional coordinates in the camera coordinate system can be obtained by parallax ranging. The binocular camera uses the feature correspondence between the left and right eye images to obtain the disparity d, and the RGBD camera can directly obtain the depth value of the feature point (i.e., disparity d). At time t, the pixel coordinates of the i-th point are expressed as (u i t ,v i t ), where u represents the horizontal coordinate in the pixel coordinate system, v represents the vertical coordinate in the pixel coordinate system, and the coordinate P in the current camera coordinate system i t =(X i t ,Y i t ,Zi t ) T It can be expressed as:

[0080]

[0081] Where b is the baseline distance of the binocular camera, (c x ,c y ) is the coordinate origin, parameter c x ,c y ,f are all camera internal parameters.

[0082] The coordinates of the feature points in the camera coordinate system can also be obtained based on the camera motion of the previous and next two frames. The camera motion is described by the rotation parameter R and the translation parameter t. In this embodiment, the ORB feature points of two consecutive frames are first matched using the LK optical flow method. The dynamic points of the previous frame and the points corresponding to the matched current frame are eliminated based on the dynamic semantic information of the previous frame to obtain a more accurate basic matrix F. The essential matrix is ​​calculated based on the basic matrix:

[0083] E=K T ·F·K

[0084] Among them, K is the camera intrinsic parameter matrix. Then perform singular value decomposition on the essential matrix E to obtain 4 sets of camera extrinsic parameters R, t. Compare the 4 sets of R, t to restore the legal number of spatial points, and select the R, t with the largest legal number as the camera extrinsic parameter. In this way, the camera motion can be obtained by two consecutive frames.

[0085]

[0086] The scene flow of the i-th feature point at time t can be expressed as:

[0087]

[0088] In an ideal situation (with correct feature point matching, correct feature point depth correspondence, and correct camera extrinsic parameter calculation), the scene flow of static features should be close to 0, so that static points can be distinguished from dynamic points directly by setting a threshold. However, in practice, feature points may be incorrectly matched, the corresponding depth values ​​of feature points may be illegal, and the camera extrinsic parameter calculation results may be unstable, resulting in some static points even corresponding to large scene flows. To overcome this problem, this embodiment associates similar points and clusters them based on information such as depth, distance, and scene flow.

[0089] To reduce the computational complexity of directly comparing the similarities of all scene flows in an image, we first perform deep clustering. Based on this, we calculate the distances between points and establish connecting lines based on these distances. We then compare the similarities of the scene flows at both ends of these connecting lines and further cluster them. Points with similar distances and consistent motion characteristics are grouped together (clusters) to determine their motion state, rather than relying on highly uncertain methods based on single outliers.

[0090] Specifically, such as Figure 3 As shown, the first step is to remove the dynamic points of the current frame obtained by matching the prior dynamic points obtained by the previous frame target detection and obtain the point set M. Then, all points in M ​​are traversed and the depth threshold is used to determine whether the depth of the current feature point is less than the existing different depth categories dC j The average depth of the feature points is used to determine whether a new cluster needs to be established. In the second step, based on the depth clustering, we calculate the similarity of the scene flow between feature points based on the distance between feature points on the 2D image, and establish connecting lines based on the similarity of the scene flow.

[0091] The 2D distance between feature points is defined using Euclidean distance:

[0092]

[0093] (u, v) is the 2D coordinate of point p, and the similarity of scene flow is defined by the angle between the scene flow vectors:

[0094]

[0095] The third step is to cluster the points at both ends of the connecting line into one category. This allows the detection of highly correlated feature point clusters based on their distance and scene flow geometry, preparing for dynamic feature point selection in subsequent steps.

[0096] The object detection thread can detect actively moving objects (such as people and vehicles). The system combines depth information to generate a mask for the actively moving object. The feature points on the mask are the prior dynamic feature points. To accurately calculate the dynamic area of ​​passively moving objects and the undetected areas of actively moving objects, this embodiment compares the scene flow information of different clusters with the scene flow information of the entire image.

[0097] like Figure 4 As shown, the scene flow information is used to judge and calculate the geometric dynamic area, and combined with the semantic dynamic feature point information of the current frame, the dynamic points in the current frame can be accurately identified and tracked.

[0098] Since the size of the scene flow is affected by the depth, according to the camera imaging principle, the deeper the depth, the larger the scene flow amplitude. Therefore, we set the adjusted scene flow S i =V t i / Z t i , multiplying the scene flow with the inverse of the depth to reduce the impact of depth on dynamic area judgment.

[0099] In addition, after removing some dynamic points based on the prior dynamic information of the previous frame, most of the objects in the scene are static. Therefore, the current scene conforms to the static world hypothesis. On this basis, the average scene flow FS of all points is calculated to represent the scene flow of the static features of the current frame, and the average scene flow aS of each category is used to represent the static features of the scene flow. j The similarity with the scene flow FS of the static features of the current frame is used to determine whether the category is a dynamic category. The similarity comparison here is also based on vector size and angle.

[0100] If there is a dynamic scene flow category, it is considered that there are passively moving objects or points on actively moving objects that have not been completely eliminated. We use the convex hull algorithm to calculate the area containing all points in this category and consider that the points in this area are dynamic feature points. In this way, the motion state of some points that cannot be clustered due to illegal depth information can be judged. In addition, the prior dynamic information used before scene flow clustering is the target detection result of the previous frame. If the dynamic information of the previous frame is continued to be used in the dynamic point selection module, inaccurate tracking will occur. Therefore, it is necessary to wait for the target detection result of the current frame. If an actively moving object is detected, the depth information of the feature point is further detected to see if it is similar to the depth information of the center point of the detection frame. If it is similar, the point is considered to be a dynamic point. Through these preparations, it is possible to determine one by one whether all feature points in the current frame are dynamic points.

[0101] A 2D semantic information detection frame can be obtained from the target detection thread, but since the detection frame contains many noise areas that are not target objects, it is necessary to obtain a 3D point cloud through depth image information, combine the 3D point cloud and 2D semantic information, and generate 3D semantic object information.

[0102] To improve efficiency, the semantic map can be constructed using only keyframes. Furthermore, to address the issue of missed detection in the object detection thread, this embodiment checks the number of actively moving objects in three consistent frames. Only when the number remains stable is the detection result considered accurate, and the local point clouds are transformed and merged.

[0103] We evaluate our system using the public datasets TUM and Bonn and compare it to the original ORB-SLAM2 system to demonstrate the improvements our approach makes over ORB-SLAM2. We also compare SSF-SLAM with other state-of-the-art dynamic SLAM systems to demonstrate the superiority of the SSF-SLAM system in terms of robustness and real-time performance. All experiments in this paper were conducted on a computer running Ubuntu 18.04, an Intel i9 CPU, and 64GB of RAM.

[0104] A. Evaluation results on the TUM RGB-D dataset

[0105] The TUM RGB-D dataset, provided by the Technical University of Munich in Germany, contains RGB images, depth images, and corresponding camera pose information from a Kinect sensor. It is specifically designed for evaluating and comparing visual odometry-based SLAM algorithms and dense 3D reconstruction techniques. To evaluate the accuracy and robustness of the proposed SSF-SLAM system, five dynamic sequence sets included in the TUM RGB-D dataset were used. The first four are high-dynamic sequences, depicting two people walking in an office, and include four different camera motions: XYZ, stationary, RPY, and hemispherical. The last is a low-dynamic sequence, depicting two people sitting at a desk and occasionally gesturing.

[0106] The experiment used absolute trajectory error (ATE) and relative pose error (RPE) as error metrics. ATE is suitable for measuring global trajectory consistency, while RPE is suitable for measuring translational and rotational drift errors. The experimental results are shown in Tables I-III. We also compared the improved method with the original ORB-SLAM2 and calculated the improved data. The improvement values ​​in the table are calculated as follows:

[0107]

[0108] Where η represents the improvement value, S represents the value of SSF-SLAM, and O represents the value of ORB-SLAM2.

[0109] Table 1 ATE results of SLAM algorithms on TUM dataset [m]

[0110]

[0111] Table 2 RPE translation results of SLAM algorithm in TUM dataset [m / s]

[0112]

[0113] Table 3 RPE rotation results of SLAM algorithm in TUM dataset [° / s]

[0114]

[0115] The results shown in Tables 1-3 show that the SSF-SLAM system significantly improves accuracy in highly dynamic sequences compared to ORB-SLAM2. For walking sequences, the average RMSE of ATE, T.RPE, and R.RPE are 96.95%, 92.59%, and 91.34% higher than those of ORB-SLAM2, respectively. Furthermore, the average standard deviation improves by 96.62%, 94.51%, and 93.47%, respectively. ORB-SLAM2 is unable to obtain accurate trajectory estimates in highly dynamic sequences. Furthermore, we compare the SSF-SLAM system with state-of-the-art dynamic SLAM systems, including DS-SLAM, DynaSLAM, YOLO-SLAM, and SG-SLAM, as shown in Tables 1-3. In terms of absolute trajectory error, DynaSLAM and YOLO-SLAM achieve a slight lead in ATE in the fr3 / w / static sequence, but their real-time performance is poor, as shown in Table 4.

[0116] As can be seen from the relative pose error, the SSF-SLAM system performs more stably in highly dynamic sequences. In the fr3 / s / static sequence, the SSF-SLAM system consistently underperforms other dynamic SLAM systems. This is because dynamic SLAM already achieves high positioning accuracy in low-dynamic sequences, leaving limited room for improvement. Overall, the system performs well in highly dynamic scenes, with only minor limitations in low-dynamic ones.

[0117] B. Evaluation results on the Bonn RGB-D dataset

[0118] The Bonn RGB-D dynamic dataset is provided by the University of Bonn in Germany. Like the TUM dataset, it contains RGB images, depth images, and corresponding camera pose information from the Kinect sensor. The difference is that the Bonn RGB-D dynamic dataset is more complex than the TUM dataset and therefore more challenging.

[0119] To further evaluate the robustness of the SSF-SLAM system, we selected nine sequences from the Bonn dataset for evaluation and compared them with ORB-SLAM2, DS-SLAM, DynaSLAM, YOLO-SLAM, and SG-SLAM. The "crowd" sequence shows three people walking randomly in a room. The "moving_no_box" sequence shows a person moving a box from the floor to a table and from the table to the floor. The "person_tracking" sequence shows a camera tracking a walking person. The "synchronous" sequence shows two people moving synchronously along the epipolar line.

[0120] Table 4 ATE results of SLAM algorithm on Bonn dataset [m]

[0121]

[0122] Table 4 shows the comparative evaluation results for nine dynamic scene sequences. In the "crowd" and "synchronous" sequences, DynaSLAM and YOLO-SLAM, which utilize more precise deep learning networks, achieve higher localization accuracy. However, the SSF-SLAM system is comparable and significantly outperforms DynaSLAM and YOLO-SLAM in terms of real-time performance. In the "moving_no_box" sequence, the SSF-SLAM system's accuracy is slightly lower than that of SG-SLAM. This is due to errors in the scene flow clustering when calculating the camera pose, which affects the accuracy of dynamic point determination. In all other sequences, the SSF-SLAM system outperforms the other algorithms. Its average ATE of 0.0307 and maximum ATE of 0.0371 are significantly lower than those of the other systems. This indicates that the worst-case performance of SSF-SLAM is significantly better than that of the other systems, indicating that our method is more robust to dynamic environments.

[0123] C. Ablation Experiment

[0124] The SSF-SLAM system combines the advantages of object detection and geometric constraints to select and eliminate dynamic feature points. To verify the functionality of each module, we designed ablation experiments. SSF-SLAM(S) is a dynamic feature selection strategy based on semantic information, SSF-SLAM(SF) is a dynamic feature selection strategy based on geometric information from scene flow clustering, and SSF-SLAM(S+SF) is a dynamic feature selection strategy that integrates semantic and geometric information. The experimental results are shown in Table 5.

[0125] Table 5 Comparison of ATE results of ablation experiments on the TUM dataset [M]

[0126]

[0127] Across five sequences, the average RMSE (RMSE) of the ATEs for SSF-SLAM(S), SSF-SLAM(S+SF), and SSF-SLAM(S+SF) were 92.70%, 72.00%, and 97.27% higher than those for ORB-SLAM2, respectively. The average SD (SD) of the ATEs was 93.17%, 70.33%, and 97.48% higher than those for ORB-SLAM2. SSF-SLAM(SF) performed poorly in high-dynamic sequences, but outperformed SSF-SLAM(S) in low-dynamic sequences. As discussed in this paper, without prior dynamic information provided by the object detection module, some dynamic features may be introduced into the scene flow calculation, reducing system accuracy. Without the scene flow clustering module, feature points on passively moving objects and some actively moving objects cannot be identified, resulting in a decrease in system accuracy. Ablation experiments demonstrate that combining semantic information with scene flow geometry can further extract information from the scene, enabling more accurate detection of dynamic objects and thus improving system accuracy.

[0128] D. Runtime Analysis

[0129] In practical applications, real-time performance is an important metric for evaluating SLAM systems. As shown in Table 6, we tested the average time cost per frame processed during system runtime and compared it with other systems. As can be seen from Table VI, DynaSLAM, DS-SLAM, and RDS-SLAM still have a long average runtime per frame even with GPU acceleration due to their use of pixel-level semantic segmentation. The object detection thread in the SSF-SLAM system runs independently of the entire system, significantly improving the processing speed per frame. Without a GPU, the increase in processing time is less than 4ms compared to ORB-SLAM2, enabling the system to run in real time.

[0130] Table 6 Real-time performance analysis of SSF-SLAM algorithm

[0131]

[0132] E. Semantic Mapping

[0133] To further evaluate the accuracy and stability of the system tracking, we constructed semantic maps on the TUM RGB-D dataset and the BonnRGB-D dataset. Figure 5These are the dense reconstruction results of the above two data sets. (a) is ORB-SLAM2(TUM), (b) is SSF-SLAM(TUM), (c) is ORB-SLAM2(Bonn), and (d) is SSF-SLAM(Bonn). In the TUM data set, the dense map obtained by ORB-SLAM2 is filled with the shadows of moving people. In the Bonn data set, since ORB-SLAM2 is affected by dynamic objects, it cannot obtain the correct camera pose, so it cannot correctly merge sub-maps and obtains an erroneous global map. The method proposed in this embodiment not only constructs a correct static map, but also provides semantic information to facilitate mobile robots to understand scenes and perform more advanced tasks.

[0134] In summary, this embodiment combines lightweight semantic information with scene flow geometry. Multi-threaded parallel processing reduces the waiting time for semantic information acquisition. Furthermore, it clusters scene flow geometry, overcoming the limitations of geometric constraints in complex scenes and reducing the impact of noise on dynamic feature judgment. Extensive experimental results demonstrate that the method provided by this embodiment significantly improves real-time performance, accuracy, and robustness in complex dynamic environments.

[0135] Furthermore, this method constructs a semantic map using 2D semantic information. Object detection and semantic mapping are each integrated into a single thread, running in parallel with other threads to ensure real-time performance. This method was compared with other advanced dynamic SLAM methods on the TUM and Bonn datasets. Experimental results validated the feasibility and effectiveness of this approach.

[0136] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made by those skilled in the art that fall within the spirit and principles of the present invention and do not require creative effort are intended to be within the scope of protection of the present invention.

Claims

1. A positioning and mapping method based on semantic and scene flow geometric information, characterized by: The following steps are involved: Get RGB image and depth image; Performing target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects; Matching feature points of two consecutive frames in the RGB image, and using the dynamic semantic information of the previous frame, matching the points corresponding to the current frame as prior dynamic points and removing them; Calculate the basic matrix between the two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. According to the essential matrix, obtain the camera extrinsic parameters, determine the camera pose, and calculate the scene flow of the feature points in combination with the depth information, and perform scene flow clustering; Combine the dynamic points obtained by the current frame target detection and the points in the geometric dynamic area obtained by scene flow clustering to select the final dynamic feature points and obtain the static feature points; Select a new key frame and perform camera pose prediction based on the static feature points; Based on the depth information, the predicted camera pose, and the keyframe feature points, the point cloud information is determined, the semantic information is converted into 3D semantic information, and the similarity matching between the point cloud information and the 3D semantic information is performed to obtain the final semantic map. The process of scene flow clustering includes: first performing deep clustering, calculating the distance between points based on deep clustering, establishing connecting lines based on the distances, comparing the similarities of the scene flows at both ends of the connecting lines, and further clustering them; points with similar distances and consistent movement characteristics are grouped together to determine their motion status; The process of deep clustering includes: according to the prior dynamic points obtained by the previous frame target detection, the dynamic points of the current frame obtained by matching, and removing them to obtain the point set ， Traverse all points in the point set and use the depth threshold to determine whether the depth of the current feature point is less than the average depth of the existing different depth categories to decide whether a new cluster needs to be established; Establishing a connecting line based on the distance, comparing the similarity of the scene flows at both ends of the connecting line, and further clustering the process includes calculating the similarity of the scene flows between feature points based on the distance between the feature points, and establishing a connecting line based on the similarity of the scene flows, and the distance between the feature points uses the Euclidean distance; Based on connecting line clustering, the points at both ends of the connecting line are classified into one category, and highly correlated feature point clusters are detected according to the distance of feature points and scene flow geometric information.

2. The positioning and mapping method based on semantic and scene flow geometric information according to claim 1, characterized in that: The process of matching the feature points of two consecutive frames in the RGB image is to match the feature points of the current frame and the previous frame through the LK optical flow method, and then combine the depth information to obtain the points of the previous frame that are within the prior dynamic frame and whose depth information is similar to the depth information of the midpoint of the prior dynamic frame, and match these points to the current frame as the prior dynamic points of the current frame for elimination.

3. The positioning and mapping method based on semantic and scene flow geometric information according to claim 1, characterized in that: The process of calculating the basic matrix between two frames after removing the prior dynamic points and calculating the essential matrix based on the camera intrinsic parameters includes removing the dynamic points of the previous frame and the points corresponding to the current frame according to the dynamic semantic information of the previous frame, obtaining the basic matrix, and calculating the essential matrix based on the basic matrix: in, K is the camera intrinsic parameter matrix, F As the basic matrix.

4. The positioning and mapping method based on semantic and scene flow geometric information as claimed in claim 3, characterized in that: The process of obtaining camera extrinsic parameters according to the essential matrix includes: performing singular value decomposition on the essential matrix E to obtain 4 sets of camera extrinsic parameters R, t, and comparing the 4 sets R , t Restore the legal number of spatial points and select R,t with the largest legal number as the camera extrinsic parameter.

5. The positioning and mapping method based on semantic and scene flow geometric information according to claim 1, characterized in that: The process of scene flow clustering to obtain geometric dynamic areas includes: multiplying the scene flow by the inverse of the depth; after eliminating a part of the dynamic points based on the prior dynamic information of the previous frame, calculating the average scene flow of all points, using the average scene flow to represent the scene flow of the static features of the current frame, and judging whether the category is a dynamic category based on the similarity between the average scene flow of each category and the scene flow of the static features of the current frame; if there is a dynamic scene flow category, it is considered that there are objects with passive motion or points on objects with active motion that have not been completely eliminated, and the convex hull algorithm is used to calculate the area containing all points of this category, and it is considered that all points in this area are dynamic feature points.

6. A positioning and mapping system based on semantic and scene flow geometric information, characterized by: include: An image acquisition module is configured to acquire an RGB image and a depth image; an object detection module configured to perform object detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects; and obtain a specific semantic mask in combination with the depth image; An optical flow matching module is configured to match feature points of two consecutive frames in the RGB image, and based on the dynamic semantic information of the previous frame, the points corresponding to the current frame obtained by matching are used as prior dynamic points and are eliminated; The scene flow clustering module is configured to calculate the basic matrix between two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. Based on the essential matrix, the camera extrinsic parameters are obtained to determine the camera pose, and the scene flow of the feature points is calculated in combination with the depth information to perform scene flow clustering. The dynamic selection module is configured to select the final dynamic feature points by combining the dynamic points obtained by the target detection in the current frame and the points in the geometric dynamic area obtained by the scene flow clustering to obtain the static feature points; A pose prediction module is configured to select a new key frame and perform camera pose prediction based on the static feature points; The semantic mapping module is configured to determine point cloud information based on depth information, predicted camera pose, and keyframe feature points, convert the semantic information into 3D semantic information, and perform similarity matching between the point cloud information and the 3D semantic information to obtain the final semantic map; The process of scene flow clustering includes: first performing deep clustering, calculating the distance between points based on deep clustering, establishing connecting lines based on the distances, comparing the similarities of the scene flows at both ends of the connecting lines, and further clustering them; points with similar distances and consistent movement characteristics are grouped together to determine their motion status; The process of deep clustering includes: according to the prior dynamic points obtained by the previous frame target detection, the dynamic points of the current frame obtained by matching, and removing them to obtain the point set ， Traverse all points in the point set and use the depth threshold to determine whether the depth of the current feature point is less than the average depth of the existing different depth categories to decide whether a new cluster needs to be established; Establishing a connecting line based on the distance, comparing the similarity of the scene flows at both ends of the connecting line, and further clustering the process includes calculating the similarity of the scene flows between feature points based on the distance between the feature points, and establishing a connecting line based on the similarity of the scene flows, and the distance between the feature points uses the Euclidean distance; Based on connecting line clustering, the points at both ends of the connecting line are classified into one category, and highly correlated feature point clusters are detected according to the distance of feature points and scene flow geometric information.

7. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the steps of the method according to any one of claims 1 to 5 are completed when the computer instructions are executed by the processor.

Images