Positioning and mapping method and system based on semantic and scene flow geometric information

Abstract

The invention provides a positioning and mapping method and system based on semantics and scene flow geometric information, which are characterized in that the semantic information of the environment is used for carrying out auxiliary detection on a dynamic target, and the obtained semantic information is tightly coupled with multi-view geometry to realize rapid and accurate identification of the dynamic target; the scene flow geometric information clustering based on depth and density is combined, so that the limitation of the common geometric constraint in the dynamic environment is effectively reduced, and the geometric dynamic region can be quickly and accurately calculated after the priori dynamic object is removed; in a complex dynamic scene, the time consumed for mapping is less, the accuracy and robustness are higher, and an intuitive semantic measurement map can be created. According to the method, the positioning precision of the system is remarkably improved, the calculation efficiency is excellent, the method is suitable for the complex environment, and the real-time performance, the accuracy and the robustness in the complex dynamic environment are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of positioning and mapping, and specifically relates to a positioning and mapping method and system based on semantic and scene flow geometric information. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Simultaneous Localization and Mapping (SLAM) has always been an important research direction in mobile robot control. SLAM aims to achieve autonomous navigation of robots by sensing the environment and obtaining their own position information at the same time. Among them, visual SLAM, as a camera-based SLAM method, has received widespread attention due to its low cost, easy availability of equipment, high-resolution image capture capabilities, and rich environmental information processing capabilities.

[0004] The development of visual SLAM has gone through several stages, and many classic visual SLAM frameworks have emerged. Among the many classic visual SLAM frameworks, ORB-SLAM2 has become a popular choice due to its excellent performance and good robustness. However, in dynamic environments, ORB-SLAM2 has difficulty handling dynamic objects and often mistakenly maps dynamic objects as part of the environment. Therefore, research on dynamic SLAM has become a current hot topic and challenge.

[0005] Most advanced dynamic SLAM methods rely on the static world assumption. However, there are a large number of dynamic objects in the real world. In a highly dynamic environment, it is often difficult to achieve good segmentation results. Misidentification of dynamic objects will lead to mismatching of feature points, thereby reducing the overall positioning and mapping accuracy of the system. Therefore, accurate identification of dynamic feature points is a prerequisite for achieving precise positioning in a dynamic environment.

[0006] In the visual SLAM system, the recognition of dynamic features is mainly divided into two ways: one is to distinguish dynamic points from static points by geometric consistency, and the other is to judge by combining semantic information with geometric information. The core idea of ​​the geometric consistency method is to identify dynamic feature points by analyzing the geometric relationship of feature points. These methods usually have low computational complexity and do not need to rely on pre-training data, but are prone to false detection when dealing with complex scenes. The method of combining semantics with geometric information introduces the results of target detection or semantic segmentation, and uses semantic information to enhance the system's understanding of the environment, thereby improving the robustness and accuracy of the system. Semantic annotation can provide prior semantic information for the selection of dynamic feature points, which helps to accurately identify dynamic feature points in complex scenes.

[0007] However, in the system using target detection, although the real-time performance is better than that of semantic segmentation, it cannot accurately identify the mask of dynamic objects, resulting in inaccurate dynamic point recognition. Even if the SLAM method using pixel-level semantic segmentation reduces the processing frame, the amount of calculation is still large, making it difficult to achieve real-time operation of the system. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes a positioning and mapping method and system based on semantics and scene flow geometric information. The present invention clusters the scene flow geometric information to reduce the influence of incorrect feature matching and motion approximation on the judgment of feature point motion attributes; and integrates the semantics and scene flow geometric information to suppress dynamic features, which not only significantly improves the positioning accuracy of the system, but also has excellent computational efficiency, is suitable for complex environments, significantly improves the real-time, accuracy and robustness in complex dynamic environments, and is conducive to further research on mobile robots.

[0009] According to some embodiments, the present invention adopts the following technical solutions:

[0010] A positioning and mapping method based on semantics and scene flow geometric information includes the following steps:

[0011] Get RGB image and depth image;

[0012] Performing target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects;

[0013] Matching feature points of two consecutive frames in the RGB image, and according to the dynamic semantic information of the previous frame, matching points corresponding to the current frame as prior dynamic points, and removing them;

[0014] Calculate the basic matrix between two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. According to the essential matrix, obtain the camera extrinsic parameters, determine the camera pose, and calculate the scene flow of feature points in combination with the depth information to perform scene flow clustering;

[0015] Combine the dynamic points obtained by the current frame target detection and the points in the geometric dynamic area obtained by scene flow clustering to select the final dynamic feature points and obtain the static feature points;

[0016] Select a new key frame and perform camera pose prediction based on the static feature points;

[0017] Based on the depth information, predicted camera pose and feature points of key frames, the point cloud information is determined, the semantic information is converted into three-dimensional semantic information, and the similarity matching between the point cloud information and the three-dimensional semantic information is performed to obtain the final semantic map.

[0018] As an optional implementation, the process of matching the feature points of two consecutive frames in the RGB image is to match the feature points of the current frame with the previous frame through the LK optical flow method, and then combine the depth information to obtain the points of the previous frame that are within the prior dynamic frame and whose depth information is similar to the depth information of the midpoints in the prior dynamic frame, and match these points to the current frame as the prior dynamic points of the current frame for elimination.

[0019] As an optional implementation, the process of calculating the basic matrix between two frames after removing the prior dynamic points and calculating the essential matrix according to the camera intrinsic parameters includes removing the dynamic points of the previous frame and the points corresponding to the matched current frame according to the dynamic semantic information of the previous frame, obtaining the basic matrix, and calculating the essential matrix according to the basic matrix:

[0020] E=K T ·F·K

[0021] Among them, K is the camera intrinsic parameter matrix and F is the basic matrix.

[0022] As an optional implementation, the process of obtaining camera extrinsic parameters based on the essential matrix includes: performing singular value decomposition on the essential matrix E to obtain 4 sets of camera extrinsic parameters R, t, comparing the 4 sets of R, t to restore the legal number of spatial points, and selecting the R, t with the largest number of legal points as the camera extrinsic parameters.

[0023] As an optional implementation, the process of scene flow clustering includes: first performing deep clustering, calculating the distance between points based on the deep clustering, establishing connecting lines based on the distances, comparing the similarities of the scene flows at both ends of the connecting lines, and further clustering them; and grouping points with similar distances and consistent movement characteristics to determine their motion states.

[0024] As a further step, the process of performing deep clustering first includes: obtaining the dynamic points of the current frame through matching based on the prior dynamic points obtained from the target detection of the previous frame, and removing them to obtain a point set, traversing all the points in the point set, and judging whether the depth of the current feature point is less than the average depth of the existing different depth categories through the depth threshold, so as to decide whether a new cluster needs to be established.

[0025] As a further step, a connection line is established based on the distance, and the similarity of the scene flows at both ends of the connection line is compared. The process of further clustering the connection line includes calculating the similarity of the scene flows between the feature points based on the distance between the feature points, and establishing a connection line based on the similarity of the scene flows, and the distance between the feature points uses the Euclidean distance;

[0026] Based on connection line clustering, the points at both ends of the connection line are classified into one category, so that highly correlated feature point clusters can be detected according to the distance of feature points and scene flow geometric information.

[0027] As an optional implementation, the process of scene flow clustering to obtain geometric dynamic areas includes: multiplying the scene flow by the inverse of the depth; after eliminating a portion of dynamic points based on the prior dynamic information of the previous frame, calculating the average scene flow of all points, using the average scene flow to represent the scene flow of the static features of the current frame, and judging whether the category is a dynamic category based on the similarity between the average scene flow of each category and the scene flow of the static features of the current frame; if there is a dynamic scene flow category, it is considered that there are points on objects with passive motion or objects with active motion that have not been completely eliminated, and a convex hull algorithm is used to calculate the area containing all points of the category, and it is considered that all points in the area are dynamic feature points.

[0028] A positioning and mapping system based on semantic and scene flow geometric information, including:

[0029] An image acquisition module, configured to acquire an RGB image and a depth image;

[0030] The target detection module is configured to perform target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects; and obtain a specific semantic mask in combination with the depth image;

[0031] An optical flow matching module is configured to match feature points of two consecutive frames in the RGB image, and according to the dynamic semantic information of the previous frame, the points corresponding to the current frame obtained by matching are used as prior dynamic points and removed;

[0032] The scene flow clustering module is configured to calculate the basic matrix between two frames after removing the prior dynamic points, and calculate the essential matrix according to the camera intrinsic parameters, obtain the camera extrinsic parameters according to the essential matrix, determine the camera pose, and calculate the scene flow of the feature points in combination with the depth information to perform scene flow clustering;

[0033] A dynamic selection module is configured to select the final dynamic feature points by combining the dynamic points of the current frame and the points in the geometric dynamic area obtained by scene flow clustering to obtain static feature points;

[0034] A pose prediction module is configured to select a new key frame and perform camera pose prediction in combination with the static feature points;

[0035] The semantic mapping module is configured to determine the point cloud information based on the depth information, the predicted camera pose and the feature points of the key frames, convert the semantic information into three-dimensional semantic information, perform similarity matching between the point cloud information and the three-dimensional semantic information, and obtain the final semantic map.

[0036] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0037] The electronic device can be mounted on a robot.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention is based on a visual SLAM system, adds a target detection thread and a semantic mapping thread, and runs in parallel with multiple threads, which effectively improves the real-time performance of the SLAM system. The target detection thread provides a priori semantic information through a target detection algorithm and can provide target detection results in real time. The semantic mapping thread can remove the point cloud located in the dynamic marking area through the semantic information provided by the target detection thread, generate a three-dimensional semantic object database, and finally obtain a static semantic scene reconstruction.

[0040] The present invention uses the prior dynamic information of the previous frame to eliminate the prior dynamic points before scene flow clustering, and combines the prior dynamic points of the current frame and the points in the dynamic area obtained by scene flow clustering to perform final dynamic feature point elimination. This not only reduces the impact of the static world assumption on high dynamic situations, but also enables the system to run in real time.

[0041] The present invention uses the semantic information of the environment to assist in the detection of dynamic targets, and tightly couples the acquired semantic information with multi-view geometry to achieve rapid and accurate recognition of dynamic targets. It also combines scene flow geometric information clustering based on depth and density to effectively reduce the limitations of commonly used geometric constraints in dynamic environments. After removing prior dynamic objects, it can quickly and accurately calculate the geometric dynamic area. In complex dynamic scenes, mapping takes less time, has higher accuracy and robustness, and can create intuitive semantic metric maps.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0044] Figure 1 is a flow chart of a positioning and mapping method based on semantics and scene flow geometric information of an embodiment;

[0045] Figure 2 is a schematic diagram of a positioning and mapping method based on semantics and scene flow geometric information in an embodiment;

[0046] Figure 3 is a schematic diagram of a scene flow clustering process of an embodiment;

[0047] Figure 4 is a schematic diagram of a process of calculating a geometric dynamic area according to scene flow information in an embodiment;

[0048] Figure 5 The diagram is a schematic diagram of semantic mapping results using different methods of an embodiment. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0052] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0053] Embodiment 1

[0054] A localization and mapping method based on semantic and scene flow geometric information, such as Figure 1 As shown, the following steps are included:

[0055] Get RGB image and depth image;

[0056] Performing target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects;

[0057] Matching feature points of two consecutive frames in the RGB image, and according to the dynamic semantic information of the previous frame, matching points corresponding to the current frame as prior dynamic points, and removing them;

[0058] Calculate the basic matrix between two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. According to the essential matrix, obtain the camera extrinsic parameters, determine the camera pose, and calculate the scene flow of feature points in combination with the depth information to perform scene flow clustering;

[0059] Combine the dynamic points obtained by the current frame target detection and the points in the geometric dynamic area obtained by scene flow clustering to select the final dynamic feature points and obtain the static feature points;

[0060] Select a new key frame and perform camera pose prediction based on the static feature points;

[0061] Based on the depth information, predicted camera pose and feature points of key frames, the point cloud information is determined, the semantic information is converted into three-dimensional semantic information, and the similarity matching between the point cloud information and the three-dimensional semantic information is performed to obtain the final semantic map.

[0062] The process of the above method is divided into multiple threads, and at least some of the threads can be operated in parallel.

[0063] First, for the convenience of description, a positioning and mapping method based on semantics and scene flow geometric information in this embodiment is referred to as SSF-SLAM.

[0064] The SSF-SLAM provided in this embodiment is based on the ORB-SLAM2 framework. ORB-SLAM2 contains three main threads: tracking thread, local mapping thread, and loop detection thread. In order to fully utilize the advantages of ORB-SLAM2, this embodiment uses ORB-SLAM2 as the framework. The principle of SSF-SLAM is as follows: Figure 2 As shown, the blue box is the original structure of ORB-SLAM2, and the red box is the added or modified content.

[0065] Most advanced dynamic SLAM methods cannot guarantee the real-time operation of the system while using semantic information to improve robustness. Therefore, in SSF-SLAM, this embodiment adds two threads: a target detection thread and a semantic mapping thread.

[0066] Multi-threaded parallel operation effectively improves the real-time performance of the SLAM system. The target detection thread provides prior semantic information through the target detection algorithm. Since the target detection algorithm does not need to calculate detailed semantic masks, it can provide target detection results in real time. The semantic mapping thread can remove the point cloud located in the dynamic marking area through the semantic information provided by the target detection thread, generate a three-dimensional semantic object database, and finally obtain a static semantic scene reconstruction.

[0067] Specifically, Figure 2 As shown in Figure 1, when SSF-SLAM is executed, the image frames captured from the RGB-D camera are first fed to the tracking thread and the target detection thread.

[0068] While the tracking thread is extracting ORB features, the target detection thread is detecting targets and marking prior dynamic objects. Target detection on images captured by the RGB-D camera can detect not only dynamic semantic information, but also semantic information of other objects. Here, the points on the target object "person" are taken as prior dynamic points, specifically the points on the "person" in the previous frame.

[0069] After the tracking thread obtains the dynamic semantic information of the previous frame, it matches the feature points of the current frame and the previous frame through the LK optical flow method, and then obtains the points of the previous frame that are in the prior dynamic frame and have a depth similar to the depth of the midpoints in the prior dynamic frame in combination with the depth information, and matches these points to the current frame as the prior dynamic points of the current frame for elimination. In this way, the static points of the current frame that match each other can be roughly obtained, and the points on non-dynamic objects in the dynamic frame can be avoided as much as possible.

[0070] After removing the dynamic points of the previous frame matched by the current frame, this embodiment uses the seven-point method based on RANSAC to calculate the basic matrix between the two frames, and calculates the essential matrix based on the camera intrinsic parameters. This can reduce the adverse effects caused by incorrect data association in the dynamic area.

[0071] Afterwards, the obtained essential matrix is ​​decomposed by singular value SVD and triangulated to obtain the camera extrinsic parameters (R, t), the scene flow of the point is calculated in combination with the depth information, and the scene flow clustering is performed.

[0072] Compared with feature extraction, feature matching, essential matrix calculation, and scene flow calculation, the target detection process is more time-consuming. If the prior dynamic information of the current frame is directly used to calculate the essential matrix and cluster the scene flow, it is necessary to wait for the completion of the target detection. Although this embodiment has adopted a target detection method with less computational effort, in order to minimize the time spent waiting for target detection, the prior dynamic information of the previous frame is used to eliminate the prior dynamic points before scene flow clustering.

[0073] Finally, the final dynamic feature points are removed by combining the prior dynamic points of the current frame and the points in the dynamic area obtained by scene flow clustering. This not only reduces the impact of the static world assumption on high dynamic situations, but also enables the system to run in real time.

[0074] The following is a detailed introduction to the content of each thread.

[0075] First, the target detection process of this embodiment is performed using NCNN. NCNN is a lightweight neural network computing framework that is optimized for mobile terminals and embedded devices. Since mobile robots generally choose ARM architecture processors with higher performance per watt, NCNN is implemented in pure C++, does not rely on any third-party libraries, supports cross-platform operations, has low precision loss and memory usage, and has fast computing speed. It can be easily integrated into the SLAM system. Therefore, it is selected as the basic framework for target detection.

[0076] NCNN supports most commonly used CNN networks, such as VGG, AlexNet, GoogleNet Inception, SqueezeNet, MobileNetV1 / V2 / V3, etc., as well as MTCNN in the detection field. In order to achieve real-time operation of the system, this embodiment uses MobileNetV3 as a direct substitute for backbone feature extraction in SSDLite. It avoids the slow operation problem of traditional SLAM systems due to complex semantic segmentation. Finally, the network is trained using the PASCAL VOC 2007 dataset.

[0077] Given the rich capabilities of NCNN, in practical applications, other deep learning methods can be flexibly used according to the performance of the hardware to strike a balance between accuracy and speed.

[0078] This embodiment uses the geometric information obtained by scene flow clustering to determine whether a feature point is a dynamic feature point. In a static scene, the movement of points in the camera coordinate system has a similar trend, while dynamic points do not. Therefore, it is possible to determine whether a point is a dynamic point based on the scene flow information of the feature point, which involves two problems: scene flow calculation and scene flow clustering.

[0079] The scene flow information of the feature points is calculated by multi-view geometry. For binocular and RGBD cameras, the stereo information of the feature points can be obtained, including the right eye feature point information and the depth information of the feature points. In this way, the three-dimensional coordinates in the camera coordinate system can be obtained by the parallax ranging method. The binocular camera uses the feature correspondence between the left and right eye images to obtain the disparity d, and the RGBD camera can directly obtain the depth value of the feature point (i.e., disparity d). At time t, the pixel coordinates of the i-th point are expressed as (u i t ,v i t ), where u represents the horizontal coordinate in the pixel coordinate system, v represents the vertical coordinate in the pixel coordinate system, and the coordinate P in the current camera coordinate system i t =(X i t ,Y i t ,Zi t ) T It can be expressed as:

[0080]

[0081] Where b is the baseline distance of the binocular camera, (c x ,c y ) is the coordinate origin, parameter c x ,c y ,f are camera internal parameters.

[0082] The coordinates of the feature points in the camera coordinate system can also be obtained according to the camera motion of the previous and next two frames. The camera motion is described by the rotation parameter R and the translation parameter t. In this embodiment, the ORB feature points of two consecutive frames are first matched by the LK optical flow method, and the dynamic points of the previous frame and the points corresponding to the matched current frame are eliminated according to the dynamic semantic information of the previous frame to obtain a more accurate basic matrix F. The essential matrix is ​​calculated according to the basic matrix:

[0083] E=K T ·F·K

[0084] Among them, K is the camera internal parameter matrix. Then, the essential matrix E is decomposed into singular values ​​to obtain four sets of camera external parameters R, t. The four sets of R, t are compared to restore the legal number of spatial points, and the R, t with the largest legal number is selected as the camera external parameter. In this way, the camera motion of two consecutive frames can be used to obtain

[0085]

[0086] The scene flow of the i-th feature point at time t can be expressed as:

[0087]

[0088] In an ideal situation (correct feature point matching, correct feature point depth correspondence, and correct camera extrinsic calculation), the scene flow of static features should be close to 0, so that static points and dynamic points can be distinguished directly by setting a threshold. In practice, however, there are incorrect matches of feature points, the depth values ​​corresponding to feature points may be illegal, and the calculation results of camera extrinsic parameters are unstable, resulting in some static points even corresponding to large scene flows. To overcome this problem, this embodiment associates similar points and clusters them based on information such as depth, distance, and scene flow.

[0089] In order to reduce the amount of calculation caused by directly comparing the similarity of all scene flows in the image, deep clustering is first performed. The distance between points is calculated based on deep clustering, and connecting lines are established based on the distance. Then, the similarity of the scene flows at both ends of the connecting line is compared and further clustered. Points with similar distances and consistent movement characteristics are grouped (clustered) to determine their motion state, rather than relying on highly uncertain methods based on single outliers.

[0090] Specifically, Figure 3 As shown in Figure 1, the first step is to remove the dynamic points of the current frame obtained by matching the prior dynamic points obtained by the target detection in the previous frame and obtain the point set M. Then, all the points in M ​​are traversed and the depth threshold is used to determine whether the depth of the current feature point is less than the existing different depth categories dC. j The average depth of the feature points is used to determine whether a new cluster needs to be established. In the second step, based on the depth clustering, we calculate the similarity of the scene flow between feature points based on the distance between feature points on the 2D image, and establish connecting lines based on the similarity of the scene flow.

[0091] The 2D distance between feature points is defined using Euclidean distance:

[0092]

[0093] (u,v) is the 2D coordinate of point p, and the similarity of scene flow is defined by the angle between scene flow vectors:

[0094]

[0095] The third step is to cluster the points at both ends of the connecting line into one category based on the connecting line. In this way, highly correlated feature point clusters can be detected based on the distance of the feature points and the scene flow geometry information, preparing for the dynamic feature point selection in the subsequent steps.

[0096] The target detection thread can detect active moving objects (such as people, cars, etc.). The system combines the depth information to obtain the mask of the active moving object. The feature points on the mask are the prior dynamic feature points. In order to accurately calculate the dynamic area of ​​the passive moving object and the undetected area on the active moving object, this embodiment compares the relationship between the scene flow information of different clusters and the scene flow information of the entire picture.

[0097] like Figure 4 As shown, the scene flow information is used to judge and calculate the geometric dynamic area, and combined with the semantic dynamic feature point information of the current frame, the dynamic points in the current frame can be accurately identified and tracked.

[0098] Since the size of the scene flow is affected by the depth, according to the camera imaging principle, the deeper the depth, the larger the scene flow amplitude. Therefore, we set the adjusted scene flow S i =V t i / Z t i , multiply the scene flow by the inverse of the depth to reduce the impact of depth on dynamic area judgment.

[0099] In addition, after removing some dynamic points based on the prior dynamic information of the previous frame, most of the objects in the scene are static, so the current scene conforms to the static world assumption. On this basis, the average scene flow FS of all points is calculated to represent the scene flow of the static features of the current frame, and the average scene flow aS of each category is used to represent the scene flow of the static features of the current frame. j The similarity with the scene flow FS of the static features of the current frame is used to determine whether the category is a dynamic category. The similarity comparison here is also based on the vector size and angle.

[0100] If there is a dynamic scene flow category, it is considered that there are passively moving objects or points on actively moving objects that have not been completely eliminated. We use the convex hull algorithm to calculate the area containing all points of this category, and consider that the points in this area are dynamic feature points. In this way, the motion state of some points that cannot be clustered in scene flow due to illegal depth information can be judged. In addition, the prior dynamic information used before scene flow clustering is the target detection result of the previous frame. If the dynamic information of the previous frame continues to be used in the dynamic point selection module, inaccurate tracking will occur. Therefore, it is necessary to wait for the target detection result of the current frame. If an actively moving object is detected, further detect whether the depth information of the feature point is similar to the depth information of the center point of the detection frame. If similar, the point is considered to be a dynamic point. Through these preparations, it is possible to determine whether all feature points in the current frame are dynamic points one by one.

[0101] The 2D semantic information detection box can be obtained from the target detection thread, but since the detection box contains many noise areas that are not target objects, it is necessary to obtain the 3D point cloud through the depth image information, combine the 3D point cloud and 2D semantic information, and generate 3D semantic object information.

[0102] To improve efficiency, the semantic map can be constructed using only key frames. In addition, to solve the problem of missed detection in the target detection thread, this embodiment checks the number of actively moving objects in three consistent frames. Only when the number remains stable, we consider the detection result accurate and transform and merge the local point clouds.

[0103] Our system will be evaluated using the public datasets TUM and Bonn and compared with the original ORB-SLAM2 system to show the improvement of our method over the ORB-SLAM2 system. We also compare SSF-SLAM with other advanced dynamic SLAM systems to show the superiority of the SSF-SLAM system in terms of robustness and real-time performance. All experiments in this paper are performed on a computer with Ubuntu 18.04 operating system, Intel I9 CPU, and 64-GB RAM.

[0104] A. Evaluation results on the TUM RGB-D dataset

[0105] The TUM RGB-D dataset is provided by the Technical University of Munich in Germany. It contains RGB images, depth images, and corresponding camera pose information from the Kinect sensor, and is specifically used to evaluate and compare SLAM algorithms and dense 3D reconstruction techniques based on visual odometers. In order to evaluate the accuracy and robustness of the proposed SSF-SLAM system, five dynamic sequence sets included in the TUM RGB-D dataset were used. Among them, the first four are high-dynamic sequences, which are scenes of two people walking in an office, including four different camera motion modes: XYZ, stationary, RPY, and hemispherical. The last one is a low-dynamic sequence, which is a scene of two people sitting at a table and occasionally making gestures.

[0106] The experiment used absolute trajectory error (ATE) and relative posture error (RPE) as error metrics. ATE is suitable for measuring the global consistency of the trajectory, while RPE is suitable for measuring the drift error of translation and rotation. The experimental results are shown in Tables I-III. We also compared the improved method with the original ORB-SLAM2 and calculated the improved data. The calculation method of the improved value in the table is as follows:

[0107]

[0108] Among them, η represents the improvement value, S represents the value of SSF-SLAM, and O represents the value of ORB-SLAM2.

[0109] Table 1 ATE results of SLAM algorithms on TUM dataset [m]

[0110]

[0111] Table 2 RPE translation results of SLAM algorithm in TUM dataset [m / s]

[0112]

[0113] Table 3 RPE rotation results of SLAM algorithm in TUM dataset [° / s]

[0114]

[0115] In the results shown in Table 1-3, compared with ORB-SLAM2, the accuracy of the SSF-SLAM system in high dynamic sequences has been significantly improved. The average RMSE of ATE, T.RPE and R.RPE in the walking sequence is 96.95%, 92.59% and 91.34% higher than that of ORB-SLAM2, respectively. In addition, the improvement of the average standard deviation is 96.62%, 94.51% and 93.47%, respectively. In high dynamic sequences, ORB-SLAM2 cannot obtain accurate estimated trajectories. In addition, we also compare the SSF-SLAM system with advanced dynamic SLAM systems such as DS-SLAM, DynaSLAM, YOLO-SLAM and SG-SLAM, as shown in Table 1-3. From the perspective of absolute trajectory error, in the fr3 / w / static sequence, the ATE results of the DynaSLAM system and the YOLO-SLAM system are slightly ahead, but their real-time performance is poor, as shown in Table 4.

[0116] It can be seen that, combined with the relative attitude error, the SSF-SLAM system performs more stably in highly dynamic sequences. In the fr3 / s / static sequence, the performance of the SSF-SLAM system is always inferior to other dynamic SLAM systems. This is because low-dynamic sequences have achieved high positioning accuracy in dynamic SLAM, so there is limited room for improvement in SSF-SLAM. In summary, the system performs well in high-dynamic scenes and has only a few limitations in low-dynamic scenes.

[0117] B. Evaluation results on the Bonn RGB-D dataset

[0118] The Bonn RGB-D dynamic dataset is provided by the University of Bonn in Germany. Like the TUM dataset, it contains RGB images, depth images, and corresponding camera pose information from the Kinect sensor. The difference is that the Bonn RGB-D dynamic dataset is more complex than the TUM dataset and is therefore more challenging.

[0119] To further evaluate the robustness of the SSF-SLAM system, we selected 9 sequences from the Bonn dataset for evaluation and compared them with ORB-SLAM2, DS-SLAM, DynaSLAM, YOLO-SLAM, and SG-SLAM. Among them, the "crowd" sequence is a scene where three people walk randomly in a room. The "moving_no_box" sequence shows a scene where a person moves a box from the floor to the table and from the table to the floor. The "person_tracking" sequence is a scene where the camera tracks a walking person. The "synchronous" sequence is a scene where two people move synchronously along the epipolar direction.

[0120] Table 4 ATE results of SLAM algorithm on Bonn dataset [m]

[0121]

[0122] The evaluation and comparison results of 9 dynamic scene sequences are shown in Table 4. In the "crowd" and "synchronous" sequences, the DynaSLAM and YOLO-SLAM systems using more accurate deep learning networks have higher positioning accuracy, but the SSF-SLAM system is not far behind them, and is much better than DynaSLAM and YOLO-SLAM in terms of real-time performance. In the "moving_no_box" sequence, the accuracy of the SSF-SLAM system is slightly lower than that of SG-SLAM. This is because the scene flow clustering has errors in calculating the camera pose, which affects the accuracy of dynamic point judgment. In other sequences, the accuracy of the SSF-SLAM system is better than other algorithms, and the average ATE result of the SSF-SLAM system is 0.0307 and the maximum ATE result is 0.0371, which is much lower than other systems. This shows that the worst performance of SSF-SLAM is much better than other systems, that is, our method is more robust to dynamic environments.

[0123] C. Ablation Experiment

[0124] The SSF-SLAM system combines the advantages of target detection and geometric constraints to select and remove dynamic feature points. In order to verify the function of each module, we designed an ablation experiment. Among them, SSF-SLAM (S): is a dynamic feature selection strategy based on semantic information, SSF-SLAM (SF): is a dynamic feature selection strategy based on geometric information of scene flow clustering, and SSF-SLAM (S+SF) is a dynamic feature selection strategy that combines semantic information and geometric information. The experimental results are shown in Table 5.

[0125] Table 5 Comparison of ATE results of ablation experiments on TUM dataset [M]

[0126]

[0127] In the five sequences, the average RMSE of ATE of SSF-SLAM(S), SSF-SLAM(S+SF) and SSF-SLAM(S+SF) is 92.70%, 72.00% and 97.27% higher than that of ORB-SLAM2, and the average SD of ATE is 93.17%, 70.33% and 97.48% higher than that of ORB-SLAM2. SSF-SLAM(SF) performs poorly in high-dynamic sequences, but outperforms SSF-SLAM(S) in low-dynamic sequences. According to the discussion in this paper, in the absence of prior dynamic information provided by the target detection module, some dynamic features may be introduced into the scene flow calculation, thereby reducing the accuracy of the system. If the scene flow clustering module is not used, the feature points on passive moving objects and some active moving objects cannot be identified, resulting in a decrease in system accuracy. Ablation experiments show that combining semantic information with scene flow geometric information can further extract information from the scene, detect dynamic targets more accurately, and thus improve the accuracy of the system.

[0128] D. Run time analysis

[0129] In practical applications, real-time performance is an important indicator for evaluating SLAM systems. As shown in Table 6, we tested the average time cost of processing each frame when the system is running and compared it with other systems. As can be seen from Table VI, DynaSLAM, DS-SLAM, and RDS-SLAM still have a long average running time per frame even with GPU acceleration because they use pixel-level semantic segmentation. The target detection thread in the SSF-SLAM system runs independently of the entire system, which significantly improves the processing speed of each frame. In the absence of a GPU, it only increases by less than 4ms compared to ORB-SLAM2, enabling the system to run in real time.

[0130] Table 6 Real-time performance analysis of SSF-SLAM algorithm

[0131]

[0132] E. Semantic Mapping

[0133] To further evaluate the tracking accuracy and stability of our system, we constructed semantic maps on the TUM RGB-D dataset and the BonnRGB-D dataset. Figure 5These are the dense reconstruction results of the above two data sets. (a) is ORB-SLAM2 (TUM), (b) is SSF-SLAM (TUM), (c) is ORB-SLAM2 (Bonn), and (d) is SSF-SLAM (Bonn). In the TUM data set, the dense map obtained by ORB-SLAM2 is filled with the shadows of moving people. In the Bonn data set, since ORB-SLAM2 is affected by dynamic objects and cannot obtain the correct camera pose, it is impossible to correctly merge sub-maps and obtain an erroneous global map. The method proposed in this embodiment not only constructs a correct static map, but also provides semantic information, which facilitates the mobile robot to understand the scene and perform more advanced tasks.

[0134] In summary, this embodiment combines lightweight semantic information with scene flow geometric information. On the one hand, multi-threaded parallelism reduces the waiting time for obtaining semantic information. On the other hand, scene flow geometric information is clustered, overcoming the limitations of geometric constraints in complex scenes and reducing the impact of noise on dynamic feature judgment. A large number of experimental results show that the method provided by this embodiment significantly improves the real-time, accuracy and robustness in complex dynamic environments.

[0135] In addition, a semantic map is constructed by two-dimensional semantic information. Object detection and semantic mapping are integrated into a single thread, which runs in parallel with other threads to ensure the real-time performance of the system. This method is compared with other advanced dynamic SLAM methods in the TUM dataset and Bonn dataset. Experimental results verify the feasibility and effectiveness of this method.

[0136] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A positioning and mapping method based on semantic and scene flow geometric information, characterized in that: The following steps are involved: Get RGB image and depth image; Performing target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects; Matching feature points of two consecutive frames in the RGB image, and according to the dynamic semantic information of the previous frame, matching points corresponding to the current frame as prior dynamic points, and removing them; Calculate the basic matrix between two frames after removing the prior dynamic points, and calculate the essential matrix based on the camera intrinsic parameters. According to the essential matrix, obtain the camera extrinsic parameters, determine the camera pose, and calculate the scene flow of feature points in combination with the depth information to perform scene flow clustering; Combine the dynamic points obtained by the current frame target detection and the points in the geometric dynamic area obtained by scene flow clustering to select the final dynamic feature points and obtain the static feature points; Select a new key frame and perform camera pose prediction based on the static feature points; Based on the depth information, predicted camera pose and feature points of key frames, the point cloud information is determined, the semantic information is converted into three-dimensional semantic information, and the similarity matching between the point cloud information and the three-dimensional semantic information is performed to obtain the final semantic map.

2. A positioning and mapping method based on semantics and scene flow geometric information as claimed in claim 1, characterized in that: The process of matching the feature points of two consecutive frames in the RGB image is to match the feature points of the current frame with the previous frame through the LK optical flow method, and then combine the depth information to obtain the points of the previous frame that are in the prior dynamic frame and whose depth information is similar to the depth information of the midpoints in the prior dynamic frame, and match these points to the current frame as the prior dynamic points of the current frame for elimination.

3. The method for positioning and mapping based on semantics and scene flow geometric information as claimed in claim 1, characterized in that: The process of calculating the basic matrix between two frames after removing the prior dynamic points and calculating the essential matrix according to the camera intrinsic parameters includes removing the dynamic points of the previous frame and the points corresponding to the matched current frame according to the dynamic semantic information of the previous frame, obtaining the basic matrix, and calculating the essential matrix according to the basic matrix: E=K T ·F·K Among them, K is the camera intrinsic parameter matrix and F is the basic matrix.

4. A positioning and mapping method based on semantics and scene flow geometric information as claimed in claim 3, characterized in that: The process of obtaining the camera extrinsic parameters according to the essential matrix includes: performing singular value decomposition on the essential matrix E to obtain 4 sets of camera extrinsic parameters R, t, comparing the 4 sets of R, t to restore the legal number of spatial points, and selecting the R, t with the largest legal number as the camera extrinsic parameters.

5. The positioning and mapping method based on semantics and scene flow geometric information as claimed in claim 1, characterized in that: The process of scene flow clustering includes: first performing deep clustering, calculating the distance between points based on deep clustering, establishing connecting lines based on the distances, comparing the similarities of the scene flows at both ends of the connecting lines, and further clustering them; and grouping points with similar distances and consistent movement characteristics to determine their motion states.

6. A positioning and mapping method based on semantics and scene flow geometric information as claimed in claim 5, characterized in that: The process of deep clustering includes: obtaining the dynamic points of the current frame through matching according to the prior dynamic points detected in the previous frame, and removing them to obtain the point set, traversing all the points in the point set, and judging whether the depth of the current feature point is less than the average depth of the existing different depth categories through the depth threshold, so as to decide whether a new cluster needs to be established.

7. The method for positioning and mapping based on semantics and scene flow geometric information as claimed in claim 5, characterized in that: A connection line is established based on the distance, and the similarity of the scene flows at both ends of the connection line is compared. The process of further clustering includes calculating the similarity of the scene flows between the feature points based on the distance between the feature points, and establishing a connection line based on the similarity of the scene flows, and the distance between the feature points uses the Euclidean distance; Based on connection line clustering, the points at both ends of the connection line are classified into one category, so that highly correlated feature point clusters can be detected according to the distance of feature points and scene flow geometric information.

8. The positioning and mapping method based on semantics and scene flow geometric information as claimed in claim 1, characterized in that: The process of scene flow clustering to obtain geometric dynamic areas includes: multiplying the scene flow by the inverse of the depth; after eliminating a part of the dynamic points according to the prior dynamic information of the previous frame, calculating the average scene flow of all points, using the average scene flow to represent the scene flow of the static features of the current frame, and judging whether the category is a dynamic category based on the similarity between the average scene flow of each category and the scene flow of the static features of the current frame; if there is a dynamic scene flow category, it is considered that there are objects with passive motion or points on objects with active motion that have not been completely eliminated, and the convex hull algorithm is used to calculate the area containing all points of the category, and it is considered that the points in the area are all dynamic feature points.

9. A positioning and mapping system based on semantic and scene flow geometric information, characterized by: include: An image acquisition module, configured to acquire an RGB image and a depth image; The target detection module is configured to perform target detection on the RGB image to obtain semantic information, wherein the semantic information includes dynamic semantic information of the target object and semantic information of other objects; and obtain a specific semantic mask in combination with the depth image; An optical flow matching module is configured to match feature points of two consecutive frames in the RGB image, and according to the dynamic semantic information of the previous frame, the points corresponding to the current frame obtained by matching are used as prior dynamic points and are eliminated; The scene flow clustering module is configured to calculate the basic matrix between two frames after removing the prior dynamic points, and calculate the essential matrix according to the camera intrinsic parameters, obtain the camera extrinsic parameters according to the essential matrix, determine the camera pose, and calculate the scene flow of the feature points in combination with the depth information to perform scene flow clustering; The dynamic selection module is configured to select the final dynamic feature points by combining the dynamic points obtained by the target detection in the current frame and the points in the geometric dynamic area obtained by the scene flow clustering to obtain the static feature points; A pose prediction module is configured to select a new key frame and perform camera pose prediction in combination with the static feature points; The semantic mapping module is configured to determine the point cloud information based on the depth information, the predicted camera pose and the feature points of the key frames, convert the semantic information into three-dimensional semantic information, perform similarity matching between the point cloud information and the three-dimensional semantic information, and obtain the final semantic map.

10. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method according to any one of claims 1 to 8 are completed.

Images