Point-line fusion-based weak-texture dynamic scene visual slam optimization method and system

Abstract

The present application relates to the technical field of computer vision, in particular to a weak-texture dynamic scene visual SLAM optimization method and system based on point-line fusion, the method comprising the following steps: performing point feature extraction processing and line feature extraction processing on a to-be-processed image to obtain geometric information containing point features and line features; performing target object detection processing on the processed image using a preset model to obtain two-dimensional semantic information of the image; determining a dynamic object using an optical flow method based on the two-dimensional semantic information of the image; determining dynamic feature points on the dynamic object using a epipolar constraint method based on the two-dimensional semantic information of the image and the geometric information, and removing the dynamic feature points from the geometric information; constructing a map based on the removed geometric information, adding a line feature extraction method and a target detection thread to enable the algorithm to extract sufficient feature information in a weak-texture dynamic scene and remove the influence of the dynamic object, complete the positioning and mapping tasks, and improve the robustness of the SLAM algorithm.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for optimizing visual SLAM in weakly textured dynamic scenes based on point-line fusion. Background Technology

[0002] Currently, simultaneous localization and mapping is one of the basic capabilities of intelligent mobile robots to estimate the state in unknown environments. Mainstream SLAM systems are divided into two main categories based on the sensors used: LiDAR-based SLAM and camera-based SLAM. Most visual SLAM systems rely on static scene assumptions. When dynamic objects are present in the environment, the localization accuracy is affected by the movement of these objects, thus severely reducing the accuracy and robustness of SLAM systems in dynamic scenes.

[0003] Among the currently available technologies, there are many different implementations of SLAM algorithms based on point and line feature fusion. These methods can solve the localization and mapping needs in most weak textured scenes and significantly improve the accuracy of SLAM systems in weak textured environments. However, practical applications are affected by many factors. When there are dynamic objects in the environment, localization and mapping are easily affected by the dynamic objects, making it difficult to extract sufficient feature information.

[0004] Therefore, it is of great significance to study a visual SLAM optimization method and system for weakly textured dynamic scenes that can extract sufficient feature information and remove the influence of dynamic objects. Summary of the Invention

[0005] The purpose of this invention is to provide a visual SLAM optimization method and system for weakly textured dynamic scenes based on point-line fusion. By adding line feature extraction methods, dynamic object detection, and dynamic point detection and removal, it can extract sufficient feature information in weakly textured dynamic scenes and remove the influence of dynamic objects to complete the localization and mapping tasks, thereby improving the robustness of the SLAM optimization method.

[0006] To address the aforementioned technical problems, the first technical solution of this invention provides a visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion, comprising the following steps:

[0007] Point feature extraction and line feature extraction are performed on the image to be processed to obtain geometric information containing point and line features;

[0008] The image is processed using a preset model to detect target objects, thereby obtaining two-dimensional semantic information of the image.

[0009] Based on the two-dimensional semantic information of the image, dynamic objects are determined using optical flow.

[0010] Based on the two-dimensional semantic information and geometric information of the image, the dynamic feature points on the dynamic object are determined using the epipolar constraint method, and the dynamic feature points are removed from the geometric information;

[0011] A map is constructed based on the removed geometric information.

[0012] In some embodiments of the first technical solution, the step of performing line feature extraction processing on the image to be processed specifically includes:

[0013] Line segment features are extracted from the image to be processed to obtain the detection front line features;

[0014] The length of the pre-detection line features is detected, short line segments are filtered and suppressed to obtain the post-detection line segments;

[0015] Calculate the line band descriptor of the detected line segment;

[0016] The line features are obtained by using the line descriptor to determine whether the line segments extracted from adjacent frames match.

[0017] In some embodiments of the first technical solution, the step of using the line descriptor to determine whether line segments extracted from adjacent frame images match, and obtaining the line features, specifically involves:

[0018] Calculate the Euclidean distance between the line descriptors of each pair of line segments;

[0019] The nearest neighbor ratio is determined using the Euclidean distance.

[0020] The nearest neighbor distance ratio is compared with a preset ratio to determine the line segment matching and obtain the line feature.

[0021] In some embodiments of the first technical solution, the preset model is the MobileNet-SSD model. The step of using the preset model to perform target object detection processing on the processed image to obtain two-dimensional semantic information of the image specifically includes:

[0022] The feature map of the processed image is extracted using a preset model;

[0023] The feature map is subjected to convolutional pooling to generate target detection information;

[0024] Based on the target detection information, two-dimensional semantic information of the image is determined according to the target information, wherein the target information includes the presence of a target, the category of the present target, and the location of the present target.

[0025] In some embodiments of the first technical solution, the step of determining dynamic objects using optical flow based on the two-dimensional semantic information of the image specifically includes:

[0026] Based on the two-dimensional semantic information of the image, the optical flow of the previous frame and the corresponding optical flow of the current frame are calculated, and the optical flow includes key points;

[0027] Calculate the difference between the key points in the previous frame and the key points in the current frame to obtain the offset distance;

[0028] The offset distance is compared with a first preset distance threshold to determine the dynamic object.

[0029] In some embodiments of the first technical solution, the step of determining the dynamic feature points on the dynamic object using the epipolar constraint method based on the two-dimensional semantic information and the geometric information of the image specifically includes:

[0030] Based on the two-dimensional semantic information and geometric information of the image, the point information observed in the previous frame and the current frame is determined;

[0031] Based on the observed point information and fundamental matrix, the epipolar line in the current frame is determined.

[0032] Based on the epipolar lines in the current frame, the observed point information, and the fundamental matrix, the epipolar line constraints are determined.

[0033] The epipolar distance between the points observed in the previous frame and the current frame is calculated based on the epipolar constraint.

[0034] The epipolar distance is compared with a second preset distance threshold to determine the dynamic feature points on the dynamic object.

[0035] In some embodiments of the first technical solution, the expression for the epipolar constraint is:

[0036]

[0037] In the above formula, P1 represents the information of the points observed by the camera in the previous frame, P2 represents the information of the points observed by the camera in the current frame, L2 represents the epipolar line of the current frame image, and F represents the fundamental matrix.

[0038] In some embodiments of the first technical solution, after removing the dynamic feature points from the geometric information and before constructing a map based on the removed geometric information, the following steps are further included:

[0039] The first determination rule determines the creation of a new keyframe;

[0040] The newly created keyframes are transmitted to the local mapping thread for local BA optimization. Line feature optimization information is added to the original point feature BA optimization, point and line features are optimized synchronously, and redundant keyframes are removed.

[0041] The first determination rule is: the number of frames between the current frame and the previous keyframe exceeds a preset number, and after removing dynamic feature points from the current frame, the number of remaining point features and line features exceeds a preset number threshold.

[0042] In some embodiments of the first technical solution, after removing redundant keyframes and before constructing a map based on the removed geometric information, the following steps are also included:

[0043] Using the bag-of-words model, loop closure detection is performed between the current image frame and the previous image frame, and the loop closure detection results are transmitted to the backend for optimization.

[0044] The second technical solution provides a visual SLAM optimization system for weakly textured dynamic scenes based on point-line fusion, which applies the visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion described in the first technical solution, including:

[0045] The extraction module is used to perform point feature extraction and line feature extraction on the image to be processed, so as to obtain geometric information containing point features and line features;

[0046] The target detection module is used to perform target object detection processing on the processed image using a preset model to obtain two-dimensional semantic information of the image;

[0047] The dynamic detection module is used to determine dynamic objects based on the two-dimensional semantic information of the image using optical flow.

[0048] The elimination module is used to determine the dynamic feature points on the dynamic object based on the two-dimensional semantic information and the geometric information of the image using the epipolar constraint method, and to eliminate the dynamic feature points from the geometric information;

[0049] A construction module is used to build a map based on the removed geometric information.

[0050] The beneficial effects of this invention are as follows:

[0051] This scheme performs point and line feature extraction on the image to be processed, obtaining geometric information containing both point and line features. It then uses a pre-defined MobileNet-SSD model to detect target objects in the image, obtaining two-dimensional semantic information. Based on this two-dimensional semantic information, optical flow is used to determine dynamic objects. Finally, based on the two-dimensional semantic and geometric information, epipolar constraint is used to determine dynamic feature points on the dynamic objects, and these dynamic feature points are removed from the geometric information. A map is then constructed based on the removed geometric information. Because this scheme extracts both point and line features, it enriches the feature information for the SLAM algorithm, solving the problem of insufficient feature information extraction in weakly textured dynamic scenes. Furthermore, since the method detects dynamic targets and removes dynamic feature points, the algorithm can eliminate the influence of dynamic objects, completing the localization and mapping tasks and improving the robustness of the SLAM algorithm. Attached Figure Description

[0052] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the overall process flow provided by a preferred embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the overall process flow structure provided by a preferred embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the polar constraint principle combining geometric methods provided by a preferred embodiment of the present invention;

[0056] Figure 4 These are the running process diagram and absolute trajectory diagram under the first sequence test provided by the preferred embodiment of the present invention;

[0057] Figure 5 These are the running process diagram and absolute trajectory diagram under the second sequence test provided by the preferred embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the system structure provided by a preferred embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0060] To address the aforementioned issues, this solution provides a visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion. Please refer to [link / reference]. Figure 1 and Figure 2 This includes the following steps:

[0061] S1 performs point feature extraction and line feature extraction on the image to be processed, obtaining geometric information containing point and line features.

[0062] Preferably, step S1 specifically includes the following steps:

[0063] S10: Read the image sequence, use the ORB feature point extraction method to extract feature points of the image to be processed, obtain point features, and perform point feature tracking and matching.

[0064] For example, an ORB-based feature extraction method is used, combined with matching methods such as the FAST keypoint detector and BRIEF descriptors to complete the matching of feature points, thereby realizing point feature extraction, tracking and matching.

[0065] S11, perform line feature extraction on the image to be processed to obtain line features;

[0066] Specifically, step S11 includes the following steps:

[0067] S110: Extract line segment features from the image to be processed to obtain the detection front line features.

[0068] For example, the Fast Line Detector (FLD) algorithm is used to extract line features and obtain the detected line features.

[0069] S111: Length detection is performed on the features before detection, short segments are filtered and suppressed to obtain the detected line segments. Since many excessively short line segments are extracted in the initial extraction of line segments, it would increase the computational load of matching if all line segments were used for feature matching. Therefore, after filtering and suppressing short line segments in step S111, these line segments can be suppressed, and excessively short lines are not used for subsequent line segment matching, thereby reducing the computational load in the matching stage.

[0070] For example, after FLD extracts line segments, a length detection is performed on the line segments. Line segments with a length lower than a set threshold are removed from the stored results and the line segment information is updated. The line segment threshold is set based on the image size. Assuming the input image is 640*480 pixels, the diagonal of the image is approximately 800 pixels. Line segments with a length less than 1 / 32 of the diagonal pixels are defined as too short lines. These line segments are suppressed, and they are not used for subsequent line segment matching, thereby reducing the amount of computation in the matching stage.

[0071] S112, Calculate the line band descriptor of the detected line segment.

[0072] S113, use the line band descriptor to determine whether the line segments extracted from adjacent frame images match, and obtain line features. Step S113 uses the LBD line band descriptor to perform line segment matching between image frames, find matching line segments between connected image frames, and thus determine whether the extracted line segments are accurate, and retain the accurate line segments as feature information for system positioning.

[0073] Specifically, step S113 includes the following steps:

[0074] S1130, calculate the Euclidean distance between the line band descriptors of each pair of line segments;

[0075] S1131, the nearest neighbor distance ratio is determined using Euclidean distance, where the nearest neighbor distance ratio (NNDR) is used as the criterion for determining the final match, which is the ratio of the minimum Euclidean distance to the second minimum Euclidean distance.

[0076] S1132, compare the nearest neighbor distance ratio with a preset ratio to determine the line segment match and obtain the line feature. Determining the line segment match means that if the ratio is less than the preset ratio, then the match corresponding to the minimum distance is considered to be correct, and the line segment is taken as the required line feature.

[0077] The preset ratio is generally between 0.6 and 0.8.

[0078] In summary, step S1 incorporates a feature extraction method that combines point and line feature fusion, enriching the feature information for the SLAM algorithm and addressing the problem of extracting sufficient feature information in dynamic scenes with weak textures.

[0079] S2, using a preset model to perform target object detection processing on the processed image to obtain two-dimensional semantic information of the image;

[0080] The preferred preset model is the MobileNet-SSD model, which uses the MNN framework to integrate the MobileNet-SSD model for object detection. The MobileNet-SSD model is used to detect common dynamic scene images, defining twenty types of objects to be detected, with humans being the primary dynamic object considered. When a human is detected, it is stored as a priori target, and subsequently combined with geometric information from the dynamic elimination algorithm to determine whether it is a true dynamic feature.

[0081] For example, in a weakly textured dynamic environment, assuming that "human" is the most likely dynamic object, the MobileNet-SSD model detects dynamic objects on the image and stores the corresponding dynamic information after MNN inference to help confirm the dynamic objects in the future.

[0082] Preferably, step S2 specifically includes the following steps:

[0083] S20 uses the preset MobileNet-SSD model to extract feature maps from the processed image.

[0084] For example, the MNN framework is a lightweight neural network inference framework that can be used with only a small amount of resources. The MobileNet-SSD model uses MobileNet as a feature extractor to extract feature maps from the input image.

[0085] S21, perform convolutional pooling on the feature map to generate object detection information.

[0086] S22, Based on the object detection information, determine the two-dimensional semantic information of the image according to the object information, wherein the object information includes the presence of an object, the category of the object, and the location of the object.

[0087] For example, by using model prediction, the presence of a target within each anchor box, as well as the target's category and location, are predicted to obtain two-dimensional semantic information of the image.

[0088] S3, based on the two-dimensional semantic information of the image, uses optical flow to determine dynamic objects, that is, to estimate the motion of the object based on the motion of pixels in consecutive frames. When the optical flow method tracks the position of the object and determines that it is a dynamic object, it is marked as a dynamic object.

[0089] Preferably, step S3 specifically includes the following steps:

[0090] S30, based on the two-dimensional semantic information of the image, calculate the optical flow of the previous frame and the corresponding optical flow of the current frame. The optical flow includes key points.

[0091] S31, calculate the difference between the key points in the previous frame and the key points in the current frame to obtain the offset distance; set the key point Pt i The distance between (i = 2, 2') and the corresponding epipolar line is set as the offset distance d. i , then d i It can be represented as:

[0092]

[0093] In the above formula, d i This is the offset distance. Let F be the key point of the current frame, F be the basic matrix, Pt1 be the key point of the previous frame, X be the first line vector, and Y be the second line vector.

[0094] For example, if point P is a static point, combining the previous equation, the offset distance of feature point P2 can be obtained as:

[0095]

[0096] S32, compare the offset distance with the first preset distance threshold to determine the dynamic object.

[0097] The preset ratio is generally 0.2.

[0098] It should be noted that, ideally, the key points of the current frame will fall on the epipolar line, but in reality, there will be noise, and the offset distance will be greater than zero. Therefore, an empirical threshold needs to be set. If the offset distance of an object is large, the object is considered to be moving.

[0099] S4. Based on the two-dimensional semantic and geometric information of the image, the epipolar constraint method is used to determine the dynamic feature points on the dynamic object and remove the dynamic feature points from the geometric information.

[0100] Preferably, step S4 specifically includes the following steps:

[0101] S40, based on the two-dimensional semantic and geometric information of the image, determine the point information observed in the previous frame and the current frame;

[0102] For example, please refer to Figure 3 O1 and O2 are the optical centers of the camera, P1 = [x1, y1, 1] and P2 = [x2, y2, 1] are the information of the points observed by the camera in the previous frame and the current frame, respectively, and the dashed lines L1 and L2 are the epipolar lines of the two frames of images.

[0103] S41, Based on the observed point information and the fundamental matrix, determine the epipolar line L2 in the current frame:

[0104]

[0105] In the above formula, L2 is the epipolar line in the current frame, F is the fundamental matrix, P1 is the information of the point observed by the camera in the previous frame, X is the first line vector, Y is the second line vector, Z is the third line vector, and x and y represent the coordinate values ​​of the feature points in the image pixel coordinates, respectively.

[0106] The fundamental matrix is ​​a 3x3 matrix used to describe the epipolar geometry between two camera views. It is calculated from a set of matched points, and the fundamental matrix is ​​calculated using functions provided by OpenCV.

[0107] S42, based on the epipolar lines in the current frame, the observed point information, and the fundamental matrix, determine the epipolar constraints. The expression for the epipolar constraints is:

[0108]

[0109] In the above formula, P1 represents the information of the points observed by the camera in the previous frame, P2 represents the information of the points observed by the camera in the current frame, L2 represents the epipolar line of the current frame image, and F represents the fundamental matrix.

[0110] S43, calculate the epipolar distance between points observed in the previous frame and the current frame based on epipolar constraints.

[0111] S44, compare the epipolar distance with the second preset distance threshold to determine the dynamic feature points on the dynamic object.

[0112] The second preset distance threshold is 0.2.

[0113] For example, the epipolar distance between the keypoint and the optical flow point is calculated. The calculated epipolar distance is the distance between the position of the observed point in the previous frame and the position of the observed point in the current frame predicted by optical flow. If the epipolar distance is greater than a set empirical threshold, then the keypoint is considered dynamic and should be removed.

[0114] In summary, steps S3, S4, and S5 are as follows: In step S3, the MNN inference engine is used to infer the MobileNet-SSD model and perform prior object detection in the image. Then, in step S4, geometric information is combined to determine the dynamic objects present in the prior detection in the environment. Finally, in step S5, the epipolar constraint method is used to determine the dynamic feature points to be removed. Through the above steps S3, S4, and S5, the impact of dynamic objects on the SLAM system can be effectively reduced, and the accuracy and robustness of localization and mapping can be improved.

[0115] S5, after removing dynamic points, proceed to tracking: After removing dynamic feature points, use the remaining point features and line features to determine the creation of a new keyframe according to the first judgment rule, and calculate the camera pose at the same time.

[0116] The first judgment rule is: the number of frames between the current frame and the previous keyframe exceeds a preset number, and after removing dynamic feature points in the current frame, the number of remaining point features and line features exceeds a preset number threshold. The preset number of frames is 5 frames.

[0117] It should be understood that the first keyframe is initialized when the algorithm starts executing, and subsequent keyframes are newly added based on whether the features extracted during subsequent tracking meet the conditions.

[0118] S6, Local Mapping: The newly created keyframes are transmitted to the local mapping thread for local BA optimization. Line feature optimization information is added to the original point feature BA optimization, and point and line features are optimized synchronously. Redundant keyframes are also removed. After adopting this setting method, the synchronous optimization of point features and line features enriches the feature information of the SLAM algorithm and solves the problem of difficulty in extracting sufficient feature information in weak texture dynamic scenes.

[0119] S7 utilizes the Bag-of-Words (BoW) model to perform loop closure detection between the current image frame and the previous image frame. The loop closure detection results are then transmitted to the backend for optimization. The loop closure detection thread here actually uses the loop closure detection part of ORB-SLAM2. This setting effectively eliminates accumulated errors and improves the accuracy of the SLAM algorithm.

[0120] Preferably, step S7 specifically includes the following steps:

[0121] S70, calculate the word vector similarity between the current keyframe and the co-view keyframe, and use the minimum value as the threshold for subsequent loop closure candidate frame selection.

[0122] S71. Based on all the word features in the current frame, search the bag of words for which past keyframes have these features. Then, count the number of similar words between the current frame and the selected keyframes. The largest number is denoted as M. All keyframes with a value greater than 0.8M are selected as the first generation of loop closure candidate frames.

[0123] S72, calculate the word vector similarity between the first-generation loop candidate frames and the current frame, and select frames with similarity higher than the threshold in the first step as the second-generation loop candidate frames.

[0124] S73, for all second-generation loop closure candidate frames, calculate the matching score sum between their common-view keyframes and the current frame, take the maximum value of these sums as L, and then take the second-generation loop closure candidate frames whose sum is greater than 0.75L as third-generation loop closure candidate frames.

[0125] S74. Based on the obtained third-generation loop closure candidate frames, traverse these candidate frames. During traversal, for the current candidate frame, perform a geometric consistency check with the current frame and apply this transformation to the map points of the current frame. Then, calculate the matching degree between the transformed map points and the map points of the candidate frames. If there are enough matching map points and their spatial distribution satisfies certain geometric constraints, the loop closure candidate frame is considered consistent. If the consistency check passes, a loop closure is detected. Then, the candidate frame and all its associated frames are placed in a group, and the score of the current frame is incremented by 1. When another loop closure candidate frame comes in, the same geometric consistency check is performed. If a loop closure is detected, the current candidate frame is compared with the frames in the previous group. If there are any identical frames, the score of the current frame is incremented by 1. This process continues until all candidate frames have been traversed. If the score of the current frame equals 3, then the loop closure is successful.

[0126] S8 constructs a map based on the removed geometric information.

[0127] In dynamic object detection, once a region containing a defined dynamic object is detected, this region is labeled and identified as a dynamic feature region. Therefore, when constructing a point cloud from the depth image, the point cloud inside the bounding box of the dynamic object during the object detection process is filtered out. This removes the dynamic object from the final point cloud map, thus constructing the map.

[0128] The effectiveness of this solution can be seen in [reference needed]. Figure 4 and Figure 5 , Figure 4 (a) is a flowchart of the test run for the first sequence of the TUM dataset. Figure 4 (b) is the absolute trajectory (APE) plot of the first sequence in the TUM dataset under test. Figure 5 (a) is a flowchart of the test run for the second sequence of the TUM dataset. Figure 5 (b) is the absolute trajectory (APE) plot of the second sequence in the TUM dataset under test.

[0129] in Figure 4 (a) and Figure 4 (b) The fr3-structure-notexture-far sequence of the TUM dataset is a weak texture scene. It can be seen that the algorithm of the present invention can enrich the feature information by extracting line features and successfully achieve tracking and matching. Figure 5 (a) and Figure 5(b) The TUM dataset fr3-walking-xyz sequence provides rich dynamic object motion processes. The algorithm of this invention removes dynamic feature points through dynamic object detection, reducing the impact of dynamic objects. As can be seen from the APE image, the camera trajectory of the algorithm of this invention during operation highly coincides with the actual trajectory provided by the dataset, proving that the algorithm of this invention can improve the localization accuracy of weak texture dynamic scenes.

[0130] As can be seen from the above, the basic steps and principles of the method proposed in this scheme will be explained in the following text in conjunction with a system using the above method.

[0131] The point-line fusion-based visual SLAM optimization system for weakly textured dynamic scenes applies the aforementioned point-line fusion-based visual SLAM optimization method for weakly textured dynamic scenes. Please refer to [link / reference]. Figure 6 ,include:

[0132] The extraction module 900 is used to perform point feature extraction and line feature extraction on the image to be processed, so as to obtain geometric information containing point features and line features;

[0133] The target detection module 901 is used to perform target object detection processing on the processed image using a preset model to obtain two-dimensional semantic information of the image;

[0134] The dynamic detection module 902 is used to determine dynamic objects based on two-dimensional semantic information of images using optical flow.

[0135] The elimination module 903 is used to determine the dynamic feature points on the dynamic object based on the two-dimensional semantic and geometric information of the image using the epipolar constraint method, and to eliminate the dynamic feature points from the geometric information;

[0136] Module 904 is used to build a map based on the removed geometric information.

[0137] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion, characterized in that, Includes the following steps: The image to be processed undergoes point feature extraction and line feature extraction to obtain geometric information containing both point and line features; The image is processed using a preset model to detect target objects, thereby obtaining two-dimensional semantic information of the image. Based on the two-dimensional semantic information of the image, dynamic objects are determined using optical flow. Based on the two-dimensional semantic information and geometric information of the image, the dynamic feature points on the dynamic object are determined using the epipolar constraint method, and the dynamic feature points are removed from the geometric information; A map is constructed based on the removed geometric information; In the step of extracting line features from the image to be processed, the specific steps are as follows: Line segment features are extracted from the image to be processed to obtain the detection front line features; The length of the pre-detection line features is detected, short line segments are filtered and suppressed to obtain the post-detection line segments; Calculate the line band descriptor of the detected line segment; The line features are obtained by using the line descriptor to determine whether the line segments extracted from adjacent frames match. In the step of determining the dynamic feature points on the dynamic object using the epipolar constraint method based on the two-dimensional semantic information and the geometric information of the image, the specific steps are as follows: Based on the two-dimensional semantic information and geometric information of the image, the point information observed in the previous frame and the current frame is determined; Based on the observed point information and fundamental matrix, the epipolar line in the current frame is determined. Based on the epipolar lines in the current frame, the observed point information, and the fundamental matrix, the epipolar line constraints are determined. The epipolar distance between the points observed in the previous frame and the current frame is calculated based on the epipolar constraint. The epipolar distance is compared with a second preset distance threshold to determine the dynamic feature points on the dynamic object.

2. The visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion according to claim 1, characterized in that, In the step of using the line descriptor to determine whether line segments extracted from adjacent frame images match, and obtaining the line features, the specific steps are as follows: Calculate the Euclidean distance between the line descriptors of each pair of line segments; The nearest neighbor ratio is determined using the Euclidean distance. The nearest neighbor distance ratio is compared with a preset ratio to determine the line segment matching and obtain the line feature.

3. The visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion according to claim 1, characterized in that, The preset model is the MobileNet-SSD model. The step of using the preset model to perform target object detection processing on the processed image to obtain two-dimensional semantic information specifically involves: The feature map of the processed image is extracted using a preset model; The feature map is subjected to convolutional pooling to generate target detection information; Based on the target detection information, two-dimensional semantic information of the image is determined according to the target information, wherein the target information includes the presence of a target, the category of the present target, and the location of the present target.

4. The visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion according to claim 1, characterized in that, In the step of determining dynamic objects using optical flow based on the two-dimensional semantic information of the image, the specific steps are as follows: Based on the two-dimensional semantic information of the image, the optical flow of the previous frame and the corresponding optical flow of the current frame are calculated, and the optical flow includes key points; Calculate the difference between the key points in the previous frame and the key points in the current frame to obtain the offset distance; The offset distance is compared with a first preset distance threshold to determine the dynamic object.

5. The visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion according to claim 1, characterized in that, The expression for the polar constraint is: In the above formula, This refers to the information of the points observed by the camera in the previous frame. This refers to the information of the points observed by the camera in the current frame. The epipolar line of the current frame image. This is the basic matrix.

6. The visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion according to claim 1, characterized in that, After removing the dynamic feature points from the geometric information, and before constructing a map based on the removed geometric information, the following steps are also included: The first determination rule determines the creation of a new keyframe; The newly created keyframes are transmitted to the local mapping thread for local BA optimization. Line feature optimization information is added to the original point feature BA optimization, point and line features are optimized synchronously, and redundant keyframes are removed. The first determination rule is: the number of frames between the current frame and the previous keyframe exceeds a preset number, and after removing dynamic feature points from the current frame, the number of remaining point features and line features exceeds a preset number threshold.

7. The visual SLAM optimization method for weakly textured dynamic scenes based on point-line fusion according to claim 6, characterized in that, After removing redundant keyframes and before constructing a map based on the removed geometric information, the following steps are also included: Using the bag-of-words model, loop closure detection is performed between the current image frame and the previous image frame, and the loop closure detection results are transmitted to the backend for optimization.

8. A visual SLAM optimization system for weakly textured dynamic scenes based on point-line fusion, characterized in that, The method for optimizing weakly textured dynamic scenes using point-line fusion based on any one of claims 1 to 7 includes: The extraction module is used to perform point feature extraction and line feature extraction on the image to be processed, so as to obtain geometric information containing point features and line features; The target detection module is used to perform target object detection processing on the processed image using a preset model to obtain two-dimensional semantic information of the image; The dynamic detection module is used to determine dynamic objects based on the two-dimensional semantic information of the image using optical flow. The elimination module is used to determine the dynamic feature points on the dynamic object based on the two-dimensional semantic information and the geometric information of the image using the epipolar constraint method, and to eliminate the dynamic feature points from the geometric information; A construction module is used to build a map based on the removed geometric information.

Images