Dynamic scene vision SLAM method and system based on point-line complementation

Abstract

The invention relates to a dynamic scene vision SLAM method and system based on point-line complementation, and belongs to the field of simultaneous mapping and positioning of mobile robots. According to the method, target prior information monitoring is carried out on an image through a semantic module, and dynamic feature points inconsistent with camera motion are identified; visual feature extraction is carried out on the image through a tracking module, and the visual feature extraction comprises point feature extraction and line feature extraction; removing the point features preliminarily extracted by the tracking module according to the dynamic feature points identified by the semantic module, fusing the obtained static point features and line features, expanding the static point features according to the fused line feature constraints, and screening out key frames according to the point and line features; and finally, completing map construction and optimization updating through a local mapping module and a loopback detection and global BA optimization module. According to the method, mapping limitation and deviation possibly caused by single scene texture information can be overcome, and the real-time performance, the mapping precision and the robustness of mapping can be improved.

Description

Technical Field

[0001] The invention belongs to the field of simultaneous mapping and positioning of mobile robots, and relates to a dynamic scene visual SLAM method and system based on point-line complementarity. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is a key technology in the field of mobile robots. Stable and reliable scene map representation is the basis for mobile robots to provide subsequent services. Due to the advantages of low power consumption and easy data acquisition of visual sensors, the research methods of SLAM systems based on visual information have always been a hot topic in the field of computer vision and robotics. In the context of a new round of scientific and technological revolution and industrial transformation, relevant policies and strategic planning have provided important guidance for robot intelligent environmental perception.

[0003] In recent years, the classical visual research methods with high influence in the academic and practical fields are mainly divided into three categories: feature point plus filtering method, feature point method plus optimization method and direct method plus optimization method. The filtering-based method relies on the feature points extracted from the image, and estimates the camera motion pose through filters such as the extended Kalman filter EKF, and performs mapping tasks based on the feature point method. This method can meet good real-time requirements under the condition of limited hardware performance. However, there are also some problems such as being difficult to run for a long time due to the accumulation of errors. The feature point + optimization method describes the feature points in the image, uses graph optimization technology, performs feature matching with sparse features as the core, constructs motion constraints, and performs global optimization to build maps. Among them, the representative ORB-SLAM method has achieved relatively successful results as a representative. Another method is the direct method plus optimization method. SVO proposes a semi-direct method visual odometer method. It cleverly combines the advantages of the feature point method and the direct method to realize a sparse direct method visual odometer. This method uses a deep filter to estimate the position of the feature points, effectively balancing the computational efficiency and accuracy.

[0004] However, most of the above-mentioned classic visual SLAM methods are based on the assumption of static scenes. In the context of more and more application scenarios that put forward higher requirements on the robustness and accuracy of SLAM systems, other dynamic objects such as people and cars in the scene are inconsistent with the scene based on static assumptions. When there are moving objects, dynamic targets will be mistaken for static feature points and participate in pose estimation and map construction, resulting in the introduction of incorrect constraint relationships in the system. This incorrect constraint will directly affect the global consistency of trajectory estimation and map, and then lead to pose estimation drift and map distortion, and even directly lead to the failure of system mapping in more serious extreme cases. On the other hand, rapid motion and in dynamic scenes will lead to a decrease in the number of reliable static feature points, and it is impossible to establish sufficient constraints between key frames, affecting the accuracy of map construction. Although there are corresponding research methods for dynamic scene filtering, considering that the coarse-grained dynamic target detection method has limited filtering effect, more accurate segmentation puts higher requirements on system real-time and computing power, so there are still certain challenges and limitations in the combination of accuracy and efficiency in dynamic scene feature removal.

[0005] In addition, point feature-based extraction is sensitive to changes in illumination. Point feature methods rely on extracting corners or other significant features from images (such as SIFT, ORB, SURF, etc.). However, in weakly textured scenes (such as smooth surfaces, flat walls, large areas of sky, etc.), the image gradient changes slightly, and corners are difficult to extract, resulting in a small number of extracted feature points. In this case, it is difficult for the point feature method to provide reliable geometric constraints, which seriously affects system performance. If the feature points are unevenly distributed in weakly textured scenes, the geometric constraints between the two frames are significantly weakened, resulting in a decrease in the positioning and mapping accuracy of the system, especially in scenes with unclear three-dimensional structures (such as single plane walls). The lack of geometric constraints may cause the system to diverge. Summary of the invention

[0006] In view of this, the object of the present invention is to provide a dynamic scene visual SLAM method and system based on point-line complementarity.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A dynamic scene visual SLAM method based on point-line complementarity, the method comprising the following steps:

[0009] S1, monitor the target prior information of the image through the semantic module, and identify the dynamic feature points that are inconsistent with the camera motion;

[0010] S2. Extracting visual features from the image through a tracking module, wherein the visual feature extraction includes point feature extraction and line feature extraction;

[0011] S3, the point features initially extracted by the tracking module are eliminated according to the dynamic feature points identified by the semantic module, the obtained static point features and line features are fused, and the static point features are expanded according to the fused line feature constraints, and then the key frames are selected according to the point and line features;

[0012] S4, using the local mapping module to process the selected key frames and construct a local map;

[0013] S5. Optimize and fuse the local map through loop detection and global BA optimization module to obtain the global map, and perform global BA optimization and map update.

[0014] Furthermore, in step S1, a target detection algorithm is used to detect target motion prior information on the collected image, wherein:

[0015] The image I at time t is divided into a potential dynamic object region D and a non-potential dynamic object region S, where D∩S=φ, D∪S=I. The consistency of camera motion is assumed. The corresponding matching point position of the next frame is calculated according to the camera motion posture transformation relationship of the previous frame. A deviation threshold is set. If the projection of the feature point in the image of the next frame and the camera projection calculated based on the static assumption deviate greater than the deviation threshold, the feature point is identified as a dynamic feature point.

[0016] Further, in step S2, an environmental RGB image and depth information are collected by an image acquisition device, and visual feature extraction processing is performed based on the collected image. The visual feature extraction includes:

[0017] S21, point feature extraction;

[0018] S2, line feature extraction;

[0019] In the point feature extraction process of step S21, the following steps are included:

[0020] S211: performing corresponding parameter calibration on the image to be processed, obtaining camera parameters, and extracting feature points of the image to be processed according to the ORB feature extraction algorithm;

[0021] S212: Based on the extracted feature points, the image area is equally divided according to the image size, a reference feature point in the area to be compared is selected by polling, a circular area method with a variable radius is used to dynamically search the reference feature point in the equally divided image area, a dynamic threshold is set to score the feature points, and the feature point with the highest score is selected as the representative feature point of the area;

[0022] S213: Screening is performed based on the feature points extracted after processing, and the feature point scores are calculated using a non-maximum suppression method combined with saliency detection, and the feature points with the highest scores are retained.

[0023] Further, in step S211, the feature point extraction process includes:

[0024] Select a point p in the image and assume that the gray value of the point is I p ;

[0025] Set a circular area with point p in the image as the center and several pixels as the radius r, and extract several pixel points on the circle;

[0026] Calculate the difference between the grayscale value of point p and the grayscale values ​​of several pixels in turn, take the absolute value of the difference, and set a threshold τ for statistics. If there are M consecutive pixels that satisfy the absolute value of the grayscale difference between the two pixels greater than the threshold τ, then point p is considered to be a fast corner point:

[0027] |I p -I i |>τ

[0028] The ORB algorithm is based on the fast corner detection method and uses the grayscale centroid method to obtain direction information. The direction information is based on the grayscale value of the image block as the weight core. The calculation process is as follows:

[0029] Define an image block S with a feature point as the center and a radius of r. The moment of the image is:

[0030]

[0031] Where I(x,y) represents the gray value corresponding to the image coordinate (x,y), c and d represent the exponents of calculating the image moment in the x direction and in the y direction;

[0032] The image moment along the x and y directions in the image block S is:

[0033]

[0034]

[0035] In the formula, m 10 and m 01 Represent the image moments of the image block in the x direction and in the y direction respectively;

[0036] The sum of the image grayscale values ​​of all pixels is The grayscale centroid of the image is C x ,C y Respectively represent the coordinates of the grayscale centroid of the image; the direction of the direction vector connecting the geometric center point of the image and the grayscale centroid of the image is the direction of the feature point, and the value of the direction is:

[0037]

[0038] The ORB algorithm uses improved BRIEF as the feature descriptor, and its process is as follows:

[0039] First, perform Gaussian filtering on the image, take a circular area with a radius of r and the key point as the center, randomly select a pixel in the circular area and compare it with the grayscale value of the current key point pixel, binarize the comparison result, repeat the evaluation randomly, and construct a 256-dimensional binary descriptor:

[0040]

[0041] Among them I p Represents the gray value of p in the pixel area, I i Represents the gray value of the i-th pixel to be calculated, σ i Represents the value of the i-th dimension of the binary descriptor.

[0042] Further, in step S212, a variable gradient radius method is used to search for feature points. The current radius r cur If the number of feature points m in the area is less than the feature point setting threshold n, the search radius is expanded; if the number of feature points m in the current radius area is greater than the feature point setting threshold, the search radius is contracted; the update speed of the search radius is proportional to the current feature point difference, and the calculation formula is as follows:

[0043] When the expanded search radius needs to be updated,

[0044]

[0045] When you need to update the search radius,

[0046]

[0047] Among them, m is the number of feature points to be compared in the selected area, n is the number of feature points currently to be searched, and r max is the maximum expansion radius of the dynamic area, r min is the minimum shrinkage radius of the dynamic area, and α is the set proportional coefficient, which is used to control the speed of radius change. In actual use, it is set using empirical values.

[0048] Further, in step S213, the global image saliency value is calculated and counted, the global image saliency value is calculated and counted, the LC detection algorithm is used to perform saliency detection on the entire image, the grayscale value of the current feature point and the grayscale interpolation of the surrounding pixels are calculated, and the feature points are evaluated from the global and local perspectives. The process is as follows:

[0049] First, convert the input RGB image into a grayscale image, then traverse the pixels in the global image, calculate the corresponding grayscale values ​​and perform statistics to obtain a grayscale value array. The value of the array element represents the frequency of occurrence of the grayscale value.

[0050] Then calculate the saliency value of a point in the pixel based on the Euclidean distance and the gray value frequency in the image:

[0051]

[0052] S(p) is the saliency value corresponding to the current pixel value, d(p,γ) is the Euclidean distance between the gray value of the current pixel and the gray value γ, and f γ is the frequency of gray value γ appearing in the image;

[0053] Assume that the median value of the global saliency feature value of the current image is S med , the obtained feature points satisfy the following conditions at the same time:

[0054]

[0055] Among them, P represents the set of feature points that meet the search radius, Score is the feature point with the highest score, is the gray value of the feature point, is the gray value of several pixels around the feature point, S i is the saliency value of the current pixel, λ and μ are the weighting coefficients of the gray value and saliency value of the local pixel, respectively.

[0056] Further, in the line feature extraction process of step S22, it includes the following steps:

[0057] S221, using the EDlines line feature extraction algorithm to detect line segments in the RGB image and extract short line segments;

[0058] S222, filtering the redundant short line segments extracted from the line features, selecting the maximum value of the extracted line segments as the line segment filtering base, and filtering the short line segments according to the minimum line segment length threshold by setting the length scale factor, and the filtering condition is:

[0059] L min =η*max(L i )

[0060] Where L i is the line segment detected in the current image. min is the minimum line segment threshold, and the line segments whose length is shorter than the minimum line segment threshold are filtered. η is an empirical value that can be pre-adjusted according to the actual scene;

[0061] S223: merging similar line segments, wherein the merging process is determined by multiple parameters, wherein the multiple parameters include the angle difference between the line segments to be processed after the short line segments are removed, the distance between the nearest neighbor vertices of the two line segments to be merged, and the distance between the midpoints of the two line segments;

[0062] For the two line segments l extracted from the two-dimensional image a and l b , calculate the angle error θ between the two line segments ab , the distance between two neighboring vertices Sd ab , and the distance Sm between the midpoints of each line segment ab ;

[0063] According to the two endpoints of the line segment, the two line segments are described in vector form. The absolute value of the angle difference between the two vectors represents the angle error θ of the two line segments. ab , so:

[0064]

[0065]

[0066] Where (x a ,y a ) and (x b ,y b ) are two line segments l a and l b The vertex of

[0067] The penalty function of adaptive length and distance is introduced to enhance the reliability of segment merging, and the angle threshold is adaptively adjusted according to the distance value. s is the distance threshold, τ θ is the angle threshold, let l a For a longer line segment:

[0068]

[0069] Therefore, the line segments for line feature merging satisfy the following constraints at the same time: θ ab Less than the set dynamic angle threshold; the distance between the nearest neighbor vertices Sd ab and the distance between the midpoints of the two line segments Sm ab The maximum value is less than the set threshold;

[0070] The merging of line segments includes the angle of the new line segment after merging and the selection of the vertex of the new line segment. The midpoint coordinates of the new line segment are set to (x new, y new ), and use the dynamic proportion method to calculate the midpoint of the new line segment:

[0071]

[0072] Where (x a ,y a ),(x b, y b ) are the coordinates of the midpoints of the two line segments, and the angle of the new line segment is θ new :

[0073]

[0074] Construct a straight line through the midpoint of the new line segment and the angle of the new line segment, and merge the l a and l b The vertices of the vertices are orthogonally projected onto the merged line to be determined. The two points on the line with the farthest distance are used as the endpoints of the new line segment, and the new line segment is connected to the longer line segment l before the merger. a The line segment constraint conditions before merging are subjected to regression testing. If the corresponding threshold conditions are not met, it is considered that the line segment merging has failed.

[0075] Furthermore, in step S3, the dynamic feature points in the obtained point feature set are firstly eliminated, and the point feature set after elimination is a static feature point set; then the static feature point set is matched with the line feature set, wherein improvement is made based on point feature optical flow matching, and reasoning is performed based on the grayscale invariant assumption:

[0076] I(u+du,v+dv,t+dt)=I(u,v,t)

[0077] The coordinate position of a point in space at time t is (x, y). At time t+dt, the point moves to (x+dx, y+dy), and the grayscale remains unchanged. Assuming that the pixel grayscale value remains constant in a short time, we have:

[0078]

[0079] in Indicates the gradient change of the gray value of the point in the x-axis direction. Indicates the gradient change of the gray value of the point in the y direction, respectively denoted as I x and I y , Represents the change value of grayscale, denoted as I t , It represents the speed in the x direction. The speed in the x direction is the same, recorded as u and v respectively, and the following equation is established:

[0080]

[0081] The above formula is a linear equation with two variables. Multiple additional equations about constraints are obtained in the entire image. Multiple points on the line segment also have the same motion. Thus, multiple overdetermined equations are established and solved by the least squares method.

[0082] In the above process, for the point-line feature fusion strategy, a method for compensating feature points of bisection line segments with variable direction is proposed, which includes the following steps:

[0083] a) When the number of static feature points in the current image area does not reach the minimum threshold and there are merged reliable line segments in the image area, the insufficient feature points are compensated by obtaining feature points between the line segments.

[0084] b) Perform binary fission on the line segment, that is, the first feature point is generated as the midpoint of the two line segments;

[0085] c) Second fission, select the midpoint of the generated line segment and the midpoint of one of the line segment endpoints;

[0086] d) The third fission, the midpoint of the newly generated line segment and the different line segment endpoints used in the previous fission, the midpoint of the line segment formed is used as the new fission point;

[0087] e) According to the above method, iteratively perform binary fission, and by continuously changing the reference endpoints of the binary fission line segment, effectively ensure the uniform distribution of the feature points after the split on the line segment until enough reliable new feature points are generated.

[0088] Further, in step S4, the keyframes are processed and mapped by the local mapping module. After the keyframes are determined, they are inserted into the local map and connected with other keyframes with common features in the map by edges. Then, the corresponding bag of words of the keyframe is calculated, and a new point cloud map is generated by triangulation. The map point cloud is screened to delete invalid map points and retain new map points. In addition, the local BA optimization is completed by the local mapping module, and the current keyframe, other keyframes connected to the current keyframe, and the observed point cloud are jointly optimized.

[0089] In step S5, the local map obtained is globally optimized through loop detection and global BA optimization modules, wherein candidate key frames are first screened according to the bag-of-words model of image recognition, and key frames that do not meet the set requirements are eliminated, and then the similarity transformation between the candidate current frame and the detected loop frame is calculated to obtain the cumulative error; then, the repeated point cloud maps are fused and the posture of the current frame is corrected in combination with the similarity transformation to achieve alignment of the two ends of the loop, and then the accumulated error of the loop detection is dispersed to the entire image through essential graph optimization; finally, the map is updated.

[0090] The present invention also proposes a system for executing the aforementioned dynamic scene visual SLAM method based on point-line complementarity, the system comprising a semantic module, a tracking module, a local map construction module, a loop detection and a global BA optimization module, wherein:

[0091] The semantic module monitors the target prior information of the image and identifies the dynamic feature points that are inconsistent with the camera motion;

[0092] The tracking module extracts visual features from the image, where visual feature extraction includes point feature extraction and line feature extraction; then the point features initially extracted by the tracking module are eliminated according to the dynamic feature points identified by the semantic module, the obtained static point features and line features are fused, and the static point features are expanded according to the fused line feature constraints, and then the key frames are selected according to the point and line features;

[0093] The local map construction module processes the filtered key frames and constructs local maps;

[0094] The loop detection thread and the global BA optimization module optimize and fuse the local maps to obtain the global map, and perform global BA optimization and map updates.

[0095] The beneficial effects of the present invention are:

[0096] (1) The present invention improves the traditional visual simultaneous mapping and positioning method based on static assumptions, combines the semantic target detection module to extract potential dynamic targets in the image from the perspective of deep learning and divide the potential dynamic target area, and then removes dynamic features based on prior dynamic target detection. This can effectively avoid the interference of dynamic features in the scene on the static scene map, and further improve the mapping and positioning accuracy and robustness in dynamic scenes.

[0097] (2) The present invention uses a method of area averaging and variable radius search for static feature points extracted from an image to ensure uniform distribution of feature points, improve the search speed of feature points, and provide fast and powerful support for subsequent feature matching and tracking.

[0098] (3) The present invention improves on the method based on straight line extraction, sets a dynamic threshold for eliminating short line segments, sets straight line merging conditions, and merges new line segments according to the line segment characteristics. The new line segments provide higher geometric constraints and more reliable features, which can effectively improve the robustness of the system in weak texture conditions and difficult feature point extraction situations.

[0099] (4) The present invention proposes a line segment splitting method with a diagonal binary division to address the problem that the number of static feature points in the image after dynamic feature point elimination may be insufficient or the feature points are unevenly distributed. By taking points on the merged line segments, feature points are used to participate in subsequent mapping and positioning tasks, which can effectively reduce the poor mapping accuracy caused by mismatching and the tracking failure caused by failure to match.

[0100] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0102] Figure 1 Schematic diagram of the overall architecture of the dynamic scene visual SLAM method and system based on point-line complementarity of the present invention;

[0103] Figure 2 It is a schematic diagram of the process of the dynamic scene visual SLAM method based on point-line complementarity of the present invention;

[0104] Figure 3 It is a schematic diagram of the characteristic point supplement process of the redirected bisection line segment fission method of the present invention. DETAILED DESCRIPTION

[0105] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0106] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0107] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0108] See also Figure 1 to Figure 3 , which is a dynamic scene visual SLAM method and system based on point-line complementarity.

[0109] Example

[0110] like Figure 1 The overall architecture schematic diagram of the dynamic scene visual SLAM method based on point-line complementarity of the present invention is shown. The present invention provides a dynamic scene visual SLAM method and system based on point-line complementarity.

[0111] This embodiment first provides a detailed implementation of a dynamic scene visual SLAM method based on point-line complementarity, such as Figure 2 The schematic diagram of the process of the dynamic scene visual SLAM method based on point-line complementarity shown in FIG. 1 includes the following steps:

[0112] S1, monitor the target prior information of the image through the semantic module, and identify the dynamic feature points that are inconsistent with the camera motion;

[0113] S2. Extracting visual features from the image through a tracking module, wherein the visual feature extraction includes point feature extraction and line feature extraction;

[0114] S3, the point features initially extracted by the tracking module are eliminated according to the dynamic feature points identified by the semantic module, the obtained static point features and line features are fused, and the static point features are expanded according to the fused line feature constraints, and then the key frames are selected according to the point and line features;

[0115] S4, using the local mapping module to process the selected key frames and construct a local map;

[0116] S5. Optimize and fuse the local map through loop detection and global BA optimization module to obtain the global map, and perform global BA optimization and map update.

[0117] In step S1 of this embodiment, a target detection algorithm is first used to detect target motion prior information for the captured image. The lightweight target detection algorithm aims to find a balance between detection accuracy and real-time performance. The dynamic points to be eliminated in the image are more likely to appear in the potential dynamic object area. Therefore, the dynamic targets in the scene are detected for the divided potential dynamic object area, and the coarse-grained dynamic feature points are extracted from the area with high confidence; the image I at time t is divided into a potential dynamic object area D and a non-potential dynamic object area S, where D∩S=φ, D∪S=I. Specifically, consider using the method provided in the present invention in a dynamic scene, and set the potential dynamic target motion objects to people and cars. The target detection method proposed in the document "An Overview of the Key Architectural Enhancements" is used to perform detection and area division on the image at time t.

[0118] By using the consistency assumption of camera motion, that is, if the feature points in the image are all static points, the corresponding matching point position of the next frame can be calculated from the camera motion posture transformation relationship of the previous frame. However, due to the movement of the dynamic point, the deviation between its projection in the image of the next frame and the camera projection calculated based on the static assumption is greater than the threshold. Therefore, the dynamic feature points that are inconsistent with the camera motion can be identified based on the threshold, and the corresponding identification is provided for subsequent processing.

[0119] In step S2 of this embodiment, an environmental RGB image and depth information are collected by an image acquisition device, and visual feature extraction is performed based on the collected image, wherein the visual feature extraction includes S21, point feature extraction and S2, line feature extraction. This embodiment uses the ORB feature point extraction method to implement feature point extraction, uses Edlines to implement line segment extraction in the image, and processes the features based on related methods to generate reliable feature point lines.

[0120] In the point feature extraction process of step S21, it includes the following key steps:

[0121] S211: performing corresponding parameter calibration on the image to be processed, obtaining camera parameters, and extracting feature points of the image to be processed according to the ORB feature extraction algorithm;

[0122] (1) Main process of FAST feature point extraction:

[0123] Select a point p in the image and assume that the gray value of the point is I p ;

[0124] Set the circular area with point p in the image as the center and 3 pixels as the radius, and extract 16 pixels on the circle;

[0125] Calculate the difference between the grayscale value of the point and the grayscale values ​​of 16 pixels in turn, take the absolute value of the difference, and set a threshold τ for statistics. If there are M consecutive pixels that satisfy the absolute value of the grayscale difference between the two pixels is greater than the threshold τ, then the point is identified as a fast corner point, where the value of M is generally 9, 12, or 15.

[0126] |I p -I i |>τ

[0127] ORB is based on the fast corner detection method, and uses the grayscale centroid method to obtain direction information to make up for the lack of scale and direction information of fast corners. The direction information is based on the grayscale value of the image block as the weight core, and its calculation process is as follows:

[0128] Define an image block S with a feature point as the center and a radius of r. The moment of the image is:

[0129]

[0130] Where I(x,y) represents the gray value corresponding to the image coordinate (x,y), c and d represent the exponents of calculating the image moment in the x direction and in the y direction;

[0131] The image moment along the x and y directions in the image block S is:

[0132]

[0133] In the formula, m 10 and m 01 Represent the image moments of the image block in the x direction and in the y direction respectively;

[0134] The sum of the image grayscale values ​​of all pixels:

[0135]

[0136] The grayscale centroid of the image is

[0137]

[0138] In the formula, C x ,C y Respectively represent the coordinates of the grayscale centroid of the image;

[0139] The direction of the direction vector connecting the geometric center point of the image and the grayscale centroid of the image is the direction of the feature point, and the value of the direction is

[0140]

[0141] (2) ORB uses improved BRIEF as the feature descriptor, and its specific process is as follows:

[0142] First, perform Gaussian filtering on the image to reduce noise interference.

[0143] Take a circular area with a radius of r and the key point as the center, randomly select a pixel in the circular area and compare it with the grayscale value of the current key point pixel, binarize the comparison result, repeat the evaluation randomly, and construct a 256-dimensional binary descriptor

[0144]

[0145] Among them I p Represents the gray value of p in the pixel area, I i Represents the gray value of the i-th pixel to be calculated, σ i Represents the value of the i-th dimension of the binary descriptor.

[0146] S212: Based on the feature points extracted in S211, the image area is equally divided according to the image size, a reference feature point in the area to be compared is selected by polling, a circular area method with a variable radius is used to dynamically search the reference feature point in the equally divided image area, a dynamic threshold is set to score the feature points, and the feature point with the highest score is selected as the representative feature point of the area; the number of feature points to be compared in the area to be selected is set to m, the number of feature points to be searched currently is set to n, and the maximum expansion radius of the dynamic area is set to r ma and the minimum shrinkage radius of the dynamic area is r min , use the variable gradient variable diameter method to search for feature points, the current radius r cur If the number of feature points m in the area is less than the feature point setting threshold n, the search radius is expanded. If the number of feature points m in the current radius area is greater than the feature point setting threshold, the search radius is contracted. The update speed of the search radius is proportional to the current feature point difference to improve the speed of searching for reliable feature points. The calculation formula is as follows:

[0147] When the expanded search radius needs to be updated,

[0148]

[0149] When you need to update the search radius,

[0150]

[0151] S213: Screening is performed based on the feature points extracted after processing, and a non-maximum suppression method combined with saliency detection is used to calculate the feature point scores, and the feature points with the highest scores are retained.

[0152] Calculation method: Calculate and count the global image saliency value, use the LC detection algorithm to perform saliency detection on the entire image, calculate the grayscale value of the current feature point and the grayscale interpolation of the surrounding pixels, and evaluate the feature points from a global and local perspective.

[0153] The image input is an RGB three-channel image. First, the RGB image is converted into a grayscale image. Then, the pixels in the global image are traversed, the corresponding grayscale values ​​are calculated and counted, and a grayscale value array is obtained. The length of the array is 256, that is, the grayscale value is divided into levels, and the value of the array element represents the frequency of occurrence of the grayscale value.

[0154] Based on this, the saliency value of a point in a pixel is calculated based on the Euclidean distance and the gray value frequency in the image.

[0155]

[0156] S(p) is the saliency value corresponding to the current pixel value, d(p,γ) is the Euclidean distance between the gray value of the current pixel and the gray value γ, and f γ is the frequency of gray value γ appearing in the image.

[0157] Suppose the median value of the global salient feature value of the current image is S med , the feature points obtained must meet the following conditions at the same time:

[0158]

[0159] Among them, P represents the set of feature points that meet the search radius, Score is the feature point with the highest score, is the gray value of the feature point, is the gray value of the 16 pixels around the feature point, S i is the saliency value of the current pixel, λ and μ are the weighting coefficients of the gray value and saliency value of the local pixel, respectively.

[0160] In the line feature extraction process of step S22, it includes the following key steps:

[0161] S221. Short line segment extraction: Use the EDlines line feature extraction algorithm to detect line segments in the image from the RGB image.

[0162] S222, short line segment elimination: Filter the redundant short line segments of the extracted line features, select the maximum value of the extracted line segments as the line segment filtering base, set the length scale factor, and filter the short line segments according to the minimum line segment length threshold. The filtering conditions are:

[0163] L min =η*max(Li )

[0164] Where L i is the line segment detected in the current image. min is the minimum line segment threshold. Line segments shorter than the threshold are discarded and do not participate in subsequent processing tasks. η is an empirical value that can be pre-adjusted according to the actual scenario, and its general value is 0.125.

[0165] S223: Merge similar line segments: To solve the problem that the detected line segments may be split and generate multiple similar line segments, similar line segments are merged. The merging process involves multiple parameters to participate in the decision-making. The multiple parameters are the angle difference between the line segments to be processed after the short line segments are removed, the distance between the nearest neighbor vertices of the two line segments to be merged, and the distance between the midpoints of the two line segments. a and l b , calculate the angle error θ between the two line segments ab , the distance between two neighboring vertices Sd ab , and the distance Sm between the midpoints of each line segment ab .

[0166] According to the two endpoints of the line segment, the two line segments can be described as vectors, and the absolute value of the angle difference between the two vectors can represent the angle error θ of the two line segments. ab , so

[0167]

[0168] Where (x a ,y a ) and (x b ,y b ) are two line segments l a and l b The vertex.

[0169] The penalty function of adaptive length and distance is introduced to enhance the reliability of segment merging, and the angle threshold is adaptively adjusted according to the distance value. s is the distance threshold, τ θ is the angle threshold, let l a For longer line segments.

[0170]

[0171] Therefore, the line segments for merging line features need to meet the following constraints at the same time:

[0172] θ ab Less than the set dynamic angle threshold;

[0173] The distance between the nearest neighbor vertices Sd aband the distance between the midpoints of the two line segments Sm ab The maximum value is less than the set threshold.

[0174] The merging of line segments mainly involves the angle of the new line segment after merging and the selection of the vertex of the new line segment.

[0175] The midpoint coordinates of the new line segment are set to (x new, y new ), considering that the longer straight line segment has a larger constraint ratio for the merged new line segment, the dynamic ratio method is used here to calculate the midpoint of the new line segment.

[0176]

[0177] Where (x a ,y a ),(x b, y b ) are the coordinates of the midpoints of the two line segments, and the angle of the new line segment is θ new

[0178]

[0179] Construct a straight line through the midpoint of the new line segment and the angle of the new line segment, and merge the l a and l b The vertices of the vertices are orthogonally projected onto the merged line to be determined. The two points on the line with the farthest distance are used as the endpoints of the new line segment, and the new line segment is connected to the longer line segment l before the merger. a The line segment constraint conditions before merging are subjected to regression testing. If the corresponding threshold conditions are not met, it is considered that the line segment merging has failed.

[0180] In step S3 of this embodiment, the dynamic feature points in the obtained point feature set are firstly removed, and the point feature set after removal is a static feature point set. Then the static feature point set is matched with the line feature set, specifically, improvement is made based on point feature optical flow matching, and reasoning is performed based on the grayscale invariant assumption.

[0181] l(u+du,v+dv,t+dt)=I(u,v,t)

[0182] The coordinate position of a point in space at time t is (x, y). At time t+dt, the point moves to (x+dx, y+dy). The grayscale remains unchanged. Assuming that the pixel grayscale value remains constant in a short time, the above formula is expanded by Tate series.

[0183]

[0184] Where R(x,y,t) is the Peano remainder, which can be approximated to zero. Combining the above equations, we can get

[0185]

[0186] Dividing both sides by dt, we get

[0187]

[0188] in Indicates the gradient change of the gray value of the point in the x-axis direction. Indicates the gradient change of the gray value of the point in the y direction, respectively denoted as I x and I y , Represents the change value of grayscale, denoted as I t , It represents the speed in the x direction. The speed in the x direction is the same, recorded as u and v respectively. The following equation can be established

[0189]

[0190] The above formula is a linear equation with two variables. In the entire image, multiple additional equations about constraints will be obtained. Multiple points on the line segment also have the same motion. Multiple equations can be established. It is an overdetermined equation and can be solved by the least squares method.

[0191] In the above process, for the point-line feature fusion strategy, a variable-direction bisection line segment feature point compensation method is proposed. After the dynamic feature points are eliminated, the static feature points may be unevenly distributed or insufficient in the image. Based on the line features, more stringent geometric constraints are provided. Each point on the merged line segment satisfies the grayscale invariance assumption and motion consistency assumption. If the static feature points in the current image area are insufficient, and there are valid line segments in the current image area, feature points can be generated according to the proposed variable-direction bisection line segment feature point compensation method to participate in subsequent matching calculations, such as Figure 3 As shown, the method for generating feature points mainly includes the following steps:

[0192] a) When the number of static feature points in the current image area does not reach the minimum threshold and there are merged reliable line segments in the image area, the insufficient feature points are compensated by obtaining feature points between the line segments.

[0193] b) Perform binary fission on the line segment, that is, the first feature point is generated as the midpoint of the two line segments;

[0194] c) Second fission, select the midpoint of the generated line segment and the midpoint of one of the line segment endpoints;

[0195] d) The third fission, the midpoint of the newly generated line segment and the different line segment endpoints used in the previous fission, the midpoint of the line segment formed is used as the new fission point;

[0196] e) According to the above method, iteratively perform binary fission, and by continuously changing the reference endpoints of the binary fission line segment, effectively ensure the uniform distribution of the feature points after the split on the line segment until enough reliable new feature points are generated.

[0197] The pseudo code of the above process is shown in Table 1:

[0198] Table 1

[0199]

[0200]

[0201] Then, when extracting key frames, the inter-frame motion of the camera is tracked according to the above-mentioned point and line features through the tracking module, and the tracked camera frames are screened, and the key frames that meet the conditions are selected to participate in the subsequent mapping task.

[0202] In step S4 of this embodiment, the key frames obtained by the above process are focused on by the local mapping module, and the key frames are processed and mapped. After the key frames are determined, they are inserted into the local map, and edge connections are established with other key frames with common features in the map to facilitate the establishment of the subsequent graph optimization model. The corresponding bag of words of the key frame is then calculated, and a new point cloud map is generated by the triangulation method. The map point cloud is screened to delete invalid map points and retain new map points. In addition, this part will also complete the local BA optimization, jointly optimize the current key frame, other key frames connected to the current key frame, and the observed point cloud, and discard invalid redundant key frames to optimize the subsequent parts.

[0203] In step S5 of the present embodiment, the obtained local map is globally optimized through loop detection and global BA optimization modules, wherein candidate key frames are first screened according to the bag-of-words model of image recognition, and key frames that do not meet the set requirements are eliminated, and then the similarity transformation between the candidate current frame and the detected loop frame is calculated to obtain the cumulative error; then, the repeated point cloud maps are fused and the posture of the current frame is corrected in combination with the similarity transformation to achieve alignment of the two ends of the loop, and then the accumulated error of the loop detection is dispersed to the entire image through essential graph optimization; finally, the map is updated.

[0204] Additionally, this embodiment further provides a dynamic scene visual SLAM system based on point-line complementarity, the system comprising a semantic module, a tracking module, a local map construction module, a loop detection and a global BA optimization module.

[0205] The semantic module is the basis for solving the problem of feature point processing in dynamic scenes. The core and focus of the system in dynamic scenes is the recognition and elimination of dynamic features. In this part, the pre-trained target detection model is used to divide the potential dynamic objects and non-potential dynamic objects. This part completes the coarse-grained classification of dynamic features.

[0206] The tracking module is the core of map construction. All processed and optimized features in the method proposed by this system will directly participate in the calculation of this part. The input of this part is the processed line features and point features, and the output is the decided key frames. The decided key frames will participate in the next part of local map construction.

[0207] The local map construction module is the part of the system that directly affects the results. This part constructs the local map based on the high-quality key frames decided.

[0208] The loop detection thread and the global BA optimization module part mainly determine whether the camera has been to a place before based on the camera pose estimation calculated in the local map construction part, and perform global pose graph optimization. This part is the optimization and error elimination of the previous part. By establishing accurate loop detection, the cumulative error of pose estimation in the visual odometer is eliminated. This part only uses feature points to participate in the loop detection bag-of-words detection model to avoid more resource computing consumption.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A dynamic scene visual SLAM method based on point-line complementarity, characterized in that: The method comprises the following steps: S1, monitor the target prior information of the image through the semantic module, and identify the dynamic feature points that are inconsistent with the camera motion; S2. Extracting visual features from the image through a tracking module, wherein the visual feature extraction includes point feature extraction and line feature extraction; S3, the point features initially extracted by the tracking module are eliminated according to the dynamic feature points identified by the semantic module, the obtained static point features and line features are fused, and the static point features are expanded according to the fused line feature constraints, and then the key frames are selected according to the point and line features; S4, using the local mapping module to process the selected key frames and construct a local map; S5. Optimize and fuse the local map through loop detection and global BA optimization module to obtain the global map, and perform global BA optimization and map update.

2. A dynamic scene visual SLAM method based on point-line complementarity according to claim 1, characterized in that: In step S1, a target detection algorithm is used to detect target motion prior information for the collected image, wherein the image I at time t is divided into a potential dynamic object area D and a non-potential dynamic object area S, wherein D∩S=φ, D∪S=I, and the consistency of camera motion is assumed. The corresponding matching point position of the next frame is calculated according to the camera motion posture transformation relationship of the previous frame, and a deviation threshold is set. If the projection of the feature point in the image of the next frame and the camera projection deviation calculated based on the static assumption are greater than the deviation threshold, the feature point is identified as a dynamic feature point.

3. A dynamic scene visual SLAM method based on point-line complementarity according to claim 2, characterized in that: In step S2, an environmental RGB image and depth information are collected by an image acquisition device, and visual feature extraction is performed based on the collected image. The visual feature extraction includes: S21, point feature extraction; S2, line feature extraction; In the point feature extraction process of step S21, the following steps are included: S211: performing corresponding parameter calibration on the image to be processed, obtaining camera parameters, and extracting feature points of the image to be processed according to the ORB feature extraction algorithm; S212: Based on the extracted feature points, the image area is equally divided according to the image size, a reference feature point in the area to be compared is selected by polling, a circular area method with a variable radius is used to dynamically search the reference feature point in the equally divided image area, a dynamic threshold is set to score the feature points, and the feature point with the highest score is selected as the representative feature point of the area; S213: Screening is performed based on the feature points extracted after processing, and the feature point scores are calculated using a non-maximum suppression method combined with saliency detection, and the feature points with the highest scores are retained.

4. A dynamic scene visual SLAM method based on point-line complementarity according to claim 3, characterized in that: In step S211, the feature point extraction process includes: Select a point p in the image and assume that the gray value of the point is I p ; Set a circular area with point p in the image as the center and several pixels as the radius r, and extract several pixel points on the circle; Calculate the difference between the grayscale value of point p and the grayscale values ​​of several pixels in turn, take the absolute value of the difference, and set a threshold τ for statistics. If there are M consecutive pixels that satisfy the absolute value of the grayscale difference between the two pixels greater than the threshold τ, then point p is considered to be a fast corner point: |I p -I i |>τ The ORB algorithm is based on the fast corner detection method and uses the grayscale centroid method to obtain direction information. The direction information is based on the grayscale value of the image block as the weight core. The calculation process is as follows: Define an image block S with a feature point as the center and a radius of r. The moment of the image is: Where I(x,y) represents the gray value corresponding to the image coordinate (x,y), c and d represent the exponents of calculating the image moment in the x direction and in the y direction; The image moment along the x and y directions in the image block S is: In the formula, m 10 and m 01 Respectively represent the image moments of the image block in the x direction and in the y direction; The sum of the image grayscale values ​​of all pixels is The grayscale centroid of the image is C x ,C y Respectively represent the coordinates of the grayscale centroid of the image; the direction of the direction vector connecting the geometric center point of the image and the grayscale centroid of the image is the direction of the feature point, and the value of the direction is: The ORB algorithm uses improved BRIEF as the feature descriptor, and its process is as follows: First, perform Gaussian filtering on the image, take a circular area with a radius of r and the key point as the center, randomly select a pixel in the circular area and compare it with the grayscale value of the current key point pixel, binarize the comparison result, repeat the evaluation randomly, and construct a 256-dimensional binary descriptor: Among them I p Represents the gray value of p in the pixel area, I i Represents the gray value of the i-th pixel to be calculated, σ i Represents the value of the i-th dimension of the binary descriptor.

5. A dynamic scene visual SLAM method based on point-line complementarity according to claim 4, characterized in that: In step S212, a variable gradient radius method is used to search for feature points. The current radius r cur If the number of feature points m in the area is less than the feature point setting threshold n, the search radius is expanded; if the number of feature points m in the current radius area is greater than the feature point setting threshold, the search radius is contracted; the update speed of the search radius is proportional to the current feature point difference, and the calculation formula is as follows: When the expanded search radius needs to be updated, When you need to update the search radius, Among them, m is the number of feature points to be compared in the selected area, n is the number of feature points currently to be searched, and r max is the maximum expansion radius of the dynamic area, r min is the minimum shrinkage radius of the dynamic area, and α is the set proportional coefficient, which is used to control the speed of radius change.

6. A dynamic scene visual SLAM method based on point-line complementarity according to claim 5, characterized in that: In step S213, the global image saliency value is calculated and counted, the global image saliency value is calculated and counted, the LC detection algorithm is used to perform saliency detection on the entire image, the grayscale value of the current feature point and the grayscale interpolation of the surrounding pixels are calculated, and the feature points are evaluated from the global and local perspectives. The process is as follows: First, convert the input RGB image into a grayscale image, then traverse the pixels in the global image, calculate the corresponding grayscale values ​​and perform statistics to obtain a grayscale value array. The value of the array element represents the frequency of occurrence of the grayscale value. Then calculate the saliency value of a point in the pixel based on the Euclidean distance and the gray value frequency in the image: S(p) is the saliency value corresponding to the current pixel value, d(p,γ) is the Euclidean distance between the gray value of the current pixel and the gray value γ, and f γ is the frequency of gray value γ appearing in the image; Assume that the median value of the global saliency feature value of the current image is S med , the obtained feature points satisfy the following conditions at the same time: Among them, P represents the set of feature points that meet the search radius, Score is the feature point with the highest score, is the gray value of the feature point, is the gray value of several pixels around the feature point, S i is the saliency value of the current pixel, λ and μ are the weighting coefficients of the gray value and saliency value of the local pixel, respectively.

7. A dynamic scene visual SLAM method based on point-line complementarity according to claim 3, characterized in that: In the line feature extraction process of step S22, it includes the following steps: S221, using the EDlines line feature extraction algorithm to detect line segments in the RGB image and extract short line segments; S222, filtering the redundant short line segments extracted from the line features, selecting the maximum value of the extracted line segments as the line segment filtering base, and filtering the short line segments according to the minimum line segment length threshold by setting the length scale factor, and the filtering condition is: L min =η*max(L i ) Where L i is the line segment detected in the current image, L min is the minimum line segment threshold, and the line segments whose length is shorter than the minimum line segment threshold are filtered. η is an empirical value that can be pre-adjusted according to the actual scene; S223: merging similar line segments, wherein the merging process is determined by multiple parameters, wherein the multiple parameters include the angle difference between the line segments to be processed after the short line segments are removed, the distance between the nearest neighbor vertices of the two line segments to be merged, and the distance between the midpoints of the two line segments; For the two line segments l extracted from the two-dimensional image a and l b , calculate the angle error θ between the two line segments ab , the distance between two neighboring vertices Sd ab , and the distance Sm between the midpoints of each line segment ab ; According to the two endpoints of the line segment, the two line segments are described in vector form. The absolute value of the angle difference between the two vectors represents the angle error θ of the two line segments. ab , so: Where (x a ,y a ) and (x b ,y b ) are two line segments l a and l b The vertex of The penalty function of adaptive length and distance is introduced to enhance the reliability of segment merging, and the angle threshold is adaptively adjusted according to the distance value. s is the distance threshold, τ θ is the angle threshold, let l a The longer line segment of the two segments: Therefore, the line segments for line feature merging satisfy the following constraints at the same time: θ ab Less than the set dynamic angle threshold; the distance between the nearest neighbor vertices Sd ab and the distance between the midpoints of the two line segments Sm ab The maximum value is less than the set threshold; The merging of line segments includes the angle of the new line segment after merging and the selection of the vertex of the new line segment. The midpoint coordinates of the new line segment are set to (x new, y new ), and use the dynamic proportion method to calculate the midpoint of the new line segment: Where (x a ,y a ),(x b, y b ) are the coordinates of the midpoints of the two line segments, and the angle of the new line segment is θ new : Construct a straight line through the midpoint of the new line segment and the angle of the new line segment, and merge the l a and l b The vertices of the orthogonal projection are projected onto the undetermined line after the merger, and the two points on the line with the farthest distance are used as the endpoints of the new line segment, and the new line segment is connected with the longer line segment l before the merger. a The line segment constraint conditions before merging are subjected to regression testing. If the corresponding threshold conditions are not met, it is considered that the line segment merging has failed.

8. A dynamic scene visual SLAM method based on point-line complementarity according to claim 7, characterized in that: In step S3, the dynamic feature points in the obtained point feature set are firstly removed, and the point feature set after removal is a static feature point set; then the static feature point set is matched with the line feature set, wherein improvement is made based on point feature optical flow matching, and reasoning is performed based on the grayscale invariant assumption: I(u+u,v+dv,t+dt)=I(u,v,t) The coordinate position of a point in space at time t is (x, y). At time t+dt, the point moves to (x+dx, y+dy), and the grayscale remains unchanged. Assuming that the pixel grayscale value remains constant in a short period of time, we have: in Indicates the gradient change of the gray value of the point in the x-axis direction. Indicates the gradient change of the gray value of the point in the y direction, respectively denoted as I x and I y , Represents the change value of grayscale, denoted as I t , It represents the speed in the x direction. The speed in the x direction is the same, recorded as u and v respectively, and the following equation is established: The above formula is a linear equation with two variables. Multiple additional equations about constraints are obtained in the entire image. Multiple points on the line segment also have the same motion. Thus, multiple overdetermined equations are established and solved by the least squares method. In the above process, for the point-line feature fusion strategy, a method for compensating feature points of bisection line segments with variable direction is proposed, which includes the following steps: a) When the number of static feature points in the current image area does not reach the minimum threshold and there are merged reliable line segments in the image area, the insufficient feature points are compensated by obtaining feature points between the line segments; b) Perform binary fission on the line segment, that is, the first feature point is generated as the midpoint of the two line segments; c) Second fission, select the midpoint of the generated line segment and the midpoint of one of the line segment endpoints; d) The third fission, the midpoint of the newly generated line segment and the different line segment endpoints used in the previous fission, the midpoint of the line segment formed is used as the new fission point; e) According to the above method, iteratively perform binary fission, and by continuously changing the reference endpoints of the binary fission line segment, effectively ensure the uniform distribution of the feature points after the split on the line segment until enough reliable new feature points are generated.

9. A dynamic scene visual SLAM method based on point-line complementarity according to claim 8, characterized in that: In step S4, the keyframes are processed and mapped by the local mapping module. After the keyframes are determined, they are inserted into the local map and connected to other keyframes with common features in the map by edges. Then, the corresponding bag of words of the keyframe is calculated, and a new point cloud map is generated by triangulation. The map point cloud is filtered to delete invalid map points and retain new map points. In addition, the local BA optimization is completed by the local mapping module, and the current keyframe, other keyframes connected to the current keyframe, and the observed point cloud are jointly optimized. In step S5, the local map obtained is globally optimized through loop detection and global BA optimization modules, wherein candidate key frames are first screened according to the bag-of-words model of image recognition, and key frames that do not meet the set requirements are eliminated, and then the similarity transformation between the candidate current frame and the detected loop frame is calculated to obtain the cumulative error; then, the repeated point cloud maps are fused and the posture of the current frame is corrected in combination with the similarity transformation to achieve alignment of the two ends of the loop, and then the accumulated error of the loop detection is dispersed to the entire image through essential graph optimization; finally, the map is updated.

10. A system for executing the point-line complementarity-based dynamic scene visual SLAM method according to any one of claims 1 to 9, characterized in that: The system includes a semantic module, a tracking module, a local map construction module, a loop detection module and a global BA optimization module, wherein: The semantic module monitors the target prior information of the image and identifies the dynamic feature points that are inconsistent with the camera motion; The tracking module extracts visual features from the image, where visual feature extraction includes point feature extraction and line feature extraction; then the point features initially extracted by the tracking module are eliminated according to the dynamic feature points identified by the semantic module, the obtained static point features and line features are fused, and the static point features are expanded according to the fused line feature constraints, and then the key frames are selected according to the point and line features; The local map construction module processes the filtered key frames and constructs local maps; The loop detection thread and the global BA optimization module optimize and fuse the local maps to obtain the global map, and perform global BA optimization and map updates.

Images