Point-line feature fusion SLAM method and system in weak texture environment

Abstract

The invention relates to the technical field of visual synchronous localization and mapping, and discloses a point-line feature fusion SLAM (Simultaneous Localization and Mapping) method and system in a weak texture environment, and the technical scheme is characterized in that the extraction of straight line features and the merging of short lines are carried out based on an edge drawing straight line detection algorithm; based on the BFMatcher, a rough matching group is rapidly obtained, and line feature matching is completed; classifying the line segments into parallel line segments or vertical line segments according to cosine angles of the line segments, and performing line segment pose optimization; a line feature word bag model is introduced, a positioning behavior is improved, and a self-adaptive point-line pose optimization model is provided; constructing MW to provide geometric constraints; in a low-texture environment, available line features can be effectively extracted and feature matching is carried out so that positioning precision and stability of an SLAM system are increased and camera positioning accuracy in the low-texture environment is increased.

Description

Technical Field

[0001] The present invention relates to the field of visual synchronous positioning and mapping technology, and more specifically, to a point-line feature fusion SLAM method and system in a weak texture environment. Background Art

[0002] Simultaneous localisation and mapping (SLAM) has always been an important part of the robotics field and autonomous driving. The current mainstream SLAM technologies are mainly divided into two-dimensional laser SLAM, three-dimensional laser SLAM and V-SLAM. Among them, V-SLAM has developed rapidly due to the low cost and low power of cameras and the ability to collect rich environmental information. V-SLAM is divided into feature point-based SLAM methods and direct SLAM methods based on the different image information used. Since the direct method is more sensitive to changes in illumination and camera parameters than the feature point method and requires more computing resources, the feature point method is widely used in texture-rich environments.

[0003] The feature point method will often cause the V-SLAM system to have problems such as feature matching failure and tracking loss when the robot moves too fast or the surrounding environment lacks texture. Furthermore, point features alone cannot provide a clearer scene description for the robot. Therefore, more advanced geometric features are needed to describe the surrounding environment. Feature line segments have more feature information than feature points. At the same time, weak texture environments are usually accompanied by strong structures. In this environment, the number of feature line segments is also much larger than feature points. In recent years, V-SLAM has also focused more and more on feature point and line fusion.

[0004] However, since the process of extracting and describing feature segments is relatively complicated, the number of feature segments is large in a strong structure environment, which can easily cause data redundancy and affect the accuracy and real-time performance of the SLAM system; on the other hand, it can easily cause camera positioning drift.

[0005] Therefore, the present invention provides a point-line feature fusion SLAM method and system in a weak texture environment, which improves the above technical problems. Summary of the invention

[0006] The disclosed embodiments aim to address the deficiencies of the prior art and provide a point-line feature fusion SLAM method and system in a weak texture environment. The present invention can effectively extract available line features and perform feature matching in a low-texture environment, thereby improving the positioning accuracy and stability of the SLAM system, while also improving the camera positioning accuracy in a low-texture environment.

[0007] The above technical objectives of the present invention are achieved through the following technical solutions: A point-line feature fusion SLAM method and system in a weak texture environment, comprising the following steps: S1, extracting straight line features and merging short lines based on an edge-drawing straight line detection algorithm to reduce line feature tracking errors;

[0008] S2, based on BFMatcher, quickly obtain the rough matching group, and remove the wrong matching pairs according to the angle and distance between the spatial line segments and the distance of the descriptor, complete the line feature matching, and improve the positioning accuracy of the SLAM system;

[0009] S3. In a low-texture environment, line segments are classified into parallel or vertical segments according to their cosine angles, and the line segment pose is optimized to reduce the camera pose drift.

[0010] S4. Aiming at the optimization of camera pose in low-texture environment, the line feature bag-of-words model is introduced and the positioning behavior is improved. An adaptive point-line pose optimization model is proposed to improve the accuracy of camera positioning in low-texture environment.

[0011] S5. Construct MW to provide geometric constraints to reduce errors caused by line features.

[0012] As a preferred technical solution of the present invention, the process of extracting straight line features and merging short lines is: using line segment endpoints to represent the detected straight line, that is, l = [p sp ,p ed ], where p sp and p ep is the pixel coordinate of the endpoint of the line segment;

[0013] Let l1 and l2 be the two line segments to be merged, and start merging the line segments when the following conditions are met: the midpoint distance between l1 and l2 is less than the minimum distance threshold; the angle difference between l1 and l2 is less than the minimum angle threshold; the distance between the descriptors of l1 and l2 is less than the minimum descriptor distance threshold;

[0014] After merging the line segments, the short line segments are removed. The improved EDLines algorithm extracts the short lines that can be merged and removes the short lines that cannot be merged.

[0015] After obtaining the final line segment, recalculate the line segment descriptor and establish the normalized line feature expression L as follows:

[0016]

[0017] Where: p s is the spatial coordinate of the starting point of the bottom line feature of the image; p e The spatial coordinates of the end point of the line feature under the image.

[0018] As a preferred technical solution of the present invention, the process of quickly obtaining a rough matching group based on BFMatcher is as follows: by calculating the depth ratio R between the midpoint and the endpoint of the line segment z To ensure that the line segments projected into three-dimensional space are real and usable:

[0019] R z The calculation formula is:

[0020]

[0021] Where: Z mid is the depth value of the midpoint of the space segment; Z s is the depth value of the starting point of the space segment; Z e is the depth value of the end point of the space line segment;

[0022] After determining the available spatial segments, the incorrect matching pairs are eliminated according to the following conditions: the ratio of the lengths of the two spatial segments meets the threshold range; the ratio of the angles of the two spatial segments meets the threshold range; the distance between the descriptors of the two segments meets the threshold range; and the rotation histogram is met.

[0023] As a preferred technical solution of the present invention, the process of classifying line segments into parallel line segments or perpendicular line segments according to their cosine angles is as follows: to distinguish between parallel and perpendicular line segments, The calculation formula is

[0024]

[0025] Where: is the spatial normalized expression of the two straight lines in the camera coordinate system;

[0026] when The value of 0 or 1 corresponds to a vertical segment and a parallel segment, respectively, defining the angle error for

[0027]

[0028] Defining Parallel Segment Error The vertical line segment error is For the formula:

[0029]

[0030] Where: w n The error term is weighted for the line response returned by the line detection algorithm.

[0031] As a preferred technical solution of the present invention, the process of optimizing the line segment posture is as follows: uniformly establish the minimum error function E for all parallel line segments and vertical line segments to be optimized in the space.L :

[0032]

[0033] Where: L is the set of line segments; L ⊥ is a set of vertical line segments; L || is a set of parallel line segments, and ρ(·) is a robust kernel function.

[0034] As a preferred technical solution of the present invention, the line feature bag-of-words model is introduced and the positioning behavior is improved. The process of the adaptive point-line pose optimization model is proposed as follows: to ensure that the current frame and the candidate keyframe can provide enough point-line features to achieve positioning, the ratio of the number of successful point-line feature matches to the number of point-line extractions R is required. pl , greater than the set threshold to enter the subsequent work, define R pl for

[0035]

[0036] Where: C p , C kp , C l , C kl The total number of point and line features extracted from the current frame and the candidate key frame respectively; M p 、M kp 、M l 、M kl The number of successfully matched point and line features for the current frame and candidate keyframes, respectively.

[0037] As a preferred technical solution of the present invention, due to the influence of noise, there is a certain deviation between the camera's observation value and the projection value of the feature point in the picture pixel, and it is necessary to adjust its own posture to minimize the deviation;

[0038] Reprojection error of point features e p It refers to the feature point p1 observed in the i-th frame image projected to the j-th frame image p1′, and the feature point u matching the j-th frame image i The error between p :

[0039]

[0040] Where: p is the set of successfully matched points; Π is the camera reprojection function; p i is the feature point observed in the i-th frame image; u i is the matching feature point in the j-th frame image.

[0041] Reprojection error of line features e l The features of the same point are similar. Calculate the error e between the projected line segment L1 and the observed line segment L2l :

[0042]

[0043] Where: X is the optimized camera pose; L1 is the feature line segment observed in the i-th frame; L2 is the matching feature line segment in the j-th frame; L1′ is the line segment projected from L1 to the j-th frame; r1 and r2 are the distances from the endpoint of the L1 projected line segment L1′ to L2 respectively; p 2s and p 2e is the endpoint of the matching line segment in the jth frame;

[0044] Since the number of point features and line features extracted in different scenes is different, the credibility is also different. The total error function e of adaptive reprojection based on point and line features is:

[0045]

[0046] As a preferred technical solution of the present invention, the process of constructing MW to provide geometric constraints is as follows: when the local mapping thread receives enough key frames, it is necessary to optimize the coordinate system more accurately;

[0047] Define the Manhattan coordinate system error e M for

[0048]

[0049] Where: e MA0 、e MA1 、e MA2 is the residual between the coordinate axis and the associated line segment, which is specifically expressed as

[0050]

[0051] Where: is the map segment associated with the Manhattan coordinate axis, is the angular error between the map line segment and the associated Manhattan coordinate axis; ρ(·) is the robust kernel function;

[0052] When the coordinate system is stable, singular value decomposition is used to preserve the orthogonality between the coordinate axes to obtain an accurate Manhattan coordinate system. When inserting a new keyframe, each map segment needs to be classified, associated with the Manhattan coordinate system, and added to the corresponding set M = {MA0, MA1, MA2} to facilitate subsequent map optimization processing.

[0053] As a preferred technical solution of the present invention, when the local mapping thread receives continuous key frames, it maps these key frames and the associated map point set P and map line set L into space and performs pose optimization;

[0054] In view of the strong structural characteristics of low-texture environments, the local map optimization function Γ is defined according to the point-line feature projection constraint, line feature parallel and vertical constraint, and Manhattan structured constraint:

[0055] Γ=argmin(A+B+C) (14)

[0056] Wherein: A is the projection constraint of point-line features, specifically expressed as formula; B is the parallel and vertical constraint of line features, specifically expressed as formula; C is the Manhattan structured constraint, specifically expressed as formula.

[0057]

[0058] Where: k c is the key frame related to P and L; k f is an irrelevant key frame; P k 、V k A collection of map points and lines that match the current keyframe.

[0059] As a preferred technical solution of the present invention, a point-line feature fusion SLAM system in a weak texture environment, the system includes a tracking thread module, a local map thread module, and a loop detection thread module;

[0060] The tracking thread module is used to extract point and line features from the image and estimate the camera's motion pose based on these features; the detected line features are classified into parallel line segments and vertical line segments according to their cosine angles; the strong structural features in low-texture environments are used to optimize the pose of line features to reduce errors.

[0061] The local map thread module is used to estimate and establish a more accurate Manhattan coordinate system after receiving enough line features. At the same time, it performs Manhattan structure optimization in the local map to reduce posture drift, and then constructs and optimizes the local map.

[0062] The loop detection thread module is used to detect loops and eliminate the accumulated drift error through pose graph optimization.

[0063] In summary, the present invention has the following beneficial effects: The present invention uses the improved EDLines algorithm to complete the extraction of straight line features and the merging of short lines, thereby reducing the line feature tracking error. The improved BFMatcher matching algorithm is used to complete the line feature matching, thereby improving the positioning accuracy of the SLAM system. Aiming at the optimization of camera pose in a low-texture environment, an adaptive point-line pose optimization model is proposed to improve the accuracy of camera positioning in a low-texture environment and avoid the camera positioning drift phenomenon in a weak-texture environment. MW is constructed to provide geometric constraints and reduce the errors caused by line features. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A framework diagram of a point-line feature fusion SLAM system in a weak texture environment provided by an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of the short line merging detected according to an embodiment of the present invention;

[0066] Figure 3 A schematic diagram of line feature matching provided by an embodiment of the present invention;

[0067] Figure 4 Improved BFMatcher flow chart provided by the embodiment of the present invention;

[0068] Figure 5 Schematic diagram of point feature reprojection error provided by an embodiment of the present invention

[0069] Figure 6 A schematic diagram of line feature reprojection error provided by an embodiment of the present invention;

[0070] Figure 7 A schematic diagram of a simulation of an algorithm data set provided by an embodiment of the present invention;

[0071] Figure 8 A schematic diagram of a mobile robot platform provided by an embodiment of the present invention;

[0072] Fig. 9 A schematic diagram of the actual deployment of the algorithm provided in the embodiment of the present invention;

[0073] Fig.10 A flow chart of a point-line feature fusion SLAM method in a weak texture environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0075] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0076] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0077] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0078] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0079] The disclosed embodiments are intended to solve the problem that the SLAM system is prone to system instability in an environment with few point features or low texture. In view of this, the disclosed embodiments propose a point-line feature fusion SLAM method and system in a weak texture environment to effectively extract available line features and perform feature matching in a low texture environment, thereby improving the positioning accuracy and stability of the SLAM system, and also improving the camera positioning accuracy in a low texture environment.

[0080] Please refer to Fig.10 , Fig.10 The flowchart of the point-line feature fusion SLAM method in a weak texture environment described in the embodiment of the present disclosure is shown. The overall process mainly includes the following 5 steps:

[0081] Step 1: Extract straight line features and merge short lines based on edge drawing straight line detection algorithm to reduce line feature tracking errors.

[0082] The point and line features are extracted from the input color image, and the corresponding depth information is extracted in combination with the depth map to restore the three-dimensional pose of the point and line features. The point feature extraction has not been modified much, and this application still uses the original feature point extraction method and descriptor.

[0083] Line segment detector (LSD) is a method for quickly detecting straight line information in an image. It has a certain degree of robustness and can cope with noise and partial occlusion. However, its detection effect may not be good for curved or discontinuous straight lines, and it may also be limited by computing resources when processing large-scale data. The EDLines algorithm is also a straight line detection algorithm that identifies straight lines based on the geometric relationship of edge point pairs. Unlike traditional line detection algorithms based on edge detection, the EDLines algorithm pays more attention to the geometric relationship between edge points and uses these geometric relationships to detect straight lines. The advantage of this method is that it can identify straight lines more accurately, rather than relying solely on the intensity information of the edge. Although the algorithm can detect more feature lines, there are many short lines. These line segments will increase the computational complexity of the SLAM system and the positioning accuracy will also decrease.

[0084] To solve the above problems, an improvement strategy is proposed: merge short lines according to the position of line segments, and remove short lines that cannot be merged.

[0085] Specifically, the detected straight line is represented by the endpoint of the line segment, that is, l = [p sp ,p ed ], where p sp and p ep is the pixel coordinate of the endpoint of the line segment. Let l1 and l2 be the two line segments to be merged, and start merging the line segments when the following conditions are met:

[0086] 1) The midpoint distance between l1 and l2 is less than the minimum distance threshold;

[0087] 2) The angle difference between l1 and l2 is less than the minimum angle threshold;

[0088] 3) The distance between l1 and l2 descriptors is less than the minimum descriptor distance threshold.

[0089] The above thresholds are all empirical fixed values. The line segments that meet the conditions are distributed according to their positions, such as Figure 2 After the line segments are merged, the short line segments are removed. The improved EDLines algorithm extracts the short lines that can be merged and removes the short lines that cannot be merged.

[0090] After obtaining the final line segment, recalculate the line segment descriptor for subsequent matching work. Establish the normalized expression L of line features as follows:

[0091]

[0092] Where: p s is the spatial coordinate of the starting point of the bottom line feature of the image; p e The spatial coordinates of the end point of the line feature under the image.

[0093] Step 2: Based on BFMatcher, a rough matching group is quickly obtained, and the wrong matching pairs are eliminated according to the angle, distance and descriptor distance between the spatial line segments to complete the line feature matching, which is used to improve the positioning accuracy of the SLAM system.

[0094] Specifically, an improvement is proposed to BFMatcher, which ensures that the line segments projected into the three-dimensional space are real and available through calculation; BFMatcher is a two-dimensional feature matching method, which compares the Hamming distance between two line feature descriptors, selects the line feature descriptor in the reference image that is most similar to the line feature descriptor to be matched as the best match and returns it. This algorithm contains a large number of false matching pairs.

[0095] The returned matching pairs are projected into three-dimensional space for verification, and incorrect matches are eliminated based on the line segment length difference, angle difference, and descriptor distance, such as Figure 4 shown.

[0096] To prevent errors caused by reading wrong depth pixels, the depth ratio R of the midpoint and endpoint of the line segment is calculated. z To ensure that the line segments projected into three-dimensional space are real and usable. z The calculation formula is

[0097]

[0098] Where: Z mid is the depth value of the midpoint of the space segment; Z s is the depth value of the starting point of the space segment; Z e It is the depth value of the end point of the space line segment.

[0099] After determining the available space segments, the incorrect matching pairs are eliminated according to the following conditions:

[0100] 1) The ratio of the lengths of the two space segments meets the threshold range;

[0101] 2) The angle ratio of the two space line segments meets the threshold range;

[0102] 3) The distance between the two line segment descriptors meets the threshold range;

[0103] 4) Comply with the rotation histogram.

[0104] Among them, the above thresholds are all empirical fixed thresholds.

[0105] Step 3: In a low-texture environment, classify the line segments into parallel or vertical segments according to their cosine angles, and perform line segment pose optimization to reduce the camera pose drift.

[0106] Specifically, in a low-texture, strong-structure environment, such as a corridor, first calculate the cosine angle between each line. to distinguish between parallel and perpendicular line segments, The calculation formula is

[0107]

[0108] Where: is the spatial normalized expression of the two straight lines in the camera coordinate system, similar to Eq.

[0109] when The value of 0 or 1 corresponds to a vertical segment and a parallel segment, respectively, defining the angle error for

[0110]

[0111] Defining Parallel Segment Error The vertical line segment error is For the formula:

[0112]

[0113] Where: w n The error term is weighted for the line response returned by the line detection algorithm.

[0114] The minimum error function E is uniformly established for all parallel and vertical line segments to be optimized in the space. L for

[0115]

[0116] Where: L is the set of line segments; L ⊥ is a set of vertical line segments; L || is a set of parallel line segments, and ρ(·) is a robust kernel function. Step 4: To optimize the camera pose in a low-texture environment, a line feature bag-of-words model is introduced to improve the positioning behavior, and an adaptive point-line pose optimization model is proposed to improve the accuracy of camera positioning in a low-texture environment.

[0117] In camera tracking, the camera projection relationship needs to be used to optimize the current camera pose. ORB-SLAM2 only relies on point features to optimize the pose. In low-texture and strong-structure environments, there may be problems such as pose optimization failure due to fewer point features. This application proposes an adaptive point-line tracking pose optimization method to address this problem.

[0118] This application first adds a line feature Bow (line feature bag-of-words model) on the original basis; then uses the point-line Bow to search for candidate key frames and perform feature matching when tracking the reference key frames; finally, based on the successfully matched point-line features, an adaptive point-line minimum projection residual model is constructed to optimize the current camera pose.

[0119] When entering the reference keyframe tracking, all candidate keyframes are first retrieved through the inverted index of the point and line Bow, and the point and line features of the current frame and the candidate keyframe are extracted and matched to obtain a successfully matched point and line feature matching pair.

[0120] In order to ensure that the current frame and the candidate keyframes can provide enough point and line features to achieve positioning, the ratio R of the number of successful point and line feature matches to the number of point and line extractions is required. pl , greater than the set threshold to enter the subsequent work, define R pl for

[0121]

[0122] Where: C p , C kp , C l , C kl The total number of point and line features extracted from the current frame and the candidate key frame respectively; M p 、M kp 、M l 、M kl The number of successfully matched point and line features for the current frame and candidate keyframes, respectively.

[0123] Due to the influence of noise, there is a certain deviation between the camera's observation value and the projection value of the feature point in the image pixel. The SLAM system needs to adjust its own posture to minimize the deviation.

[0124] Reprojection error of point features e p It refers to the feature point p1 observed in the i-th frame image projected to the j-th frame image p1′, and the feature point u matching the j-th frame image i The error between p As Figure 5 As shown, the expression is

[0125]

[0126] Where: p is the set of successfully matched points; Π is the camera reprojection function; p i is the feature point observed in the i-th frame image; u i is the matching feature point in the j-th frame image.

[0127] Reprojection error of line features e l The same point features are similar, and the error between the projected line segment L1 and the observed line segment L2 is calculated, such as Figure 6 As shown. This paper uses the distance between the end point of the line segment and the projected line segment to calculate e l , the expression is

[0128]

[0129] Where: X is the optimized camera pose; L1 is the feature line segment observed in the i-th frame; L2 is the matching feature line segment in the j-th frame; L1′ is the line segment projected from L1 to the j-th frame; r1 and r2 are the distances from the endpoint of the L1 projected line segment L1′ to L2 respectively; p 2s and p 2e is the endpoint of the matched line segment in the jth frame.

[0130] Since the number of point features and line features extracted in different scenes is different, the credibility is also different. To solve this problem, this application designs an adaptive reprojection total error function e based on point and line features:

[0131]

[0132] Where: e ip is the projection error between each pair of matching points; r j1 、r j2 are the endpoint distance errors of each matching line pair.

[0133] In summary, in a low-texture environment, the current pose is continuously optimized by establishing the adaptive reprojection error function, making full use of point and line features and adjusting the adaptive coefficients according to the point and line feature errors, thereby optimizing and obtaining a more accurate current pose.

[0134] Step 5: Constructing MW provides geometric constraints to reduce the errors caused by line features.

[0135] Specifically, MW means assuming that the world is a structure composed of axes intersecting at right angles. To ensure the accuracy of the Manhattan coordinate system, this application first makes a rough estimate of the first key frame, divides the frame image into multiple grids, and restores it to the spatial coordinate system, while calculating the plane normal vector of each grid. Subsequently, a histogram strategy is used to filter out normal vectors with large directional deviations, and extract normal vectors with similar directions as the Manhattan coordinate system. As the image is input, the coordinate system gradually becomes stable, indicating that the Manhattan coordinate system has been successfully initialized.

[0136] When the local mapping thread receives enough key frames, it is necessary to optimize the coordinate system more accurately. Define the Manhattan coordinate system error e M for

[0137]

[0138] Where: e MA0 、e MA1 、e MA2 is the residual between the coordinate axis and the associated line segment, which is specifically expressed as

[0139]

[0140] Where: is the map segment associated with the Manhattan coordinate axis, is the angular error between the map line segment and the associated Manhattan coordinate axis; ρ(·) is the robust kernel function.

[0141] When the coordinate system is stable, singular value decomposition is used to preserve the orthogonality between the coordinate axes to obtain an accurate Manhattan coordinate system. When inserting a new keyframe, each map line segment needs to be classified and associated with the Manhattan coordinate system as much as possible. That is, if the map line segment is approximately parallel to the Manhattan coordinate axis, it is considered to be associated and added to the corresponding set M = {MA0, MA1, MA2} to facilitate subsequent map optimization processing.

[0142] After receiving continuous key frames, this thread maps these key frames and the associated map point set P and map line set L into space and performs pose optimization to reduce the drift effect of cumulative errors on the camera pose.

[0143] In view of the strong structural characteristics of low-texture environments, this application defines the local map optimization function Γ based on point-line feature projection constraints, line feature parallel and vertical constraints, and Manhattan structured constraints:

[0144] Γ=argmin(A+B+C) (14)

[0145] Wherein: A is the projection constraint of point-line features, specifically expressed as formula; B is the parallel and vertical constraint of line features, specifically expressed as formula; C is the Manhattan structured constraint, specifically expressed as formula.

[0146]

[0147] Where: k c is the key frame related to P and L; k f is an irrelevant key frame; P k 、V k is the set of map points and lines that match the current keyframe; ⊥ 、E || 、E p 、E l are defined in Formula, Formula, Formula, and Formula respectively; Same as E || Similar, that is, the map segment is parallel to the Manhattan coordinate axis, so the formula is adopted.

[0148] Please refer to Figure 1 , Figure 1 A point-line feature fusion SLAM system in a weak texture environment according to an embodiment of the present disclosure is shown, the system comprising: a tracking thread module, a local map thread module, and a loop detection thread module;

[0149] The tracking thread module is used to extract point and line features from the image and estimate the camera's motion pose based on these features. The detected line features are first classified into parallel line segments and vertical line segments based on their cosine angles. At the same time, the strong structural features in low-texture environments are used to optimize the pose of the line features to reduce the error.

[0150] The local map thread module is used to receive the keyframes output by the tracking thread and optimize the local map. To ensure the simplicity of the map, it is necessary to remove redundant map points, lines and keyframes. After receiving enough line features, the local map thread module estimates and establishes a more accurate Manhattan coordinate system. At the same time, it optimizes the Manhattan structure in the local map to reduce the position drift, builds the local map and optimizes the local map to reduce the error.

[0151] The loop detection thread module is used to detect loops and eliminate accumulated drift errors through pose graph optimization; specifically, as the camera explores the extension of the path during the mapping process, accumulated errors continue to be generated. In addition to the above optimizations to reduce the camera's accumulated errors, these accumulated errors are corrected through the loop detection thread to improve the accuracy and robustness of the SLAM system. The loop detection thread uses the word Bow to obtain a group of candidate keyframes and matches their point features with the current frame. When a candidate keyframe contains a sufficient number of inliers with the current frame, the system accepts the loop keyframe and performs a loop fusion operation. To reduce the system's computational burden, this thread only uses point features for processing.

[0152] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A point-line feature fusion SLAM method in a weak texture environment, characterized in that: The method comprises the following steps: S1, extracting straight line features and merging short lines based on edge drawing straight line detection algorithm to reduce line feature tracking errors; S2, based on BFMatcher, quickly obtain the rough matching group, and remove the wrong matching pairs according to the angle and distance between the spatial line segments and the distance of the descriptor, complete the line feature matching, and improve the positioning accuracy of the SLAM system; S3. In a low-texture environment, line segments are classified into parallel or vertical segments according to their cosine angles, and the line segment pose is optimized to reduce the camera pose drift. S4. Aiming at the optimization of camera pose in low-texture environment, the line feature bag-of-words model is introduced and the positioning behavior is improved. An adaptive point-line pose optimization model is proposed to improve the accuracy of camera positioning in low-texture environment. S5. Construct MW to provide geometric constraints to reduce errors caused by line features.

2. The point-line feature fusion SLAM method in a weak texture environment according to claim 1, characterized in that: The process of extracting straight line features and merging short lines is as follows: using line segment endpoints to represent the detected straight line, i.e. l ​​= [p sp ,p ed ], where p sp and p ep is the pixel coordinate of the endpoint of the line segment; Let l1 and l2 be the two line segments to be merged, and start merging the line segments when the following conditions are met: the midpoint distance between l1 and l2 is less than the minimum distance threshold; the angle difference between l1 and l2 is less than the minimum angle threshold; the distance between the descriptors of l1 and l2 is less than the minimum descriptor distance threshold; After merging the line segments, the short line segments are removed. The improved EDLines algorithm extracts the short lines that can be merged and removes the short lines that cannot be merged. After obtaining the final line segment, recalculate the line segment descriptor and establish the normalized line feature expression L as follows: Where: p s is the spatial coordinate of the starting point of the bottom line feature of the image; p e is the spatial coordinate of the end point of the bottom line feature of the image.

3. The point-line feature fusion SLAM method in a weak texture environment according to claim 1, characterized in that: The process of quickly obtaining a rough matching group based on BFMatcher is as follows: by calculating the depth ratio R between the midpoint and endpoint of the line segment z To ensure that the line segments projected into three-dimensional space are real and usable: R z The calculation formula is: Where: Z mid is the depth value of the midpoint of the space segment; Z s is the depth value of the starting point of the space segment; Z e is the depth value of the end point of the space line segment; After determining the available spatial segments, the incorrect matching pairs are eliminated according to the following conditions: the ratio of the lengths of the two spatial segments meets the threshold range; the ratio of the angles of the two spatial segments meets the threshold range; the distance between the descriptors of the two segments meets the threshold range; and the rotation histogram is met.

4. The point-line feature fusion SLAM method in a weak texture environment according to claim 1, characterized in that: The process of classifying line segments as parallel or perpendicular based on their cosine angles is as follows: to distinguish between parallel and perpendicular line segments, The calculation formula is Where: is the spatial normalized expression of the two straight lines in the camera coordinate system; when The value of 0 or 1 corresponds to a vertical segment and a parallel segment, respectively, defining the angle error for Defining Parallel Segment Error The vertical line segment error is For the formula: Where: w n The error term is weighted for the line response returned by the line detection algorithm.

5. The point-line feature fusion SLAM method in a weak texture environment according to claim 1, characterized in that: The process of optimizing the line segment pose is as follows: uniformly establish the minimum error function E for all parallel and vertical line segments to be optimized in the space. L : Where: L is the set of line segments; L ⊥ is a set of vertical line segments; L || is a set of parallel line segments, and ρ(·) is a robust kernel function.

6. The point-line feature fusion SLAM method in a weak texture environment according to claim 1, characterized in that: The line feature bag-of-words model is introduced to improve the positioning behavior, and the process of the adaptive point-line pose optimization model is proposed as follows: In order to ensure that the current frame and the candidate keyframe can provide enough point-line features to achieve positioning, the ratio of the number of successful point-line feature matches to the number of point-line extractions R is required. pl , greater than the set threshold to enter the subsequent work, define R pl for Where: C p , C kp , C l , C kl The total number of point and line features extracted for the current frame and candidate key frames respectively; M p 、M kp 、M l 、M kl The number of successfully matched point and line features for the current frame and candidate keyframes, respectively.

7. The point-line feature fusion SLAM method in a weak texture environment according to claim 6, characterized in that: Due to the influence of noise, there is a certain deviation between the camera's observation value and the projection value of the feature point in the image pixel, and it is necessary to adjust its own posture to minimize the deviation; Reprojection error of point features e p It refers to the feature point p1 observed in the i-th frame image projected to the j-th frame image p1′, and the feature point u matching the j-th frame image i The error between p : Where: p is the set of successfully matched points; Π is the camera reprojection function; p i is the feature point observed in the i-th frame image; u i is the matching feature point in the j-th frame image. Reprojection error of line features e l The features of the same point are similar. Calculate the error e between the projected line segment L1 and the observed line segment L2 l : Where: X is the optimized camera pose; L1 is the feature line segment observed in the i-th frame; L2 is the matching feature line segment in the j-th frame; L1′ is the line segment projected from L1 to the jth frame; r1 and r2 are the distances from the endpoint of the L1 projected line segment L1′ to L2 respectively; p 2s and p 2e is the endpoint of the matching line segment in the jth frame; Since the number of point features and line features extracted in different scenes is different, the credibility is also different. The total error function e of adaptive reprojection based on point and line features is:

8. The point-line feature fusion SLAM method in a weak texture environment according to claim 1, characterized in that: The process of constructing the geometric constraints provided by MW is as follows: when the local mapping thread receives enough keyframes, the coordinate system needs to be optimized more accurately; Define the Manhattan coordinate system error e M for Where: e MA0 、e MA1 、e MA2 is the residual between the coordinate axis and the associated line segment, which is specifically expressed as Where: is the map segment associated with the Manhattan coordinate axis, is the angular error between the map line segment and the associated Manhattan coordinate axis; ρ(·) is the robust kernel function; When the coordinate system is stable, singular value decomposition is used to preserve the orthogonality between the coordinate axes to obtain an accurate Manhattan coordinate system. When inserting a new keyframe, each map segment needs to be classified, associated with the Manhattan coordinate system, and added to the corresponding set M = {MA0, MA1, MA2} to facilitate subsequent map optimization processing.

9. The point-line feature fusion SLAM method in a weak texture environment according to claim 8, characterized in that: After receiving continuous key frames, the local mapping thread maps these key frames and the associated map point set P and map line set L into space and performs pose optimization. In view of the strong structural characteristics of low-texture environments, the local map optimization function Γ is defined according to the point-line feature projection constraint, line feature parallel and vertical constraint, and Manhattan structured constraint: Γ=argmin(A+B+C) (14) Wherein: A is the projection constraint of point-line features, specifically expressed as formula; B is the parallel and vertical constraint of line features, specifically expressed as formula; C is the Manhattan structured constraint, specifically expressed as formula. Where: k c is the key frame related to P and L; k f is an irrelevant key frame; P k 、V k A collection of map points and lines that match the current keyframe.

10. A point-line feature fusion SLAM system in a weak texture environment, characterized in that: The system is used to implement the point-line feature fusion SLAM method in a weak texture environment according to any one of claims 1 to 9, and the system includes a tracking thread module, a local map thread module, and a loop detection thread module; The tracking thread module is used to extract point and line features from the image and estimate the camera's motion pose based on these features; the detected line features are classified into parallel line segments and vertical line segments according to their cosine angles; the strong structural features in low-texture environments are used to optimize the pose of line features to reduce errors. The local map thread module is used to estimate and establish a more accurate Manhattan coordinate system after receiving enough line features. At the same time, it performs Manhattan structure optimization in the local map to reduce posture drift, and then constructs and optimizes the local map. The loop detection thread module is used to detect loops and eliminate the accumulated drift error through pose graph optimization.

Images