A visual SLAM method based on point and line features in low-texture environment

By extracting and screening building structure line features in visual SLAM technology, combining RANSAC algorithm and reprojection error model, the problem of insufficient feature matching in low-texture environments is solved, the accuracy and robustness of the system are improved, and the accuracy and robustness of the pose estimation are achieved. More stable visual SLAM performance.

CN114627309BInactive Publication Date: 2024-11-01CHANGCHUN UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210241165.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-11
Publication Date
2024-11-01
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing visual SLAM technology is difficult to effectively utilize linear information in low-texture environments, resulting in insufficient feature matching and failure of pose estimation, and insufficient system stability and robustness.

Method used

Extract point features and line features in the image, filter out line features of building structures, combine RANSAC algorithm to perform feature matching and error matching removal, use reprojection error model to estimate camera position, and improve system stability through back-end optimization and loopback detection. .

Benefits of technology

It improves the accuracy and robustness of the system's pose estimation in low-texture environments, and enhances the positioning and mapping capabilities of the structured environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114627309B_ABST
    Figure CN114627309B_ABST
Patent Text Reader

Abstract

A visual SLAM method based on point and line features in a low-texture environment relates to the technical field of computer vision. The present invention is extended on the ORB-SLAM2 system, extracts point features and line features in an image, screens out building structure line features based on ordinary line features, and obtains point features and building structure line features; uses point features and building structure line features for feature matching, and uses the RANSAC algorithm to eliminate mismatched pairs; estimates the camera pose based on the reprojection error model derived from adjacent frames; and sequentially executes backend optimization and loop detection threads, thereby improving the stability and robustness of the system in low-texture scenes, and finally realizing a visual SLAM method based on point and line features for low-texture environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and more specifically to a visual SLAM method based on point and line features in a low-texture environment. Background Art

[0002] Simultaneous Localization And Mapping (SLAM) refers to estimating the pose and building a map by acquiring data through sensors in an unknown environment and uncertain position. Visual SLAM uses cameras as sensors to obtain rich geometric, texture, color and other information in the environment, and performs positioning and mapping by capturing feature information in images. As a prerequisite for mobile robots to complete intelligent tasks, visual SLAM technology has become a research hotspot in the field of robotics.

[0003] Although many mature visual SLAM systems have been developed, they still face many problems in practical applications, making it difficult for visual SLAM technology to be truly popularized. For example, most current visual SLAM algorithms use point features for tracking and matching, which means that the rich straight line information in the image is not fully utilized. When the camera is in an environment with missing textures (such as empty rooms and corridors), the number of point features will drop sharply and be unevenly distributed, resulting in camera tracking failure, or the inability to perform pose estimation and positioning due to the lack of sufficient feature matching pairs.

[0004] Compared with point features, line features are a higher-level feature. Line features are not only more robust to changes in lighting and perspective, but also can more intuitively express the geometric information of the scene. In a low-texture environment where feature points are missing, line features can improve the stability and robustness of the system. Summary of the invention

[0005] The purpose of the present invention is to provide a point-line feature-based visual SLAM method in a low-texture environment in view of the shortcomings of the above-mentioned technology, which can improve the stability and robustness of the system in a low-texture environment.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] A visual SLAM method based on point and line features in a low-texture environment, the steps are as follows:

[0008] Step (1) extracting point features and line features from the image, screening out building structure line features based on common line features, and obtaining point features and building structure line features;

[0009] Step (2) performing feature matching based on point features and building structure line features, and using the RANSAC algorithm to eliminate mismatched pairs;

[0010] Step (3) estimating the camera pose based on the reprojection error model derived from adjacent frames;

[0011] Step (4) sequentially executes the backend optimization and loop detection threads, thereby improving the stability and robustness of the system in low-texture scenes, and ultimately realizing a point-line feature-based visual SLAM method for low-texture scenes.

[0012] Preferably, the specific method of step (1) of the present invention is as follows: first, extract the ORB point features and LSD line features in the image, and calculate the binary descriptors of the features, and then, based on the ordinary line features, screen out the building structure line features that conform to the main directions of the Manhattan world.

[0013] Preferably, in the online feature screening stage, the method calculates the dominant direction of the Manhattan world through the vanishing point, and screens the structural line features consistent with the dominant direction in the ordinary line features; in three-dimensional space, a group of parallel straight lines will not intersect, or can be said to intersect at the same infinite point, which may be imaged in the image plane under the perspective projection of the camera, and the imaging point is called the vanishing point. All building structure lines in a dominant direction can be determined by a vanishing point, and the midpoint of the line segment is connected to each vanishing point to obtain the corresponding reference straight line. If the line segment is close to a certain reference straight line, it is considered that the line segment belongs to this vanishing point and belongs to the dominant direction corresponding to this vanishing point.

[0014] Preferably, the specific method of step (2) of the present invention is as follows: using the descriptors of the ORB point features and LSD line features of the extracted image, first use the descriptors to perform feature matching, and then use the RANSAC algorithm to eliminate false matches on this basis.

[0015] Preferably, the method uses the RANSAC algorithm to eliminate false matches. The RANSAC algorithm assumes that the data contains correct data and abnormal data (noise), and also assumes that there is a method to calculate the model parameters that match the correct data. The algorithm has the characteristics of randomness and hypothesis. The randomness is reflected in the random selection of sampled data according to the probability of the correct data appearing, and the randomness simulates the approximate correct results. The hypothesis is reflected in the assumption that the selected sampled data are all correct data, and these data are used to calculate other points to obtain the best model parameters.

[0016] Preferably, in step (3) of the present invention, the camera pose is estimated based on the reprojection error model derived from adjacent frames, and the specific method is as follows:

[0017] By combining the reprojection errors of spatial point features and spatial line features, and adding the reprojection errors of building structure line features, we can get the error sum of all spatial points, spatial lines and structural lines in three-dimensional space, and use this reprojection error sum to solve the camera pose.

[0018] Preferably, the method for sequentially executing the backend optimization and loop detection threads in step (4) of the present invention is as follows:

[0019] Back-end optimization includes local map-based optimization and loop detection-based optimization. The local map optimization first determines whether the current frame is a key frame. If so, a local map associated with the current key frame is established, and then a graph optimization model is established based on the local map to optimize the camera's posture. The loop detection-based optimization first determines whether the current key frame is a loop frame. If so, a posture graph optimization model is constructed based on the loop information to eliminate the camera's accumulated error.

[0020] Preferably, the vanishing point in three-dimensional space is determined using formula (1):

[0021] u T v=0 (1)

[0022] Wherein, u represents a 3×N matrix, N is the number of straight lines, and v represents the 3×1 homogeneous coordinates of the vanishing point.

[0023] Preferably, the direction of the structure line in the world coordinate system is calculated using the following formula (2):

[0024] η∝R wc K -1 v (2)

[0025] Among them, η is the direction of the structure line in the world coordinate system, which is also called the dominant direction, R wc is the rotation matrix from the camera coordinate system to the world coordinate system, K -1 is the inverse of the camera intrinsic parameter.

[0026] Preferably, the vanishing point in the image is calculated using formula (3):

[0027] v i =KR cw η i (3)

[0028] Among them, v i is the vanishing point in the image, K is the camera intrinsic parameter, R cw is the rotation matrix from the world coordinate system to the camera coordinate system, η i is the dominant direction in the world coordinate system.

[0029] Preferably, the inter-frame reprojection error and are calculated using formula (4):

[0030]

[0031] Among them, the k-th frame pose of the camera is T cw,k , the i-th spatial point p observed in the k-th frame w,i The reprojection error is Ep k,i , the jth spatial line L is observed in the kth frame w,j The reprojection error is El k,j , the camera posture maintained in the kth frame is R k , the pose of the camera in the kth frame estimated by the structured line is Z k , ∑p and ∑l represent the observed covariance of points and lines, ρ p , l , s is the Huber robust kernel function.

[0032] Compared with the prior art, the present invention introduces line features based on the ORB-SLAM2 algorithm, and screens out building structure line features containing building structure information based on traditional line features, and uses point features and building structure line features to perform pose estimation, which can improve the accuracy of pose estimation.

[0033] The present invention can combine point features in the environment with building structure line features to perform pose estimation, which can reduce the pose estimation error of the system and improve the robustness of the system in structured, low-texture environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process of the present invention;

[0035] Figure 2 It is a schematic diagram of the structure in the method of the present invention that conforms to the Manhattan world hypothesis;

[0036] Figure 3 Schematic diagram of extracting point features and building structure line features in the method of the present invention:

[0037] Figure 4 Schematic diagram of the map and camera trajectory constructed by the method of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described below in conjunction with the embodiments and drawings, but is not intended to limit the implementation of the present invention.

[0039] like Figure 1 As shown in the figure, a visual SLAM method based on point and line features in a low-texture environment is implemented as follows:

[0040] Step (1) extracting point features and line features from the image, screening out building structure line features based on common line features, and obtaining point features and building structure line features;

[0041] The specific process is as follows:

[0042] Firstly, ORB point features and LSD line features are extracted from the image, and the binary descriptors of the features are calculated. Then, based on the ordinary line features, the building structure line features that conform to the main directions of the Manhattan world are screened out.

[0043] In the online feature screening stage, this method calculates the dominant direction of the Manhattan world through the vanishing point, and screens the structural line features consistent with the dominant direction in the ordinary line features; in three-dimensional space, a group of parallel straight lines will not intersect, or it can be said that they intersect at the same infinite point, which may be imaged in the image plane under the perspective projection of the camera. The imaging point is called the vanishing point. All building structure lines in a dominant direction can be determined by a vanishing point. The midpoint of the line segment is connected to each vanishing point to obtain the corresponding reference straight line. If the line segment is close to a reference straight line, it is considered that the line segment belongs to this vanishing point and belongs to the dominant direction corresponding to this vanishing point.

[0044] Step (2) using point features and building structure line features for feature matching, and using the RANSAC algorithm to eliminate mismatched pairs;

[0045] The specific process is as follows:

[0046] Using the descriptors of the ORB point features and LSD line features extracted from the image, the descriptors are first used for feature matching, and then the RANSAC algorithm is used to eliminate false matches on this basis. This method uses the RANSAC algorithm to eliminate false matches. The RANSAC algorithm assumes that the data contains correct data and abnormal data (noise), and also assumes that there is a method to calculate the model parameters that conform to the correct data. The algorithm has the characteristics of randomness and hypothesis. The randomness is reflected in the random selection of sampled data according to the probability of the correct data appearing. The randomness simulates the approximate correct results. The hypothesis is reflected in the assumption that the selected sampled data are all correct data, and these data are used to calculate other points to obtain the best model parameters.

[0047] Step (3) estimating the camera pose based on the reprojection error model derived from adjacent frames;

[0048] The specific process is as follows:

[0049] By combining the reprojection errors of spatial point features and spatial line features, and adding the reprojection errors of building structure line features, we can get the sum of errors of all spatial points, spatial lines and structural lines in three-dimensional space, and use this sum of reprojection errors to solve the camera pose.

[0050] In step (4), the backend optimization and loop detection threads are executed sequentially, thereby improving the stability and robustness of the system in low-texture scenes, and finally realizing a point-line feature-based visual SLAM method for low-texture scenes.

[0051] The specific process is as follows:

[0052] Back-end optimization includes local map-based optimization and loop detection-based optimization. The local map optimization first determines whether the current frame is a key frame. If so, a local map associated with the current key frame is established, and then a graph optimization model is established based on the local map to optimize the camera's posture. The loop detection-based optimization first determines whether the current key frame is a loop frame. If so, a posture graph optimization model is constructed based on the loop information to eliminate the camera's accumulated error.

[0053] Formula (1) is used to determine the vanishing point in three-dimensional space:

[0054] u T v=0 (1)

[0055] Wherein, u represents a 3×N matrix, N is the number of straight lines, and v represents the 3×1 homogeneous coordinates of the vanishing point.

[0056] The direction of the structural line in the world coordinate system is calculated using the formula:

[0057] η∝R wc K -1 v (2)

[0058] Among them, η is the direction of the structure line in the world coordinate system, which is also called the dominant direction, R wc is the rotation matrix from the camera coordinate system to the world coordinate system, K -1 is the inverse of the camera intrinsic parameter.

[0059] Formula (3) is used to calculate the vanishing point in the image:

[0060] v i =KR cw η i (3)

[0061] Among them, v i is the vanishing point in the image, K is the camera intrinsic parameter, R cw is the rotation matrix from the world coordinate system to the camera coordinate system, η i is the dominant direction in the world coordinate system.

[0062] Formula (4) is used to calculate the inter-frame reprojection error and:

[0063]

[0064] Among them, the k-th frame pose of the camera is T cw,k , the i-th spatial point p observed in the k-th frame w,i The reprojection error is Ep k,i , the jth spatial line L is observed in the kth frame w,j The reprojection error is El k,j , the camera posture maintained in the kth frame is R k , the pose of the camera in the kth frame estimated by the structured line is Z k , ∑p and ∑l represent the observed covariance of points and lines, ρ p , l , s is the Huber robust kernel function.

[0065] Example 1

[0066] A visual SLAM method based on point and line features in a low-texture environment:

[0067] Step 1: First, extract the ORB point features and LSD line features in the image, and calculate the binary descriptors of the features. Then, based on the ordinary line features, screen out the building structure line features that are consistent with the main direction of the Manhattan world. In the online feature screening stage, this method calculates the dominant direction of the Manhattan world through the vanishing point, and screens the structural line features that are consistent with the dominant direction in the ordinary line features.

[0068] Step 2: Using the descriptors of the ORB point features and LSD line features extracted from the image, first use the descriptors for feature matching, and then use the RANSAC algorithm to eliminate false matches based on this.

[0069] Step 3: Comprehensively consider the reprojection errors of spatial point features and spatial line features, and add the reprojection errors of building structure line features on this basis. The sum of errors of all spatial points, spatial lines, and structural lines in three-dimensional space can be obtained, and the camera pose can be solved using this sum of reprojection errors.

[0070] Step 4: Backend optimization includes local map-based optimization and loop detection-based optimization. The local map optimization first determines whether the current frame is a key frame. If so, a local map associated with the current key frame is established, and then a graph optimization model is established based on the local map to optimize the camera's posture. The loop detection-based optimization first determines whether the current key frame is a loop frame. If so, a posture graph optimization model is constructed based on the loop information to eliminate the camera's accumulated error.

[0071] like Figure 1 As shown, this embodiment can improve the accuracy of pose estimation by combining point features in the environment with building structure line features for pose estimation; through back-end optimization and loop detection threads, a globally consistent map is constructed, thereby improving the robustness of the system in structured, low-texture environments.

Claims

1. A visual SLAM method based on point and line features in a low-texture environment, characterized in that: The steps include: Step (1) extracting point features and line features from the image, screening out building structure line features based on common line features, and obtaining point features and building structure line features; The specific process is as follows: Firstly, ORB point features and LSD line features are extracted from the image, and the binary descriptors of the features are calculated. Then, based on the common line features, the building structure line features that conform to the main directions of the Manhattan world are screened out. In the online feature screening stage, this method calculates the dominant direction of the Manhattan world through the vanishing point, and screens the structural line features that are consistent with the dominant direction in the ordinary line features; in three-dimensional space, a group of parallel straight lines will not intersect, or can be said to intersect at the same infinite point, which may be imaged in the image plane under the perspective projection of the camera. The imaging point is called the vanishing point. All building structure lines in a dominant direction can be determined by a vanishing point. The midpoint of the line segment is connected to each vanishing point to obtain the corresponding reference straight line. If the distance between the line segment and a reference straight line is close, it is considered that the line segment belongs to this vanishing point and belongs to the dominant direction corresponding to this vanishing point; Formula (1) is used to determine the vanishing point in three-dimensional space: u T v=0 (1) Where u represents a 3×N matrix, N is the number of straight lines, and v represents the 3×1 homogeneous coordinates of the vanishing point; Step (2) uses point features and building structure line features for feature matching, and uses the RANSAC algorithm to eliminate mismatched pairs; the specific process is as follows: Using the descriptors of the ORB point features and LSD line features extracted from the image, the descriptors are first used for feature matching, and then the RANSAC algorithm is used to eliminate false matches on this basis; the RANSAC algorithm assumes that the data contains correct data and abnormal data (noise), and also assumes that there is a method to calculate the model parameters that conform to the correct data. The algorithm has the characteristics of randomness and hypothesis. The randomness is reflected in the random selection of sampled data according to the probability of the correct data appearing, and the randomness simulates the approximate correct results. The hypothesis is reflected in the assumption that the selected sampled data are all correct data, and these data are used to calculate other points to obtain the best model parameters; Formula (2) is used to calculate the direction of the structural line in the world coordinate system: η∝R wc K -1 v (2) Among them, η is the direction of the structure line in the world coordinate system, which is also called the dominant direction, R wc is the rotation matrix from the camera coordinate system to the world coordinate system, K -1 is the inverse of the camera intrinsic parameter; Formula (3) is used to calculate the vanishing point in the image: in i =KR cw η i (3) Among them, v i is the vanishing point in the image, K is the camera intrinsic parameter, R cw is the rotation matrix from the world coordinate system to the camera coordinate system, η i is the dominant direction in the world coordinate system; Step (3) estimates the camera pose based on the reprojection error model derived from adjacent frames; the specific process is as follows: By combining the reprojection errors of spatial point features and spatial line features, and adding the reprojection errors of building structure line features, we can get the error sum of all spatial points, spatial lines and structural lines in three-dimensional space, and use this error sum to solve the camera pose. Formula (4) is used to calculate the inter-frame reprojection error and: Among them, the k-th frame pose of the camera is T cw,k , the i-th spatial point p observed in the k-th frame w,i The reprojection error is Ep k,i , the jth spatial line L is observed in the kth frame w,j The reprojection error is El k,j , the camera posture maintained in the kth frame is R k , the pose of the camera in the kth frame estimated by the structured line is Z k , ∑p and ∑l represent the observed covariance of points and lines, ρ p , l , s is the Huber robust kernel function; Step (4) sequentially executes back-end optimization and loop detection threads, thereby improving the stability and robustness of the system in a low-texture environment, and ultimately realizing a visual SLAM method based on point and line features in a low-texture environment; the back-end optimization includes local map optimization and loop detection optimization; the local map optimization first determines whether the current frame is a key frame, and if so, establishes a local map associated with the current key frame, and then establishes a graph optimization model based on the local map to optimize the camera's posture; the loop detection optimization first determines whether the current key frame is a loop frame, and if so, constructs a posture graph optimization model based on the loop information, thereby eliminating the camera's accumulated error.

Citation Information

Patent Citations

  • SLAM system rapid relocation algorithm based on line features

    CN110489501A

  • Binocular vision SLAM method based on point-line feature fusion

    CN112509044A