A multi-modal feature association method based on structured feature extraction

Abstract

The application discloses a multi-mode feature correlation method based on structured feature extraction, extracts point and line features from binocular images, estimates plane features through line features, and then constructs correlation between different features by using spatial geometric relations between features. Mainly includes the following steps: first, extract point and line features, then estimate the plane features of intersecting lines according to reliable line features, then construct the correlation between different features by using the spatial geometric relations between features and create a semantic map, finally combine the correlation with the back end to optimize the system. The application improves the algorithm for estimating the plane by using the line feature, gives the mathematical model of the spatial geometric relation between features, and uses the correlation for the back end optimization of the SLAM system, reduces the cumulative error of the system, and provides better positioning capability.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image graphics, and particularly relates to a multi-modal feature association method based on structured feature extraction. BACKGROUND

[0002] Under the weak texture scene SLAM, the SLAM based on feature point matching is prone to tracking failure, resulting in the positioning failure of the SLAM system. The system using line and surface features can reflect the geometric structure in the moving environment, is less affected by noise, and can generate a semantic map that can restore the real scene better. However, the existing SLAM technology can only optimize the system through point and line features, and the optimization of the minimum parameterized features in the back end cannot reliably associate the features and realize the three-dimensional reconstruction with a higher restoration degree.

[0003] In the RGB-D camera and the laser radar, the surface features are usually extracted from the depth image and the laser point cloud, but the binocular camera cannot provide the dense point cloud for directly extracting the plane, and the existing SLAM technology proposes a method of using the intersection line to estimate the plane and generate the dense point cloud, thereby obtaining the mapping effect with a larger amount of information. However, the system lacks the screening of the line features, which affects the accuracy of the estimation of the plane and the precision of the back end optimization. Therefore, how to use a more effective plane estimation algorithm in the binocular SLAM system to realize the feature extraction has important significance. SUMMARY

[0004] To solve the above problems, the application discloses a multi-modal feature association method based on structured feature extraction, improves the algorithm for estimating the plane using the line features, and provides an association method between different types of features, thereby providing a multi-modal feature association model for the back end optimization.

[0005] To achieve the above purpose, the technical scheme of the application is as follows:

[0006] A multi-modal feature association method based on structured feature extraction comprises the following steps:

[0007] (1) extracting point and line features;

[0008] (2) estimating the plane features of the intersection line;

[0009] (3) associating different types of features according to the reliable degree of the feature types.

[0010] (4) combining the association relationship between the features to establish a back end optimization model.

[0011] In step (1), the point and line features are extracted, and the specific steps are as follows:

[0012] (1.1) extracting the ORB feature points of the image according to the gray image;

[0013] (1.2) Extract line features in left and right images using LSD algorithm, and get the line feature candidate set by matching with LBD descriptor.

[0014] (1.3) For each line feature in the candidate set, filter out the false matches according to the projection position of the end points, the actual length of the line segment and the direction vector, etc., to get the line feature with high accuracy.

[0015] In step (2), the plane feature of the intersection line is estimated, and the specific steps are as follows:

[0016] (2.1) Calculate the included angle of the intersection line, and ensure that the included angle is greater than a certain threshold to filter out parallel lines.

[0017] (2.2) Calculate the length of the four line segments between the end points of the two intersection lines, and compare it with the length of itself to ensure that the intersection lines are close to each other.

[0018] (2.3) According to the appropriate intersection line, calculate the normal vector n of the plane π :

[0019] n π =n l1 ×n l2

[0020] Where n l1 and n l2 are the direction vectors of the two intersection lines.

[0021] (2.4) Calculate the distance value of the intersection line end points relative to the estimated plane, and ensure that the four end points are in the same plane. For the plane parameters d i of the end points p i (i=1, 2, 3, 4):

[0022]

[0023] Where n π represents the plane normal vector obtained by the cross product of the intersection line direction vector, represents a 3D point, p ix , p iy , p iz , which corresponds to the x, y, z coordinates.

[0024] (2.5) According to the plane parameters d i , calculate the discrete value D:

[0025] D = Max | d i - d π |, i = 1, 2, 3, 4

[0026] Where d π is d iThe arithmetic mean of the intersection lines. When D is less than the threshold and the intersection lines meet the above conditions, then a plane is defined. .

[0027] (2.6) When D is less than the threshold and the intersection lines meet the above conditions, then define a plane. This yields a candidate plane set.

[0028] in: Describes a plane, a unit vector. d represents the normal vector of the plane. π This represents the distance from the origin.

[0029] In step (3), different types of features are associated based on the reliability of the feature types. The specific steps are as follows:

[0030] (3.1) Associate point features with line features, and calculate the angle between the vectors formed by the lines connecting the points and the endpoints of the lines, i.e., calculate p. i With l j Dot product of vectors at endpoints:

[0031]

[0032] Where p i For point features, p s With p e For line features l j The two endpoints, Let be the vector between two points. If Then it is assumed that point features are related to line features; if And the shortest distance d from the endpoint l If the value is less than a threshold, the point feature is considered to be associated with the line feature. Finally, the output is the set S of the number of points associated with the line feature. l {n1, n2, ..., n j}

[0033] (3.2) Associate point features with surface features. Calculate the distance d between the feature point and the plane and compare it with the theroshold:

[0034]

[0035] Where n π Let be the normal vector of the plane. Describes a 3D point, d π `theroshold` is the distance from the plane to the origin, and `theroshold` is the threshold for the distance of associated points from the plane. Then, the distances between feature points smaller than the threshold and the center point of the intersection lines are calculated, and it is determined whether they are less than the length of any intersection line. If there exists an intersection line whose length is greater than the distance between points, then the point feature is considered to be associated with the surface feature.

[0036] (3.3) Associate line features with plane features. Calculate the distance between the center point of the line and the plane, and filter out the line features with a distance greater than a threshold. Then calculate the distance between the center point and one end point of the line and the center point of the intersecting line, and determine whether it is less than any intersecting line. If there is an intersecting line with a length greater than the distance between the two points, it is considered that the line feature is associated with the plane feature.

[0037] (3.4) Output the number set S of point features and line features associated with the plane feature gp {np1, np2,..., np k} and S gl {nl1, nl2,..., nl k}.

[0038] Where: np k and nl k are the number of point features and line features associated with the plane feature π k .

[0039] In step (4), a back-end optimization model is established based on the association relationship between features, and the specific steps are as follows:

[0040] (4.1) Calculate the feature residual term of point, line and plane features.

[0041] (4.2) Use the point set associated with the line feature set and the point feature set and the line feature number set associated with the plane feature to adjust and optimize the bundle, to obtain a higher precision estimation result.

[0042] (4.3) Perform local BA during tracking to optimize camera pose by reducing re-projection error and structural error.

[0043] (4.4) When it is detected that the system has a loop (arrives at a place that has been passed before), stop local BA and perform global BA instead, and all camera poses and point, line and plane landmarks will be corrected according to the current observation results, to eliminate cumulative error and update key frame pose and global map.

[0044] The beneficial effects of the present application are:

[0045] Most existing binocular SLAM lacks effective means to generate dense point clouds, and usually can only optimize the system using point and line features. The present application estimates the plane using the intersection relationship of line features and generates point clouds, uses the correlation degree between features and the real degree of feature existence, gives a mathematical model of the spatial geometric relationship between features, and uses the spatial geometric relationship between features to construct the association relationship between different features and create a semantic map. While generating a semantic map with high restoration degree, it provides better positioning ability. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A structural diagram of a multi-modal feature association method based on structured feature extraction;

[0047] Figure 2 A flowchart of a feature association model method of a multi-modal feature SLAM system;

[0048] Figure 3 A schematic diagram of a plane generated by intersecting lines;

[0049] Figure 4 A schematic diagram of line feature association points. DETAILED DESCRIPTION

[0050] The present application will be further clarified by the following description and specific embodiments, which should be understood as merely exemplary and not limiting the scope of the present application.

[0051] As Figure 1 shown is a method flowchart of the present application.

[0052] Step S1: Extracting point and line features. Specifically including:

[0053] S1.1, extracting ORB feature points of the image according to the gray image;

[0054] S1.2, using the LSD algorithm to extract line features in the left and right images, and using the LBD descriptor to match to obtain a line feature candidate set.

[0055] S1.3, screening each line feature in the candidate set in the left and right images, deleting false matches according to the projection position of the end point, the actual length of the line segment, and the direction vector, to obtain line features with high accuracy.

[0056] Step S2: Estimating the plane feature of the intersecting line. Specifically including:

[0057] S2.1, calculating the included angle of the intersecting line, ensuring that the included angle is greater than a certain threshold, and screening out parallel lines.

[0058] S2.2, calculating the length of the four line segments between the end points of the two intersecting lines, comparing with the length of itself, and ensuring that the intersecting lines are close to each other.

[0059] S2.3, calculating the normal vector n π of the plane according to the appropriate intersecting line

[0060] n π = n l1 × n l2

[0061] where n l1 and n l2are the direction vectors of the two intersecting lines.

[0062] S2.4, calculate the discrete degree of the distance value of the end points of the intersecting line relative to the estimated plane, ensure that the four end points are in the same plane, for the end point p i the plane parameter d i of (i=1, 2, 3, 4)

[0063]

[0064] where n π represents the normal vector of the plane obtained by the cross product of the direction vectors of the intersecting line, represents a 3D point, p ix , p iy , p iz , that is, the coordinates corresponding to x, y, and z.

[0065] S2.5, according to the plane parameter d

[0066] D=Max|d i -d π |, i=1, 2, 3, 4

[0067] where d π is the arithmetic mean of d i . When D is less than the threshold value and the intersecting line meets the above conditions, the plane

[0068] S2.6, when D is less than the threshold value and the intersecting line meets the above conditions, the plane is defined, thereby obtaining a candidate plane set.

[0069] where: represents a plane, the unit vector represents the normal vector of the plane, d π represents the distance from the origin.

[0070] Step S3: According to the reliability degree of the feature type, different types of features are associated. Specifically, it includes:

[0071] S3.1, associate point features with line features, calculate the angle between the vector formed by the connection line of the point and the end point of the line, that is, calculate the dot product of the vector of p i and the end point of l j :

[0072]

[0073] where p i is a point feature, p s and p e are two end points of the line feature l j , is a vector between two points. If a point feature is considered to be associated with a line feature; if and the shortest distance d l from the point to the line is less than a threshold, the point feature is considered to be associated with the line feature. Finally, the set S l {n1, n2,..., n j} of the number of point features associated with the line feature is output.

[0074] S3.2, associate point features with surface features. Calculate the distance d

[0075]

[0076] where n π is the normal vector of the surface, p represents a 3D point, d π is the distance from the origin to the surface, and the threshold is the threshold distance of the point from the surface. Then calculate the distance between the feature point and the center point of the intersection line that is less than the threshold, and determine whether it is less than the length of any intersection line. If there is an intersection line whose length is greater than the distance between the points, the point feature is considered to be associated with the surface feature.

[0077] S3.3, associate line features with surface features. Calculate the distance from the center point of the line to the surface, and filter out line features whose distance is greater than the threshold. Then calculate the distance between the center point and one of the end points of the line and the center point of the intersection line, and determine whether it is less than the length of any intersection line. If there is an intersection line whose length is greater than the distance between the two points, the line feature is considered to be associated with the surface feature.

[0078] S3.4, output the set S gp {np1, np2,..., np k} of the number of point features associated with the surface feature and the set S gl {nl1, nl2,..., nl k} of the number of line features associated with the surface feature.

[0079] where: np k and nl k are the number of point features and line features associated with the surface feature p k .

[0080] Step S4: establish a back-end optimization model based on the association relationship between features. Specifically, it includes:

[0081] S4.1, calculate the feature residual term of point, line and surface features.

[0082] S4.2, the point set associated with the line feature set is associated with the point feature set of the plane feature associated surface and the line feature number set for bundle adjustment optimization, and a higher precision estimation result is obtained.

[0083] S4.3, local BA is performed during tracking, and camera poses are optimized by reducing re-projection errors and structural errors.

[0084] S4.4, when it is detected that the system has a loop (arriving at a place that has been passed before), the local BA is stopped and global BA is performed instead, all camera poses and point, line and plane landmarks are corrected according to the current observation result, accumulated errors are eliminated, and key frame poses and global maps are updated.

[0085] It should be noted that the above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application. For ordinary skilled persons in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which fall within the protection scope of the claims of the present application.

Claims

1. A multi-modal feature correlation method based on structured feature extraction, characterized in that, The method comprises the following steps: (1) extracting point and line features; the specific steps are: (1.1) extracting ORB feature points of the image according to a gray image; (1.2) extracting line features in left and right images by using an LSD algorithm, and obtaining a line feature candidate set by matching through an LBD descriptor; (1.3) screening each line feature in the candidate set in the left and right images, and deleting false matches according to the projection positions of end points, actual lengths and direction vectors of line segments, to obtain line features with high accuracy; (2) estimating plane features of intersecting lines; the specific steps are: (2.1) calculating the included angles of intersecting lines, and screening out parallel lines to ensure that the included angles are greater than a certain threshold value; (2.2) calculating the lengths of four line segments between end points of two intersecting lines, and comparing the lengths with the lengths of the intersecting lines to ensure that the intersecting lines are close to each other; (2.3) Calculate the normal vector of the plane according to the appropriate intersection line : ； wherein and are the direction vectors of the two intersecting lines; (2.4) Calculate the discrete degree of distance value of the intersection line endpoints relative to the estimated plane, ensure that the four endpoints are in the same plane, for the endpoints 1,2,3,4 plane parameters : ； wherein represents a plane normal vector obtained by cross product of intersection line direction vectors, represents a 3D point, i.e. the coordinates of corresponding (2.5) calculating the discrete value D according to the plane parameters , calculating the discrete value D: ； wherein is the arithmetic mean; when is less than a threshold value and the intersection line meets the above conditions, the plane is defined (2.6) When If the intersection line meets the above conditions, the plane is defined Thus, a candidate plane set is obtained; wherein: represents a plane, the unit vector represents the normal vector of the plane, represents the distance from the origin; (3) correlating different types of features according to the reliability of the feature types; (4) constructing a back-end optimization model and performing pose optimization in combination with the correlation between features.

2. The method of claim 1, wherein the structured feature extraction based multi-modal feature correlation method is characterized by, In the step (3), the different types of features are correlated according to the reliability of the feature types, and the specific steps are: (3.1) Correlate point features with line features, calculate the angle between the vector of the point and the line end point, i.e. calculate and vector dot product of the end point: ； wherein is a point feature, with is a line feature of two end points, is a vector between two points; if , the point feature is considered to be associated with the line feature; if , and the shortest distance to the end points is less than a threshold, the point feature is considered to be associated with the line feature; finally output the set of number of points associated with the line feature ; (3.2) correlating point features and plane features; Computing the distance of a feature point from a plane and Comparison: ； wherein is a normal vector of the plane, represents a 3D point, is the distance of the plane from the origin, is a threshold value for the distance of the associated point from the plane; the distance of the feature point from the center point of the intersection line is then calculated, and it is determined whether it is less than the length of any of the intersection lines; if there is an intersection line whose length is greater than the distance between the points, it is considered that the point feature is associated with the surface feature; (3.3) correlating line features and plane features; calculating the distance from the center point of the line to the plane, and screening out line features greater than a threshold value; secondly, calculating the distances from the center point and one end point of the line to the center point of the intersecting line, and judging whether the distances are less than any intersecting line; if there is an intersecting line with a length greater than the distance between the two points, it is considered that the line feature is correlated with the plane feature; (3.4) outputting a set of point feature and line feature quantities associated with the planar feature and ; wherein: and is the number of point features and line features associated with the planar feature .

3. The method of claim 1, wherein the method further comprises: In the step (4), the different types of features are correlated according to the reliability of the feature types, and the specific steps are: (4.1) calculating feature residual terms of point, line and plane features; (4.2) using the point set associated with the line feature set, the point feature set associated with the plane feature face and the line feature quantity set for bundle adjustment optimization to obtain higher-precision estimation results; (4.3) performing local BA during tracking, and optimizing the camera pose by reducing the re-projection error and the structural error; (4.4) when it is detected that the system has a loop, stopping local BA and executing global BA instead, and all camera poses and point, line and plane landmarks are corrected according to the current observation results, the cumulative error is eliminated, and the key frame pose and the global map are updated.

Images