A point-line feature based visual positioning method against illumination changes

Abstract

The application discloses a kind of anti-light change visual positioning methods based on point line feature, comprising the following steps: S1: using superPoint to detect point feature to original image, LSD detects line feature;S2: the line segment detected is merged and excluded;S3: the association of point feature extracted in step S1 with the line feature after merging in step S2 is established, and the matching of line feature is carried out;S4: the triangulation of line feature is carried out;S5: selecting key frame;S6: construct graph optimization with point constraint and line constraint, output final positioning and trajectory information.The beneficial effects of the present application are: through the image point feature and line feature obtained, and then through the matching and triangulation of points, the matching and triangulation of lines are completed, after selecting corresponding key frame, construct graph optimization with point constraint and line constraint, output final positioning and trajectory information, so as to be applicable to texture details simple, feature sparse environment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a visual positioning method based on point and line features that resists changes in illumination. Background Technology

[0002] Visual localization is a core research area in computer vision. The input consists of various videos or images. By analyzing the feature information of these videos or images and applying an algorithm model, the final result shows the sensor's position and movement trajectory on a map. Visual localization has wide applications in many fields, such as mobile robots, autonomous driving, AR / VR, and 3D reconstruction. Currently, many effective visual localization algorithms exist, but most operate under stable lighting conditions and without sudden environmental changes. They cannot function well in environments with simple textures and few features. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a visual positioning method based on point and line features that is resistant to changes in illumination.

[0004] The objective of this invention is achieved through the following technical solution: a visual positioning method based on point and line features to resist changes in illumination, comprising the following steps:

[0005] S1: Use SuperPoint to detect point features and LSD to detect line features on the original image;

[0006] S2: Merge and exclude the detected line segments;

[0007] S3: Establish the association between the point features extracted in step S1 and the line features merged in step S2, and perform line feature matching.

[0008] S4: Triangulate the line features;

[0009] S5: Select keyframes;

[0010] S6: Construct a graph optimization with point and line constraints, and output the final positioning and trajectory information.

[0011] Preferably, in step S2, the merging condition is:

[0012]

[0013] Where θ is the angle formed by the line segment and the x-axis, p m Let p be the coordinates of the midpoint of the line segment. e The coordinates of the endpoints of the line segment that are closer to the line segment to be merged.

[0014] After merging, remove those with length values ​​less than δ. l The line segment.

[0015] Preferably, in step S2, both the merging and removing operations are performed twice, with the first δ... θ The value is 0.05, the second δ θ The first δ is 0.03. d The value is 5, and the second δ is... d The value is 3, the first δ ep The value is 15, the second δ ep The value is 50, the first δ l The value is 30, and the second δ is... l It is 60.

[0016] Preferably, in step S3, the point feature p i =(x i ,y i Line feature l j =(A j B j C j ,x j1 ,y j1 ,x j2 ,y j2 ), of which (A j B j C j ) is the linear parameter, (x j1 ,y j1 ,x j2 ,y j2 Let p be the two endpoints of the line segment. i With l j Distance between:

[0017]

[0018] When d ij <3 and p i If the projection on the coordinate axes is onto the projection of the line segment, then p i Belongs to l j ;

[0019] Calculate the line feature l on image k m and line features l on image k+1 n Confidence score:

[0020]

[0021] Where, N pm For l m and l n The number of points matched on N m For l m The number of point features, N n For ln The number of point features on S, when S mn Greater than δ s And N pm Greater than δ N Then l m and l n Match successful.

[0022] Preferably, in step S3, δ s It is 0.8, δ N The value is 2.

[0023] Preferably, step S4 further includes the following step:

[0024] S41: Represent a 3D spatial line using Plück coordinates.

[0025]

[0026] Where v is the direction vector of the line, n is the normal vector determined by the line and the origin, and 6 is the degree of freedom vector;

[0027] S42: Triangulate the line features.

[0028]

[0029] X1 and X2 are two 3D points on the line features that have been triangulated.

[0030] Preferably, in step S5, when the time elapsed since the last keyframe has passed... The angle difference between the pose of the frame and the previous keyframe exceeds Or the number of tracked map points is greater than the number located in and If one of the following conditions is met, then the current frame is a keyframe.

[0031] Preferably, in step S5, It is 10. It is 0.15. It is 25. It is 50.

[0032] Preferably, in step S6, the point constraint is:

[0033] E p =x p -π(RX p +t);

[0034] Where, x p A point with 3D coordinates, X p For the observation in the current frame, π is the camera intrinsic parameter projection, R is the rotation part of the pose in the current frame, and t is the translation part of the pose in the current frame.

[0035] The line constraints are:

[0036] E l =e(l l ,π(HL i )P;

[0037] e(l l ,l p )=[d(x l1 ,l p ),d(x l2 ,l p )] T ;

[0038]

[0039] Where d(x,l) is the distance from point x to line l, x l1 With x l2 For the line l l The two endpoints.

[0040] The present invention has the following advantages: By acquiring image point features and line features, the present invention completes line matching and triangulation through point matching and triangulation. After selecting the corresponding keyframes, it constructs a graph optimization with point constraints and line constraints, and outputs the final positioning and trajectory information, thus making it applicable to environments with simple texture details and few features. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the visual positioning method process. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0045] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0046] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0048] In this embodiment, as Figure 1 As shown, a visual localization method based on point and line features to resist changes in illumination includes the following steps:

[0049] S1: Use SuperPoint to detect point features and LSD to detect line features on the original image;

[0050] S2: Merge and exclude the detected line segments;

[0051] S3: Establish the association between the point features extracted in step S1 and the line features merged in step S2, and perform line feature matching.

[0052] S4: Triangulate the line features;

[0053] S5: Select keyframes;

[0054] S6: Construct a graph optimization with point and line constraints, and output the final localization and trajectory information. By acquiring image point and line features, and then performing point matching and triangulation to complete line matching and triangulation, after selecting the corresponding keyframes, a graph optimization with point and line constraints is constructed, outputting the final localization and trajectory information. This approach is applicable to environments with simple texture details and sparse features. Specifically, in step S1, superPoint is used to detect point features, and LSD is used to detect line features, but feature descriptors for line segments are not extracted. In step S3, the main purpose of establishing the association between the point features extracted in step S1 and the line features merged in step S2 is to overcome lighting variations and weak texture scenes.

[0055] Furthermore, in step S2, the merging condition is:

[0056]

[0057] Where θ is the angle formed by the line segment and the x-axis, p m Let p be the coordinates of the midpoint of the line segment. e The coordinates of the endpoints of the line segment that are closer to the line segment to be merged.

[0058] After merging, remove those with length values ​​less than δ. l The line segment. Furthermore, in step S2, both the merging and removing operations are performed twice; the first δ... θ The value is 0.05, the second δ θ The first δ is 0.03. d The value is 5, and the second δ is... d The value is 3, the first δ ep The value is 15, the second δ ep The value is 50, the first δ l The value is 30, and the second δ is... l It is 60.

[0059] In this embodiment, in step S3, the point feature p i =(x i ,y i Line feature l j =(A j B j C j ,x j1 ,y j1 ,x j2 ,y j2 ), of which (A j B j C j ) is the linear parameter, (x j1 ,y j1 ,x j2 ,y j2Let p be the two endpoints of the line segment. i With l j Distance between:

[0060]

[0061] When d ij <3 and p i If the projection on the coordinate axes is onto the projection of the line segment, then p i Belongs to l j ;

[0062] Calculate the line feature l on image k m and line features l on image k+1 n Confidence score:

[0063]

[0064] Where, N pm For l m and l n The number of points matched on N m For l m The number of point features, N n For l n The number of point features on S, when S mn Greater than δ s And N pm Greater than δ N Then l m and l n Match successful. Preferably, in step S3, δ s It is 0.8, δ N The value is 2.

[0065] In this embodiment, step S4 further includes the following step:

[0066] S41: Represent a 3D spatial line using Plück coordinates.

[0067]

[0068] Where v is the direction vector of the line, n is the normal vector determined by the line and the origin, and 6 is the degree of freedom vector;

[0069] S42: Triangulate the line features.

[0070]

[0071] X1 and X2 are two 3D points on the line features that have been triangulated.

[0072] In this embodiment, in step S5, when the time elapsed since the previous keyframe has passed... The angle difference between the pose of the frame and the previous keyframe exceeds Or the number of tracked map points is greater than the number located in and If one of the following conditions is met, the current frame is a keyframe. Specifically, because continuous image information is redundant and requires a large amount of computation, keyframes are used to filter the input image. Further, in step S5, It is 10. It is 0.15. It is 25. It is 50.

[0073] In this embodiment, in step S6, the point constraint is:

[0074] E p =x p -π(RX p +t);

[0075] Where, x p A point with 3D coordinates, X p For the observation in the current frame, π is the camera intrinsic parameter projection, R is the rotation part of the pose in the current frame, and t is the translation part of the pose in the current frame.

[0076] The line constraints are:

[0077] E l =e(l l ,π(HL i ));

[0078] e(l l ,l p )=[d(x l1 ,l p ),d(xl2,l p )] T ;

[0079]

[0080] Where d(x,l) is the distance from point x to line l, x l1 With x l2 For the line l l The two endpoints.

[0081] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual positioning method for resisting illumination changes based on point and line features, characterized in that: Includes the following steps: S1: Use SuperPoint to detect point features and LSD to detect line features on the original image; S2: Merge and exclude the detected line segments; S3: Establish the association between the point features extracted in step S1 and the line features merged in step S2, and perform line feature matching; S4: Triangulate the line features; S5: Select keyframes; S6: Construct a graph optimization with point and line constraints, and output the final positioning and trajectory information; In step S2, the merging condition is: ； in, The angle formed by the line segment and the x-axis. Let be the coordinates of the midpoint of the line segment. The coordinates of the endpoints of the line segment that are closer to the line segment to be merged. After merging, remove those with length values ​​less than The line segment.

2. The visual positioning method for resisting illumination changes based on point and line features according to claim 1, characterized in that: In step S2, both the merging and removing operations are performed twice, the first time... It was 0.05, the second time. The value was 0.03, the first time. It is 5, the second time. For 3, the first time It is 15, the second time. It's 50, the first time. For 30, the second time It is 60.

3. The visual positioning method for resisting illumination changes based on point and line features according to claim 2, characterized in that: In step S3, point features Line features ,in For linear parameters, Given the two endpoints of the line segment, calculate and Distance between: ； when and The projection on the coordinate axes onto the projection of the line segments is then belong ; Calculate the line features on image k Line features on image k+1 Confidence score: ； in, for and The number of points matched on the top, for Number of point features on, for The number of point features on the surface, when Greater than and Greater than ,but and Match successful.

4. The visual positioning method for resisting illumination changes based on point and line features according to claim 3, characterized in that: In step S3 It is 0.

8. The value is 2.

5. The visual positioning method for resisting illumination changes based on point and line features according to claim 4, characterized in that: Step S4 also includes the following steps: S41: Represent a 3D spatial line using Plück coordinates. ； in, The direction vector of the line. 6 is the normal vector determined by the line and the origin, and 6 is the vector of degrees of freedom. S42: Triangulate the line features. ； in, and For 3D points that have been triangulated on two line features.

6. The visual positioning method for resisting illumination changes based on point and line features according to claim 5, characterized in that: In step S5, when the time has passed since the previous keyframe... The angle difference between the pose of the frame and the previous keyframe exceeds Or the number of tracked map points is greater than the number located in and If one of the following conditions is met, then the current frame is a keyframe.

7. The visual positioning method for resisting illumination changes based on point and line features according to claim 6, characterized in that: In step S5 It is 10. It is 0.

15. It is 25. It is 50.

8. The visual positioning method for resisting illumination changes based on point and line features according to claim 7, characterized in that: In step S6, the point constraint is: ； in, A point with 3D coordinates. For the observations in the current frame, Projecting camera intrinsic parameters. The rotation part of the current frame pose. This represents the translation portion of the current frame pose; The line constraints are: ； ； ； in, For point to the straight line distance, and It is a straight line The two endpoints.

Images