Positioning methods and equipment

Abstract

This invention relates to the field of positioning technology, and more particularly to a positioning method and device. The method includes: determining two-dimensional line segments and visible points in each frame of images acquired by a camera, wherein the visible points include the endpoints of the two-dimensional line segments and interior points between the endpoints; fitting a three-dimensional line based on the two-dimensional line segments and determining the collinearity constraints of the two-dimensional line segments; constructing a cost objective function based on the photometric errors of the visible points in each frame of images, the photometric errors of the two-dimensional line segments in each frame of images, and the collinearity constraints; and minimizing the cost objective function to determine the pose parameters of the camera. This invention proposes a direct visual odometry scheme based on points and lines. This scheme extends the photometric error with respect to lines and adds collinearity constraints on points on the lines, thereby significantly reducing the number of variables to accelerate the optimization process and ensuring that the collinearity constraints are satisfied.

Description

Technical Field

[0001] This invention relates to the field of positioning technology, and in particular to a positioning method and device. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a fundamental module in many robotics and computer vision applications, used to achieve autonomous navigation or augmented reality (AR). For lightweight SLAM systems without closed-loop components, it is also called Visual Odometry (VO). In resource-constrained embedded devices, VO can play a crucial role in real-time pose estimation scenarios. Visual localization methods for SLAM or VO typically include feature point methods and direct methods. Feature point methods have long been a research hotspot in this field. Recent studies have shown that direct methods also exhibit high accuracy and robustness. Therefore, how to better achieve visual localization based on direct methods has become a problem that needs to be solved. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a positioning method and device, and propose a direct visual odometry scheme based on points and lines. This scheme extends the photometric error with respect to lines and adds collinearity constraints on points on the line, thereby significantly reducing the number of variables to accelerate the optimization speed and ensuring that the collinearity constraints are satisfied.

[0004] In a first aspect, embodiments of the present invention provide a positioning method, including:

[0005] Determine the two-dimensional line segments and visible points of each frame image captured by the camera, wherein the visible points include the endpoints of the two-dimensional line segments and the interior points between the endpoints;

[0006] Fit a three-dimensional straight line based on the two-dimensional straight line segment and determine the collinearity constraint of the two-dimensional straight line segment;

[0007] Based on the photometric error of the visible points of each frame image, the photometric error of the two-dimensional straight line segments of each frame image, and the collinearity constraint, a cost objective function is constructed.

[0008] Minimize the cost objective function to determine the camera pose parameters.

[0009] Optionally, the photometric error of the two-dimensional straight line segment is determined based on the photometric errors of the endpoints and interior points of the two-dimensional straight line segment.

[0010] Optionally, determining the photometric error at any interior point x of the two-dimensional straight line segment includes:

[0011] Determine the three-dimensional point P by the x-back projection of the interior point of the two-dimensional line segment. x ;

[0012] Determine the three-dimensional point P x At the projection point x′ of the target frame image;

[0013] Based on the projection point x′, determine the photometric error of the interior point x of the two-dimensional straight line segment.

[0014] Optionally, the three-dimensional point P is the result of the back projection of the interior point x on the two-dimensional line segment. x It is determined based on the depth of the endpoints of the corresponding two-dimensional straight line segment.

[0015] Optionally, the three-dimensional straight line fitted by the two-dimensional straight line is represented by Plück coordinates, and the first and second parameters of the Plück coordinates are both determined based on the endpoint depth of the corresponding two-dimensional straight line segment.

[0016] Optionally, the step of fitting a three-dimensional line based on the two-dimensional line segment and determining the collinearity constraint of the two-dimensional line segment includes:

[0017] Based on the support region width of the two-dimensional line segment under the current camera posture and the fitable three-dimensional line group, establish the distance function from the three-dimensional endpoint of the back projection of the two-dimensional line segment to each three-dimensional line in the fitable three-dimensional line group.

[0018] Based on the distance function, the collinearity constraint of the two-dimensional line segment is determined.

[0019] Optionally, determining the two-dimensional line segments of each frame image captured by the camera includes:

[0020] A sliding window strategy is used to traverse the image data stream acquired by the camera and identify two-dimensional line segments from each frame image contained in the current sliding window.

[0021] Optionally, minimizing the cost objective function to determine the camera pose parameters includes:

[0022] Based on the known camera pose parameters and the determined endpoint depths of the two-dimensional straight line segment, the three-dimensional straight line is fitted by minimizing the collinearity constraint in the cost objective function;

[0023] Under the collinearity constraint, the camera pose parameters when the camera captures each frame of an image are determined by minimizing the photometric error of the visible point, the photometric error of the two-dimensional line segment, and the collinearity constraint by adjusting the camera pose parameters and the endpoint depth of the two-dimensional line segment.

[0024] Optionally, based on the known camera pose parameters and the endpoint depths of the determined two-dimensional line segment, the three-dimensional line is fitted by minimizing the collinearity constraint in the cost objective function, including:

[0025] Based on the known camera pose parameters and the determined endpoint depths of the two-dimensional line segments, determine the distances from the three-dimensional points of the back projections of the related two-dimensional line segments in each frame image to the selectable three-dimensional line.

[0026] By minimizing the distance, a three-dimensional line corresponding to each two-dimensional line segment with a correlation is fitted.

[0027] Optionally, the method further includes: when minimizing the distance, if the calculated minimum distance is greater than a set value, then the association relationship of the corresponding two-dimensional line segment is released;

[0028] If the calculated minimum distance is less than or equal to the set value, then the corresponding two-dimensional line segment is back-projected onto the same three-dimensional line, and the corresponding two-dimensional line segments are connected and merged.

[0029] Secondly, embodiments of the present invention provide a positioning device, comprising:

[0030] The first determining module is used to determine the two-dimensional line segments and visible points of each frame image captured by the camera, wherein the visible points include the endpoints of the two-dimensional line segments and the interior points between the endpoints;

[0031] The collinearity constraint module is used to fit a three-dimensional line to the two-dimensional line segment and determine the collinearity constraint of the two-dimensional line segment;

[0032] The cost module is used to construct a cost objective function based on the photometric error of the visible points of each frame image, the photometric error of the two-dimensional straight line segments of each frame image, and the collinearity constraint.

[0033] The second determining module is used to minimize the cost objective function to determine the pose parameters of the camera.

[0034] Thirdly, embodiments of the present invention provide an electronic device, including: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method described in any of the first aspects by calling the program instructions.

[0035] Fourthly, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method described in any of the first aspects. Attached Figure Description

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1-a A schematic diagram of collinear points identified in different scenarios;

[0038] Figure 1-b A schematic diagram illustrating the complexity introduced by collinear constraints in related technologies;

[0039] Figure 2 A flowchart of a positioning method provided in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a direct dot-line model provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram illustrating the introduction of collinearity constraints on three-dimensional straight lines using related technologies.

[0042] Figure 5 A schematic diagram of a three-dimensional straight line L, a sliding window, and Hessian matrices of different models provided in an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of merging two-dimensional line segments provided in an embodiment of the present invention;

[0044] Figure 7 A comparative schematic diagram of different visual odometers provided in an embodiment of the present invention;

[0045] Figure 8 A comparative schematic diagram of different visual odometers provided in an embodiment of the present invention;

[0046] Figure 9 This is a schematic diagram of the structure of a positioning device provided in an embodiment of the present invention;

[0047] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] The original photometric error is defined for individual points, making it difficult to apply to lines. Furthermore, collinearity constraints between points on a line are either ignored or introduce heavy computational burdens. This invention proposes a direct visual odometry scheme based on points and lines. This scheme extends the photometric error for lines by adding collinearity constraints for points on the line. Depth estimation of collinear points is adjusted by combining collinearity constraints with photometric errors. This significantly reduces the number of variables to accelerate optimization and ensures that collinearity constraints are satisfied.

[0050] In man-made scenes, points and lines are widely distributed. Direct visual SLAM, or VO, based on features, points, and lines, is being extensively studied.

[0051] Line matching: One of the challenges of direct visual localization based on lines is line matching. Among related algorithms, descriptor-based line matching, such as line matching based on line band descriptors (LBD), is widely used. Related technologies have also proposed line detection schemes based on line segment detectors (LSD) and deep learning. However, line segments observed by the camera may partially enter or leave the camera's field of view during camera movement, causing changes in the appearance of the corresponding line segments. These appearance changes may lead to line matching failures. To improve line matching performance, this invention employs a line tracking approach based on spatial and temporal coherence. This approach samples several points from a straight line and tracks these points using optical flow, thereby establishing line associations between related line segments without introducing additional computation time.

[0052] Feature-based methods: Feature-based SLAM may experience performance degradation in low-texture regions. In low-texture environments, lines can be used to supplement point-based methods, such as the ORB-SLAM algorithm. However, this strategy can significantly increase the computational load on both the front-end and back-end. In recent years, significant progress has been made in techniques for extracting 3D information from images using deep learning. 3D depth information obtained through deep learning has been incorporated into traditional SLAM algorithms. However, these algorithms require relatively powerful graphics processing units (GPUs), making them less suitable for embedded systems with limited computing resources.

[0053] Direct methods: Direct visual odometry (DMO) is based on pixel depth and calculates camera pose by minimizing the photometric error of pixels. Photometric error typically refers to the difference in intensity between pixels in the reference and target images. In related techniques, photometric error is defined for pixels and is difficult to generalize to lines. Furthermore, collinearity constraints in related techniques are mainly used to adjust depth estimation. Combining collinearity constraints with photometric error is also a problem that needs to be overcome. For example, related techniques combine collinearity constraints with photometric error to optimize 3D lines with keyframe poses and points, but this method cannot guarantee that points projected onto the target image fully satisfy the collinearity constraints and increases computational cost. In addition, 3D lines are initialized with the depth of collinear points. If the depth quality of collinear points is poor, the 3D lines cannot be initialized and therefore cannot be used to normalize the depth estimation of collinear points. Furthermore, 3D lines in the back-projection plane are determined by the first observed 2D lines, which are characterized by two parameters. While this reduces the number of unknown parameters, accelerates optimization, and avoids singularities in some parameters, estimation errors in the 2D lines may lead to suboptimal results.

[0054] In direct methods, collinear points are often used to handle textureless problems. However, the correlation between collinear points is usually ignored, which can lead to large-scale least-squares problems and reduce computational accuracy. See also Figure 1-a As shown, collinear points dominate corner features in different scenarios. In related techniques, each point in a collinear network introduces a separate depth, leading to a large-scale least-squares problem. Furthermore, due to the ambiguity of the matching, the depth of collinear 2D points may be inaccurate. Figure 1-b As shown, simply introducing collinearity constraints may further increase computational complexity. Therefore, this embodiment of the invention will focus on illustrating how to introduce collinearity constraints into the direct algorithm to improve accuracy and reduce computational complexity.

[0055] To address the aforementioned problems in related technologies, this invention proposes a direct visual odometry scheme based on points and lines. This scheme extends the photometric error regarding lines and adds collinearity constraints on points along the line, thereby significantly reducing the number of variables to accelerate optimization and ensuring that the collinearity constraints are satisfied. Furthermore, this scheme employs a two-step method to accelerate the optimization of the cost objective function with minimal accuracy loss.

[0056] See Figure 2 This is a flowchart illustrating a positioning method provided in an embodiment of the present invention. This method is a direct visual odometry method based on points and lines. Figure 2 As shown, the processing flow of this method includes:

[0057] 101. Determine the two-dimensional line segments and visible points of each frame image captured by the camera, wherein the visible points include the endpoints of the two-dimensional line segments and the interior points between the endpoints.

[0058] In this embodiment of the invention, a sliding window strategy is used to traverse the image data stream acquired by the camera. Each of the aforementioned frames can be multiple frames contained within the sliding window. The number of frames contained in the sliding window can be determined according to actual needs, such as 3, 4, or 5 frames, etc., which will not be listed here. Furthermore, each of the aforementioned frames can be a keyframe image contained within the sliding window. A keyframe refers to an image frame where the camera meets certain motion conditions or scene changes.

[0059] Furthermore, the visible points identified from the aforementioned image frames can be pixels with a gradient greater than a certain value. Optionally, the visible points can include the endpoints of the identified two-dimensional line segments and the interior points between the endpoints. Optionally, embodiments of the present invention can identify two-dimensional line segments in each image frame using the LSD method.

[0060] 102. Fit a three-dimensional line to a two-dimensional line segment and determine the collinearity constraint of the two-dimensional line segment.

[0061] Optionally, the 3D points projected from the interior points of a 2D line segment are determined based on the endpoint depths of the corresponding 2D line segment. Therefore, the 3D line fitted from the 2D line can be determined using the endpoint depths of the 2D line segment. Specifically, the 3D line fitted from the 2D line is represented using Plück coordinates, where the first and second parameters are determined based on the endpoint depths of the corresponding 2D line segment. Optionally, for any 3D line, it can be determined based on the endpoint depths of the first observed 2D line segment. Subsequently observed 2D line segments with the same parameters can be considered collinear. This collinearity constraint restores the four degrees of freedom of the 3D line. Given the camera pose and the endpoint depths of the 2D line segments, it minimizes the distance from the projected points of the 2D line segments to the 3D line, ensuring that all 2D line segments corresponding to the same line target in space are collinear.

[0062] 103. Based on the photometric error of visible points in each frame of the image, the photometric error of two-dimensional straight line segments in each frame of the image, and the collinearity constraint, construct the cost objective function.

[0063] In this embodiment of the invention, the photometric error of points is extended to the photometric error of lines (i.e., the photometric error of two-dimensional straight line segments) in the constructed cost objective function, and collinearity constraints are added. This can significantly reduce the number of variables to speed up the optimization process and ensure that the collinearity constraints are met.

[0064] 104. Minimize the cost objective function to determine the camera pose parameters. Optionally, this embodiment of the invention employs a two-step method to minimize the cost objective function, thereby accelerating the optimization process and reducing the loss of accuracy. The two-step method for minimizing the cost objective function includes: (1) fitting a three-dimensional straight line by minimizing the collinearity constraint in the cost objective function based on the known camera pose parameters and the determined endpoint depth of the two-dimensional straight line segment. (2) under the premise of satisfying the collinearity constraint, minimizing the photometric error of the visible point, the photometric error of the two-dimensional straight line segment, and the collinearity constraint by adjusting the camera pose parameters and the endpoint depth of the two-dimensional straight line segment, thereby determining the camera pose when the camera captures each frame of images.

[0065] See Figure 3 This is a schematic diagram of a direct dot-line model provided in an embodiment of the present invention. Figure 3 As shown, assume the camera captures the Ith... i and I j A frame image. Where Ω represents the image domain. The camera captures the I-th frame image. i Frame image to I j The rotation matrix of a frame image is denoted by R, and the translation vector by t, or more simply, by T. Here, R ∈ SO3, where SO3 represents a rotation group. t ∈ R 3 , R 3 Let T represent three-dimensional space. T∈SE(3), where SE(3) represents a special Euclidean group. The camera captures the I-th... i and I j The poses of the frames are T i and T j Assume x∈Ω is I i A point in the system, whose depth is inversely proportional to d. Assume x is also within the system of I. j It is observed at x′∈Ωj. Therefore, the relationship between x and x′ is expressed as formula (1):

[0066]

[0067] Among them, Π c :R 3 →Ω represents the projection function that projects three-dimensional space onto the image domain.

[0068] Ω×R→R 3 , which represents the back projection function that back projects the image domain to three-dimensional space.

[0069] R ij Indicates from the Ith i and I j The rotation matrix of the frame image, t ij Indicates from the Ith i and I jThe translation vector of the frame image.

[0070] The photometric error for points x and x′ is expressed by formula (2):

[0071]

[0072] Among them, t i and t j Indicates the Ith i and I j The exposure time of a frame image. i a j b i and b j This represents the affine brightness transformation parameters. x N represents the weighting factor associated with the gradient. x Let γ represent a set of neighborhood points of point x. γ is the Huber norm.

[0073] The three-dimensional line projected from the back projection of a two-dimensional line segment can be represented using Plück coordinates. The first and second parameters of the Plück coordinates can be determined based on the endpoint depths of the corresponding two-dimensional line segment. Specifically, for example... Figure 3 As shown, the three-dimensional straight line L l Including points p1 and p2, the Plück coordinates of a three-dimensional line are expressed by formula (3):

[0074]

[0075] Where × represents the cross product. Plück coordinates are homogeneous coordinates. m and d are the first and second parameters of the Plück coordinates, respectively. Here, d is normalized, which makes the subsequent formulas simpler. Using the Plück coordinates in (3), the distance from point p to L can be expressed as formula (4):

[0076] e(L, p) = mp × d.

[0077] In this embodiment of the invention, the three-dimensional point P is the back projection of the interior point x on the two-dimensional line segment. x Determined based on the depth of the endpoints of the corresponding two-dimensional straight line segment. For example... Figure 3 As shown, assume that x1 and x2 are the two endpoints of the two-dimensional line segment l, and the depths of endpoints x1 and x2 are d1 and d2, respectively. The three-dimensional points of the back projection of endpoints x1 and x2 are p1 and p2, respectively. The representation of the three-dimensional points p1 and p2 is given in formula (5):

[0078]

[0079] Based on formula (3), the Plück coordinates L of the three-dimensional line determined by p1 and p2 can be calculated. l This can be expressed as formula (6):

[0080] L l =[m l ;d l ].

[0081] Among them, L l It is a function of d1 and d2.

[0082] like Figure 3 As shown, assume x is an interior point of a two-dimensional line segment l, distinct from its endpoints x1 and x2. Given camera pose parameters K, x is back-projected onto the three-dimensional line L. l The back projection line L x Having directional parameter d x Represented as formula (7):

[0083]

[0084] in, Let x be the homogeneous coordinate of x. The point x is back-projected onto the 3D line L. l The back projection point P x The closed-form solution is expressed as formula (8):

[0085]

[0086]

[0087] Among them, [d l ] × and [d x ] × d is determined according to formulas (6) and (7), respectively. l and d x A slant matrix.

[0088] This invention extends the photometric error of points to the photometric error of lines. The line photometric error of a two-dimensional straight line segment includes: determining the endpoints of the two-dimensional straight line segment and several interior points between the endpoints; calculating the photometric errors of the endpoints and interior points of the two-dimensional straight line segment respectively; and determining the line photometric error of the two-dimensional straight line segment based on the photometric errors of the endpoints and interior points.

[0089] like Figure 3 As shown, assume the Ith i Frame and I j Three-dimensional line L was observed in all images. l 3D line L l In the I iThe frame image is projected as l, with endpoints x1 and x2, and inverse depths d1 and d2 for x1 and x2, respectively. According to the above description, the three-dimensional points projected onto the three-dimensional line from l are determined. Therefore, different formulas can be used to calculate the photometric errors of the endpoints and interior points of the two-dimensional line segment. For the interior point x, formula (8) can be used to calculate its projection onto the three-dimensional line L. l P x P x Project to I j Let the point be x′. x′ can be expressed as formula (9):

[0090] x′=Π c (R ij p x +t ij ).

[0091] Substituting formula (9) into formula (2), the photometric error of the interior point x of the two-dimensional straight line segment can be calculated. For endpoints x1 and x2, the photometric errors of endpoints x1 and x2 can be calculated using formula (2). and Assuming that several interior points of line l form a set X, the photometric error formula for the two-dimensional line segment 1 is expressed as formula (10):

[0092]

[0093] Among them, E lj It depends on the inverse depths of x1 and x2, and is independent of the number of interior points x.

[0094] Based on the above description, this embodiment of the invention incorporates two-dimensional straight line segments into the photometric error. Since a straight line is an unbounded object, it can exist within the camera's field of view for a considerable period. Therefore, this embodiment of the invention establishes line associations by tracing points on the straight line and introduces collinearity constraints to adjust the depth of the endpoints.

[0095] Specifically, by adjusting the endpoint depth of each two-dimensional line segment, the photometric error about the point is minimized to obtain the line association. The line association refers to establishing a relationship between line segments corresponding to the same object in each frame of the image. After obtaining the line association, collinearity constraints can be applied to the points on the two-dimensional line segments. In this embodiment of the invention, the three-dimensional line is determined by the back projection of the first observed two-dimensional line segment. Estimation errors of the two-dimensional line segments may lead to suboptimal solutions. This embodiment of the invention recovers the four degrees of freedom of the three-dimensional line and represents the three-dimensional line in the world coordinate system.

[0096] Suppose a three-dimensional straight line with Plückian coordinates L lies at M. l Under the pose, it was observed that its indexes form a set of O LOptionally, the transformation matrix in this pose can be represented as T. i , i∈O L In T i Under the given posture, the two-dimensional line segment is back-projected onto the three-dimensional line L at two points p. i，1 and p i，2 Among them, p can be calculated according to formula (5). i，1 and p i，2 .use p i，1 and p i，2 Convert the coordinates of p to global coordinates. i，1 and p i，2 The global coordinates are represented as q i，1 and q i，2 This embodiment of the invention employs the LSD algorithm for line detection. Due to noise, quantization, or motion blur, each line detected by the LSD algorithm is associated with a line support region. Let ρ i Represents a two-dimensional straight line segment l i Supported area width. ρ i It can reflect l i The uncertainty of the endpoints, in turn, affects p on the three-dimensional line. i，1 and p i，2 The uncertainty of ρ. i The larger the value, the higher the uncertainty at the upper endpoint of the three-dimensional line. Therefore, 1 / ρ can be used. i To measure collinearity constraints. Based on formula (4), the collinearity constraint of the three-dimensional line L can be described as follows: Based on the support region width of the two-dimensional line segment under the current camera pose and the fitable three-dimensional line group, establish the distance function from the three-dimensional endpoint of the back projection of the two-dimensional line segment to each three-dimensional line in the fitable three-dimensional line group; determine the collinearity constraint of the two-dimensional line segment based on the distance function. Optionally, the collinearity constraint can be expressed by formula (11):

[0097]

[0098] In summary, considering the photometric errors of points and lines, as well as the collinearity constraints of line associations, the cost objective function for points, lines, and frames constructed based on formulas (2), (10), and (11) can be expressed as formula (12):

[0099]

[0100] In formula (12), i represents traversing all frames. Let i represent the set of all visible points in the i-th frame. Let represent the set of all two-dimensional line segments in the i-th frame. Let obs(p) represent the set of frames visible to point p, and obs(l) represent the set of frames visible to two-dimensional line segment l. It is a set of three-dimensional straight lines. For obs(l), if any sampling point on the two-dimensional straight line segment l is visible in the j-th frame of the image, then the two-dimensional straight line segment l is also visible in the j-th frame of the image. In formula (12), the first term represents the photometric error of the point, the second term represents the photometric error about the line, and the third term represents the collinearity constraint.

[0101] Furthermore, the role of collinearity constraints differs in related technologies and in the embodiments of this invention. For example... Figure 3 As shown, the three-dimensional point P is the result of the back projection of the interior point on the two-dimensional line segment l. x The depths are determined by the reverse directions of the endpoints x1 and x2 of the two-dimensional line segment, thus avoiding the introduction of new variables. P1, P x and P2, as well as P1, P x and P2 in the I j The projected points x1′, x′, and x2′ in the frame image are completely collinear. In fact, regardless of how many points are sampled from the two-dimensional line segment l, the number of variables does not increase, and the collinearity requirement is satisfied. In this embodiment of the invention, by minimizing the photometric error about the line, each two-dimensional line segment in the keyframe corresponds to a three-dimensional line. Collinearity constraints ensure that the three-dimensional lines corresponding to the two-dimensional line segments of the same object in the corresponding space remain consistent. That is, even without collinearity terms, the three-dimensional points back-projected from each two-dimensional line segment still satisfy collinearity. In related technologies, collinearity constraints are used to ensure that the three-dimensional points corresponding to two-dimensional line segments in different keyframes are collinear. Figure 4 As shown, a three-dimensional straight line L is explicitly introduced for the three-dimensional points P1 and P2. x Applying collinear constraints to P1 and P2 does not guarantee that P1 and P2 are collinear. x and P2 in the I j The projections of the images are collinear. Furthermore, increasing the number of sampling points for the two-dimensional line segment l also increases the number of variables.

[0102] Furthermore, this embodiment of the invention employs a sliding window strategy to balance the cost of accuracy and efficiency. In the VO algorithm of related technologies, an edge-mapping algorithm is typically used. A straight line can exist for a long time within the camera's field of view, and the solution of this embodiment allows for continuous updating of the relevant parameters of the straight line as long as it can be observed. If an edge-mapping algorithm is used, updating the relevant parameters of the straight line may result in a dense Hessian matrix. Although the sparsity of the Hessian matrix can be maintained by removing some constraints, this approach is clearly suboptimal. Additionally, for identified associated two-dimensional straight line segments, spurious associations may exist. When the back-projection point of a two-dimensional straight line segment is too far from the target straight line, a spurious association can be considered to exist between the two-dimensional straight line segments corresponding to the target straight line. During optimization, it is desirable to remove line associations with spurious associations. However, once a spurious association is edged out, it is difficult to re-establish the association.

[0103] Given the problems with edge detection algorithms, this invention proposes a local optimization adjustment strategy. Specifically, this invention employs a sliding window strategy. For multiple frames of images within the sliding window, the camera pose parameters, visible points, and parameters of two-dimensional line segments are adjusted to ensure that related two-dimensional line segments satisfy collinearity constraints. Specifically, this can be achieved by: determining the distance from the back-projected 3D points of each related two-dimensional line segment in each frame to a selectable 3D line, based on the known camera pose parameters and the determined endpoint depths of the two-dimensional line segments; and fitting the 3D line corresponding to each related two-dimensional line segment by minimizing the distance. When minimizing the distance, if the calculated minimum distance is greater than a set value, the association relationship of the corresponding two-dimensional line segment is released. If the calculated minimum distance is less than or equal to the set value, the corresponding two-dimensional line segment is back-projected to the same 3D line, and the corresponding two-dimensional line segments are connected and merged. Through this invention, erroneous associations and large photometric errors can be easily eliminated.

[0104] See Figure 5 This is a schematic diagram of a three-dimensional line L, a sliding window, and Hessian matrices for different models. (Example) Figure 5 As shown, embodiments of the present invention can employ a sliding window of size 3. For example... Figure 5 As shown in (a), invisible points on the 3D line L are marked as fixed points, such as gray points. For 2D line segments, regardless of the number of sampled points, only the back depths of the two endpoints are optimized. The 3D points projected from the interior points of each 2D line segment are determined based on the endpoint depths of the 2D line segment. Figure 5 (b) and (c) in the figure demonstrate the Hessian matrix using the two-step minimum polarization method and the direct minimization method, respectively. Figure 5Images (d) and (e) illustrate the Hessian matrices for Direct Sparse Odometry (DSO) and Direct Point-Line Visual Odometry (DPLVO), respectively. DSO considers only the photometric error at each point, while DPLVO imposes collinearity constraints on collinear points. Both methods introduce inverse depth for each pixel. Clearly, the two-step minimization algorithm of this embodiment yields the minimum Hessian matrix.

[0105] Using points and camera poses to describe lines increases the number of parameters related to the lines. Correspondingly, optimizing lines based on points and poses also increases the computational load. Therefore, this embodiment of the invention employs a two-step minimization method to optimize the cost objective function in (12). First, for the last term in formula (12), points, lines, and camera poses are related. Given the camera pose parameters and the endpoint depths of the two-dimensional line segment, this problem is transformed into fitting a three-dimensional line based on known visible points. Among them, E L The parameters are determined according to formula (11). On the other hand, if the parameters of the three-dimensional line are fixed, the photometric errors of points and lines can be effectively minimized by adjusting the camera pose parameters and the endpoint depths of the two-dimensional line segments. By repeating the above two steps, the objective cost function can be minimized. And the two-step minimization algorithm always converges.

[0106] The front-end of this invention is used to manage points, lines, and frames. The main difference between this invention and related technologies lies in the initialization of three-dimensional lines and the merging of two-dimensional line segments.

[0107] Initialization of 3D Lines: In related technologies, some points are sampled from newly detected 2D line segments. Without considering collinearity constraints, the 3D points projected from each sampled point are estimated. The 3D line is initialized by fitting these 3D points. If the accuracy of the estimated 3D points is low, the initialization accuracy of the 3D line may decrease or even fail, thus affecting the stability of the algorithm. In this embodiment of the invention, collinearity constraints are applied during 3D line initialization. Specifically, for a newly detected 2D line segment l, 2D points collinear with the 2D line segment l in subsequent images can be tracked by minimizing photometric errors. Due to the ambiguity of lines, it is difficult to generate accurate point-to-point correspondences using this tracking method. However, accurate line segment correspondences can be generated through tracking. For example, by tracking points on the 2D line segment l, a 2D line segment l′ is fitted in subsequent frame images. Assume that π and π′ are the back-projection surfaces of l and l′, respectively. If the angle between π and π′ is greater than 3°, a 3D line can be fitted based on l and l′. Based on the transformation relationship between π and π′, a three-dimensional straight line can be calculated using linear triangulation, and the inverse depth of the endpoints of the two-dimensional straight line segment l can be obtained. Based on the fitted three-dimensional straight line, endpoint tracing of the two-dimensional straight line segment can be further performed. During endpoint tracing, if a newly updated three-dimensional straight line is available, the three-dimensional points on the three-dimensional straight line can be updated using the average value.

[0108] Merging of two-dimensional line segments: The LSD detector can return multiple two-dimensional line segments from an image frame. For example... Figure 6 As shown in the left figure, some detected two-dimensional line segments belong to the same line. Therefore, this embodiment of the method can merge two-dimensional line segments with similar parameters. The merging effect is as follows. Figure 6 As shown in the right figure. It should be noted that the embodiment of the present invention does not merge adjacent and parallel two-dimensional line segments. In this embodiment, by merging two-dimensional line segments, drift can be reduced and the introduction of new parameters can be avoided.

[0109] In this embodiment of the invention, the three-dimensional point P is the back projection of the interior point x on the two-dimensional line segment. x The depth is determined based on the endpoint depth of the corresponding two-dimensional straight line segment. The proof of this conclusion includes: [The following is a separate, unrelated sentence: Three-dimensional point P...] x Located on line L l The back projection line L of point x x At the intersection. For example... Figure 3 As shown, L x Passing through the camera's origin. x The Plück coordinates are in the form of: L x =[0;d x ]. Wherein, d x It can be determined according to formula (7). According to formula (4), P xThe following system of equations (13) should be satisfied:

[0110] m l -p x ×d l =0 3×1

[0111] p x ×d x =0 3×1 .

[0112] When P x ×d l =-[d l ] × P x , and P x ×d x =-[d x ] × P x Then, the system of equations (13) can be written as the system of equations (14):

[0113] [d l ] × p x =-m l

[0114] [d x ] × p x =0 3×1 .

[0115] According to formula (8), A l and b l By definition, the system of equations (14) can be written as A l P x =b l Therefore, P x There is a closed solution Because A l and b l It depends only on d1 and d2, therefore P x Determined by d1 and d2.

[0116] The two-step minimization algorithm always converges. The proof of this conclusion includes: expressing the cost value calculated according to formula (12) in the k-th iteration as E. k In the (k+1)th iteration, based on the known camera pose and reverse depth obtained in the kth iteration, the last term E in formula (12) can be minimized. L Assume that the cost of formula (12) after this step is... Since the first two terms of formula (12) do not involve three-dimensional lines, their values ​​remain unchanged. Therefore, In the second step, the parameters of the 3D line are fixed, and the depth of the endpoints of the 2D line segment is reduced by adjusting the camera pose parameters and the depth of the endpoints. Of course, the first two terms of formula (12) can be reduced simultaneously. After this step, the cost value of formula (12) is E. k+1 It is obvious that... According to the monotone convergence theorem (if a sequence is decreasing and bounded, then the sequence has a limit), the two-step minimization algorithm always converges.

[0117] Result verification:

[0118] The performance of the algorithm in this embodiment of the invention is compared using the ICL-NUIM and TUM monoVO datasets. The ICL-NUIM dataset contains 8 indoor sequences, while the TUM monoVO dataset contains 50 closed-loop sequences from indoor and outdoor environments. To handle nondeterministic behavior, each sequence is run 10 times, and the results are summarized using a cumulative error curve. This curve shows the number of tracking sequences with errors below a certain threshold, indicating the accuracy and robustness of the algorithm. The ICL-NUIM dataset provides the ground truth trajectory for each sequence. Therefore, the absolute trajectory error of the algorithm can be evaluated. On the other hand, the TUM monoVO dataset does not provide the ground truth trajectory. Instead, it provides the basic facts of the closed loops. Therefore, the performance of the algorithm can be evaluated using alignment error (ealign), rotation drift (er), and scale drift (es).

[0119] The method of this invention can be abbreviated as EDPLVO. EDPLVO is compared with related technologies such as DSO, DPLVO, and ORB-SLAM. The results for the ICL-NUIM dataset and the TUM monoVO dataset are as follows: Figure 7 and Figure 8 As shown. Clearly, the EDPLVO algorithm in this embodiment of the invention is superior to other algorithms. For a fair comparison, a line management method can be used to replace the corresponding method in EDPLVO, giving both algorithms the same front end, and this method can be named EDPLVO++. For example... Figure 7 As shown, DPL VO++ provides better results than DPL VO. ORB-SLAM performs poorly due to the low-texture environments in both datasets. Our algorithm achieves better accuracy than DPLVO because the proposed linear photometric error in this embodiment introduces precise collinearity to minimize photometric accuracy and fully optimizes the four degrees of freedom of the 3D line.

[0120] Ablation Study: This embodiment of the invention further investigates the impact of different components in the algorithm, considering two variants of the algorithm:

[0121] EDPLVO-LA: Line association not established. That is, the collinearity constraint in formula (12) is removed.

[0122] EDPLVO Joint: Variables in the joint adjustment cost function (12). Embodiments of this invention can use a four-degree-of-freedom representation to parameterize a three-dimensional straight line.

[0123] See Figure 8 Due to the adjustment of collinearity points, EDPLVO-LA outperforms DSO and is close to DPLVO++. The difference between the results of EDPLVO and EDPLVOJoint is small, but EDPLVO is significantly more efficient.

[0124] In summary, this invention proposes a novel direct visual odometry scheme based on points and lines. This scheme demonstrates that the depth of a point within a two-dimensional line segment is determined by the depth of the endpoints of the two-dimensional line segment. Compared to related techniques, this strategy significantly reduces the number of variables in the optimization process and satisfies collinearity. Furthermore, since this strategy accurately achieves collinearity in the optimization process, it facilitates the initialization of three-dimensional lines and improves the accuracy of the direct algorithm. Further, this invention introduces a two-step activation method to accelerate the optimization speed. Experimental results show that the algorithm in this embodiment significantly reduces the computation time of the optimization and achieves more accurate results compared to existing techniques.

[0125] See Figure 9 This is a schematic diagram of a positioning device provided in an embodiment of the present invention. Figure 9 As shown, the above-mentioned device may include:

[0126] The first determining module 201 is used to determine the two-dimensional line segments and visible points of each frame image captured by the camera, wherein the visible points include the endpoints of the two-dimensional line segments and the interior points between the endpoints;

[0127] Collinearity constraint module 202 is used to fit a three-dimensional straight line based on the two-dimensional straight line segment and determine the collinearity constraint of the two-dimensional straight line segment;

[0128] Cost module 203 is used to construct a cost objective function based on the photometric error of the visible points of each frame image, the photometric error of the two-dimensional straight line segments of each frame image, and the collinearity constraint;

[0129] The second determining module 204 is used to minimize the cost objective function to determine the pose parameters of the camera.

[0130] The positioning device of this invention can perform the functions shown in Figures 1 to 12. Figure 8 The illustrated embodiment involves a positioning method. For parts not described in detail in this embodiment, please refer to Figures 1 to 12. Figure 8The relevant descriptions of the embodiments shown are provided below. The execution process and technical effects of this technical solution are illustrated in Figures 1 to 12. Figure 8 The descriptions in the illustrated embodiments will not be repeated here.

[0131] It should be understood that Figure 9 The division of the various modules in the positioning device shown is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing elements; they can be fully implemented in hardware; or some modules can be implemented in software via processing elements, while others are implemented in hardware. For example, the first determining module 201 and the second determining module 204 can be separate processing elements, or they can be integrated into a chip in the electronic device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0132] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, these modules can be integrated together as a System-On-a-Chip (SOC).

[0133] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. This electronic device can be used to perform the positioning method described above. Figure 10 As shown, the electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, a communication interface 420, a memory 430, and a communication bus 440 connecting different system components (including processors 410, communication interfaces 420, and memory 430).

[0134] Communication bus 440 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0135] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0136] Memory 430 may include a computer system readable medium in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to execute embodiments of the present invention as shown in Figures 1 to 12. Figure 8 The positioning method involved in the illustrated embodiment.

[0137] A program / utility having a set (at least one) of program modules can be stored in memory 430. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically execute embodiments of the present invention as shown in Figures 1 to 12. Figure 8 The positioning method involved in the illustrated embodiment.

[0138] Processor 410 executes various functional applications and data processing by running programs stored in memory 430, such as implementing the functions shown in Figures 1 to 12 of this specification. Figure 8 The positioning method involved in the illustrated embodiment.

[0139] In a specific implementation, this application also provides a computer storage medium, wherein the computer storage medium may store a program, and the program, when executed, may include some or all of the steps provided in the various embodiments of this application. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0140] In a specific implementation, the present invention also provides a computer program product, which includes executable instructions that, when executed on a computer, cause the computer to perform some or all of the steps in the above method embodiments.

[0141] In this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0142] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In several embodiments provided by this invention, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The above description is merely a specific embodiment of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the protection scope of this invention. The protection scope of this invention should be determined by the scope of the claims.

Claims

1. A positioning method, characterized in that, include: Determine the two-dimensional line segments and visible points of each frame image captured by the camera, wherein the visible points include the two endpoints of the two-dimensional line segments and the interior points between the two endpoints; Fit a three-dimensional straight line based on the two-dimensional straight line segment and determine the collinearity constraint of the two-dimensional straight line segment; Based on the photometric error of the visible points of each frame image, the photometric error of the two-dimensional straight line segments of each frame image, and the collinearity constraint, a cost objective function is constructed. Minimize the cost objective function to determine the camera pose parameters; The photometric error of the two-dimensional straight line segment is determined based on the photometric errors of the endpoints and interior points of the two-dimensional straight line segment. The determination of the photometric error at any interior point x of the two-dimensional straight line segment includes: Based on the endpoint depth of the corresponding two-dimensional line segment, determine the three-dimensional point P by the back projection of the interior point x of the two-dimensional line segment. x ; Determine the three-dimensional point P x At the projection point x′ of the target frame image; Based on the projection point x′, determine the photometric error of the interior point x of the two-dimensional straight line segment; Wherein, the step of fitting a three-dimensional straight line based on the two-dimensional straight line segment and determining the collinearity constraint of the two-dimensional straight line segment includes: Based on the support region width of the two-dimensional line segment under the current camera posture and the fitable three-dimensional line group, establish the distance function from the three-dimensional endpoint of the back projection of the two-dimensional line segment to each three-dimensional line in the fitable three-dimensional line group. Based on the distance function, the collinearity constraint of the two-dimensional line segment is determined.

2. The method according to claim 1, characterized in that, The three-dimensional line fitted from the two-dimensional line is represented by Plück coordinates, and the first and second parameters of the Plück coordinates are determined based on the endpoint depths of the corresponding two-dimensional line segments.

3. The method according to claim 1, characterized in that, The determination of the two-dimensional straight line segments of each frame image captured by the camera includes: A sliding window strategy is used to traverse the image data stream acquired by the camera and identify two-dimensional line segments from each frame image contained in the current sliding window.

4. The method according to claim 1, characterized in that, Minimizing the cost objective function to determine the camera pose parameters includes: Based on the known camera pose parameters and the endpoint depths of the corresponding two-dimensional line segments, the three-dimensional line is fitted by minimizing the collinearity constraint in the cost objective function; Under the collinearity constraint, the camera pose parameters when the camera captures each frame of images are determined by minimizing the photometric error of the visible point, the photometric error of the two-dimensional line segment, and the collinearity constraint by adjusting the camera pose parameters and the endpoint depth of the corresponding two-dimensional line segment.

5. The method according to claim 4, characterized in that, Based on the known camera pose parameters and the endpoint depths of the corresponding two-dimensional line segments, the three-dimensional line is fitted by minimizing the collinearity constraint in the cost objective function, including: Based on the known camera pose parameters and the endpoint depths of the corresponding two-dimensional line segments, determine the distances from the three-dimensional points of the back projection of each related two-dimensional line segment in each frame image to the selectable three-dimensional line. By minimizing the distance, a three-dimensional line corresponding to each two-dimensional line segment with a correlation is fitted.

6. The method according to claim 5, characterized in that, The method further includes: When minimizing the distance, if the calculated minimum distance is greater than a set value, the association relationship of the corresponding two-dimensional line segment is terminated; If the calculated minimum distance is less than or equal to the set value, then the corresponding two-dimensional line segment is back-projected onto the same three-dimensional line, and the corresponding two-dimensional line segments are connected and merged.

7. An electronic device, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 6 by calling the program instructions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method of any one of claims 1 to 6.

Images