A method for joint camera pose estimation of underground tunnel image points and lines based on vanishing point prior

Abstract

The present application proposes a method for estimating the camera pose of underground tunnel images based on vanishing point priors, wherein the method includes: determining the point feature matching results and line feature matching results between the current frame image and the previous frame image captured by the camera, and determining the vanishing point corresponding to the current frame image based on the first line feature corresponding to the current frame image, and determining the first pose information corresponding to the camera based on the vanishing point and the camera's internal parameters, determining the second pose information corresponding to the camera based on the first pose information and the point feature matching results, and optimizing the second pose information based on the constructed cost function, the point feature matching results and the line feature matching results to obtain the target pose information of the camera. As a result, the accuracy of camera pose calculation is improved in scenes with weak textures and high robustness requirements such as underground tunnels.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for estimating underground tunnel image point and line combined with camera pose based on vanishing point prior. Background Art

[0002] In computer vision, Simultaneous Localization And Mapping (SLAM), and 3D reconstruction, it is necessary to determine the position and orientation of a camera in 3D space. The internal and external parameters of a camera determine its own characteristics. The internal parameters are usually determined when the camera leaves the factory or can be obtained through calibration. In order to obtain the complete position and orientation, the external parameters must be calculated.

[0003] The lighting in the coal mine tunnel scene is dim, and there are a large number of low-texture areas such as the tunnel wall. These areas have the characteristics of slow gradient changes, low contrast, and unclear texture information. Therefore, the images collected in low-texture scenes such as coal mine tunnels have the problem of unclear features, and it is difficult to provide accurate and effective pose information. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of this application is to propose a method for joint camera pose estimation of underground tunnel images based on vanishing point priors, in which some pose parameters are calculated through the reliability of a single vanishing point as a priori, to guide the pose optimization screening process based on feature calculation, and to organically integrate point features and line features to increase the feature richness and pose solution reliability in tunnel weak texture scenes, thereby improving the accuracy of camera pose calculation in scenes with weak texture and high robustness requirements such as underground tunnels.

[0006] The second objective of the present application is to propose a device for estimating underground tunnel image point and line combined with camera pose based on vanishing point prior.

[0007] The third objective of the present application is to provide an electronic device.

[0008] A fourth objective of the present application is to provide a computer-readable storage medium.

[0009] A fifth object of the present application is to provide a computer program product.

[0010] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a method for estimating underground tunnel image point lines and camera pose based on vanishing point prior, including:

[0011] Performing line feature extraction and point feature extraction on the current frame image and the previous frame image captured by the camera, respectively, to obtain a first line feature and a first point feature corresponding to the current frame image, and a second line feature and a second point feature corresponding to the previous frame image;

[0012] Matching the first point feature and the second point feature to obtain a point feature matching result;

[0013] Based on the point feature matching result, matching the first line feature and the second line feature to obtain a line feature matching result;

[0014] Based on the first line feature, determine a vanishing point corresponding to the current frame image, and based on the vanishing point and an intrinsic parameter of the camera, determine first position information corresponding to the camera;

[0015] Determine second pose information corresponding to the camera based on the first pose information and the point feature matching result;

[0016] Based on the constructed cost function, the point feature matching result and the line feature matching result, the second pose information is optimized to obtain the target pose information of the camera.

[0017] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a device for estimating underground tunnel image point lines and camera pose based on vanishing point prior, comprising:

[0018] An extraction module is used to perform line feature extraction and point feature extraction on a current frame image and a previous frame image captured by a camera, respectively, to obtain a first line feature and a first point feature corresponding to the current frame image, and a second line feature and a second point feature corresponding to the previous frame image;

[0019] A first matching module, used for matching the first point feature and the second point feature to obtain a point feature matching result;

[0020] A second matching module matches the first line feature and the second line feature based on the point feature matching result to obtain a line feature matching result;

[0021] A first determination module, configured to determine a vanishing point corresponding to the current frame image based on the first line feature, and to determine first position information corresponding to the camera based on the vanishing point and an intrinsic parameter of the camera;

[0022] A second determination module, configured to determine second posture information corresponding to the camera based on the first posture information and the point feature matching result;

[0023] An acquisition module is used to optimize the second pose information based on the constructed cost function, the point feature matching result and the line feature matching result to obtain the target pose information of the camera.

[0024] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:

[0025] A processor, and a memory communicatively connected to the processor;

[0026] The memory stores computer-executable instructions;

[0027] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the embodiment of the first aspect.

[0028] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment.

[0029] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, a computer program, which implements the method described in the first aspect when executed by a processor.

[0030] The method for joint camera pose estimation of underground tunnel images based on vanishing point prior provided in the present application calculates some pose parameters as prior through the reliability of a single vanishing point, guides the pose optimization screening process based on feature calculation, and organically integrates point features and line features to increase the feature richness and pose solution reliability in tunnel weak texture scenes, thereby improving the accuracy of camera pose calculation in scenes with weak texture and high robustness requirements such as underground tunnels.

[0031] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A schematic flow chart of a method for estimating underground tunnel image point and line combined with camera pose based on vanishing point prior provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of determining the number of matching pairs provided in an embodiment of the present disclosure; and

[0035] Figure 3 A schematic diagram of the structure of a device for estimating underground tunnel image points and lines combined with a camera pose based on vanishing point prior provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the method and device for estimating the point and line of underground tunnel images and the camera pose based on vanishing point prior in an embodiment of the present application with reference to the accompanying drawings.

[0038] Figure 1 A flowchart of a method for estimating underground tunnel image points and lines combined with a camera pose based on vanishing point prior provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the underground tunnel image point-line combined with camera pose estimation method based on vanishing point prior includes the following steps:

[0040] Step 101, performing line feature extraction and point feature extraction on the current frame image and the previous frame image captured by the camera, respectively, to obtain the first line feature and the first point feature corresponding to the current frame image, and the second line feature and the second point feature corresponding to the previous frame image.

[0041] Among them, the current frame image and the previous frame image are images collected by the camera at different times, and most of them are image pairs with certain viewing angle changes and time intervals.

[0042] In some embodiments, line segment detection is performed on an image to obtain a first line segment detection result, two line segments whose angle difference is less than an angle threshold and whose distance between the two closest endpoints is less than a second distance threshold in the first line segment detection result are merged, and two line segments whose angle difference is less than the angle threshold and whose distance between the midpoints is less than a third distance threshold are merged to obtain a second line segment detection result, and line segments whose line segment lengths in the second line segment detection result are less than a length threshold are filtered out to obtain line features. The line features may be straight lines in the image such as lane edges and pipelines.

[0043] In some possible implementations, line segment detection is performed on the current frame image to obtain a first line segment detection result corresponding to the current frame image, and two line segments in the first line segment detection result corresponding to the current frame image whose angle difference is less than an angle threshold and whose distance between two closest endpoints is less than a second distance threshold are merged, and two line segments in which the angle difference is less than the angle threshold and whose distance between midpoints is less than a third distance threshold are merged to obtain a second line segment detection result corresponding to the current frame image, and line segments in the second line segment detection result corresponding to the current frame image whose line segment length is less than a length threshold are filtered out to obtain a first line feature corresponding to the current frame image.

[0044] In some possible implementations, line segment detection is performed on the previous frame image to obtain a first line segment detection result corresponding to the previous frame image, and two line segments in the first line segment detection result corresponding to the previous frame image whose angle difference is less than an angle threshold and whose distance between two closest endpoints is less than a second distance threshold are merged, and two line segments in which the angle difference is less than the angle threshold and whose distance between midpoints is less than a third distance threshold are merged to obtain a second line segment detection result corresponding to the previous frame image, and line segments in the second line segment detection result corresponding to the previous frame image whose line segment length is less than a length threshold are filtered out to obtain the first line feature corresponding to the previous frame image.

[0045] In some embodiments, a multiple comparison method (Least Significant Difference, LSD) may be used to perform line segment detection on the image to obtain a first line segment detection result.

[0046] In some embodiments, the length threshold may be determined based on the width, length, and a preset scale factor of the image.

[0047] The calculation formula of the length threshold can be:

[0048] L min =γ[min(W,H)]

[0049] Among them, L min is the length threshold, W is the image width, H is the image height, and γ is the proportionality coefficient. For example, the value of the proportionality coefficient may be 0.125, 0.2, etc. This disclosure does not limit this.

[0050] It should be noted that in the embodiment of the present disclosure, after merging two line segments into one line segment, they can continue to be merged with other line segments until there are no more line segments that can be merged, thereby retaining long line segments with high repeatability and stability for use in subsequent steps.

[0051] In some embodiments, a Superpoint network or a dual decoupling network (Depth and Descriptor Network, D2-Net) may be used to extract point features from the current frame image and the previous frame image respectively.

[0052] Step 102: Match the first point feature and the second point feature to obtain a point feature matching result.

[0053] In some embodiments, the Superglue algorithm may be used to match the first point feature and the second point feature.

[0054] Step 103: Based on the point feature matching result, the first line feature and the second line feature are matched to obtain a line feature matching result.

[0055] In some embodiments, obtaining the line feature matching result includes the following steps:

[0056] (1) Determine a first target point corresponding to each first line segment according to a first distance between a first feature point corresponding to the current frame image in the point feature matching result and each first line segment in the current frame image.

[0057] Specifically, the distance between each first feature point of the current frame image and each first line segment is calculated, and the first feature point whose first distance to the first line segment is less than the fourth distance threshold is determined as the first target point corresponding to the first line segment.

[0058] (2) Determine the second target point corresponding to each second line segment according to the second distance between the second feature point corresponding to the previous frame image in the point feature matching result and each second line segment in the previous frame image.

[0059] Specifically, the distance between each second feature point of the previous frame image and each second line segment is calculated, and the second feature point whose second distance to the second line segment is less than the fourth distance threshold is determined as the second target point of the second line segment.

[0060] (3) Combine a first line segment and a second line segment into a line segment pair.

[0061] It should be noted that each line segment pair includes a first line segment and a second line segment. Specifically, a first line segment in the current frame image and each second line segment in the previous frame image can form a line segment pair, and a second line segment in the previous frame image and each first line segment in the current frame image can form a line segment pair.

[0062] (4) According to the point feature matching results, determine the number of matching pairs between the first target point corresponding to the first line segment and the second target point corresponding to the second line segment in each line segment pair.

[0063] It can be understood that based on the point feature matching result, the first target point corresponding to the first line segment in the line segment pair is matched with the second target point corresponding to the second line segment to determine the number of successfully matched point pairs.

[0064] Figure 2 A schematic diagram of determining the number of matching pairs provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the first line segment L k+1 The corresponding first target points are E and F, and the second line segment is L k The corresponding second target points are A, B, and C. The point feature matching results indicate that A and D are a pair of matching points, B and E are a pair of matching points, and C and F are a pair of matching points. Therefore, L k+1 and L k The corresponding number of matching pairs is 2.

[0065] (5) Determine the maximum value of a first number of first target points corresponding to the first line segment and a second number of second target points corresponding to the second line segment in each line segment pair.

[0066] (6) The line segment pairs whose ratio between the number of matching pairs and the maximum value is greater than the score threshold are determined as line feature matching results.

[0067] like Figure 2 As shown, the first line segment L k+1 The first number of the corresponding first target points is 2, and the second line segment L k The first number of the corresponding second target points is 3, the maximum value is 3, and L k+1 and L k The corresponding matching number is 2, and the ratio is 0.67. If the score threshold is 0.6, then L k+1 and L k It can be used as a set of paired line segments in the line feature matching results.

[0068] Therefore, the strategy of using point-line association to guide line segment matching saves computing resources. Since the results of a large number of point feature matching are robust to lighting changes, and the association between points and lines is not affected by environmental conditions, the obtained line segment matching results have good robustness and adaptability to the lighting change environment in underground tunnels.

[0069] Step 104: determine the vanishing point corresponding to the current frame image based on the first line feature, and determine the first position information corresponding to the camera based on the vanishing point and the camera's internal parameters.

[0070] In some embodiments, determining the vanishing point corresponding to the current frame image includes the following steps:

[0071] (1) Based on the slope of each first line segment in the first line feature, cluster the first line segments in the first line feature to obtain a cluster set, wherein the slope difference between every two line segments in each cluster set is less than a slope threshold.

[0072] In the embodiment of the present disclosure, the straight lines can be clustered and divided according to the lane design standard with the lane edge as the main direction. The first detected line segments are clustered according to the slope similarity using the K-means clustering algorithm.

[0073] (2) Each time, two first line segments are sampled from different cluster sets, and the distance between the intersection point of the two first line segments and the straight lines where the remaining first line segments except the two first line segments are located is determined, and the total number of first line segments whose distance is less than the first distance threshold is determined as the number of inner points corresponding to the intersection point.

[0074] (3) The intersection point with the largest number of inliers is determined as the vanishing point corresponding to the current frame image.

[0075] In some embodiments, based on the vanishing point and the intrinsic parameters of the camera, the formula for determining the first position information corresponding to the camera is:

[0076]

[0077] Among them, α is the first pitch angle, β is the first yaw angle, k is the intrinsic parameter of the camera, and v P is the coordinate of the vanishing point, and the first pitch angle and the first yaw angle are the first position information.

[0078] It should be noted that, assuming that the straight line passes through point X and the direction vector is V, any point P on the straight line can be expressed as:

[0079] P=X+S*V

[0080] Where S is the first scale factor. The camera projection equation can be used to further obtain the pixel coordinates of the points on the straight line in the corresponding image space. The camera projection equation is as follows:

[0081]

[0082] Among them, λ is the second scale factor, (u,v) is the coordinate of the corresponding point in the pixel coordinate system, (X,Y,Z) is the coordinate of the corresponding point in the world coordinate system, k is the camera intrinsic parameter matrix, (R,T) is the rotation and translation matrix, and the imaging coordinates of any point P on the straight line in the image are:

[0083]

[0084] Since all parallel lines in space that are not parallel to the imaging plane intersect at infinity and are reflected in the vanishing point of the image, when S is infinite, the coordinate expression of the vanishing point on the image is as follows:

[0085]

[0086] The camera intrinsic parameter matrix k can be obtained by camera calibration and generally does not change. Considering that the present disclosure only requires a stable vanishing point in a single direction, the lane edge is used as the Z-axis direction, and according to the vanishing point coordinate V in step S2, p (u vp ,v vp ) can be calculated to obtain the corresponding direction vector V as (0,0,1), and the rotation matrix R is expressed by the yaw matrix, pitch matrix and roll matrix.

[0087] R=R yaw R pitc□ R roll =(a,b,r)

[0088] Therefore, we can get λV p = kr, since the columns of the rotation matrix are unit vectors,

[0089] In addition, the pitch angle, roll angle, and yaw angle are expressed as follows:

[0090]

[0091] Therefore, the rotation matrix can be expressed as:

[0092]

[0093] Among them, C γ is cosine cos(γ), C β is cosine cos(β), C α is cosine cos(α), S α is the sine sin(α), S β is sine sin(β), S γ is sine sin(γ).

[0094]

[0095] Therefore, we can get:

[0096]

[0097] Then combined with the vanishing point V p (u vp ,vvp ), we can get the first pitch angle α and the first yaw angle β.

[0098] Therefore, the stability of vanishing point detection is improved by screening and retaining long line segment features through the improved LSD algorithm.

[0099] Step 105: Determine the second pose information corresponding to the camera based on the first pose information and the point feature matching result.

[0100] In some embodiments, determining the second pose information may include the following steps:

[0101] (1) Randomly extract four sets of matching point pairs from the point feature matching results, and determine the transformation matrix between the previous frame image and the current frame image based on the four sets of matching point pairs.

[0102] In some embodiments, the solvePnPRansac algorithm in Opencv can be used to solve the transformation matrix T between two frames. 12 .

[0103] (2) Based on the transformation matrix, determine an initial candidate pose information, wherein the initial candidate pose information includes a second pitch angle and a second yaw angle.

[0104] In some embodiments, the pose of the current frame can be determined according to the transformation matrix, and the rotation matrix of the camera can be further decomposed to recover the second pitch angle θ from the rotation matrix. p With the second yaw angle θ y .

[0105] (3) The operation of randomly extracting four sets of matching point pairs is performed cyclically to obtain multiple initial candidate pose information.

[0106] (4) Based on the similarity between the first pose information and each initial candidate pose information, the target candidate pose information is screened out from multiple initial candidate pose information.

[0107] In some embodiments, the initial candidate pose information corresponding to a similarity greater than a similarity threshold may be determined as the target candidate pose information, wherein the similarity threshold may be 0.95, 0.9, etc. This disclosure does not limit this.

[0108] In some embodiments, direction vectors corresponding to the first pitch angle, the first yaw angle, the second pitch angle, and the second yaw angle may be constructed first.

[0109] Among them, the direction vector corresponding to the first pitch angle is:

[0110] The direction vector corresponding to the first yaw angle is:

[0111] The direction vector corresponding to the second pitch angle is:

[0112] The direction vector corresponding to the second yaw angle is:

[0113] Afterwards, a first similarity between the first pitch angle and the second pitch angle, and a second similarity between the first yaw angle and the second yaw angle are calculated, and the smallest similarity between the first similarity and the second similarity is determined as the similarity between the first pose information and the initial candidate pose information.

[0114] Among them, the first similarity is: The second similarity is:

[0115] (5) Fit the transformation matrix corresponding to the target candidate pose to obtain the target transformation matrix.

[0116] In some embodiments, the Ransac algorithm may be used for fitting, which is not limited in the present disclosure.

[0117] (6) Based on the target transformation matrix, determine the second pose information corresponding to the camera.

[0118] In some embodiments, the line feature matching results are filtered based on the target transformation matrix.

[0119] Specifically, the line segments of the previous frame image in the matching line segment pair can be converted to the next frame image through the target transformation matrix, and the matching degree between the converted line segments and the line segments of the current frame image in the matching line segment pair is calculated, and the matching line segment pairs with a matching degree lower than the matching degree threshold are deleted, thereby further improving the accuracy of the line feature matching results.

[0120] Step 106 , based on the constructed cost function, the point feature matching result and the line feature matching result, the second pose information is optimized to obtain the target pose information of the camera.

[0121] In some embodiments, determining the target posture information may specifically include the following steps:

[0122] (1) The first coordinate corresponding to the first feature point in each matching point pair in the point feature matching result in the image coordinate system and the second coordinate corresponding to the second feature point in the world coordinate system are determined as a set of point projection pairs, wherein the first feature point is a feature point in the current frame image, and the second feature point is a feature point in the previous frame image.

[0123] In some embodiments, the three-dimensional point coordinates corresponding to the second feature point in the world coordinate system may be calculated by triangulation measurement.

[0124] (2) The position of the first line segment in each matching line segment pair in the line feature matching result in the image coordinate system and the third coordinates corresponding to the two endpoints of the second line segment in the world coordinate system are determined as a set of line projection pairs, wherein the first line segment is the line segment in the current frame image, and the second line segment is the line segment in the previous frame image.

[0125] (3) Based on the cost function, the point projection pair and the line projection pair, determine the projection error corresponding to the second pose information.

[0126] In some embodiments, the cost function may be:

[0127]

[0128] Where C(T) is the projection error, T 1 is the second pose information, π is the projection function, N is the total number of point projection pairs, p i is the second coordinate of the projection pair of the i-th point, q i is the first coordinate of the projection pair of the i-th point, ‖π(T 1 ·p i )-q i ‖ indicates that through T 1 and π will p i The distance between the coordinate projected to the image coordinate system and the first coordinate, represents the point feature projection error, σ is the weight corresponding to the point feature projection error, M is the total number of line projection pairs, l j1 is the third coordinate corresponding to one endpoint of the second line segment in the j-th line projection pair, l j2 is the third coordinate corresponding to the other endpoint of the second line segment in the j-th line projection pair, m j is the position of the first line segment in the j-th line projection pair, d(π(T 1 ·l j1 ),m j ) indicates that the 1 and π will l j1 The distance between the coordinate projected to the image coordinate system and the first line segment, d(π(T 1 lj2,mj represents the distance between the coordinates of lj2 projected to the image coordinate system by T1 and π and the first line segment, represents the line feature projection error, and ω is the weight corresponding to the line feature projection error.

[0129] (4) The second pose information is updated, and the projection error corresponding to the updated second pose information is calculated.

[0130] In some embodiments, the update function of the second pose information may be:

[0131] ΔT=-(J T J+μ·diag(J T J)) -1 J T μ)

[0132] Among them, ΔT can be the adjustment value of the second pose information, J is the Jacobian matrix, r is the residual, which can be C(T), and the iteration step size can be controlled by adjusting the value of μ.

[0133] In the disclosed embodiment, ΔT+T is the updated second posture information.

[0134] In some embodiments, the number of times the second pose information is updated may be set.

[0135] (5) The pose information with the smallest projection error is determined as the target pose information corresponding to the camera.

[0136] It should be noted that the use of a single vanishing point to calculate the pose prior information can eliminate obviously erroneous matching results as early as possible, improve the accuracy of pose estimation, and accelerate the iterative process of pose estimation.

[0137] In this embodiment, some pose parameters are calculated through the reliability of a single vanishing point as a priori to guide the pose optimization screening process based on feature calculation, and point features and line features are organically integrated to increase the feature richness and pose solution reliability and efficiency in weak texture scenes in tunnels, thereby improving the accuracy of camera pose calculation in scenes with weak texture and high robustness requirements such as underground tunnels.

[0138] In order to implement the above-mentioned embodiments, the present application also proposes a device for estimating underground tunnel image points and lines in conjunction with a camera pose based on vanishing point priors.

[0139] Figure 3 The present invention provides a schematic diagram of a device for estimating the position and posture of underground tunnel image points and lines combined with a camera based on a vanishing point prior. Figure 3 As shown, the underground tunnel image point line combined with camera pose estimation device 300 based on vanishing point prior includes:

[0140] An extraction module 301 is used to perform line feature extraction and point feature extraction on a current frame image and a previous frame image captured by a camera, respectively, to obtain a first line feature and a first point feature corresponding to the current frame image, and a second line feature and a second point feature corresponding to the previous frame image;

[0141] A first matching module 302 is used to match the first point feature and the second point feature to obtain a point feature matching result;

[0142] The second matching module 303 is used to match the first line feature and the second line feature based on the point feature matching result to obtain a line feature matching result;

[0143] A first determination module 304 is used to determine a vanishing point corresponding to the current frame image based on the first line feature, and determine first position information corresponding to the camera based on the vanishing point and the camera's internal parameters;

[0144] A second determination module 305 is used to determine second posture information corresponding to the camera based on the first posture information and the point feature matching result;

[0145] The acquisition module 306 is used to optimize the second pose information based on the constructed cost function, the point feature matching result and the line feature matching result to obtain the target pose information of the camera.

[0146] Furthermore, in a possible implementation of the embodiment of the present application, the second matching module 303 is configured to:

[0147] Determine a first target point corresponding to each first line segment according to a first distance between a first feature point corresponding to the current frame image in the point feature matching result and each first line segment in the current frame image;

[0148] Determine a second target point corresponding to each second line segment according to a second distance between a second feature point corresponding to the previous frame image in the point feature matching result and each second line segment in the previous frame image;

[0149] Combining a first line segment and a second line segment into a line segment pair;

[0150] According to the point feature matching result, determining the number of matching pairs between the first target point corresponding to the first line segment and the second target point corresponding to the second line segment in each line segment pair;

[0151] Determine the maximum value of a first number of first target points corresponding to the first line segment and a second number of second target points corresponding to the second line segment in each line segment pair;

[0152] The line segment pairs whose ratio between the number of matching pairs and the maximum value is greater than the score threshold are determined as line feature matching results.

[0153] Further, in a possible implementation of the embodiment of the present application, the first determining module 304 is configured to:

[0154] Based on the slope of each first line segment in the first line feature, clustering the first line segments in the first line feature to obtain a cluster set, wherein the slope difference between every two line segments in each cluster set is less than a slope threshold;

[0155] Each time, two first line segments are sampled from different cluster sets, and the distance between the intersection point of the two first line segments and the lines where the remaining first line segments except the two first line segments are located is determined, and the total number of first line segments whose distances are less than the first distance threshold is determined as the number of inner points corresponding to the intersection point;

[0156] The intersection point with the largest number of inliers is determined as the vanishing point corresponding to the current frame image.

[0157] Furthermore, in a possible implementation of the embodiment of the present application, based on the vanishing point and the intrinsic parameters of the camera, the formula for determining the first position information corresponding to the camera is:

[0158]

[0159] Among them, α is the first pitch angle, β is the first yaw angle, k is the intrinsic parameter of the camera, and v P is the coordinate of the vanishing point, and the first pitch angle and the first yaw angle are the first position information.

[0160] Further, in a possible implementation of the embodiment of the present application, the second determining module 305 is configured to:

[0161] Randomly extract four sets of matching point pairs from the point feature matching results, and determine the transformation matrix between the previous frame image and the current frame image based on the four sets of matching point pairs;

[0162] Based on the transformation matrix, determining an initial candidate pose information, wherein the initial candidate pose information includes a second pitch angle and a second yaw angle;

[0163] The operation of randomly extracting four sets of matching point pairs is executed cyclically to obtain multiple initial candidate pose information;

[0164] Based on the similarity between the first pose information and each initial candidate pose information, the target candidate pose information is screened out from the multiple initial candidate pose information;

[0165] Fit the transformation matrix corresponding to the target candidate pose to obtain the target transformation matrix;

[0166] Based on the target transformation matrix, the second pose information corresponding to the camera is determined.

[0167] Furthermore, in a possible implementation of the embodiment of the present application, a screening module is also included, which is used to:

[0168] Based on the target transformation matrix, the line feature matching results are filtered.

[0169] Furthermore, in a possible implementation of the embodiment of the present application, the acquisition module 306 is used to:

[0170] Determine a set of point projection pairs by the first coordinate corresponding to the first feature point in the image coordinate system and the second coordinate corresponding to the second feature point in the world coordinate system in each matching point pair in the point feature matching result, wherein the first feature point is a feature point in the current frame image, and the second feature point is a feature point in the previous frame image;

[0171] Determine the position of the first line segment in each matching line segment pair in the line feature matching result in the image coordinate system and the third coordinates corresponding to the two endpoints of the second line segment in the world coordinate system as a set of line projection pairs, wherein the first line segment is a line segment in the current frame image, and the second line segment is a line segment in the previous frame image;

[0172] Determine a projection error corresponding to the second pose information based on the cost function, the point projection pair, and the line projection pair;

[0173] Updating the second pose information and calculating the projection error corresponding to the updated second pose information;

[0174] The pose information with the smallest projection error is determined as the target pose information corresponding to the camera.

[0175] Furthermore, in a possible implementation of the embodiment of the present application, the cost function is:

[0176]

[0177] Where C(T) is the projection error, T 1 is the second pose information, π is the projection function, N is the total number of point projection pairs, p i is the second coordinate of the projection pair of the i-th point, q i is the first coordinate of the projection pair of the i-th point, ‖π(T 1 ·p i )-q i ‖ indicates that through T 1 and π will p i The distance between the coordinate projected to the image coordinate system and the first coordinate, represents the point feature projection error, σ is the weight corresponding to the point feature projection error, M is the total number of line projection pairs, l j1 is the third coordinate corresponding to one endpoint of the second line segment in the j-th line projection pair, l j2 is the third coordinate corresponding to the other endpoint of the second line segment in the j-th line projection pair, m j is the position of the first line segment in the j-th line projection pair, d(π(T 1 ·l j1 ),m j ) indicates that the 1 and π will l j1The distance between the coordinate projected to the image coordinate system and the first line segment, d(π(T 1 lj2,mj represents the distance between the coordinates of lj2 projected to the image coordinate system by T1 and π and the first line segment, represents the line feature projection error, and ω is the weight corresponding to the line feature projection error.

[0178] Furthermore, in a possible implementation of the embodiment of the present application, line feature extraction includes the following steps:

[0179] Performing line segment detection on the image to obtain a first line segment detection result;

[0180] In the first line segment detection result, two line segments whose angle difference is less than the angle threshold and whose distance between the two closest endpoints is less than the second distance threshold are merged, and two line segments whose angle difference is less than the angle threshold and whose distance between the midpoints is less than the third distance threshold are merged to obtain a second line segment detection result;

[0181] The line segments whose lengths in the second line segment detection result are less than the length threshold are filtered out to obtain line features.

[0182] It should be noted that the aforementioned explanation of the embodiment of the method for estimating the point and line of underground tunnel images with a camera based on vanishing point prior is also applicable to the device for estimating the point and line of underground tunnel images with a camera based on vanishing point prior of this embodiment, and will not be repeated here.

[0183] In this embodiment, some pose parameters are calculated through the reliability of a single vanishing point as a priori to guide the pose optimization screening process based on feature calculation, and point features and line features are organically integrated to increase the feature richness and pose solution reliability in weak texture scenes in tunnels, thereby improving the accuracy of camera pose calculation in scenes with weak texture and high robustness requirements such as underground tunnels.

[0184] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0185] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0186] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0187] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0188] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0189] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0190] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0191] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0192] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0193] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0194] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0195] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0197] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for estimating the point-line combined with camera pose of underground tunnel images based on vanishing point prior, characterized in that: The following steps are involved: Performing line feature extraction and point feature extraction on the current frame image and the previous frame image captured by the camera, respectively, to obtain a first line feature and a first point feature corresponding to the current frame image, and a second line feature and a second point feature corresponding to the previous frame image; Matching the first point feature and the second point feature to obtain a point feature matching result; Based on the point feature matching result, matching the first line feature and the second line feature to obtain a line feature matching result; Based on the first line feature, determine a vanishing point corresponding to the current frame image, and based on the vanishing point and an intrinsic parameter of the camera, determine first position information corresponding to the camera; Determine second pose information corresponding to the camera based on the first pose information and the point feature matching result; Based on the constructed cost function, the point feature matching result and the line feature matching result, optimizing the second pose information to obtain the target pose information of the camera; The determining, based on the first line feature, a vanishing point corresponding to the current frame image includes: Based on the slope of each first line segment in the first line feature, clustering the first line segments in the first line feature to obtain a cluster set, wherein a slope difference between every two line segments in each cluster set is less than a slope threshold; Each time, two first line segments are sampled from different cluster sets, and the distance between the intersection point of the two first line segments and the lines where the remaining first line segments except the two first line segments are located is determined, and the total number of first line segments whose distances are less than the first distance threshold is determined as the number of inner points corresponding to the intersection point; The intersection point with the largest number of inliers is determined as the vanishing point corresponding to the current frame image.

2. The method according to claim 1, characterized in that The matching of the first line feature and the second line feature based on the point feature matching result to obtain a line feature matching result includes: Determine a first target point corresponding to each first line segment according to a first distance between a first feature point corresponding to the current frame image in the point feature matching result and each first line segment in the current frame image; Determine a second target point corresponding to each second line segment according to a second distance between a second feature point corresponding to the previous frame image in the point feature matching result and each second line segment in the previous frame image; Combining one of the first line segments and one of the second line segments into a line segment pair; According to the point feature matching result, determining the number of matching pairs between the first target point corresponding to the first line segment and the second target point corresponding to the second line segment in each line segment pair; Determine a maximum value among a first number of first target points corresponding to the first line segment and a second number of second target points corresponding to the second line segment in each of the line segment pairs; The line segment pairs for which the ratio between the number of matching pairs and the maximum value is greater than the score threshold are determined as the line feature matching results.

3. The method according to claim 1, characterized in that The formula for determining the first position information corresponding to the camera based on the vanishing point and the intrinsic parameters of the camera is: Among them, α is the first pitch angle, β is the first yaw angle, k is the intrinsic parameter of the camera, and v P is the coordinate of the vanishing point, and the first pitch angle and the first yaw angle are the first posture information.

4. The method according to claim 1, characterized in that: Determining second pose information corresponding to the camera based on the first pose information and the point feature matching result includes: Randomly extracting four groups of matching point pairs from the point feature matching results, and determining a transformation matrix between the previous frame image and the current frame image based on the four groups of matching point pairs; Based on the transformation matrix, determining an initial candidate pose information, wherein the initial candidate pose information includes a second pitch angle and a second yaw angle; The operation of randomly extracting four groups of matching point pairs is executed cyclically to obtain multiple initial candidate pose information; Based on the similarity between the first pose information and each of the initial candidate pose information, screening out target candidate pose information from the multiple initial candidate pose information; Fitting the transformation matrix corresponding to the target candidate pose to obtain the target transformation matrix; Based on the target transformation matrix, second pose information corresponding to the camera is determined.

5. The method according to claim 4, characterized in that Also includes: The line feature matching results are screened based on the target transformation matrix.

6. The method according to claim 1, characterized in that The step of optimizing the second pose information based on the constructed cost function, the point feature matching result, and the line feature matching result to obtain the target pose information of the camera includes: Determine a first coordinate corresponding to a first feature point in each matching point pair in the point feature matching result in the image coordinate system and a second coordinate corresponding to a second feature point in the world coordinate system as a set of point projection pairs, wherein the first feature point is a feature point in the current frame image, and the second feature point is a feature point in the previous frame image; Determine a set of line projection pairs by taking the position of the first line segment in each matching line segment pair in the line feature matching result in the image coordinate system and the third coordinates corresponding to the two endpoints of the second line segment in the world coordinate system, wherein the first line segment is a line segment in the current frame image, and the second line segment is a line segment in the previous frame image; Determine a projection error corresponding to the second pose information based on the cost function, the point projection pair, and the line projection pair; Updating the second pose information, and calculating the projection error corresponding to the updated second pose information; The pose information with the smallest projection error is determined as the target pose information corresponding to the camera.

7. The method according to claim 6, characterized in that The cost function is: Where C(T) is the projection error, T1 is the second pose information, π is the projection function, N is the total number of the point projection pairs, p i is the second coordinate of the projection pair of the i-th point, q i is the first coordinate of the projection pair of the i-th point, ‖π(T1·p i )-q i ‖ means that p is converted to i The distance between the coordinate projected to the image coordinate system and the first coordinate, represents the point feature projection error, σ is the weight corresponding to the point feature projection error, M is the total number of line projection pairs, l j1 is the third coordinate corresponding to one endpoint of the second line segment in the j-th line projection pair, l j2 is the third coordinate corresponding to the other endpoint of the second line segment in the j-th line projection pair, m j is the position of the first line segment in the j-th line projection pair, d(π(T1·l j1 ),m j ) indicates that l is connected through T1 and π j1 The distance between the coordinate projected to the image coordinate system and the first line segment, d(π(T1·l j2 ),m j ) indicates that l is connected through T1 and π j2 The distance between the coordinate projected to the image coordinate system and the first line segment, represents the line feature projection error, and ω is the weight corresponding to the line feature projection error.

8. The method according to claim 1, characterized in that The line feature extraction comprises the following steps: Performing line segment detection on the image to obtain a first line segment detection result; Merge two line segments in the first line segment detection result whose angle difference is less than the angle threshold and whose distance between the two closest endpoints is less than the second distance threshold, and merge two line segments whose angle difference is less than the angle threshold and whose distance between the midpoints is less than the third distance threshold, to obtain a second line segment detection result; Line segments whose lengths in the second line segment detection result are less than a length threshold are filtered out to obtain line features.

9. A device for estimating the point-line combined with camera pose of underground tunnel images based on vanishing point prior, characterized in that: include: An extraction module is used to perform line feature extraction and point feature extraction on a current frame image and a previous frame image captured by a camera, respectively, to obtain a first line feature and a first point feature corresponding to the current frame image, and a second line feature and a second point feature corresponding to the previous frame image; A first matching module, used for matching the first point feature and the second point feature to obtain a point feature matching result; A second matching module, configured to match the first line feature and the second line feature based on the point feature matching result to obtain a line feature matching result; A first determination module, configured to determine a vanishing point corresponding to the current frame image based on the first line feature, and to determine first position information corresponding to the camera based on the vanishing point and an intrinsic parameter of the camera; A second determination module, configured to determine second posture information corresponding to the camera based on the first posture information and the point feature matching result; An acquisition module, configured to optimize the second pose information based on the constructed cost function, the point feature matching result, and the line feature matching result, so as to acquire the target pose information of the camera; The first determining module is further configured to cluster the first line segments in the first line feature based on the slope of each first line segment in the first line feature to obtain a cluster set, wherein a slope difference between every two line segments in each cluster set is less than a slope threshold; Each time, two first line segments are sampled from different cluster sets, and the distance between the intersection point of the two first line segments and the lines where the remaining first line segments except the two first line segments are located is determined, and the total number of first line segments whose distances are less than the first distance threshold is determined as the number of inner points corresponding to the intersection point; The intersection point with the largest number of inliers is determined as the vanishing point corresponding to the current frame image.

Images