Incremental three-dimensional line segment reconstruction method, system, equipment and medium

Abstract

The invention discloses an incremental three-dimensional line segment reconstruction method, system and device and a medium, and is based on the field of image processing, and the method comprises the steps: obtaining a plurality of 2D line segments; matching each 2D line segment to generate a corresponding 3D line segment hypothesis; a 3D line segment hypothesis graph is constructed according to the generated 3D line segment hypotheses, nodes in the 3D line segment hypothesis graph represent the 3D line segment hypotheses, and edges in the 3D line segment hypothesis graph are connected with nodes sharing the same 2D line segment; performing iterative calculation on the 3D line segment hypothesis diagram to construct a new 3D line segment hypothesis diagram and a new 3D line segment map; according to the method, the system, the equipment and the medium, linear trajectory merging is carried out on the 3D line segments in the final 3D line segment map, joint optimization is carried out on the 3D points, the 3D line segments and the blanking points, then outer point filtering is carried out, the reconstructed 3D line segments and linear trajectories are obtained, and the 3D line segments and the linear trajectories can be completely and correctly generated through the method, the system, the equipment and the medium.

Description

Technical Field

[0001] The invention belongs to the field of image processing and relates to an incremental three-dimensional line segment reconstruction method, system, equipment and medium. Background Art

[0002] Typically, most 3D reconstruction algorithms use 3D points to represent scene structures. However, using 3D points to represent weakly textured regions and slender structures is both difficult and redundant. Therefore, 3D line segments have recently become popular because they can effectively and concisely represent these scenes in artificial environments.

[0003] Given a camera pose, 3D line reconstruction aims to generate 3D line segments and straight line trajectories. However, since the traditional method uses one-to-many 2D line segment matching, a large number of external point matches are introduced, which weakens the robustness of the reconstruction algorithm. In addition, in the traditional method, since the reconstruction of the 3D line segment corresponding to each 2D line segment is run independently, it is difficult to ensure that the external point matches identified when reconstructing different 3D line segments are consistent, which makes it difficult for the traditional method to identify the true external point matches, and therefore cannot completely and correctly generate 3D line segments and straight line trajectories. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide an incremental three-dimensional line segment reconstruction method, system, device and medium, which can completely and correctly generate 3D line segments and straight line trajectories.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] In one aspect of the present invention, the incremental three-dimensional line segment reconstruction method of the present invention comprises:

[0007] Get several 2D line segments;

[0008] Generate a corresponding 3D line segment hypothesis for each 2D line segment match;

[0009] Constructing a 3D line segment hypothesis graph according to the generated 3D line segment hypotheses, wherein nodes in the 3D line segment hypothesis graph represent 3D line segment hypotheses, and edges in the 3D line segment hypothesis graph connect nodes that share the same 2D line segment;

[0010] Iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map;

[0011] The 3D line segments in the final 3D line segment map are merged into straight line trajectories, and the 3D points, 3D line segments and hidden points are jointly optimized, and then the external points are filtered to obtain the reconstructed 3D line segments and straight line trajectories.

[0012] The incremental three-dimensional line segment reconstruction method of the present invention is further improved in that:

[0013] Furthermore, the process of generating corresponding 3D line segment hypotheses for each 2D line segment matching is as follows:

[0014] Based on the hybrid RANSAC framework, a corresponding 3D line segment hypothesis is generated for each 2D line segment match.

[0015] Furthermore, the hybrid RANSAC framework includes a Line-Line solver, a Two-Points solver and a Point-VP solver.

[0016] Furthermore, the process of iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map is as follows:

[0017] Assign strength scores to each node in the 3D line segment hypothesis graph;

[0018] Select the node with the highest strength score as the best 3D line segment hypothesis, and add the best 3D line segment hypothesis to the new 3D line segment map;

[0019] Extending the straight line trajectory of the optimal 3D line segment hypothesis;

[0020] filtering outliers based on the reprojection of the best 3D line hypothesis and removing the 3D line hypothesis generated by the outlier matching from the 3D line hypothesis graph;

[0021] It is determined whether the strength score of each node in the 3D line segment hypothesis graph is less than a preset threshold. When the strength score of each node in the 3D line segment hypothesis graph is less than the preset threshold, the final new 3D line segment map is obtained. Otherwise, the node with the highest strength score is reselected as the best 3D line segment hypothesis.

[0022] Furthermore, the node Strength score for:

[0023]

[0024] in, For two 3D line segments and L c The similarity score between Representation and Node A collection of connected nodes.

[0025] Furthermore, the process of jointly optimizing the 3D points, 3D line segments and vanishing points is as follows:

[0026] The 3D points, 3D line segments, and vanishing points are jointly optimized by minimizing E, where E is:

[0027]

[0028] Among them, E P (P) is the square of the point reprojection error, E L (L) is the square of the line reprojection error, E PL (P,L) encodes the perpendicular distance between the associated SfM point and the 3D line segment, the angle between the 3D orientation of the associated VP and the 3D line segment, and the orthogonality regularization term of the VP.

[0029] Furthermore, the square of the line reprojection error E L (L) is:

[0030]

[0031] w ∠ (L k ,l k )=exp(α(1-cos(∠(L k , l k ))))

[0032] Among them, l k is the detected 2D line segment, L k Project the 3D line segment L onto l k The 2D infinite line on the image, e perp (L k ,l k ) is l k The two endpoints to L k The root of the sum of the squares of the vertical distances, α equals 10.0.

[0033] In a second aspect of the present invention, the incremental three-dimensional line segment reconstruction system of the present invention comprises:

[0034] An acquisition module, used to acquire a number of 2D line segments;

[0035] A generation module, used to generate corresponding 3D line segment hypotheses for each 2D line segment matching;

[0036] A construction module, configured to construct a 3D line segment hypothesis graph according to the generated 3D line segment hypotheses, wherein nodes in the 3D line segment hypothesis graph represent 3D line segment hypotheses, and edges in the 3D line segment hypothesis graph connect nodes that share the same 2D line segment;

[0037] An iterative calculation module, used for iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map;

[0038] The post-processing module is used to merge the 3D line segments in the final 3D line segment map into straight line trajectories, and jointly optimize the 3D points, 3D line segments and hidden points, and then perform external point filtering to obtain the reconstructed 3D line segments and straight line trajectories.

[0039] In a third aspect of the present invention, the computer device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the incremental three-dimensional line segment reconstruction method when executing the computer program.

[0040] In a fourth aspect of the present invention, the computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the steps of the incremental three-dimensional line segment reconstruction method are implemented.

[0041] The present invention has the following beneficial effects:

[0042] The incremental three-dimensional line segment reconstruction method, system, device and medium described in the present invention, when in specific operation, generates corresponding 3D line segment hypotheses for each 2D line segment matching, effectively integrates the geometric constraints provided by 3D points and vanishing points, can solve the degradation problem and improve the reconstruction accuracy. In addition, the 3D line segment hypothesis graph is iteratively calculated to construct a new 3D line segment hypothesis graph and a new 3D line segment map to enhance the robustness to external points; finally, the 3D line segments in the final 3D line segment map are merged into straight line trajectories, and the 3D points, 3D line segments and vanishing points are jointly optimized, and then the external points are filtered to improve the reconstruction quality, thereby generating longer straight line trajectories, and then completely and correctly generating 3D line segments and straight line trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0044] Figure 1a is a schematic diagram of a 2D line segment;

[0045] Figure 1b 3D line segment map reconstructed for L3D++;

[0046] Figure 1c 3D line segment map reconstructed for LIMAP;

[0047] Figure 1d A 3D line segment map reconstructed by the present invention;

[0048] Figure 2 The schematic diagram of the method of the present invention;

[0049] Figure 3a This is the schematic diagram of the Line-Line solver;

[0050] Figure 3b This is the schematic diagram of the Two-Points solver;

[0051] Figure 3c This is the schematic diagram of the Point-VP solver;

[0052] Figure 4 is a schematic diagram of the present invention in one iteration;

[0053] Figure 5 3D line segment maps on Hypersim and Tanks and Temples datasets in validation experiments;

[0054] Figure 6 This is the intermediate result image of incremental reconstruction on the Hypersim dataset ai_001_001 scene in the verification experiment;

[0055] Figure 7 is a system structure diagram of the present invention;

[0056] Figure 8 It is a structural diagram of the iterative calculation module. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only an embodiment of a part of the present invention, not all embodiments, and is not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concepts disclosed in the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0058] The accompanying drawings show schematic diagrams of structures according to embodiments disclosed in the present invention. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0059] Embodiment 1

[0060] refer to Figure 2 The incremental three-dimensional line segment reconstruction method based on the three-dimensional line segment hypothesis graph of the present invention comprises the following steps:

[0061] 1) Get several 2D line segments;

[0062] 2) Based on the hybrid RANSAC framework, a corresponding 3D line segment hypothesis is generated for each 2D line segment matching;

[0063] The process of step 2) is:

[0064] set up For 2D line segment matching (l i ,l j ) generated 3D line segment hypothesis, where l i and j For two source 2D line segments, since one-to-many matching is used, there are a large number of outlier matches, which will generate a large number of outlier 3D line segment hypotheses.

[0065] In the traditional two-view 3D line segment reconstruction method, LIMAP uses four solvers to generate multiple 3D line segment hypotheses for a 2D line segment match. The four solvers are Line+Line solver, Multiple Points solver, Line+Point solver and Line+VP solver. This method incorporates all generated 3D line segment hypotheses into the reconstruction process, which increases redundancy and improves the external point rate. In order to solve this problem, the goal of the present invention is to reconstruct an accurate 3D line segment hypothesis from each 2D line segment match. In addition, in order to solve the degradation problem and improve the accuracy of two-view reconstruction, the present invention uses a hybrid RANSAC framework to integrate multiple solvers to effectively utilize the geometric constraints provided by 3D points and vanishing points.

[0066] In the hybrid RANSAC framework, the model is defined as a 3D infinite line. The present invention has three types of data, namely, a 2D line segment matching (l i ,l j ), 3D point set and vanishing point set Among them, the collection Contains i and j The associated SfM points, where the 2D feature points corresponding to each element are also in l i and j on; collection Contains i or j The associated vanishing point, where each element is l i or j The corresponding vanishing point is or The present invention relates to three solvers: Line-Line solver, Two-Points solver and Point-VP solver. For Line-Line solver, take l i and l j The intersection line of the back-projection plane is the 3D infinite length straight line L inf ,like Figure 3a As shown, the Line-Line solver is similar to the Line+Line solver of LIMAP; for the Two-Points solver, two points belonging to the set 3D points to determine L inf ,like Figure 3b As shown; for the Point-VP solver, use a set A 3D point and a belonging to the set The 3D direction corresponding to the vanishing point is used to determine L inf ,like Figure 3c shown.

[0067] It should be noted that the reference Figure 1a , Figure 1b , Figure 1c and Figure 1d As shown, the traditional Line-Line solver is inherently unstable in the degenerate case, while the two solvers provided by the present invention are not affected. Compared with the MultiplePoints, Line+Point and Line+VP solvers of LIMAP, the Two-Points solver and Point-VP solver in the present invention can estimate a 3D infinite length line using the least data.

[0068] Given the estimated model L inf , first verify the 2D line segment matching (l i ,l j ) is an interior point. Specifically, i and l j The endpoints are projected onto L inf On the top, generate L i and L j , when L i and L j No overlap, or L inf and l i The reprojection score is equal to 0, or L inf and l j When the reprojection score of is equal to 0, then (l i ,l j ) is regarded as an external point, L inf Will be discarded.

[0069] It should be noted that the definition of the reprojection score is: let L be a 3D line segment or a 3D infinite line, l be a 2D line segment, π(L) be the 2D line segment or 2D infinite line projected from L onto the image where l is located, and the reprojection score is:

[0070]

[0071] Among them, s r Represents a normalized score, s r Used to quantify a certain type of distance r, the set Contains three normalized scores related to the angle a 2( , vertical distance p 2( and overlap ratio o 2( , where L is a 3D infinite line, then Not included Specifically, angle a 2( is the angle between l and π(L), and the vertical distance between the end points is p 2( is the maximum vertical distance from the two endpoints of l to the 2D infinite line where π(L) is located, and is used to determine the overlap ratio o 2( , first project π(L) orthogonally onto l, and when the endpoint of the projection exceeds l, clip it to obtain a 2D line segment Π(π(L)). Let o 2( is the length of Π(π(L)) divided by the length of l, that is, and The calculation formula is Among them, τ r is the scale factor corresponding to a certain distance r, and the scale factor is set as Spend, Pixel, when o 2( >0, then is set to 1; otherwise, is set to 0.

[0072] For identification In the interior point, first calculate the 3D point To L inf The vertical distance t is calculated to obtain scale invariance, and the scale factor σ = median (σ k ), where σ k =d k / f k , d k is the depth of p in its observed image, f kis the focal length of the observed image. The scale factor encodes the distance that a 3D point can move in 3D when reaching 1 pixel error in the image. When the vertical distance t is less than σ, the 3D point p is considered an inlier. Let The inner point in is the angle between L imf The vanishing point is less than 5 degrees.

[0073] In the entire RANSAC process, since there is only one 2D line segment matching data, the Line-Line solver can only produce one model. Therefore, the model estimated by the Line-Line solver is used to initialize the hybrid RANSAC. In each RANSAC iteration, according to the process of the standard hybrid RANSAC framework, the Two-Points solver or the Point-VP solver is selected to estimate the model. When the 2D line segment is matched (l i ,l j ) is not an interior point, then the model is discarded; otherwise, according to and The number of inliers in the model is used to score the model. When the number of new inliers exceeds the number of inliers in the best model, the best model is updated. Finally, when the Two-Points solver is selected K B RANSAC stops when the Point-VP solver is selected K2 times, where

[0074]

[0075] Among them, ∈ p and ∈ Dp Respectively in and The estimated proportion of inliers in the , the probability P is set to 0.999, and the final 3D line segment hypothesis For the best L inf The corresponding L i and L j The union of .

[0076] 3) constructing a 3D line segment hypothesis graph according to the generated 3D line segment hypotheses, wherein the nodes in the 3D line segment hypothesis graph represent the 3D line segment hypotheses, and the edges in the 3D line segment hypothesis graph connect the nodes that share the same 2D line segment;

[0077] 4) performing iterative calculation on the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map;

[0078] The process of step 4) is:

[0079] 41) Score the strength of each node in the 3D line segment hypothesis graph;

[0080] Specifically, when the 3D line segment is assumed is an interior point, then the 2D line segment matches (l i ,l j ) is an interior point, and There are several l gathered around i and l j The 3D line segment hypothesis is reconstructed from two views with the respective correctly matched 2D line segments; when is an external point, then It's unlikely that there are many other 3D line segment hypotheses clustered around.

[0081] For measurement is the possibility of an interior point, first collect l i and its 2D matching line segment The corresponding 3D line segment hypothesis and j and its 2D matching line segment The corresponding 3D line segment hypothesis And build the collection Then, from Remove in,

[0082] To measure the similarity of two 3D line segments, let two 3D line segments L B The similarity score p(L B ,L2) is:

[0083]

[0084] Among them, p 2( (L B ,L2) represents the 2D similarity score, p N( (L B ,L2) represents the 3D similarity score, s r is a normalized score, s r Quantify a certain type of distance r, let and and L B and the straight line trajectory corresponding to L2, when L B and L2 are two 3D line segments, then and For L B and the source 2D line segment of L2, let For the collection Remove The set of elements remaining after the elements in , similarly, let For the collection Remove The set of elements remaining after the elements in .

[0085] To calculate the 2D similarity score p 2( (L B ,L2), for the set For each element in L B The reprojection score of Each element in calculates the L2 reprojection score, the 2D similarity score p 2( (L B ,L2) is the minimum value of all reprojection scores, p 2( (L B ,L2) is represented by a set of normalized scores. Contains the normalized scores of all reprojection scores, let m and n be the sets and The number of elements in , the total number of reprojection scores is m+n, The total number of normalized scores in is 3m+3n, and each reprojection score involves 3 normalized scores and

[0086] and Similar, collection Contains three normalized scores related to angle a N( , the scale-invariant endpoint vertical distance p N( and overlap ratio o N( , specifically, angle a N( For L B and L2, let p N( For L B and the maximum perpendicular distance between the endpoints of L2 will penalize longer 3D line segments, which are crucial for concise and clear reconstruction. B and L2 are projected onto each other's straight lines to obtain two 3D line segment projections. Let p N( is the maximum vertical distance from the two endpoints of the 3D line segment projection to each other’s 3D line segment projection. To obtain scale invariance, p N( Divide by min(σ B ,σ2), where is the depth value of the midpoint of the two 3D endpoints, and the two 3D endpoints are the 2D endpoints of the kth trajectory element (2D line segment) back-projected to L i The projection point on the 3D infinite line, is the focal length of the image where the kth trajectory element (2D line segment) is located, and the overlap ratio is o N( By adding L Band L2 are projected onto each other's straight lines and the maximum length ratio between the 3D line segment projection and the original 3D line segment is calculated to determine, and The calculation formula is Among them, τ r is the scale factor corresponding to a certain distance r, Spend, When o N( >0, then is set to 1; otherwise, is set to 0, the set for and The union of .

[0087] For incremental reconstruction, a weighted undirected graph is constructed Among them, the node represents the assumption that the initial edge ε connects two nodes that share the same source 2D segment. Representation and Node A set of connected nodes, based on the way graph G is constructed, the initial equal Let the weight of the edge be the similarity score of the two nodes. When the similarity score is equal to 0, the corresponding edge will be deleted. In graph theory, the strength of a node is the sum of the weights of the edges connected to the node. The strength score is:

[0088]

[0089] The larger the strength score of the node, the greater the possibility that the node is an inner point, because the neighboring nodes give the node stronger geometric support.

[0090] 42) selecting the node with the highest strength score as the best 3D line segment hypothesis, and adding the best 3D line segment hypothesis to the new 3D line segment map;

[0091] Specifically, upon initialization, the new 3D line segment map is blank.

[0092] 43) extending the straight line trajectory of the optimal 3D line segment hypothesis;

[0093] The process of step 43) is:

[0094] refer to Figure 4 (a) For the node with the highest strength score, the goal of the present invention is to extend its straight trajectory by adding more trajectory elements, i.e., 2D line segments. To this end, all 2D line segments matching the initial trajectory elements are collected and put into the set {l E} and use The reprojection is tested, such as Figure 4 As shown in (b), when the 2D line segment l N ∈{l E}and If the reprojection score is greater than 0, the 2D line segment l N Add to In the straight line trajectory, then l N The two endpoints are back-projected to The 3D infinite line L inf Come up and extend The above extension process is repeated until all 2D line segments matching the trajectory elements and The reprojection score of is equal to 0, i.e., all 2D line segments matching the trajectory elements fail the reprojection test.

[0095] 44) filtering the outlier matches according to the reprojection of the best 3D line segment hypothesis, and removing the 3D line segment hypotheses generated by the outlier matches from the 3D line segment hypothesis graph;

[0096] The process of step 44) is:

[0097] After extending the straight line trajectory, when one source 2D line segment is a trajectory element and the other source 2D line segment is not, the node is considered as an outlier, because the two source 2D line segments come from a 2D line segment match, but one of the source 2D line segments does not pass the reprojection test, so the 2D line segment match is considered as an outlier, and the node reconstructed according to the 2D line segment match is an outlier. In order to prevent the outlier node from providing erroneous geometric support for other nodes, thereby affecting the best node selection in the next round of incremental reconstruction, in the present invention, the above-mentioned outlier nodes are filtered out from the graph G, as shown in FIG. Figure 4 (b) shows that, because l [ and The reprojection score of is equal to 0, so delete Figure 4 Nodes in (c) Outlier nodes may be In addition, the nodes whose source 2D segments are all trajectory elements are also filtered out. The reason is that the source 2D segment of the node has corresponded to the reconstructed 3D segment, and the node cannot connect to the nodes whose source 2D segments are not trajectory elements. Therefore, the node cannot affect the next best node selection, so it is filtered out.

[0098] 45) Determine whether the strength score of each node in the 3D line segment hypothesis graph is less than a preset threshold. When the strength score of each node in the 3D line segment hypothesis graph is less than the preset threshold, output the new 3D line segment map. Otherwise, recalculate the strength scores of all nodes and go to step 42).

[0099] 5) The 3D line segments in the final 3D line segment map are merged into straight line trajectories, and the 3D points, 3D line segments and hidden points are jointly optimized, and then the external points are filtered to obtain the reconstructed 3D line segments and straight line trajectories.

[0100] In step 5), the process of merging the straight line trajectories of the 3D line segments in the final 3D line segment map is as follows:

[0101] To improve efficiency and robustness, 2D line segments are matched only in adjacent images. However, this local matching may cause the same 3D line segment entity to be observed by multiple straight line trajectories, but there is no match between the elements of each trajectory. In order to merge disconnected straight line trajectories, the present invention establishes a fully connected weighted undirected graph, wherein the nodes in the weighted undirected graph are 3D line segments, and the weights of the edges are the 3D similarity scores of the two nodes. Then, the edges are arranged in descending order according to their weights to obtain a list of edges, and the traversal starts from the first element in the list of edges, and the Union-Find algorithm is used to merge the two nodes corresponding to the edge. Initially, the 3D line segment of each group is the node itself. In each traversal, when the 3D similarity score between the 3D line segments of the two groups is greater than 0, principal component analysis (PCA) and the 3D line segments corresponding to the two groups are applied to fit a 3D line segment to be merged. When the reprojection scores of all trajectory elements and the 3D line segments to be merged in the two groups are greater than 0, the Union-Find groups where the two nodes are located are merged, and the 3D line segments and straight line trajectories of the merged new group are the union of the 3D line segments to be merged and the straight line trajectories of the two groups. Repeat the above merging operation until no more 3D line segments can be merged.

[0102] In step 5), the process of jointly optimizing 3D points, 3D line segments and vanishing points is as follows:

[0103] To improve the quality of 3D line segment reconstruction, the 3D line segments, SfM points and vanishing points are jointly optimized by minimizing the following energy function:

[0104]

[0105] Among them, E P is the square of the point reprojection error, E L is the square of the line reprojection error, E L for:

[0106]

[0107] w ∠ (L k , l k )=exp(α(1-cos(∠(L k , l k ))))

[0108] Among them, l k is the detected 2D line segment, L k Project the 3D line segment L onto l k The 2D infinite line on the image, e perp (L k ,l k ) is l k The two endpoints to L k The root of the sum of the squares of the vertical distances, α equals 10.0, function E PL (P,L) encodes the perpendicular distance between the associated SfM point and the 3D line segment, the angle between the 3D direction of the associated VP and the 3D line segment, and the orthogonal regularization term of the VP. The SfM point associated with the 3D line segment is determined by the 2D point-line association, SfM point trajectory and line trajectory, and the VP associated with the 3D line segment is determined by the 2D line-VP association and line trajectory.

[0109] In step 5), the process of outlier filtering is:

[0110] Reprojection is used to filter outlier trajectory elements. Specifically, when the reprojection score of any trajectory element and its corresponding 3D line segment is equal to 0, the trajectory element is identified as an outlier. All inlier trajectory elements are back-projected onto the 3D infinite straight line where the original 3D line segment is located to obtain a series of 3D projections. The new 3D line segment is the union of these 3D projections. To enhance robustness, only 3D line segments that are visible in at least four images are retained.

[0111] Confirmatory test

[0112] In this experiment, the proposed IncreLM is compared with two state-of-the-art methods, L3D++ and LIMAP. The synthetic Hypersim dataset and the real Tanks and Temples dataset are used in the experiment. The 2D line segments are generated by the LSD line detector or the DeepLSD line detector, and the 2D line segment matching is generated by the GlueStick matcher. The 2D line segment matching input to all methods is the same. All experiments are performed on a computer with a 64GHz processor.

[0113] a) Evaluation of 3D line segment reconstruction;

[0114] Since there is no ground truth (GT) 3D line segment, the GT point cloud is used as the GT model, and the following five indicators are used to evaluate the reconstructed 3D line segments and straight line trajectories;

[0115] a1) Length recall within τ mm (Rτ): the sum of the lengths of 3D line segments within τ mm of the GT model;

[0116] a2) Percentage of inliers within τ mm (Pτ): percentage of 3D line segments within τ mm of the GT model;

[0117] a3) Average support (#support): average image support and 2D line segment support of all straight line trajectories;

[0118] a4) Number of 3D line segments (#3D lines): the number of reconstructed 3D line segments;

[0119] a5) Running time (time[s]): The running time of the reconstruction algorithm.

[0120] Three-dimensional line segment reconstruction was performed on the first 8 scenes of the Hypersim dataset, and the average indicators are shown in Table 1. The present invention reconstructed the most 3D line segments, achieved the highest length recall, and obtained the most image support numbers, and had a good 2D line segment support number, indicating that the present invention can generate a more complete 3D line segment map and a longer straight line trajectory. It should be noted that since L3D++ uses GPU parallel computing and omits the processing of 3D points and hidden points in the reconstruction stage, it has the fastest running speed. However, L3D++ performs poorly in terms of map integrity. In addition, both the present invention and LIMAP integrate 3D points and hidden points to improve the reconstruction quality, but the present invention surpasses LIMAP in speed, which proves the high efficiency of the present invention.

[0121] Table 2 shows the average indicators in the Tanks and Temples train dataset. The present invention removes the scene Ignatius with almost no straight line structure. The present invention achieves the highest results in length recall, average support number and number of 3D line segments, and runs significantly faster than LIMAP. In addition, in the experiment based on LSD line segments, the present invention also achieves the highest percentage of inliers at 5 mm and 10 mm, which further proves the superiority of the present invention.

[0122] Figure 5 The quantitative results are shown, where the first row is a schematic diagram of 2D line segments detected by the DeepLSD algorithm, the second row is a 3D line segment map reconstructed by L3D++, the third row is a 3D line segment map reconstructed by LIMAP, and the last row is a 3D line segment map reconstructed by the IncreLM of the present invention. The present invention produces a more complete 3D line segment map and is able to reconstruct very dense 3D line segments, which is helpful for many downstream computer vision tasks, such as visual positioning and surface reconstruction.

[0123] Table 1

[0124]

[0125] Table 2

[0126]

[0127] b) incremental reconstruction evaluation;

[0128] exist Figure 6 The intermediate results of incremental reconstruction are shown in , where the number of currently reconstructed 3D segments, the length recall within 5 mm, and the percentage of inliers within 5 mm are reported in sequence. As the incremental reconstruction proceeds, the percentage of inliers decreases while the length recall increases, which indicates that the present invention can preferentially reconstruct relatively correct 3D segments. Therefore, the previous 3D segment map provides valuable feedback for subsequent modeling, and the filtered outlier nodes are more likely to represent true outliers. This mechanism greatly enhances the robustness of outlier matching, embodying the core idea of ​​the present invention, using accurate 3D segment maps to detect outliers and guide subsequent modeling, which helps to produce more accurate and complete 3D segment maps.

[0129] c) 3D line segment reconstruction of two views;

[0130] This experiment conducted an ablation experiment on the solver used in hybrid RANSAC, and the results are shown in Table 3, where the checked items indicate that the corresponding solver was used. As shown in Table 3, the use of any single solver alone failed to obtain satisfactory results. The 3D point-based solvers (i.e., Two-Points solver and Point-VP solver) had a higher percentage of inliers but lower length recall. Only 0.2% of 2D line segment matches were eligible to use the Two-Points solver (i.e., only 0.2% of 2D line segment matches had at least two associated SfM points), and only 0.6% of line matches were eligible to use the Point-VP solver (i.e., only 0.6% of 2D line segment matches had at least one associated SfM point and one associated VP), which indicates that although 3D points are reliable, they are too sparse for effective 3D line segment reconstruction. In contrast, the Line-Line solver has a higher length recall but a lower inlier percentage, indicating that dense but inaccurate 2D line segment matching is not suitable for robust 3D line segment reconstruction. By integrating these solvers into a unified hybrid RANSAC framework, the length recall is improved and the inlier percentage is between the Line-Line solver and the 3D point-based solver.

[0131] Table 3

[0132]

[0133] d) Linear trajectory merging;

[0134] This experiment conducted an ablation experiment on the straight line trajectory merging module. As shown in Table 4, the use of the straight line trajectory merging module can increase the average trajectory length before the joint optimization, and the final trajectory length does not increase more. The reason is that the present invention merges 3D line segments with less than four supporting images that should have been filtered, which allows new 3D line segments with shorter trajectories to be added to the final result, thereby suppressing the growth of the average trajectory length. In addition, the use of the straight line trajectory merging module significantly improves the length recall, which proves the effectiveness of the present invention.

[0135] Table 4

[0136]

[0137] It should be noted that in Table 4, the results marked with "*" are the results input to the joint optimization module, and the other results are the final output results.

[0138] Embodiment 2

[0139] refer to Figure 7 The incremental three-dimensional line segment reconstruction system of the present invention comprises:

[0140] An acquisition module, used to acquire a number of 2D line segments;

[0141] A generation module, used to generate corresponding 3D line segment hypotheses for each 2D line segment matching;

[0142] A construction module, configured to construct a 3D line segment hypothesis graph according to the generated 3D line segment hypotheses, wherein nodes in the 3D line segment hypothesis graph represent 3D line segment hypotheses, and edges in the 3D line segment hypothesis graph connect nodes that share the same 2D line segment;

[0143] An iterative calculation module, used for iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map;

[0144] The post-processing module is used to merge the 3D line segments in the final 3D line segment map into straight line trajectories, and jointly optimize the 3D points, 3D line segments and hidden points, and then perform external point filtering to obtain the reconstructed 3D line segments and straight line trajectories.

[0145] refer to Figure 8 As an implementation mode of the present invention, the iterative calculation module includes:

[0146] A scoring module is used to score the strength of each node in the 3D line segment hypothesis graph;

[0147] A selection module, for selecting a node with the highest strength score as the best 3D line segment hypothesis, and adding the best 3D line segment hypothesis to a new 3D line segment map;

[0148] An extension module, used for extending the straight line trajectory of the optimal 3D line segment hypothesis;

[0149] A filtering module, configured to filter outliers according to the reprojection of the best 3D line segment hypothesis and delete the 3D line segment hypothesis generated by the outlier matching from the 3D line segment hypothesis graph;

[0150] The judgment module is used to judge whether the strength score of each node in the 3D line segment hypothesis map is less than a preset threshold. When the strength score of each node in the 3D line segment hypothesis map is less than the preset threshold, the final new 3D line segment map is obtained. Otherwise, the node with the highest strength score is reselected as the best 3D line segment hypothesis.

[0151] As an implementation of the present invention, the process of generating a corresponding 3D line segment hypothesis for each 2D line segment matching is: based on a hybrid RANSAC framework, generating a corresponding 3D line segment hypothesis for each 2D line segment matching.

[0152] As an implementation manner of the present invention, the hybrid RANSAC framework includes a Line-Line solver, a Two-Points solver and a Point-VP solver.

[0153] As an implementation mode of the present invention, a node Strength score for:

[0154]

[0155] in, For two 3D line segments and L c The similarity score between Representation and Node A collection of connected nodes.

[0156] As an implementation mode of the present invention, the process of jointly optimizing 3D points, 3D line segments and vanishing points is as follows:

[0157] The 3D points, 3D line segments, and vanishing points are jointly optimized by minimizing E, where E is:

[0158]

[0159] Among them, E P (P) is the square of the point reprojection error, E L (L) is the square of the line reprojection error, E PL (P,L) encodes the perpendicular distance between the associated SfM point and the 3D line segment, the angle between the 3D orientation of the associated VP and the 3D line segment, and the orthogonality regularization term of the VP.

[0160] As an implementation of the present invention, the square of the line reprojection error E L (L) is:

[0161]

[0162] w ∠ (L k ,l k )=exp(α(1-cos(∠(L k , l k ))))

[0163] Among them, l k is the detected 2D line segment, L k Project the 3D line segment L onto l k The 2D infinite line on the image, e perp (L k ,l k ) is l k The two endpoints to L k The root of the sum of the squares of the vertical distances, α equals 10.0.

[0164] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0165] Embodiment 3

[0166] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the incremental three-dimensional line segment reconstruction method are implemented, for example, including: obtaining a number of 2D line segments; generating a corresponding 3D line segment hypothesis for each 2D line segment match; constructing a 3D line segment hypothesis graph based on each generated 3D line segment hypothesis, wherein the nodes in the 3D line segment hypothesis graph represent 3D line segment hypotheses, and the edges in the 3D line segment hypothesis graph connect nodes that share the same 2D line segment; iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map; merging the straight line trajectories of the 3D line segments in the final 3D line segment map, and jointly optimizing the 3D points, 3D line segments and hidden points, and then filtering the external points to obtain the reconstructed 3D line segments and straight line trajectories. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, network interface, and memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0167] Embodiment 4

[0168] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the incremental three-dimensional line segment reconstruction method are implemented, for example, including: obtaining a number of 2D line segments; generating a corresponding 3D line segment hypothesis for each 2D line segment match; constructing a 3D line segment hypothesis graph according to each generated 3D line segment hypothesis, wherein the nodes in the 3D line segment hypothesis graph represent 3D line segment hypotheses, and the edges in the 3D line segment hypothesis graph connect the nodes sharing the same 2D line segment; iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map; merging the 3D line segments in the final 3D line segment map into straight line trajectories, and jointly optimizing the 3D points, 3D line segments and vanishing points, and then filtering the external points to obtain the reconstructed 3D line segments and straight line trajectories. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory, etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, and the like.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0170] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0171] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An incremental three-dimensional line segment reconstruction method, characterized in that: include: Get several 2D line segments; Generate a corresponding 3D line segment hypothesis for each 2D line segment match; Constructing a 3D line segment hypothesis graph according to the generated 3D line segment hypotheses, wherein nodes in the 3D line segment hypothesis graph represent 3D line segment hypotheses, and edges in the 3D line segment hypothesis graph connect nodes that share the same 2D line segment; Iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map; The 3D line segments in the final 3D line segment map are merged into straight line trajectories, and the 3D points, 3D line segments and hidden points are jointly optimized, and then the external points are filtered to obtain the reconstructed 3D line segments and straight line trajectories.

2. The incremental three-dimensional line segment reconstruction method according to claim 1, characterized in that: The process of generating corresponding 3D line segment hypotheses for each 2D line segment match is: Based on the hybrid RANSAC framework, a corresponding 3D line segment hypothesis is generated for each 2D line segment match.

3. The incremental three-dimensional line segment reconstruction method according to claim 2, characterized in that: The hybrid RANSAC framework includes a Line-Line solver, a Two-Points solver and a Point-VP solver.

4. The incremental three-dimensional line segment reconstruction method according to claim 1, characterized in that: The process of iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph is as follows: Assign strength scores to each node in the 3D line segment hypothesis graph; Select the node with the highest strength score as the best 3D line segment hypothesis, and add the best 3D line segment hypothesis to the new 3D line segment map; Extending the straight line trajectory of the optimal 3D line segment hypothesis; filtering outliers based on the reprojection of the best 3D line hypothesis and removing the 3D line hypothesis generated by the outlier matching from the 3D line hypothesis graph; It is determined whether the strength score of each node in the 3D line segment hypothesis graph is less than a preset threshold. When the strength score of each node in the 3D line segment hypothesis graph is less than the preset threshold, the final new 3D line segment map is obtained. Otherwise, the node with the highest strength score is reselected as the best 3D line segment hypothesis.

5. The incremental three-dimensional line segment reconstruction method according to claim 4, characterized in that: node Strength score for: in, For two 3D line segments and L c The similarity score between Representation and Node A collection of connected nodes.

6. The incremental three-dimensional line segment reconstruction method according to claim 1, characterized in that: The process of jointly optimizing 3D points, 3D line segments and vanishing points is as follows: The 3D points, 3D line segments, and vanishing points are jointly optimized by minimizing E, where E is: Among them, E P (P) is the square of the point reprojection error, E L (L) is the square of the line reprojection error, E PL (P,L) is used to encode the perpendicular distance between the associated SfM point and the 3D line segment, the angle between the 3D direction of the associated VP and the 3D line segment, and the orthogonality regularization term of the VP.

7. The incremental three-dimensional line segment reconstruction method according to claim 6, characterized in that: The square of the line reprojection error E L (L) is: w ∠ (L k ,L k )=exp(α(1-cos(∠(L k ,L k )))) Among them, l k is the detected 2D line segment, L k Project the 3D line segment L onto l k The 2D infinite line on the image, e perp (L k , l k ) is l k The two endpoints to L k The root of the sum of the squares of the vertical distances, α equals 10.

0.

8. An incremental three-dimensional line segment reconstruction system, characterized in that: include: An acquisition module, used to acquire a number of 2D line segments; A generation module, used to generate corresponding 3D line segment hypotheses for each 2D line segment matching; A construction module, configured to construct a 3D line segment hypothesis graph according to the generated 3D line segment hypotheses, wherein nodes in the 3D line segment hypothesis graph represent 3D line segment hypotheses, and edges in the 3D line segment hypothesis graph connect nodes that share the same 2D line segment; An iterative calculation module, used for iteratively calculating the 3D line segment hypothesis graph to construct a new 3D line segment hypothesis graph and a new 3D line segment map; The post-processing module is used to merge the 3D line segments in the final 3D line segment map into straight line trajectories, and jointly optimize the 3D points, 3D line segments and hidden points, and then perform external point filtering to obtain the reconstructed 3D line segments and straight line trajectories.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the incremental three-dimensional line segment reconstruction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the incremental three-dimensional line segment reconstruction method according to any one of claims 1 to 7 are implemented.

Images