Three-dimensional line segment reconstruction method, device, electronic device and storage medium
By modeling the best suggestions for two-dimensional line segments in image concentration into 0-1 planning problems and performing clustering and joint optimization, the problem of mismatch between the two-dimensional support line segment suggestions in three-dimensional line segment reconstruction is solved, and more accurate is achieved and complete three-dimensional line segment map construction.
Patent Information
- Application Number
- CN202410359798.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-03-27
AI Technical Summary
In the existing three-dimensional segment reconstruction methods, the one-to-many matching method cannot accurately reconstruct the three-dimensional segment due to the mismatch of the support relationship between the two-dimensional support segment and the three-dimensional segment proposal.
Model the best three-dimensional line segment suggestions for all two-dimensional line segments in the image center as a 0-1 planning problem. Select the three-dimensional line segment suggestions by maximizing the consistency score function, and build the initial three-dimensional line segment map based on clustering and joint optimization algorithms, enhance the The support relationship between two-dimensional support segments and three-dimensional segment suggestions.
A more accurate and complete three-dimensional line segment map reconstruction is achieved, improving the accuracy and consistency of three-dimensional line segment reconstruction.
Smart Images

Figure CN118397171B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional line segment reconstruction method, device, electronic equipment and storage medium. Background Art
[0002] Traditional 3D reconstruction methods, such as Structure from Motion (SfM) methods, mainly rely on sparse key points to generate point clouds. Unlike 3D points, 3D line segments (3D LS) encode high-level structural information about the scene, have lower redundancy, and can provide better representation for areas with weak textures and slender structures. In addition, 3D line segments often appear in man-made environments. The reconstructed 3D line segment map (3D LS map) has been applied to many other tasks, including visual localization, surface reconstruction, and point-based SfM result optimization.
[0003] Based on the reconstruction algorithm's requirement for 2D line segment matching, one of the existing 3D line segment reconstruction methods is a 3D line segment reconstruction method based on one-to-many matching. This method allows a 2D line segment in one image to match one or more 2D line segments in another image, and each pair of matches can generate one or more 3D line segment proposals (3D LS proposals) of the reference 2D line segment. This method mainly studies proposal generation, best proposal selection, and 3D line segment track building. This method can utilize more candidate matches and prior knowledge to generate rich 3D line segment proposals, thereby having the opportunity to reconstruct more complete and accurate 3D line segments.
[0004] Because 3D line segment reconstruction methods based on one-to-many matching essentially evaluate the quality of 3D line segment proposals using 2D supporting line segments, these 2D supporting line segments are assumed to be the 2D observations of the 3D line segment proposals. However, the support relationship between the 2D supporting line segments and the 3D line segment proposals may not match, making it impossible to accurately reconstruct the 3D line segments. Summary of the Invention
[0005] The present invention provides a three-dimensional line segment reconstruction method, device, electronic equipment and storage medium to solve the defects in the prior art.
[0006] The present invention provides a three-dimensional line segment reconstruction method, comprising:
[0007] Extracting two-dimensional line segments within each image in the image set and determining the neighborhood images of each image;
[0008] For any image in the image set, generating a three-dimensional line segment proposal for a two-dimensional line segment in the any image;
[0009] The problem of selecting all the best three-dimensional line segment proposals corresponding to all the two-dimensional line segments in the image set is modeled as a 0-1 programming problem; in the 0-1 programming problem, a consistency score function corresponding to any three-dimensional line segment proposal of any two-dimensional line segment in any image is determined based on the sum of the maximum similarity scores between any three-dimensional line segment proposal and all neighboring images of the any image; the maximum similarity score between any three-dimensional line segment proposal and any neighboring image of the any image is the maximum value of the similarity scores between the any three-dimensional line segment proposal and the best three-dimensional line segment proposals of all two-dimensional line segments in the any neighboring images that match the any two-dimensional line segment;
[0010] Solving the 0-1 programming problem to obtain all optimal three-dimensional line segment suggestions corresponding to the image set, and clustering all two-dimensional line segments in the image set based on all the optimal three-dimensional line segment suggestions to reconstruct an initial three-dimensional line segment map;
[0011] Acquire a three-dimensional point cloud, and identify a three-dimensional plane based on the three-dimensional point cloud; the three-dimensional point cloud includes sampling points in the initial three-dimensional line segment map and SfM points determined based on the image set using a structure-from-motion method;
[0012] Calculating a first summation result of reprojection errors of each 3D point in the 3D point cloud and a second summation result of reprojection errors of the 3D line segments in the initial 3D line segment map;
[0013] Calculating a first association relationship between the three-dimensional point and its associated three-dimensional plane;
[0014] Calculating a second association relationship between points and lines in the initial three-dimensional line segment map;
[0015] Calculating a third association relationship between each three-dimensional line segment in the initial three-dimensional line segment map and its associated three-dimensional plane;
[0016] Based on the first summation result, the second summation result, the first association relationship, the second association relationship and the third association relationship, a loss function is determined, and by minimizing the loss function, the initial three-dimensional line segment map is optimized to obtain a target three-dimensional line segment map.
[0017] According to a three-dimensional line segment reconstruction method provided by the present invention, the second association relationship includes: the first distance from the three-dimensional point to the associated three-dimensional line segment in the three-dimensional line segment map, the sine value of the angle between the direction corresponding to the vanishing point in the three-dimensional line segment map and the direction of the straight line where the associated three-dimensional line segment is located, and the orthogonal regularization term between any two vanishing points of the initial three-dimensional line segment map.
[0018] According to a three-dimensional line segment reconstruction method provided by the present invention, the third association relationship includes: a second distance from a first target point on a three-dimensional infinite straight line of a texture line segment in the initial three-dimensional line segment map to a three-dimensional plane associated with the texture line segment, an absolute value of the cosine of an angle between a direction of the texture line segment and a normal vector of the three-dimensional plane associated with the texture line segment, a third distance from a second target point on a three-dimensional infinite straight line of a structure line segment in the initial three-dimensional line segment map to a target three-dimensional infinite straight line, and a sine value of a direction angle between the three-dimensional infinite straight line of the structure line segment and the target three-dimensional infinite straight line;
[0019] The texture line segment is associated with one three-dimensional plane, the structure line segment is associated with two three-dimensional planes, and the target three-dimensional infinite straight line is the intersection of the two three-dimensional planes associated with the structure line segment.
[0020] According to a three-dimensional line segment reconstruction method provided by the present invention, the first target point is a projection point of the midpoint of the texture line segment on the three-dimensional infinite length straight line of the texture line segment;
[0021] The second target point is a projection point of the midpoint of the structural line segment on the three-dimensional infinite length straight line of the structural line segment.
[0022] According to a 3D line segment reconstruction method provided by the present invention, clustering all 2D line segments in the image set based on all optimal 3D line segment suggestions to reconstruct an initial 3D line segment map includes:
[0023] For any image in the image set, a similarity graph is constructed using valid two-dimensional line segments in the image as nodes, lines connecting the nodes and matching nodes in the neighboring images of the image as edges, and similarities between the best three-dimensional line segment proposals of the two nodes as edge weights; the valid two-dimensional line segments refer to two-dimensional line segments of the best three-dimensional line segment proposals that have at least one two-dimensional supporting line segment in the image, and the two-dimensional supporting line segments of the three-dimensional line segment proposals refer to two-dimensional line segments in the neighboring images of the image that correspond to the maximum non-zero similarity score and match the two-dimensional line segment;
[0024] Based on an agglomerative clustering algorithm with target constraints, clustering the nodes in the similarity graph to reconstruct the initial three-dimensional line segment map;
[0025] The target constraint is as follows: for two two-dimensional line segments in the same image, if the angle between the two two-dimensional line segments is higher than a first threshold, or the minimum value of the maximum two-dimensional endpoint perpendicular distance of the two two-dimensional line segments is higher than a second threshold, then the two two-dimensional line segments belong to different three-dimensional line segments in the initial three-dimensional line segment map; the maximum two-dimensional endpoint perpendicular distance refers to the maximum perpendicular distance from the endpoint of one of the two two-dimensional line segments to the two-dimensional infinite straight line where the other two-dimensional line segment is located.
[0026] According to a three-dimensional line segment reconstruction method provided by the present invention, generating a three-dimensional line segment suggestion of a two-dimensional line segment in any one of the images includes:
[0027] For any two-dimensional line segment in any one of the images, matching the any two-dimensional line segment with a two-dimensional line segment in a neighboring image of the any one of the images to obtain a matching pair of the any two-dimensional line segment;
[0028] Projecting each two-dimensional line segment in the matching pair into three-dimensional space to obtain two planes, and determining the association relationship between any two-dimensional line segment and the vanishing point;
[0029] generating a 3D line segment proposal for any 2D line segment based on the intersection of the two planes, the vanishing point associated with any 2D line segment, and the shared 3D points and the target non-shared 3D points of each 2D line segment in the matching pair;
[0030] The target non-shared three-dimensional point refers to a non-shared three-dimensional point whose reprojection error with another two-dimensional line segment in the matching pair is less than a preset number of pixels.
[0031] The present invention also provides a three-dimensional line segment reconstruction device, comprising:
[0032] An extraction and determination module is used to extract two-dimensional line segments in each image in the image set and determine the neighborhood image of each image;
[0033] a suggestion generation module, configured to generate, for any image in the image set, a three-dimensional line segment suggestion of a two-dimensional line segment in the any image;
[0034] a problem modeling module, configured to model the problem of selecting all best suggestions corresponding to all images in the image set as a 0-1 programming problem; wherein a consistency score function corresponding to any three-dimensional line segment suggestion of any two-dimensional line segment in any image in the 0-1 programming problem is determined based on the sum of the maximum similarity scores between any three-dimensional line segment suggestion and all neighboring images of the any image; wherein the maximum similarity score between any three-dimensional line segment suggestion and any neighboring image of the any image is the maximum value of the similarity scores between the any three-dimensional line segment suggestion and the best three-dimensional line segment suggestions of all two-dimensional line segments in the any neighboring images that match the any two-dimensional line segment;
[0035] a problem-solving module for solving the 0-1 programming problem to obtain all optimal three-dimensional line segment suggestions corresponding to the image set, and clustering all two-dimensional line segments in the image set based on all the optimal three-dimensional line segment suggestions to reconstruct an initial three-dimensional line segment map;
[0036] Joint optimization module for:
[0037] Acquire a three-dimensional point cloud, and identify a three-dimensional plane based on the three-dimensional point cloud; the three-dimensional point cloud includes sampling points in the initial three-dimensional line segment map and SfM points determined based on the image set using a structure-from-motion method;
[0038] Calculating a first summation result of reprojection errors of each 3D point in the 3D point cloud and a second summation result of reprojection errors of the 3D line segments in the initial 3D line segment map;
[0039] Calculating a first association relationship between the three-dimensional point and its associated three-dimensional plane;
[0040] Calculating a second association relationship between points and lines in the initial three-dimensional line segment map;
[0041] Calculating a third association relationship between each three-dimensional line segment in the initial three-dimensional line segment map and its associated three-dimensional plane;
[0042] Based on the first summation result, the second summation result, the first association relationship, the second association relationship and the third association relationship, a loss function is determined, and by minimizing the loss function, the initial three-dimensional line segment map is optimized to obtain a target three-dimensional line segment map.
[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-described three-dimensional line segment reconstruction methods when executing the computer program.
[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described three-dimensional line segment reconstruction methods.
[0045] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-described three-dimensional line segment reconstruction methods.
[0046] The 3D line segment reconstruction method, device, electronic device, and storage medium provided by the present invention model the problem of selecting the best suggestion corresponding to all images in an image set as a 0-1 programming problem. In this 0-1 programming problem, the consistency score function corresponding to any 3D line segment suggestion for any 2D line segment in any image is determined based on the sum of the maximum similarity scores between the 3D line segment suggestion and all neighboring images in the image. The maximum similarity score between the 3D line segment suggestion and a neighboring image in the image is the maximum similarity score between the 3D line segment suggestion and the best 3D line segment suggestion of all 2D line segments matching the 2D line segment in the neighboring images. The introduction of this consistency score function, which calculates the consistency score of the 3D line segment suggestion using the best 3D line segment suggestion of all 2D line segments, can enhance the support relationship between the 2D supporting line segments and the 3D line segment suggestion, thereby achieving accurate construction of the target 3D line segment map. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without any creative work.
[0048] Figure 1 1 is a flow chart of the three-dimensional line segment reconstruction method provided by the present invention;
[0049] Figure 2 is a schematic diagram of selecting the local best suggestion by using the local consistency criterion provided by the present invention;
[0050] Figure 3 Schematic diagram of the soft association relationship between a texture line segment and its associated three-dimensional plane in the three-dimensional line segment reconstruction method provided by the present invention;
[0051] Figure 4 It is a schematic diagram of the soft association relationship between a structural line segment and its associated three-dimensional plane in the three-dimensional line segment reconstruction method provided by the present invention;
[0052] Figure 5 is a schematic diagram of generating a three-dimensional line segment suggestion in the three-dimensional line segment reconstruction method provided by the present invention;
[0053] Figure 6 It is a 3D line segment map obtained on the Hypersim and Tanks and Temples datasets using the DeepLSD line detector;
[0054] Figure 7 The target 3D line segment map is obtained by using this method on the Hypersim and Tanks and Temples datasets using the LSD line detector;
[0055] Figure 8 It is a schematic diagram showing how the objective function value and the average support relationship consistency percentage change with the number of iterations in the process of iteratively solving the nonlinear 0-1 programming problem in the three-dimensional line segment reconstruction method provided by the present invention;
[0056] Figure 9 3D line segment reconstruction method provided by the present invention is a schematic diagram of the consistency analysis of the 3D line segment trajectory;
[0057] Figure 10 Schematic diagram of the structure of the three-dimensional line segment reconstruction device provided by the present invention;
[0058] Figure 11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0059] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0060] The terms "first" and "second" in the specification and claims of the present invention may explicitly or implicitly refer to one or more of the features. In the description of the invention, unless otherwise specified, "plurality" means two or more. Furthermore, the term "and / or" in the specification and claims refers to at least one of the connected items, and the character " / " generally indicates an "or" relationship between the connected items.
[0061] In the prior art, 3D line segment reconstruction methods can be categorized into non-matching, one-to-one matching, and one-to-many matching based on the requirements of 2D line segment matching in 3D reconstruction algorithms. For non-matching methods, 3D line segments are primarily fitted using point clouds. These methods primarily focus on point cloud processing to obtain more structural information, but have difficulty handling sparse point clouds. For one-to-one matching methods, 3D reconstruction algorithms only allow one-to-one matching (i.e., a 2D line segment in one image can only match one 2D line segment in another image), and 3D line segments are primarily triangulated using multi-view geometry. These methods primarily focus on 2D line segment matching and triangulation and are sensitive to matching outliers. For one-to-many matching methods, 3D reconstruction algorithms allow one-to-many matching (i.e., a 2D line segment in one image can match one or more 2D line segments in another image), and each matching pair generates one or more 3D line segment proposals (3D LS proposals) for the reference 2D line segment. These methods primarily address proposal generation, best proposal selection, and 3D LS track building. Compared to one-to-one matching, one-to-many matching can leverage more candidate matches and prior knowledge to generate richer 3D line segment proposals, potentially reconstructing more complete and accurate 3D line segments. However, due to the large number of outliers in the 3D line segment proposal set, one-to-many matching also suffers from more inconsistencies.
[0062] Because existing 3D line segment reconstruction methods based on one-to-many matching may cause mismatches between the support relationships between identified 2D supporting line segments and 3D line segment suggestions, accurate 3D line segment reconstruction is impossible. Therefore, an embodiment of the present invention provides a 3D line segment reconstruction method.
[0063] Figure 1 FIG. 1 is a flow chart of a three-dimensional line segment reconstruction method provided in an embodiment of the present invention. Figure 1 As shown, the method includes:
[0064] S1, extracting the two-dimensional line segments in each image in the image set and determining the neighborhood images of each image;
[0065] S2, for any image in the image set, generating a three-dimensional line segment suggestion for the two-dimensional line segment in the any image;
[0066] S3, modeling the problem of selecting all the best three-dimensional line segment proposals corresponding to all the two-dimensional line segments in the image set as a 0-1 programming problem; in the 0-1 programming problem, the consistency score function corresponding to any three-dimensional line segment proposal of any two-dimensional line segment in any image is determined based on the sum of the maximum similarity scores between the any three-dimensional line segment proposal and all neighboring images of the any image; the maximum similarity score between the any three-dimensional line segment proposal and any neighboring image of the any image is: the maximum value of the similarity scores between the any three-dimensional line segment proposal and the best three-dimensional line segment proposals of all two-dimensional line segments in the any neighboring images that match the any two-dimensional line segment;
[0067] S4, solving the 0-1 programming problem to obtain all optimal 3D line segment suggestions corresponding to the image set, and clustering all 2D line segments in the image set based on all the optimal 3D line segment suggestions to reconstruct an initial 3D line segment map;
[0068] S5, using the initial three-dimensional line segment map, jointly optimize points, lines, surfaces, and hidden points to obtain the target three-dimensional line segment map.
[0069] Specifically, the three-dimensional line segment reconstruction method provided in the embodiment of the present invention is executed by a three-dimensional line segment reconstruction device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.
[0070] First, step S1 is executed to extract two-dimensional line segments within each image in the image set and determine the neighboring images of each image. The image set includes multiple images, each of which carries prior information. This prior information may include three-dimensional point clouds, depth maps, camera pose information, intrinsic parameter matrices, and world coordinates of the center.
[0071] The two-dimensional line segments in each image can be extracted using conventional line extraction operators.
[0072] The neighborhood images of each image can be determined by the prior information carried by each image. The number of neighborhood images of each image is n v It can be multiple.
[0073] Then, step S2 (the proposal generation step) is executed. The image set is traversed. For each image in the image set, 3D line segment proposals are generated for each 2D line segment in the image. Here, each 2D line segment corresponds to a corresponding 3D line segment proposal set, which includes one or more 3D line segment proposals. The 3D endpoints of each 3D line segment proposal lie on the back-projected ray of the 2D endpoint of the corresponding 2D line segment.
[0074] Then, step S3 is executed to model the best suggestion selection problem corresponding to all images in the image set as a 0-1 programming problem. The 0-1 programming problem can be expressed by the following formula (1):
[0075]
[0076] in, represents an image set, represents all two-dimensional line segments in image i, Represents the j-th two-dimensional line segment l in image i ij All three-dimensional line segment suggestions, Indicates l ij The k-th 3D line segment suggestion, represents all 0-1 decision variables, for The predicted value of for A 0-1 decision variable, representing Is it the best 3D line segment suggestion? express is the best 3D line segment suggestion. c (·) represents the consistency score function, Suggested for 3D line segments Corresponding consistency score function, 3D line segment suggestion The higher the consistency, The bigger it is.
[0077] The goal of the 0-1 programming problem expressed in formula (1) is to maximize the sum of the consistency scores of the best 3D line segment proposals.
[0078] The consistency score functions used by traditional methods such as L3D++ and LIMAP have the same form, which is formula (2):
[0079]
[0080] in, Indicates the neighborhood image of image i except for the generated All neighboring images except the neighboring image of , is the image p and l ij Matched 2D line segments, Is to use matching pairs (l ij ,l pq ) generated by ij , s(·,·)∈{0}∪[0.5,1] is a similarity score function used to measure the comprehensive similarity of two 2D line segments, which takes into account factors such as collinearity and reprojection error of the two 2D line segments. for and similarity score.
[0081] For a 3D line segment suggestion L, the consistency score s corresponding to L c The 2D line segments that contribute to (L) are defined as the 2D support segments of L. Intuitively, these 2D support segments are assumed to be the 2D observations of L. The 2D support segments suggested by the 3D line segments defined by formula (2) are the 2D support segments that correspond to the non-zero maximum similarity score in each neighborhood image and are consistent with l. ij Matched 2D line segments.
[0082] Traditional methods such as L3D++ and LIMAP obtain the best 3D line segment proposals by searching for the maximum consistency score of all 3D line segment proposals for each 2D line segment. Since formula (2) can be understood as a local consistency criterion, that is, 3D line segment proposals generated by matching inliers of the same 2D line segment should be very close in 3D space, the best 3D line segment proposal selected based on this local consistency criterion is called the local best proposal.
[0083] However, there is an inconsistency problem in the support relationship between the local best proposal selected by the local consistency criterion and the two-dimensional support line segments. Figure 2 A simple example is shown in . Assume that a 3D line segment suggestion L1 belongs to a 2D line segment l r , and by matching pairs (l r , ). Assume that another 3D line segment proposal L2 is generated by matching (l r , ) is generated, and L2 contributes to the consistency score s of L1 c (L1), so by definition, the generated L2 is a two-dimensional support segment of L1. Locally optimal recommendations back, and L1 may not be collinear. In this case, L1 and The support relationship between them is inconsistent. Should not contribute to the consistency score c (L1).
[0084] The reason for the above inconsistency problem is that the consistency between the 3D line segment proposal and the best 3D line segment proposal of other 2D line segments is not considered when calculating the consistency score of the 3D line segment proposal. This is also the reason why the local consistency criterion is called "local".
[0085] In order to solve the problem of inconsistency in support relationships in the local consistency criterion, the embodiment of the present invention uses a global consistency criterion to calculate the consistency score, that is, the best three-dimensional line segment proposals corresponding to the same three-dimensional straight line should be very close in three-dimensional space. The best three-dimensional line segment proposal selected according to the global consistency criterion is called the global best proposal. The global consistency criterion is called "global" because the consistency score of the three-dimensional line segment proposal is calculated using the best three-dimensional line segment proposals of other two-dimensional line segments. In other words, global information is used to evaluate the consistency of the three-dimensional line segment proposal.
[0086] Since the best 3D line segment suggestions for other 2D line segments cannot be known in advance, in the embodiment of the present invention, the best 3D line segment suggestions for other 2D line segments are encoded into 0-1 decision variables. Based on this, the consistency score function used in formula (1) in the embodiment of the present invention is:
[0087]
[0088] in, represents the neighborhood image of image i, represents the qth two-dimensional line segment l in image p pq All three-dimensional line segment suggestions, Indicates l pq The r-th three-dimensional line segment suggestion, for A 0-1 decision variable, representing Whether it is the best 3D line segment suggestion. Represents a 3D line segment suggestion Maximize the similarity score corresponding to image p, Suggested for 3D line segments With image p in l ij Similarity scores between 3D line segment proposals of all matched 2D line segments.
[0089] From formula (3), we can see that each image can only provide one consistency score, and the two-dimensional support line segment suggested by the three-dimensional line segment is also the non-zero maximum value corresponding to the similarity score in each neighborhood image and is consistent with l ij Matched 2D line segments. Note that a similarity score greater than 0 indicates that the two 3D line segment proposals are approximately collinear. Formally, it can be seen that the 2D supporting line segment defined by formula (3) and the supported 3D line segment proposals are approximately collinear. Therefore, using the consistency score function of formula (3) can enhance the consistency of the support relationship.
[0090] Then, step S4 is executed to solve the 0-1 programming problem. That is, formula (1) is solved in combination with formula (3) to obtain all the best 3D line segment suggestions corresponding to the image set. Furthermore, based on all the best 3D line segment suggestions, the 2D line segments within each image in the image set can be clustered to reconstruct an initial 3D line segment map.
[0091] Formula (1) and Formula (3) together constitute a nonlinear 0-1 programming problem, which can be approximately solved by iteratively solving a linear 0-1 programming problem. The linear 0-1 programming problem can be defined as Formula (1) equipped with the following consistency score function (4):
[0092]
[0093] in, is the global best proposal obtained in the previous iteration. In practice, we first select the local best proposal as the initial global best proposal, and then update the global best proposal by repeatedly solving Equation (1) and Equation (4). Each 2D segment can select a new global best proposal in each iteration, but only those global best proposals with at least a specified number of 2D support segments can be seen in the next iteration. The specified number can be expressed as τ n .
[0094] If the rate of change of the objective function value of the nonlinear 0-1 programming problem is lower than the preset threshold τ c , the iteration stops, and the final global best proposal is the best 3D line segment proposal for each 2D line segment. During the 3D line segment construction process, the 2D line segments to which the best 3D line segment proposal has no 2D supporting line segments are ignored.
[0095] When reconstructing the initial 3D segment map, due to the lack of coplanar constraints, the topological relationships between 3D points, 3D segments, 3D planes, and vanishing points may be inconsistent. To this end, embodiments of the present invention introduce the concept of 3D planes and various soft associations to optimize the initial 3D segment map. These soft associations take into account the topological connectivity between different geometric entities. After determining the initial 3D segment map, step S5 is executed, which jointly optimizes points, lines, planes, and vanishing points to obtain the target 3D segment map.
[0096] Step S5 may include the following process:
[0097] Acquire a three-dimensional point cloud, and identify a three-dimensional plane based on the three-dimensional point cloud; the three-dimensional point cloud includes sampling points in the initial three-dimensional line segment map and SfM points determined based on the image set using a structure-from-motion method;
[0098] Calculating a first summation result of reprojection errors of each 3D point in the 3D point cloud and a second summation result of reprojection errors of the 3D line segments in the initial 3D line segment map;
[0099] Calculating a first association relationship between the three-dimensional point and its associated three-dimensional plane;
[0100] Calculating a second association relationship between points and lines in the initial three-dimensional line segment map;
[0101] Calculating a third association relationship between each three-dimensional line segment in the initial three-dimensional line segment map and its associated three-dimensional plane;
[0102] Based on the first summation result, the second summation result, the first association relationship, the second association relationship and the third association relationship, a loss function is determined, and by minimizing the loss function, the initial three-dimensional line segment map is optimized to obtain a target three-dimensional line segment map.
[0103] The 3D point cloud may include sampling points in the initial 3D line segment map and SfM points determined using the structure-from-motion method from the image set. The sampling points are 3D points on each 3D line segment in the initial 3D line segment map.
[0104] Through the 3D point cloud, a plane detection algorithm can be used to identify the 3D plane. Then, the initial soft association relationship between point and plane, line and plane can be constructed through the internal points of the 3D plane.
[0105] First, the first summation result E of the reprojection error of each 3D point can be calculated P And the second summation result E of the reprojection error of each 3D line segment in the initial 3D line segment map L .
[0106] Calculate the first correlation E PΠ The first association relationship may be an association relationship between each 3D point and its associated 3D plane. Here, the 3D plane associated with each 3D point refers to the 3D plane where the 3D point theoretically lies, but in practice, due to errors or noise, the 3D point is not on the 3D plane. The first association relationship may be a distance relationship.
[0107] Calculate the second association E PLThe second association relationship may be the association relationship between each 3D point in the initial 3D line segment map and the 3D line segment in which it resides, the association relationship between the straight line corresponding to each vanishing point in the initial 3D line segment map and the straight line in which the corresponding 3D line segment resides, and the association relationship between each vanishing point. The association relationship between each 3D point and the 3D line segment in which it resides may be a distance relationship, the association relationship between the straight line corresponding to each vanishing point in the initial 3D line segment map and the straight line in which the corresponding 3D line segment resides may be a direction angle relationship, and the association relationship between each vanishing point may be an orthogonal relationship between the directions corresponding to each vanishing point.
[0108] Calculate the third correlation E LΠ The third association relationship may be an association relationship between each three-dimensional line segment in the initial three-dimensional line segment map and its associated three-dimensional plane, which may include a distance relationship and a direction angle relationship.
[0109] Using the first summation result E P , the second summation result E L , the first association relationship E PΠ , the second association relationship E PL And the third relationship E LΠ , determine the loss function. The loss function E can be:
[0110] E=E P +E L +E PL +E PΠ +E LΠ (5)
[0111] By minimizing the loss function E, the initial 3D segment map can be optimized to ultimately obtain a target 3D segment map. This target 3D segment map is more accurate and reliable than the initial 3D segment map before optimization.
[0112] The 3D line segment reconstruction method provided in an embodiment of the present invention models the problem of selecting the best proposal corresponding to all images in an image set as a 0-1 programming problem. The maximum similarity score between the 3D line segment proposal and all neighboring images in the image is determined by the sum of the maximum similarity scores between the 3D line segment proposal and a particular neighboring image in the image. The maximum similarity score between the 3D line segment proposal and the best 3D line segment proposals for all 2D line segments in the neighboring images that match the 2D line segment is the maximum value. The introduction of this consistency score function, which calculates the consistency score of the 3D line segment proposal using the best 3D line segment proposals for all 2D line segments, can enhance the support relationship between 2D supporting segments and 3D line segment proposals, thereby accurately constructing the target 3D line segment map.
[0113] Based on the above embodiment, the first association relationship may be the distance between each 3D point and its associated 3D plane. This distance may be a 3D Euclidean distance. Thus, minimizing this distance ensures that each 3D point is on, or approximately on, its associated 3D plane.
[0114] Based on the above embodiment, the second association relationship includes: a first distance from each 3D point in the 3D point cloud to the associated 3D line segment in the initial 3D line segment map; the sine value of the angle between the direction corresponding to the vanishing point in the initial 3D line segment map and the direction of the line containing the associated 3D line segment; and an orthogonal regularization term between any two vanishing points in the initial 3D line segment map. It is understood that the first distance can be a 3D Euclidean distance. The orthogonal regularization term refers to the orthogonal regularization term between the directions of the two vanishing points.
[0115] Based on the above embodiment, the third association relationship includes: a second distance from a first target point on the three-dimensional infinite straight line of the texture segment in the initial three-dimensional segment map to a three-dimensional plane associated with the texture segment, an absolute value of the cosine of an angle between the direction of the texture segment and a normal vector of the three-dimensional plane associated with the texture segment, a third distance from a second target point on the three-dimensional infinite straight line of the structure segment in the initial three-dimensional segment map to a target three-dimensional infinite straight line, and a sine value of a direction angle between the three-dimensional infinite straight line of the structure segment and the target three-dimensional infinite straight line;
[0116] The texture line segment is associated with one three-dimensional plane, the structure line segment is associated with two three-dimensional planes, and the target three-dimensional infinite straight line is the intersection of the two three-dimensional planes associated with the structure line segment.
[0117] Specifically, in this embodiment of the present invention, a texture segment is a 3D segment with one associated 3D plane, and a structure segment is a 3D segment with two associated 3D planes. Therefore, each 3D segment in the initial 3D segment map has at most two associated 3D planes.
[0118] Based on this, Figure 3 As shown, the third association relationship may include: a soft association relationship between the texture line segment L1 and its associated three-dimensional plane, that is, a three-dimensional infinite straight line L of the texture line segment L1 in the initial three-dimensional line segment map. inf1 The second distance d1 from the first target point to the three-dimensional plane associated with the texture segment L1 and the absolute value of the cosine of the angle between the direction of the texture segment L1 and the normal vector of the three-dimensional plane associated with the texture segment L1 can be a three-dimensional Euclidean distance.
[0119] like Figure 4As shown, the third association relationship may also include: a soft association relationship between the structure line segment L2 and its associated three-dimensional plane, that is, the three-dimensional infinite straight line L of the structure line segment L2 in the initial three-dimensional line segment map inf2 The third infinite straight line l from the second target point to the target inf3 The third distance d2 and the three-dimensional infinite straight line L of the structural line segment L2 inf2 and the target three-dimensional infinite straight line L inf3 The third distance may be a three-dimensional Euclidean distance.
[0120] The first target point can be the three-dimensional infinite straight line L of the texture segment L1. inf1 Any point on the second target point can be the three-dimensional infinite straight line L of the structural line segment L2 inf2 In addition, the first target point can also be the three-dimensional infinite straight line L of the texture segment L1. inf1 The point closest to the origin of the coordinate system, the second target point can also be the three-dimensional infinite straight line L of the structural line segment L2 inf2 The point on the coordinate system that is closest to the origin.
[0121] Based on the above embodiment, the first target point is a projection point of the midpoint of the texture segment on the three-dimensional infinite straight line of the texture segment;
[0122] The second target point is a projection point of the midpoint of the structural line segment on the three-dimensional infinite length straight line of the structural line segment.
[0123] Specifically, using conventional methods to determine the first target point and the second target point may cause them to deviate significantly from the area where the three-dimensional line segment is located. Figure 3 As shown, the midpoint o1 of the texture segment L1 can be selected to be on the three-dimensional infinite straight line L of the texture segment L1. inf1 The projection point on is taken as the first target point.
[0124] Likewise, if Figure 4 As shown, the midpoint o2 of the structure segment L2 can be selected to be the three-dimensional infinite straight line L inf2 The projection point on is taken as the second target point.
[0125] Building on the above examples, to achieve scale invariance, the distances involved in formula (5) can be divided by a scale factor σ = d / f, where d is the median depth of the 3D point across all images, and f is the focal length of the camera (in pixels). Essentially, σ represents the 3D error of a point when it reaches a 1-pixel error on the image. Scale-invariant optimization is important because it prevents changes in scene scale from affecting the weights of the terms in formula (5).
[0126] In particular, the process of clustering all two-dimensional line segments in the image set in step S4 is as follows:
[0127] For any image in the image set, a similarity graph is constructed using valid two-dimensional line segments in the image as nodes, lines connecting the nodes and matching nodes in the neighboring images of the image as edges, and similarities between the best three-dimensional line segment proposals of the two nodes as edge weights; the valid two-dimensional line segments refer to two-dimensional line segments of the best three-dimensional line segment proposals that have at least one two-dimensional supporting line segment in the image, and the two-dimensional supporting line segments of the three-dimensional line segment proposals refer to two-dimensional line segments in the neighboring images of the image that correspond to the maximum non-zero similarity score and match the two-dimensional line segment;
[0128] Based on an agglomerative clustering algorithm with target constraints, clustering the nodes in the similarity graph to reconstruct the initial three-dimensional line segment map;
[0129] The target constraint is as follows: for two two-dimensional line segments in the same image, if the angle between the two two-dimensional line segments is higher than a first threshold, or the minimum value of the maximum two-dimensional endpoint perpendicular distance of the two two-dimensional line segments is higher than a second threshold, then the two two-dimensional line segments belong to different three-dimensional line segments in the initial three-dimensional line segment map; the maximum two-dimensional endpoint perpendicular distance refers to the maximum perpendicular distance from the endpoint of one of the two two-dimensional line segments to the two-dimensional infinite straight line where the other two-dimensional line segment is located.
[0130] Specifically, when reconstructing the initial 3D line segment map, the image set is traversed. For any image i in the image set, a similarity graph is constructed, with valid 2D line segments within image i as nodes, the lines connecting these nodes with matching nodes in the neighboring images of image i as edges, and the similarity between the best 3D line segment proposals of the two nodes as the edge weight. A valid 2D line segment is a best 3D line segment proposal with at least one 2D supporting line segment, and the 2D supporting line segments of any 3D line segment proposal are the 2D line segments in each neighboring image of image i that have the maximum similarity score and match the 2D line segment.
[0131] In order to enhance the consistency of 3D line segments, it is necessary to avoid two non-collinear 2D line segments in the same image being assigned to the same 3D line segment trajectory. To this end, a cannot-link constraint, namely the target constraint, is imposed during clustering. i ,x j )∈C ≠ means that the node x in the node set Y of all nodes in the similar graph is i and node x jcannot be assigned to the same category, C ≠ Represents the set of all unconnectable nodes.
[0132] Let l1 and l2 be two 2D line segments, and their minimum-maximum 2D endpoint perpendicular distance, i.e., the minimum of the maximum 2D endpoint perpendicular distance, is defined as:
[0133] d mp (l1,l2)=min{d p (l1,l2),d p (l2,l1)} (6)
[0134] Among them, d p (l1,l2) is the maximum vertical distance from the endpoint of l1 to the two-dimensional infinite line where l2 is located, d p (l2, l1) is the maximum perpendicular distance from the endpoint of l2 to the two-dimensional infinite line where l1 is located.
[0135] Compared with the maximum of the maximum 2D endpoint perpendicular distance, the minimum-maximum 2D endpoint perpendicular distance is more robust when the lengths of l1 and l2 differ greatly.
[0136] In particular, the definition of the minimum-maximum 3D endpoint perpendicular distance is just a direct generalization of the 2D form. In practice, if l1 and l2 come from the same image, but their angle is above the first threshold τ a Or the minimum-maximum two-dimensional endpoint vertical distance is higher than the second threshold τ p , then add l1 and l2 to C ≠ middle.
[0137] Given a similarity graph and C ≠ , an agglomerative constrained clustering algorithm based on target constraints can be executed to cluster the nodes in the similar graph and reconstruct the initial three-dimensional line segment map.
[0138] First, build an edge list in descending order of the edge weights in the similar graphs, and then merge the edges that are not in C in sequence starting from the edge list. ≠ The two edges of the node.
[0139] Finally, the two-dimensional line segments of the same type form a three-dimensional line segment trajectory.
[0140] For each 3D line segment trajectory, we apply principal component analysis (PCA) to all directions of the best 3D line segment proposal and use the principal eigenvectors of the best 3D line segment proposal and the average position of the 3D endpoints to fit a 3D infinite line. We project all 3D endpoints onto the 3D infinite line to obtain new robust 3D endpoints.
[0141] Specifically, the steps for generating 3D line segment proposals in step S2 are as follows:
[0142] For any two-dimensional line segment in any one of the images, matching the any two-dimensional line segment with a two-dimensional line segment in a neighboring image of the any one of the images to obtain a matching pair of the any two-dimensional line segment;
[0143] Projecting each two-dimensional line segment in the matching pair into three-dimensional space to obtain two planes, and determining the association relationship between any two-dimensional line segment and the vanishing point;
[0144] generating a 3D line segment proposal for any 2D line segment based on the intersection of the two planes, the vanishing point associated with any 2D line segment, and the shared 3D points and the target non-shared 3D points of each 2D line segment in the matching pair;
[0145] The target non-shared three-dimensional point refers to a non-shared three-dimensional point whose reprojection error with another two-dimensional line segment in the matching pair is less than a preset number of pixels.
[0146] Specifically, the 3D line segment proposal for each 2D line segment in each image is generated using the same steps, that is, when generating any 2D line segment l in any image r, r When the three-dimensional line segment is suggested, the two-dimensional line segment l r As a reference two-dimensional line segment, the two-dimensional line segment l r Match all two-dimensional line segments in all neighborhood images of image r to obtain two-dimensional line segments l r Among them, the two-dimensional line segment l r In each neighborhood image of image r, K two-dimensional line segments of the previous target number can be obtained. For example, Figure 5 As shown, l m Yes r A matching two-dimensional line segment, l r With l m form a matching pair.
[0147] The traditional 3D line segment reconstruction algorithm (LIMAP) can use the following four methods to generate the 3D infinite length line L corresponding to the 3D line segment suggestion L: inf :1) Face-to-face intersection method (Line+Line): Intersection l r and l m The back-projection plane is L inf ; 2) Multiple Points: Use l r and l m Fit the shared 3D points of the 3D infinite line and then project it onto l r L is obtained on the back projection plane inf , where the shared 3D point is lr and l m 3) Line+Point method: add soft constraints to make the L obtained by the “surface intersection method” inf Passing through a shared 3D point, so each shared 3D point can generate an L inf 4) Vanishing point method (Line+VP): First detect the vanishing point, then add soft constraints to make the L obtained by the "surface intersection method" inf The direction of and its associated vanishing point have the same 3D direction.
[0148] Generated L inf Must be in l r On the back-projection plane, l r The three-dimensional line segment suggestion L has two three-dimensional endpoints defined as l inf and l r The intersection of the back-projected rays of two 3D endpoints.
[0149] However, due to the limitations of keypoint detection and matching capabilities, some keypoints with the same name are not simultaneously r and l m The surroundings are not detected, or even if they are all detected, they are not matched, so the shared 3D points cannot be triangulated through SfM.
[0150] To solve this problem, unlike the LIMAP algorithm, in addition to using shared 3D points, the target non-shared 3D points are also used to perform the above-mentioned "multi-point fitting method" and "one-point method". Among them, the target non-shared 3D points are m The reprojection error is less than the preset amount τ k Unshared 3D points of pixels.
[0151] Figure 5 In, l r Point 1 and l on m Point 4 on the match, l r Point 2 on l m Point 3 on the L matches, and points 2 and 3 are on L inf The corresponding 3D SfM point is the shared 3D point. Point 1 and point 4 are on L inf The above corresponds to two different 3D SfM points, both of which are non-shared 3D points.
[0152] Here, both shared 3D points and non-shared 3D points can be used to generate 3D line segment proposals, which is conducive to reconstructing a more complete target 3D line segment map.
[0153] Finally, based on the above embodiments, the three-dimensional line segment reconstruction method provided in the embodiments of the present invention is implemented based on the line mapping and positioning library LIMAP, and its effect has been verified.
[0154] Use the number of neighborhood images n v = 20, the number of the top targets of the matching results K = 10 to build matching pairs for line segment matching, the preset number τ k = 1 pixel to generate 3D line segment suggestions, specify the number τ n =2, preset threshold τ c = 0.001 is used to select the best 3D line segment proposal, the first threshold τ a =2 degrees, the second threshold τ p = 2 pixels are used to construct the unconnectable constraint. Quantitative and qualitative experiments were conducted using Hypersim (synthetic image set) and Tanks and Temples (real image set). The 3D line segment reconstruction method provided in the present invention (abbreviated as Ours) was compared with L3D++ and LIMAP, using a traditional LSD line detector and a learning-based DeepLSD line detector, respectively.
[0155] (1) Evaluation on 3D LS Reconstruction
[0156] Since there is no ground truth (GT) 3D line segment, the GT point cloud is used to evaluate the target 3D line segment map. According to the LIMAP evaluation process, the following indicators are used for evaluation:
[0157] ① Length recall within τmm: the sum of the lengths of three-dimensional line segments within τmm.
[0158] ②Inlier percentage: The percentage of three-dimensional line segments whose error is within vmm.
[0159] ③Average supports: the average number of images and 2D line segments on all lines.
[0160] Table 1 reports the average reconstruction metrics for the first eight scenes in the Hypersim dataset, each with 100 images. Compared to the other two methods, this method achieves higher recall and longer 3D line segments, but its precision is slightly lower than that of L3D++. Because inconsistent elements in 3D line segment trajectories are assigned to different consistent trajectories, the number of 2D line segments in a 3D line segment is sometimes less than that of LIMAP.
[0161] Table 1 3D line segment reconstruction results on the Hypersim dataset
[0162]
[0163] Table 2 reports the average reconstruction metrics on the Tanks and Temples training set (excluding the Ignatius scene as it contains almost no rectilinear structures). Since the provided ground truth point cloud only contains the primary object, we computed an axis-aligned bounding box as recommended by LIMAP, expanded it by 0.1 meters, and evaluated only the 3D line segments within this area. As shown in Table 2, our method significantly outperforms other methods in terms of completeness and accuracy, while also achieving longer 3D line segments.
[0164] Table 2 3D line segment reconstruction results on the Tanks and Temples training set
[0165]
[0166] Figure 6 Shows 3D line segment maps obtained on the Hypersim and Tanks and Temples datasets using the DeepLSD line detector. Figure 6 The first line is the original image, the second line is the 3D line segment map obtained by L3D++, the third line is the 3D line segment map obtained by LIMAP, and the fourth line is the target 3D line segment map obtained by this method.
[0167] By comparison, this method can not only reconstruct fine structures, but also reconstruct a more complete 3D line segment map of the target as a whole.
[0168] Figure 7 The 3D line segment maps of the target obtained by using this method on the Hypersim and Tanks and Temples datasets are shown using the LSD line detector.
[0169] (2) Evaluation on Best Proposal Selection
[0170] To compare the best proposal selection methods based on local consistency and global consistency, we evaluate the best proposal map consisting of the best 3D line segment proposals of all 2D line segments. Specifically, we evaluate the initial best proposal map (i.e., obtained by directly solving Equation (1)) and the filtered best proposal map (i.e., each best 3D line segment proposal must have at least one 2D supporting line segment).
[0171] Table 3 Average metrics on the Hypesim dataset
[0172]
[0173] As shown in Table 3, the filtering operation can significantly improve the precision while sacrificing a small amount of recall. Compared with the local consistency-based method, our method achieves higher recall and precision on both the original and filtered best proposal maps, proving that our method can select more accurate best proposals.
[0174] In addition, to clarify the reasons behind the success of this method, we will analyze the consistency of support relations. Given the best 3D line segment proposal of all 2D supporting line segments of a 3D line segment proposal, the average consistency of all support relations of the 3D line segment proposal can be quantitatively evaluated using the following consistency percentage:
[0175] ① Angle consistency percentage: The percentage of 2D support segments whose angle difference between the best suggestion of the 2D support segment and the supported 3D segment suggestion is within τ degrees.
[0176] ② Distance consistency percentage within τmm: The percentage of 2D supporting line segments whose optimal suggestion for a 2D supporting line segment and the minimum-maximum 3D endpoint perpendicular distance of the supported 3D line segment suggestion are within τmm.
[0177] Experiments were conducted on the Hypersim dataset. Table 3 reports the average consistency percentage, total number of filtered best proposals, and the total number of all 2D support segments. Compared to local consistency-based methods, this method achieves more 2D support segments and a higher consistency percentage despite a reduction in the number of filtered best 3D segment proposals. This demonstrates that this method not only avoids the introduction of inconsistent 2D support segments and the resulting erroneous best proposals, but also finds more consistent 2D support segments.
[0178] Figure 8 The paper shows how the objective function value (Objective value) and the average support relationship consistency percentage (Percentage) change with the number of iterations in the iterative solution of the nonlinear 0-1 programming problem. It can be seen that the iterative algorithm proposed in this paper can approximately solve the nonlinear 0-1 programming problem and can gradually enhance the consistency of the support relationship while having a relatively fast convergence speed.
[0179] (3) Ablation Study on Consistency Enhancing
[0180] Table 4. Related results of ablation experiments
[0181]
[0182] Ablation experiments were conducted on the Hypersim dataset for the steps of 3D line segment proposal generation (G), best proposal selection (S), and 3D line segment construction (B). The relevant results are recorded in Table 4. The checked ones indicate that this method is used, otherwise the LIMAP method is used. As shown in Table 4, each step and its combination improves the recall rate. In particular, the 3D line segment proposal generation step improves the recall rate and precision rate; the best proposal selection step significantly increases the length of the 3D line segment, while improving the recall rate and precision rate; the 3D line segment construction step significantly improves the recall rate while sacrificing some accuracy. Since this method assigns inconsistent elements in the 3D line segment to different 3D line segments, it will reduce the number of 2D line segments on the trajectory.
[0183] Figure 9 A schematic diagram of the consistency analysis of three-dimensional line segment trajectories is shown. Figure 9 In parts (a) and (c), using the LIMAP method, two non-collinear 2D line segments are assigned to the same 3D line segment (the 2D line segment projected by the 3D line segment is the long line segment in part (c)). Figure 9 In parts (b) and (d) of the figure, using this method allows two non-collinear 2D line segments to be assigned to different 3D line segments, thereby improving the consistency of the 3D line segments. Ultimately, using both methods simultaneously achieves the best performance balance.
[0184] (4) Evaluation on Joint Optimization
[0185] Comparing the second distance of the texture line in this method with the existing line-surface distance (called closet-point), the main difference lies in how to determine the first target point on the three-dimensional infinite line during optimization. Here, only E is used. L +E LΠ Perform joint optimization and disable scale invariance. Table 5 records the comparison results. This method achieves better performance, proving that the second distance calculated in this method is more reasonable.
[0186] Table 5 Comparison results of the second distance of the texture line in this method and the existing line-surface distance
[0187]
[0188] Table 6. Ablation experiment results of joint optimization
[0189]
[0190] Table 6 records the ablation experiment of joint optimization. Since joint optimization does not affect the average support of 3D line segments, they are not shown in the table. For simplicity, only the results of some important combination items are shown. The ones with a check mark indicate that the item is used for optimization, otherwise it is not used. All methods involving optimization of 3D points take into account the reprojection error of 3D points. As shown in Table 6, only E L The optimization reduces the recall rate and precision rate. PL , can reduce the use of only E L The adverse effects of optimization are introduced by E PΠ and E LΠ , can greatly improve the recall rate and precision rate. Especially E L +E LΠ The optimization achieved the best precision on P1 and P10 and significantly improved the recall, which shows that the coplanarity constraint is crucial for 3D line segment reconstruction. Finally, the combination of all data items achieved the best recall and second-best precision among all results.
[0191] In summary, a three-dimensional line segment reconstruction method based on global consistency is provided in an embodiment of the present invention. The main ideas include: 1) generating multiple candidate three-dimensional line segment proposals for each two-dimensional line segment, and using a method based on global consistency to determine the best three-dimensional line segment proposal, thereby enhancing the consistency of the two-dimensional supporting line segment and the supported three-dimensional line segment proposal. 2) By imposing a cannot-link constraint, two non-collinear two-dimensional line segments in the same image are avoided from being assigned to the observation trajectory of the same three-dimensional line segment, thereby improving the consistency of the three-dimensional line segments. 3) By introducing three-dimensional planes and various association relationships, three-dimensional points, three-dimensional line segments, three-dimensional planes and vanishing points are jointly optimized, thereby enhancing the consistency of the topological relationships between geometric entities. Experimental results show that enhancing consistency can improve the integrity and accuracy of three-dimensional line segment reconstruction, while also obtaining longer and more robust three-dimensional line segments.
[0192] like Figure 10 As shown, based on the above embodiment, an embodiment of the present invention provides a three-dimensional line segment reconstruction device, including:
[0193] An extraction and determination module 101 is used to extract two-dimensional line segments in each image in the image set and determine the neighborhood image of each image;
[0194] A suggestion generating module 102 is configured to generate, for any image in the image set, a three-dimensional line segment suggestion for a two-dimensional line segment in the image;
[0195] A problem modeling module 103 is configured to model the problem of selecting all the best three-dimensional line segment proposals corresponding to all the two-dimensional line segments in the image set as a 0-1 programming problem; in the 0-1 programming problem, a consistency score function corresponding to any three-dimensional line segment proposal for any two-dimensional line segment in any image is determined based on the sum of the maximum similarity scores between any three-dimensional line segment proposal and all neighboring images of the any image; the maximum similarity score between any three-dimensional line segment proposal and any neighboring image of the any image is the maximum value of the similarity scores between any three-dimensional line segment proposal and the best three-dimensional line segment proposals of all two-dimensional line segments in any neighboring image that match the any two-dimensional line segment;
[0196] a problem-solving module 104 for solving the 0-1 programming problem to obtain all optimal 3D line segment suggestions corresponding to the image set, and clustering all 2D line segments in the image set based on all the optimal 3D line segment suggestions to reconstruct an initial 3D line segment map;
[0197] The joint optimization module 105 is used to:
[0198] Acquire a three-dimensional point cloud, and identify a three-dimensional plane based on the three-dimensional point cloud; the three-dimensional point cloud includes sampling points in the initial three-dimensional line segment map and key points determined based on the image set using a structure-from-motion method;
[0199] Calculating a first summation result of reprojection errors of each 3D point in the 3D point cloud and a second summation result of reprojection errors of the 3D line segments in the initial 3D line segment map;
[0200] Calculating a first association relationship between the three-dimensional point and its associated three-dimensional plane;
[0201] Calculating a second association relationship between points and lines in the initial three-dimensional line segment map;
[0202] Calculating a third association relationship between each three-dimensional line segment in the initial three-dimensional line segment map and its associated three-dimensional plane;
[0203] Based on the first summation result, the second summation result, the first association relationship, the second association relationship and the third association relationship, a loss function is determined, and by minimizing the loss function, the initial three-dimensional line segment map is optimized to obtain a target three-dimensional line segment map.
[0204] Based on the above embodiments, in the three-dimensional line segment reconstruction device provided in the embodiments of the present invention, the second association relationship includes: the first distance from the three-dimensional point to the associated three-dimensional line segment in the initial three-dimensional line segment map, the sine value between the direction corresponding to the vanishing point in the initial three-dimensional line segment map and the direction of the straight line where the associated three-dimensional line segment is located, and the orthogonal regularization term between any two vanishing points of the initial three-dimensional line segment map.
[0205] On the basis of the above embodiment, in the three-dimensional line segment reconstruction device provided in the embodiment of the present invention, the third association relationship includes: a second distance from a first target point on the three-dimensional infinite straight line of the texture line segment in the initial three-dimensional line segment map to a three-dimensional plane associated with the texture line segment, an absolute value of the cosine of an angle between the direction of the texture line segment and a normal vector of the three-dimensional plane associated with the texture line segment, a third distance from a second target point on the three-dimensional infinite straight line of the structure line segment in the initial three-dimensional line segment map to a target three-dimensional infinite straight line, and a sine value of a direction angle between the three-dimensional infinite straight line of the structure line segment and the target three-dimensional infinite straight line;
[0206] The texture line segment is associated with one three-dimensional plane, the structure line segment is associated with two three-dimensional planes, and the target three-dimensional infinite straight line is the intersection of the two three-dimensional planes associated with the structure line segment.
[0207] Based on the above embodiment, in the three-dimensional line segment reconstruction device provided in the embodiment of the present invention, the first target point is the projection point of the midpoint of the texture line segment on the three-dimensional infinite length straight line of the texture line segment;
[0208] The second target point is a projection point of the midpoint of the structural line segment on the three-dimensional infinite length straight line of the structural line segment.
[0209] On the basis of the above embodiments, in the three-dimensional line segment reconstruction device provided in the embodiments of the present invention, the problem-solving module is specifically used to:
[0210] For any image in the image set, a similarity graph is constructed using valid two-dimensional line segments in the image as nodes, lines connecting the nodes and matching nodes in the neighboring images of the image as edges, and similarities between the best three-dimensional line segment proposals of the two nodes as edge weights; the valid two-dimensional line segments refer to two-dimensional line segments of the best three-dimensional line segment proposals that have at least one two-dimensional supporting line segment in the image, and the two-dimensional supporting line segments of the three-dimensional line segment proposals refer to two-dimensional line segments in the neighboring images of the image that correspond to the maximum non-zero similarity score and match the two-dimensional line segment;
[0211] Based on an agglomerative clustering algorithm with target constraints, clustering the nodes in the similarity graph to reconstruct the initial three-dimensional line segment map;
[0212] The target constraint is as follows: for two two-dimensional line segments in the same image, if the angle between the two two-dimensional line segments is higher than a first threshold, or the minimum value of the maximum two-dimensional endpoint perpendicular distance of the two two-dimensional line segments is higher than a second threshold, then the two two-dimensional line segments belong to different three-dimensional line segments in the initial three-dimensional line segment map; the maximum two-dimensional endpoint perpendicular distance refers to the maximum perpendicular distance from the endpoint of one of the two two-dimensional line segments to the two-dimensional infinite straight line where the other two-dimensional line segment is located.
[0213] On the basis of the above embodiments, in the three-dimensional line segment reconstruction device provided in the embodiments of the present invention, the suggestion generation module is specifically configured to:
[0214] For any two-dimensional line segment in any one of the images, matching the any two-dimensional line segment with a two-dimensional line segment in a neighboring image of the any one of the images to obtain a matching pair of the any two-dimensional line segment;
[0215] Projecting each two-dimensional line segment in the matching pair into three-dimensional space to obtain two planes, and determining the association relationship between any two-dimensional line segment and the vanishing point;
[0216] generating a 3D line segment proposal for any 2D line segment based on the intersection of the two planes, the vanishing point associated with any 2D line segment, and the shared 3D points and the target non-shared 3D points of each 2D line segment in the matching pair;
[0217] The target non-shared three-dimensional point refers to a non-shared three-dimensional point whose reprojection error with another two-dimensional line segment in the matching pair is less than a preset number of pixels.
[0218] Specifically, the functions of each module in the three-dimensional line segment reconstruction device provided in the embodiment of the present invention correspond one-to-one to the operation process of each step in the above method embodiment, and the effects achieved are also consistent. Please refer to the above embodiment for details, and no further details will be given in the embodiment of the present invention.
[0219] Figure 11 An example of a physical structure diagram of an electronic device is shown below. Figure 11As shown, the electronic device may include: a processor (Processor) 110, a communication interface (Communications Interface) 120, a memory (Memory) 130, and a communication bus 140. The processor 110, the communication interface 120, and the memory 130 communicate with each other via the communication bus 140. The processor 110 may call the logic instructions in the memory 130 to execute the three-dimensional line segment reconstruction method provided in the above embodiments.
[0220] In addition, the logic instructions in the above-mentioned memory 130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0221] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional line segment reconstruction method provided in the above embodiments.
[0222] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the three-dimensional line segment reconstruction method provided in the above embodiments.
[0223] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.
[0224] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.
[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. A 3D line segment reconstruction method, It is characterized in that include: Extracting two-dimensional line segments within each image in the image set and determining a neighborhood image of each image; For any image in the image set, generating a three-dimensional line segment proposal for a two-dimensional line segment in the any image; The problem of selecting all the best three-dimensional line segment proposals corresponding to all the two-dimensional line segments in the image set is modeled as a 0-1 programming problem; the consistency score function corresponding to any three-dimensional line segment proposal of any two-dimensional line segment in any image in the 0-1 programming problem is determined based on the sum of the maximum similarity scores between any three-dimensional line segment proposal and all neighboring images of the any image; the maximum similarity score between any three-dimensional line segment proposal and any neighboring image of the any image refers to: the maximum value of the similarity scores between any three-dimensional line segment proposal and the best three-dimensional line segment proposals of all two-dimensional line segments in any neighboring image that match the any two-dimensional line segment; Solving the 0-1 programming problem to obtain all optimal three-dimensional line segment suggestions corresponding to the image set, and clustering all two-dimensional line segments in the image set based on all the optimal three-dimensional line segment suggestions to reconstruct an initial three-dimensional line segment map; Acquire a three-dimensional point cloud, and identify a three-dimensional plane based on the three-dimensional point cloud; the three-dimensional point cloud includes sampling points in the initial three-dimensional line segment map and SfM points determined by using a structure from motion method based on the image set; Calculating a first summation result of reprojection errors of each 3D point in the 3D point cloud and a second summation result of reprojection errors of the 3D line segments in the initial 3D line segment map; Calculating a first association relationship between the three-dimensional point and its associated three-dimensional plane; Calculating a second association relationship between points and lines in the initial three-dimensional line segment map; Calculating a third association relationship between each three-dimensional line segment in the initial three-dimensional line segment map and its associated three-dimensional plane; Determine a loss function based on the first summation result, the second summation result, the first association relationship, the second association relationship, and the third association relationship, and optimize the initial three-dimensional line segment map by minimizing the loss function to obtain a target three-dimensional line segment map; The step of clustering all the two-dimensional line segments in the image set based on all the best three-dimensional line segment suggestions and reconstructing an initial three-dimensional line segment map comprises: For any image in the image set, a similarity graph is constructed with valid two-dimensional line segments in the image as nodes, lines connecting the nodes and matching nodes in the neighborhood images of the image as edges, and similarities between the best three-dimensional line segment suggestions of the two nodes as edge weights; the valid two-dimensional line segments refer to two-dimensional line segments of the best three-dimensional line segment suggestions with at least one two-dimensional supporting line segment in the image, and the two-dimensional supporting line segments of the three-dimensional line segment suggestions refer to two-dimensional line segments in the neighborhood images of the image that correspond to the maximum non-zero similarity score and match the two-dimensional line segment; Based on an agglomerative clustering algorithm with target constraints, clustering the nodes in the similarity graph, and reconstructing the initial three-dimensional line segment map; Among them, the target constraint is: for two two-dimensional line segments in the same image, if the angle between the two two-dimensional line segments is higher than a first threshold, or the minimum value of the maximum two-dimensional endpoint vertical distance of the two two-dimensional line segments is higher than a second threshold, then the two two-dimensional line segments belong to different three-dimensional line segments in the initial three-dimensional line segment map; the maximum two-dimensional endpoint vertical distance refers to the maximum vertical distance from the endpoint of one of the two two-dimensional line segments to the two-dimensional infinite straight line where the other two-dimensional line segment is located.
2. The three-dimensional line segment reconstruction method according to claim 1, It is characterized in that The second association relationship includes: the first distance from the three-dimensional point to the associated three-dimensional line segment in the initial three-dimensional line segment map, the sine value of the angle between the direction corresponding to the vanishing point in the initial three-dimensional line segment map and the direction of the straight line where the associated three-dimensional line segment is located, and the orthogonal regularization term between any two vanishing points of the initial three-dimensional line segment map.
3. The three-dimensional line segment reconstruction method according to claim 1, It is characterized in that The third association relationship includes: a second distance from a first target point on a three-dimensional infinite straight line of a texture segment in the initial three-dimensional segment map to a three-dimensional plane associated with the texture segment, an absolute value of a cosine of an angle between a direction of the texture segment and a normal vector of the three-dimensional plane associated with the texture segment, a third distance from a second target point on a three-dimensional infinite straight line of a structure segment in the initial three-dimensional segment map to a target three-dimensional infinite straight line, and a sine value of a direction angle between the three-dimensional infinite straight line of the structure segment and the target three-dimensional infinite straight line; The texture line segment is associated with one three-dimensional plane, the structure line segment is associated with two three-dimensional planes, and the target three-dimensional infinite straight line is the intersection of the two three-dimensional planes associated with the structure line segment.
4. The three-dimensional line segment reconstruction method according to claim 3, It is characterized in that The first target point is a projection point of the midpoint of the texture line segment on the three-dimensional infinite straight line of the texture line segment; The second target point is a projection point of the midpoint of the structural line segment on the three-dimensional infinite straight line of the structural line segment.
5. The three-dimensional line segment reconstruction method according to any one of claims 1 to 4, It is characterized in that Generating a three-dimensional line segment suggestion of a two-dimensional line segment in any one of the images comprises: For any two-dimensional line segment in any one of the images, matching the any two-dimensional line segment with a two-dimensional line segment in a neighborhood image of the any one of the images to obtain a matching pair of the any two-dimensional line segment; Projecting each two-dimensional line segment in the matching pair into a three-dimensional space to obtain two planes, and determining the association relationship between any two-dimensional line segment and the vanishing point; Generate a 3D line segment suggestion for any 2D line segment based on the intersection of the two planes, the vanishing points associated with any 2D line segment, and the shared 3D points and the target non-shared 3D points of each 2D line segment in the matching pair; The target non-shared three-dimensional point refers to a non-shared three-dimensional point whose reprojection error with another two-dimensional line segment in the matching pair is less than a preset number of pixels.
6. A three-dimensional line segment reconstruction device, It is characterized in that include: An extraction and determination module is used to extract two-dimensional line segments in each image in the image set and determine the neighborhood image of each image; a suggestion generation module, for generating, for any image in the image set, a three-dimensional line segment suggestion of a two-dimensional line segment in the any image; A problem modeling module, used for modeling all the best suggestion selection problems corresponding to all the images in the image set into a 0-1 programming problem; the consistency score function corresponding to any three-dimensional line segment suggestion of any two-dimensional line segment in any image in the 0-1 programming problem is determined based on the sum of the maximum similarity scores between any three-dimensional line segment suggestion and all neighboring images of the any image; the maximum similarity score between any three-dimensional line segment suggestion and any neighboring image of the any image refers to: the maximum value of the similarity scores between any three-dimensional line segment suggestion and the best three-dimensional line segment suggestions of all two-dimensional line segments in any neighboring image that match the any two-dimensional line segment; A problem solving module, used for solving the 0-1 programming problem, obtaining all the best three-dimensional line segment suggestions corresponding to the image set, and clustering all the two-dimensional line segments in the image set based on all the best three-dimensional line segment suggestions, and reconstructing an initial three-dimensional line segment map; The step of clustering all the two-dimensional line segments in the image set based on all the best three-dimensional line segment suggestions and reconstructing an initial three-dimensional line segment map comprises: For any image in the image set, a similarity graph is constructed with valid two-dimensional line segments in the image as nodes, lines connecting the nodes and matching nodes in the neighborhood images of the image as edges, and similarities between the best three-dimensional line segment suggestions of the two nodes as edge weights; the valid two-dimensional line segments refer to two-dimensional line segments of the best three-dimensional line segment suggestions with at least one two-dimensional supporting line segment in the image, and the two-dimensional supporting line segments of the three-dimensional line segment suggestions refer to two-dimensional line segments in the neighborhood images of the image that correspond to the maximum non-zero similarity score and match the two-dimensional line segment; Based on an agglomerative clustering algorithm with target constraints, clustering the nodes in the similarity graph, and reconstructing the initial three-dimensional line segment map; The target constraint is as follows: for two two-dimensional line segments in the same image, if the angle between the two two-dimensional line segments is higher than a first threshold, or the minimum value of the maximum two-dimensional endpoint vertical distance of the two two-dimensional line segments is higher than a second threshold, then the two two-dimensional line segments belong to different three-dimensional line segments in the initial three-dimensional line segment map; the maximum two-dimensional endpoint vertical distance refers to the maximum vertical distance from the endpoint of one of the two two-dimensional line segments to the two-dimensional infinite straight line where the other two-dimensional line segment is located; Joint Optimization Module for: Acquire a three-dimensional point cloud, and identify a three-dimensional plane based on the three-dimensional point cloud; the three-dimensional point cloud includes sampling points in the initial three-dimensional line segment map and SfM points determined by using a structure from motion method based on the image set; Calculating a first summation result of reprojection errors of each 3D point in the 3D point cloud and a second summation result of reprojection errors of the 3D line segments in the initial 3D line segment map; Calculating a first association relationship between the three-dimensional point and its associated three-dimensional plane; Calculating a second association relationship between points and lines in the initial three-dimensional line segment map; Calculating a third association relationship between each three-dimensional line segment in the initial three-dimensional line segment map and its associated three-dimensional plane; Based on the first summation result, the second summation result, the first association relationship, the second association relationship and the third association relationship, a loss function is determined, and by minimizing the loss function, the initial three-dimensional line segment map is optimized to obtain a target three-dimensional line segment map.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the three-dimensional line segment reconstruction method according to any one of claims 1 to 5 is implemented.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the three-dimensional line segment reconstruction method according to any one of claims 1 to 5 is implemented.
9. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the three-dimensional line segment reconstruction method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Multi-view image three-dimensional line segment reconstruction method and system, electronic equipment and medium
CN116246038A
Visual localization method based on lightweight structured line map
CN116662600A
Piecewise planar reconstruction of three-dimensional scenes
US20100315412A1
Three-dimensional reconstruction method and apparatus, and computer-readable storage medium
WO2024037562A1
