Three-dimensional reconstruction method, device, terminal and storage medium based on line segment matching

Abstract

The present invention discloses a three-dimensional reconstruction method, device, terminal and storage medium based on line segment matching, the method comprising: extracting line segments from at least two plane images taken at different angles, and confirming multiple line segment structures of each plane image based on the line segments; matching the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images; calculating the relative posture parameters between any two plane images using the matched line segment structures; calculating the absolute posture parameters of the camera when taking each plane image using the matching number and the relative posture parameters; constructing the final three-dimensional model based on the line segments and the absolute posture parameters. In the above manner, the present invention reconstructs the three-dimensional model by line segment matching, which is more accurate than the three-dimensional reconstruction method of feature point matching.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional models, and in particular to a three-dimensional reconstruction method, device, terminal and storage medium based on line segment matching. Background Art

[0002] Image-based 3D reconstruction mainly includes two stages: sparse reconstruction and dense reconstruction. Sparse reconstruction is based on data association (image matching) to achieve feature point reconstruction, which is generally completed through motion recovery structure. Dense reconstruction is to obtain a denser 3D point cloud after sparse reconstruction.

[0003] Existing image matching and 3D reconstruction technologies rely heavily on the extraction and matching of feature points. However, the feature point method is greatly affected by the environment and may have a large number of mismatches. The traditional 3D reconstruction method based on feature points has low accuracy, and the effect varies greatly in different data, and is not robust enough. Although there have been some research results that use line segments as features for image matching and 3D reconstruction, it is difficult to get rid of the dependence on feature point extraction. Such methods mainly combine points and lines, and have strict requirements on the position of line segments (for example, requiring line segments to be parallel or perpendicular), which has great limitations. Summary of the invention

[0004] The present application provides a three-dimensional reconstruction method, device, terminal and storage medium based on line segment matching to solve the problem of low accuracy of the existing three-dimensional reconstruction method based on feature point matching.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a three-dimensional reconstruction method based on line segment matching, including: extracting line segments from at least two plane images taken at different angles, and confirming multiple line segment structures of each plane image based on the line segments; matching the line segment structures of any two plane images respectively to obtain the number of matches of the line segment structures of any two plane images; calculating the relative posture parameters between any two plane images using the matched line segment structures; calculating the absolute posture parameters of the camera when taking each plane image using the matching number and the relative posture parameters; and constructing the final three-dimensional model based on the line segments and the absolute posture parameters.

[0006] As a further improvement of the present invention, the step of confirming multiple line segment structures of each plane image based on line segments includes: setting the influence area of ​​each line segment in each plane image; confirming that any two line segments are adjacent when it is detected that at least one endpoint of the first line segment is within the influence area of ​​the second line segment, and the intersection of the straight line where the first line segment is located and the straight line where the second line segment is located is within the influence area of ​​the second line segment; and forming a line segment structure from any two line segments.

[0007] As a further improvement of the present invention, the step of matching the line segment structures of any two plane images includes: setting a description vector for each line segment structure to describe the local features of the line segment structure; and using the description vector of the line segment structure to match two line segment structures that do not belong to the same plane image.

[0008] As a further improvement of the present invention, the step of matching two line segment structures that do not belong to the same plane image using the description vector of the line segment structure includes: obtaining the angle between the line segments in each line segment structure; calculating the angle difference between the target line segment structure and each line segment structure of the other plane image, and screening out a first set of line segment structures whose angle difference is less than a preset angle threshold; calculating the distance between the target line segment structure and each line segment structure in the first set of line segment structures using the description vector, and screening out a second set of line segment structures whose distance is less than a preset distance threshold; confirming the line segment structure with the smallest distance from the second set of line segment structures, and confirming that the line segment structure with the smallest distance matches the target line segment structure.

[0009] As a further improvement of the present invention, after the step of obtaining the matching number of line segment structures of any two planar images, the method further includes: deleting incorrectly matched line segment structures based on epipolar constraints and topological distribution of the line segment structures.

[0010] As a further improvement of the present invention, the step of using the matching number and the relative pose parameters to calculate the absolute pose parameters of the camera when taking each plane image includes: selecting the first plane image and the second plane image with the largest number of matches, and constructing a world coordinate system with the camera corresponding to the first plane image as the origin, so that the relative pose parameters of the first plane image and the second plane image are the absolute pose parameters of the camera corresponding to the second plane image; selecting the third plane image with the largest number of matches with the first plane image or the second plane image from the remaining plane images, and using the relative pose parameters of the third plane image and the first plane image or the second plane image to calculate the absolute pose parameters of the camera corresponding to the third plane image, and repeating the above steps until the absolute pose parameters of the camera corresponding to each plane image are obtained.

[0011] As a further improvement of the present invention, the step of constructing the final three-dimensional model based on line segments and absolute pose parameters includes: based on the line segments and absolute pose parameters, using the bundle adjustment method of points and line segments to perform sparse reconstruction to obtain an initial three-dimensional model; using a multi-view stereo vision library to perform dense reconstruction on the initial three-dimensional model to obtain a final three-dimensional model.

[0012] To solve the above technical problems, another technical solution adopted in the present application is: to provide a three-dimensional reconstruction device based on line segment matching, including: an extraction module, used to extract line segments from at least two plane images taken at different angles, and confirm multiple line segment structures of each plane image based on the line segments; a matching module, used to match the line segment structures of any two plane images respectively, and obtain the number of matches of the line segment structures of any two plane images; a relative posture calculation module, used to calculate the relative posture parameters between any two plane images using the matched line segment structures; an absolute posture calculation module, used to calculate the absolute posture parameters of the camera when taking each plane image using the matching number and relative posture parameters; a three-dimensional reconstruction module, used to construct a final three-dimensional model based on the line segments and absolute posture parameters.

[0013] To solve the above technical problems, another technical solution adopted in the present application is: providing a terminal, which includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing the above-mentioned three-dimensional reconstruction method based on line segment matching; the processor is used to execute the program instructions stored in the memory to reconstruct the three-dimensional model.

[0014] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a storage medium storing a program file capable of implementing the above three-dimensional reconstruction method based on line segment matching.

[0015] The beneficial effects of the present application are as follows: the present invention gets rid of the dependence on feature point extraction by matching the line segment structure composed of line segments in the plane image. Compared with the feature point matching method, the present invention has higher robustness and accuracy. Moreover, the present invention does not require the line segments to be parallel or perpendicular to each other, but only needs to be coplanar, and has strong versatility. In addition, when matching, any two images are matched with each other, so that three-dimensional reconstruction can be performed based on disordered images, without requiring the images to be continuous and orderly. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic flow chart of a three-dimensional reconstruction method based on line segment matching according to a first embodiment of the present invention;

[0017] Figure 2 is a flow chart of a three-dimensional reconstruction method based on line segment matching according to a second embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of determining a line segment structure in an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of a line segment structure type in an embodiment of the present invention;

[0020] Figure 5is a schematic flow chart of a three-dimensional reconstruction method based on line segment matching according to a third embodiment of the present invention;

[0021] Figure 6 is a schematic diagram of constructing a description vector of a line segment structure according to an embodiment of the present invention;

[0022] Figure 7 is a schematic flow chart of a three-dimensional reconstruction method based on line segment matching according to a fourth embodiment of the present invention;

[0023] Figure 8 A schematic diagram of topological distribution constraints of a line segment structure according to an embodiment of the present invention;

[0024] Fig. 9 is a schematic flow chart of a three-dimensional reconstruction method based on line segment matching according to a fifth embodiment of the present invention;

[0025] Fig.10 A schematic diagram of a line segment coplanar constraint according to an embodiment of the present invention;

[0026] Fig.11 is a schematic structural diagram of a three-dimensional reconstruction device based on line segment matching according to an embodiment of the present invention;

[0027] Fig.12 is a schematic diagram of the structure of a terminal according to an embodiment of the present invention;

[0028] Fig.13 It is a schematic diagram of the structure of the storage medium of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The terms "first", "second", "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] Figure 1 FIG. 1 is a flow chart of a three-dimensional reconstruction method based on line segment matching according to a first embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present invention is not limited to the first embodiment. Figure 1 The process sequence shown is limited. Figure 1 As shown, the method comprises the steps of:

[0033] Step S101: extracting line segments from at least two planar images taken at different angles, and confirming multiple line segment structures of each planar image based on the line segments.

[0034] In step S101, the image of the target object is first captured by a camera from multiple different angles to obtain multiple two-dimensional plane images, and then the line segment detection algorithm (Line Segment Detector, LSD) is used to extract line segments from each plane image, and the line segment structure composed of the line segments in each plane image is determined.

[0035] Step S102: matching the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images.

[0036] In step S102, the line segment structures of any two plane images are matched one by one, so as to confirm the line segment structures that match each other in the two plane images and the matching number of the line segment structures that match each other.

[0037] Step S103: Calculate the relative pose parameters between any two planar images using the matched line segment structure.

[0038] In step S103, in the process of matching the memory line segment structure, according to the epipolar geometry constraint, combined with the intrinsic parameters of the camera, based on the five-point method and the random sampling consensus algorithm (RANSAC), the essential matrix between each two images can be obtained, and then the rotation matrix and translation vector, that is, the relative pose parameters between the two plane images, are obtained through singular value decomposition. It should be noted that when the intrinsic parameters of the camera are unknown, we can also obtain the relevant parameters from the EXIF ​​information of the image.

[0039] Step S104: using the matching number and the relative pose parameters to calculate the absolute pose parameters of the camera when taking each planar image.

[0040] In step S104, after obtaining the relative pose parameters between every two plane images, the pair of plane images with the largest number of matches is determined as the initial plane images in combination with the number of matches between the plane images, and the camera corresponding to one of the plane images is used as the origin to construct the world coordinate system, and then the absolute pose parameters of the camera corresponding to each plane image are calculated in combination with the relative pose parameters between the plane images.

[0041] Step S105: constructing a final three-dimensional model based on the line segments and absolute pose parameters.

[0042] Specifically, step S105 includes:

[0043] Based on line segments and absolute pose parameters, the bundle adjustment method of points and line segments is used to perform sparse reconstruction to obtain an initial 3D model. The multi-view stereo vision library is then used to perform dense reconstruction on the initial 3D model to obtain the final 3D model.

[0044] The three-dimensional reconstruction method based on line segment matching in the first embodiment of the present invention uses the line segment structure composed of line segments in the plane image for matching, thereby getting rid of the dependence on feature point extraction. Compared with the feature point matching method, the robustness and accuracy are higher. Moreover, the present invention does not require the line segments to be parallel or perpendicular to each other, but only needs to be coplanar, and has strong versatility. In addition, when matching, any two images are matched with each other, so that three-dimensional reconstruction can be performed based on disordered images without requiring the images to be continuous and orderly.

[0045] Figure 2FIG. 1 is a flow chart of a three-dimensional reconstruction method based on line segment matching according to a second embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present invention is not limited to the method of FIG. 1 . Figure 2 The process sequence shown is limited. Figure 2 As shown, the method comprises the steps of:

[0046] Step S201: extracting line segments from at least two planar images taken at different angles.

[0047] In this embodiment, Figure 2 Step S201 and Figure 1 The process is similar to step S101 in , and for the sake of brevity, it will not be repeated here.

[0048] Step S202: setting the influence area of ​​each line segment in each plane image.

[0049] Step S203: When it is detected that among any two line segments, at least one endpoint of the first line segment is within the influence area of ​​the second line segment, and the intersection of the straight line where the first line segment is located and the straight line where the second line segment is located is within the influence area of ​​the second line segment, it is confirmed that the any two line segments are adjacent and are two coplanar line segments.

[0050] In step S202 to step S203, if Figure 3 As shown, for each line segment in a plane image, an influence area R of (|l1|+2w)×2w is defined, where |l1| is the length of line segment l1, and w is a user-defined parameter (w=20 in this embodiment). When at least one endpoint of a line segment is within the influence area of ​​line segment l1, and when the intersection of the straight line where the line segment is located and the straight line where line segment l1 is located is within the influence area of ​​line segment l1, the line segment is determined to be adjacent to line segment l1 and constitute a line segment structure.

[0051] Step S204: Construct a line segment structure from any two line segments.

[0052] In step S204, the line segment structure generally includes three types, such as Figure 4 As shown in A, the intersection of the two line segments is located on the extension line of the two line segments, which is the first line segment structure; Figure 4 As shown in B, the intersection of the two line segments is located on one of the line segments and on the extension line of the other line segment, which is the second line segment structure; Figure 4 As shown in C, when the intersection of two line segments is located on both line segments at the same time, it is the third type of line segment structure.

[0053] Step S205: matching the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images.

[0054] In this embodiment, Figure 2Step S205 and Figure 1 The process is similar to step S102 in , and will not be described again for the sake of brevity.

[0055] Step S206: Calculate the relative pose parameters between any two planar images using the matched line segment structure.

[0056] In this embodiment, Figure 2 Step S206 and Figure 1 The process is similar to step S103 in , and will not be described again for the sake of brevity.

[0057] Step S207: using the matching number and the relative pose parameters to calculate the absolute pose parameters of the camera when taking each planar image.

[0058] In this embodiment, Figure 2 Step S207 and Figure 1 The process is similar to step S104 in , and will not be described again for the sake of brevity.

[0059] Step S208: constructing a final three-dimensional model based on the line segments and absolute pose parameters.

[0060] In this embodiment, Figure 2 Step S208 and Figure 1 The process is similar to step S105 in , and will not be described again for the sake of brevity.

[0061] The three-dimensional reconstruction method based on line segment matching in the second embodiment of the present invention is based on the first embodiment. An influence area is set for each line segment, and adjacent line segments are determined according to the influence area. A line segment structure is constructed with adjacent line segments. There is no strict requirement on the position of the line segments, thereby ensuring that enough line segments can be extracted, making the line segment matching result more accurate.

[0062] Figure 5 FIG. 1 is a flow chart of a three-dimensional reconstruction method based on line segment matching according to a third embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present invention is not limited to the method of FIG. Figure 5 The process sequence shown is limited. Figure 5 As shown, the method comprises the steps of:

[0063] Step S301: extracting line segments from at least two planar images taken at different angles, and confirming multiple line segment structures of each planar image based on the line segments.

[0064] In this embodiment, Figure 5 Step S201 and Figure 1 The process is similar to step S101 in , and for the sake of brevity, it will not be repeated here.

[0065] Step S302: setting a description vector for each line segment structure to describe the local features of the line segment structure.

[0066] In step S302, after obtaining the line segment structure of each plane image, refer to Figure 6 , analogous to SIFT features, a gradient direction histogram is constructed in the local area around the intersection of the line segment structure. For the line segment structure AOB, two concentric circles are constructed with the intersection O as the center, where the radius of the small circle is r and the radius of the large circle is twice the radius of the small circle. The two concentric circles are divided into 4 parts by the straight lines where the two line segments are located. Each part contains a sector and an annular area. Each annular area is divided into three sub-areas, so that the area of ​​the sub-area is equal to the area of ​​the corresponding sector. There are a total of 16 areas, 8 groups of gradient direction histograms are constructed, and a 128-dimensional vector is generated as the description vector of the line segment structure.

[0067] Step S303: using the description vector of the line segment structure, matching two line segment structures that do not belong to the same plane image, and obtaining the matching number of line segment structures of any two plane images.

[0068] In step S303, when matching the line segment structure, the distance between the description vectors of the line segment structure can be calculated to confirm whether the line segment structure matches. Considering that the line segments in the line segment structure are coplanar, the angle between the two line segments will not change too much during the image transformation. Therefore, in this embodiment, in order to reduce the amount of calculation and improve the accuracy of the matching result, before calculating the distance between the description vectors, the angles between different line segment structures can also be compared. The specific steps are as follows:

[0069] 1. Get the angle between the line segments in each line segment structure.

[0070] Specifically, refer to the three types of line segment structures described in the second embodiment to obtain the angles between line segments in each line segment structure.

[0071] 2. Calculate the angle difference between the target line segment structure and each line segment structure of another plane image, and select a first set of line segment structures whose angle difference is less than a preset angle threshold.

[0072] It should be noted that the preset angle threshold is pre-set. Specifically, when a target line segment structure is matched with each line segment structure of another plane image, the angle difference between the target line segment structure and the line segment structure of another plane image is calculated, and when the angle difference is less than the preset angle threshold, the line segment structure is classified into the first line segment structure set.

[0073] 3. Calculate the distance between the target line segment structure and each line segment structure in the first line segment structure set using the description vector, and select the second line segment structure set whose distance is less than a preset distance threshold.

[0074] It should be noted that the preset distance threshold is preset.

[0075] 4. Confirm the line segment structure with the shortest distance from the second line segment structure set, and confirm that the line segment structure with the shortest distance matches the target line segment structure.

[0076] Specifically, when a planar image has multiple line segment structures that meet the requirements of the preset angle threshold and the preset distance threshold, the line segment structure with the smallest distance is selected to match the target line segment structure.

[0077] Step S304: Calculate the relative pose parameters between any two planar images using the matched line segment structure.

[0078] In this embodiment, Figure 5 Step S304 and Figure 1 The process is similar to step S103 in , and will not be described again for the sake of brevity.

[0079] Step S305: Calculate the absolute pose parameters of the camera when taking each planar image using the matching number and relative pose parameters.

[0080] In this embodiment, Figure 5 Step S305 and Figure 1 The process is similar to step S104 in , and will not be described again for the sake of brevity.

[0081] Step S306: constructing a final three-dimensional model based on the line segments and absolute pose parameters.

[0082] In this embodiment, Figure 5 Step S306 and Figure 1 The process is similar to step S105 in , and will not be described again for the sake of brevity.

[0083] The three-dimensional reconstruction method based on line segment matching in the third embodiment of the present invention is based on the first embodiment. Each line segment structure is described by a description vector, and the similarity between the line segment structures corresponding to two description vectors can be confirmed by calculating the distance between the description vectors. When the distance between the description vectors is less than a preset distance threshold, the two corresponding line segment structures are considered to match. In addition, before calculating the distance between the description vectors, this embodiment performs preliminary screening through the angle difference between the line segment structures, thereby reducing the amount of data processing.

[0084] Figure 7 FIG. 4 is a flow chart of a three-dimensional reconstruction method based on line segment matching according to a fourth embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present invention is not limited to the method of FIG. Figure 7 The process sequence shown is limited. Figure 7 As shown, the method comprises the steps of:

[0085] Step S401: extracting line segments from at least two planar images taken at different angles, and confirming multiple line segment structures of each planar image based on the line segments.

[0086] In this embodiment, Figure 7 Step S401 and Figure 1 The process is similar to step S101 in , and for the sake of brevity, it will not be repeated here.

[0087] Step S402: matching the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images.

[0088] In this embodiment, Figure 7 Step S402 and Figure 1 The process is similar to step S102 in , and will not be described again for the sake of brevity.

[0089] Step S403: deleting incorrectly matched line segment structures based on the epipolar constraints and the topological distribution of the line segment structures.

[0090] In step S403, after completing the matching of the line segment structure, the basic matrix between the two plane images can be obtained according to the line segment structure matching structure, and some incorrectly matched line segment structures are deleted based on the epipolar constraint. In addition, the topological distribution between the LJL structures should be unchanged, and we have also deleted some incorrect matches accordingly, please refer to Figure 8 , each line segment structure divides the area around it into 4 quadrants. It can be considered that for two matching line segment structures, at least 50% of the line segment structures around them should be matched, and among these matching line segment structures, at least 80% should be located in the same quadrant.

[0091] Step S404: Calculate the relative pose parameters between any two planar images using the matched line segment structure.

[0092] In this embodiment, Figure 7 Step S404 and Figure 1 The process is similar to step S103 in , and will not be described again for the sake of brevity.

[0093] Step S405: Calculate the absolute pose parameters of the camera when taking each planar image using the matching number and relative pose parameters.

[0094] In this embodiment, Figure 7 Step S405 and Figure 1 The process is similar to step S104 in , and will not be described again for the sake of brevity.

[0095] Step S406: constructing a final three-dimensional model based on the line segments and absolute pose parameters.

[0096] In this embodiment, Figure 7 Step S406 and Figure 1 The process is similar to step S105 in , and will not be described again for the sake of brevity.

[0097] The three-dimensional reconstruction method based on line segment matching in the fourth embodiment of the present invention is based on the first embodiment. By deleting some incorrectly matched line segment structures based on polar line constraints and the topological distribution of the line segment structure, the accuracy of the final matching result is further improved, making the triple reconstruction effect better.

[0098] Fig. 9 FIG. 5 is a flow chart of a three-dimensional reconstruction method based on line segment matching according to a fifth embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present invention is not limited to the method of FIG. Fig. 9 The process sequence shown is limited. Fig. 9 As shown, the method comprises the steps of:

[0099] Step S501: extracting line segments from at least two planar images taken at different angles, and confirming multiple line segment structures of each planar image based on the line segments.

[0100] In this embodiment, Fig. 9 Step S501 and Figure 1 The process is similar to step S101 in , and for the sake of brevity, it will not be repeated here.

[0101] Step S502: Match the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images.

[0102] In this embodiment, Fig. 9 Step S502 and Figure 1 The process is similar to step S102 in , and will not be described again for the sake of brevity.

[0103] Step S503: Calculate the relative pose parameters between any two planar images using the matched line segment structure.

[0104] In this embodiment, Fig. 9 Step S503 and Figure 1 The process is similar to step S103 in , and will not be described again for the sake of brevity.

[0105] Step S504: Select the first plane image and the second plane image with the largest number of matches, and construct a world coordinate system with the camera corresponding to the first plane image as the origin. The relative pose parameters of the first plane image and the second plane image are the absolute pose parameters of the camera corresponding to the second plane image.

[0106] In step S504, after obtaining the matching number of line segment structures of any two plane images, two plane images with the largest number of matching are selected from them, and any one of the two plane images is used as the first plane image, and the other is used as the second plane image. A world coordinate system is constructed with the posture of the first camera that shoots the first plane image as the far point, and the rotation matrix of the first camera is used as the unit matrix to obtain the absolute posture parameters of the first camera, and then the distance between the first camera and the second camera that shoots the second plane image is used as the unit length, and based on the relative posture parameters between the first plane image and the second plane image, the absolute posture parameters of the second camera can be obtained.

[0107] Step S505: Select the third plane image with the largest number of matches with the first plane image or the second plane image from the remaining plane images, and use the relative pose parameters of the third plane image and the first plane image or the second plane image to calculate the absolute pose parameters of the camera corresponding to the third plane image, and repeat the above steps until the absolute pose parameters of the camera corresponding to each plane image are obtained.

[0108] In step S505, after determining the absolute posture parameters of the first plane image and the second plane image, a third plane image is selected from the remaining plane images, and the number of matching line segment structures between the third plane image and the first plane image or the second plane image is second only to the number of matching line segment structures between the first plane image and the second plane image.

[0109] Among them, the calculation process of the absolute pose parameters of the camera corresponding to the third plane image is as follows:

[0110] (1) When the number of matches between the third plane image and the first plane image is the largest, the absolute pose parameters of the camera corresponding to the third plane image are solved by referring to the process of solving the absolute pose parameters of the camera corresponding to the second plane image.

[0111] (2) When the number of matches between the third plane image and the second plane image is the largest, refer to Fig.10 , number the first plane image, the second plane image, and the third plane image as 1, 2, and 3 respectively, assuming that the center of the camera corresponding to the three plane images is C i , i = 1, 2, 3 (i refers to the camera number), the absolute pose parameter is (R i ,T i ),i=1,2,3, where R i is the rotation matrix, T i is the translation vector, we know that:

[0112] R j =Rij R i ;

[0113] T j =R ij T i +T ij ;

[0114]

[0115] Where R ij is the relative rotation matrix between cameras i and j, T ij is the relative translation vector between cameras i and j. Now we know the absolute pose parameters of cameras 1 and 2 (R i ,T i ), i = 1, 2, and the relative pose parameter R between cameras 2 and 3 23 ,t 23 (t 23 is a unit vector), solve the absolute pose parameters (R3, T3) of camera 3, where R3 = R 23 R2, T3 = R 23 T2+T 23 , assuming T 23 =λ 23 t 23 , where λ 23 is the length of the translation vector between cameras 2 and 3, then we only need to solve λ 23 , the absolute pose parameters of camera 3 can be obtained. This can be achieved by the coplanar constraint of line segments and the simplified trifocal point / line constraint. The coplanar constraint of line segments refers to the constraint relationship constructed by two coplanar line segments in space, such as Fig.10 As shown, L a , L b are two line segments in the same plane in space, two line segments of a line segment structure, where L a There are projections on the first and second images, that is, two-dimensional line segments And L b There are projections on the second and third images, namely two-dimensional line segments To simplify the formula, the internal parameter matrix is ​​unified into the unit matrix, and the two do not affect the results. Consider the line segment L i A point P on (i=a,b) is preferably the center of the line segment. Assume that its projection point on the second image is Depth Then we have:

[0116]

[0117] (i=a,k=1 or i=b,k=3);

[0118] Calculation can be obtained (i=a, k=1 or i=b, k=3). Assume that line segment L a , L b The plane is Then its normal vector is in is line segment L a The direction vector of is line segment L b The direction vector of the line segment L a Is a plane (That is, point C1 and line segment The plane formed by (That is, point C2 and line segment The intersection of the planes formed by

[0119]

[0120]

[0121] Assume M is a plane For any point on the plane, since point P is also in the plane, we have According to the above formulas, we can get:

[0122] (i=a,k=1 or i=b,k=3);

[0123] The left side of the above formula is a constant, so in the two cases of i=a, k=1 and i=b, k=3, the value on the right side of the above formula is equal, and then we can get:

[0124]

[0125] where t ij is the unit translation vector between cameras i and j, λ ij is the length of the translation vector between cameras i and j, that is, satisfies T ij =λ ij t ij According to the above calculation process, the absolute posture parameters of the third plane image are calculated.

[0126] After obtaining the absolute pose parameters of the first, second, and third plane images, select from the remaining plane images, and refer to the process of calculating the absolute pose parameters of the third plane image to calculate the absolute pose parameters of the camera corresponding to the newly selected plane image, and repeat the above process until the absolute pose parameters of the camera corresponding to each plane image are obtained. Take an example to illustrate:

[0127] Assume that there are five plane images, A, B, C, D, and E. The number of matching line segment structures between the plane images is as follows:

[0128] AB: 200;

[0129] AC: 150;

[0130] AD: 100;

[0131] AE: 120;

[0132] BC: 160;

[0133] BD: 140;

[0134] BE: 180;

[0135] CD: 300;

[0136] CE: 110;

[0137] DE: 180;

[0138] First, select the CD with the largest number of matches as the initial matching plane image and initialize it. Take the center of the camera corresponding to C as the coordinate origin to establish the world coordinate system. The absolute rotation matrix of the camera corresponding to C is the unit matrix I, and the absolute translation vector is 0. Using the relative posture parameters between CDs, the absolute rotation matrix of the camera corresponding to D can be obtained as the relative rotation matrix Rcd between CDs, and the absolute translation vector is the relative translation vector Tcd between CDs, that is, the distance between CDs is taken as the unit length. So far, the absolute posture parameters of the camera corresponding to C and the camera corresponding to D have been determined.

[0139] Then, select the one among A, B, and E that matches C or D the most, and calculate the absolute pose parameters. Compare AC, AD, BC, BD, EC, and ED, and find that ED (DE) has the most matches, so solve the absolute pose parameters of the camera corresponding to E based on the triple C, D, and E.

[0140] Select a plane image from A and B again and calculate the corresponding camera absolute pose parameters. Among them, BE has the largest number of matches, so select B. At this time, you can select the triple B, E, C or B, E, D to calculate the absolute pose parameters of the camera corresponding to B. Here, the number of matches of ED is more than that of EC, so select the triple B, E, D for calculation to ensure a more accurate result.

[0141] Finally, it is necessary to solve the absolute pose parameters of the camera corresponding to A. AB has the largest number of matches, and BE has the largest number of matches, so the triplet A, B, and E is selected to solve the absolute pose parameters of the camera corresponding to A. So far, the absolute pose parameters of the cameras corresponding to A, B, C, D, and E have been calculated.

[0142] Step S506: constructing a final three-dimensional model based on the line segments and absolute pose parameters.

[0143] In this embodiment, Fig. 9 Step S506 and Figure 1 The process is similar to step S105 in , and will not be described again for the sake of brevity.

[0144] The three-dimensional reconstruction method based on line segment matching in the fifth embodiment of the present invention is based on the first embodiment. It uses two coplanar line segments in the line segment structure to calculate the absolute pose parameters of the camera corresponding to the plane image. There is no need to select coplanar line segments for calculation. Therefore, it has stronger robustness. In addition, in this embodiment, plane images are selected from multiple plane images in disorder to calculate the absolute pose parameters of the camera, and the calculation is performed in order from large to small according to the number of matches between the plane images, so that the final absolute pose parameters are more accurate.

[0145] Fig.11 FIG. 1 is a schematic diagram of the structure of a three-dimensional reconstruction device based on line segment matching according to an embodiment of the present invention. Fig.11 As shown, the device 10 includes an extraction module 11, a matching module 12, a relative pose calculation module 13, an absolute pose calculation module 14 and a three-dimensional reconstruction module 15.

[0146] An extraction module 11, configured to extract line segments from at least two plane images taken at different angles, and to confirm multiple line segment structures of each plane image based on the line segments;

[0147] A matching module 12, used for matching the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images;

[0148] A relative posture calculation module 13 is used to calculate the relative posture parameters between any two plane images using the matched line segment structure;

[0149] The absolute posture calculation module 14 is used to calculate the absolute posture parameters of the camera when taking each planar image by using the matching number and the relative posture parameters;

[0150] The 3D reconstruction module 15 is used to construct a final 3D model based on line segments and absolute pose parameters.

[0151] Optionally, the operation of confirming the multiple line segment structures of each plane image based on line segments can also be: setting the influence area of ​​each line segment in each plane image; confirming that any two line segments are adjacent when it is detected that at least one endpoint of the first line segment is within the influence area of ​​the second line segment, and the intersection of the straight line where the first line segment is located and the straight line where the second line segment is located is within the influence area of ​​the second line segment; and forming a line segment structure from any two line segments.

[0152] Optionally, the matching module 12 can also perform the operation of matching the line segment structures of any two planar images by: setting a description vector for each line segment structure to describe the local features of the line segment structure; and matching two line segment structures that do not belong to the same planar image using the description vector of the line segment structure.

[0153] Optionally, the matching module 12 uses the description vector of the line segment structure to match two line segment structures that do not belong to the same plane image. The operations can also be as follows: obtaining the angle between the line segments in each line segment structure; calculating the angle difference between the target line segment structure and each line segment structure of the other plane image, and screening out a first set of line segment structures whose angle difference is less than a preset angle threshold; using the description vector to calculate the distance between the target line segment structure and each line segment structure in the first set of line segment structures, and screening out a second set of line segment structures whose distance is less than a preset distance threshold; confirming the line segment structure with the shortest distance from the second set of line segment structures, and confirming that the line segment structure with the shortest distance matches the target line segment structure.

[0154] Optionally, after obtaining the number of matches of the line segment structures of any two planar images, the matching module 12 is further configured to delete incorrectly matched line segment structures based on epipolar constraints and topological distribution of the line segment structures.

[0155] Optionally, the absolute pose calculation module 14 uses the matching number and relative pose parameters to calculate the absolute pose parameters of the camera when taking each plane image. The operation can also be as follows: select the first plane image and the second plane image with the largest number of matches, and construct a world coordinate system with the camera corresponding to the first plane image as the origin, then the relative pose parameters of the first plane image and the second plane image are the absolute pose parameters of the camera corresponding to the second plane image; select the third plane image with the largest number of matches with the first plane image or the second plane image from the remaining plane images, and use the relative pose parameters of the third plane image and the first plane image or the second plane image to calculate the absolute pose parameters of the camera corresponding to the third plane image, and repeat the above steps until the absolute pose parameters of the camera corresponding to each plane image are obtained.

[0156] Optionally, the operation of the three-dimensional reconstruction module 15 to construct the final three-dimensional model based on line segments and absolute pose parameters can also be: based on the line segments and absolute pose parameters, sparse reconstruction is performed using the bundle adjustment method of points and line segments to obtain an initial three-dimensional model; and dense reconstruction of the initial three-dimensional model is performed using a multi-view stereo vision library to obtain a final three-dimensional model.

[0157] See also Fig.12 , Fig.12 FIG. 1 is a schematic diagram of the structure of a terminal according to an embodiment of the present invention. Fig.12 As shown, the terminal 60 includes a processor 61 and a memory 62 coupled to the processor 61 .

[0158] The memory 62 stores program instructions for implementing the three-dimensional reconstruction method based on line segment matching described in any of the above embodiments.

[0159] The processor 61 is used to execute program instructions stored in the memory 62 to reconstruct a three-dimensional model.

[0160] The processor 61 may also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip having the ability to process signals. The processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0161] See also Fig.13 , Fig.13 Schematic diagram of the structure of the storage medium of the embodiment of the present invention. The storage medium of the embodiment of the present invention stores a program file 71 that can implement all the above methods, wherein the program file 71 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0162] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0163] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A three-dimensional reconstruction method based on line segment matching, characterized in that: include: Extracting line segments from at least two planar images taken at different angles, and confirming a plurality of line segment structures of each planar image based on the line segments; Match the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images; The relative pose parameters between any two plane images are calculated using the matched line segment structure; The absolute pose parameters of the camera when taking each planar image are calculated using the matching number and the relative pose parameters; Constructing a final three-dimensional model based on the line segments and the absolute pose parameters; The step of matching the line segment structures of any two planar images comprises: Setting a description vector for each line segment structure to describe the local features of the line segment structure; Using the description vector of the line segment structure, matching two line segment structures that do not belong to the same plane image; The step of matching two line segment structures that do not belong to the same plane image by using the description vector of the line segment structure comprises: Get the angle between the line segments in each line segment structure; Calculating the angle difference between the target line segment structure and each line segment structure of another plane image, and screening out a first set of line segment structures whose angle difference is less than a preset angle threshold; Calculating the distance between the target line segment structure and each line segment structure in the first line segment structure set by using the description vector, and screening out a second line segment structure set whose distance is less than a preset distance threshold; confirming the line segment structure with the smallest distance from the second line segment structure set, and confirming that the line segment structure with the smallest distance matches the target line segment structure; The step of calculating the absolute posture parameters of the camera when taking each planar image by using the matching number and the relative posture parameters comprises: Selecting the first plane image and the second plane image with the largest number of matches, and constructing a world coordinate system with the camera corresponding to the first plane image as the origin, then the relative pose parameters of the first plane image and the second plane image are the absolute pose parameters of the camera corresponding to the second plane image; Select a third plane image from the remaining plane images that has the largest number of matches with the first plane image or the second plane image, and use the relative pose parameters of the third plane image and the first plane image or the second plane image to calculate the absolute pose parameters of the camera corresponding to the third plane image, and repeat the above steps until the absolute pose parameters of the camera corresponding to each plane image are obtained.

2. The three-dimensional reconstruction method based on line segment matching according to claim 1, characterized in that: The step of confirming a plurality of line segment structures of each planar image based on the line segments comprises: Set the influence area of ​​each line segment in each plane image; When it is detected that, of any two line segments, at least one endpoint of a first line segment is within an influence area of ​​a second line segment, and an intersection point of a straight line where the first line segment is located and a straight line where the second line segment is located is within an influence area of ​​the second line segment, the any two line segments are determined to be adjacent; The arbitrary two line segments constitute the line segment structure.

3. The three-dimensional reconstruction method based on line segment matching according to claim 1, characterized in that: After the step of obtaining the matching number of line segment structures of any two planar images, the method further includes: Mismatched line segment structures are deleted based on epipolar constraints and the topological distribution of the line segment structures.

4. The three-dimensional reconstruction method based on line segment matching according to claim 1, characterized in that: The step of constructing the final three-dimensional model based on the line segments and the absolute posture parameters comprises: Based on the line segments and the absolute posture parameters, sparse reconstruction is performed using a bundle adjustment method of points and line segments to obtain an initial three-dimensional model; The initial three-dimensional model is densely reconstructed using a multi-view stereo vision library to obtain the final three-dimensional model.

5. A three-dimensional reconstruction device based on line segment matching using the three-dimensional reconstruction method based on line segment matching according to claim 1, characterized in that: include: An extraction module, configured to extract line segments from at least two planar images taken at different angles, and to confirm multiple line segment structures of each planar image based on the line segments; A matching module, used for matching the line segment structures of any two plane images respectively to obtain the matching number of the line segment structures of any two plane images; A relative pose calculation module is used to calculate the relative pose parameters between any two plane images using the matching line segment structure; An absolute posture calculation module, used to calculate the absolute posture parameters of the camera when taking each planar image by using the matching number and the relative posture parameters; A three-dimensional reconstruction module is used to construct a final three-dimensional model based on the line segments and the absolute posture parameters.

6. A terminal, characterized in that: The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the three-dimensional reconstruction method based on line segment matching as described in any one of claims 1 to 4; The processor is used to execute the program instructions stored in the memory to reconstruct a three-dimensional model.

7. A storage medium, characterized in that: A program file capable of implementing the three-dimensional reconstruction method based on line segment matching as described in any one of claims 1 to 4 is stored.

Images