Three-dimensional reconstruction method, device, apparatus and storage medium

Abstract

The application discloses a three-dimensional reconstruction method, device and equipment, and a storage medium, and relates to the technical field of three-dimensional reconstruction. The technical scheme provided by the application comprises the following steps: constructing a first three-dimensional map based on an image sequence by using a global reconstruction algorithm, and determining a first missing frame in the image sequence; optimizing the first three-dimensional map based on the first missing frame by using an incremental reconstruction algorithm, obtaining a second three-dimensional map, and determining a second missing frame in the image sequence; in the case that the second three-dimensional map does not satisfy a preset completeness condition, constructing a local three-dimensional map corresponding to the second missing frame by using a hole filling algorithm, filling the local three-dimensional map into the second three-dimensional map, and obtaining a target three-dimensional map. Through the above technical means, a highly complete three-dimensional map can be constructed, the success rate and robustness of three-dimensional reconstruction are effectively improved, and the problem of poor robustness of three-dimensional reconstruction in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, apparatus, device and storage medium. Background Technology

[0002] SfM (Structure from Motion) is a very popular 3D reconstruction algorithm in recent years. It aims to recover the camera position and orientation corresponding to the images by taking a series of images of the same scene, and to construct a 3D point cloud of the scene.

[0003] In existing technologies, the global rotation matrix of the image is typically determined first, followed by the global translation vector. The image is then triangulated using both the global rotation matrix and the global translation vector to construct a 3D point cloud. The calculation of the global translation vector depends on the output of the global rotation matrix; therefore, the accuracy of the global rotation matrix determines the accuracy of the 3D point cloud. However, the global rotation matrix is ​​susceptible to interference from external points, leading to inaccurate calculation results, a high reconstruction failure rate, and poor algorithm robustness. Summary of the Invention

[0004] This application provides a 3D reconstruction method, apparatus, device, and storage medium. It rapidly constructs an initial 3D map through a global reconstruction algorithm, then sequentially determines the shooting pose of each lost frame and optimizes the initial 3D map through an incremental reconstruction algorithm, and finally fills the hole areas in the local 3D map through a hole-filling strategy, resulting in a highly complete 3D map. This effectively improves the success rate and robustness of 3D reconstruction and solves the problem of poor robustness in existing 3D reconstruction technologies.

[0005] Firstly, this application provides a three-dimensional reconstruction method, including:

[0006] A first 3D map is constructed based on the image sequence using a global reconstruction algorithm, and the first lost frame in the image sequence is identified.

[0007] Based on the first lost frame, the first 3D map is optimized using an incremental reconstruction algorithm to obtain a second 3D map, and the second lost frame in the image sequence is determined.

[0008] If it is determined that the second 3D map does not meet the preset completeness condition, a local 3D map corresponding to the second lost frame is constructed by the hole-filling algorithm, and the local 3D map is filled into the second 3D map to obtain the target 3D map.

[0009] Secondly, this application provides a three-dimensional reconstruction apparatus, comprising:

[0010] The global reconstruction module is configured to construct a first 3D map based on an image sequence using a global reconstruction algorithm, and to determine the first lost frame in the image sequence;

[0011] The incremental reconstruction module is configured to optimize the first three-dimensional map based on the first lost frame using an incremental reconstruction algorithm to obtain a second three-dimensional map, and to determine the second lost frame in the image sequence.

[0012] The hole-filling reconstruction module is configured to construct a local 3D map corresponding to the second lost frame using a hole-filling algorithm when it is determined that the second 3D map does not meet the preset integrity conditions, and then fill the local 3D map into the second 3D map to obtain the target 3D map.

[0013] Thirdly, this application provides a three-dimensional reconstruction device, comprising:

[0014] One or more processors; a storage device storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the three-dimensional reconstruction method as described in the first aspect.

[0015] Fourthly, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the three-dimensional reconstruction method as described in the first aspect.

[0016] In this application, a first 3D map is constructed based on an image sequence using a global reconstruction algorithm, and a first lost frame in the image sequence is identified. Based on the first lost frame, an incremental reconstruction algorithm is used to optimize the first 3D map to obtain a second 3D map, and a second lost frame in the image sequence is identified. If the second 3D map does not meet a preset completeness condition, a local 3D map corresponding to the second lost frame is constructed using a hole-filling algorithm, and the local 3D map is then filled into the second 3D map to obtain the target 3D map. Through these techniques, the global reconstruction algorithm, as an initialization operation, can quickly construct a 3D map and determine the shooting pose of the image frames. However, the accuracy of the 3D map constructed by the global reconstruction algorithm is relatively low. Therefore, an incremental reconstruction algorithm can be used to gradually recover the shooting pose of the lost frames in the image sequence. For each recovered shooting pose of a lost frame, the 3D map and the shooting pose are jointly optimized to improve the accuracy of the 3D map. If an image frame contains areas with severe occlusion or complex textures, and the image frame has not yet recovered its shooting pose after incremental reconstruction due to limitations in image resolution and the number of feature points, then the image frame can be upsampled using a hole-filling algorithm before feature points are extracted to increase the number of feature points in the image frame and construct a local 3D map corresponding to the image frame. This local 3D map can then be used to fill the 3D map constructed by the incremental reconstruction algorithm, resulting in a highly complete 3D map, which effectively improves the success rate and robustness of 3D reconstruction. Attached Figure Description

[0017] Figure 1 This is a flowchart of a three-dimensional reconstruction method provided in an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating the construction of a first 3D map using the global SfM algorithm provided in this application embodiment;

[0019] Figure 3 This is a flowchart of the process for filtering target feature matching pairs provided in the embodiments of this application;

[0020] Figure 4 This is a flowchart of determining the relative transformation parameters of adjacent image frames provided in an embodiment of this application;

[0021] Figure 5 This is a flowchart illustrating the determination of a second shooting pose for an image frame, as provided in an embodiment of this application.

[0022] Figure 6 This is a flowchart of determining the first shooting pose of the third image frame provided in an embodiment of this application;

[0023] Figure 7 This is a flowchart of constructing a first three-dimensional map provided in an embodiment of this application;

[0024] Figure 8 This is a flowchart of generating a second three-dimensional map provided in an embodiment of this application;

[0025] Figure 9 This is a flowchart illustrating the generation of a target 3D map based on a hole-filling algorithm provided in an embodiment of this application;

[0026] Figure 10 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application;

[0027] Figure 11 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] The 3D reconstruction method provided in this embodiment can be executed by a 3D reconstruction device, which can be implemented through software and / or hardware. The 3D reconstruction device can consist of two or more physical entities, or it can consist of a single physical entity. For example, the 3D reconstruction device can be an unmanned aerial vehicle (UAV) performing high-altitude mapping of the area to be mapped; it can also be a server wirelessly connected to the UAV. The UAV sends the image sequence of the area to be mapped to the server, and the server constructs a 3D map of the area based on the image sequence. The unmanned aerial vehicle (UAV) refers to a device such as a drone that can automatically execute tasks based on preset parameters.

[0031] The 3D reconstruction device is equipped with at least one type of operating system. Based on this operating system, the device can install at least one application. This application can be a built-in application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the 3D reconstruction device has at least one application capable of executing 3D reconstruction methods.

[0032] For ease of understanding, this embodiment uses a server as the main entity executing the 3D reconstruction method as an example for description.

[0033] In one embodiment, when the unmanned equipment travels along a pre-planned flight path, it captures a sequence of images of the area to be mapped according to a preset shooting strategy and sends the image sequence to a server. Based on each image frame in the image sequence, the server first determines the shooting pose of each image frame and constructs a 3D map of the area to be mapped using a global SfM algorithm. Then, it performs global BA (Bundle Adjustment) optimization on the 3D map. The global SfM algorithm first determines the global rotation matrix of the image frame, then determines the global translation vector of the image frame based on the global rotation matrix, and triangulates the image frame based on the global rotation matrix and global translation vector to construct a 3D point cloud of the image. The calculation of the global translation vector depends on the output of the global rotation matrix; that is, the accuracy of the global rotation matrix determines the accuracy of the 3D point cloud. However, the global rotation matrix is ​​easily affected by external point interference, leading to inaccurate calculation results. Even with global BA optimization, the error cannot be eliminated. Therefore, the global SfM algorithm has a high reconstruction failure rate and poor robustness.

[0034] To address the issue of poor robustness of the global SfM algorithm in the prior art, this embodiment provides a three-dimensional reconstruction method.

[0035] Figure 1 A flowchart of a three-dimensional reconstruction method provided in an embodiment of this application is given. (Reference) Figure 1 The three-dimensional reconstruction method specifically includes:

[0036] S110. Construct a first 3D map based on the image sequence using a global reconstruction algorithm, and determine the first lost frame in the image sequence.

[0037] In this embodiment, the global reconstruction algorithm is the global SfM algorithm, and the first 3D map refers to the 3D map of the area to be mapped constructed by the global SfM algorithm based on the image sequence of the area to be mapped. For example, the unmanned equipment photographs the area to be mapped according to a pre-planned flight path, sorts all the captured image frames according to their timestamps to generate an image sequence, and sends the image sequence to the server. The server uses the global SfM algorithm as an initialization operation and quickly reconstructs the first 3D map based on each image frame in the image sequence.

[0038] In this embodiment, Figure 2 This is a flowchart illustrating the construction of a first 3D map using the global SfM algorithm provided in this application embodiment. For example... Figure 2 As shown, the steps of the global SfM algorithm to construct the first 3D map specifically include S1101-S1104:

[0039] S1101. Extract feature points from each image frame in the image sequence and determine target feature matching pairs between adjacent image frames.

[0040] For example, suppose the image sequence is {I1, I2, ... I... i ,…,I N}, I i Let I be the i-th image frame in the image sequence, and N be the total number of image frames in the image sequence. The SIFT (Scale-invariant feature transform) algorithm is used to extract features from image frame I. i Extracting image frame I i Feature points To obtain from image frame I i The j-th feature point extracted, M is the image frame I i The total number of feature points in the image frame I. i With image frame I i+1 Perform feature matching to determine image frame I i With image frame I i+1 Feature matching pairs between Image frame I i The nth feature point in the image frame I i+1 The m-th feature point is a feature matching pair.

[0041] In this embodiment, the target feature matching pair refers to the feature matching pair subsequently used to estimate the relative transformation parameters between adjacent image frames. Since mismatches may occur when performing feature matching on adjacent image frames, the generated feature matching pairs can be filtered to obtain truly matching pairs, avoiding the introduction of outliers into the subsequent pose estimation process and improving the accuracy of 3D reconstruction. For example, Figure 3 This is a flowchart illustrating the process of filtering target feature matching pairs provided in an embodiment of this application. For example... Figure 3 As shown, the steps for filtering target feature matching pairs specifically include S11011-S11012:

[0042] S11011. Match the feature points of adjacent image frames to determine the first feature matching pair.

[0043] S11012. The mismatch pairs in the first feature matching pair are removed by the mismatch removal algorithm to obtain the target feature matching pair.

[0044] For example, suppose image frame I i With image frame I i+1 After feature matching, image frame I was determined. i With image frame I i+1 The first feature matching pair is Image frame I is constructed based on the first feature matching pair. i With image frame I i+1 The epipolar constraint graph between the two features is used to filter out mismatched pairs from the first feature matching pairs using the ACRANSAC (A Contrario RANSAC) algorithm. Assume the mismatched pairs are... Then After removing from the first feature matching pairs, the resulting target feature matching pairs are:

[0045] S1102. Determine the relative transformation parameters of adjacent image frames based on the target feature matching pairs of adjacent image frames.

[0046] For example, Figure 4 This is a flowchart illustrating the determination of relative transformation parameters between adjacent image frames, provided in an embodiment of this application. For example... Figure 4 As shown, the step of determining the relative transformation parameters of adjacent image frames specifically includes S11021-S11022:

[0047] S11021. Based on the target feature matching pairs and feature points of three consecutive image frames in the image sequence, construct a triplet of three consecutive image frames, and the three feature points in the triplet are matched with each other.

[0048] For example, from image frame I i With image frame Ii+1 Target feature matching pairs select image frames I i The first feature point, which will be the image frame I i The first feature point and image frame I i+2 Match all feature points in the image frame I to determine the match between the image frame I and the corresponding feature points. i+2 The third feature point that matches the first feature point is then compared with the first feature point in image frame I. i+1 The second feature point matched in the image, and the corresponding third feature point, constitute image frame I. i Image frame I i+1 and image frame I i+2 The triples are used to match the three feature points in each triple pairwise. For example, suppose image frame I... i With image frame I i+1 Target feature matching pairs include The first feature point obtained includes If the first feature point With image frame I i+2 Feature points in If they match, then As the first feature point The third feature point is obtained, and the first feature point is obtained from the target feature matching pair. The corresponding second feature point This leads to the triplet.

[0049] S11022. When the number of triplets in three consecutive image frames is greater than or equal to a preset threshold, the relative transformation parameters of the corresponding adjacent image frames are determined based on the triplets of the three consecutive image frames.

[0050] In this embodiment, the preset quantity threshold can be understood as the minimum number of triplets corresponding to pairwise matching of three consecutive image frames. For example, when image frame I... i Image frame I i+1 and image frame I i+2 When the number of triples is greater than or equal to a preset threshold, it indicates that image frame I... i Image frame I i+1 and image frame I i+2 The matching constraints between them are relatively large, and the accuracy of the relative transformation parameters determined based on this triplet is high. At this point, image frame I can be determined based on the first feature point and the corresponding second feature point in the triplet. i and image frame I i+1 The relative transformation parameters between them are used to determine image frame I based on the second feature point and the corresponding third feature point in the triplet. i+1 and image frame I i+2 The relative transformation parameters between them are used to subsequently transform the image frame I.i and image frame I i+1 The first shooting pose and the relative transformation parameters are used to recover image frame I. i+2 The first shooting pose.

[0051] In this embodiment, after determining image frame I i and image frame I i+1 The relative transformation parameters and image frame I between them i+1 and image frame I i+2 After determining the relative transformation parameters between the two parameters, the corresponding rotation matrix is ​​extracted from these two relative transformation parameters, and the image frame I is determined based on the rotation matrix. i and image frame I i+1 The rotation angle between them, and image frame I i+1 and image frame I i+2 The rotation angle between adjacent image frames. Since the rotation angle between adjacent image frames is not very large, if the sum of these two rotation angles exceeds 2°, it indicates that the currently calculated relative transformation parameters are not accurate enough, and subsequent image frame I cannot be accurately recovered based on the relative transformation parameters. i+2 The first shooting pose. At this time, image frame I can be captured. i+2 Mark it as the first lost frame in the image sequence, and obtain image frame I from the image sequence. i+3 To construct image frame I i Image frame I i+1 and image frame I i+3 The triplet.

[0052] When image frame I i Image frame I i+1 and image frame I i+2 When the number of triples is greater than or equal to a preset threshold, it indicates that image frame I... i Image frame I i+1 and image frame I i+2 The matching constraints between the pairs are relatively small, resulting in a large error in the relative transformation parameters determined based on these triples. Consequently, image frame I cannot be accurately recovered based on these relative transformation parameters. i+2 The first shooting pose. At this time, image frame I can be captured. i+2 Mark it as the first lost frame in the image sequence, and obtain image frame I from the image sequence. i+3 To construct image frame I i Image frame I i+1 and image frame I i+3 The triplet is used to determine the image frame I. i With image frame I i+1 The relative transformation parameters and image frame I i+1 With image frame I i+3 The relative transformation parameters.

[0053] It should be noted that the adjacent or consecutive image frames mentioned in this embodiment are not necessarily sequentially adjacent or consecutive image frames in the image sequence, but rather refer to sequentially adjacent or consecutive image frames among the remaining image frames after excluding lost frames. For example, image frame I... i+2 After the first lost frame, image frame I i+1 With image frame I i+3 They became adjacent image frames.

[0054] S1103. Determine the second shooting pose of each image frame based on the relative transformation parameters and the first shooting pose of two adjacent image frames.

[0055] In this embodiment, the first shooting pose of two adjacent image frames can be understood as the pre-determined shooting pose of two adjacent image frames in an image sequence. For example, the shooting poses of two adjacent shooting points on a flight path are pre-set. When the unmanned device (UAV) travels to these two shooting points, it captures two adjacent image frames using the corresponding pre-set shooting poses. The UAV can determine the first shooting pose of the corresponding image frame based on the position of the shooting point and the corresponding shooting pose, and associate the image frame with the corresponding first shooting pose to the server, so that the first shooting poses of two adjacent image frames in the image sequence obtained by the server are pre-determined by the UAV. For example, if the shooting poses of the first and second shooting points on the flight path are pre-set, then when the UAV travels to the first shooting point, it captures image frame I1 according to the shooting pose set for the first shooting point, and when the UAV travels to the second shooting point, it captures image frame I2 according to the shooting pose set for the second shooting point. The UAV determines the first shooting pose of image frame I1 based on the position and shooting pose of the first shooting point, and associates and saves image frame I1 with the corresponding first shooting pose. The unmanned device determines the first shooting pose of image frame I2 based on the position and shooting posture of the second shooting point, and saves image frame I2 in association with the corresponding first shooting pose. After the unmanned device sends the image sequence to the server, the server obtains the image sequence and the first shooting poses of image frames I1 and I2. The server can determine the first shooting pose of each image frame based on the first shooting poses of image frames I1 and I2, as well as the essential matrix between each adjacent image frame in the image sequence.

[0056] It should be noted that the two adjacent image frames with the first shooting pose predetermined in the image sequence can be any two adjacent image frames. In this embodiment, image frame I1 and image frame I2 are used as examples for description.

[0057] In this embodiment, the global SfM algorithm first recovers the first shooting pose of each image frame based on target feature matching pairs of adjacent image frames. Then, it performs global joint optimization on the first shooting poses of each corresponding image frame to obtain the second shooting pose of the image frame with higher accuracy. For example, Figure 5 This is a flowchart illustrating the process of determining the second shooting pose of an image frame, as provided in an embodiment of this application. Figure 5 As shown, the step of determining the second shooting pose of the image frame specifically includes S11031-S11032:

[0058] S11031. Determine the first shooting pose of the third image frame based on the first shooting pose of the first image frame and the second image frame in the three consecutive image frames, and the relative transformation parameters between the second image frame and the third image frame in the three consecutive image frames.

[0059] In this embodiment, the first image frame and the second image frame are image frames whose first shooting pose has been recovered from three consecutive image frames, and the third image frame is image frames whose first shooting pose has not been recovered from three consecutive image frames. For example, when the first shooting poses of image frames I1 and I2 are predetermined, a triplet of image frames I1, I2, and I3 is constructed. Based on this triplet, it is determined whether image frame I3 is the first lost frame. If so, image frame I4 is obtained from the image sequence; otherwise, the first shooting pose of image frame I3 is determined based on the first shooting poses of image frames I1 and I2 and the relative transformation parameters between image frames I2 and I3. After determining the first shooting pose of image frame I3, image frame I4 is obtained from the image sequence, and a triplet of image frames I2, I3, and I4 is constructed. This process is repeated to determine the first lost frame in the image sequence and to recover the first shooting poses of the remaining image frames.

[0060] In this embodiment, Figure 6 This is a flowchart illustrating the determination of the first shooting pose of the third image frame, provided in an embodiment of this application. Figure 6 As shown, the step of determining the first shooting pose of the third image frame specifically includes S110311-S110313:

[0061] S110311. Based on the first shooting pose of the first image frame and the second image frame, the target feature matching pair of the first image frame and the second image frame is triangulated to generate a first three-dimensional point.

[0062] This embodiment describes the example where image frame I3 is not the first lost frame. For example, the target feature matching pair of image frame I1 and image frame I2 is obtained from all triples of image frame I1, image frame I2 and image frame I3, and the target feature matching pair is triangulated to construct the first three-dimensional point corresponding to the target feature matching pair in the world coordinate system.

[0063] S110312. Multiply the relative transformation parameter between the second image frame and the third image frame by the first shooting pose of the second image frame, and project the first three-dimensional point into the third image frame through the product to obtain the two-dimensional point of the first three-dimensional point in the third image frame.

[0064] For example, the relative transformation parameters between image frames I2 and I3 are multiplied by the first shooting pose of the second image frame, and the calculated product can be regarded as the shooting pose of image frame I3. The first three-dimensional point is projected onto the pixel coordinate system of image frame I3 using this shooting pose and camera intrinsic parameters, thus constructing a two-dimensional point of the first three-dimensional point in image frame I3.

[0065] S110313. Construct a reprojection residual equation based on the two-dimensional points and feature points in the third image frame, solve the reprojection residual equation, and obtain the first shooting pose of the third image frame.

[0066] For example, feature points of image frame I3 are obtained from all triples of image frames I1, I2, and I3. Based on the target feature matching pairs corresponding to the two-dimensional points, the feature points of image frame I3 that belong to the same triplet as the target feature matching pair are subtracted from the two-dimensional points. Based on the differences between each two-dimensional point and its corresponding feature point, a first-fold projection residual equation is constructed using a least-squares algorithm. Solving the first-fold projection residual equation yields the first rotation matrix of image frame I3. Based on the first rotation matrix, a first three-dimensional point is reprojected onto a two-dimensional point in image frame I3. Based on the difference between this two-dimensional point and its corresponding feature point in image frame I3, a second-fold projection residual equation is constructed using a least-squares algorithm. Solving the second-fold projection residual equation yields the first translation matrix of image frame I3. The first shooting pose of image frame I3 can be determined based on the first rotation matrix and the first translation matrix.

[0067] S11032. The first shooting pose of each image frame is globally optimized by the bundle adjustment algorithm to obtain the second shooting pose of each image frame.

[0068] For example, after determining the first shooting pose of the remaining image frames in the image sequence except for the first lost frame, global BA optimization is performed by combining the first three-dimensional points constructed from each adjacent image frame with the first shooting pose to obtain a second shooting pose with higher accuracy for the image frames.

[0069] S1104. Construct a first three-dimensional map based on the feature points of the image frame and the second shooting pose.

[0070] For example, all the first 3D points optimized by global BA can be used to form a first 3D map of the area to be mapped. However, because AC RANSAC filtering is performed on the first feature matching pairs of adjacent image frames during feature matching, the number of all first 3D points is relatively small, resulting in a low density of the generated first 3D map. Therefore, a dense first 3D map can be constructed based on the first feature matching pairs of adjacent image frames and the second shooting pose of the image frames. In this embodiment, Figure 7 This is a flowchart illustrating the construction of a first three-dimensional map provided in an embodiment of this application. For example... Figure 7 As shown, the steps for constructing the first 3D map specifically include S11041-S11043:

[0071] S11041. Assign two sets of first feature matching pairs containing the same feature points to the same feature point set.

[0072] For example, suppose image frame I i With image frame I i+1 The first feature matching pair is Image Frame I i+1 With image frame I i+2 The first feature matching pair is Then feature points and They are grouped into the same set of feature points.

[0073] S11042. Triangulate the feature point set according to the second shooting pose of the image frame to generate the first three-dimensional point cloud.

[0074] For example, suppose a set of feature points includes feature points and According to image frame I i Image frame I i+1 and image frame I i+1 The second shooting pose, for feature points and Triangulation is performed to construct feature points. and Three-dimensional points in the world coordinate system. Similarly, by combining the three-dimensional points constructed from each set of feature points, the first three-dimensional point cloud is obtained.

[0075] S11043. The second shooting pose of the first 3D point cloud and the image frame is globally optimized by the bundle adjustment algorithm to obtain the third shooting pose of the first 3D map and the image frame.

[0076] For example, global basis optimization (BA) is performed by combining the first 3D point cloud and the second shooting poses of each image frame to obtain the third shooting pose of the first 3D map and the image frames. Compared with the first 3D point cloud and the second shooting pose, the first 3D map and the third shooting pose have higher accuracy.

[0077] S120. Based on the first lost frame, the first three-dimensional map is optimized using an incremental reconstruction algorithm to obtain the second three-dimensional map, and the second lost frame in the image sequence is determined.

[0078] Since some first-frame loss occurs in the image sequence and the shooting pose has not been recovered, and the accuracy of the first 3D map constructed by the global SfM algorithm is low, this embodiment proposes to add the first-frame loss to the first 3D map one by one based on an incremental reconstruction algorithm to generate a more accurate second 3D map. The incremental reconstruction algorithm is the incremental SfM algorithm.

[0079] In this embodiment, Figure 8 This is a flowchart of generating a second three-dimensional map provided in an embodiment of this application.

[0080] like Figure 8 As shown, the steps for generating the second 3D map specifically include S1201-S1204:

[0081] S1201. Determine the matching frame of the first lost frame from the recovered frames of the image sequence, and determine the third shooting pose of the first lost frame based on the third shooting pose of the matching frame.

[0082] In this embodiment, the recovered frame is the image frame in the image sequence whose third shooting pose has been recovered. The first lost frame is matched with each recovered frame in the image sequence to determine the number of target feature matching pairs between the first lost frame and each recovered frame. The recovered frame with the largest number of target feature matching pairs is taken as the matching frame of the first lost frame. Based on the target feature matching pairs between the first lost frame and the corresponding matching frame, the relative transformation parameter between the first lost frame and the corresponding matching frame is determined. This relative transformation parameter is then multiplied by the third shooting pose of the matching frame to recover the third shooting pose of the first lost frame.

[0083] It should be noted that if the number of target feature matching pairs between the first lost frame and each recovered frame is less than the preset number threshold, it indicates that the matching constraint of the recovered frame on the first lost frame is low, that is, the shooting pose of the first lost frame cannot be recovered based on the target feature matching pairs. In this case, the first lost frame can be marked as the second lost frame in the image sequence.

[0084] S1202. Based on the third shooting pose of the first lost frame and the matching frame, the target feature matching pair of the first lost frame and the matching frame is triangulated to generate a second three-dimensional point.

[0085] For example, based on the third shooting pose of the first lost frame and the matching frame, the target feature matching pair of the first lost frame and the matching frame is triangulated to construct the three-dimensional points of the feature points of the target feature matching pair in the world coordinate system.

[0086] S1203. Add the second three-dimensional point cloud to the first three-dimensional map, and optimize the first three-dimensional map with the added second three-dimensional point cloud using the bundle adjustment algorithm.

[0087] For example, after adding the second 3D point to the first 3D map, global BA optimization is performed on the first 3D map with the added second 3D point cloud, as well as the third shooting pose of the first lost frame and all recovered frames. Further, the optimized first lost frame is marked as a recovered frame, and the next first lost frame is obtained. The matching frame of the first lost frame is determined from all recovered frames, and steps S1201-S1203 are repeated until the second 3D point added to the last first lost frame is optimized.

[0088] S1204. After optimizing the second three-dimensional point cloud of the last first lost frame, the first three-dimensional map with each second three-dimensional point added is globally optimized by the bundle adjustment algorithm to obtain the second three-dimensional map.

[0089] For example, global BA optimization is performed on the first 3D map with all second 3D points added, the optimized third shooting pose of all restored frames, and the camera intrinsic parameters to construct a least squares function. Solving the least squares function yields the second 3D map and the fourth shooting pose of each restored frame.

[0090] In this embodiment, after constructing the first 3D map using the global SfM algorithm, an incremental SfM algorithm is used to gradually add the 3D scene of the first lost frame to the first 3D map, thereby constructing a second 3D map with higher accuracy. Moreover, since the number of first lost frames is small, even though the incremental SfM algorithm requires a global BA optimization after constructing the 3D scene of each first lost frame, the time required to generate the second 3D map is much less than that of directly using the incremental SfM algorithm, ensuring high-precision map construction while improving map construction efficiency.

[0091] S130. If it is determined that the second three-dimensional map does not meet the preset completeness condition, a local three-dimensional map corresponding to the second lost frame is constructed by the hole-filling algorithm, and the local three-dimensional map is filled into the second three-dimensional map to obtain the target three-dimensional map.

[0092] In this embodiment, the preset completeness condition refers to the condition that the second 3D map must meet to completely describe the 3D scene of the area to be surveyed. For example, when the second 3D map meets the preset completeness condition, it is determined that the second 3D map construction is successful, and thus the second 3D map is used as the target 3D map of the area to be surveyed. When the second 3D map does not meet the preset completeness condition, it is determined that the second 3D map construction has failed, and hole-filling operations are performed on the empty areas in the second 3D map to complete the empty areas and obtain the target 3D map of the area to be surveyed. For example, the higher the ratio of the number of recovered frames in the second 3D map to the number of image frames in the image sequence, the higher the completeness of the second 3D map. Therefore, this embodiment sets a preset ratio threshold as the minimum ratio at which the second 3D map can completely describe the 3D scene of the area to be surveyed. If the ratio of the number of recovered frames in the second 3D map to the number of image frames in the image sequence is less than or equal to the preset ratio threshold, it is determined that the second 3D map does not meet the preset completeness condition; if the ratio of the number of recovered frames in the second 3D map to the number of image frames in the image sequence is greater than the preset ratio threshold, it is determined that the second 3D map meets the preset completeness condition.

[0093] In one embodiment, Figure 9 This is a flowchart illustrating the generation of a target 3D map based on a hole-filling algorithm, as provided in an embodiment of this application. Figure 9 As shown, the steps for generating a target 3D map based on the hole-filling algorithm specifically include S1301-S1303:

[0094] S1301. Based on the distance between the second lost frame and the recovered frame, add the lost frame and the recovered frame in the corresponding neighborhood to the group to be solved.

[0095] For example, when the unmanned device captures an image frame, its RTK positioning system determines the device's geographic coordinates and uses these coordinates as the geographic coordinates of the currently captured image frame. The coordinate distances between the second lost frame and each recovered frame can be determined based on the geographic coordinates of the second lost frame and each recovered frame. Based on these coordinate distances, the recovered frames are sorted in ascending order of their corresponding coordinate distances, and the first preset number of recovered frames at the top of the sorted order are considered as recovered frames within the corresponding neighborhood of the second lost frame. All second lost frames and all recovered frames within their corresponding neighborhoods are then added to a group to be solved.

[0096] It should be noted that, since there may be a correlation between the second lost frames and the second lost frames, this embodiment will add all the second lost frames and the corresponding recovered frames in the neighborhood to a group to be solved, so as to completely search out the correlation between each image frame from the group to be solved.

[0097] S1302. Determine the matching results between each image frame in the group to be solved, and classify the two matching image frames into the same connected group according to the matching results.

[0098] For example, any image frame in the group to be solved is designated as the fourth image frame, and any image frame other than the fourth image frame is designated as the fifth image frame. Based on the geographic coordinates of the fourth image frame and each of the fifth image frames, the coordinate distances between the fourth image frame and each of the fifth image frames are calculated. The fifth image frames are then sorted in ascending order of their coordinate distances to the fourth image frame, so that the earlier the fifth image frame is, the closer it is to the fourth image frame. If the fourth image frame is a second lost frame, a second preset number of second lost frames and a second preset number of recovered frames are selected from the sequence of fifth image frames, starting from the beginning, to form an image pair with the fourth image frame. If the fourth image frame is a recovered frame, a third preset number of fifth image frames are selected from the sequence of fifth image frames, starting from the beginning, to form an image pair with the fourth image frame. The third preset number is twice the second preset number.

[0099] Each image pair is upsampled before feature points are extracted to obtain more effective feature points, increase the number of feature matching pairs, and deeply explore the correlation between two image frames. Based on the feature points of the two image frames of an image pair, feature matching is performed on the two image frames to determine the number of target feature matching pairs for the image pair. If the number of target feature matching pairs for the image pair is greater than a preset first threshold, the two image frames of the image pair are determined to be matched; otherwise, the two image frames are determined not to be matched. The first threshold is the minimum number of feature matching pairs required for two image frames to be matched, as set in this embodiment. It can be understood that if the number of feature matching pairs for two image frames exceeds the preset threshold, it indicates that the two image frames highly overlap, and the shooting pose of one image frame imposes a significant constraint on the shooting pose of the other image frame. Therefore, it can be determined that there is a correlation between the two matched image frames.

[0100] Iterate through the matching results of each image pair, and group the two matching image frames into the same connected group. For example, if image frame I... i With image frame I j Matching, image frame I j With image frame I k Matching, even if image frame I i With image frame I k Mismatch will also result in image frame I i Image frame I j and image frame I kThe images are stored in the same connected group. In this case, 3D reconstruction based on the image frames in the connected group can construct a local 3D map that is associated with the second 3D map and covers the hole areas. This allows the local 3D map to be aligned to the second 3D map based on the relationship between the local 3D map and the second 3D map, thereby filling in the hole areas in the second 3D map.

[0101] S1303. Perform 3D reconstruction based on the image frames in the connected group to obtain a local 3D map. Align the local 3D map with the second 3D map to obtain the target 3D map.

[0102] For example, the fourth shooting pose of the recovered frame, which forms an image pair with the second lost frame in the connected group, is determined. Based on the target feature matching pairs of each image pair in the connected group, two sets of target feature matching pairs containing the same feature points are grouped into the same feature point set. The feature point set is triangulated based on the fourth shooting poses of each image frame in the connected group to generate a second 3D point cloud. Global basis adjustment (BA) is then performed on the second 3D point cloud and the fourth shooting poses of each image frame in the connected group using a bundle adjustment algorithm to obtain a local 3D map and the fifth shooting poses of each image frame.

[0103] Obtain the fourth and fifth shooting poses of each recovered frame in the connected group. The fourth shooting pose can be considered as the shooting pose of the recovered frame in the coordinate system of the second 3D map, and the fifth shooting pose can be considered as the shooting pose of the recovered frame in the coordinate system of the local 3D map. Determine the relative transformation parameters between the fourth and corresponding fifth shooting poses. Use these parameters to transform the local 3D map to the coordinate system of the second 3D map, and also transform the second lost frame in the connected group to the coordinate system of the second 3D map, to recover the fourth shooting pose of the second lost frame and obtain the local 3D map in the coordinate system of the second 3D map. Combine the local 3D map in the same coordinate system with the second 3D map to obtain a complete target 3D map of the area to be mapped, completing the hole-filling operation of the overall 3D map.

[0104] In summary, the 3D reconstruction method provided in this application constructs a first 3D map based on an image sequence using a global reconstruction algorithm and identifies the first lost frame in the image sequence. Based on the first lost frame, an incremental reconstruction algorithm is used to optimize the first 3D map to obtain a second 3D map, and the second lost frame in the image sequence is identified. If the second 3D map does not meet the preset completeness condition, a local 3D map corresponding to the second lost frame is constructed using a hole-filling algorithm, and the local 3D map is filled into the second 3D map to obtain the target 3D map. Through the above technical means, the global reconstruction algorithm can quickly construct a 3D map and determine the shooting pose of the image frame as an initialization operation. However, the accuracy of the 3D map constructed by the global reconstruction algorithm is relatively low. An incremental reconstruction algorithm can be used to gradually recover the shooting pose of the lost frames in the image sequence. For each recovered shooting pose of a lost frame, the 3D map and the shooting pose are jointly optimized to improve the accuracy of the 3D map. If an image frame contains areas with severe occlusion or complex textures, and the image frame has not yet recovered its shooting pose after incremental reconstruction due to limitations in image resolution and the number of feature points, then the image frame can be upsampled using a hole-filling algorithm before feature points are extracted to increase the number of feature points in the image frame and construct a local 3D map corresponding to the image frame. The local 3D map is then filled into the 3D map constructed by the incremental reconstruction algorithm to obtain a highly complete 3D map, which effectively improves the success rate and robustness of 3D reconstruction.

[0105] Based on the above embodiments, Figure 10 This is a schematic diagram of a three-dimensional reconstruction device provided in an embodiment of this application. (Reference) Figure 10 The three-dimensional reconstruction device provided in this embodiment specifically includes: a global reconstruction module 21, an incremental reconstruction module 22, and a hole-filling reconstruction module 23.

[0106] The global reconstruction module is configured to construct a first 3D map based on the image sequence using a global reconstruction algorithm, and to determine the first lost frame in the image sequence.

[0107] The incremental reconstruction module is configured to optimize the first 3D map based on the first lost frame using an incremental reconstruction algorithm to obtain the second 3D map and determine the second lost frame in the image sequence.

[0108] The hole-filling and reconstruction module is configured to construct a local 3D map corresponding to the second lost frame using a hole-filling algorithm when it is determined that the second 3D map does not meet the preset integrity conditions, and then fill the local 3D map into the second 3D map to obtain the target 3D map.

[0109] Based on the above embodiments, the global reconstruction module includes: a feature matching submodule, configured to extract feature points of each image frame in the image sequence and determine target feature matching pairs of adjacent image frames; a relative transformation determination submodule, configured to determine relative transformation parameters of adjacent image frames based on the target feature matching pairs of adjacent image frames; a second pose determination submodule, configured to determine the second shooting pose of each image frame based on the relative transformation parameters and the first shooting pose of two adjacent image frames; and a first map construction submodule, configured to construct a first three-dimensional map based on the feature points and the second shooting pose of the image frames.

[0110] Based on the above embodiments, the feature matching submodule includes: a feature matching unit configured to match feature points of adjacent image frames to determine a first feature matching pair; and a mismatch elimination unit configured to eliminate mismatch pairs in the first feature matching pair using a mismatch elimination algorithm to obtain a target feature matching pair.

[0111] Based on the above embodiments, the relative transformation determination submodule includes: a triplet construction unit, configured to construct a triplet of three consecutive image frames based on target feature matching pairs and feature points in three consecutive image frames in the image sequence, wherein the three feature points in the triplet are mutually matched; and a relative transformation determination unit, configured to determine the relative transformation parameters of corresponding adjacent image frames based on the triplet of three consecutive image frames when the number of triplets in the three consecutive image frames is greater than or equal to a preset number threshold.

[0112] Based on the above embodiments, the second pose determination submodule includes: a first pose determination unit configured to determine the first shooting pose of the third image frame based on the first shooting pose of the first image frame and the second image frame in three consecutive image frames, and the relative transformation parameters between the second image frame and the third image frame in three consecutive image frames; and a second pose determination unit configured to perform global optimization of the first shooting pose of each image frame through a bundle adjustment algorithm to obtain the second shooting pose of each image frame.

[0113] Based on the above embodiments, the first pose determination unit includes: a first three-dimensional point generation subunit, configured to perform triangulation processing on the target feature matching pair of the first image frame and the second image frame according to the first shooting pose of the first image frame and the second image frame to generate a first three-dimensional point; a three-dimensional point projection subunit, configured to multiply the relative transformation parameter between the second image frame and the third image frame by the first shooting pose of the second image frame, and project the first three-dimensional point into the third image frame through the product to obtain a two-dimensional point of the first three-dimensional point in the third image frame; and the first pose determination subunit is configured to construct a reprojection residual equation based on the two-dimensional point and feature points in the third image frame, solve the reprojection residual equation, and obtain the first shooting pose of the third image frame.

[0114] Based on the above embodiments, the first map construction submodule includes: a feature point set partitioning unit, configured to partition two sets of first feature matching pairs containing the same feature points into the same feature point set; a three-dimensional point cloud generation unit, configured to perform triangulation processing on the feature point set according to the second shooting pose of the image frame to generate a first three-dimensional point cloud; and a first map generation unit, configured to perform global optimization on the first three-dimensional point cloud and the second shooting pose of the image frame through a bundle adjustment algorithm to obtain a first three-dimensional map and a third shooting pose of the image frame.

[0115] Based on the above embodiments, the incremental reconstruction module includes: a third pose determination submodule, configured to determine the matching frame of the first lost frame from the recovered frames of the image sequence, and determine the third shooting pose of the first lost frame according to the third shooting pose of the matching frame; a second 3D point generation submodule, configured to perform triangulation processing on the target feature matching pair of the first lost frame and the matching frame according to the third shooting pose of the first lost frame and the matching frame, and generate second 3D points; an optimization submodule, configured to add the second 3D points to the first 3D map, and optimize the first 3D map with the added second 3D point cloud through a bundle adjustment algorithm; and a second map generation submodule, configured to optimize the second 3D point cloud of the last first lost frame, and then perform global optimization on the first 3D map with each added second 3D point through a bundle adjustment algorithm to obtain the second 3D map.

[0116] Based on the above embodiments, the hole-filling reconstruction module includes: a group to be solved submodule, configured to add the lost frame and the corresponding recovered frame in the neighborhood to the group to be solved according to the distance between the second lost frame and the recovered frame; a connected group generation submodule, configured to determine the matching result between each image frame in the group to be solved, and divide two matching image frames into the same connected group according to the matching result; and a target 3D map generation submodule, configured to perform 3D reconstruction based on the image frames in the connected group to obtain a local 3D map, and align the local 3D map with the second 3D map to obtain the target 3D map.

[0117] The 3D reconstruction apparatus provided in this application, as described above, constructs a first 3D map based on an image sequence using a global reconstruction algorithm and identifies a first lost frame in the image sequence. Based on the first lost frame, it optimizes the first 3D map using an incremental reconstruction algorithm to obtain a second 3D map and identifies a second lost frame in the image sequence. If the second 3D map does not meet a preset completeness condition, a local 3D map corresponding to the second lost frame is constructed using a hole-filling algorithm, and the local 3D map is filled into the second 3D map to obtain the target 3D map. Through the above technical means, the global reconstruction algorithm, as an initialization operation, can quickly construct a 3D map and determine the shooting pose of the image frame. However, the accuracy of the 3D map constructed by the global reconstruction algorithm is relatively low. Therefore, an incremental reconstruction algorithm can be used to gradually recover the shooting pose of the lost frames in the image sequence. For each recovered shooting pose of a lost frame, the 3D map and the shooting pose are jointly optimized to improve the accuracy of the 3D map. If an image frame contains areas with severe occlusion or complex textures, and the image frame has not yet recovered its shooting pose after incremental reconstruction due to limitations in image resolution and the number of feature points, then the image frame can be upsampled using a hole-filling algorithm before feature points are extracted to increase the number of feature points in the image frame and construct a local 3D map corresponding to the image frame. The local 3D map is then filled into the 3D map constructed by the incremental reconstruction algorithm to obtain a highly complete 3D map, which effectively improves the success rate and robustness of 3D reconstruction.

[0118] The three-dimensional reconstruction device provided in this application embodiment can be used to execute the three-dimensional reconstruction method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0119] Figure 11 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application, with reference to... Figure 11 The 3D reconstruction device includes a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 and the number of memories 32 in the 3D reconstruction device can be one or more. The processor 31, memory 32, communication device 33, input device 34, and output device 35 of the 3D reconstruction device can be connected via a bus or other means.

[0120] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the three-dimensional reconstruction method in any embodiment of this application (e.g., the global reconstruction module 21, incremental reconstruction module 22, and hole-filling reconstruction module 23 in the three-dimensional reconstruction device). The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 32 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] The communication device 33 is used for data transmission.

[0122] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 32, thereby realizing the above-mentioned three-dimensional reconstruction method.

[0123] Input device 34 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 35 may include display devices such as a display screen.

[0124] The three-dimensional reconstruction equipment provided above can be used to perform the three-dimensional reconstruction method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0125] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a three-dimensional reconstruction method. The three-dimensional reconstruction method includes: constructing a first three-dimensional map based on an image sequence using a global reconstruction algorithm, and identifying a first lost frame in the image sequence; optimizing the first three-dimensional map based on the first lost frame using an incremental reconstruction algorithm to obtain a second three-dimensional map, and identifying a second lost frame in the image sequence; and, if it is determined that the second three-dimensional map does not meet a preset integrity condition, constructing a local three-dimensional map corresponding to the second lost frame using a hole-filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain a target three-dimensional map.

[0126] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0127] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the three-dimensional reconstruction method described above, but can also perform related operations in the three-dimensional reconstruction method provided in any embodiment of this application.

[0128] The three-dimensional reconstruction apparatus, storage medium, and three-dimensional reconstruction device provided in the above embodiments can execute the three-dimensional reconstruction method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the three-dimensional reconstruction method provided in any embodiment of this application.

[0129] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: A first 3D map is constructed based on the image sequence using a global reconstruction algorithm, and the first lost frame in the image sequence is identified. The first lost frame is added to the first three-dimensional map one by one based on the incremental reconstruction algorithm to construct the second three-dimensional map, and the first lost frame that is not successfully added to the first three-dimensional map is determined as the second lost frame in the image sequence; If it is determined that the second 3D map does not meet the preset completeness condition, the second lost frame and the corresponding recovered frame are added to the group to be solved according to the distance between the second lost frame and the recovered frame. The matching results between each image frame in the group to be solved are determined, and two matching image frames are assigned to the same connected group according to the matching results; three-dimensional reconstruction is performed on the image frames in the connected group to obtain a local three-dimensional map, and the local three-dimensional map is aligned with the second three-dimensional map to obtain the target three-dimensional map.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, The construction of the first 3D map based on image sequences using a global reconstruction algorithm includes: Extract feature points from each image frame in the image sequence and determine target feature matching pairs between adjacent image frames; Based on the target feature matching pairs of the adjacent image frames, determine the relative transformation parameters of the adjacent image frames; The second shooting pose of each image frame is determined based on the relative transformation parameters and the first shooting pose of two adjacent image frames; The first three-dimensional map is constructed based on the feature points of the image frame and the second shooting pose.

3. The three-dimensional reconstruction method according to claim 2, characterized in that, The step of extracting feature points from each image frame in the image sequence and determining target feature matching pairs between adjacent image frames includes: The feature points of adjacent image frames are matched to determine the first feature matching pair; The mismatched pairs in the first feature matching pair are removed by a mismatch removal algorithm to obtain the target feature matching pair.

4. The three-dimensional reconstruction method according to claim 2, characterized in that, The step of determining the relative transformation parameters of adjacent image frames based on the target feature matching pairs of adjacent image frames includes: Based on the target feature matching pairs and feature points of three consecutive image frames in the image sequence, a triplet of three consecutive image frames is constructed, wherein the three feature points in the triplet are matched with each other. If the number of triplets in the three consecutive image frames is greater than or equal to a preset threshold, the relative transformation parameters of the corresponding adjacent image frames are determined based on the triplets in the three consecutive image frames.

5. The three-dimensional reconstruction method according to claim 2, characterized in that, Determining the second shooting pose of each image frame based on the relative transformation parameters and the first shooting poses of two adjacent image frames includes: The first shooting pose of the third image frame is determined based on the first shooting pose of the first and second image frames in three consecutive image frames, and the relative transformation parameters between the second image frame and the third image frame in the three consecutive image frames. The first shooting pose of each image frame is globally optimized by the bundle adjustment algorithm to obtain the second shooting pose of each image frame.

6. The three-dimensional reconstruction method according to claim 5, characterized in that, The step of determining the first shooting pose of the third image frame based on the first shooting pose of the first and second image frames in three consecutive image frames, and the relative transformation parameters between the second image frame and the third image frame in the three consecutive image frames, includes: Based on the first shooting pose of the first image frame and the second image frame, the target feature matching pair of the first image frame and the second image frame is triangulated to generate a first three-dimensional point. Multiply the relative transformation parameter between the second image frame and the third image frame by the first shooting pose of the second image frame, and project the first three-dimensional point into the third image frame through the product to obtain the two-dimensional point of the first three-dimensional point in the third image frame. Based on the two-dimensional points and feature points in the third image frame, a reprojection residual equation is constructed, and the reprojection residual equation is solved to obtain the first shooting pose of the third image frame.

7. The three-dimensional reconstruction method according to claim 3, characterized in that, The step of constructing the first three-dimensional map based on the feature points of the image frame and the second shooting pose includes: Two sets of first feature matching pairs containing the same feature points are grouped into the same feature point set; Based on the second shooting pose of the image frame, the feature point set is triangulated to generate a first three-dimensional point cloud. The first 3D point cloud and the second shooting pose of the image frame are globally optimized by the bundle adjustment algorithm to obtain the third shooting pose of the first 3D map and the image frame.

8. The three-dimensional reconstruction method according to claim 1, characterized in that, The incremental reconstruction algorithm is used to add the first lost frames one by one to the first 3D map to construct the second 3D map, including: The matching frame of the first lost frame is determined from the recovered frames of the image sequence, and the third shooting pose of the first lost frame is determined based on the third shooting pose of the matching frame. Based on the third shooting pose of the first lost frame and the matching frame, the target feature matching pair of the first lost frame and the matching frame is triangulated to generate a second three-dimensional point. The second 3D point is added to the first 3D map, and the first 3D map with the added second 3D point cloud is optimized using a bundle adjustment algorithm; After optimizing the second 3D point cloud of the last first lost frame, the first 3D map with each second 3D point added is globally optimized using the bundle adjustment algorithm to obtain the second 3D map.

9. A three-dimensional reconstruction device, characterized in that, include: The global reconstruction module is configured to construct a first 3D map based on an image sequence using a global reconstruction algorithm, and to determine the first lost frame in the image sequence; The incremental reconstruction module is configured to add the first lost frame to the first three-dimensional map one by one based on the incremental reconstruction algorithm to construct the second three-dimensional map, and to determine the first lost frame that was not successfully added to the first three-dimensional map as the second lost frame in the image sequence. The hole-filling and reconstruction module is configured to, when it is determined that the second 3D map does not meet the preset integrity conditions, add the second lost frame and the corresponding recovered frame in the neighborhood to the group to be solved based on the distance between the second lost frame and the recovered frame; The matching results between each image frame in the group to be solved are determined, and two matching image frames are assigned to the same connected group according to the matching results; three-dimensional reconstruction is performed on the image frames in the connected group to obtain a local three-dimensional map, and the local three-dimensional map is aligned with the second three-dimensional map to obtain the target three-dimensional map.

10. A three-dimensional reconstruction device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the three-dimensional reconstruction method as described in any one of claims 1-8.

11. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the three-dimensional reconstruction method as described in any one of claims 1-8.

Images