A 3D reconstruction method, device and medium for incremental sub-scene merging

Abstract

The present invention relates to the field of aerial surveying, and discloses a three-dimensional reconstruction method, device and medium for incremental sub-scene merging. The method comprises obtaining an initial overall scene map of a drone aerial survey image. G , and divide it into sub-graphs of different sub-scenes G s ; The sub-scenes are expanded using a maximum clique to increase the number of common connection points in the sub-scenes to obtain the expanded sub-scenes; for the expanded sub-scenes, the consistency of the optimized gradient direction of the overlapping image camera parameters is calculated from the perspective of the optimized gradient direction of the common overlapping image camera parameters between different sub-scenes; incremental sub-scenes are merged according to the consistency of the optimized gradient direction to obtain the final real-scene three-dimensional model; the present invention realizes high-precision merging of sub-scenes, can reduce the cumulative error occurring in the sub-scene merging process, ensures the accuracy of the final merged model, and provides high-precision data support for the three-dimensional reconstruction of drone images and the application of real-scene three-dimensional models.

Description

Technical Field

[0001] The present invention relates to the field of aerial surveying, and in particular to a three-dimensional reconstruction method, device and medium for incremental sub-scene merging. Background Art

[0002] As a key platform for collecting image data of target scenes through aerial photogrammetry, drones (UAVs) are crucial for high-precision and efficient 3D reconstruction of captured images. With the improvement of UAV endurance and the widespread use of drone nests, the effective cruising time of UAVs has been significantly increased, while their flight range has also expanded. This increased efficiency in UAV data collection has placed higher demands on data processing capabilities, necessitating new data processing methods. Currently, segmenting and merging large-scale UAV imagery is an effective way to improve data processing efficiency. Therefore, high-precision merging of segmented sub-scenes is a current research hotspot.

[0003] When merging submodels, the order and strategy of merging submodels are the main factors limiting their accuracy. Unreliable merging order and measurement can easily lead to cumulative errors, affecting the accuracy of the final merged model. For each independent subscene, similarity transformations between subscenes are typically estimated based on the relationship between common connection points, and the subscenes are then fused into the global coordinate system for optimization. However, this approach lacks consideration of the merging order and strategy.

[0004] Currently, some methods use a binary tree structure to index sub-scenes and merge sub-scenes using a bottom-up hierarchical aggregation method. However, this method lacks global considerations, resulting in unrobust merging. Chen Yu, on the other hand, uses the overlapping relationship between sub-scenes to construct a global minimum height tree and seek the optimal seed model in the entire scene. Based on this, he progressively merges sub-scenes to avoid error accumulation during the incremental merging process. However, the above method does not consider the problem of optimizing the gradient direction of overlapping images between sub-scenes, and cannot guarantee whether the selected merging order has a consistent optimization direction, which is prone to error accumulation. Summary of the Invention

[0005] The purpose of the present invention is to propose a three-dimensional reconstruction method for incremental sub-scene merging to solve the technical problem of low merging accuracy due to error accumulation when merging sub-scenes in UAV measurement.

[0006] The present invention starts from the perspective of optimizing the gradient direction of overlapping images between sub-scenes and performs incremental merging thereof to reduce the error accumulation problem occurring during the sub-scene merging process.

[0007] Specifically, the present invention provides a 3D reconstruction method for incremental sub-scene merging, comprising the following steps:

[0008] S1. Obtain the initial overall scene map of the UAV aerial survey image G , and divide it into sub-graphs of different sub-scenes G s ;

[0009] S2. Expand the sub-scene using a maximal clique to increase the number of common connection points in the sub-scene, and obtain the expanded sub-scene;

[0010] S3. For the expanded sub-scenes, starting from the perspective of the optimized gradient direction of the camera parameters of the overlapping images between different sub-scenes, calculate the consistency of the optimized gradient direction of the overlapping image camera parameters;

[0011] S4. Incrementally merge sub-scenes based on the consistency of the optimized gradient direction to obtain the final real-scene 3D model.

[0012] A storage medium stores instructions and data for implementing a three-dimensional reconstruction method for incremental sub-scene merging.

[0013] A three-dimensional reconstruction device for incremental sub-scene merging includes: a processor and the storage medium; the processor loads and executes instructions and data in the storage medium to implement a three-dimensional reconstruction method for incremental sub-scene merging.

[0014] The beneficial effects provided by the present invention are:

[0015] The high-precision merging of subscenes is achieved by adopting three steps: subscene graph expansion based on maximal cliques, subscene gradient consistency estimation, and incremental sub-model merging. This can reduce the cumulative error in the subscene merging process, ensure the accuracy of the final merged model, provide high-precision positioning and orientation results for the 3D reconstruction of UAV images, and provide high-precision data support for the application of real-scene 3D models. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic flow chart of the method of the present invention;

[0017] Figure 2 This is a schematic diagram of the sub-scene graph expansion based on the maximum clique;

[0018] Figure 3 Schematic diagram of sub-scene gradient consistency estimation;

[0019] Figure 4 Schematic diagram of incremental sub-model merging;

[0020] Figure 5 It is a schematic diagram of the working of the hardware device of an embodiment of the present invention. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0022] Before formally explaining the present invention, the scheme of the present invention is first generally explained for easy understanding.

[0023] Please refer to Figure 1 The present invention provides a 3D reconstruction method for incremental sub-scene merging, comprising the following steps:

[0024] S1. Obtain the initial overall scene map of the UAV aerial survey image G , and divide it into sub-graphs of different sub-scenes G s ;

[0025] It should be noted that the present invention creates an initial scene graph for the entire scene before segmentation. , and simplify and eliminate gross errors. After segmentation, each independent sub-scene graph is represented as (in , n is the number of sub-scene graphs).

[0026] S2. Expand the sub-scene using a maximal clique to increase the number of common connection points in the sub-scene, and obtain the expanded sub-scene;

[0027] Since there are no common image nodes between sub-scene graphs, directly merging the sub-scene reconstruction models can easily lead to failure in overall model reconstruction. In an undirected graph, all nodes in a maximal clique have interconnected edges. Expanding the local maximal clique of the sub-scene graph can improve the connection strength between the sub-scene graphs, such as Figure 2 shown.

[0028] As an embodiment, step S2 is specifically as follows:

[0029] S21. Subgraph in each sub-scene G s The split edge position, traverse all the image nodes corresponding to the split edges ;

[0030] S22. In the overall scene graph In the search, find the node All nodes with connection relationships are used to construct a local undirected graph and calculate all maximal clusters;

[0031] It should be noted that maximal clique is a basic concept in graph theory, and there are many classic calculation methods, such as the Born_Kerbosch algorithm.

[0032] S23, in the included node Find out whether there is a node belonging to a different sub-scene graph in the maximal clique. If so, remove the nodes that do not belong to the sub-scene graph. Image node Add to this sub-scene graph and retain and The connecting edges between , go to step S24; if it does not exist, end the current operation;

[0033] S24, for newly added nodes , repeat steps S22 to S23 until the iteration number threshold or the newly added node ratio constraint is met;

[0034] S25. Treat each expanded sub-scene graph as a single node , then the connection relationship between the sub-scene graphs Newly added image node and connecting edges Description, that is, the entire flight strip structure is expressed as .

[0035] S3. For the expanded sub-scenes, starting from the perspective of the optimized gradient direction of the camera parameters of the overlapping images between different sub-scenes, calculate the consistency of the optimized gradient direction of the overlapping image camera parameters;

[0036] It should be noted that the connectivity between the expanded sub-scene graphs is enhanced. The primary factor affecting robust sub-scene fusion is the order in which they are merged. Even if individual sub-scenes are reconstructed with high accuracy, an inappropriate merging order can lead to poor overall reconstructed model accuracy. Traditional methods that rely on common overlapping images and the number of connection points to determine the merging order fail to consider optimizing gradient directions between sub-models, leading to potential error accumulation during the merging process.

[0037] As an embodiment, the present invention selects the merging order of sub-models based on the gradient consistency of the sub-models as a metric based on the sub-scene parallel SfM reconstruction.

[0038] Specifically, for a given two interconnected sub-models and , calculate the steepest optimization gradient direction of all parameters in the sub-model respectively, and select the sub-model from them and The gradient of the camera parameters corresponding to the overlapping images forms a new gradient vector and , and ensure that the same image parameter position is in the vector and Consistent (e.g. Figure 3As shown), the subgraph and The gradient consistency between and Calculated according to formula (1).

[0039] (1).

[0040] S4. Incrementally merge sub-scenes based on the consistency of the optimized gradient direction to obtain the final real-scene 3D model.

[0041] It should be noted that the incremental sub-model merging strategy starts from the optimal seed sub-model, iteratively selects the optimal sub-model to be merged, and incorporates it into the unified global coordinate system for adjustment optimization to improve the robustness of sub-model merging and avoid error accumulation in sub-scene merging.

[0042] As an embodiment, the present invention optimizes the consistency of directional gradients based on the common image camera parameters between sub-scenes and adopts an incremental merging method, such as Figure 4 The specific steps are as follows:

[0043] ① Initial sub-model selection: For the sub-scene model with the selected optimal internal orientation parameters, calculate the gradient consistency cost between each sub-model connected to it, and select the sub-model with the minimum cost as the initial seed model for merging;

[0044] ②Selection of the optimal sub-model to be merged: For the merged reconstructed model, traverse the sub-models to be merged that have a connection relationship with it in turn, find the overlapping images shared with them and calculate the gradient consistency cost, and select the sub-model with the minimum cost as the optimal sub-model to be merged;

[0045] ③Sub-model merging:

[0046] The common image poses and 3D point coordinates between the two sub-models are searched, the similarity change matrix between the two sub-models is estimated using similarity change, and the sub-models are transformed into a unified coordinate framework.

[0047] It should be noted that similarity change is a common matrix change operation.

[0048] For the merging of initial sub-models, the sub-model with the selected optimal internal orientation parameters is used as the global coordinate system; for the merged sub-models in the scene, the merged sub-models are used as the global coordinate system.

[0049] After transforming the sub-models to be merged into the global coordinate system, a local bundle adjustment is performed on all image parameters and 3D point coordinates of the two models to be merged. Finally, a GNSS-assisted weighted bundle adjustment is performed to improve the absolute orientation accuracy of the merged model.

[0050] It should be noted that local bundle adjustment and GNSS-assisted bundle adjustment are both classic adjustment operations.

[0051] See Figure 5 , Figure 5 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a 3D reconstruction device 401 for incremental sub-scene merging, a processor 402 and a storage medium 403.

[0052] A 3D reconstruction device 401 for incremental sub-scene merging: The 3D reconstruction device 401 for incremental sub-scene merging implements the 3D reconstruction method for incremental sub-scene merging.

[0053] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the three-dimensional reconstruction method of incremental sub-scene merging.

[0054] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the three-dimensional reconstruction method of incremental sub-scene merging.

[0055] The beneficial effects of the present invention are:

[0056] The high-precision merging of subscenes is achieved by adopting three steps: subscene graph expansion based on maximal cliques, subscene gradient consistency estimation, and incremental sub-model merging. This can reduce the cumulative error in the subscene merging process, ensure the accuracy of the final merged model, provide high-precision positioning and orientation results for the 3D reconstruction of UAV images, and provide high-precision data support for the application of real-scene 3D models.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D reconstruction method using incremental sub-scene merging, characterized by: The method comprises the following steps: S1. Obtain the initial overall scene map of the UAV aerial survey image G , and divide it into sub-graphs of different sub-scenes G s ; S2. Expand the sub-scene using a maximal clique to increase the number of common connection points in the sub-scene, and obtain the expanded sub-scene; S3. For the expanded sub-scenes, starting from the perspective of the optimized gradient direction of the camera parameters of the overlapping images between different sub-scenes, calculate the consistency of the optimized gradient direction of the overlapping image camera parameters; S4. Incrementally merge sub-scenes based on the consistency of the optimized gradient direction to obtain the final real-scene 3D model.

2. The 3D reconstruction method of incremental sub-scene merging according to claim 1, characterized in that: Step S2 is specifically as follows: S21. Subgraph in each sub-scene G s The split edge position, traverse all the image nodes corresponding to the split edges ; S22. In the overall scene graph In the search, find the node All nodes with connection relationships are used to construct a local undirected graph and calculate all maximal clusters; S23, in the included node Find out whether there is a node belonging to a different sub-scene graph in the maximal clique. If so, remove the nodes that do not belong to the sub-scene graph. Image node Add to this sub-scene graph and retain and The connecting edges between , go to step S24; if it does not exist, end the current operation; S24, for newly added nodes , repeat steps S22 to S23 until the iteration number threshold or the newly added node ratio constraint is met; S25. Treat each expanded sub-scene graph as a single node , then the connection relationship between the sub-scene graphs Newly added image node and connecting edges Description, that is, the entire flight strip structure is expressed as .

3. The 3D reconstruction method of incremental sub-scene merging according to claim 1, characterized in that: In step S3, the calculation formula for the consistency of the optimized gradient direction of the overlapping image camera parameters is as follows: in, and is the gradient vector.

4. The 3D reconstruction method of incremental sub-scene merging according to claim 3, characterized in that: and The determination process is as follows: For a given two interconnected sub-scene models and , calculate the steepest optimization gradient direction of all parameters in the sub-scene model respectively, and select the sub-scene model from it and The gradient of the camera parameters corresponding to the overlapping images forms a new gradient vector and .

5. The 3D reconstruction method of incremental sub-scene merging according to claim 1, characterized in that: Step S4 is as follows: S41, selecting an initial sub-scene model; S42, selecting the optimal sub-scene model to be merged; S43: perform incremental sub-scene model merging.

6. The 3D reconstruction method of incremental sub-scene merging according to claim 5, characterized in that: The process of selecting the initial sub-scene model is as follows: For the sub-scene model with the selected optimal internal orientation parameters, the gradient consistency cost between each sub-scene model with a connection relationship is calculated, and the sub-scene model with the minimum cost is selected as the initial seed scene model for merging.

7. The 3D reconstruction method of incremental sub-scene merging according to claim 5, characterized in that: The process of selecting the optimal sub-scene model to be merged is as follows: For the merged reconstructed model, traverse the sub-scene models to be merged that have a connection relationship with it in turn, find the overlapping images shared with it and calculate the gradient consistency cost, and select the sub-scene model with the minimum cost as the optimal sub-scene model to be merged.

8. The 3D reconstruction method of incremental sub-scene merging according to claim 5, characterized in that: The process of incremental merging is as follows: Search for the common image poses and 3D point coordinates between the two sub-scene models, use similarity changes to estimate the similarity change matrix between the two sub-scene models, and transform the sub-scene models into a unified coordinate framework; For the merging of the initial sub-scene models, the sub-scene model with the selected optimal internal orientation parameters is used as the global coordinate system; for the merged sub-scene models in the scene, the merged sub-scene models are used as the global coordinate system; After transforming the sub-scene models to be merged into the global coordinate system, all image parameters and 3D point coordinates of the two models to be merged are optimized using the local bundle adjustment method. Combined with GNSS-assisted weighted bundle adjustment, the final real-scene 3D model is obtained.

9. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the three-dimensional reconstruction method of incremental sub-scene merging as described in any one of claims 1 to 6.

10. A 3D reconstruction device for incremental sub-scene merging, characterized by: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a three-dimensional reconstruction method for incremental sub-scene merging as described in any one of claims 1 to 6.

Images