Three-dimensional reconstruction method and device for incremental sub-scene combination and medium

Abstract

The invention relates to the field of aerial survey, and discloses a three-dimensional reconstruction method and device for incremental sub-scene merging and a medium, and the method comprises the steps: obtaining an initial overall scene graph G of an aerial survey image of an unmanned aerial vehicle, and segmenting the initial overall scene graph G into sub-graphs Gs of different sub-scenes; expanding the sub-scenes by using a maximum clique, and increasing the number of common connection points of the sub-scenes to obtain expanded sub-scenes; for the expanded sub-scenes, starting from the angle of the optimized gradient direction of the co-overlapping image camera parameters among different sub-scenes, calculating the consistency of the optimized gradient direction of the overlapping image camera parameters; performing incremental sub-scene combination according to the consistency of the optimized gradient direction to obtain a final live-action three-dimensional model; according to the method, high-precision merging of the sub-scenes is realized, accumulative errors occurring in the sub-scene merging process can be reduced, the precision of the final merging model is ensured, and high-precision data support is provided for unmanned aerial vehicle image three-dimensional reconstruction and live-action three-dimensional model application.

Description

Technical Field

[0001] The present invention relates to the field of aerial surveying, and in particular to a three-dimensional reconstruction method, device and medium for incremental sub-scene merging. Background Art

[0002] As an important platform for collecting target scene image data by aerial photogrammetry, it is of great significance to reconstruct the collected images with high precision and high efficiency. With the improvement of drone endurance and the widespread application of drone nests, the effective cruising time of drones has been significantly improved, and the flight range has also been expanded. Then, the improvement of drone data collection efficiency has put higher requirements on data processing capabilities, and new data processing methods are urgently needed. At present, block-merging processing of large-scale drone images is an effective way to improve data processing efficiency. Therefore, high-precision merging of sub-scenes after block-merging is a hot topic in current research.

[0003] In the sub-model merging, the merging order and merging strategy of the sub-models become the main reasons that restrict their merging accuracy. Unreliable merging order and measurement can easily lead to error accumulation, affecting the accuracy of the final merged model. For each independent sub-scene, the similarity transformation between sub-scenes is usually estimated based on the relationship between common connection points, and the sub-scenes are fused into the global coordinate system for optimization, without considering the merging order and merging strategy.

[0004] At present, some methods use a binary tree structure to index sub-scenes and merge sub-scenes in a bottom-up hierarchical aggregation method. However, this method lacks global considerations, resulting in an unrobust merger. Chen Yu uses the overlapping relationship between sub-scenes to construct a global minimum height tree and seeks the optimal seed model in the entire scene. Based on this, the sub-scenes are progressively merged to avoid error accumulation during the incremental merging process. However, the above method does not consider the problem of optimizing the gradient direction of overlapping images between sub-scenes, and cannot guarantee whether the selected merging order has a consistent optimization direction, which is prone to error accumulation. Summary of the invention

[0005] The purpose of the present invention is to propose a three-dimensional reconstruction method for incremental sub-scene merging to solve the technical problem that the merging accuracy is not high due to error accumulation when merging sub-scenes in UAV measurement.

[0006] The present invention starts from the perspective of optimizing the gradient direction of overlapping images between sub-scenes and performs incremental merging of them to reduce the error accumulation problem that occurs during the sub-scene merging process.

[0007] Specifically, the present invention provides a 3D reconstruction method for incremental sub-scene merging, comprising the following steps: S1. Obtain the initial overall scene map of the drone aerial survey image G , and divide it into sub-images of different sub-scenes G s ; S2, the sub-scene is expanded using a maximum clique to increase the number of common connection points of the sub-scene, and an expanded sub-scene is obtained; S3, for the expanded sub-scenes, starting from the perspective of the optimized gradient direction of the camera parameters of the overlapping images between different sub-scenes, calculating the consistency of the optimized gradient direction of the overlapping image camera parameters; S4. Incrementally merge sub-scenes according to the consistency of optimized gradient directions to obtain the final real-scene 3D model.

[0008] A storage medium stores instructions and data for implementing a three-dimensional reconstruction method for incremental sub-scene merging.

[0009] A three-dimensional reconstruction device for incremental sub-scene merging includes: a processor and the storage medium; the processor loads and executes instructions and data in the storage medium to implement a three-dimensional reconstruction method for incremental sub-scene merging.

[0010] The beneficial effects provided by the present invention are: The high-precision merging of subscenes is achieved by adopting three steps: subscene graph expansion based on maximal cliques, subscene gradient consistency estimation, and incremental sub-model merging. This can reduce the cumulative error in the subscene merging process, ensure the accuracy of the final merged model, provide high-precision positioning and orientation results for the 3D reconstruction of UAV images, and provide high-precision data support for the application of real-life 3D models. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic flow chart of the method of the present invention; Figure 2 This is a schematic diagram of the expansion of the sub-scene graph based on the maximum clique; Figure 3 Schematic diagram of sub-scene gradient consistency estimation; Figure 4 Schematic diagram of incremental sub-model merging; Figure 5 It is a schematic diagram of the working of the hardware device of an embodiment of the present invention. DETAILED DESCRIPTION

[0012] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0013] Before formally describing the present invention, the scheme of the present invention is first generally described for easy understanding.

[0014] Please refer to Figure 1 The present invention provides a three-dimensional reconstruction method for incremental sub-scene merging, comprising the following steps: S1. Obtain the initial overall scene map of the drone aerial survey image G , and divide it into sub-images of different sub-scenes G s ; It should be noted that the present invention creates an initial scene graph for the entire scene before segmentation. , and simplify and eliminate gross errors. After segmentation, each independent sub-scene graph is represented as (in , n is the number of sub-scene graphs).

[0015] S2, the sub-scene is expanded using a maximum clique to increase the number of common connection points of the sub-scene, and an expanded sub-scene is obtained; Since there are no common image nodes between sub-scene graphs, directly merging the sub-scene reconstruction models can easily lead to failure in overall model reconstruction. In an undirected graph, all nodes in the maximal clique have interconnected edges. Expanding the local maximal clique of the sub-scene graph can improve the connection strength between the sub-scene graphs, such as Figure 2 shown.

[0016] As an embodiment, step S2 is specifically as follows: S21. Sub-images in each sub-scene G s The split edge position, traverse all the image nodes corresponding to the split edges ; S22. In the overall scene graph In the search, find the node All nodes with connection relationships are used to construct local undirected graphs and calculate all maximal clusters; It should be noted that maximal clique is a basic concept in graph theory, and there are many classic calculation methods, such as the Born_Kerbosch algorithm.

[0017] S23, in the included node Find out whether there are nodes belonging to different sub-scene graphs in the maximal clique of , if so, remove the nodes that do not belong to the sub-scene graph Image node Add to this sub-scene graph and keep and The connecting edges between , go to step S24; if it does not exist, end the current operation; S24, for newly added nodes , repeat steps S22 to S23 until the iteration number threshold or the newly added node ratio constraint is met; S25. Treat each expanded sub-scene graph as a single node , then the connection relationship between the sub-scene graphs By the newly added image node and connecting edges Description, that is, the entire flight strip structure is expressed as .

[0018] S3, for the expanded sub-scenes, starting from the perspective of the optimized gradient direction of the camera parameters of the overlapping images between different sub-scenes, calculating the consistency of the optimized gradient direction of the overlapping image camera parameters; It should be noted that the connection between the expanded sub-scene graphs is enhanced. The main factor affecting the robust fusion of sub-scenes is the choice of the merging order between sub-scenes. Even if the reconstruction accuracy of a single sub-scene is high, an insecure merging order can easily lead to a deterioration in the accuracy of the overall reconstructed model. The traditional method that relies on common overlapping images and the number of connection points to determine the merging order does not consider the problem of optimizing the gradient direction between sub-models, and is prone to error accumulation during the merging process.

[0019] As an embodiment, the present invention selects the merging order of sub-models based on the sub-scene parallel SfM reconstruction and takes the gradient consistency of the sub-models as a metric.

[0020] Specifically, for a given two interconnected sub-models and , calculate the steepest optimization gradient direction of all parameters in the sub-model respectively, and select the sub-model from them and The gradients of the camera parameters corresponding to the overlapping images form a new gradient vector and , and ensure that the same image parameter position is in the vector and Consistent (such as Figure 3 As shown), then the subgraph and The gradient consistency between and Calculate according to formula (1).

[0021] (1).

[0022] S4. Incrementally merge sub-scenes according to the consistency of optimized gradient directions to obtain the final real-scene 3D model.

[0023] It should be noted that the incremental sub-model merging strategy starts from the optimal seed sub-model, iteratively selects the optimal sub-model to be merged, and incorporates it into the unified global coordinate system for adjustment optimization to improve the robustness of sub-model merging and avoid error accumulation in sub-scene merging.

[0024] As an embodiment, the present invention optimizes the consistency of directional gradients according to the common image camera parameters between sub-scenes, and adopts an incremental merging method, such as Figure 4 As shown, the specific steps are as follows: ① Initial sub-model selection: For the sub-scene model with the selected optimal internal orientation parameters, the gradient consistency cost between each sub-model connected to it is calculated, and the sub-model with the minimum cost is selected as the initial seed model for merging; ②Selection of the optimal sub-model to be merged: For the merged reconstructed model, traverse the sub-models to be merged that have a connection relationship with it in turn, find the overlapping images shared with it respectively and calculate the gradient consistency cost, and select the sub-model with the minimum cost as the optimal sub-model to be merged; ③Sub-model merging: The common image poses and 3D point coordinates between the two sub-models are searched, and the similarity change matrix between the two sub-models is estimated using similarity change, and the sub-models are transformed into a unified coordinate framework.

[0025] It should be noted that similarity change is a common matrix change operation.

[0026] For the merging of initial sub-models, the sub-model with the selected optimal internal orientation parameter is used as the global coordinate system; for the merged sub-models in the scene, the merged sub-models are used as the global coordinate system.

[0027] After transforming the sub-models to be merged into the global coordinate system, all image parameters and 3D point coordinates of the two models to be merged are optimized by local bundle adjustment. Finally, GNSS-assisted weighted bundle adjustment is combined to improve the absolute orientation accuracy of the merged model.

[0028] It should be noted that local bundle adjustment and GNSS-assisted bundle adjustment are both classic adjustment operations.

[0029] See also Figure 5 , Figure 5 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a 3D reconstruction device 401 for incremental sub-scene merging, a processor 402 and a storage medium 403.

[0030] A three-dimensional reconstruction device 401 for incremental sub-scene merging: The three-dimensional reconstruction device 401 for incremental sub-scene merging implements the three-dimensional reconstruction method for incremental sub-scene merging.

[0031] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the three-dimensional reconstruction method of incremental sub-scene merging.

[0032] Storage medium 403: the storage medium 403 stores instructions and data; the storage medium 403 is used to implement the three-dimensional reconstruction method of incremental sub-scene merging.

[0033] The beneficial effects of the present invention are: The high-precision merging of subscenes is achieved by adopting three steps: subscene graph expansion based on maximal cliques, subscene gradient consistency estimation, and incremental sub-model merging. This can reduce the cumulative error in the subscene merging process, ensure the accuracy of the final merged model, provide high-precision positioning and orientation results for the 3D reconstruction of UAV images, and provide high-precision data support for the application of real-life 3D models.

[0034] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D reconstruction method for incremental sub-scene merging, characterized by: The method comprises the following steps: S1. Obtain the initial overall scene map of the drone aerial survey image G , and divide it into sub-images of different sub-scenes G s ; S2, the sub-scene is expanded using a maximum clique to increase the number of common connection points of the sub-scene, and an expanded sub-scene is obtained; S3, for the expanded sub-scenes, starting from the perspective of the optimized gradient direction of the camera parameters of the overlapping images between different sub-scenes, calculating the consistency of the optimized gradient direction of the overlapping image camera parameters; S4. Incrementally merge sub-scenes according to the consistency of optimized gradient directions to obtain the final real-scene 3D model.

2. The 3D reconstruction method of incremental sub-scene merging according to claim 1, characterized in that: Step S2 is specifically as follows: S21. Sub-images in each sub-scene G s The split edge position, traverse all the image nodes corresponding to the split edges ; S22. In the overall scene graph In the search, find the node All nodes with connection relationships are used to construct local undirected graphs and calculate all maximal clusters; S23, in the included node Find out whether there are nodes belonging to different sub-scene graphs in the maximal clique of , if so, remove the nodes that do not belong to the sub-scene graph Image node Add to this sub-scene graph and keep and The connecting edges between , go to step S24; if it does not exist, end the current operation; S24, for newly added nodes , repeat steps S22 to S23 until the iteration number threshold or the newly added node ratio constraint is met; S25. Treat each expanded sub-scene graph as a single node , then the connection relationship between the sub-scene graphs By the newly added image node and connecting edges Description, that is, the entire flight strip structure is expressed as .

3. The incremental sub-scene merging 3D reconstruction method according to claim 1, characterized in that: In step S3, the calculation formula for the consistency of the optimized gradient direction of the overlapping image camera parameters is as follows: in, and is the gradient vector.

4. The 3D reconstruction method of incremental sub-scene merging according to claim 3, characterized in that: and The determination process is as follows: For a given two interconnected sub-models and , calculate the steepest optimization gradient direction of all parameters in the sub-model respectively, and select the sub-model from them and The gradients of the camera parameters corresponding to the overlapping images form a new gradient vector and .

5. The 3D reconstruction method of incremental sub-scene merging according to claim 1, characterized in that: Step S4 is as follows: S41, selecting an initial sub-model; S42, selecting the optimal sub-model to be merged; S43, performing incremental sub-model merging.

6. The incremental sub-scene merging 3D reconstruction method according to claim 5, characterized in that: The process of selecting the initial sub-model is as follows: For the sub-scene model with the selected optimal internal orientation parameters, the gradient consistency cost between each sub-model connected to it is calculated, and the sub-model with the minimum cost is selected as the initial seed model for merging.

7. The incremental sub-scene merging 3D reconstruction method according to claim 5, characterized in that: The process of selecting the best sub-model to be merged is as follows: For the merged reconstructed model, the sub-models to be merged that are connected to it are traversed in turn, the overlapping images shared with it are found and the gradient consistency cost is calculated, and the sub-model with the minimum cost is selected as the optimal sub-model to be merged.

8. The incremental sub-scene merging 3D reconstruction method according to claim 5, characterized in that: The process of incremental merging is as follows: Search for the common image poses and 3D point coordinates between the two sub-models, estimate the similarity change matrix between the two sub-models using similarity changes, and transform the sub-models into a unified coordinate framework; For the merging of the initial sub-models, the sub-model with the selected optimal internal orientation parameter is used as the global coordinate system; for the merged sub-models in the scene, the merged sub-models are used as the global coordinate system; After transforming the sub-models to be merged into the global coordinate system, all image parameters and 3D point coordinates of the two models to be merged are optimized by local bundle adjustment. Combined with GNSS-assisted weighted bundle adjustment, the final real-scene 3D model is obtained.

9. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a three-dimensional reconstruction method for incremental sub-scene merging as described in any one of claims 1 to 6.

10. A 3D reconstruction device for incremental sub-scene merging, characterized in that: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a three-dimensional reconstruction method for incremental sub-scene merging as described in any one of claims 1 to 6.

Images