Multi-machine parallel rapid three-dimensional reconstruction method

Abstract

The invention relates to the field of three-dimensional reconstruction, and discloses a multi-machine parallel rapid three-dimensional reconstruction method, which comprises the following steps of: dividing a plurality of blocks according to GPS (Global Positioning System) coordinates in EXIF (Extensible Extensible Interchange Format) information of an image; independently executing three-dimensional reconstruction by each block; a center block is selected; calculating a transformation matrix between each block and the central block; rotating, translating and zooming the three-dimensional model by each block according to the corresponding transformation matrix; combining the result output of each block; the plurality of blocks are a plurality of sub-regions, the number of images in the plurality of blocks is equal, and the adjacent blocks have minimum overlap. The number of images in each block is approximately equal, so that the processing time of each processing node is approximately equal, and finally, all nodes execute the three-dimensional reconstruction step in approximately equal time, thereby overcoming the problems that the air triangulation processing data volume is limited by a single-machine memory and the consumption is long in single-machine three-dimensional reconstruction. And rapid three-dimensional reconstruction of a large number of images is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a multi-machine parallel fast three-dimensional reconstruction method. Background Art

[0002] The three-dimensional model can accurately and comprehensively reflect the overall appearance of a scene or object. People can perform a lot of analysis and simulation work on the digital three-dimensional model. At present, there is no more convenient equipment and tool than drones for global situational awareness of large scenes. Drones can easily take pictures of the scene in a short time and construct a three-dimensional model of the scene through image-based three-dimensional reconstruction algorithms.

[0003] At present, the 3D model is generated based on images. The algorithm processing is image data processing, aerial triangulation, multi-view stereo 3D reconstruction, and output of 3D model. Among them, the aerial triangulation part is difficult to parallelize because it needs to solve the position and posture of all images of the entire scene as a whole. The scale of data processing is limited by the memory size of a single machine. At the same time, the processing time of aerial triangulation increases exponentially with the increase in the number of images. When processing large data, the processing time is unacceptable, which leads to the problem of slow 3D model generation. Especially in some occasions, such as emergency and field operations, the scenes to be processed are very large and the timeliness requirements for processing are very high. Therefore, fast 3D reconstruction is very necessary. Based on this, we propose a fast 3D reconstruction method with multi-machine parallelization to solve the above problems. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a multi-machine parallel fast 3D reconstruction method to solve the problem of aerial triangulation. Since the positions and postures of all images of the entire scene need to be solved as a whole, it is difficult to parallelize them. In this way, the scale of data processing is limited by the memory size of a single machine. At the same time, the processing time of aerial triangulation increases exponentially with the increase in the number of images. When processing larger data, the processing time is unacceptable, thus resulting in the problem of slow 3D model generation.

[0005] In a first aspect, the present invention provides the following technical solution, a multi-machine parallel fast three-dimensional reconstruction method, comprising the following steps:

[0006] S1. Divide the image into several blocks according to the GPS coordinates in the EXIF ​​information;

[0007] S2, each block is executed separately for 3D reconstruction;

[0008] S3, select the center block;

[0009] S4, each block calculates the transformation matrix between itself and the central block;

[0010] S5, each block rotates, translates and scales the three-dimensional model according to the corresponding transformation matrix;

[0011] S6. Merge the output of each block.

[0012] Preferably, the plurality of blocks are a plurality of sub-regions, the number of images in the plurality of blocks in the step S1 is equal, and there is a minimum overlap between adjacent blocks.

[0013] Preferably, in step S2, the images in a plurality of blocks are sent to a computing node, and the computing node is capable of completely executing a three-dimensional reconstruction algorithm, wherein the three-dimensional reconstruction includes aerial triangulation and multi-view stereoscopic three-dimensional reconstruction.

[0014] Preferably, the aerial triangulation includes the precise positions and postures of several block images and a sparse three-dimensional point cloud, wherein each three-dimensional point represents the position of a point in the actual scene and has three-dimensional coordinates, and each three-dimensional point is calculated from pixel points on two or more images; the multi-view stereo three-dimensional reconstruction is a textured three-dimensional model with real geographic coordinates.

[0015] Preferably, the step S3 comprises the following steps:

[0016] S301. Calculate the connectivity scores C of all adjacent blocks. ij , i and j represent two blocks in the scene respectively, and the calculation method is: the number of common points in the two blocks i and j is the connectivity score C of the two blocks ij ;

[0017] S302: The entire scene is considered as a weighted directed graph G = (V, E), where V represents all vertices, i.e., all blocks, and E represents the weights between nodes. ij Represents the weight between nodes i and j, and its calculation method is:

[0018]

[0019] ∑ mn C mn Represents the sum of all connectivity scores in the scene;

[0020] S303, record the sum of the shortest path weights from the target block to each node in the graph G, and record them as: E ab 、E ac 、E ac 、E ad ..., and find the sum, denoted as TA ;

[0021] S304, take the representative vertices of other blocks in the scene as the starting point, and calculate the sum of the weights from the vertex to all other vertices, which are recorded as: T A , T B , T C , T D ……;

[0022] S305 , selecting a block with the smallest sum of vertex weights as the center block, and simultaneously recording the path vertices from all other blocks to the block with the smallest sum of vertex weights.

[0023] Preferably, the step S4 comprises the following steps:

[0024] S401, first calculate the shortest path from other blocks to the central block, and take the transformation matrix of all blocks of a node, S AB represents the common point of two blocks, and its calculation method is the same as that in step S3. AB Each point in has different three-dimensional coordinates in each block. The transformation matrix M from other blocks to the block with the minimum sum of vertex weights is calculated by the seven-parameter method. BA ;

[0025] S402, then calculate the transformation matrix of the block with the shortest path node 2 to the center block from the inside to the outside, then its transformation matrix is ​​M CA =M CB *M BA ; Thus, the transformation matrix from each block to the central block is calculated in turn.

[0026] Preferably, the three-dimensional model is rotated, translated and scaled in step S5 by transforming the coordinates of each vertex through a transformation matrix from each block to the center block.

[0027] In a second aspect, a multi-machine parallel fast 3D reconstruction system includes:

[0028] Image acquisition and processing module: used to collect images of the target area and divide them into several blocks according to the GPS coordinates in the image EXIF ​​information;

[0029] Regional 3D reconstruction module: used to perform 3D reconstruction on each block separately;

[0030] Calculation module: used to select the central block according to the algorithm and calculate the transformation matrix between each block and the central block;

[0031] Total 3D reconstruction module: used to rotate, translate and scale the 3D model of each block according to the corresponding transformation matrix, and merge the output results of each block.

[0032] In the third aspect, the invention provides the following technical solution: a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multi-machine parallel fast three-dimensional reconstruction method when executing the computer program.

[0033] In a fourth aspect, the present invention provides the following technical solution: a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned multi-machine parallel fast three-dimensional reconstruction method.

[0034] The present invention has the following beneficial effects:

[0035] 1. The present invention makes the number of images in each block roughly equal, so the processing time of each processing node will be roughly the same, and finally all nodes will complete the three-dimensional reconstruction steps at roughly the same time, thereby overcoming the problem that the amount of data processed by aerial triangulation in single-machine three-dimensional reconstruction is limited by the memory of the single machine and the long consumption, so as to facilitate the rapid three-dimensional reconstruction of a large number of images.

[0036] 2. The present invention divides the GPS coordinates in the image EXIF ​​information into several blocks; each block is then independently executed for three-dimensional reconstruction; a central block is then selected; each block then calculates the transformation matrix between the central block; each block then rotates, translates and scales the three-dimensional model according to the corresponding transformation matrix; and the output of each block is then merged; thereby the process of three-dimensional reconstruction using images can be perfectly parallelized, thereby getting rid of the limitation of the physical memory of a single machine, and at the same time greatly improving the reconstruction efficiency, which is very suitable for emergency situations and other occasions where three-dimensional model results need to be quickly produced. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a multi-machine parallel fast three-dimensional reconstruction method of the present invention;

[0038] Figure 2 It is a block division schematic diagram of the present invention;

[0039] Figure 3 A schematic diagram for selecting a central block of the present invention;

[0040] Figure 4 It is a schematic diagram of the calculation transformation matrix of the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Embodiment 1

[0043] Reference Figure 1 In a first embodiment of the present invention, the present invention provides a multi-machine parallel fast three-dimensional reconstruction method, characterized in that it includes the following steps:

[0044] S1, divide the image into several blocks according to the GPS coordinates in the EXIF ​​information;

[0045] S2, each block is executed separately for 3D reconstruction;

[0046] S3, select the center block;

[0047] S4, each block calculates the transformation matrix between itself and the central block;

[0048] S5, each block rotates, translates and scales the three-dimensional model according to the corresponding transformation matrix;

[0049] S6. Merge the output of each block.

[0050] Specifically, for the 3D reconstruction of large scenes and massive images, this method divides the large scene into different sub-areas (hereinafter referred to as a sub-area as a block) according to the shooting range of the scene, and performs aerial triangulation and multi-view stereo 3D reconstruction on each block; when dividing the blocks, a certain amount of overlapping images are left between adjacent blocks, and finally, coordinate transformation is performed between different blocks through overlapping images, and then the 3D models of each block are merged, and finally the overall model is output.

[0051] Reference Figure 2 , several blocks are several sub-regions, the number of images in several blocks in step S1 is equal, and there is a minimum overlap between adjacent blocks.

[0052] Specifically, all drones currently have a route planning function. Users can pre-plan the flight route of the drone according to the operation area and specify the points where the drone needs to take images, so that the images taken by the aircraft can be evenly and regularly distributed in space. A typical distribution scenario is as follows: Figure 2As shown in the figure, the blocks are divided; each black dot in the figure represents an image taken at this location, and each wireframe in the figure indicates that the images represented by the black dots in the frame are divided into the same block. The division of blocks basically follows two principles: a. The number of images in each block is roughly equal; b. There is minimal overlap between adjacent blocks.

[0053] In step S2, the images in several blocks are sent to the computing node, which can fully execute the 3D reconstruction algorithm. The 3D reconstruction includes aerial triangulation and multi-view stereo 3D reconstruction; aerial triangulation includes the precise position and posture of several block images and a sparse 3D point cloud, where each 3D point represents the position of a point in the actual scene and has 3D coordinates. At the same time, each 3D point is calculated from pixel points on two or more images; multi-view stereo 3D reconstruction is a textured 3D model with real geographic coordinates.

[0054] Specifically, an image in a block is sent to a computing node, where a node refers to a computing module that can fully execute a 3D reconstruction algorithm, including but not limited to a computing board, a PC, and a workstation. After receiving the image data, the processing node executes the complete 3D reconstruction steps. Since the number of images in each block is roughly equal, the processing time of each processing node will be roughly the same. Finally, all nodes will complete the 3D reconstruction steps at roughly the same time. The output results include the results of aerial triangulation and the results of multi-view stereo 3D reconstruction. The results of aerial triangulation include: a. The precise position and posture of each image; b. A sparse 3D point cloud, where each 3D point represents the position of a point in the actual scene and has 3D coordinates. At the same time, each 3D point is calculated from pixels on two or more images, so each 3D point will also be associated with pixels on two or more images. The result of multi-view stereo 3D reconstruction is a textured 3D model with real geographic coordinates.

[0055] Reference Figure 3 , step S3 comprises the following steps:

[0056] S301. Calculate the connectivity scores C of all adjacent blocks. ij , i and j represent two blocks in the scene respectively, and the calculation method is: the number of common points in the two blocks i and j is the connectivity score C of the two blocks ij ;

[0057] S302: The entire scene is considered as a weighted directed graph G = (V, E), where V represents all vertices, i.e., all blocks, and E represents the weights between nodes. ijRepresents the weight between nodes i and j, and its calculation method is:

[0058]

[0059] ∑ mn C mn Represents the sum of all connectivity scores in the scene;

[0060] S303, record the sum of the shortest path weights from the target block to each node in the graph G, and record them as: E ab 、E ac 、E ac 、E ad ..., and find the sum, denoted as T A ;

[0061] S304, take the representative vertices of other blocks in the scene as the starting point, and calculate the sum of the weights from the vertex to all other vertices, which are recorded as: T A , T B , T C , T D ……;

[0062] S305 , selecting a block with the smallest sum of vertex weights as the center block, and simultaneously recording the path vertices from all other blocks to the block with the smallest sum of vertex weights.

[0063] Specifically, the selection of the center block is roughly divided into the following steps:

[0064] a. Calculate the connectivity score C of all adjacent blocks ij , i and j represent two blocks in the scene respectively, and the calculation method is: the number of common points in the two blocks i and j is the connectivity score C of the two blocks ij ;

[0065] b. Consider the entire scene as a weighted directed graph G = (V, E), where V represents all vertices in the graph, i.e., all blocks, and E represents the weights between nodes. ij Represents the weight between nodes i and j, and its calculation method is:

[0066] E ij =∑ m，n C mn / C ij ,

[0067] ∑ mn C mn Represents the sum of all connectivity scores in the scene.

[0068] c. Each point in the graph represents a vertex in G, that is, a block. For blockA and blockB, blockA is regarded as the starting point and blockB is regarded as the end point. Dijkstra algorithm is used to calculate the shortest path from blockA to blockB. Obviously, the shortest path from blockA to blockB will be as shown by the dotted line in the figure, and the sum of the shortest path weights is 54.411. The sum of the shortest path weights from blockA to each node in graph G is recorded in turn, and recorded as: E ab , E ac , E ac , E ad ..., and find the sum, denoted as T A ;

[0069] d. Take the representative vertices of other blocks in the scene as the starting point, and calculate the sum of the weights from this vertex to all other vertices, which are recorded as: T A , T B , T C , T D ……;

[0070] e. Select the block represented by the vertex with the smallest total weight as the central block, assuming it is blockA, and record the path vertices from all other blocks to blockA.

[0071] Reference Figure 4 , step S4 comprises the following steps:

[0072] S401, first calculate the shortest path from other blocks to the central block, and take the transformation matrix of all blocks of a node, S AB represents the common point of two blocks, and its calculation method is the same as that in step S3. AB Each point in has different three-dimensional coordinates in each block. The transformation matrix M from other blocks to the block with the minimum sum of vertex weights is calculated by the seven-parameter method. BA ;

[0073] S402, then calculate the transformation matrix of the block with the shortest path node 2 to the center block from the inside to the outside, then its transformation matrix is ​​M CA =M CB *M BA ; Thus, the transformation matrix from each block to the central block is calculated in turn.

[0074] Specifically, we adopt an inside-out strategy to calculate the transformation matrix of all blocks except the center block A to the center block one by one. First, we calculate the transformation matrix of all blocks whose shortest path to block A is a node. Figure 4 The transformation matrix from block to blockA represented by the semi-transparent gray nodes in the middle, assuming that the point set S AB represents the common point of two blocks, and its calculation method is the same as that in step S3. AB Each point in blockA has different three-dimensional coordinates in blockB. The transformation matrix M from blockB to blockA is calculated by the seven-parameter method. BA ; In the previous step, the transformation matrix from the block represented by the gray node in the figure to the central block has been calculated. Then, according to the strategy from inside to outside, the transformation matrix from the block with the shortest path node 2 to the gray node in the figure is calculated, that is, the transformation matrix from the block represented by the black node in the figure to the block represented by the gray node; For example: the central block is A, block B is adjacent to block A, and the transformation matrix M is calculated BA , blockC and blockB are adjacent, and the transformation matrix is ​​calculated to be M CB , then the final transformation matrix from blockC to the center blockA is M CA =M CB *M BA .

[0075] In step S5, the three-dimensional model is rotated, translated and scaled to transform the coordinates of each vertex through the transformation matrix from each block to the center block.

[0076] Specifically, in step S4, the transformation matrix from all blocks to the central block has been calculated. For the 3D model finally generated by each block, the coordinates of each vertex can be transformed by the transformation matrix, thereby realizing fast 3D reconstruction on multiple machines in parallel.

[0077] Embodiment 2:

[0078] In a second embodiment of the present invention, the present invention provides a multi-machine parallel fast three-dimensional reconstruction system, comprising:

[0079] Image acquisition and processing module: used to collect images of the target area and divide them into several blocks according to the GPS coordinates in the image EXIF ​​information;

[0080] Regional 3D reconstruction module: used to perform 3D reconstruction on each block separately;

[0081] Calculation module: used to select the center block according to the algorithm and calculate the transformation matrix between each block and the center block:

[0082] Total 3D reconstruction module: used to rotate, translate and scale the 3D model of each block according to the corresponding transformation matrix, and merge the output results of each block.

[0083] Embodiment 3

[0084] The third embodiment of the present invention is based on the same inventive concept and proposes a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the steps of a multi-machine parallel fast three-dimensional reconstruction method of the above embodiment.

[0085] Embodiment 4

[0086] The fourth embodiment of the present invention is based on the same inventive concept. The present invention proposes a terminal, which includes: a processor and a memory; the processor and the memory communicate with each other; the memory is used to store instructions; the processor is used to execute the instructions in the memory, and execute a multi-machine parallel fast three-dimensional reconstruction method of the above embodiment.

[0087] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0088] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-machine parallel fast 3D reconstruction method, characterized by: The following steps are involved: S1. Divide the image into several blocks according to the GPS coordinates in the EXIF ​​information; S2, each block is executed separately for 3D reconstruction; S3, select the center block; S4, each block calculates the transformation matrix between itself and the central block; S5, each block rotates, translates and scales the three-dimensional model according to the corresponding transformation matrix; S6. Merge the output of each block.

2. The multi-machine parallel fast 3D reconstruction method according to claim 1, characterized in that: The blocks are several sub-regions. In step S1, the number of images in the blocks is equal, and there is a minimum overlap between adjacent blocks.

3. The multi-machine parallel fast 3D reconstruction method according to claim 1, characterized in that: In step S2, the images in a plurality of blocks are sent to a computing node, and the computing node is capable of completely executing a three-dimensional reconstruction algorithm, wherein the three-dimensional reconstruction includes aerial triangulation and multi-view stereoscopic three-dimensional reconstruction.

4. The multi-machine parallel fast 3D reconstruction method according to claim 3, characterized in that: The aerial triangulation includes the precise position and posture of several block images and a sparse three-dimensional point cloud, wherein each three-dimensional point represents the position of a point in the actual scene and has three-dimensional coordinates, and each three-dimensional point is calculated from pixel points on two or more images; the multi-view stereo three-dimensional reconstruction is a textured three-dimensional model with real geographic coordinates.

5. The multi-machine parallel fast 3D reconstruction method according to claim 1, characterized in that: The step S3 comprises the following steps: S301. Calculate the connectivity scores C of all adjacent blocks. ij , i and j represent two blocks in the scene respectively, and the calculation method is: the number of common points in the two blocks i and j is the connectivity score C of the two blocks ij ; S302: The entire scene is considered as a weighted directed graph G = (V, E), where V represents all vertices, i.e., all blocks, and E represents the weights between nodes. ij Represents the weight between nodes i and j, and its calculation method is: ∑ mn C mn Represents the sum of all connectivity scores in the scene; S303, record the sum of the shortest path weights from the target block to each node in the graph G, and record them as: E ab 、E ac 、E ac 、E ad ..., and find the sum, denoted as T A ; S304, take the representative vertices of other blocks in the scene as the starting point, and calculate the sum of the weights from the vertex to all other vertices, which are recorded as: T A , T B , T C , T D ……; S305 , selecting a block with the smallest sum of vertex weights as the center block, and simultaneously recording the path vertices from all other blocks to the block with the smallest sum of vertex weights.

6. The multi-machine parallel fast 3D reconstruction method according to claim 1, characterized in that: The step S4 comprises the following steps: S401, first calculate the shortest path from other blocks to the central block, and take the transformation matrix of all blocks of a node, S AB represents the common point of two blocks, and its calculation method is the same as that in step S3. AB Each point in has different three-dimensional coordinates in each block. The transformation matrix M from other blocks to the block with the minimum sum of vertex weights is calculated by the seven-parameter method. BA ; S402, then calculate the transformation matrix of the block with the shortest path node 2 to the center block from the inside to the outside, then its transformation matrix is ​​M CA =M CB *M BA ; Thus, the transformation matrix from each block to the central block is calculated in turn.

7. The multi-machine parallel fast 3D reconstruction method according to claim 1, characterized in that: In step S5, the three-dimensional model is rotated, translated and scaled by transforming the coordinates of each vertex through the transformation matrix from each block to the center block.

8. A multi-machine parallel fast 3D reconstruction system, characterized by: A multi-machine parallel fast three-dimensional reconstruction method for any one of claims 1 to 7, comprising: Image acquisition and processing module: used to collect images of the target area and divide them into several blocks according to the GPS coordinates in the image EXIF ​​information; Regional 3D reconstruction module: used to perform 3D reconstruction on each block separately; Calculation module: used to select the central block according to the algorithm and calculate the transformation matrix between each block and the central block; Total 3D reconstruction module: used to rotate, translate and scale the 3D model of each block according to the corresponding transformation matrix, and merge the output results of each block.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-machine parallel fast three-dimensional reconstruction method as described in any one of claims 1 to 7 is implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for rapid three-dimensional reconstruction using multiple machines in parallel is implemented.

Images