Method and system for scalable scene 3d reconstruction based on uav onboard localization

Abstract

The application discloses a kind of based on unmanned aerial vehicle airborne positioning extensible scene three-dimensional reconstruction method and system, based on the color image and depth image obtained by depth camera carried on unmanned aerial vehicle are carried out three-dimensional reconstruction, including in the unit ball in 6D state space, particle is uniformly sampled, obtains particle group template;Surface measurement is carried out to each frame depth image input, the pixel-by-pixel projection of depth image is projected into three-dimensional space and the normal of each three-dimensional point is calculated, while calculating segmented normal map;According to the correlation between three-dimensional point normal, adaptively allocate reconstruction voxel memory in GPU;Only using depth image and the three-dimensional model that exists in current GPU active space has been constructed, the camera pose of fast motion is tracked;With the continuous transmission of depth image frame sequence, based on truncated signed distance field TSDF fusion measurement value with sensor noise, construct dense three-dimensional point cloud model, extract three-dimensional surface when needing visualization, generate three-dimensional mesh model.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, specifically relating to an scalable scene 3D reconstruction technology solution based on UAV-borne real-time positioning. Background Technology

[0002] Real-world 3D models, as a crucial data foundation for representing the real world, are widely used in smart cities, interior design, disaster prevention and relief, augmented reality, mixed reality, and many other fields. Traditional 3D reconstruction typically uses 2D color images as input and outputs a reconstructed 3D model. However, limited by fixed input data, the reconstructed 3D models are often incomplete and lack realism. Furthermore, the massive computational demands of multi-view depth estimation also lead to low reconstruction efficiency, making it difficult to extend traditional 3D reconstruction techniques to applications with high real-time requirements, such as augmented reality, mixed reality, drone navigation, and disaster recovery in unknown, enclosed environments. While Simultaneous Localization and Mapping (SLAM) technology can achieve real-time scene reconstruction using a monocular color camera, the sparse point cloud model generated by a monocular SLAM system only supports basic scene reconstruction and struggles to provide a realistic and intuitive 3D scene representation.

[0003] Time-of-flight or structured light-based depth cameras (RGB-D cameras) provide reliable, dense depth measurements in highly integrated, miniaturized devices. With a single monocular color lens and a set of depth sensing devices, depth cameras can capture color (RGB image) and depth images of the surrounding environment at sufficient resolution and real-time rates, enabling real-time scene reconstruction and model updates. As depth cameras have become increasingly cost-effective, they have gained favor in academia and industry, leading to the rapid development of depth camera-based 3D reconstruction technology over the past decade. Combining these advantages, consumer-grade depth cameras even outperform some more expensive 3D scanning systems, particularly in consumer-grade solutions.

[0004] Currently, most depth camera-based 3D reconstruction operations involve handheld depth cameras, with the acquired color and depth images processed in real-time on high-performance data processing terminals. This includes autonomous camera localization and pose determination (Camera Tracking) and 3D model construction and updating (Update Reconstruction). Existing depth camera-based 3D reconstruction methods have three main drawbacks: First, the system's applicability is limited to slow camera movement, typically less than 1 m / s; under fast camera movement, the reconstruction system is prone to crashing. Second, they rely on high-performance computing devices; real-time performance is only achievable with the support of high-performance graphics computing devices. On onboard computers with limited computing resources, real-time performance is difficult to achieve. Third, limited by the limited graphics card memory resources, the reconstructable scene range is restricted to a fixed and very limited size; when the camera movement exceeds this range, the reconstruction system will cease operation. These technical limitations severely restrict the real-time reconstruction of large-scale 3D models from UAVs under rapid movement using only inexpensive, miniaturized, and computationally limited onboard computers. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a method for real-time positioning and scalable scene 3D reconstruction on UAVs. This method enables real-time reconstruction of large-scale scene 3D models on the onboard computer of a fast-moving UAV.

[0006] To achieve the above objectives, the technical solution of this invention is a scalable scene 3D reconstruction method based on UAV airborne positioning. This method performs 3D reconstruction based on color and depth images acquired by a depth camera mounted on a UAV, and includes the following steps:

[0007] Step 1: Uniformly sample the particles within a unit sphere in the 6D state space to obtain a particle swarm template, where any one particle represents an effective rigid body motion in three-dimensional space.

[0008] Step 2: Perform surface measurement on each frame of the input depth image, project the depth image pixel by pixel into the three-dimensional space according to the camera intrinsic parameters, calculate the normal of each three-dimensional point, and calculate the segmentation normal map at the same time.

[0009] Step 3: Based on the correlation between the normals of the three-dimensional points, the memory of the reconstructed voxels is adaptively allocated in the GPU to realize the sparse representation of the surface. After the camera leaves the current reconstruction area, the allocated voxel space is managed.

[0010] Step 4: Using only the depth image and the existing 3D model in the current GPU activity space, track the pose of the fast-moving camera and calculate the relative position and orientation of the camera when capturing the current frame.

[0011] Step 5: As the depth image frame sequence continues to be received, the measurement values ​​with sensor noise are fused based on the truncated symbolic distance field TSDF to construct a dense 3D point cloud model. When visualization is required, the 3D surface is extracted to generate a 3D mesh model.

[0012] Furthermore, in step 1, a fixed number of particles are uniformly presampled within a unit sphere in the 6D state space, with each particle representing a rigid body motion in three-dimensional space. The presampled set of particles is called a particle swarm template. Multi-level particle swarm templates are constructed and used in combination with depth images of different resolutions to support pose estimation.

[0013] Furthermore, in step 2, depth image preprocessing is performed in the following manner:

[0014] The depth camera intrinsic parameter matrix K and the depth image I d Calculate the vertex graph

[0015] Step based on vertex graph Calculate the corresponding normal map

[0016] In the normal diagram Based on this, the segmentation normal map is calculated using the region growing method.

[0017] Furthermore, in step 3, the scene surface sparse representation with normal correlation constraints is performed in the following manner:

[0018] When a new frame of depth imagery is input into the reconstruction system, it iterates through each pixel in the depth imagery, and for each depth measurement value I... d (u) Construct a ray with the photography center

[0019] Set the cutoff range of TSDF to μ, along the ray The direction is in I d (u)-μ to I d Within the (u)+μ cutoff band, a line segment is created. The coordinates of the voxels on the line segment are obtained from coarse to fine. The hash value h is calculated, and it is checked whether the voxel block b corresponding to the hash value has been allocated memory in the GPU. If not, it is allocated, and new entries are created in the hash table and voxel block array. The segmentation normal map is used as a mask to allocate voxel blocks for pixels within the segmentation plane region at the coarsest resolution level. Pixels outside the segmentation region skip the coarsest level and are directly allocated voxel space at the second coarser level. In their corresponding coarsest level hash table entry, a special marker implicitly points to the location of the sub-block.

[0020] After the voxel memory allocation is completed, the roughness r(b) of the surface where the voxel block b is located is calculated based on the correlation of the normals, and is used as the execution standard for fusing and subdividing voxels.

[0021] When roughness is greater than the segmentation threshold t s When the roughness is less than the merging threshold t, segment the voxels of the current level; m At that time, merge the voxels of the current level;

[0022] Define a spherical activity space containing the current camera's field of view. For different depth cameras, based on the effective depth range l, set the center of the activity space to be located at a distance l / 2 from the depth camera, with a radius of l. When the camera moves, causing the current scene to leave the camera's field of view, its corresponding surface voxels are also moved out of the activity space, and the voxels are transferred from the CPU to the GPU. When the camera revisits a previously reconstructed area, the voxels are transferred from the GPU to the CPU.

[0023] Furthermore, in step 4, fast camera pose estimation is achieved using only depth images in the following way.

[0024] Step 4.1: Sample from the particle swarm template to generate a segmentation normal map from the current depth image. As a mask, a six-degree-of-freedom pose particle is used to project the depth image onto the three-dimensional world. It checks whether the particle is in the active space, filters out the particle set outside the active space, and calculates the proportion of the segmented block in the L0 level voxel block. If it is less than the corresponding threshold, the pose particle is eliminated.

[0025] Step 4.2: Based on the six-DOF pose particles constrained by Step 4.1, calculate the depth image of frame t. With the candidate optimal pose in the k-th iteration fitness between fitness Greater than the optimal pose in the previous iteration The set of particles is used as the particle sampling range for the next iteration;

[0026] Step 4.3: In each iteration, set up three combinations of particle swarm templates and depth images with different resolutions, and use them sequentially in the iterations;

[0027] Step 4.4: Set the iteration termination condition as follows: the six-degree-of-freedom pose changes before and after the optimization iteration are less than the corresponding thresholds or the number of iterations is greater than the upper limit of the iteration. Repeat steps 4.2 and 4.3 until the iteration termination condition is triggered.

[0028] Furthermore, in step 5, scene surface reconstruction using depth images with sensor noise is achieved in the following way:

[0029] Step 5.1 introduces a confidence factor α based on the truncated symbolic distance field TSDF fusion, which represents the number of times the model points are observed. Weighted average in σ is an empirical value, and γ represents the normalized radial distance between the depth camera measurement error and the center of the photograph.

[0030] Step 5.2: Extract the scene surface using ray casting. When interpolating non-global voxels, if the voxel containing a point X = (X,Y,Z) is at the same level as its surrounding voxels, the interpolation function is solved using the known trilinear interpolation coefficients through the eight corner points of the voxel containing that point. If the voxel containing that point is at a different level from its surrounding voxels, the interpolation function is: F(X,Y,Z) = z1XYZ + a2XY + a3YZ + a4XZ + a5X + a6Y + a7Z + a8. Equations in the form of interpolation functions are given for the eight grid points surrounding that point, forming a system of linear equations.

[0031] On the other hand, the present invention also provides an scalable scene 3D reconstruction system based on UAV airborne positioning, which is used to implement the scalable scene 3D reconstruction method based on UAV airborne positioning as described above.

[0032] Moreover, it includes the following modules,

[0033] The first module is used to uniformly sample particles within a unit sphere in a 6D state space to obtain a particle swarm template, where any particle represents an effective rigid body motion in three-dimensional space.

[0034] The second module is used to perform surface measurements on each frame of the input depth image, project the depth image pixel by pixel into the three-dimensional space according to the camera intrinsic parameters, calculate the normal of each three-dimensional point, and calculate the segmentation normal map at the same time.

[0035] The third module is used to adaptively allocate reconstruction voxel memory in the GPU based on the correlation between the three-dimensional point normals, realize the sparse representation of the surface, and manage the allocated voxel space after the camera leaves the current reconstruction area.

[0036] The fourth module is used to track the pose of a fast-moving camera using only depth images and existing 3D models in the current GPU active space, and to calculate the camera's relative position and orientation when the current frame is captured.

[0037] The fifth module is used to construct a dense 3D point cloud model by fusing measurements with sensor noise based on the truncated symbolic distance field (TSDF) as depth image frame sequences are continuously input. When visualization is required, the 3D surface is extracted to generate a 3D mesh model.

[0038] Alternatively, it may include a processor and a memory, with the memory used to store program instructions and the processor used to call the stored instructions in the memory to execute a scalable scene 3D reconstruction method based on UAV onboard positioning as described above.

[0039] Alternatively, it may include a readable storage medium storing a computer program that, when executed, implements a scalable scene 3D reconstruction method based on UAV airborne positioning as described above.

[0040] The technical solution provided by this invention automates the creation of detailed 3D models of complex structures using swarm drones. The key difference between this invention and existing technologies lies in:

[0041] First, the camera pose estimation of the present invention is obtained through random optimization, whereas in the prior art it is usually ICP and keyframe matching.

[0042] Second, the reconstructed surface of the present invention adopts a layered sparse representation, while the prior art uses a regular voxel mesh;

[0043] Third, the deep fusion of existing technologies is simply a matter of the number of observations. This invention adds a normalized radial distance constraint on the center of the depth camera measurement error.

[0044] Accordingly, the present invention has the following advantages:

[0045] 1. Able to work in complete darkness;

[0046] 2. It can robustly estimate fast camera movement;

[0047] 3. Capable of reconstructing 3D models of large-scale scenes in real time;

[0048] 4. The entire system can run in real time on an onboard computer with limited computing resources.

[0049] The present invention is simple and convenient to implement, highly practical, and solves the problems of low practicality and inconvenience in actual application of related technologies. It can improve user experience and has significant market value. Attached Figure Description

[0050] Figure 1 This is a flowchart of an embodiment of the present invention for a UAV-borne real-time positioning and scalable scene 3D reconstruction method;

[0051] Figure 2 This is a flowchart of a method for constructing sparse representations of scene surfaces according to an embodiment of the present invention;

[0052] Figure 3 This is a flowchart of the pose estimation method according to an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram illustrating the effect of an embodiment of the present invention. Part (a) shows a drone performing autonomous 3D scanning using the method of the present invention. Parts (b) and (c) show a 3D model generated in real time and a 3D model rendered using a normal map, respectively. Different colors represent surface undulations. Detailed Implementation

[0054] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0055] See Figure 1 This invention proposes a scalable scene 3D reconstruction method based on UAV airborne positioning. The method reconstructs 3D images from color and depth images acquired by a depth camera mounted on a UAV, and includes the following steps:

[0056] Step 1: Uniformly sample the particles within a unit sphere in the 6D state space to obtain a particle swarm template, where any one particle represents an effective rigid body motion in three-dimensional space.

[0057] This invention further proposes to uniformly presample a fixed number of particles within a unit sphere in a 6D state space, where each particle represents a rigid body motion in three-dimensional space. The presampled particle set is called the particle swarm template Ω. This invention constructs multi-level particle swarm templates, which are combined with depth images of different resolutions to balance the accuracy and efficiency of pose estimation.

[0058] In the embodiment, step 1 is preferably implemented as follows: a fixed number of particles are uniformly pre-sampled within a unit sphere in the 6D state space, where each particle represents a rigid body motion in three-dimensional space, i.e., the camera pose s = (R, t) = (q). x ,q y ,q z (x, y, z), R is the rotation matrix, q x ,q y ,q z Let t = (x, y, z) be the imaginary part of three rotational quaternions. T This is a three-dimensional translation. The presampled particle set is called the particle swarm template Ω, with a center point c and a 6D axis length r = (r d ) d＝1:6 This means that the particle swarm template is sampled starting from the center of the unit sphere, i.e. This invention constructs a multi-level particle swarm template, sampling the number of particles within a unit sphere as follows: 1024, 3072, and 10240, which are then combined with depth images of different resolutions to balance the accuracy and efficiency of pose estimation.

[0059] Among them, (r d) d＝1:6 This represents the axis length of a 6D element; r d This indicates the specific length value of each dimension; d represents the dimension label. 0 represents a three-dimensional vector, and 0 represents the three-dimensional zero vector.

[0060] Step 2: Perform surface measurement on each frame of the input depth image, project the depth image pixel by pixel into the three-dimensional space according to the camera intrinsic parameters, calculate the normal of each three-dimensional point, and calculate the segmentation normal map at the same time.

[0061] The present invention further proposes that, in step 2, the depth image preprocessing method specifically includes:

[0062] Step 2.1, using the depth camera intrinsic parameter matrix K and the depth image I... d Calculate the vertex graph

[0063] Step 2.2, based on the vertex graph Calculate the corresponding normal map

[0064] Step 2.3, in the normal diagram Based on this, the segmentation normal map is calculated using the region growing method.

[0065] All three steps above are based on parallel processing in the GPU using CUDA.

[0066] In step 2 of the embodiment, the depth image preprocessing method mainly involves constructing three auxiliary maps, namely vertex maps. Normal diagram and segmentation normal map All three images are generated sequentially in parallel on the GPU using CUDA. The preferred implementation method is as follows:

[0067] Step 2.1, set the depth camera intrinsic parameter matrix to K, and convert the depth image I in two-dimensional space... d Each pixel u = Converted to vertices v in three-dimensional space i (u), the conversion method is: Finally, the vertex graph is obtained.

[0068] Where (x,y) represents the horizontal and vertical coordinates of the image pixels.

[0069] Step 2.2, based on the vertex graph Calculate the corresponding normal map The calculation method is as follows: in This represents the direction vector of the three-dimensional points on both sides of point u horizontally, and similarly... This represents the direction vectors of point u perpendicular to the three-dimensional points on either side. Normalizing the cross product result yields the normal n of the current point. i (u), where the direction of the normal points away from the center of the image, reverse the direction of the normal so that it points towards the center of the image. Points located at the edge of the depth image do not have their normals calculated.

[0070] Step 2.3: Obtain the segmentation normal map using the region growing method. normal diagram One pixel is designated as the seed pixel. The angle between the normal to its neighboring pixels and the Manhattan distance between them are calculated. If both indicators are less than the given threshold, then this pixel is accepted as a unit for region growing. In this embodiment of the invention, the normal angle threshold is preferably set to 0.4°, and the Manhattan distance threshold is the length plus the width of the image. After growing, if the number of obtained pixels is less than 2% of the total number of pixels in the image, the region is rejected; otherwise, the segmented region will be too small. In a region growing process, the seed pixel and the pixels of the segmented region are not counted repeatedly.

[0071] Step 3: Based on the correlation between the normals of the three-dimensional points, the reconstruction voxel memory is adaptively allocated in the GPU to realize the sparse representation of the surface. After the camera leaves the current reconstruction area, the allocated voxel space is managed.

[0072] In step 3, this embodiment of the invention implements a hierarchical representation of surface voxels based on a divisible voxel data structure. That is, an L-level voxel can be subdivided into 8 L-1 level voxels, and vice versa. Simultaneously, these voxels are managed using a hash structure incorporating hierarchical tree features. Specific implementations can employ existing technologies, which will not be elaborated upon here, as shown in the appendix. Figure 2 As shown, the preferred implementation method of the embodiment is as follows:

[0073] Step 3.1, Photographic Ray Construction: When a new frame of depth image is input into the reconstruction system, iterate through each pixel in the depth image, and for each depth measurement value I... d (u) Construct a ray with the photography center The method can be denoted as: in It represents the pose of the previous frame (the current frame is the k-th frame), indicating the transformation amount that converts frame k in the camera coordinate system to the global frame g, where... Let be the rotation transformation matrix. For translation transformation vector, 0 T It is represented as the transpose of the three-dimensional zero vector.

[0074] in, It is a special European style group. It is a special orthogonal group.

[0075] Step 3.2, Voxel Allocation from Coarse to Fine: Set the TSDF cutoff range to μ, along the ray... The direction is in I d (u)-μ to I d Create a line segment within the truncation band of (u)+μ, obtain the coordinates of the voxels on this line segment from thick to thin, calculate its hash value h, and check whether the voxel block b corresponding to this hash value has been allocated memory in the GPU. The hash value is calculated as follows: Among them (b) x ,b y ,b z (P1, P2, P3) = (73856093, 19349669, 83492791) are the three-dimensional coordinates of voxel block b, and (P1, P2, P3) = (73856093, 19349669, 83492791) are constants. If the voxel block b corresponding to the hash value h(b) has not been allocated memory in the GPU, then it will be allocated, that is, a new entry will be created in the hash table and the voxel block array;

[0076] In this invention, the segmentation normal map serves as a mask. Furthermore, it is proposed that voxel blocks are allocated to pixels within the segmented planar region at the coarsest resolution level; pixels outside the segmented region bypass the coarsest level and are directly allocated voxel space at the second coarser level, with a special marker implicitly pointing to the location of the sub-block in their corresponding coarsest-level hash table entry. Here, entry represents an entry, and sub-block represents a child voxel block.

[0077] Step 3.3, Surface Roughness Assessment: After voxel memory allocation is completed, the roughness of the surface where voxel block b is located is calculated. This is the execution standard for fusing and subdividing voxels. This invention uses the correlation of normals to define the surface roughness.

[0078] The example calculates the roughness r(b) of voxel block b by the correlation of normals between voxels in the voxel block. The calculation method is as follows: Where ΔF k This indicates that the portion of voxel block b containing TSDFs has k elements, Δ is the gradient operator, and ΔF k (i) represents the normal at voxel i. This represents the mean normal value of k voxels. It should be noted that the normal value of the voxels is calculated by the gradient of the TSDF value, rather than directly using the normal value in the normal map, for two reasons: First, the TSDF stored in the voxel is the result of multiple weighted fusions of observed data (see step 5.1), and has high reliability; second, the normal map obtained by directly converting the measurement values ​​from the depth image contains certain noise interference.

[0079] Step 3.4, Voxel Block Segmentation and Merging: When the voxel block roughness is greater than the segmentation threshold t s When the roughness of the sub-voxel block is less than the merging threshold t, segment the current level voxels; when the roughness of the sub-voxel block is less than the merging threshold t. m At that time, the voxels of the current level are merged. In specific implementations, a threshold value can be preset according to the situation. In this embodiment of the invention, t is set... s =0.8, t m =0.4, with voxel sizes of L0=8mm, L1=5mm, and L2=2mm, three resolution levels from coarse to fine. After segmentation and merging, the voxel resolution level is recorded.

[0080] Step 3.5, Voxel Memory Management: Define a spherical active space containing the current camera's field of view. For different depth cameras, based on their effective depth range l, set the center of the active space to be located at a distance l / 2 from the depth camera, with a radius of l. When the camera moves, causing the current scene to leave the camera's field of view, its corresponding surface voxels are also removed from the active space. This is done by marking the voxel hash indices to be removed in a hash table from fine to coarse. If several sibling voxels belonging to the same voxel need to be removed, only the parent voxel is marked. Then, the hash indices and their corresponding voxel blocks are copied and transferred to an intermediate buffer, and then passed from the intermediate buffer to the CPU for storage. Finally, these indices and voxel blocks are deleted in the GPU. The operation of transferring voxels from the CPU to the GPU is the reverse process. This operation is triggered when the camera revisits a previously reconstructed area. That is, if a voxel block is identified as "previously reconstructed," it is transferred from the host back to the GPU, returning to step 3.2 for iteration to perform depth fusion with new measurement data.

[0081] Step 4: Using only the depth image and the existing 3D model in the current GPU activity space, track the pose of the fast-moving camera and calculate the relative position and orientation of the camera when capturing the current frame.

[0082] In this embodiment, the main function of step 4 is to quickly estimate the optimal pose T of the current image by using only the depth image as input and combining it with the particle swarm template. * The flowchart is attached. Figure 3 As shown, its specific implementation method is as follows:

[0083] Step 4.1, Particle Filtering: Sampling from the particle swarm template, using the segmentation normal map generated from the current depth image. As a mask, six-DOF pose particles are used to project the depth image onto the 3D world. First, it checks if the particle is within the active space. After filtering out particle sets outside the active space, the proportion of the segmented block located in the L0-level voxel block is calculated. If it is less than a certain threshold, the pose particle is discarded. This method can quickly orient the particle to the vicinity of the optimal pose, reducing unnecessary computation and accelerating the convergence speed of iterative optimization. In specific implementations, the threshold value can be preset according to the situation.

[0084] Steps 4.2-4.3 iteratively evaluate particle fitness and update the particle swarm template to update the particle sampling range.

[0085] Step 4.2: Based on the six-DOF pose particles constrained by Step 4.1, calculate the depth image of frame t. With the candidate optimal pose in the k-th iteration fitness between The method is as follows in and Representing pose respectively The rotation and translation transformation quantities in x ij For depth images pixels, This represents the set of overlapping pixels between the depth image of frame t and the depth image of frame t-1 at the k-th iteration. The fitness... Greater than the optimal pose of the previous iteration The set of particles is used as the particle sampling range Ω for the next iteration, i.e.

[0086] Where (i,j) represents the row and column numbers of the depth image pixels;

[0087] ψ represents the calculation of the TSDF value;

[0088] τ is a constant, which is 1 by default in the examples;

[0089] Indicates the number of overlapping pixels;

[0090] exp represents taking the natural exponential function;

[0091] Ω k and Ω k+1 These represent the particle sampling ranges for the k-th and (k+1)-th iterations, respectively.

[0092] This represents the pose of the i-th particle in the k-th iteration;

[0093] f(*) represents the fitness of the calculated pose*.

[0094] Step 4.3: In each iteration, three combinations of particle swarm templates and depth images with different resolutions (1024, 1 / 8), (3072, 1 / 16), and (10240, 1 / 32) are set and used sequentially in the iteration to reduce the iteration time consumption.

[0095] Step 4.4, Optimization Termination: The condition for terminating the iteration is that the six-degree-of-freedom pose changes before and after the optimization iteration are both less than a preset threshold (preferred in the example, 1×10). -6 If the number of iterations exceeds a preset iteration limit, this embodiment of the invention preferably sets the iteration limit to 10. Repeat steps 4.2 and 4.3 until the iteration termination condition is triggered to obtain the optimal pose T*.

[0096] Step 5: As the depth image frame sequence continues to be received, the measurement values ​​with sensor noise are fused based on the truncated signed distance field (TSDF) to construct a dense 3D point cloud model. When visualization is required, the 3D surface is extracted to generate a 3D mesh model.

[0097] The embodiment obtains the optimal pose T of the current image through step 4. * Then, in step 5, scene surface reconstruction is performed by combining the depth image with sensor noise. The specific implementation method is as follows:

[0098] Step 5.1: Based on traditional TSDF fusion, this invention introduces a confidence factor α, which measures the number of times a model point is observed. Weighted average in, This represents the weight of a model point when it was last observed. In this embodiment, σ is preferably set to an empirical value of 0.6, and γ represents the normalized radial distance between the depth camera measurement error and the center of the photograph. Furthermore, the color information of the point cloud (from the RGB image) is also fused in this step.

[0099] Step 5.2: This invention uses ray casting to extract the scene surface. Unlike traditional methods, when interpolating non-global voxels (i.e., subdivided voxels), if the voxel containing a point X = (X,Y,Z) is at the same level as its surrounding voxels, the interpolation can be solved using the eight corner points of that voxel and the known trilinear interpolation coefficients. If the voxel containing that point is at a different level than its surrounding voxels, such as a point located at the boundary of different resolution levels, the interpolation function is: F(X,Y,Z) = a1XYZ + a2XY + a3YZ + a4XZ + a5X + a6Y + a7Z + a8. Equations of this form need to be given for the eight grid points surrounding this point, forming a system of linear equations. in

[0100] in,

[0101] a1…a8 represent the interpolation function coefficients;

[0102] (X1,Y1,Z1)…(X8,Y8,Z8) represents the three-dimensional coordinates of eight grid points around the interpolation location.

[0103] Appendix Figure 4 Part (a) illustrates a UAV performing autonomous 3D scanning using the method of this invention, whose position information for motion planning is derived from the pose estimated by the method in step 4 of this invention. Figure 4 Sections (b) and (c) respectively show the real-time generated 3D model and the 3D model rendered using normal maps, with different colors representing surface undulations.

[0104] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0105] In some possible embodiments, a scalable scene 3D reconstruction system based on UAV airborne positioning is provided, including the following modules:

[0106] The first module is used to uniformly sample particles within a unit sphere in a 6D state space to obtain a particle swarm template, where any particle represents an effective rigid body motion in three-dimensional space.

[0107] The second module is used to perform surface measurements on each frame of the input depth image, project the depth image pixel by pixel into the three-dimensional space according to the camera intrinsic parameters, calculate the normal of each three-dimensional point, and calculate the segmentation normal map at the same time.

[0108] The third module is used to adaptively allocate reconstruction voxel memory in the GPU based on the correlation between the three-dimensional point normals, realize the sparse representation of the surface, and manage the allocated voxel space after the camera leaves the current reconstruction area.

[0109] The fourth module is used to track the pose of a fast-moving camera using only depth images and existing 3D models in the current GPU active space, and to calculate the camera's relative position and orientation when the current frame is captured.

[0110] The fifth module is used to construct a dense 3D point cloud model by fusing measurements with sensor noise based on the truncated symbolic distance field (TSDF) as depth image frame sequences are continuously input. When visualization is required, the 3D surface is extracted to generate a 3D mesh model.

[0111] In some possible embodiments, a scalable scene 3D reconstruction system based on UAV airborne positioning is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a scalable scene 3D reconstruction method based on UAV airborne positioning as described above.

[0112] In some possible embodiments, a scalable scene 3D reconstruction system based on UAV airborne positioning is provided, including a readable storage medium storing a computer program, which, when executed, implements the scalable scene 3D reconstruction method based on UAV airborne positioning as described above.

[0113] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A scalable scene 3D reconstruction method based on UAV airborne positioning, characterized in that: 3D reconstruction is performed based on color and depth images acquired by a depth camera mounted on a drone, including the following steps: Step 1: Uniformly sample the particles within a unit sphere in the 6D state space to obtain a particle swarm template, where any one particle represents an effective rigid body motion in three-dimensional space. Step 2: Perform surface measurement on each frame of the input depth image, project the depth image pixel by pixel into the three-dimensional space according to the camera intrinsic parameters, calculate the normal of each three-dimensional point, and calculate the segmentation normal map at the same time. Step 3: Based on the correlation between the normals of the three-dimensional points, adaptively allocate the memory of the reconstructed voxels in the GPU to realize the sparse representation of the surface. After the camera leaves the current reconstruction area, manage the allocated voxel space. Step 4: Using only the depth image and the existing 3D model in the current GPU activity space, track the pose of the fast-moving camera and calculate the relative position and orientation of the camera when capturing the current frame. Step 5: As the depth image frame sequence continues to be received, the measurement values ​​with sensor noise are fused based on the truncated symbolic distance field TSDF to construct a dense 3D point cloud model. When visualization is required, the 3D surface is extracted to generate a 3D mesh model.

2. The scalable scene 3D reconstruction method based on UAV airborne positioning according to claim 1, characterized in that: In step 1, a fixed number of particles are uniformly presampled within a unit sphere in the 6D state space. Each particle represents a rigid body motion in three-dimensional space. The presampled particle set is called a particle swarm template. Multi-level particle swarm templates are constructed and used in combination with depth images of different resolutions to support pose estimation.

3. The scalable scene 3D reconstruction method based on UAV airborne positioning according to claim 1, characterized in that: In step 2, depth image preprocessing is performed in the following manner: The depth camera intrinsic parameter matrix K and the depth image I d Calculate the vertex graph Step based on vertex graph Calculate the corresponding normal map In the normal diagram Based on this, the segmentation normal map is calculated using the region growing method.

4. The scalable scene 3D reconstruction method based on UAV airborne positioning according to claim 1, characterized in that: In step 3, the sparse representation of the scene surface with normal correlation constraints is performed in the following way. When a new frame of depth imagery is input into the reconstruction system, it iterates through each pixel in the depth imagery, and for each depth measurement value I... d (u) Construct a ray with the photography center Set the TSDF cutoff range to μ, along the ray The direction is in I d (u)-μ to I d Create a line segment within the truncation band of (u)+μ, obtain the coordinates of the voxels on the line segment from thick to thin, calculate its hash value h, and check whether the voxel block b corresponding to the hash value has been allocated memory in the GPU. If not, allocate memory for it and create new entries in the hash table and voxel block array. Using the segmentation normal map as a mask, voxel blocks are assigned to pixels within the segmentation plane region at the coarsest level of resolution. Pixels outside the segmented region skip the coarsest level and are allocated voxel space directly at the second coarsest level. In their corresponding coarsest level hash table entry, a special marker implicitly points to the location of the sub-block. After the voxel memory allocation is completed, the roughness r(b) of the surface where the voxel block b is located is calculated based on the correlation of the normals, and is used as the execution standard for fusing and subdividing voxels. When roughness is greater than the segmentation threshold t s At that time, segment the voxels of the current level; When roughness is less than the merging threshold t m At that time, merge the voxels of the current level; Define a spherical activity space containing the current camera's field of view. For different depth cameras, based on the effective depth range l, set the center of the activity space to be located at a distance l / 2 from the depth camera, with a radius of l. When the camera moves, causing the current scene to leave the camera's field of view, its corresponding surface voxels are also moved out of the activity space, and the voxels are transferred from the CPU to the GPU. When the camera revisits a previously reconstructed area, the voxels are transferred from the GPU to the CPU.

5. The scalable scene 3D reconstruction method based on UAV airborne positioning according to claim 1, characterized in that: In step 4, fast camera pose estimation using only depth images is achieved in the following way. Step 4.1: Sample from the particle swarm template to generate a segmentation normal map from the current depth image. As a mask, a six-degree-of-freedom pose particle is used to project the depth image onto the three-dimensional world. It checks whether the particle is in the active space, filters out the particle set outside the active space, and calculates the proportion of the segmented block in the L0 level voxel block. If it is less than the corresponding threshold, the pose particle is eliminated. Step 4.2: Based on the six-DOF pose particles constrained by Step 4.1, calculate the depth image of frame t. With the candidate optimal pose in the k-th iteration fitness between fitness Greater than the optimal pose in the previous iteration The set of particles is used as the particle sampling range for the next iteration; Step 4.3: In each iteration, set up three combinations of particle swarm templates and depth images with different resolutions, and use them sequentially in the iterations; Step 4.4: Set the iteration termination condition as follows: the six-degree-of-freedom pose changes before and after the optimization iteration are less than the corresponding thresholds or the number of iterations is greater than the upper limit of the iteration. Repeat steps 4.2 and 4.3 until the iteration termination condition is triggered.

6. The scalable scene 3D reconstruction method based on UAV airborne positioning according to claim 1, characterized in that: In step 5, scene surface reconstruction using depth images with sensor noise is achieved in the following way: Step 5.1 introduces a confidence factor α based on the truncated symbolic distance field TSDF fusion, which represents the number of times the model points are observed. Weighted average in σ is an empirical value, and γ represents the normalized radial distance between the depth camera measurement error and the center of the photograph. Step 5.2: Extract the scene surface using ray casting. When interpolating non-global voxels, if the voxel containing a point X = (X,Y,Z) is at the same level as its surrounding voxels, the interpolation function is solved using the known trilinear interpolation coefficients through the eight corner points of the voxel containing that point. If the voxel containing that point is at a different level from its surrounding voxels, the interpolation function is: F(X,Y,Z) = z1XYZ + a2XY + a3YZ + a4XZ + a5X + a6Y + a7Z + a8. Equations in the form of interpolation functions are given for the eight grid points surrounding that point, forming a system of linear equations.

7. A scalable scene 3D reconstruction system based on UAV airborne positioning, characterized in that: This method is used to implement a scalable scene 3D reconstruction method based on UAV airborne positioning as described in any one of claims 1-6.

8. The scalable scene 3D reconstruction system based on UAV airborne positioning according to claim 7, characterized in that: Includes the following modules, The first module is used to uniformly sample particles within a unit sphere in a 6D state space to obtain a particle swarm template, where any particle represents an effective rigid body motion in three-dimensional space. The second module is used to perform surface measurements on each frame of the input depth image, project the depth image pixel by pixel into the three-dimensional space according to the camera intrinsic parameters, calculate the normal of each three-dimensional point, and calculate the segmentation normal map at the same time. The third module is used to adaptively allocate reconstruction voxel memory in the GPU based on the correlation between the three-dimensional point normals, realize the sparse representation of the surface, and manage the allocated voxel space after the camera leaves the current reconstruction area. The fourth module is used to track the pose of a fast-moving camera using only depth images and existing 3D models in the current GPU activity space, and to calculate the relative position and orientation of the camera when the current frame is captured. The fifth module is used to construct a dense 3D point cloud model by fusing measurements with sensor noise based on the truncated symbolic distance field (TSDF) as depth image frame sequences are continuously input. When visualization is required, the 3D surface is extracted to generate a 3D mesh model.

9. The scalable scene 3D reconstruction system based on UAV airborne positioning according to claim 7, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the scalable scene 3D reconstruction method based on UAV airborne positioning as described in any one of claims 1-6.

10. The scalable scene 3D reconstruction system based on UAV airborne positioning according to claim 7, characterized in that: It includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements a scalable scene 3D reconstruction method based on UAV airborne positioning as described in any one of claims 1-6.

Images