A method for constructing a training dataset for true depth estimation based on 3D mesh and 3D Gaussian splashing

Abstract

This invention discloses a method for constructing a realistic depth estimation training dataset based on a 3D mesh and 3D Gaussian splashing, relating to the field of UAV aerial surveying technology. The method involves acquiring aerial image sequences using a visible light camera mounted on a UAV platform, and filtering and preprocessing the image quality; generating a dense point cloud and a 3D mesh; training and generating a 3DGS model based on camera pose and image sequences consistent with the 3D mesh; unifying the 3D mesh, camera pose, and 3DGS model to the same world coordinate system; and using Z-buffer rasterization to output pixel-by-pixel depth and validity masks; generating RGB images of the corresponding viewpoint in the 3DGS renderer; and encapsulating the dataset according to a recommended data organization structure. This invention can accurately generate realistic RGB and geometrically accurate depth, with strict pixel alignment between the RGB images and depth maps, adapting to the needs of real-world application scenarios such as UAV aerial surveying and actual terrain mapping.

Description

Technical Field

[0001] This invention belongs to the field of UAV aerial surveying technology, specifically relating to a method for constructing a training dataset for true depth estimation based on a three-dimensional mesh and three-dimensional Gaussian splashing. Background Technology

[0002] Depth estimation is a core foundational technology in fields such as computer vision, UAV aerial surveying, and 3D reconstruction. It obtains 3D depth information of a scene through image analysis and is widely used in many scenarios such as autonomous driving, terrain mapping, virtual reality, and scene understanding. The training effect of depth estimation models is highly dependent on pixel-level strictly aligned RGB image and depth map sample pairs, and has stringent requirements for sample density, geometric accuracy, realism, and scale consistency. Especially in large-scale real-world application scenarios such as UAV aerial surveying, high-quality depth estimation training datasets become a key prerequisite for improving the model's generalization ability and actual accuracy.

[0003] With the development of UAV aerial surveying and 3D reconstruction technologies, in order to improve the cost and difficulty of collecting real-world scene data, the current construction of depth estimation training datasets mainly involves building virtual scenes using simulation engines and directly generating synthetic RGB-Depth sample pairs for model training. However, although this approach can achieve perfect alignment between RGB and depth maps and generate samples on a large scale, the simulated images differ significantly from the real world in terms of reflection characteristics, vegetation details, aerial large-scene textures, and lighting changes. Depth estimation models trained based on synthetic data are prone to insufficient generalization ability in real-world scenarios, leading to a significant decrease in the accuracy of depth estimation models and making it difficult to adapt to the needs of real-world application scenarios such as UAV aerial surveying and actual terrain mapping. Summary of the Invention

[0004] The purpose of this invention is to provide a method for constructing a realistic depth estimation training dataset based on a 3D mesh and 3D Gaussian splash, which can accurately generate realistic RGB and geometrically accurate depth, and the pixels of the RGB image and the depth map are strictly aligned, adapting to the needs of real application scenarios such as UAV aerial surveying and actual terrain mapping, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for constructing a true depth estimation training dataset based on 3D meshes and 3D Gaussian splashes includes the following steps:

[0007] S1. The UAV platform is equipped with a visible light camera to collect aerial image sequences and to screen and preprocess the image quality.

[0008] S2. The preprocessed images are subjected to aerial triangulation and multi-view geometric reconstruction using aerial surveying software to generate dense point clouds and 3D meshes of the scene, and the camera pose corresponding to each image is exported at the same time.

[0009] S3. Train and generate a 3DGS model based on camera pose and image sequence consistent with the 3D mesh, and impose quality constraints on the 3DGS model.

[0010] S4. Unify the 3D mesh, camera pose, and 3DGS model into the same world coordinate system and clearly define the coordinate axis conventions;

[0011] S5. Import the unified coordinate 3D mesh into the rendering engine, set the projection matrix and view matrix for each camera view, and use Z-buffer rasterization to output pixel-by-pixel depth and validity mask to obtain the Mesh rendering depth map.

[0012] S6. Using the same camera intrinsic matrix and camera pose as the Mesh rendering depth map rendering, generate the corresponding RGB image in the 3DGS renderer.

[0013] S7. Package the dataset according to the recommended data organization structure, then filter and clean the samples by indicators such as reprojection consistency and invalid region ratio, and divide the training, validation and test sets.

[0014] Preferably, the visible light camera is a camera capable of acquiring RGB images and synchronizing with GNSS or IMU time. When acquiring aerial image sequences, it acquires images along a predetermined flight path and records the timestamp, GNSS coordinates, and IMU attitude information for each frame. During image quality screening, blurred or abnormally exposed frames are removed. Image distortion is corrected as needed, and equivalent distortion-free camera intrinsic parameters are calculated. The resolution and pixel coordinate origin of all images are defined uniformly.

[0015] Preferably, the aerial survey software is the DJI-Terra software system used for aerial survey processing, aerial triangulation, point cloud or network reconstruction, and camera pose export. During aerial triangulation, feature points are first extracted from the image, corresponding feature points are identified between images with different degrees of overlap, 2D-2D geometric constraints are established, and then the solution is optimized by bundle adjustment. Combined with the initial GNSS or IMU pose of the image, the camera extrinsic and intrinsic parameters of each image are accurately solved, and sparse point clouds of the scene are generated to complete aerial triangulation.

[0016] In multi-view geometric reconstruction, the disparity between images is calculated using a dense matching algorithm based on the sparse point cloud obtained from aerial triangulation and the camera pose, generating a dense point cloud of the scene. Then, surface reconstruction and triangulation are performed on the dense point cloud to transform the discrete point cloud into a three-dimensional mesh composed of triangular facets, thus completing the multi-view geometric reconstruction.

[0017] Preferably, the reconstructed 3D mesh needs to undergo geometric cleaning and repair, including removing discrete group patches, filling holes, and unifying the normal direction.

[0018] Preferably, the same world coordinate system is a geographic reference coordinate system of WGS84 or UTM-Zone50N. If there is a difference between the reconstructed coordinate system and the GNSS coordinate system, the similarity transformation is solved by using the control point or camera GPS center point set to unify the reconstructed point cloud, grid vertices, 3DGS Gaussian center points, and camera extrinsic parameters to the same world coordinate system.

[0019] Preferably, a virtual camera is generated to expand the training viewpoint in a unified world coordinate system. The camera center is randomly or grid-sampled within the flight path coverage area, and the height is perturbed around the reference height. The virtual camera pose is parameterized by Euler angles, and the Euler angles are converted into a rotation matrix to generate the camera pose of the new training viewpoint.

[0020] Preferably, in step S5, the depth of each pixel is defined as the optical axis depth or Euclidean distance and is specified in the metadata. A valid mask is output synchronously, marking valid depth pixels and invalid regions such as holes, out-of-range, and low-confidence regions. Then, a consistency check is performed on the depth map, the depth is back-projected into a point cloud, the alignment error with the original mesh or point cloud is statistically analyzed, and abnormal depth pixels with abrupt changes or isolated noise are removed or marked.

[0021] Preferably, in step S6, the RGB image is output through additional channels for occlusion and dynamic object filtering, the additional channels including alpha or visibility channels, normal channels, and uncertainty channels.

[0022] Preferably, the recommended data organization structure includes an images folder, a depth folder, a mask folder, poses.json, and scen_meta.json. After filtering and cleaning, the dataset is divided into training set, validation set, and test set according to scene or flight path, and the division strategy is recorded.

[0023] Preferably, the 3DGS model performs the following process for quality constraints:

[0024] A1. During the training of the 3DGS model, set upper and lower scale thresholds, monitor and prune Gaussian voxels that exceed the upper and lower thresholds in real time, and constrain the spatial extension range of Gaussian voxels to match the actual geometric features of the scene.

[0025] A2. Introduce an L1 sparse regularization term into the joint loss function of the 3DGS model to penalize Gaussian voxels without effective image pixel support, dynamically adjust the weight of the regularization term, remove or shrink invalid Gaussian voxels in the space, and suppress the generation of floating artifacts.

[0026] A3. Divide the original aerial image sequence into training viewpoint and unfurled viewpoint. Train the 3DGS model using only the training viewpoint. Render the unfurled viewpoint using the trained 3DGS model. Calculate the PSNR, SSIM, and LPIPS similarity indices between the rendered image and the original unfurled viewpoint image. Combine the visual effects to check the 3DGS model's reconstruction capability. If the preset threshold is not met, readjust the Gaussian scale threshold and the sparse regularization term weights and iterate until the 3DGS model meets the fidelity requirements.

[0027] The present invention proposes a method for constructing a true depth estimation training dataset based on a 3D mesh and 3D Gaussian splashing, which has the following advantages compared with existing technologies:

[0028] 1. This invention uses real visible light image sequences captured by UAVs as the data source, rather than virtual scenes from simulation engines. This ensures that RGB images are highly consistent with the real world in terms of reflection characteristics, vegetation details, textures of large-scale aerial scenes, and lighting changes. At the same time, it outputs pixel-by-pixel depth maps through Z-buffer rasterization of a 3D mesh. The depth information is obtained based on multi-view geometric reconstruction of the real scene, with high geometric accuracy, which can be directly adapted to the depth estimation model training needs of real scenes such as UAV aerial surveying and terrain mapping.

[0029] 2. This invention relies on an unmanned aerial vehicle (UAV) platform to collect aerial images and combines aerial surveying software to complete aerial triangulation and 3D reconstruction. The technical approach is in line with the actual application process of UAV aerial surveying. The generated dataset is adapted to the needs of large-scale scenarios such as UAV aerial surveying, making up for the shortcomings of traditional depth datasets which are mostly applicable to small-scale scenarios and difficult to adapt to large-scale aerial surveying, and expanding the practical application boundaries of depth estimation models.

[0030] 3. The three-dimensional mesh of this invention is obtained by reconstructing and triangulating the surface of dense point cloud of real scene. Its rasterized output depth map has extremely high geometric accuracy and can accurately reflect the three-dimensional spatial structure of the scene. The 3DGS model is good at restoring real visual texture and details, and the generated RGB image visual effect is closer to the real aerial photography scene. The combination of the two allows the dataset to have both geometrically accurate depth information and high-fidelity visual information.

[0031] 4. This invention is based on real data collected by UAV aerial surveys. It combines 3D mesh and 3DGS model to unify them into the same world coordinate system, ensuring the consistency of data scale and adapting to the needs of large-scale UAV aerial surveys. At the same time, it can generate virtual camera centers by random / grid sampling within the flight path coverage area. Combined with elevation perturbation and Euler angles, it can generate camera poses for new training perspectives. It can expand the perspective samples based on real data, improve the sample density and scene coverage of the dataset, and further improve the generalization ability of the depth estimation model. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the process of the present invention;

[0033] Figure 2 This is a schematic diagram of the coordinate system and scale calibration of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the alignment mechanism between Mesh rendering depth and 3DGS rendering RGB in this invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention provides, for example Figure 1-3 The method shown here for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing includes the following steps:

[0037] S1. The UAV platform is equipped with a visible light camera to collect aerial image sequences and to screen and preprocess the image quality.

[0038] The visible light camera is capable of acquiring RGB images and synchronizing with GNSS or IMU time. During aerial image sequence acquisition, it follows a predetermined flight path to ensure forward or lateral overlap and improve reconstruction quality. It also records the timestamp, GNSS coordinates, and IMU attitude information for each frame. During image quality screening, blurred and abnormally exposed frames are removed. Image distortion is corrected as needed, and the equivalent distortion-free camera intrinsic parameter K' is calculated. The resolution and pixel coordinate origin of all images are unified.

[0039] The data acquisition platform was a DJI Motrice 400 (M400) drone, equipped with a visible light camera on a Zenmuse L2 sensor to acquire RGB data and record GNSS / IMU data;

[0040] S2. The preprocessed images are subjected to aerial triangulation and multi-view geometric reconstruction using aerial surveying software to generate dense point clouds and 3D meshes of the scene. At the same time, the camera poses corresponding to each image are exported, including the rotation matrix R, translation vector t, and camera center C.

[0041] The aerial survey software is the DJI-Terra software system used for aerial survey processing, aerial triangulation, point cloud or network reconstruction, and camera pose export. During aerial triangulation, feature points are first extracted from the image, and corresponding feature points are identified between images with different degrees of overlap to establish 2D-2D geometric constraints. Then, the solution is optimized by bundle adjustment. Combined with the initial GNSS or IMU pose of the image, the camera extrinsic and intrinsic parameters of each image are accurately solved. At the same time, sparse point clouds of the scene are generated to complete aerial triangulation.

[0042] In multi-view geometry reconstruction, the disparity between images is calculated using a dense matching algorithm based on the sparse point cloud obtained from aerial triangulation and the camera pose, generating a dense point cloud of the scene. Then, surface reconstruction and triangulation are performed on the dense point cloud to transform the discrete point cloud into a three-dimensional mesh composed of triangular facets, thus completing multi-view geometry reconstruction and improving the stability of depth rendering.

[0043] S3. A 3DGS model is trained and generated based on camera pose and image sequence consistent with the 3D mesh. This 3DGS model is used as a high-fidelity RGB rendering source, and the quality of the 3DGS model is constrained. The fidelity is evaluated by rendering with the viewpoint left open.

[0044] The 3DGS model undergoes the following process for quality constraints:

[0045] A1. During the training of the 3DGS model, set upper and lower scale thresholds, monitor and prune Gaussian voxels that exceed the upper and lower thresholds in real time, and constrain the spatial extension range of Gaussian voxels to match the actual geometric features of the scene.

[0046] A2. Introduce an L1 sparse regularization term into the joint loss function of the 3DGS model to penalize Gaussian voxels without effective image pixel support, dynamically adjust the weight of the regularization term, remove or shrink invalid Gaussian voxels in the space, and suppress the generation of floating artifacts.

[0047] A3. Divide the original aerial image sequence into training viewpoint and unfurled viewpoint. Train the 3DGS model using only the training viewpoint. Render the unfurled viewpoint using the trained 3DGS model. Calculate the PSNR, SSIM, and LPIPS similarity indices between the rendered image and the original unfurled viewpoint image. Combine the visual effects to check the 3DGS model's reconstruction capability. If the preset threshold is not met, readjust the Gaussian scale threshold and the sparse regularization term weights and iterate until the 3DGS model meets the fidelity requirements.

[0048] The reconstructed 3D mesh needs to undergo geometric cleaning and repair, including removing discrete group patches, filling holes, and unifying the normal direction.

[0049] S4. Unify the 3D mesh, camera pose, and 3DGS model to the same world coordinate system, clearly defining coordinate axis conventions, including whether the coordinate system is right-handed or left-handed, the Z-axis orientation, and the camera optical axis direction. Record these conventions in the metadata to avoid inconsistencies between the rendering engine and reconstruction software coordinates.

[0050] like Figure 2 As shown, the same world coordinate system is the geographic reference coordinate system of WGS84 or UTM-Zone50N, with the unit being meters. If there is a difference between the reconstructed coordinate system and the GNSS coordinate system, the similarity transformation is solved by using the control point or camera GPS center point set to unify the reconstructed point cloud, grid vertices, 3DGS Gaussian center points, and camera extrinsic parameters to the same world coordinate system.

[0051] Specifically, the 7DoF similarity transformation algorithm is used to solve for the unified transformation matrix T. The specific solution process is as follows:

[0052] First, arbitrarily reconstruct the lower point of the coordinate system. Mapped to points in a unified world coordinate system The mapping formula is:

[0053] ,

[0054] In the formula, As a scale factor, It is a three-dimensional rotation matrix. It is a translation vector; These represent the coordinates of a three-dimensional point in the reconstructed coordinate system and the unified world coordinate system, respectively.

[0055] The remapping formula is written in homogeneous coordinate form, thus ensuring that the camera extrinsic parameters, mesh vertices, and DGS Gaussian center are all expressed in the same coordinate system. The homogeneous coordinate form expression is:

[0056]

[0057] in, To unify the coordinate transformation matrix, it consists of scale-rotation terms. With translation terms constitute, for Zero vector;

[0058] In a unified world coordinate system, a virtual camera is generated to expand the training viewpoint. The camera center is randomly or grid-sampled within the flight path coverage area, and the height is perturbed around the reference height. The virtual camera pose is parameterized by Euler angles, and the Euler angles are converted into a rotation matrix to generate the camera pose of the new training viewpoint.

[0059] Specifically, Euler angles include roll, pitch, and yaw. Yaw is uniformly sampled from 0 to 360°, pitch is sampled from -45° to +45°, and roll is sampled from -10° to +10° (threshold is configurable). The formula for converting Euler angles to a rotation matrix is:

[0060] ,

[0061] in, They represent circumference respectively. , , Euler angle rotation of the axis (either in degrees or radians must be consistent in the implementation); This is the rotation matrix corresponding to the virtual camera pose. Let be the basic rotation matrix about each coordinate axis, and this formula gives a commonly used combination order (first...). Axis roll, then Axis pitch, final (axis yaw), used to parametrically map Euler angles to three-dimensional rotations.

[0062]

[0063]

[0064]

[0065] in, Corresponding to The values ​​are as follows; the sampling ranges for each angle are set as described above, and are respectively... , , (Its threshold is configurable).

[0066] Virtual Camera Center It can be obtained by projecting geographic coordinates onto UTM. and elevation The composition, specifically the formula, is as follows:

[0067]

[0068] in For the camera center coordinates in a unified world coordinate system (e.g., UTM); These are the eastward and northward coordinate components of the UTM, respectively; Elevation of the camera center; For reference altitude (such as average altitude of the route or a given reference altitude); The term represents a high-order disturbance and follows an interval... Uniform distribution on Control the amplitude of the disturbance.

[0069] S5. Import the unified coordinate 3D mesh into the rendering engine, which includes OpenGL, Blender, and EGL off-screen rendering. Set the projection matrix and view matrix for each camera view. The projection matrix is ​​derived from the camera intrinsic parameters, and the view matrix is ​​derived from the camera pose. Use Z-buffer rasterization to output pixel-wise depth and effective mask. The effective mask includes unnamed mesh pixels, near and far clipping, and hole regions. Backproject the pixel-wise depth into a point cloud, and perform alignment error statistics with the mesh or point cloud. Remove or mark abnormal depths for abrupt changes or isolated noise to obtain the Mesh rendering depth map.

[0070] The depth of pixel-by-pixel depth is defined as optical axis depth or Euclidean distance and is specified in the metadata. A valid mask is output synchronously to mark valid depth pixels and invalid regions such as holes, out-of-range, and low-confidence regions. Then, a consistency check is performed on the depth map, the depth is back-projected into a point cloud, the alignment error with the original mesh or point cloud is calculated, and abnormal depth pixels with abrupt changes or isolated noise are removed or marked.

[0071] Specifically, during camera model and coordinate transformation, let the world coordinate system be W, the camera coordinate system be C, and the pixel coordinate system be I. The expression for the camera intrinsic parameter matrix K (without distortion) is:

[0072]

[0073] Where K is the camera's native distortion-free intrinsic parameter matrix. The equivalent focal length in pixels (corresponding to the scale factors in the horizontal and vertical directions, respectively). Principal point coordinates;

[0074] World to camera coordinates (represented by the camera center):

[0075] Let the center of the camera be... World Point The rotation from the world coordinate system to the camera coordinate system is Then the coordinates of the world point in the camera coordinate system satisfy

[0076] ,

[0077] in, Let the point be the three-dimensional coordinates in the camera coordinate system. The position of the camera center in the world coordinate system. Describing the directional transformation from the world coordinate system to the camera coordinate system, equation (1.1) can be equivalently written as an extrinsic parameter translation form.

[0078]

[0079] in, This is the translation term of the world-to-camera transformation.

[0080] The pinhole is projected onto pixel coordinates, making

[0081]

[0082] Then the corresponding pixel homogeneous coordinates satisfy

[0083]

[0084] in, These are the three-dimensional components of the point in the camera coordinate system. The depth component is along the camera's optical axis. This indicates that they are equal in a homogeneous sense (with a non-zero scale factor difference).

[0085] Furthermore, the optical axis depth is expressed as: ,

[0086] Euclidean distance depth is expressed as: ;

[0087] In the formula, These are the three-dimensional components of the point in the camera coordinate system. The depth component is along the camera's optical axis. Coordinates are Pixel depth value;

[0088] Specify the selected depth definition and its unit (meters) in the metadata to ensure consistency between the rendered output and subsequent geometry / learning modules.

[0089] S6. Using the same camera intrinsic matrix and camera pose as the Mesh rendering depth map rendering, generate the corresponding RGB image in the 3DGS renderer. To ensure strict alignment, the RGB image rendering resolution, principal point definition, pixel origin and depth are consistent.

[0090] The RGB image is output through additional channels for occlusion and dynamic object filtering. These additional channels include an alpha or visibility channel, a normal channel, and an uncertainty channel, such as... Figure 3 The diagram shows the alignment mechanism between Mesh rendering depth and 3DGS rendering RGB.

[0091] S7. Package the dataset according to the recommended data organization structure, then filter and clean the samples by indicators such as reprojection consistency and invalid region ratio, and divide the training, validation and test sets.

[0092] The recommended data organization structure includes the images folder, depth folder, mask folder, poses.json, and scen_meta.json. After filtering and cleaning, the dataset is divided into training set, validation set and test set according to scene or route, and the division strategy is recorded to ensure that the evaluation process is reproducible.

[0093] The images folder contains images / xxxx.png, which is an RGB image rendered by 3DGS (or has the same name as the original image), with the same resolution and depth.

[0094] The depth folder contains depth / xxxx.exr or depth / xxxx.npy, a depth map rendered by the Mesh, float32, in meters; lossless formats such as EXR / NPY are recommended.

[0095] The mask folder contains mask / xxxx.png: Validity mask: 0 invalid / holey, 1 valid (expandable multi-level labels);

[0096] poses.json contains the camera extrinsic parameters (R,t or 4×4 homogeneous matrix), intrinsic parameters K / K' (K is the camera's native distortion-free intrinsic parameter matrix, K' is the equivalent distortion-free intrinsic parameter matrix), coordinate system convention, timestamp, GPS, etc. for each frame;

[0097] The scen_meta.json file contains scene-level information such as UTM area code, EPSG, scale, reconstruction parameters, and quality statistics.

[0098] For example, sample screening and cleaning methods include: ① reprojection consistency threshold; ② depth missing rate threshold; ③ RGB rendering and original image PSNR / SSIM constraints (optional); ④ dynamic object region mask filtering;

[0099] Example of a cleaning rule: Define the percentage of black or transparent areas as follows:

[0100] ,

[0101] in, Indicates satisfaction or The number of pixels, Represents the total number of pixels in the image. This characterizes the proportion of black / transparent pixels in the entire image. Based on this metric, when The corresponding sample is removed at certain times, and the threshold can be configured according to the data quality requirements.

[0102] If an invalid mask exists, the proportion of the invalid region can also be defined as follows:

[0103] ,

[0104] in This represents the number of invalid mask pixels. This indicates the proportion of invalid regions in the entire image, and a method is adopted accordingly. The same screening strategy was used for sample cleaning.

[0105] In summary, by leveraging the geometric accuracy of meshes and the high realism of 3DGS rendering capabilities, we achieve "dense, accurate, and aligned" RGB-Depth sample pairs, reducing the sparsity and misalignment issues in point cloud projection. By unifying to WGS84 / UTM and clearly defining coordinate axis conventions, the data possesses traceable geographic references and scale consistency, facilitating cross-line / cross-scene fusion and reproduction. By generating depth and RGB through rendering, large-scale datasets can be built without additional depth sensors, reducing costs and enabling repeated rendering to generate multi-resolution, multi-view augmented data. Finally, we output masks and quality metrics to facilitate filtering of holes / low-confidence regions, improving model robustness and accuracy.

[0106] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing, characterized in that: Includes the following steps: S1. The UAV platform is equipped with a visible light camera to collect aerial image sequences and to screen and preprocess the image quality. S2. The preprocessed images are subjected to aerial triangulation and multi-view geometric reconstruction using aerial surveying software to generate dense point clouds and 3D meshes of the scene, and the camera pose corresponding to each image is exported at the same time. S3. Train and generate a 3DGS model based on camera pose and image sequence consistent with the 3D mesh, and impose quality constraints on the 3DGS model. S4. Unify the 3D mesh, camera pose, and 3DGS model into the same world coordinate system and clearly define the coordinate axis conventions; S5. Import the unified coordinate 3D mesh into the rendering engine, set the projection matrix and view matrix for each camera view, and use Z-buffer rasterization to output pixel-by-pixel depth and validity mask to obtain the Mesh rendering depth map. S6. Using the same camera intrinsic matrix and camera pose as the Mesh rendering depth map rendering, generate the corresponding RGB image in the 3DGS renderer. S7. Package the dataset according to the recommended data organization structure, then filter and clean the samples by indicators such as reprojection consistency and invalid region ratio, and divide the training, validation and test sets.

2. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 1, characterized in that: The visible light camera is capable of acquiring RGB images and synchronizing with GNSS or IMU time. During aerial image sequence acquisition, it acquires images along a predetermined flight path and records the timestamp, GNSS coordinates, and IMU attitude information for each frame. During image quality screening, blurred and abnormally exposed frames are removed. Image distortion is corrected as needed, and equivalent distortion-free camera intrinsic parameters are calculated. The resolution and pixel coordinate origin of all images are defined uniformly.

3. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 1, characterized in that: The aerial survey software is the DJI-Terra software system used for aerial survey processing, aerial triangulation, point cloud or network reconstruction, and camera pose export. During aerial triangulation, feature points are first extracted from the image, and corresponding feature points are identified between images with different degrees of overlap to establish 2D-2D geometric constraints. Then, the solution is optimized by bundle adjustment. Combined with the initial GNSS or IMU pose of the image, the camera extrinsic and intrinsic parameters of each image are accurately solved. At the same time, sparse point clouds of the scene are generated to complete aerial triangulation. In multi-view geometric reconstruction, the disparity between images is calculated using a dense matching algorithm based on the sparse point cloud obtained from aerial triangulation and the camera pose, generating a dense point cloud of the scene. Then, surface reconstruction and triangulation are performed on the dense point cloud to transform the discrete point cloud into a three-dimensional mesh composed of triangular facets, thus completing the multi-view geometric reconstruction.

4. The method for constructing a true depth estimation training dataset based on a 3D mesh and 3D Gaussian splashing as described in claim 3, characterized in that: The reconstructed 3D mesh needs to undergo geometric cleaning and repair, including removing discrete group patches, filling holes, and unifying the normal direction.

5. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 1, characterized in that: The same world coordinate system is the geographic reference coordinate system of WGS84 or UTM-Zone50N. If there is a difference between the reconstructed coordinate system and the GNSS coordinate system, the similarity transformation is solved by using the control point or camera GPS center point set to unify the reconstructed point cloud, grid vertices, 3DGS Gaussian center points, and camera extrinsic parameters to the same world coordinate system.

6. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 5, characterized in that: A virtual camera is generated to expand the training viewpoint in a unified world coordinate system. The camera center is randomly or grid-sampled within the flight path coverage area, and the height is perturbed around the reference height. The virtual camera pose is parameterized by Euler angles, and the Euler angles are converted into rotation matrices to generate the camera pose of the new training viewpoint.

7. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 1, characterized in that: In step S5, the depth of pixel-by-pixel depth is defined as optical axis depth or Euclidean distance and is specified in the metadata. A valid mask is output synchronously, marking valid depth pixels and invalid regions such as holes, out-of-range, and low-confidence regions. Then, a consistency check is performed on the depth map, the depth is back-projected into a point cloud, the alignment error with the original mesh or point cloud is calculated, and abnormal depth pixels with abrupt changes or isolated noise are removed or marked.

8. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 1, characterized in that: In step S6, the RGB image is output through additional channels for occlusion and dynamic object filtering. The additional channels include an alpha or visibility channel, a normal channel, and an uncertainty channel.

9. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 1, characterized in that: The recommended data organization structure includes the images folder, depth folder, mask folder, poses.json, and scen_meta.json. After filtering and cleaning, the dataset is divided into training set, validation set, and test set according to scene or flight path, and the division strategy is recorded.

10. The method for constructing a training dataset for true depth estimation based on a 3D mesh and 3D Gaussian splashing as described in claim 1, characterized in that: The 3DGS model undergoes the following process for quality constraints: A1. During the training of the 3DGS model, set upper and lower scale thresholds, monitor and prune Gaussian voxels that exceed the upper and lower thresholds in real time, and constrain the spatial extension range of Gaussian voxels to match the actual geometric features of the scene. A2. Introduce an L1 sparse regularization term into the joint loss function of the 3DGS model to penalize Gaussian voxels without effective image pixel support, dynamically adjust the weight of the regularization term, remove or shrink invalid Gaussian voxels in the space, and suppress the generation of floating artifacts. A3. Divide the original aerial image sequence into training viewpoint and unfurled viewpoint. Train the 3DGS model using only the training viewpoint. Render the unfurled viewpoint using the trained 3DGS model. Calculate the PSNR, SSIM, and LPIPS similarity indices between the rendered image and the original unfurled viewpoint image. Combine the visual effects to check the 3DGS model's reconstruction capability. If the preset threshold is not met, readjust the Gaussian scale threshold and the sparse regularization term weights and iterate until the 3DGS model meets the fidelity requirements.

Images