Unmanned aerial vehicle oblique image three-dimensional reconstruction optimization method based on deep learning

Abstract

This invention discloses a deep learning-based optimization method for 3D reconstruction of UAV oblique imagery. The method includes image acquisition, key view pair selection, generation of multi-scale depth prior maps, calculation of oblique consistency gating coefficients, generation of initial depth maps, depth map refinement, semantically enhanced dense point cloud reconstruction, and adaptive surface reconstruction. This invention relates to the field of 3D reconstruction technology, specifically to a deep learning-based optimization method for 3D reconstruction of UAV oblique imagery. This scheme improves matching reliability and depth estimation accuracy by constructing an oblique consistency gating mechanism to dynamically weight cross-view matching costs; it also improves reconstruction accuracy and detail representation in complex structural regions by constructing a deformable attention refinement network to adaptively optimize depth estimation; and it achieves adaptive allocation of reconstruction accuracy by constructing a semantic curvature joint adaptive octree, thereby improving 3D model reconstruction efficiency and model quality.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, specifically to a deep learning-based optimization method for 3D reconstruction of UAV oblique images. Background Technology

[0002] With the rapid development of UAV platforms, oblique photogrammetry technology, and deep learning algorithms, 3D reconstruction technology based on UAV oblique imagery has been widely applied in fields such as smart city construction, natural resource surveys, digital twins, building information modeling, emergency mapping, and cultural heritage protection. Compared with traditional manual mapping methods, UAV oblique photogrammetry can quickly acquire multi-angle image information of target areas. Combined with deep learning, multi-view stereo reconstruction, and surface reconstruction technologies, it can realize the automated construction of large-scale, high-precision 3D models, and has become an important technological development direction for the acquisition of 3D spatial information.

[0003] However, existing UAV oblique image 3D reconstruction methods still have certain shortcomings. During multi-view matching, they are easily affected by factors such as viewpoint changes, occlusion, texture repetition, and lighting changes, leading to the accumulation of matching errors and a decrease in depth estimation accuracy. Existing depth optimization methods have limited ability to recover complex building edges and detailed areas, making it difficult to balance depth accuracy and structural integrity. In addition, traditional surface reconstruction methods usually adopt a fixed-scale partitioning strategy, lacking the joint utilization of semantic information and geometric complexity, which can easily lead to the loss of details in complex areas or waste of computational resources, making it difficult to simultaneously meet the application requirements of high-precision 3D reconstruction and high-efficiency modeling. Summary of the Invention

[0004] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a deep learning-based optimization method for 3D reconstruction of UAV oblique imagery. Addressing the problem that cross-view matching in existing UAV oblique imagery 3D reconstruction processes is easily affected by large tilt angles, occlusion regions, and texture inconsistencies, leading to large matching errors and poor depth estimation stability, this solution constructs a tilt consistency gating mechanism that integrates local texture stability, occlusion probability, semantic boundary consistency, and tilt angle difference. This dynamically weights the cross-view matching cost, effectively suppressing matching noise in low-reliability regions and improving the reliability of matching results and depth estimation accuracy. Furthermore, it addresses the limitations of existing depth map refinement methods, which rely on fixed sampling positions and struggle to adapt to complex building edges. To address the issue of insufficient depth detail recovery in complex regions, this solution constructs a deformable attention refinement network that adaptively predicts sampling positions based on current depth, confidence information, and multi-scale features. It also iteratively optimizes depth estimation using depth residuals, achieving refined reconstruction of complex structural regions and improving edge quality and detail representation in depth maps. Furthermore, to address the problem of fixed octree partitioning in existing UAV 3D surface reconstruction processes, which struggles to balance reconstruction accuracy in complex regions with overall computational efficiency, this solution constructs an adaptive octree by combining semantic complexity and local curvature information. This is further enhanced by incorporating structurally constrained Poisson surface reconstruction, edge sharpening, and multi-level mesh generation methods to adaptively allocate reconstruction accuracy across different regions.

[0005] The technical solution adopted in this invention is as follows: a deep learning-based optimization method for 3D reconstruction of UAV oblique images, which includes the following steps:

[0006] Step S1: Image acquisition. Acquire multi-angle tilted images collected by the UAV along the preset flight path. Perform time stamp alignment, radiometric normalization, size unification, and lens distortion correction on all images in sequence to obtain a standardized input image set. Simultaneously record the initial exterior orientation, focal length parameters, principal point parameters, and attitude parameters of each image as input data for subsequent view selection and geometric solution.

[0007] Step S2: Filter key view pairs. Based on the UAV attitude parameters and initial exterior orientation parameters recorded in Step S1, calculate the camera center baseline length, principal optical axis tilt angle difference and image overlap relationship between any two images. Construct a view pair scoring function that integrates the above three geometric reliability indicators. Filter out key view pairs that meet the conditions according to the scoring threshold to form a key view set for subsequent feature learning and cost volume construction.

[0008] Step S3: Generate multi-scale depth prior maps. Input the key views selected in Step S2 into the deep learning encoding network to extract local texture features, structural boundary features, and spatial layout features layer by layer. After encoding the image tilt position and pose information, inject them into the feature representation of each layer to obtain a feature map with fused pose encoding. Based on this, through a multi-branch convolutional fusion mechanism, generate depth prior maps at multiple scales to form a multi-scale depth prior map set, which provides an initial depth reference for the subsequent construction of the matching cost volume.

[0009] Step S4: Calculate the tilt consistency gating coefficient. Using the pose-aware features generated in Step S3, construct a cross-view feature matching cost body among key views. For each pixel, calculate the local texture stability, inter-view occlusion probability, and semantic boundary consistency score. Combine the above three quality indicators with the cosine value of the tilt angle difference between the view pairs and generate a pixel-wise tilt consistency gating coefficient after mapping by an activation function. Use the gating coefficient to dynamically weight the original matching cost body pixel-wise to form a gating cost body, thereby achieving adaptive suppression of matching noise in unreliable regions.

[0010] Step S5: Generate an initial depth map. Input the gated cost volume generated in step S4 into the depth regression network, perform softmax normalization on the matching cost on the discrete depth set to obtain the pixel-wise depth probability distribution, and calculate the pixel-level initial depth map in the form of expected value. At the same time, estimate the surface normal vector of each pixel based on the initial depth map, construct the normal consistency loss function, and jointly optimize it with the depth regression loss to output the initial depth map and the corresponding depth confidence map.

[0011] Step S6: Depth Map Refinement. Based on the initial depth map output in Step S5, a deformable attention refinement network is designed. In each iteration, the current depth value, confidence map, and multi-scale pose perception features are fused to predict the non-uniform spatial sampling offset of each pixel, guiding the deformable convolution to adaptively perceive along structural edges and depth discontinuities. Based on the offset-guided features, a depth correction value is output through a depth residual decoding sub-network, and the depth value is iteratively updated using a learnable stride coefficient. After multiple iterations, a high-precision refined depth map is output.

[0012] Step S7: Semantic Enhancement Dense Point Cloud Reconstruction. The refined depth map output from Step S6 is combined with the intrinsic parameter matrix, rotation matrix, and translation vector of each image to back-project the effective pixels to the world coordinate system, thus converging the back-projected point clouds of all images. Weighted centering is performed in the voxel grid with the number of multi-view coverages as the weight to remove redundancy, resulting in an initial dense point cloud. A semantic segmentation network is run on each image, and the pixel-by-pixel semantic probability vector is assigned to the corresponding 3D point after multi-view weighted averaging. Dual filtering is performed by combining statistical outlier detection conditions and semantic confidence thresholds to remove geometric outliers and high-confidence non-building structure category points. Each point in the filtered point cloud is fully assigned four attributes: normal vector, RGB color, semantic probability vector, and local curvature, outputting a semantically dense point cloud carrying complete geometric and semantic information.

[0013] Step S8: Adaptive surface reconstruction. Based on the semantically dense point cloud output in Step S7, the subdivision factor is calculated by combining the semantic complexity within voxels and the normalized average curvature. A semantic curvature joint adaptive octree is constructed, and the mesh resolution is automatically increased in regions with mixed categories or abrupt curvature changes. The Poisson implicit function containing the regularization term of the building wall plane constraint and the curvature modulation anisotropic smoothing regularization term is solved on the adaptive octree. The initial triangular mesh is extracted by the Marching Cubes algorithm. Constraint sharpening processing based on the mean of the neighborhood normal is performed on the vertices of the building area to enhance the sharpness of the building edges. Finally, the sharpened mesh is simplified in stages by a quadratic error metric method to generate a three-dimensional mesh model with multiple levels of detail.

[0014] Furthermore, in step S1, the radiometric normalization process standardizes the pixel values ​​using the pixel mean and standard deviation of each image as parameters, and introduces a minimal constant to prevent the denominator from being zero; the distortion correction uses a distortion model containing two-order radial distortion coefficients and two-order tangential distortion coefficients to correct the image normalization coordinates. The output of step S1 is a set of standardized images and a corresponding set of corrected coordinate parameters.

[0015] Furthermore, in step S2, the view pair reliability scoring function is composed of a weighted sum of three parts: a baseline length normalization term, a tilt angle difference Gaussian decay term, and an overlap saturation term, with the sum of the three weighting coefficients being 1. Only when the score is not lower than a preset threshold is the view pair included in the key view set to participate in subsequent feature learning and cost body construction.

[0016] Furthermore, in step S3, the tilt position and attitude joint encoding function takes the planar coordinates, roll angle and pitch angle of each image center as input, and the encoding result is weighted by learnable injection coefficients and superimposed on the convolutional feature map; the multi-scale depth prior map is obtained by weighted fusion of multi-branch point-by-point correlation convolution results, and the sum of the fusion weights of each branch is 1.

[0017] Further, in step S4, local texture stability is defined as the ratio of the variance to the mean of pixel values ​​within a local window centered on the target pixel; the occlusion probability is obtained by mapping the absolute difference between the initial depth prior map and the cross-view reprojection depth using an exponential function; the semantic boundary consistency score is obtained by mapping the boundary response value output by the semantic boundary detection network. The mapping is inverted to obtain the above four indicators, which are then linearly weighted and summed. After activation, the gating coefficient is output. The higher the gating coefficient, the more reliable the match at that pixel.

[0018] Furthermore, in step S5, the surface normal vector is calculated from the partial derivatives of the initial depth map along the horizontal and vertical directions; the normal consistency loss is defined as the sum of the first-norm distances between the normal vector at each effective pixel and the average normal vector of its neighborhood. By jointly minimizing the depth regression loss and the normal consistency loss, the depth estimation maintains global consistency at the geometric level.

[0019] Further, in step S6, the offset prediction subnetwork takes the channel dimension concatenation of the previous round depth value, confidence map and multi-scale features as input to predict the two-dimensional spatial offset for multiple sampling points of each pixel; the deformable convolution, after superimposing the predicted offset on the basic offset coordinates, samples from the feature map through bilinear interpolation and sums them with learnable scalar weights; the depth residual decoding subnetwork outputs scalar residuals, which are damped and updated with learnable step size coefficients, gradually converging to the refined depth estimate.

[0020] Further, in step S7, the semantically enhanced dense point cloud reconstruction specifically includes the following steps:

[0021] Step S71: 3D point backprojection. For all valid pixels in each image, combine the refined depth value, intrinsic parameter matrix, rotation matrix, and translation vector to backproject the pixel coordinates to the world coordinate system to obtain the corresponding 3D points. Aggregate the backprojected 3D points of all images to construct an initial point cloud. In a voxel grid with a preset side length, use the number of times each pixel is covered in multiple images as the weight to perform weighted centering redundancy removal processing on 3D points belonging to the same voxel. Pixels with more coverage times are given higher fusion weights. Each voxel outputs a weighted average of the representative 3D point coordinates to form the initial dense point cloud after redundancy removal.

[0022] Step S72: Semantic-guided filtering. Calculate the k-nearest neighbor average distance for each 3D point in the initial dense point cloud. Calculate the mean and standard deviation of this distance across the global point cloud. Points whose average neighbor distance exceeds the mean plus a threshold of several times the standard deviation are identified as geometric outliers and marked. Run a semantic segmentation network on each image, outputting a pixel-by-pixel multi-category semantic probability vector. Back-project this vector onto the corresponding 3D point. The semantic probabilities of the same voxel covered by multiple views are weighted and averaged using fusion weights to obtain the semantic probability vector for each 3D point. Perform dual filtering on the point cloud using joint logical conditions, retaining only points that simultaneously satisfy geometric non-outlier and do not belong to the high-confidence non-building structure category, forming the filtered point cloud. The non-building structure category set and semantic confidence threshold can be configured according to the application scenario.

[0023] Step S73: Complete assignment of point cloud attributes. The following four types of attributes are completely assigned to each 3D point in the filtered point cloud: the normal vector is determined by the eigenvector corresponding to the smallest eigenvalue of the local covariance matrix formed by the k nearest neighbors, representing the local surface normal direction, and is corrected for consistency with the inner product sign of the nearest camera's line of sight; the RGB color attribute is obtained by weighted averaging of the pixel colors of multiple images covering the voxel using fusion weights; the semantic probability vector is directly inherited from the result of the multi-view weighted averaging in step S72; the local curvature is defined as the ratio of the smallest eigenvalue of the covariance matrix to the sum of the three eigenvalues, which approaches zero at flat surfaces and approaches one-third at corner points; the final output is a semantically dense point cloud carrying complete geometric and semantic information, with each point containing five types of attributes: 3D coordinates, normal vector, RGB color, semantic probability vector, and local curvature.

[0024] Furthermore, in step S8, the adaptive surface reconstruction specifically includes the following steps:

[0025] Step S81: Construct a semantic curvature joint adaptive octree. For a subset of points within each voxel of the octree, calculate two metrics: semantic complexity and normalized mean curvature. Semantic complexity is obtained by inverting the maximum value of the mean of each category of the semantic probability vector of each point within the voxel. It approaches zero when the semantic category within the voxel is pure and approaches a larger value when the categories are mixed. Normalized mean curvature is obtained by dividing the mean of the local curvature of all points within the voxel by the reference curvature and truncating it to no more than 1. Linearly weight the above two metrics with adjustable semantic weight coefficients to obtain a joint subdivision factor. Based on the linear interpolation and rounding of the joint subdivision factor between the minimum and maximum octree depths, determine the actual subdivision depth of each voxel. The higher the joint subdivision factor, the smaller the sub-voxel is divided into, in order to capture the fine structure at the intersection of semantic boundaries and the abrupt change in geometric curvature. Large flat semantically pure regions with lower subdivision factors maintain a coarser resolution, realizing on-demand allocation of computing resources.

[0026] Step S82: Poisson surface reconstruction. The implicit scalar function is solved on the adaptive octree constructed in step S81, and its zero isosurface is the reconstructed surface. The objective functional consists of three terms: the principal term constrains the implicit function space gradient field to be as consistent as possible with the objective vector field obtained by interpolating the point cloud normal vectors; the plane constraint regularization term targets the semantic category point set of the building wall and minimizes the second-order directional derivative of the implicit function along the normal vector direction of each point, so that the reconstructed surface remains linear and flat in the wall normal direction; the anisotropic smoothing regularization term applies anisotropic weights to the implicit function Hessian matrix with a curvature-modulated diagonal matrix. The smoothing constraint is weaker in areas with greater curvature, thus maintaining flatness in flat areas while preserving geometric details at corners and edges; the objective functional is discretized into a system of linear equations in the octree function space, and the implicit function is obtained by iteratively solving the conjugate gradient method. Then, the initial triangular mesh is extracted at the isosurface determined by the mean of the point cloud implicit field function values ​​using the Marching Cubes algorithm.

[0027] Step S83: Multi-level detail generation. Normal constraint sharpening is applied to vertices belonging to the building region in the initial triangular mesh. Specifically, the mean unit normal vector of the neighboring facets of each vertex is calculated as a local normal direction reference. The projection of the displacement of each neighboring vertex relative to the current vertex in this normal direction is weighted by an adjustable sharpening intensity coefficient and superimposed onto the vertex coordinates, enhancing the building's edges and corners to obtain a sharpened mesh. Based on the sharpened mesh, a quadratic error matrix based on the sum of the outer products of the shared facets is established for each vertex using a quadratic error metric. The mesh is then hierarchically simplified according to a preset facet retention ratio sequence, generating multiple levels of detail 3D meshes ranging from full precision to the coarsest representation. The facet retention ratio decreases from 1 to a set minimum value; the larger the level number, the simpler the mesh. Finally, a set of 3D meshes covering multiple levels of detail is output to meet the differentiated requirements for rendering accuracy and computational efficiency in different scenarios.

[0028] The beneficial effects achieved by the present invention using the above solution are as follows:

[0029] (1) In the existing UAV tilt image 3D reconstruction process, cross-view matching is easily affected by large tilt angle, occlusion area and texture inconsistency, resulting in large matching error and poor depth estimation stability. This solution constructs a tilt consistency gating mechanism that integrates local texture stability, occlusion probability, semantic boundary consistency and tilt angle difference, and dynamically weights the cross-view matching cost, effectively suppressing matching noise in low reliability areas and improving the reliability of matching results and depth estimation accuracy.

[0030] (2) To address the problem that existing depth map refinement methods have fixed sampling positions and are difficult to adapt to complex building edges and detailed areas, resulting in insufficient depth detail recovery capabilities, this solution constructs a deformable attention refinement network, which adaptively predicts the sampling position based on the current depth, confidence information and multi-scale features, and combines depth residual iterative optimization of depth estimation to achieve fine reconstruction of complex structural areas, thereby improving the edge quality and detail representation capability of the depth map.

[0031] (3) In view of the problem that the octree division is fixed in the existing UAV 3D surface reconstruction process and it is difficult to balance the reconstruction accuracy of complex areas and the overall computational efficiency, this solution constructs an adaptive octree by combining semantic complexity and local curvature information, and combines structural constraint Poisson surface reconstruction, edge sharpening and multi-level mesh generation methods to achieve adaptive allocation of reconstruction accuracy in different areas, thereby improving the overall reconstruction efficiency and model quality while ensuring the integrity of architectural details. Attached Figure Description

[0032] Figure 1 A schematic diagram illustrating the deep learning-based UAV tilted image 3D reconstruction optimization method provided by this invention;

[0033] Figure 2 This is a schematic diagram of step S7;

[0034] Figure 3 This is a schematic diagram of step S8.

[0035] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0037] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0038] Example 1, see Figure 1The present invention provides a deep learning-based method for optimizing 3D reconstruction of UAV oblique images, which includes the following steps:

[0039] Step S1: Image acquisition. Acquire multi-angle tilted images collected by the UAV along the preset flight path, preprocess the raw images, and simultaneously record the camera's orientation, focal length, principal point, and attitude parameters.

[0040] Step S2: Select key view pairs, construct a view reliability evaluation mechanism, and select key view pairs for 3D reconstruction;

[0041] Step S3: Generate a multi-scale depth prior map. Input the key view into the deep learning encoding network to extract multi-level features of the image. At the same time, fuse the image position and pose information to generate a depth prior map through multi-scale feature fusion.

[0042] Step S4: Calculate the skew consistency gating coefficient, establish a cross-view matching cost body between key views, calculate the skew consistency gating coefficient, dynamically weight the matching cost, and generate a more reliable gating cost body;

[0043] Step S5: Generate an initial depth map, input the gated cost volume into the depth regression network, predict the pixel-level initial depth map and corresponding confidence map, and construct normal consistency constraints to optimize the depth regression process;

[0044] Step S6: Depth map refinement. A deformable attention refinement network is constructed for the initial depth map. During the iteration process, the adaptive sampling position is predicted, and the depth result is continuously corrected through the depth residual to obtain a refined depth map.

[0045] Step S7: Semantic Enhancement Dense Point Cloud Reconstruction. The 3D point cloud is reconstructed using the refined depth map. Combined with multi-view fusion, statistical outlier detection and semantic segmentation results, the point cloud is deredundant, filtered and attribute-assigned so that the generated dense point cloud has complete geometric and semantic information.

[0046] Step S8: Adaptive surface reconstruction. An adaptive surface reconstruction model is constructed based on semantically enhanced dense point clouds. The surface reconstruction process is jointly controlled by semantic complexity and curvature information. Combined with structural constraints, edge sharpening and multi-level mesh simplification, a three-dimensional mesh model that balances accuracy and efficiency is generated.

[0047] Example 2, see Figure 1This embodiment is based on the above embodiment. In step S1: the image acquisition involves acquiring multi-angle tilted images collected by the UAV along a preset flight path. All images undergo timestamp alignment, radiometric normalization, size standardization, and distortion correction to obtain a standardized input image set. Simultaneously, the initial exterior orientation, focal length, principal point, and attitude parameters of each image are recorded as input data for subsequent view selection and geometric solving. The original image set is represented as follows: ;in, Represents the original image set. Indicates the first Original image, Indicates the total number of images; for the first... Radiometric standardization of the images yields standardized images: ;in, Represents the normalized image in pixel coordinates Pixel value at that location, Indicates the first The average of the images, Indicates the first Standard deviation of the image This represents a very small constant to prevent the denominator from being zero; when lens distortion exists in the image, coordinate correction is performed, and the correction relationship is... ;in, Represents the normalized coordinates before distortion. Indicates the coordinates after distortion correction. , , Represents the radial distortion coefficient. , The output of step S1 represents the tangential distortion coefficients, and is a normalized image set. And the corresponding set of correction coordinate parameters.

[0048] Example 3, see Figure 1 This embodiment is based on the above embodiment. In step S2, the key view pairs are selected by calculating the baseline length, tilt angle difference, and overlap relationship between any two images based on the UAV attitude parameters and initial exterior orientation parameters, and thereby selecting key view pairs for subsequent 3D reconstruction. Image and the first The camera centers corresponding to the images are respectively and The corresponding principal optical axis unit vectors are respectively and Then the baseline vector and tilt angle difference between the two images are defined as: ;in, Represents the camera center baseline vector. Image The angle between the principal optical axis and the baseline vector. Indicates the first The principal optical axis unit vector of the image. Describing the Euclidean norm, Indicate transpose; construct the view for the reliability scoring function: ;in, Indicates image pair Reliability rating Denotes the weighting coefficients and satisfies , Represents the baseline normalization constant. Indicates the center value of the target tilt angle. Indicates the angular attenuation scale. Represents the overlap estimate between two images; when At that time, view the pair Add to key view collection ;when At this point, subsequent feature learning and cost body construction are not performed; the output of step S2 is a set of key views. .

[0049] Example 4, see Figure 1 This embodiment is based on the above embodiment. In step S3, the generation of a multi-scale depth prior map involves inputting the key views selected in step S2 into a deep learning encoding network to extract local texture features, structural boundary features, and spatial layout features. The image center position and pose information are then encoded into these features to generate the multi-scale depth prior map. The first input image The feature maps in the layer network are ;in, This represents the output feature map of the l-th layer. This represents the input feature map of the previous layer. This represents the bias term at level l. Indicates the first Layer convolution kernel parameters, This represents the convolution operation. express Activation function; after injecting the skew position encoding into the feature representation, we get: ;in, This represents the feature map after fusion of pose encoding. Indicates the attitude coding injection coefficients. This represents the joint encoding function for position and attitude. Indicates the coordinates of the image center. Indicates the roll angle. Representing the pitch angle; further constructing a multi-scale depth prior map: ;in, Representing scale The depth prior map below, Indicates the number of branches. Indicates the first Each branch at scale The fusion weights below satisfy , Representing scale The predicted convolutional kernel for the corresponding branch, This represents point-by-point correlation operations; the output of step S3 is a set of multi-scale depth prior maps. .

[0050] Example 5, see Figure 1 This embodiment is based on the above embodiment. In step S4, the tilt consistency gating coefficient is calculated, a cross-view matching cost volume is constructed between key views, and tilt consistency gating coefficients are generated by combining local texture stability, tilt angle difference, occlusion probability, and semantic boundary consistency. The cost volume is then dynamically weighted, assuming image pairs... At pixel and hypothesis depth The matching cost is: ;in, Represents pixels In image pair and depth Matching cost under certain conditions Image At pixel The feature vector at that location, Represents the depth-related feature transformation matrix. Image The feature vector at the corresponding position. This represents the reprojection position obtained based on the exterior orientation parameters and the assumed depth. Denotes the norm; the local texture stability, occlusion probability, and boundary consistency scores are defined as follows: ; ; ;in, Image At pixel Local texture stability, Indicates A local window centered on the center. Represents variance. This represents the mean. Represents a minimal constant. Indicates the probability of occlusion. Image Initial depth estimation, Indicates the image Reprojection to image Subsequent depth estimation, This indicates that sensitive scales are being obscured. Indicates the semantic boundary consistency score. Indicates the boundary response scale. Represent the boundary response value output by the semantic boundary network; construct the skewed consistency gating coefficient: ;in, Represents pixels The gating coefficient at the location, Indicates the gating weight, Image With images The tilt angle difference between them; ultimately forming the gating cost body: ;in, This represents the matching cost body after gating.

[0051] By performing the above operations, this solution addresses the problem that cross-view matching in existing UAV tilted image 3D reconstruction is easily affected by large tilt angles, occlusion areas, and texture inconsistencies, resulting in large matching errors and poor depth estimation stability. This solution constructs a tilt consistency gating mechanism that integrates local texture stability, occlusion probability, semantic boundary consistency, and tilt angle difference to dynamically weight the cross-view matching cost, effectively suppressing matching noise in low-reliability areas and improving the reliability of matching results and depth estimation accuracy.

[0052] Example 6, see Figure 1 This embodiment is based on the above embodiment. In step S5, the initial depth map is generated by inputting the gated cost volume into the depth regression network, outputting a pixel-level initial depth map and a confidence map, and introducing a normal consistency constraint during the regression process; the probability distribution output by the depth regression network is as follows: ;in, Indicates the first Image pixels In depth The probability distribution at that location, Represents a discrete set of depths; the initial depth map is calculated from the probability distribution: ;in, Represents pixels The initial depth value; calculate the surface normal vector based on the initial depth map: ;in, Represents pixels Surface normal vector at that location, and Let the horizontal and vertical partial derivatives be represented respectively; construct the normal consistency loss function: ;in, Indicates the loss of normal consistency. Image The set of valid pixels in Represents pixels The average normal vector within the neighborhood; the initial depth map is obtained by minimizing the depth regression loss and the normal consistency loss. and depth confidence plot .

[0053] Example 7, see Figure 1 This embodiment is based on the above embodiment. In step S6, the depth map is refined. For the initial depth map in step S5, a deformable attention refinement network is designed to output a refined depth map. The next iteration at the pixel level Calculate the sampling offset at: ;in, For the first The next iteration in imaging pixel place as A two-dimensional spatial offset vector generated from each sampling point; For parameters The offset prediction subnetwork; For channel splicing; The initial depth value is the same as the previous depth value. ; For the confidence map in step S5, at the pixel point The value at; For step S3 Layer fusion of pose-encoded feature vectors; extraction of deformable convolutional features guided by offset: ;in, For the first Learnable weight scalar for each sampling point; For the first The two-dimensional offset corresponding to each sampling point; For the standard convolution kernel The base offset coordinates of each sampling point; This indicates bilinear interpolation sampling of the feature map; depth residual decoding and iterative update: ;in, For the first The learnable step size coefficient for each iteration; The decoding subnetwork for outputting scalar depth residuals has the following parameters: Iteration Next order ; Indicates the first The refined depth map after the next iteration.

[0054] By performing the above operations, this solution addresses the problem that existing depth map refinement methods have fixed sampling positions, making it difficult to adapt to complex building edges and detailed regions, resulting in insufficient depth detail recovery capabilities. This solution constructs a deformable attention refinement network that adaptively predicts the sampling position based on the current depth, confidence information, and multi-scale features. It also combines depth residuals to iteratively optimize depth estimation, thereby achieving fine reconstruction of complex structural regions and improving the edge quality and detail representation capabilities of depth maps.

[0055] Example 8, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S7, the semantically enhanced dense point cloud reconstruction backprojects the refined depth map into a 3D point cloud, introduces semantic segmentation for category attribute assignment, and combines statistical outlier detection and semantic guided filtering to assign complete geometric and radial attributes to each point. Specifically, it includes the following steps:

[0056] Step S71: 3D point back projection, for image For all valid pixels, the refined depth values ​​will be back-projected to the world coordinate system: ;in, For images pixel The corresponding three-dimensional point in the world coordinate system; For images rotation matrix; This is the intrinsic parameter matrix; It is a translation vector; The homogeneous coordinates of the pixels are used; by aggregating all valid pixels from the image, an initial point cloud is constructed. On a side with length of Perform weighted centering redundancy removal in the voxel grid: ; ;in, For pixels The fusion weight; This is the refinement factor; For pixels The count of overlapping images is obtained by statistical analysis of camera overlap relationships, with higher weighting for more overlaps. voxels Internal image A set of pixels; voxels The representative three-dimensional point coordinates after internal weighted average;

[0057] Step S72: Semantic guided filtering, for each point calculate Average distance to nearest neighbors and statistical global distribution: ; ;in, For the first Coordinates of nearest neighbors; and Let be the global mean and standard deviation of the average nearest neighbor distance, respectively; satisfying... The point is a geometric outlier. The outlier determination factor is used; a semantic segmentation network is applied to each image to output pixel-wise semantic probabilities, which are then back-projected onto 3D points. ;in, For semantic segmentation networks At pixel Output Category probability vector; The semantic probability vector of the three-dimensional points is obtained by weighted averaging of multiple views. Indicates its first Each component; joint filtering conditions, retaining points that satisfy the following conditions constitute... : ;in, This is the set of non-building structure categories that need to be filtered out. This is the semantic confidence threshold; For logical AND, Logical NOT; a point is retained only if it simultaneously satisfies geometric non-outlier and does not belong to the high-confidence filtering category.

[0058] Step S73: Completely assign point cloud attributes. Each point in the array is assigned the following complete properties:

[0059] Normal vector: ;in, for Centroid of nearest neighbor points; It is the local covariance matrix; The eigenvector corresponding to the smallest eigenvalue is taken, which is the local surface normal direction; the normal vector direction is corrected for consistency by the sign of the inner product with the nearest camera's line of sight.

[0060] Color attributes: ;in, Weighted average for multiple views Color vector; For S1 standardized images at pixel points The RGB color vector at that location;

[0061] Local curvature: ;in, for The three eigenvalues; It approaches 0 at flat surfaces and approaches 0 at corner points. ; To prevent extremely small positive numbers with a denominator of zero, the final output is a semantically dense point cloud carrying complete attributes. .

[0062] Example 9, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S8, the adaptive surface reconstruction, based on the semantically dense point cloud in step S7, constructs a semantic curvature joint adaptive octree, performs Poisson surface reconstruction with structural constraints, performs sharpening processing on the building edges, and generates a multi-level detail 3D mesh model, specifically including the following steps:

[0063] Step S81: Construct a semantic curvature joint adaptive octree for voxels. interior point subset Calculate the semantic complexity: ;in, For semantic complexity; For point The Class semantic probability; the inner layer sums and averages, then the maximum value is calculated for all classes. The maximum value approaches 1 when the classes within a voxel are pure. Approaching 0; approaching when categories are mixed Normalized mean curvature: ;in, Normalized mean curvature; The normalized curvature reference value is a positive real scalar; joint subdivision depth: ;in, For joint segmentation factors; These are semantic weight coefficients; and These are the minimum and maximum octree subdivision depths, respectively; Round down; the higher the subdivision factor, the smaller the voxel is divided into sub-voxels to capture fine structure.

[0064] Step S82: Reconstruct the Poisson surface and solve the implicit function on the adaptive octree. The objective functional is: ;in, The zero isosurface is the reconstructed surface; Its spatial gradient vector field; For point cloud normal vectors The target vector field is constructed by interpolation; and The weights of the two regularization terms are: The plane constraint regularization term is: ;in, For building wall category point set, A set of semantic category indexes for building walls; for At this point Matrix, elements are ; for The second-order directional derivative along the normal vector direction is minimized to keep the reconstructed surface linear in the normal direction; the anisotropic smoothing regularization term is: ;in, Anisotropic smoothing control diagonal matrix; Construct a diagonal matrix using the three scalars within the parentheses as diagonal elements; curvature The larger the value, the closer the first two diagonal elements are to 0, the weaker the smoothing constraint, and the more geometric details are preserved. For matrix Norm; the objective functional is discretized into a system of linear equations in the octree function space, and solved by the conjugate gradient method. ,pass The algorithm on the isosurface Extracting the initial triangular mesh ,in It is determined by the mean value of the implicit field function in the point cloud;

[0065] Step S83: Generate multiple levels of detail, and perform normal constraint sharpening on the vertices of the building region: ;in, For the first Current coordinates of each vertex; As vertex The set of facets to which it belongs; For dough The unit normal vector; The mean of the normal vectors of the neighborhood facets; As vertex The set of vertices in the ring neighborhood; It is the projection scalar of the relative displacement of the neighboring vertices onto the normal direction; This represents the sharpening intensity factor; To prevent extremely small positive numbers from being divided by zero; Let be the vertex coordinates after sharpening; let the sharpened mesh be . Generate based on quadratic error metric indivual level: ; ;in, As vertex The second-order error measure; For shared vertices A collection of facets; For dough The coefficient vector of the unit homogeneous equations in the plane; for The fundamental error quadratic form matrix; Indicates execution on the grid Simplified to the original number of face pieces times; Represents a grid simplification function based on a quadratic error metric; For the first layer Dough sheet retention ratio, Preserve the complete grid. The larger the mesh size, the coarser the grid; the more pixels are output. 3D mesh .

[0066] By performing the above operations, this solution addresses the problem that existing UAV 3D surface reconstruction methods suffer from fixed octree partitioning, making it difficult to balance reconstruction accuracy in complex regions with overall computational efficiency. This solution constructs an adaptive octree by combining semantic complexity and local curvature information, and combines structural constraint Poisson surface reconstruction, edge sharpening, and multi-level mesh generation methods to achieve adaptive allocation of reconstruction accuracy for different regions. This improves overall reconstruction efficiency and model quality while ensuring the integrity of architectural details.

[0067] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0068] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

[0069] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A deep learning-based optimization method for 3D reconstruction of UAV oblique images, characterized in that, The method includes the following steps: Step S1: Image acquisition. Acquire multi-angle tilted images collected by the UAV along the preset flight path. Perform time stamp alignment, radiometric normalization, size unification, and lens distortion correction on all images in sequence to obtain a standardized input image set. Simultaneously record the initial exterior orientation, focal length parameters, principal point parameters, and attitude parameters of each image as input data for subsequent view selection and geometric solution. Step S2: Filter key view pairs. Based on the UAV attitude parameters and initial exterior orientation parameters recorded in Step S1, calculate the camera center baseline length, principal optical axis tilt angle difference and image overlap relationship between any two images. Construct a view pair scoring function that integrates the above three geometric reliability indicators. Filter out key view pairs that meet the conditions according to the scoring threshold to form a key view set for subsequent feature learning and cost volume construction. Step S3: Generate multi-scale depth prior maps. Input the key views selected in Step S2 into the deep learning encoding network to extract local texture features, structural boundary features, and spatial layout features layer by layer. After encoding the image tilt position and pose information, inject them into the feature representation of each layer to obtain a feature map with fused pose encoding. Based on this, through a multi-branch convolutional fusion mechanism, generate depth prior maps at multiple scales to form a multi-scale depth prior map set, which provides an initial depth reference for the subsequent construction of the matching cost volume. Step S4: Calculate the tilt consistency gating coefficient. Using the pose-aware features generated in Step S3, construct a cross-view feature matching cost body among key views. For each pixel, calculate the local texture stability, inter-view occlusion probability, and semantic boundary consistency score. Combine the above three quality indicators with the cosine value of the tilt angle difference between the view pairs and generate a pixel-wise tilt consistency gating coefficient after mapping by an activation function. Use the gating coefficient to dynamically weight the original matching cost body pixel-wise to form a gating cost body, thereby achieving adaptive suppression of matching noise in unreliable regions. Step S5: Generate an initial depth map. Input the gated cost volume generated in step S4 into the depth regression network, perform softmax normalization on the matching cost on the discrete depth set to obtain the pixel-wise depth probability distribution, and calculate the pixel-level initial depth map in the form of expected value. At the same time, estimate the surface normal vector of each pixel based on the initial depth map, construct the normal consistency loss function, and jointly optimize it with the depth regression loss to output the initial depth map and the corresponding depth confidence map. Step S6: Depth Map Refinement. Based on the initial depth map output in Step S5, a deformable attention refinement network is designed. In each iteration, the current depth value, confidence map, and multi-scale pose perception features are fused to predict the non-uniform spatial sampling offset of each pixel, guiding the deformable convolution to adaptively perceive along structural edges and depth discontinuities. Based on the offset-guided features, a depth correction value is output through a depth residual decoding sub-network, and the depth value is iteratively updated using a learnable stride coefficient. After multiple iterations, a high-precision refined depth map is output. Step S7: Semantic Enhancement Dense Point Cloud Reconstruction. The refined depth map output from Step S6 is combined with the intrinsic parameter matrix, rotation matrix, and translation vector of each image to back-project the effective pixels to the world coordinate system, thus converging the back-projected point clouds of all images. Weighted centering is performed in the voxel grid with the number of multi-view coverages as the weight to remove redundancy, resulting in an initial dense point cloud. A semantic segmentation network is run on each image, and the pixel-by-pixel semantic probability vector is assigned to the corresponding 3D point after multi-view weighted averaging. Dual filtering is performed by combining statistical outlier detection conditions and semantic confidence thresholds to remove geometric outliers and high-confidence non-building structure category points. Each point in the filtered point cloud is fully assigned four attributes: normal vector, RGB color, semantic probability vector, and local curvature, outputting a semantically dense point cloud carrying complete geometric and semantic information. Step S8: Adaptive surface reconstruction. Based on the semantically dense point cloud output in Step S7, the subdivision factor is calculated by combining the semantic complexity within voxels and the normalized average curvature. A semantic curvature joint adaptive octree is constructed, and the mesh resolution is automatically increased in regions with mixed categories or abrupt curvature changes. The Poisson implicit function containing the regularization term of the building wall plane constraint and the curvature modulation anisotropic smoothing regularization term is solved on the adaptive octree. The initial triangular mesh is extracted by the Marching Cubes algorithm. Constraint sharpening processing based on the mean of the neighborhood normal is performed on the vertices of the building area to enhance the sharpness of the building edges. Finally, the sharpened mesh is simplified in stages by a quadratic error metric method to generate a three-dimensional mesh model with multiple levels of detail.

2. The deep learning-based UAV oblique image 3D reconstruction optimization method according to claim 1, characterized in that: In step S1, the radiometric normalization process standardizes the pixel values ​​using the pixel mean and standard deviation of each image as parameters, and introduces a minimal constant to prevent the denominator from being zero; the distortion correction uses a distortion model containing two-order radial distortion coefficients and two-order tangential distortion coefficients to correct the image normalization coordinates. The output of step S1 is a set of standardized images and a set of corresponding correction coordinate parameters.

3. The method for optimizing 3D reconstruction of UAV oblique images based on deep learning according to claim 1, characterized in that: In step S2, the reliability scoring function of the view pair is composed of a weighted sum of three parts: a baseline length normalization term, a tilt angle difference Gaussian decay term, and an overlap saturation term. The sum of the three weighting coefficients is 1. Only when the score is not lower than a preset threshold is the view pair included in the key view set and participate in subsequent feature learning and cost body construction.

4. The deep learning-based UAV oblique image 3D reconstruction optimization method according to claim 1, characterized in that: In step S3, the tilt position and attitude joint encoding function takes the planar coordinates, roll angle and pitch angle of each image center as input, and the encoding result is weighted by learnable injection coefficients and superimposed on the convolutional feature map; the multi-scale depth prior map is obtained by weighted fusion of multi-branch point-by-point correlation convolution results, and the sum of the fusion weights of each branch is 1.

5. The deep learning-based UAV oblique image 3D reconstruction optimization method according to claim 1, characterized in that: In step S4, the local texture stability is defined as the ratio of the variance to the mean of pixel values ​​within a local window centered on the target pixel; the occlusion probability is obtained by mapping the absolute difference between the initial depth prior map and the cross-view reprojection depth using an exponential function. The semantic boundary consistency score is calculated by evaluating the boundary response value output by the semantic boundary detection network. Obtained by inverting the mapping; The above four indicators were linearly weighted and summed. After activation, the gating coefficient is output. The higher the gating coefficient, the more reliable the match at that pixel.

6. The deep learning-based UAV oblique image 3D reconstruction optimization method according to claim 1, characterized in that: In step S5, the surface normal vector is calculated from the partial derivatives of the initial depth map along the horizontal and vertical directions; the normal consistency loss is defined as the sum of the first-norm distances between the normal vector at each effective pixel and the average normal vector of its neighborhood. By jointly minimizing the depth regression loss and the normal consistency loss, the depth estimation maintains global consistency at the geometric level.

7. The deep learning-based UAV oblique image 3D reconstruction optimization method according to claim 1, characterized in that: In step S6, the offset prediction subnetwork takes the channel dimension concatenation of the previous round depth value, confidence map, and multi-scale features as input to predict the two-dimensional spatial offset for multiple sampling points of each pixel; the deformable convolution, after superimposing the predicted offset on the basic offset coordinates, samples from the feature map through bilinear interpolation and sums them with learnable scalar weights; the depth residual decoding subnetwork outputs scalar residuals, which are damped and updated with learnable step size coefficients, gradually converging to a refined depth estimate.

8. The deep learning-based UAV oblique image 3D reconstruction optimization method according to claim 1, characterized in that: In step S7, the semantically enhanced dense point cloud reconstruction specifically includes the following steps: Step S71: 3D point backprojection. For all valid pixels in each image, combine the refined depth value, intrinsic parameter matrix, rotation matrix, and translation vector to backproject the pixel coordinates to the world coordinate system to obtain the corresponding 3D points. Aggregate the backprojected 3D points of all images to construct an initial point cloud. In a voxel grid with a preset side length, use the number of times each pixel is covered in multiple images as the weight to perform weighted centering redundancy removal processing on 3D points belonging to the same voxel. Pixels with more coverage times are given higher fusion weights. Each voxel outputs a weighted average of the representative 3D point coordinates to form the initial dense point cloud after redundancy removal. Step S72: Semantic-guided filtering. Calculate the k-nearest neighbor average distance for each 3D point in the initial dense point cloud. Calculate the mean and standard deviation of this distance across the global point cloud. Points whose average neighbor distance exceeds the mean plus a threshold of several times the standard deviation are identified as geometric outliers and marked. Run a semantic segmentation network on each image, outputting a pixel-by-pixel multi-category semantic probability vector. Back-project this vector onto the corresponding 3D point. The semantic probabilities of the same voxel covered by multiple views are weighted and averaged using fusion weights to obtain the semantic probability vector for each 3D point. Perform dual filtering on the point cloud using joint logical conditions, retaining only points that simultaneously satisfy geometric non-outlier and do not belong to the high-confidence non-building structure category, forming the filtered point cloud. The non-building structure category set and semantic confidence threshold can be configured according to the application scenario. Step S73: Complete assignment of point cloud attributes. The following four types of attributes are completely assigned to each 3D point in the filtered point cloud: the normal vector is determined by the eigenvector corresponding to the smallest eigenvalue of the local covariance matrix formed by the k nearest neighbors, representing the local surface normal direction, and is corrected for consistency with the inner product sign of the nearest camera's line of sight; the RGB color attribute is obtained by weighted averaging of the pixel colors of multiple images covering the voxel using fusion weights; the semantic probability vector is directly inherited from the result of the multi-view weighted averaging in step S72; the local curvature is defined as the ratio of the smallest eigenvalue of the covariance matrix to the sum of the three eigenvalues, which approaches zero at flat surfaces and approaches one-third at corner points; the final output is a semantically dense point cloud carrying complete geometric and semantic information, with each point containing five types of attributes: 3D coordinates, normal vector, RGB color, semantic probability vector, and local curvature.

9. The deep learning-based UAV oblique image 3D reconstruction optimization method according to claim 1, characterized in that: In step S8, the adaptive surface reconstruction specifically includes the following steps: Step S81: Construct a semantic curvature joint adaptive octree. For a subset of points within each voxel of the octree, calculate two metrics: semantic complexity and normalized mean curvature. Semantic complexity is obtained by inverting the maximum value of the mean of each category of the semantic probability vector of each point within the voxel. It approaches zero when the semantic category within the voxel is pure and approaches a larger value when the categories are mixed. Normalized mean curvature is obtained by dividing the mean of the local curvature of all points within the voxel by the reference curvature and truncating it to no more than 1. Linearly weight the above two metrics with adjustable semantic weight coefficients to obtain a joint subdivision factor. Based on the linear interpolation and rounding of the joint subdivision factor between the minimum and maximum octree depths, determine the actual subdivision depth of each voxel. The higher the joint subdivision factor, the smaller the sub-voxel is divided into, in order to capture the fine structure at the intersection of semantic boundaries and the abrupt change in geometric curvature. Large flat semantically pure regions with lower subdivision factors maintain a coarser resolution, realizing on-demand allocation of computing resources. Step S82: Poisson surface reconstruction. The implicit scalar function is solved on the adaptive octree constructed in step S81, and its zero isosurface is the reconstructed surface. The objective functional consists of three terms: the principal term constrains the implicit function space gradient field to be as consistent as possible with the objective vector field obtained by interpolating the point cloud normal vectors; the plane constraint regularization term targets the semantic category point set of the building wall and minimizes the second-order directional derivative of the implicit function along the normal vector direction of each point, so that the reconstructed surface remains linear and flat in the wall normal direction; the anisotropic smoothing regularization term applies anisotropic weights to the implicit function Hessian matrix with a curvature-modulated diagonal matrix. The smoothing constraint is weaker in areas with greater curvature, thus maintaining flatness in flat areas while preserving geometric details at corners and edges; the objective functional is discretized into a system of linear equations in the octree function space, and the implicit function is obtained by iteratively solving the conjugate gradient method. Then, the initial triangular mesh is extracted at the isosurface determined by the mean of the point cloud implicit field function values ​​using the Marching Cubes algorithm. Step S83: Multi-level detail generation. Normal constraint sharpening is applied to vertices belonging to the building region in the initial triangular mesh. Specifically, the mean unit normal vector of the neighboring facets of each vertex is calculated as a local normal direction reference. The projection of the displacement of each neighboring vertex relative to the current vertex in this normal direction is weighted by an adjustable sharpening intensity coefficient and superimposed onto the vertex coordinates, enhancing the building's edges and corners to obtain a sharpened mesh. Based on the sharpened mesh, a quadratic error matrix based on the sum of the outer products of the shared facets is established for each vertex using a quadratic error metric. The mesh is then hierarchically simplified according to a preset facet retention ratio sequence, generating multiple levels of detail 3D meshes ranging from full precision to the coarsest representation. The facet retention ratio decreases from 1 to a set minimum value; the larger the level number, the simpler the mesh. Finally, a set of 3D meshes covering multiple levels of detail is output to meet the differentiated requirements for rendering accuracy and computational efficiency in different scenarios.

Images