A method and system for three-dimensional reconstruction of unmanned aerial vehicle oblique photography based on machine vision

Abstract

This invention discloses a machine vision-based UAV oblique photogrammetry 3D reconstruction method and system, belonging to the field of photogrammetric 3D reconstruction technology. It includes: generating a synchronized image dataset and a synchronized point cloud dataset; generating an enhanced image dataset; constructing a multi-scale feature pyramid composed of high-resolution texture features, medium-resolution structural features, and low-resolution semantic features using a hierarchical window attention backbone network containing deformable convolutional coupling structures of SwinTransformer-V2 and DCNv4; forming geometric anchor point guidance markers; outputting a fused feature set in an improved DUSt3R 3D reconstruction network; generating an initial visual dense point cloud; generating a fused and enhanced dense point cloud; and outputting 3D reconstruction results that meet the accuracy requirements of engineering surveying. This invention can significantly reduce the point cloud void rate and improve the continuous surface coverage capability in complex construction site scenarios.

Description

Technical Field

[0001] This invention relates to the field of photographic 3D reconstruction technology, and in particular to a machine vision-based UAV oblique photographic 3D reconstruction method and system. Background Technology

[0002] With the widespread application of UAV oblique photogrammetry technology in engineering surveying and digital twin fields, 3D reconstruction methods based on multi-view images have gradually become the mainstream technical approach for acquiring large-scale, high-precision spatial data. Existing technologies typically employ traditional visual reconstruction frameworks based on motion reconstruction structures and multi-view stereo vision, extracting and matching feature points across different images to recover camera pose and generate dense point clouds. However, in real-world engineering scenarios, numerous weakly textured regions, highly reflective surfaces, and repetitive texture structures make it difficult to stably extract or match feature points, leading to camera pose calculation failures or broken matching chains, resulting in problems such as model holes, distortion, or even reconstruction failure.

[0003] Purely visual reconstruction methods suffer from inherent scale uncertainty and error accumulation in large-scale survey areas. Even when combined with RTK or control points for constraints, scale drift and elevation error accumulation are still prone to occur over long distances or in edge regions, making it difficult to meet the stringent accuracy requirements of engineering surveying. Meanwhile, although lidar point cloud data has high-precision absolute spatial coordinates, its point cloud density is low and it lacks texture information, making it difficult to achieve high-quality surface reconstruction and realistic texture representation when used alone.

[0004] The fusion of visual data and LiDAR data usually adopts a post-processing registration method, that is, point cloud is introduced for alignment correction after visual reconstruction is completed. This method fails to solve the instability problem in the visual reconstruction process from the source and makes it difficult to make full use of the complementary advantages of the two types of data. Summary of the Invention

[0005] One objective of this invention is to propose a machine vision-based UAV oblique photogrammetry 3D reconstruction method and system. This invention can significantly reduce point cloud void rate and improve continuous surface coverage in complex construction site scenarios.

[0006] A machine vision-based UAV oblique photogrammetry 3D reconstruction method according to an embodiment of the present invention includes:

[0007] Data is collected by using a drone equipped with a tilting camera and a lidar, and time synchronization and extrinsic parameter calibration are completed by combining RTK / PPK to generate synchronized image datasets and synchronized point cloud datasets.

[0008] Perform image quality enhancement processing on the synchronized image dataset to generate an enhanced image dataset;

[0009] A hierarchical window attention backbone network containing deformable convolutional coupling structures of SwinTransformer-V2 and DCNv4 is used to extract multi-scale features from the enhanced image dataset, and a multi-scale feature pyramid composed of high-resolution texture features, medium-resolution structural features and low-resolution semantic features is constructed.

[0010] The synchronous point cloud dataset is projected onto the corresponding view plane of the enhanced image dataset based on the extrinsic parameter calibration results, generating a sparse depth anchor map, an edge skeleton map, and a point cloud coverage confidence mask, which are then encoded to form geometric anchor guide markers.

[0011] In the improved DUSt3R 3D reconstruction network, the multi-scale feature pyramid and geometric anchor point guide markers are fused through a cross-modal cross-attention mechanism to output a fused feature set;

[0012] Based on the fused feature set, pixel-level local 3D coordinates and point-level confidence are regressed within the DUSt3R point map regression head to generate an initial visual dense point cloud;

[0013] Using a synchronized point cloud dataset as the absolute geometric skeleton, global rotation alignment, translation alignment and scale correction are performed on the initial visual dense point cloud based on point-level confidence to generate a fused and enhanced dense point cloud.

[0014] Hole repair and surface continuity completion are performed on the fused and enhanced dense point cloud to construct a fused and enhanced 3D surface model. The fused and enhanced 3D surface model is then texture-mapped with the enhanced image dataset to generate a real-scene 3D reconstruction model with absolute spatial coordinates and realistic texture expression. The output is a 3D reconstruction result that meets the accuracy requirements of engineering surveying and mapping.

[0015] Optionally, the method of using a drone equipped with a tilting camera and a lidar to collaboratively collect data, and combining RTK / PPK to complete time synchronization and extrinsic parameter calibration, includes:

[0016] Using a multi-rotor UAV equipped with a five-lens oblique photography camera, a lidar sensor, and an RTK / PPK dual-redundant positioning module, collaborative flight data acquisition is performed on the target survey area to obtain oblique photography image datasets, lidar raw point cloud datasets, and corresponding pose datasets. The oblique photography image datasets, lidar raw point cloud datasets, and pose datasets are synchronized in time according to a unified time synchronization reference. An absolute spatial coordinate system for the survey area is established based on the RTK / PPK positioning results. External parameter calibration is performed on the oblique photography camera and lidar sensor to generate synchronized image datasets and synchronized point cloud datasets.

[0017] Optionally, the image quality enhancement process includes brightness normalization, reflection suppression, and local contrast enhancement.

[0018] Optionally, the hierarchical window attention backbone network employing a deformable convolutional coupling structure of SwinTransformer-V2 and DCNv4 is used to extract multi-scale features from the enhanced image dataset, including:

[0019] Each enhanced image in the enhanced image dataset is input into the initial embedding unit. Using a two-dimensional convolutional unit with a kernel size of 4×4 and a stride of 4, non-overlapping block partitioning and channel mapping are performed on each enhanced image, so that each enhanced image is mapped from the original pixel space to the initial feature tensor.

[0020] The initial feature tensor is input into a hierarchical window attention backbone network containing a three-level hierarchical processing structure. In each level of hierarchical processing, the Laplacian operator is used to perform high-frequency spatial gradient calculation on the current input feature tensor to generate a visual edge saliency feature map.

[0021] After combining the visual edge saliency feature map with the corresponding input feature tensor, the input is fed into the offset prediction convolutional unit to generate an adaptive offset field. The corresponding input feature tensor is then fed into the modulation prediction convolutional unit to generate a modulation coefficient field.

[0022] Based on the adaptive offset field and modulation coefficient field, deformable convolutional local enhancement processing is performed on the corresponding input feature tensor through deformable convolutional local enhancement unit in each level of hierarchical processing to generate locally enhanced feature tensors;

[0023] The local enhancement feature tensor is divided into multiple local windows. Query mapping, key mapping and value mapping are performed on the feature vectors in each local window to generate a local query matrix, a local key matrix and a local value matrix.

[0024] In the window attention unit, window attention calculation is performed on the local query matrix, local key matrix and local value matrix in each local window of each level of hierarchical processing to generate the window attention feature tensor.

[0025] In the shift window reorganization unit, the window attention feature tensor is processed by shift window reorganization, and a second window attention interaction calculation is performed in the reorganized window to generate a cross-window related feature tensor.

[0026] In the feature fusion unit, the local enhancement feature tensor, the window attention feature tensor, the cross-window association feature tensor and the corresponding input feature tensor are coupled and fused to generate the fused feature tensor corresponding to the hierarchical processing.

[0027] For each level of layered processing, the fusion feature tensor is updated by layered downsampling. When the layer level is the first or second level, a downsampling convolutional unit with a kernel size of 2×2 and a stride of 2 is used to perform spatial downsampling and channel mapping on the corresponding fusion feature tensor to generate the output feature tensor of the corresponding layer. When the layer level is the third level, the fusion feature tensor corresponding to the third level is directly used as the output feature tensor of the third level.

[0028] Channel projection processing is performed on the output feature tensors at each level to generate high-resolution texture features, medium-resolution structural features, and low-resolution semantic features.

[0029] High-resolution texture features, medium-resolution structural features, and low-resolution semantic features are combined to construct a multi-scale feature pyramid for the corresponding enhanced image.

[0030] Optionally, projecting the synchronized point cloud dataset onto the corresponding view plane of the enhanced image dataset based on the extrinsic parameter calibration results includes:

[0031] Establish a correspondence between each frame of synchronized point cloud in the synchronized point cloud dataset and the corresponding enhanced image in the enhanced image dataset. Based on the extrinsic parameter calibration results of the corresponding enhanced image, transform each laser point in each frame of synchronized point cloud from the lidar coordinate system to the camera coordinate system of the corresponding enhanced image to generate the camera coordinate point set of the corresponding enhanced image.

[0032] Perform view plane projection processing on each camera coordinate point in each camera coordinate point set, and map each camera coordinate point located in the imaging view frustum of the corresponding enhanced image to the pixel plane of the corresponding enhanced image to generate a point cloud projection coordinate set.

[0033] Based on the point cloud projection coordinate set and the depth values ​​of the corresponding camera coordinate points, a sparse depth anchor map and an effective projection indicator map of the corresponding enhanced image are constructed.

[0034] Based on the sparse depth anchor point map and the effective projection indicator map, the edge skeleton map of the corresponding enhanced image is extracted;

[0035] Based on the coverage distribution of the point cloud projection coordinate set in the corresponding augmented image pixel plane, a point cloud coverage confidence mask for the corresponding augmented image is constructed.

[0036] Based on the maximum effective depth value in the m-th enhanced image, local depth log normalization mapping compression is performed on all depth anchor values ​​in the sparse depth anchor map to generate a scale-normalized depth map.

[0037] The scale-normalized depth map, edge skeleton map, and point cloud coverage confidence mask are stacked sequentially according to the channel dimension, so that the scale-normalized depth value, edge skeleton value, and point cloud coverage confidence value at the same pixel location are arranged sequentially in the channel dimension, generating the geometric guidance tensor of the corresponding enhanced image.

[0038] Perform convolutional encoding and downsampling mapping on the geometric guidance tensor to generate geometric anchor point guidance marks for the corresponding enhanced image.

[0039] Optionally, in the improved DUSt3R 3D reconstruction network, the multi-scale feature pyramid and geometric anchor guide markers are fused through a cross-modal cross-attention mechanism, including:

[0040] The geometric anchor point guide markers are mapped to the spatial resolution corresponding to the multi-scale feature pyramid to generate high-resolution geometric guide features, medium-resolution geometric guide features and low-resolution geometric guide features;

[0041] Channel alignment processing is performed on the three visual scale features and the three geometric guidance features respectively to obtain channel-aligned visual scale features and channel-aligned geometric features;

[0042] The channel-aligned visual scale features and channel-aligned geometric features are respectively expanded into visual scale feature labeling sequences and geometric feature labeling sequences according to their spatial position order;

[0043] In the cross-modal cross-attention layer of the improved DUST3R 3D reconstruction network, a cross-modal query matrix is ​​generated using a visual scale feature label sequence, and a cross-modal key matrix and a cross-modal value matrix are generated using a geometric feature label sequence.

[0044] Based on the cross-modal query matrix, cross-modal key matrix, and cross-modal value matrix, cross-modal cross-attention calculation guided by confidence bias is performed at each scale type to generate a geometrically injected feature label sequence;

[0045] Gated fusion processing is performed on the visual scale feature label sequence and the geometric injection feature label sequence to generate a cross-modal fusion label sequence;

[0046] The cross-modal fusion label sequence under each scale type is restored into a two-dimensional feature grid of the corresponding scale type according to its spatial label number, generating high-resolution fusion features, medium-resolution fusion features and low-resolution fusion features;

[0047] The high-resolution fusion features, medium-resolution fusion features, and low-resolution fusion features are combined to form a fusion feature set.

[0048] Optionally, the step of regressing pixel-level local 3D coordinates and point-level confidence within the DUSt3R point map regression head based on the fused feature set includes:

[0049] The high-resolution, medium-resolution, and low-resolution fusion features in the fusion feature set are input into the DUSt3R point map regression head. The medium-resolution and low-resolution fusion features are then enlarged in spatial scale using a bilinear interpolation upsampling algorithm and mapped to the spatial resolution corresponding to the high-resolution fusion features, generating high-resolution aligned structural features and high-resolution aligned semantic features.

[0050] High-resolution fused features, high-resolution aligned structural features, and high-resolution aligned semantic features are concatenated along the channel dimension, and channel compression and local context encoding are performed on the concatenated features to generate a dot plot regression feature tensor.

[0051] Perform pixel-level unfolding and restoration processing to map the point map regression feature tensor from the high-resolution feature space to the original pixel space of the enhanced image, generating a pixel-level point map regression feature tensor.

[0052] Input the pixel-level point map regression feature tensor into the pixel-level local 3D coordinate regression branch to regress the pixel-level local 3D coordinates corresponding to each pixel position and generate a pixel-level local 3D point map.

[0053] Input the pixel-level point map regression feature tensor into the point-level confidence regression branch, regress the point-level confidence corresponding to each pixel position, and generate a point-level confidence map;

[0054] The pixel-level local 3D point map is correlated pixel by pixel with the point-level confidence map to generate a pixel-level visual point map with confidence attributes.

[0055] Based on the pixel-level visual point maps corresponding to all enhanced images, an initial visually dense point cloud is constructed.

[0056] Optionally, the step of using the synchronized point cloud dataset as the absolute geometric skeleton and performing global rotation alignment, translation alignment, and scale correction on the initial visual dense point cloud based on point-level confidence includes:

[0057] Pixel-level local 3D coordinates and point-level confidence scores are extracted from each enhanced image from the initial visual dense point cloud. The pixel-level local 3D coordinates are used as visual points to be aligned, and the point-level confidence scores are used as the reliability attributes of the visual points to be aligned.

[0058] The synchronized point cloud dataset is used as the absolute geometric skeleton points, and the pixel position relationship between the visual points to be aligned and the absolute geometric skeleton points is established based on the point cloud projection coordinate set, generating a set of visual-laser associated point pairs.

[0059] Based on point-level confidence, construct geometric skeleton constraint weights;

[0060] Based on the visual-laser associated point pair set and geometric skeleton constraint weights, the single-view scale parameter, single-view rotation matrix and single-view translation vector from the local camera coordinate system to the absolute spatial coordinate system of the survey area are solved independently for each enhanced image.

[0061] Based on the single-view scale parameters, single-view rotation matrix and single-view translation vector corresponding to each enhanced image, spatial rotation alignment, translation alignment and scale correction are performed on all pixel-level local three-dimensional coordinates in the corresponding image to generate absolute coordinate visual points.

[0062] Based on the geometric skeleton constraint weights, an adaptive weighted update is performed on the absolute coordinate visual points and the corresponding absolute geometric skeleton points to generate geometric constraint visual points;

[0063] All the geometrically constrained visual points corresponding to the enhanced images are grouped and organized according to the absolute spatial coordinate system of the survey area to generate a fused and enhanced dense point cloud.

[0064] Optionally, the step of performing hole repair and surface continuity completion on the fused and enhanced dense point cloud to construct a fused and enhanced 3D surface model includes:

[0065] All geometrically constrained visual points and their point-level confidence scores in the fusion-enhanced dense point cloud are uniformly indexed according to the absolute spatial coordinate system of the survey area to form a set of points to be reconstructed on the surface.

[0066] A local neighborhood search is performed on the point set to be reconstructed on the surface. Based on the neighborhood distance distribution and point-level confidence distribution of each geometrically constrained visual point, the hole boundary points in the fused and enhanced dense point cloud are identified.

[0067] Based on the cavity boundary points, candidate cavity regions are generated, and point completion processing based on neighborhood geometric constraints is performed on the candidate cavity regions to generate a complete point set;

[0068] The completed point set is merged into the point set to be reconstructed to generate a surface continuity completed point set, and local normal estimation is performed on the surface continuity completed point set to obtain the local normal vector;

[0069] A set of triangular facets is constructed based on the surface continuity completion point set and local normal vectors to generate a fused and enhanced 3D surface model;

[0070] The triangular facets in the fused and enhanced 3D surface model are projected onto the enhanced image dataset, and the texture source image is determined based on the visibility markers of the triangular facets, the pixel projection area, and the point-level confidence.

[0071] Based on the texture source image, perform texture mapping on the fused and enhanced 3D surface model to generate a textured 3D surface model;

[0072] The spatial coordinates, triangular facet topological relationships, and texture color information in the textured 3D surface model are uniformly encapsulated to generate a real-world 3D reconstruction model with absolute spatial coordinates and realistic texture expression, thus forming a 3D reconstruction result.

[0073] The beneficial effects of this invention are:

[0074] This invention introduces a hierarchical window attention and deformable convolution coupled backbone network, enabling the network to have stable geometric representation capabilities in weak texture and strong reflective scenes. It can still form stable feature responses in concrete walls, glass curtain walls and areas with repetitive textures, and can significantly reduce point cloud void rate and improve continuous surface coverage in complex construction site scenes.

[0075] This invention achieves the transformation of LiDAR geometric information from post-processing constraints to feedforward guidance constraints by constructing a geometric anchor guidance mechanism and introducing cross-modal cross-attention fusion within the network. Based on the generation of sparse depth anchor maps, edge skeleton maps, and point cloud coverage confidence masks from point cloud projection, geometric guidance markers are directly injected into the visual feature stream through the cross-modal attention mechanism. This allows the DUSt3R point map regression process to be simultaneously constrained by semantic information and absolute geometric priors, effectively suppressing false surface generation and depth drift problems in water accumulation areas, shadowed areas, and high-reflectivity areas. It enables error constraint in the early stages of reconstruction, significantly improving the stability and geometric consistency of point map regression. Attached Figure Description

[0076] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0077] Figure 1 This is a flowchart of a UAV oblique photogrammetry 3D reconstruction method based on machine vision proposed in this invention;

[0078] Figure 2 This is a schematic diagram of the multi-scale feature extraction structure based on a hierarchical window attention and deformable convolution coupled backbone network in the UAV oblique photogrammetry 3D reconstruction method based on machine vision proposed in this invention. Detailed Implementation

[0079] Example 1: Reference Figures 1-2 A machine vision-based UAV oblique photogrammetry 3D reconstruction method includes:

[0080] Using a multi-rotor UAV equipped with a five-lens oblique photography camera, a lidar sensor, and an RTK / PPK dual-redundant positioning module, collaborative flight data acquisition is performed on the target survey area to obtain oblique photography image datasets, lidar raw point cloud datasets, and corresponding pose datasets. The oblique photography image datasets, lidar raw point cloud datasets, and pose datasets are synchronized in time according to a unified time synchronization reference. An absolute spatial coordinate system for the survey area is established based on the RTK / PPK positioning results. External parameter calibration is performed on the oblique photography camera and lidar sensor to generate synchronized image datasets and synchronized point cloud datasets.

[0081] Brightness normalization, reflection suppression, and local contrast enhancement are performed on the synchronized image dataset to generate an enhanced image dataset.

[0082] A hierarchical window attention backbone network containing deformable convolutional coupling structures of SwinTransformer-V2 and DCNv4 is used to extract multi-scale features from the enhanced image dataset, and a multi-scale feature pyramid composed of high-resolution texture features, medium-resolution structural features and low-resolution semantic features is constructed.

[0083] In this embodiment, each enhanced image in the enhanced image dataset is input into the initial embedding unit. Using a two-dimensional convolutional unit with a kernel size of 4×4 and a stride of 4, non-overlapping block partitioning and channel mapping are performed on each enhanced image, so that each enhanced image is mapped from the original pixel space to the initial feature tensor.

[0084] In Example 1, each enhanced image in the enhanced image dataset is input into the initial embedding unit. The initial embedding unit uses a two-dimensional convolutional unit with a kernel size of 4×4 and a stride of 4 to traverse each enhanced image in a scanning order from left to right and from top to bottom. At each stride position, the corresponding 4×4 pixel block is extracted. All pixel values ​​within each 4×4 pixel block are processed by channel-weighted accumulation and linear mapping to map each 4×4 pixel block into a feature vector. According to the spatial arrangement order of each 4×4 pixel block in the original enhanced image, all feature vectors are rearranged into a two-dimensional feature grid to generate the initial feature tensor of the corresponding enhanced image. The spatial resolution of the initial feature tensor is one-quarter of the original spatial resolution of the corresponding enhanced image, and the number of channels of the initial feature tensor is determined by the output channel configuration of the initial embedding unit.

[0085] The initial feature tensor is input into a hierarchical window attention backbone network containing a three-level hierarchical processing structure. In each level of hierarchical processing, the Laplacian operator is used to perform high-frequency spatial gradient calculation on the current input feature tensor to generate a visual edge saliency feature map. The visual edge saliency feature map is then combined with the corresponding input feature tensor and input into the offset prediction convolution unit to generate an adaptive offset field. The corresponding input feature tensor is then input into the modulation prediction convolution unit to generate a modulation coefficient field.

[0086] In Example 1, the hierarchical window attention backbone network with a three-level hierarchical processing structure is composed of a first-level hierarchical processing module, a second-level hierarchical processing module, and a third-level hierarchical processing module connected in a cascaded manner. Each level of hierarchical processing module includes an offset prediction convolution unit, a modulation prediction convolution unit, a deformable convolution local enhancement unit, a window attention unit, a shift window reorganization unit, and a feature fusion unit. Furthermore, the feature resolution is progressively reduced between adjacent hierarchical processing modules through spatial downsampling.

[0087] Based on the adaptive offset field and modulation coefficient field, deformable convolutional local enhancement processing is performed on the corresponding input feature tensor through deformable convolutional local enhancement unit in each level of hierarchical processing to generate locally enhanced feature tensors;

[0088] ;

[0089] in, Indicates the first In the hierarchical processing, the first The spatial coordinates of the local enhancement feature tensor generated from the enhanced image Place, No. Characteristic response values ​​on each output channel This indicates the sequence number of the enhanced image in the enhanced image dataset, and , This represents the total number of augmented images in the augmented image dataset. This represents the hierarchical sequence number in the hierarchical window attention backbone network, and This represents the spatial index of the feature tensor in the horizontal direction. This represents the spatial index of the feature tensor in the vertical direction. This represents the output channel index of the local enhancement feature tensor. Indicates the first The total number of preset sampling points in the hierarchical processing. Indicates the first The number of channels in the input feature tensor during hierarchical processing; Indicates the first In the hierarchical processing of the first level The sampling point at the th sampling point The input channel and the first Deformable convolution weight parameters between each output channel Indicates the first Enhanced image in the first The input feature tensor obtained after hierarchical processing Indicates the first In the hierarchical processing of the first level The reference sampling offset of each sampling point in the lateral direction. Indicates the first In the hierarchical processing of the first level The reference sampling offset of each sampling point in the longitudinal direction. Indicates that for the first Enhanced image in the first In the hierarchical processing of the first level The adaptive lateral offset corresponding to each sampling point is generated by the visual edge saliency feature map constraint, which drives the sampling points to preferentially cluster towards regions with abrupt changes in image texture and structural edges within the pure visual receptive field. Indicates that for the first Enhanced image in the first In the hierarchical processing of the first level The adaptive vertical offset corresponding to each sampling point is also controlled by the visual edge saliency feature map. This represents the channel index of the input feature tensor; Indicates that for the first Enhanced image in the first In the hierarchical processing of the first level The modulation coefficients corresponding to each sampling point.

[0090] The local enhancement feature tensor is divided into multiple local windows. Query mapping, key mapping and value mapping are performed on the feature vectors in each local window to generate a local query matrix, a local key matrix and a local value matrix.

[0091] In the window attention unit, window attention calculation is performed on the local query matrix, local key matrix and local value matrix in each local window of each level of hierarchical processing to generate the window attention feature tensor.

[0092] In the shift window reorganization unit, the window attention feature tensor is processed by shift window reorganization, and a second window attention interaction calculation is performed in the reorganized window to generate a cross-window related feature tensor.

[0093] In Example 1, the window attention feature tensor is divided into local windows according to a preset window division method. While keeping the feature tensor channel dimension unchanged, the feature data corresponding to each local window is synchronously translated in the horizontal and vertical directions in the spatial dimension according to a preset shift step size to obtain the shifted feature distribution. The preset shift step size is half of the corresponding local window size.

[0094] Based on the shifted feature distribution, the feature data is re-partitioned into local windows with the same window size as the original local window partitioning method. Within the re-partitioned local windows, the feature data is aligned and reassembled to generate a shifted and reassembled feature distribution. The shifted and reassembled feature distribution is then mapped again into a shift query matrix, a shift key matrix, and a shift value matrix. A second-order window self-attention calculation is performed within the re-partitioned local windows to extract the global spatial dependencies across windows and generate a cross-window associated feature tensor.

[0095] In the feature fusion unit, the local enhancement feature tensor, the window attention feature tensor, the cross-window association feature tensor and the corresponding input feature tensor are coupled and fused to generate the fused feature tensor corresponding to the hierarchical processing.

[0096] In Example 1, channel alignment is performed on the local enhancement feature tensor, window attention feature tensor, cross-window association feature tensor, and corresponding input feature tensor. The four types of feature tensors after channel alignment are then subjected to spatial global average pooling to extract channel-level statistics, which are then input into a multilayer perceptron for nonlinear mapping of dimensionality reduction and expansion. An adaptive gating vector is output through the Sigmoid activation function and used as the corresponding channel weight coefficient. The corresponding channel weight coefficients are applied to the channel-aligned local enhancement feature tensors and then weighted element-wise to generate a fused feature tensor.

[0097] For each level of layered processing, the fusion feature tensor is updated by layered downsampling. When the layer level is the first or second level, a downsampling convolutional unit with a kernel size of 2×2 and a stride of 2 is used to perform spatial downsampling and channel mapping on the corresponding fusion feature tensor to generate the output feature tensor of the corresponding layer. When the layer level is the third level, the fusion feature tensor corresponding to the third level is directly used as the output feature tensor of the third level.

[0098] Channel projection processing is performed on the output feature tensors at each level to generate high-resolution texture features, medium-resolution structural features, and low-resolution semantic features.

[0099] In Example 1, channel projection processing is performed on the output feature tensor of the first-level layered processing to generate high-resolution texture features, channel projection processing is performed on the output feature tensor of the second-level layered processing to generate medium-resolution structural features, and channel projection processing is performed on the output feature tensor of the third-level layered processing to generate low-resolution semantic features.

[0100] High-resolution texture features, medium-resolution structural features, and low-resolution semantic features are combined to construct a multi-scale feature pyramid for the corresponding enhanced image.

[0101] The synchronous point cloud dataset is projected onto the corresponding view plane of the enhanced image dataset based on the extrinsic parameter calibration results, generating a sparse depth anchor map, an edge skeleton map, and a point cloud coverage confidence mask, which are then encoded to form geometric anchor guide markers.

[0102] In this embodiment, a correspondence is established between each frame of synchronized point cloud in the synchronized point cloud dataset and the corresponding enhanced image in the enhanced image dataset. Based on the external parameter calibration results of the corresponding enhanced image, each laser point in each frame of synchronized point cloud is transformed from the lidar coordinate system to the camera coordinate system of the corresponding enhanced image to generate the camera coordinate point set of the corresponding enhanced image.

[0103] Perform view plane projection processing on each camera coordinate point in each camera coordinate point set, and map each camera coordinate point located in the imaging view frustum of the corresponding enhanced image to the pixel plane of the corresponding enhanced image to generate a point cloud projection coordinate set.

[0104] In Example 1, for each camera coordinate point in the set of camera coordinate points corresponding to the m-th enhanced image, an intrinsic parameter mapping is performed based on the intrinsic parameter matrix of the camera corresponding to the m-th enhanced image to obtain a homogeneous projection vector. The horizontal and vertical components in the homogeneous projection vector are divided by the scale component in the homogeneous projection vector to obtain the projected pixel coordinates of the camera coordinate point in the pixel plane of the m-th enhanced image.

[0105] The projected pixel coordinates include the pixel positions in the horizontal direction and the pixel positions in the vertical direction. Only the projected pixel coordinates that satisfy the following conditions are retained: the pixel position in the horizontal direction is greater than or equal to zero and less than the number of pixels in the original width of the enhanced image; the pixel position in the vertical direction is greater than or equal to zero and less than the number of pixels in the original height of the enhanced image; and the depth value of the corresponding camera coordinate point in the camera coordinate system is greater than zero. The point cloud projection coordinate set corresponding to the m-th enhanced image is generated.

[0106] Based on the point cloud projection coordinate set and the depth values ​​of the corresponding camera coordinate points, a sparse depth anchor map and an effective projection indicator map of the corresponding enhanced image are constructed.

[0107] In Example 1, based on the point cloud projection coordinate set, the camera coordinate points projected to each pixel position are classified according to their pixel positions, and the depth value of the camera coordinate point corresponding to each pixel position is extracted.

[0108] For pixel locations with projections from one or more camera coordinate points, a minimum depth retention strategy is used to determine the sparse depth anchor value at the pixel location, and the effective projection indicator value at the pixel location is set to 1. For pixel locations without projections from any camera coordinate points, the sparse depth anchor value at the pixel location is set to 0, and the effective projection indicator value at the pixel location is set to 0, thus constructing a sparse depth anchor map and an effective projection indicator map for the corresponding enhanced image.

[0109] Based on the sparse depth anchor point map and the effective projection indicator map, the edge skeleton map of the corresponding enhanced image is extracted;

[0110] In Example 1, for the m-th enhanced image, the depth difference between the current pixel position and its horizontally adjacent pixel position is calculated only if the current pixel position and its horizontally adjacent pixel position have effective projection at the same time. The depth difference is used as the effective horizontal depth gradient value at the current pixel position. The effective vertical depth gradient value is calculated in the same way. If the current pixel position and its corresponding adjacent pixel position do not have effective projection at the same time, the depth gradient value in the corresponding direction is set to zero.

[0111] The effective depth gradient magnitude at the current pixel position is calculated by summing the squares of the effective horizontal depth gradient and the effective vertical depth gradient and then taking the square root. Based on the effective projection indicator map, the number of effective projections in the local neighborhood around each pixel position in the m-th enhanced image is counted to obtain the effective neighborhood support count value at the corresponding pixel position.

[0112] Based on the effective depth gradient magnitude and the effective neighborhood support count, pixel positions that simultaneously satisfy the condition that the effective depth gradient magnitude is greater than or equal to the depth edge determination threshold and the effective neighborhood support count is greater than or equal to the support count threshold are filtered. The edge response value at the corresponding pixel position is set to 1, and the edge response value at the other pixel positions is set to 0, generating an edge response map. The edge response map is then subjected to skeleton thinning processing based on non-maximum suppression, which suppresses non-peak responses in the local gradient direction, removes redundant edge pixels, and generates an edge skeleton map corresponding to the m-th enhanced image with a single pixel width.

[0113] Based on the coverage distribution of the point cloud projection coordinate set in the corresponding augmented image pixel plane, a point cloud coverage confidence mask for the corresponding augmented image is constructed.

[0114] In Example 1, for any pixel location in the m-th enhanced image, a local neighborhood window with a side length of twice the radius of the local neighborhood window plus one is constructed with the corresponding pixel location as the center, and the number of projection points within the local neighborhood window is counted to obtain the local coverage count value at the pixel location.

[0115] If a pixel location in the m-th enhanced image has at least one projection point, the projection point presence indicator value at the corresponding pixel location is set to 1; if a pixel location in the m-th enhanced image does not have any projection points, the projection point presence indicator value at the pixel location is set to 0.

[0116] Based on the projection point presence indication value, the local coverage count values ​​at all pixel locations with projection points in the m-th enhanced image are statistically analyzed. The maximum local coverage count value is used as the normalization benchmark. The local coverage count value at each pixel location is divided by the maximum local coverage count value. For pixel locations with a projection point presence indication value of 0, their point cloud coverage confidence value is directly assigned to 0, thus obtaining the point cloud coverage confidence value at the corresponding pixel location and generating the point cloud coverage confidence mask corresponding to the m-th enhanced image.

[0117] Based on the maximum effective depth value in the m-th enhanced image, local depth log normalization mapping compression is performed on all depth anchor values ​​in the sparse depth anchor map to generate a scale-normalized depth map.

[0118] In Example 1, the sparse depth anchor map corresponding to the m-th enhanced image is traversed, all effective depth anchor values ​​greater than zero are extracted, and the local maximum effective depth value is calculated. To eliminate the significant differences in magnitude and dimension between absolute depth values ​​and edge skeleton maps and point cloud coverage confidence masks, and to avoid logarithmic singularity overflow errors caused by zero-depth backgrounds (regions without projection points), a local depth logarithmic normalization model with a smoothing bias term is adopted for any pixel location. Depth anchor point map at the location Perform nonlinear compression mapping to generate a scale-normalized depth map. :

[0119] ;

[0120] in, This represents a scale-normalized depth map after mapping, which nonlinearly compresses the physical quantity of absolute depth, which could originally be hundreds of meters, to a smaller scale. The dimensionless pure scalar of the interval, the constant 1 in the formula is a smoothing bias term, ensuring that when At that time, its corresponding This ensures the monotonicity of depth and zero-point safety.

[0121] The scale-normalized depth map, edge skeleton map, and point cloud coverage confidence mask are stacked sequentially according to the channel dimension, so that the scale-normalized depth value, edge skeleton value, and point cloud coverage confidence value at the same pixel location are arranged sequentially in the channel dimension, generating the geometric guidance tensor of the corresponding enhanced image.

[0122] Perform convolutional encoding and downsampling mapping on the geometric guidance tensor to generate geometric anchor point guidance marks for the corresponding enhanced image.

[0123] In Example 1, the geometric guidance tensor corresponding to the m-th enhanced image is input into the geometric coding unit. The geometric coding unit uses a two-dimensional convolutional coding unit with a kernel size of 4×4 and a stride of 4 to traverse the geometric guidance tensor in blocks. For the values ​​of all input channels in each 4×4 spatial region, channel weighted accumulation and linear mapping are performed to map each 4×4 spatial region into an output feature vector. According to the spatial arrangement order of each 4×4 spatial region in the m-th enhanced image, all output feature vectors are rearranged into a two-dimensional feature grid to generate the geometric anchor guidance mark corresponding to the m-th enhanced image. The spatial resolution of the geometric anchor guidance mark is one-quarter of the original spatial resolution of the m-th enhanced image, and the number of channels of the geometric anchor guidance mark is determined by the output channel configuration of the geometric coding unit.

[0124] In the improved DUSt3R 3D reconstruction network, the multi-scale feature pyramid and geometric anchor point guide markers are fused through a cross-modal cross-attention mechanism to output a fused feature set;

[0125] In this embodiment, the geometric anchor point guide marks are mapped to the spatial resolution corresponding to the multi-scale feature pyramid to generate high-resolution geometric guide features, medium-resolution geometric guide features and low-resolution geometric guide features.

[0126] In Example 1, the geometric anchor point guidance mark corresponding to the m-th enhanced image is directly determined as the high-resolution geometric guidance feature corresponding to the m-th enhanced image.

[0127] The geometric anchor point guidance markers corresponding to the m-th enhanced image are subjected to average pooling processing according to the two-by-two local regions, so that the geometric anchor point guidance marker values ​​in each two-by-two local region are aggregated into a geometric guidance feature value, generating the medium-resolution geometric guidance feature corresponding to the m-th enhanced image.

[0128] The medium-resolution geometric guidance feature corresponding to the m-th enhanced image is again processed by average pooling according to the two-by-two local regions, so that the medium-resolution geometric guidance feature value in each two-by-two local region is aggregated into a geometric guidance feature value, generating the low-resolution geometric guidance feature corresponding to the m-th enhanced image; the number of channels of the geometric guidance feature remains unchanged during the first and second two-fold spatial downsampling processes.

[0129] Channel alignment processing is performed on the three visual scale features and the three geometric guidance features respectively to obtain channel-aligned visual scale features and channel-aligned geometric features;

[0130] In Example 1, the high-resolution texture features, medium-resolution structural features, and low-resolution semantic features in the multi-scale feature pyramid are used as high-resolution texture visual scale features, medium-resolution structural visual scale features, and low-resolution semantic visual scale features, respectively, to obtain three types of visual scale features.

[0131] For the visual scale features of the m-th enhanced image at any scale type, the visual scale features at the corresponding scale type are input into the visual channel projection unit. The visual channel projection unit uses one-dimensional convolution to perform channel-weighted summation on all input channel feature values ​​at each spatial location in the visual scale features at the corresponding scale type, and then superimposes the visual projection bias term of the corresponding output channel to generate the channel-aligned visual scale features of the m-th enhanced image at that scale type. The channel-aligned geometric features are obtained using the same method. The channel-aligned visual scale features and the channel-aligned geometric features have the same spatial resolution and the same output channels at the same scale type.

[0132] The channel-aligned visual scale features and channel-aligned geometric features are respectively expanded into visual scale feature labeling sequences and geometric feature labeling sequences according to their spatial position order;

[0133] In Example 1, for the channel-aligned visual scale features of the m-th enhanced image at any scale type, the channel feature vectors at each spatial location are arranged sequentially into a visual scale feature label sequence according to the spatial order from left to right and from top to bottom.

[0134] For the channel alignment geometric features of the m-th enhanced image under the same scale type, the channel feature vectors at each spatial location are arranged sequentially into a geometric feature label sequence according to the spatial order from left to right and from top to bottom; the spatial label number in the visual scale feature label sequence corresponds one-to-one with the spatial label number in the geometric feature label sequence, and each spatial label number corresponds to a spatial location under the same scale type.

[0135] In the cross-modal cross-attention layer of the improved DUST3R 3D reconstruction network, a cross-modal query matrix is ​​generated using a visual scale feature label sequence, and a cross-modal key matrix and a cross-modal value matrix are generated using a geometric feature label sequence.

[0136] Based on the cross-modal query matrix, cross-modal key matrix, and cross-modal value matrix, cross-modal cross-attention calculation guided by confidence bias is performed at each scale type to generate a geometrically injected feature label sequence;

[0137] In Example 1, the dot product of the cross-modal query matrix and the cross-modal key matrix is ​​calculated to obtain the initial cross-modal similarity matrix. To prevent invalid projection regions of the sparse LiDAR point cloud from interfering with visual features, the point cloud overlay confidence mask is spatially downsampled, mapped, and serialized according to the corresponding scale type to construct an attention bias matrix, which is then injected into the initial cross-modal similarity matrix. Softmax normalization is performed on the similarity matrix after bias injection to generate the cross-modal attention weight matrix. The cross-modal value matrix is ​​then weighted and multiplied using the cross-modal attention weight matrix to generate the geometric injection feature label sequence for the m-th enhanced image at that scale type. :

[0138] ;

[0139] in, This represents the cross-modal query matrix generated from the visual scale feature label sequence. and These represent the cross-modal key matrix and cross-modal value matrix generated from the geometric feature marker sequence, respectively. Indicates the channel dimension scaling factor. Represents the transpose of the cross-modal bond matrix. This represents the point cloud coverage confidence mask bias matrix after scale mapping and serialization expansion. This represents the scale factor used to adjust the bias intensity; the attention calculation mechanism enables high-confidence geometric anchors to receive exponential attention amplification during cross-fusion, while regions with extremely low confidence or missing information have their interference with visual features forcibly suppressed.

[0140] Gated fusion processing is performed on the visual scale feature label sequence and the geometric injection feature label sequence to generate a cross-modal fusion label sequence;

[0141] In Example 1, the visual scale feature label sequence and the geometric injection feature label sequence are concatenated along the channel dimension to obtain the concatenated label sequence. The concatenated label sequence is then input into the gating mapping layer. The concatenated label sequence is linearly mapped through the weight matrix and bias term of the gating mapping layer, and cross-modal gating weights are generated through the Sigmoid activation function.

[0142] The geometrically injected feature label sequence is weighted channel-by-channel based on the cross-modal gating weights, and the visual scale feature label sequence is weighted channel-by-channel based on the compensation weight obtained by subtracting the cross-modal gating weights. The weighted geometrically injected feature label sequence and the weighted visual scale feature label sequence are then added element-by-element to generate the cross-modal fusion label sequence for the m-th enhanced image at that scale type.

[0143] ;

[0144] in, Indicates the first Enhanced images at scale The cross-modal gating weights generated below, Represents a sequence of visual feature labels. This represents the geometric injection feature marker sequence.

[0145] The cross-modal fusion label sequence under each scale type is restored into a two-dimensional feature grid of the corresponding scale type according to its spatial label number, generating high-resolution fusion features, medium-resolution fusion features and low-resolution fusion features;

[0146] In Example 1, for the cross-modal fusion marker sequence of the m-th enhanced image at any scale type, according to the one-to-one correspondence between the spatial marker number and the spatial position, each fusion marker in the cross-modal fusion marker sequence is refilled to the corresponding spatial position, restoring it to a two-dimensional feature grid at that scale type, and generating the fusion feature of the m-th enhanced image at that scale type.

[0147] When the scale type is high-resolution texture scale, generate high-resolution fusion features corresponding to the m-th enhanced image; when the scale type is medium-resolution structure scale, generate medium-resolution fusion features corresponding to the m-th enhanced image; when the scale type is low-resolution semantic scale, generate low-resolution fusion features corresponding to the m-th enhanced image.

[0148] The high-resolution fusion features, medium-resolution fusion features, and low-resolution fusion features are combined to form a fusion feature set;

[0149] The fused feature set is the output of the enhanced image m after being fused by multi-scale feature pyramids and geometric anchor point guided markers in the improved DUSt3R 3D reconstruction network through a cross-modal cross-attention mechanism.

[0150] Based on the fused feature set, pixel-level local 3D coordinates and point-level confidence are regressed within the DUSt3R point map regression head to generate an initial visual dense point cloud;

[0151] In this embodiment, the high-resolution fusion features, medium-resolution fusion features and low-resolution fusion features in the fusion feature set are input into the DUSt3R point map regression head, and the medium-resolution fusion features and low-resolution fusion features are spatially scaled up using the bilinear interpolation upsampling algorithm and mapped to the spatial resolution corresponding to the high-resolution fusion features to generate high-resolution aligned structural features and high-resolution aligned semantic features.

[0152] High-resolution fused features, high-resolution aligned structural features, and high-resolution aligned semantic features are concatenated along the channel dimension, and channel compression and local context encoding are performed on the concatenated features to generate a dot plot regression feature tensor.

[0153] Perform pixel-level unfolding and restoration processing to map the point map regression feature tensor from the high-resolution feature space to the original pixel space of the enhanced image, generating a pixel-level point map regression feature tensor.

[0154] In Example 1, the pixel-level point map regression feature value at any original pixel spatial location and on any output channel in the pixel-level point map regression feature tensor is obtained by weighting and summing all input channel feature values ​​at the corresponding spatial location in the point map regression feature tensor according to the unfolding weights of the pixel unfolded convolution unit at the corresponding 4x4 pixel region offset position and the corresponding output channel, and then adding the bias term of the corresponding output channel. The 4x4 pixel region offset position includes a horizontal offset index and a vertical offset index. Both the horizontal and vertical offset indices are used to represent the pixel offset position within the 4x4 pixel region in the 4x4 spatial unfolding process. The output channel index of the pixel-level point map regression feature tensor is used to represent the channel position in the pixel-level point map regression feature tensor.

[0155] Input the pixel-level point map regression feature tensor into the pixel-level local 3D coordinate regression branch to regress the pixel-level local 3D coordinates corresponding to each pixel position and generate a pixel-level local 3D point map.

[0156] In Example 1, the pixel-level local 3D coordinate regression branch is composed of a shared pixel-level point map regression feature tensor input, a first 3x3 convolutional layer, a first normalization layer, a first activation layer, and a three-channel coordinate output layer connected sequentially.

[0157] The pixel-level point map regression feature values ​​of the pixel location and its surrounding three-by-three local neighborhood are input into the first three-by-three convolutional layer. The first three-by-three convolutional layer performs spatial weighted aggregation of the pixel-level point map regression feature values ​​of the pixel location and its surrounding three-by-three local neighborhood according to the preset convolution weights to obtain the local spatial context features corresponding to the pixel location.

[0158] Local spatial context features are input into the first normalization layer, where channel normalization is performed to obtain normalized local features. These normalized local features are then input into the first activation layer, where nonlinear activation is performed to obtain nonlinear local features. The nonlinear local features are then input into the three-channel coordinate output layer, where channel mapping is performed to output the first, second, and third coordinate channel response values. The first coordinate channel response value is determined as the horizontal local coordinate, the second coordinate channel response value as the vertical local coordinate, and the third coordinate channel response value is input into the Softplus activation function to determine the depth local coordinate. The horizontal, vertical, and depth local coordinates are physical coordinates in a local camera coordinate system constructed with the optical center of the camera corresponding to the m-th enhanced image as the origin and the camera coordinate axes as the reference directions.

[0159] By combining the horizontal local coordinates, vertical local coordinates, and depth local coordinates, pixel-level local 3D coordinates are formed. The same processing is performed on all pixel positions in the original pixel space of the m-th enhanced image to generate a pixel-level local 3D point map corresponding to the m-th enhanced image.

[0160] Input the pixel-level point map regression feature tensor into the point-level confidence regression branch, regress the point-level confidence corresponding to each pixel position, and generate a point-level confidence map;

[0161] In Example 1, the point-level confidence regression branch is composed of a shared pixel-level point map regression feature tensor input, a second 3x3 convolutional layer, a second normalization layer, a second activation layer, and a single-channel confidence output layer connected sequentially.

[0162] The pixel-level point map regression feature values ​​of the pixel location and its surrounding 3x3 local neighborhood are input into the second 3x3 convolutional layer. Spatially weighted aggregation of these feature values ​​is performed according to preset convolution weights to obtain confidence context features. These confidence context features are then input into the second normalization layer for channel normalization, resulting in normalized confidence features. The normalized confidence features are then input into the second activation layer for nonlinear activation, yielding nonlinear confidence features. These nonlinear confidence features are then input into a single-channel confidence output layer for channel mapping, outputting confidence response values. The confidence response values ​​are then input into a Sigmoid activation function to map point-level confidence between zero and one. The same processing is applied to all pixel locations in the original pixel space of the m-th enhanced image to generate the point-level confidence map corresponding to the m-th enhanced image.

[0163] The pixel-level local 3D point map is correlated pixel by pixel with the point-level confidence map to generate a pixel-level visual point map with confidence attributes.

[0164] Based on the pixel-level visual point maps corresponding to all enhanced images, an initial visually dense point cloud is constructed.

[0165] In Example 1, all pixel-level visual point maps corresponding to all enhanced images in the enhanced image dataset are aggregated and organized to generate an initial visual dense point cloud; the initial visual dense point cloud includes the pixel-level visual point map corresponding to each pixel position in the original pixel space of each enhanced image in the enhanced image dataset.

[0166] The improved DUSt3R 3D reconstruction network in this embodiment includes an initial embedding unit, a hierarchical window attention backbone network, a geometric anchor guided coding unit, a cross-modal cross-attention fusion layer, and a DUSt3R point map regression head, connected sequentially. The geometric anchor guided coding unit receives a scale-normalized depth map, an edge skeleton map, and a point cloud overlay confidence mask compressed by local logarithmic mapping generated from the projection of the synchronous point cloud dataset, and forms geometric anchor guided markers through convolutional coding and downsampling mapping. The cross-modal cross-attention fusion layer maps the geometric anchor guided markers to the spatial resolution corresponding to the multi-scale feature pyramid, and generates a cross-modal query matrix with a visual scale feature marker sequence for each scale type, and generates a cross-modal key matrix and a cross-modal value matrix with a geometric feature marker sequence, ultimately forming high-resolution fusion features, medium-resolution fusion features, and low-resolution fusion features.

[0167] Compared with the existing DUSt3R 3D reconstruction network, the improvements of the improved DUSt3R 3D reconstruction network are as follows:

[0168] The original feature extraction structure, which mainly relied on visual Transformer representation, was replaced with a hierarchical window attention backbone network that included a deformable convolutional coupling structure of SwinTransformer-V2 and DCNv4. The sampling offset of the deformable convolution was guided by high-frequency spatial gradient prior, which enabled the network to simultaneously capture local geometric details, model cross-window spatial dependencies, and express multi-scale semantics, thereby improving the feature stability in weakly textured surfaces, highly reflective curtain walls, repetitive structural regions, and large-view tilted images.

[0169] Instead of using LiDAR point clouds as post-processing registration data, geometric anchor point guidance markers are formed by sparse depth anchor point maps, edge skeleton maps, and point cloud coverage confidence masks. These markers participate in cross-modal cross-attention fusion within the network, allowing the DUSt3R point map regression process to be subject to feedforward physical constraints of absolute depth, structural boundaries, and point cloud coverage reliability.

[0170] In the similarity matrix calculation layer of cross-modal cross-attention, a point cloud coverage confidence mask bias and gated fusion mechanism are introduced. This makes the high-confidence geometric anchors exert stronger constraints on visual features, while the interference of point cloud missing or unreliable projection regions on visual features is suppressed. This reduces the scale uncertainty, local depth drift and false reconstruction of reflective areas in large-scale UAV oblique photography scenarios of traditional DUSt3R. As a result, the output initial visual dense point cloud has higher geometric consistency, boundary integrity and engineering surveying applicability before entering post-processing.

[0171] Using a synchronized point cloud dataset as the absolute geometric skeleton, global rotation alignment, translation alignment and scale correction are performed on the initial visual dense point cloud based on point-level confidence to generate a fused and enhanced dense point cloud.

[0172] In this embodiment, pixel-level local 3D coordinates and point-level confidence scores corresponding to each enhanced image are extracted from the initial visual dense point cloud. The pixel-level local 3D coordinates are used as visual points to be aligned, and the point-level confidence scores are used as the reliability attributes of the visual points to be aligned.

[0173] The synchronized point cloud dataset is used as the absolute geometric skeleton points, and the pixel position relationship between the visual points to be aligned and the absolute geometric skeleton points is established based on the point cloud projection coordinate set, generating a set of visual-laser associated point pairs.

[0174] In Example 1, for the m-th enhanced image, the laser point in the synchronous point cloud projected to the pixel position is determined as the absolute geometric skeleton point corresponding to the visual point to be aligned at the pixel position. When there are multiple laser points at the same pixel position, the laser point with the smallest depth value is selected as the absolute geometric skeleton point of the pixel position by comparing the depth values ​​corresponding to each laser point.

[0175] Extract the original three-dimensional physical coordinates of the absolute geometric skeleton points in the absolute spatial coordinate system of the test area, and combine the pixel-level local three-dimensional coordinates, the absolute geometric skeleton points in the absolute spatial coordinate system, and the point-level confidence to form a set of visual-laser associated point pairs.

[0176] Based on point-level confidence, construct geometric skeleton constraint weights;

[0177] In Example 1, by performing reverse mapping on point-level confidence, pixel positions with smaller point-level confidence are mapped to larger geometric skeleton constraint weights, and pixel positions with larger point-level confidence are mapped to smaller geometric skeleton constraint weights. The geometric skeleton constraint weights are obtained by linear interpolation between preset minimum and maximum weight values.

[0178] Based on the visual-laser associated point pair set and geometric skeleton constraint weights, the single-view scale parameter, single-view rotation matrix and single-view translation vector from the local camera coordinate system to the absolute spatial coordinate system of the survey area are solved independently for each enhanced image.

[0179] In Example 1, the three-dimensional coordinates of each visual point to be aligned in the m-th enhanced image in the local camera coordinate system are scaled using a single-view scale parameter, spatially rotated using a single-view rotation matrix, and spatially translated using a single-view translation vector, so that they are mapped to the absolute spatial coordinate system of the survey area.

[0180] For the m-th enhanced image, all visual-laser associated point pairs within the image are traversed. For each associated point pair, the spatial distance error between the coordinates of the visual point after scale, rotation, and translation transformations and the corresponding absolute geometric skeleton point is calculated. The spatial distance error is squared and multiplied by the corresponding geometric skeleton constraint weight. The weighted squared errors of all associated point pairs within the image are accumulated to obtain the single-view weighted residual sum of squares. By minimizing the single-view weighted residual sum of squares, the optimal single-view scale parameters, single-view rotation matrix, and single-view translation vector for the m-th enhanced image are obtained using the singular value decomposition algorithm.

[0181] ;

[0182] in, This represents the optimal single-view scale parameter corresponding to the m-th enhanced image obtained by solving the problem. This represents the optimal single-view rotation matrix. This represents the optimal single-view translation vector. , , Let these represent the scale parameter, rotation matrix, and translation vector to be optimized for the m-th enhanced image, respectively. This indicates the sequence number of the enhanced image in the enhanced image dataset. Indicates the first The pixel position of the enhanced image in the original pixel space. Indicates the first There is a set of pixel locations in the enhanced image that are visually associated with laser point pairs. Indicates the first Image enhancement at pixel location Geometric skeleton constraint weights at the location, Indicates the first Image enhancement at pixel location Pixel-level local 3D coordinates at the location Indicates the relationship with the first Image enhancement at pixel location The absolute geometric skeleton point corresponding to the visual point to be aligned at the location, located in the absolute spatial coordinate system of the test area. This represents the L2 norm.

[0183] Based on the single-view scale parameters, single-view rotation matrix and single-view translation vector corresponding to each enhanced image, spatial rotation alignment, translation alignment and scale correction are performed on all pixel-level local three-dimensional coordinates in the corresponding image to generate absolute coordinate visual points.

[0184] Based on the geometric skeleton constraint weights, an adaptive weighted update is performed on the absolute coordinate visual points and the corresponding absolute geometric skeleton points to generate geometric constraint visual points;

[0185] In Example 1, for the region in the m-th enhanced image where both absolute coordinate visual points and absolute geometric skeleton points exist at the pixel location, the geometric skeleton constraint weights at the pixel location are normalized, and the normalized geometric skeleton constraint weights are determined as the fusion weights of the absolute geometric skeleton points. The three-dimensional coordinates of the absolute coordinate visual points and the absolute geometric skeleton points are weighted and summed according to the fusion weights to obtain the geometric constraint visual points at the pixel locations.

[0186] For visual points with absolute coordinates that do not have corresponding absolute geometric skeleton points, retain their spatial coordinates after global rotation alignment, translation alignment and scale correction, and retain their point-level confidence attributes.

[0187] All the geometrically constrained visual points corresponding to the enhanced images are organized into a set according to the absolute spatial coordinate system of the survey area to generate a fused and enhanced dense point cloud.

[0188] In Example 1, the geometrically constrained visual points and their point-level confidence scores corresponding to each pixel position in the original pixel space of all enhanced images in the enhanced image dataset are uniformly organized. All geometrically constrained visual points are arranged in a set according to their spatial positions in the absolute spatial coordinate system of the survey area to generate a fused enhanced dense point cloud. The fused enhanced dense point cloud includes the geometrically constrained visual points and their point-level confidence scores corresponding to all pixel positions.

[0189] Hole repair and surface continuity completion are performed on the fused and enhanced dense point cloud to construct a fused and enhanced 3D surface model. The fused and enhanced 3D surface model is then texture-mapped with the enhanced image dataset to generate a real-scene 3D reconstruction model with absolute spatial coordinates and realistic texture expression. The output is a 3D reconstruction result that meets the accuracy requirements of engineering surveying and mapping.

[0190] In this embodiment, all geometrically constrained visual points and their point-level confidence scores in the fusion-enhanced dense point cloud are uniformly indexed according to the absolute spatial coordinate system of the survey area to form a set of points to be reconstructed on the surface.

[0191] In Example 1, all geometrically constrained visual points are uniformly numbered according to their spatial positions in the absolute spatial coordinate system of the survey area, and each geometrically constrained visual point is bound to its corresponding point-level confidence level to form a set of points to be reconstructed on the surface.

[0192] A local neighborhood search is performed on the point set to be reconstructed on the surface. Based on the neighborhood distance distribution and point-level confidence distribution of each geometrically constrained visual point, the hole boundary points in the fused and enhanced dense point cloud are identified.

[0193] In Example 1, a preset number of nearest neighbor points that are spatially closest to the geometrically constrained visual point are searched in the set of points to be reconstructed on the surface, forming a set of neighborhood points corresponding to the geometrically constrained visual point.

[0194] Calculate the Euclidean distance between the geometrically constrained visual point and each nearest neighbor in the neighborhood point set. Sum all the Euclidean distances and divide by the number of nearest neighbors in the neighborhood point set to obtain the average neighborhood distance. Geometrically constrained visual points whose average neighborhood distance is greater than or equal to the hole boundary distance threshold and whose point-level confidence is greater than or equal to the point-level confidence threshold are identified as hole boundary points.

[0195] The hole boundary distance threshold is determined by statistical analysis of the neighborhood average distance distribution of all geometrically constrained visual points in the fused and enhanced dense point cloud and combined with a preset proportional coefficient. The point-level confidence threshold is determined by statistical analysis of the point-level confidence distribution of all geometrically constrained visual points and selected based on the confidence quantile value.

[0196] Based on the cavity boundary points, candidate cavity regions are generated, and point completion processing based on neighborhood geometric constraints is performed on the candidate cavity regions to generate a complete point set;

[0197] In Example 1, all cavity boundary points are clustered according to their spatial adjacency to obtain multiple cavity candidate regions. The three-dimensional spatial coordinates of all cavity boundary points in the cavity boundary point set corresponding to the cavity candidate region are summed coordinate by coordinate and divided by the number of cavity boundary points in the cavity boundary point set to obtain the center point of the cavity candidate region.

[0198] Calculate the Euclidean distance between the center point of the candidate cavity region and the cavity boundary point. Divide the point-level confidence level corresponding to the cavity boundary point by the sum of the Euclidean distance and the stability length constant to obtain the supplementary point weighting coefficient corresponding to the cavity boundary point.

[0199] Based on the point supplementation weighting coefficient, the three-dimensional spatial coordinates of all cavity boundary points in the cavity boundary point set corresponding to the cavity candidate region are weighted and summed. The weighted summation result is divided by the sum of all point supplementation weighting coefficients to obtain the cavity completion point corresponding to the cavity candidate region. The same point supplementation process is performed on all cavity candidate regions to generate a completion point set.

[0200] The completed point set is merged into the point set to be reconstructed to generate a surface continuity completed point set, and local normal estimation is performed on the surface continuity completed point set to obtain the local normal vector;

[0201] In Example 1, for each point in the set of surface continuity completion points, a set of surrounding surface neighborhood points is searched in the set of surface continuity completion points. The spatial difference vector of each neighborhood point in the set of surface neighborhood points relative to the current point is calculated, and a unit vector is solved to minimize the sum of squares of the projections of all spatial difference vectors in the direction of the unit vector. The unit vector is then used as the initial normal vector.

[0202] Extract the coordinates of the camera optical center corresponding to the enhanced image that has a visual relationship with the current point, construct the viewpoint direction vector from the current point to the camera optical center, calculate the dot product of the initial normal vector and the viewpoint direction vector, if the dot product is less than zero, reverse the initial normal vector, and determine the processed unit vector as the local normal vector corresponding to the current point.

[0203] A set of triangular facets is constructed based on the surface continuity completion point set and local normal vectors to generate a fused and enhanced 3D surface model;

[0204] In Example 1, neighborhood triangulation is performed on the points in the surface continuity completion point set to form candidate triangular patches. The side lengths of the three sides of each candidate triangular patch are calculated, and the directional consistency between the local normal vectors corresponding to the three vertices of each candidate triangular patch is calculated.

[0205] When the side lengths of the three sides of a candidate triangle are all less than or equal to the triangle side length threshold, and the directional consistency between the local normal vectors corresponding to the three vertices of the candidate triangle is greater than or equal to the local normal consistency threshold, the corresponding candidate triangle is retained. All retained candidate triangles are combined into a triangle set, and the fused and enhanced 3D surface model is constructed by the surface continuity completion point set and the triangle set.

[0206] The side length threshold of the triangular facet is obtained by statistically analyzing the Euclidean distance distribution of all adjacent point pairs in the continuous completion point set of the surface and selecting its preset quantile as a unified side length constraint. The local normal consistency threshold is obtained by statistically analyzing the cosine value distribution of the angle between the local normal vectors corresponding to all adjacent points and selecting its preset quantile as a direction consistency constraint.

[0207] The triangular facets in the fused and enhanced 3D surface model are projected onto the enhanced image dataset, and the texture source image is determined based on the visibility markers of the triangular facets, the pixel projection area, and the point-level confidence.

[0208] In Example 1, the three vertices of the triangular facet are projected onto the pixel plane of each enhanced image in the enhanced image dataset to obtain the projection area of ​​the triangular facet in each enhanced image. For the projection area of ​​the triangular facet in any enhanced image, the average of the visibility marker, pixel projection area, and point-level confidence scores corresponding to the three vertices of the triangular facet relative to the enhanced image are calculated. The visibility score, projection area score, and confidence score are weighted and summed to obtain the texture selection score of the triangular facet in the enhanced image. The enhanced image with the highest texture selection score is determined as the texture source image corresponding to the triangular facet.

[0209] ;

[0210] in, Indicates the first The triangular facet in the first Texture selection scoring in enhanced images Indicates the first The triangular facet is relative to the first The visibility marker for amplitude-enhanced images is obtained by projecting the three vertices of a triangular facet onto the pixel plane of the amplitude-enhanced image and determining whether the projected area is occluded by other surfaces. Indicates the first The triangular facet is projected onto the first... The pixel projection area formed after image enhancement is obtained by counting the number of pixels covered by the triangular patch projection area on the pixel plane. Indicates the first The average point-level confidence score is obtained by summing the point-level confidence scores corresponding to the three vertices of a triangular facet and then dividing by the number of vertices. and These represent the weights for visibility score, projected area score, and confidence score, respectively. This represents the stable area constant used to avoid a denominator of zero.

[0211] Based on the texture source image, perform texture mapping on the fused and enhanced 3D surface model to generate a textured 3D surface model;

[0212] In Example 1, the texture sampling point in the triangular facet is projected onto the pixel plane of the texture source image to obtain the texture sampling position of the texture sampling point in the texture source image. The texture sampling point is located in the absolute space coordinate system of the test area. The color value is extracted from the texture sampling position in the texture source image to obtain the texture color value corresponding to the texture sampling point. The texture color value is bound to the corresponding triangular facet of the fused and enhanced three-dimensional surface model to generate a textured three-dimensional surface model.

[0213] The spatial coordinates, triangular facet topological relationships, and texture color information in the textured 3D surface model are uniformly encapsulated to generate a real-world 3D reconstruction model with absolute spatial coordinates and realistic texture expression, thus forming a 3D reconstruction result.

[0214] In Example 1, the real-scene 3D reconstruction model, the fused and enhanced dense point cloud, the fused and enhanced 3D surface model, and the textured 3D surface model are combined to form a 3D reconstruction result. The 3D reconstruction result is used to express the spatial geometric structure, fused and enhanced point cloud structure, continuous 3D surface structure, and real texture expression results of the target survey area in the absolute spatial coordinate system of the survey area.

[0215] A machine vision-based UAV oblique photogrammetry 3D reconstruction system is used to execute a machine vision-based UAV oblique photogrammetry 3D reconstruction method, including:

[0216] The data acquisition and synchronization module is used to collect data in collaboration with a tilting camera and LiDAR mounted on a drone, and to perform time synchronization and extrinsic parameter calibration by combining RTK / PPK, generating synchronized image datasets and synchronized point cloud datasets.

[0217] The image enhancement module is used to perform image quality enhancement processing on the synchronous image dataset to generate an enhanced image dataset;

[0218] The feature extraction module is used to extract multi-scale features from the augmented image dataset using a hierarchical window attention backbone network containing deformable convolutional coupling structures of SwinTransformer-V2 and DCNv4, and to construct a multi-scale feature pyramid.

[0219] The geometry building module is used to project the synchronized point cloud dataset onto the augmented image dataset and generate geometric anchor point guide markers;

[0220] The fusion module is used to fuse multi-scale features and geometric anchors in the improved DUSt3R network to guide 3D reconstruction;

[0221] The point cloud generation module is used to regress pixel-level 3D coordinates and confidence levels, generate visual point clouds and align them with laser point clouds to obtain fused and enhanced point clouds.

[0222] The model output module is used to perform surface reconstruction and texture mapping, and output the 3D reconstruction results.

[0223] Example 2: In a large-scale integrated construction scenario, the survey area simultaneously included newly poured concrete walls, water accumulation in the foundation pit, glass curtain wall material storage areas, exposed soil areas, and temporary steel structure support areas. During conventional oblique photogrammetry modeling, the implementers found that the concrete wall area had a significantly insufficient number of feature points due to its monotonous texture. The glass curtain wall area exhibited drastic brightness variations of the same component under different viewing angles due to reflections. The water accumulation area of ​​the foundation pit showed a large number of mismatched points. The model generated by the traditional SfM / MVS software in this survey area showed continuous holes and local warping. The deviation between the elevation of the foundation pit edge and the on-site checkpoints exceeded the allowable range for project acceptance.

[0224] In this embodiment, the implementer used a multi-rotor UAV equipped with a five-lens oblique photography camera, a LiDAR sensor, and an RTK / PPK dual-redundant positioning module to perform data acquisition. A single mission acquired 6840 enhanced forward-tilted images, approximately 82 million raw LiDAR point clouds, and 6840 sets of RTK / PPK pose records. After unified time synchronization, 43 images with a time deviation exceeding 8ms were removed, retaining 6797 synchronized images. After extrinsic parameter calibration, the average image projection reprojection error was 0.73 pixels, and the average deviation between the synchronized point cloud and the image edge projection was 1.92 pixels.

[0225] The implementer input synchronous images into the image enhancement process. Statistical analysis revealed that the overexposed pixel ratio in the glass curtain wall area decreased from 19.6% to 5.1%, and the average local contrast in the water accumulation area increased from 0.18 to 0.41. 12,600 training samples were extracted from the enhanced image dataset, including 3,600 samples of weak-texture concrete, 2,400 samples of reflective curtain walls, 1,800 samples of water accumulation boundaries, 3,000 samples of earthwork slopes, and 1,800 samples of steel structure occlusion. During training, the effective response rate of the traditional DUSt3R backbone on weak-texture samples was 71.4%. After adopting a hierarchical window attention and deformable convolution coupled backbone network, the effective response rate increased to 89.7%. On reflective curtain wall samples, the traditional backbone had a depth error ratio of 16.8%, which was reduced to 5.9% in this embodiment.

[0226] During the geometric anchor point generation process, the implementer projects the synchronous point cloud onto the corresponding augmented image view plane. In an augmented image containing reflections from concrete walls and curtain walls, there are a total of 18,432 effective laser projection points, of which 17,106 are effective pixel anchor points after applying a minimum depth retention strategy. The implementer simultaneously generates an effective projection indicator map to avoid pixels without points participating in gradient calculations with zero depth. During edge skeleton extraction, the system calculates depth difference only when adjacent pixels have effective projections, ultimately obtaining 2,314 edge skeleton pixels; if sparse depth map direct difference is used, the same image will generate 19,680 pseudo edge responses, indicating that this embodiment can avoid noisy skeletons caused by point cloud sparsity.

[0227] In the cross-modal fusion stage, the point cloud coverage confidence mask is mapped to high, medium, and low scales and injected with a cross-attention layer. For regions with a point cloud coverage confidence value higher than 0.75, the average attention weight of the geometrically injected features is 0.63; for regions with a coverage confidence value lower than 0.15, the average attention weight of the geometrically injected features is suppressed to 0.11. The implementer examined a set of water boundary images; the traditional visual model generated 22.3% of false surface points near the water surface, which was reduced to 6.4% in this embodiment after confidence bias guidance.

[0228] During the point map regression stage, the system outputs local 3D coordinates and point-level confidence for each pixel. In a set of weakly textured wall images, traditional methods can only recover 61.2% of the wall points, while this embodiment outputs a pixel-level visual point coverage of 94.6%. Among them, areas with point-level confidence below 0.35 are mainly concentrated at reflective boundaries and water surface boundaries, with an overlap rate of 87.1% with manually labeled difficult areas. The system uses the synchronized point cloud as the absolute geometric skeleton and independently solves the single-view scale parameters, rotation matrix, and translation vector for each enhanced image. In a long strip edge image, the average spatial distance between the visual points and laser skeleton points before correction was 0.184m, which decreased to 0.031m after single-view weighted alignment. In areas with point-level confidence below 0.3, the geometric skeleton constraint weights are automatically increased, and the average elevation error of the area decreases from 0.216m to 0.028m.

[0229] After generating the enhanced dense point cloud, the implementer performed void repair and surface continuity completion. Traditional methods resulted in 42 obvious voids at the interface between the concrete wall and the curtain wall, with the largest void area being approximately 13.7㎡. This embodiment identified 39 candidate void regions and generated a completion point set, reducing the largest residual void area to 1.8㎡. During triangular facet construction, approximately 14.8 million valid triangular facets were generated. Candidate facets were discarded due to excessive side length or inconsistent normals, accounting for 7.6%, resulting in a final surface continuity score of 96.2%.

[0230] During texture mapping, the system calculates a comprehensive score for each triangular facet based on visibility, pixel projected area, and point-level confidence, and selects the enhanced image with the highest score as the texture source. Traditional closest-view texture selection methods produce noticeable streaks and misalignments in glass curtain wall areas, with an abnormal texture seam rate of 14.1%; this embodiment reduces this to 3.7%. Verified at 42 on-site checkpoints, the traditional SfM / MVS model has an average planar error of 8.9cm, an average elevation error of 13.6cm, and a maximum elevation error of 24.8cm; the real-scene 3D reconstruction model generated in this embodiment has an average planar error of 3.2cm, an average elevation error of 2.6cm, and a maximum elevation error of 4.1cm. When verifying the earthwork volume of a foundation pit area, the manual verification volume was 38,640m³, the traditional model calculation volume was 42,110m³ with an error of 8.98%, and this embodiment's calculation volume was 38,980m³ with an error of 0.88%, meeting the requirements for engineering surveying and earthwork verification applications.

[0231] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A machine vision-based UAV oblique photogrammetry 3D reconstruction method, characterized in that, include: Data is collected by using a drone equipped with a tilting camera and a lidar, and time synchronization and extrinsic parameter calibration are completed by combining RTK / PPK to generate synchronized image datasets and synchronized point cloud datasets. Perform image quality enhancement processing on the synchronized image dataset to generate an enhanced image dataset; A hierarchical window attention backbone network containing deformable convolutional coupling structures of SwinTransformer-V2 and DCNv4 is used to extract multi-scale features from the enhanced image dataset, and a multi-scale feature pyramid composed of high-resolution texture features, medium-resolution structural features and low-resolution semantic features is constructed. The synchronous point cloud dataset is projected onto the corresponding view plane of the enhanced image dataset based on the extrinsic parameter calibration results, generating a sparse depth anchor map, an edge skeleton map, and a point cloud coverage confidence mask, which are then encoded to form geometric anchor guide markers. In the improved DUSt3R 3D reconstruction network, the multi-scale feature pyramid and geometric anchor point guide markers are fused through a cross-modal cross-attention mechanism to output a fused feature set; Based on the fused feature set, pixel-level local 3D coordinates and point-level confidence are regressed within the DUSt3R point map regression head to generate an initial visual dense point cloud; Using a synchronized point cloud dataset as the absolute geometric skeleton, global rotation alignment, translation alignment and scale correction are performed on the initial visual dense point cloud based on point-level confidence to generate a fused and enhanced dense point cloud. Hole repair and surface continuity completion are performed on the fused and enhanced dense point cloud to construct a fused and enhanced 3D surface model. The fused and enhanced 3D surface model is then texture-mapped with the enhanced image dataset to generate a real-scene 3D reconstruction model with absolute spatial coordinates and realistic texture expression. The output is a 3D reconstruction result that meets the accuracy requirements of engineering surveying and mapping.

2. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The method of using a drone equipped with a tilting camera and LiDAR to collaboratively collect data, and combining RTK / PPK to complete time synchronization and extrinsic parameter calibration, includes: Using a multi-rotor UAV equipped with a five-lens oblique photography camera, a lidar sensor, and an RTK / PPK dual-redundant positioning module, collaborative flight data acquisition is performed on the target survey area to obtain oblique photography image datasets, lidar raw point cloud datasets, and corresponding pose datasets. The oblique photography image datasets, lidar raw point cloud datasets, and pose datasets are synchronized in time according to a unified time synchronization reference. An absolute spatial coordinate system for the survey area is established based on the RTK / PPK positioning results. External parameter calibration is performed on the oblique photography camera and lidar sensor to generate synchronized image datasets and synchronized point cloud datasets.

3. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The image quality enhancement process includes brightness normalization, reflection suppression, and local contrast enhancement.

4. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The hierarchical window attention backbone network, employing a deformable convolutional coupling structure of SwinTransformer-V2 and DCNv4, performs multi-scale feature extraction on the enhanced image dataset, including: Each enhanced image in the enhanced image dataset is input into the initial embedding unit. Using a two-dimensional convolutional unit with a kernel size of 4×4 and a stride of 4, non-overlapping block partitioning and channel mapping are performed on each enhanced image, so that each enhanced image is mapped from the original pixel space to the initial feature tensor. The initial feature tensor is input into a hierarchical window attention backbone network containing a three-level hierarchical processing structure. In each level of hierarchical processing, the Laplacian operator is used to perform high-frequency spatial gradient calculation on the current input feature tensor to generate a visual edge saliency feature map. After combining the visual edge saliency feature map with the corresponding input feature tensor, the input is fed into the offset prediction convolutional unit to generate an adaptive offset field. The corresponding input feature tensor is then fed into the modulation prediction convolutional unit to generate a modulation coefficient field. Based on the adaptive offset field and modulation coefficient field, deformable convolutional local enhancement processing is performed on the corresponding input feature tensor through deformable convolutional local enhancement unit in each level of hierarchical processing to generate locally enhanced feature tensors; The local enhancement feature tensor is divided into multiple local windows. Query mapping, key mapping and value mapping are performed on the feature vectors in each local window to generate a local query matrix, a local key matrix and a local value matrix. In the window attention unit, window attention calculation is performed on the local query matrix, local key matrix and local value matrix in each local window of each level of hierarchical processing to generate the window attention feature tensor. In the shift window reorganization unit, the window attention feature tensor is processed by shift window reorganization, and a second window attention interaction calculation is performed in the reorganized window to generate a cross-window related feature tensor. In the feature fusion unit, the local enhancement feature tensor, the window attention feature tensor, the cross-window association feature tensor and the corresponding input feature tensor are coupled and fused to generate the fused feature tensor corresponding to the hierarchical processing. For each level of layered processing, the fusion feature tensor is updated by layered downsampling. When the layer level is the first or second level, a downsampling convolutional unit with a kernel size of 2×2 and a stride of 2 is used to perform spatial downsampling and channel mapping on the corresponding fusion feature tensor to generate the output feature tensor of the corresponding layer. When the layer level is the third level, the fusion feature tensor corresponding to the third level is directly used as the output feature tensor of the third level. Perform channel projection processing on the output feature tensors at each level to generate high-resolution texture features, medium-resolution structural features, and low-resolution semantic features; High-resolution texture features, medium-resolution structural features, and low-resolution semantic features are combined to construct a multi-scale feature pyramid for the corresponding enhanced image.

5. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The step of projecting the synchronized point cloud dataset onto the corresponding view plane of the enhanced image dataset based on the extrinsic parameter calibration results includes: Establish a correspondence between each frame of synchronized point cloud in the synchronized point cloud dataset and the corresponding enhanced image in the enhanced image dataset. Based on the extrinsic parameter calibration results of the corresponding enhanced image, transform each laser point in each frame of synchronized point cloud from the lidar coordinate system to the camera coordinate system of the corresponding enhanced image to generate the camera coordinate point set of the corresponding enhanced image. Perform view plane projection processing on each camera coordinate point in each camera coordinate point set, and map each camera coordinate point located in the imaging view frustum of the corresponding enhanced image to the pixel plane of the corresponding enhanced image to generate a point cloud projection coordinate set. Based on the point cloud projection coordinate set and the depth values ​​of the corresponding camera coordinate points, a sparse depth anchor map and an effective projection indicator map of the corresponding enhanced image are constructed. Based on the sparse depth anchor point map and the effective projection indicator map, the edge skeleton map of the corresponding enhanced image is extracted; Based on the coverage distribution of the point cloud projection coordinate set in the corresponding augmented image pixel plane, a point cloud coverage confidence mask for the corresponding augmented image is constructed. Based on the maximum effective depth value in the m-th enhanced image, local depth log normalization mapping compression is performed on all depth anchor values ​​in the sparse depth anchor map to generate a scale-normalized depth map. The scale-normalized depth map, edge skeleton map, and point cloud coverage confidence mask are stacked sequentially according to the channel dimension, so that the scale-normalized depth value, edge skeleton value, and point cloud coverage confidence value at the same pixel location are arranged sequentially in the channel dimension, generating the geometric guidance tensor of the corresponding enhanced image. Perform convolutional encoding and downsampling mapping on the geometric guidance tensor to generate geometric anchor point guidance marks for the corresponding enhanced image.

6. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The improved DUSt3R 3D reconstruction network integrates multi-scale feature pyramids and geometric anchor point guidance markers through a cross-modal cross-attention mechanism, including: The geometric anchor point guide markers are mapped to the spatial resolution corresponding to the multi-scale feature pyramid to generate high-resolution geometric guide features, medium-resolution geometric guide features and low-resolution geometric guide features; Channel alignment processing is performed on the three visual scale features and the three geometric guidance features respectively to obtain channel-aligned visual scale features and channel-aligned geometric features; The channel-aligned visual scale features and channel-aligned geometric features are respectively expanded into visual scale feature labeling sequences and geometric feature labeling sequences according to their spatial position order; In the cross-modal cross-attention layer of the improved DUST3R 3D reconstruction network, a cross-modal query matrix is ​​generated using a visual scale feature label sequence, and a cross-modal key matrix and a cross-modal value matrix are generated using a geometric feature label sequence. Based on the cross-modal query matrix, cross-modal key matrix, and cross-modal value matrix, cross-modal cross-attention calculation guided by confidence bias is performed at each scale type to generate a geometrically injected feature label sequence; Gated fusion processing is performed on the visual scale feature label sequence and the geometric injection feature label sequence to generate a cross-modal fusion label sequence; The cross-modal fusion label sequence under each scale type is restored into a two-dimensional feature grid of the corresponding scale type according to its spatial label number, generating high-resolution fusion features, medium-resolution fusion features and low-resolution fusion features; The high-resolution fusion features, medium-resolution fusion features, and low-resolution fusion features are combined to form a fusion feature set.

7. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The process of regressing pixel-level local 3D coordinates and point-level confidence scores within the DUSt3R point map regression head based on the fused feature set includes: The high-resolution, medium-resolution, and low-resolution fusion features in the fusion feature set are input into the DUSt3R point map regression head. The medium-resolution and low-resolution fusion features are then enlarged in spatial scale using a bilinear interpolation upsampling algorithm and mapped to the spatial resolution corresponding to the high-resolution fusion features, generating high-resolution aligned structural features and high-resolution aligned semantic features. High-resolution fused features, high-resolution aligned structural features, and high-resolution aligned semantic features are concatenated along the channel dimension, and channel compression and local context encoding are performed on the concatenated features to generate a dot plot regression feature tensor. Perform pixel-level unfolding and restoration processing to map the point map regression feature tensor from the high-resolution feature space to the original pixel space of the enhanced image, generating a pixel-level point map regression feature tensor. Input the pixel-level point map regression feature tensor into the pixel-level local 3D coordinate regression branch to regress the pixel-level local 3D coordinates corresponding to each pixel position and generate a pixel-level local 3D point map. Input the pixel-level point map regression feature tensor into the point-level confidence regression branch, regress the point-level confidence corresponding to each pixel position, and generate a point-level confidence map; The pixel-level local 3D point map is correlated pixel by pixel with the point-level confidence map to generate a pixel-level visual point map with confidence attributes. Based on the pixel-level visual point maps corresponding to all enhanced images, an initial visually dense point cloud is constructed.

8. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The method of using a synchronized point cloud dataset as the absolute geometric skeleton and performing global rotation alignment, translation alignment, and scale correction on the initial visual dense point cloud based on point-level confidence includes: Pixel-level local 3D coordinates and point-level confidence scores are extracted from each enhanced image from the initial visual dense point cloud. The pixel-level local 3D coordinates are used as visual points to be aligned, and the point-level confidence scores are used as the reliability attributes of the visual points to be aligned. The synchronized point cloud dataset is used as the absolute geometric skeleton points, and the pixel position relationship between the visual points to be aligned and the absolute geometric skeleton points is established based on the point cloud projection coordinate set, generating a set of visual-laser associated point pairs. Based on point-level confidence, construct geometric skeleton constraint weights; Based on the visual-laser associated point pair set and geometric skeleton constraint weights, the single-view scale parameter, single-view rotation matrix and single-view translation vector from the local camera coordinate system to the absolute spatial coordinate system of the survey area are solved independently for each enhanced image. Based on the single-view scale parameters, single-view rotation matrix and single-view translation vector corresponding to each enhanced image, spatial rotation alignment, translation alignment and scale correction are performed on all pixel-level local three-dimensional coordinates in the corresponding image to generate absolute coordinate visual points. Based on the geometric skeleton constraint weights, an adaptive weighted update is performed on the absolute coordinate visual points and the corresponding absolute geometric skeleton points to generate geometric constraint visual points; All the geometrically constrained visual points corresponding to the enhanced images are grouped and organized according to the absolute spatial coordinate system of the survey area to generate a fused and enhanced dense point cloud.

9. The UAV oblique photogrammetry 3D reconstruction method based on machine vision according to claim 1, characterized in that, The process of performing hole repair and surface continuity completion on the fused and enhanced dense point cloud to construct a fused and enhanced 3D surface model includes: All geometrically constrained visual points and their point-level confidence scores in the fusion-enhanced dense point cloud are uniformly indexed according to the absolute spatial coordinate system of the survey area to form a set of points to be reconstructed on the surface. A local neighborhood search is performed on the point set to be reconstructed on the surface. Based on the neighborhood distance distribution and point-level confidence distribution of each geometrically constrained visual point, the hole boundary points in the fused and enhanced dense point cloud are identified. Based on the cavity boundary points, candidate cavity regions are generated, and point completion processing based on neighborhood geometric constraints is performed on the candidate cavity regions to generate a complete point set; The completed point set is merged into the point set to be reconstructed to generate a surface continuity completed point set, and local normal estimation is performed on the surface continuity completed point set to obtain the local normal vector; A set of triangular facets is constructed based on the surface continuity completion point set and local normal vectors to generate a fused and enhanced 3D surface model; The triangular facets in the fused and enhanced 3D surface model are projected onto the enhanced image dataset, and the texture source image is determined based on the visibility markers of the triangular facets, the pixel projection area, and the point-level confidence. Based on the texture source image, perform texture mapping on the fused and enhanced 3D surface model to generate a textured 3D surface model; The spatial coordinates, triangular facet topological relationships, and texture color information in the textured 3D surface model are uniformly encapsulated to generate a real-world 3D reconstruction model with absolute spatial coordinates and realistic texture expression, thus forming a 3D reconstruction result.

10. A machine vision-based UAV oblique photogrammetry 3D reconstruction system, used to execute the machine vision-based UAV oblique photogrammetry 3D reconstruction method according to any one of claims 1-9, characterized in that, include: The data acquisition and synchronization module is used to collect data in collaboration with a tilting camera and LiDAR mounted on a drone, and to perform time synchronization and extrinsic parameter calibration by combining RTK / PPK, generating synchronized image datasets and synchronized point cloud datasets. The image enhancement module is used to perform image quality enhancement processing on the synchronous image dataset to generate an enhanced image dataset; The feature extraction module is used to extract multi-scale features from the augmented image dataset using a hierarchical window attention backbone network containing deformable convolutional coupling structures of SwinTransformer-V2 and DCNv4, and to construct a multi-scale feature pyramid. The geometry building module is used to project the synchronized point cloud dataset onto the augmented image dataset and generate geometric anchor point guide markers; The fusion module is used to fuse multi-scale features and geometric anchors in the improved DUSt3R network to guide 3D reconstruction; The point cloud generation module is used to regress pixel-level 3D coordinates and confidence levels, generate visual point clouds and align them with laser point clouds to obtain fused and enhanced point clouds. The model output module is used to perform surface reconstruction and texture mapping, and output the 3D reconstruction results.

Images