An unmanned aerial vehicle large-scale fine surface reconstruction method and system fusing implicit neural representation and surface elements

Abstract

The application discloses a UAV large-scale fine surface reconstruction method and system fusing implicit neural representation and surface elements, and relates to the field of UAVs.The method comprises the following steps: acquiring multi-view aerial images of a UAV, restoring initial surface geometry and discretizing the initial surface geometry into surface elements with spatial positions and normals, and binding learnable neural descriptors.A deformation graph node is arranged in association with adjacent surface elements, so that the surface elements are synchronously updated according to the node pose.A symbolic distance field is constructed by aggregating neural descriptors of surface elements in the neighborhood of a query point and by regressing the neural descriptors through a neural network.Predicted images are generated based on the camera pose, and a reconstruction loss is fed back to update the node pose and the neural descriptors.Finally, the surface elements are driven by a correction pose, a dense symbolic distance field is generated, and a three-dimensional curved surface model is extracted.The method improves the local geometric detail preservation accuracy of complex and fine curved surface regions in the three-dimensional reconstruction of a large-scale UAV scene.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicles (UAVs), and more particularly to a method and system for large-scale fine surface reconstruction of UAVs that integrates implicit neural representations and surface elements. Background Technology

[0002] With the rapid development of UAV data acquisition technology, the demand for large-scale 3D reconstruction based on multi-view aerial imagery continues to grow in fields such as urban surveying, infrastructure inspection, and geographic information acquisition. Using high-resolution multi-view images captured by UAVs as input, restoring the 3D spatial structure of the target area and generating a detailed 3D surface model are the core technological components supporting these applications.

[0003] In related technologies, researchers have proposed a 3D surface reconstruction method based on implicit neural networks. This method uses a neural network to globally parameterize the symbolic distance field of the target scene. Arbitrary 3D spatial coordinates are input into a multilayer perceptron, which outputs the corresponding symbolic distance values. Then, using the photometric error of multi-view images as the optimization objective, gradients are backpropagated to iteratively update the global network parameters. Finally, isosurfaces are extracted from the converged symbolic distance field to obtain a 3D surface model.

[0004] However, when the above methods are applied to large-scale drone scenarios, the global neural network must simultaneously compress a large amount of geometric and appearance information from different locations and scales in space into the same set of network parameters. Loss gradients from different regions of the scene are continuously superimposed on the same set of parameters, and the feature encodings of different regions compete with each other for the parameter representation space. As the scale of the scene increases, this competition and interference intensifies, causing the geometric morphology of complex regions such as the edges of fine components in the reconstructed surface model to become overly smooth, and local fine geometric features to become distorted and blurred. Summary of the Invention

[0005] This application provides a method and system for large-scale fine surface reconstruction of UAVs that integrates implicit neural representation and surface elements, which can improve the accuracy of local geometric details preservation in complex fine surface regions during 3D reconstruction of large-scale UAV scenes.

[0006] Firstly, this application provides a method for large-scale fine-grained surface reconstruction of unmanned aerial vehicles (UAVs) that integrates implicit neural representations and surface elements. This method is applied to a large-scale fine-grained surface reconstruction system for UAVs. The method includes: acquiring aerial images of a target area taken by the UAV from multiple perspectives; recovering the initial surface geometry based on the aerial images and discretizing it into a set of surface elements, each surface element having a spatial position and surface normal; binding learnable neural descriptors to the surface elements; arranging deformation graph nodes within the target area; establishing associations between surface elements and their neighboring deformation graph nodes, so that the spatial position and surface normal of the surface elements are updated synchronously with the pose parameters of the associated deformation graph nodes; and retrieving the neighborhood of a query point within the target area. The surface elements within the target area are aggregated, and the neural descriptors bound to the retrieved surface elements are input into a neural network. The symbolic distance value of the query point is obtained by regression, and this symbolic distance value constitutes a symbolic distance field within the target area. Based on the camera pose corresponding to the aerial image, a predicted image is generated from the symbolic distance field through differentiable rendering. The gradient is backpropagated based on the reconstruction loss between the predicted image and the aerial image to update the pose parameters and neural descriptors of the deformable map nodes, thereby obtaining the corrected pose parameters and the target neural descriptor. The surface elements are driven to the corrected position using the corrected pose parameters. A dense symbolic distance field is generated based on the surface elements at the corrected position and the bound target neural descriptor. A three-dimensional surface model covering the target area is extracted from the dense symbolic distance field.

[0007] In the above embodiments, the system discretizes the initial surface geometry restored from UAV aerial imagery into surface elements with spatial position and surface normal. These surface elements are then bound to learnable neural descriptors, and the pose updates of the surface elements are driven by deformation graph nodes. The symbolic distance value of the query point is obtained by aggregating the neural descriptors bound to neighboring surface elements and then regressing them via a neural network. The parameterized carrier of the symbolic distance field is dispersed across sparse surface elements and deformation graph nodes. This processing method is suitable for 3D reconstruction tasks of large target areas such as urban building complexes and power transmission towers, ensuring that the geometric forms of fine components such as building parapets, ridgelines, and tower members are preserved in the large-scene reconstruction results.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the steps of binding learnable neural descriptors to surface elements, arranging deformable graph nodes within a target area, and establishing associations between surface elements and adjacent deformable graph nodes specifically include: assigning a learnable neural descriptor to each surface element, registering the neural descriptor, the spatial position of the surface element, and the surface normal as binding attributes of the surface element; downsampling the spatial position of the surface element within the target area according to a preset voxel size, using the downsampled spatial position as the initial position of the deformable graph node, and initializing each deformable graph node... The pose parameters used to characterize local rigid transformations are transformed; within the target region, a preset number of deformation map nodes that are closest to each surface element are retrieved as associated deformation map nodes of the surface element, and skin weights are constructed based on the spatial distance between the surface element and the associated deformation map nodes; the pose parameters of the associated deformation map nodes are weighted and composited based on the skin weights, and the weighted and composited pose parameters are applied to the spatial position and surface normal of the surface element, so that the spatial position and surface normal of the surface element are updated synchronously with the pose parameters of the associated deformation map nodes, and the neural descriptors bound to the surface element maintain index consistency with the updated spatial position.

[0009] In the above embodiments, the system registers neural descriptors, spatial positions, and surface normals together as surface element binding attributes, and downsamples the initial positions of deformable graph nodes according to a preset voxel size. For each surface element, a skin weight is constructed based on nearest neighbors and spatial distance. The weighted composite pose parameters are then applied synchronously to the spatial position and surface normal of the surface element, and the bound neural descriptors maintain index consistency with the updated positions. This binding mechanism is suitable for multi-flight, multi-time-period UAV acquisition scenarios. It provides an adjustable space for camera pose measurement deviations and geometric misalignments between flight paths during the acquisition process, ensuring that curved surfaces with different orientations, such as building facades and bridge structures, maintain the correspondence between appearance features and geometric positions after pose correction.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the steps of retrieving surface elements in the neighborhood of a query point within a target area, aggregating the neural descriptors bound to the retrieved surface elements and inputting them into a neural network to regress and obtain the symbolic distance value of the query point specifically include: establishing a spatial index structure based on the spatial position of surface elements within the target area, and retrieving a preset number of surface elements closest to the query point as neighborhood surface elements based on the spatial index structure; calculating the aggregation weight of the neighborhood surface elements to the query point based on the spatial distance between the neighborhood surface elements and the query point and the orientation deviation of the surface normal of the neighborhood surface elements relative to the query point, wherein the aggregation weight increases as the spatial distance and orientation deviation decrease; performing a weighted summation of the neural descriptors bound to the neighborhood surface elements based on the aggregation weight to obtain the aggregation feature corresponding to the query point; concatenating the aggregation feature with the relative position vector of the query point relative to the spatial position of the neighborhood surface elements and inputting it into a neural network to regress and obtain the symbolic distance value of the query point, wherein the symbolic distance value constitutes a symbolic distance field as the query point traverses within the target area.

[0011] In the above embodiments, the system establishes a spatial index structure based on the spatial location of surface elements, retrieves the nearest neighbor surface elements of the query point according to a preset number, and constructs an aggregation weight by combining spatial distance and surface normal orientation deviation. Surface elements that are closer and have more consistent normal orientation contribute more to the query point. The aggregation features are concatenated with the relative position vector and then input into the neural network to regress the symbolic distance value. This aggregation method is suitable for urban scene reconstruction tasks containing complex structures such as thin-walled walls, dense railings, and facilities attached to the building facade. It effectively distinguishes the opposite curved surfaces on both sides of the thin wall during the symbolic distance field regression process, avoiding the adhesion of adjacent opposite-sided components in the reconstruction results.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the steps of generating a predicted image from a symbolic distance field through differentiable rendering based on the camera pose corresponding to the aerial image, backpropagating gradients based on the reconstruction loss of the predicted image and the aerial image, and updating the pose parameters and neural descriptors of the deformable map nodes specifically include: determining the camera optical center based on the camera pose corresponding to the aerial image, and projecting rays from the camera optical center along each pixel direction of the aerial image to the target region; performing spherical tracing along the ray on the symbolic distance field, and taking the intersection of the ray and the zero isosurface of the symbolic distance field as the surface intersection point corresponding to the pixel direction; retrieving surface elements in the neighborhood of the surface intersection point, aggregating the neural descriptors bound to the retrieved surface elements, and inputting the aggregation result into a color decoding network to regress and obtain the pixel color at the surface intersection point, and backfilling the pixel color according to the pixel direction to form the predicted image; constructing a reconstruction loss based on the difference in pixel color between the predicted image and the aerial image, and backpropagating the gradient of the reconstruction loss to the pose parameters and neural descriptors of the deformable map nodes respectively, and iteratively updating the pose parameters and neural descriptors of the deformable map nodes through gradient descent until the reconstruction loss converges, thereby obtaining the corrected pose parameters and target neural descriptors.

[0013] In the above embodiment, the system projects rays along the pixel direction from the camera's optical center and performs spherical tracing on the symbolic distance field, using the intersection of the ray and the zero isosurface as the surface intersection point. Neural descriptors bound to surface elements within the neighborhood of the surface intersection point are aggregated and then used by a color decoding network to regress pixel colors, forming a reconstruction loss with the aerial image. Gradients are then backpropagated for the pose parameters of the deformable map nodes and the neural descriptors, respectively. This differentiable rendering path is suitable for application scenarios with initial camera pose deviations in UAV oblique photogrammetry tasks, enabling pose alignment and geometric appearance updates to be jointly completed under the same image supervision, resulting in a surface geometry and appearance representation consistent with multi-view aerial image observations.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of extracting a 3D surface model covering the target region from the dense symbolic distance field, the method further includes: performing differentiable rendering of the dense symbolic distance field according to the camera pose to obtain a rendered image that is pixel-wise aligned with the aerial image; calculating the absolute value of the color difference between the aerial image and the rendered image at the same pixel position to obtain a residual image; back-projecting pixels in the residual image whose residuals are greater than a preset residual threshold along the imaging direction corresponding to the camera pose onto the 3D surface model to obtain a surface region to be refined on the 3D surface model; supplementing sampling points in the surface region to be refined as new surface elements, and using the distance-weighted interpolation result of the target neural descriptors in the neighborhood as the initial neural descriptors bound to the new surface elements; re-searching for neighborhood surface elements in the surface region to be refined according to the generation method of the dense symbolic distance field, aggregating the bound neural descriptors, and regressing the symbolic distance value to obtain a refined symbolic distance field; extracting isosurfaces from the refined symbolic distance field to replace the surface portion of the 3D surface model in the surface region to be refined to obtain a refined 3D surface model.

[0015] In the above embodiments, the system performs differentiable rendering of the dense symbolic distance field according to the camera pose to obtain a pixel-aligned rendered image. Pixels in the residual image that exceed a preset residual threshold are back-projected along the imaging direction to the 3D surface model to locate the surface region to be refined. After adding new surface elements and initial neural descriptors in this region, the system reverts to the symbolic distance field. This refinement process is suitable for scenarios in infrastructure inspection where high geometric accuracy is required for key parts such as bridge expansion joints, power transmission line hardware, and building eaves decorations. It concentrates optimization resources on local surface regions with high image residuals, resulting in a refined 3D surface model with improved geometric accuracy.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of extracting the three-dimensional surface model covering the target area from the dense symbolic distance field, the method further includes: sampling points on the refined three-dimensional surface model at a preset sampling interval, and filtering out aerial images that are not self-occluded by the refined three-dimensional surface model according to the camera pose to obtain visible aerial images corresponding to the sampling points; projecting the sampling points onto the visible aerial images according to the camera pose and sampling colors at the projected pixel positions to obtain candidate texture colors of the sampling points under each visible aerial image; retrieving surface elements in the neighborhood of the sampling points within the target area, and using the distance-weighted average of the retrieved surface normals as the surface normal of the sampling points; converting the included angle into a viewpoint weight corresponding to the candidate texture color according to a preset mapping relationship that increases as the included angle between the surface normal and the viewpoint direction of the visible aerial image decreases; weighting and summing the candidate texture colors with the viewpoint weights and arranging them according to the sampling point positions to obtain a texture map bound to the refined three-dimensional surface model.

[0017] In the above embodiments, the system samples points on the refined 3D curved surface model at preset sampling intervals, filters out self-occluded aerial images according to camera pose to obtain visible aerial images, and generates viewpoint weights based on the angle between the surface normal of the sampling points and the visible viewpoint direction, and then weights and sums the candidate texture colors. This texture generation method is suitable for application scenarios where the number of visible images corresponding to each sampling point in large-scene aerial photography data from UAVs varies greatly with the viewpoint distribution. It allows the orthogonal images of building facades to obtain higher weights in texture mapping, resulting in a texture map that matches the geometric orientation of the curved surface, supporting the visual representation of urban 3D reality models in geographic information products.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, the step of supplementing sampling points in the surface region to be refined as new surface elements specifically includes: back-projecting the residual values ​​of the residual images corresponding to each pixel position in the surface region to be refined onto the surface of the three-dimensional surface model along the imaging direction to obtain the surface residual values ​​at each surface point in the surface region to be refined; dividing the surface region to be refined into multiple residual level sub-regions according to the surface residual values, wherein the residual level sub-region with larger surface residual values ​​corresponds to a higher supplementary sampling density; performing supplementary sampling on the surface of the three-dimensional surface model according to the supplementary sampling density corresponding to each residual level sub-region to obtain the new surface elements; determining the transition sampling density at the boundary of two adjacent residual level sub-regions with different residual levels by distance-weighted interpolation of the supplementary sampling densities on both sides, and supplementing the transition surface elements according to the transition sampling density, so that the refined symbolic distance field remains continuous between different residual level sub-regions.

[0019] In the above embodiment, the system back-projects the residual values ​​corresponding to each pixel position in the surface area to be refined onto the surface of the three-dimensional surface model along the imaging direction to obtain surface residual values, and divides the surface area to be refined into multiple residual level sub-regions according to the surface residual values. The sub-region with larger residuals is assigned a higher supplementary sampling density. At the same time, at the boundary of adjacent sub-regions of different residual levels, the transition sampling density is determined by weighted interpolation based on the distance between the supplementary sampling densities on both sides, and transition surface elements are supplemented. This adaptive sampling method is suitable for application scenarios where the geometric complexity distribution is uneven within the same surface area to be refined in UAV aerial photography, such as the coexistence of high-frequency detail areas like window frames and air conditioner outdoor units on building facades with large areas of flat walls. It concentrates computing resources on the local surface with severe geometric distortion in a residual-driven manner, avoiding redundant sampling overhead in low residual flat areas. At the same time, it maintains the continuity of the refined symbolic distance field between different residual level sub-regions through smooth interpolation of transition sampling density at the boundary, avoiding the appearance of step-like discontinuities in the isosurface extraction results at the sub-region boundary due to abrupt changes in sampling density. This supports the geometrically smooth connection between local refinement results and the global surface model in large-scale UAV 3D reconstruction.

[0020] In a second aspect, embodiments of this application provide a large-scale fine surface reconstruction system for unmanned aerial vehicles (UAVs). The UAV large-scale fine surface reconstruction system includes: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the UAV large-scale fine surface reconstruction system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a large-scale fine surface reconstruction system for unmanned aerial vehicles (UAVs), cause the large-scale fine surface reconstruction system for UAVs to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a large-scale fine surface reconstruction system for unmanned aerial vehicles (UAVs), cause the UAV large-scale fine surface reconstruction system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the UAV large-scale fine surface reconstruction system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By employing a hybrid representation method that discretizes the initial surface geometry of UAV aerial imagery into surface elements with spatial position and surface normal, binds a learnable neural descriptor to each surface element, and drives the pose update of surface elements by deformation graph nodes, the parameterization of the symbolic distance field is distributed across sparse deformation graph nodes and dense surface elements. The gradient of the image reconstruction loss is only backpropagated to the associated surface elements and deformation graph nodes within the local neighborhood of the query point. This effectively solves the problem of excessive smoothing of the geometry of fine components such as building edges, roof ridge outlines, and tower members, as well as local detail distortion and blurring, caused by the competition between different regional features in the same set of network parameters in large-scale UAV scenarios such as urban surveying and park aerial photography. This enables the discriminative expression of the geometric features of fine components in the reconstruction of 3D curved surfaces of large target areas, allowing the reconstructed 3D curved surface model to effectively restore the local geometric information carried by high-resolution aerial imagery, supporting the geometric restoration needs of key parts such as building facade accessories and small facility components in urban surveying and infrastructure inspection.

[0026] 2. By adopting a design that registers neural descriptors, spatial positions, and surface normals together as surface element binding attributes, uses voxel downsampling to preserve positions as the initial positions of deformable graph nodes, and constructs skin weights based on nearest neighbor association and spatial distance to weighted composite of pose parameters, the pose adjustment of deformable graph nodes, after weighted composite processing, synchronously acts on the spatial position and surface normal of surface elements. The bound neural descriptors maintain index consistency with the updated position, effectively solving the problems of geometric misalignment between flight strips, index misalignment between camera pose deviation and neural features, and feature drift caused by the separation of pose correction and geometric representation processing in existing technologies during multiple UAV sorties and multi-time period acquisition tasks. This achieves consistent updates of geometric pose and appearance description on the same deformable carrier, allowing pose errors and initial geometric deviations in UAV aerial photography data to be absorbed under a unified optimization framework, supporting the consistent maintenance of surface pose and appearance features in large-scale target areas such as urban building clusters and continuous linear infrastructure.

[0027] 3. By employing a method that uses a spatial index structure to retrieve the nearest neighbor surface elements of the query point, and combines spatial distance and surface normal orientation deviation to construct aggregate weights, weighted sums of the bound neural descriptors, and then concatenated with the relative position vector before regressing the symbolic distance value via a neural network, surface elements that are closer and have more consistent normal orientation contribute more to the symbolic distance value of the query point. The symbolic distance field regression is determined only by the local neighborhood geometry, effectively solving the problems of crosstalk between opposite curved surfaces on both sides of thin-walled structures, abnormal thickness of the zero isosurface, and adhesion of adjacent opposite-sided components in the reconstruction results under the global symbolic distance field expression method in the prior art. This enables the distinguishable expression of complex local structures such as thin-walled walls, dense railings, and attachments close to the building facade in urban scenes, and makes the zero isosurface of the symbolic distance field converge stably at the position constrained by the surface element normal, supporting the requirement of UAV oblique photogrammetry to restore the geometric boundaries of dense small components. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a method for large-scale fine surface reconstruction of UAVs that integrates implicit neural representation and surface elements, as described in this application.

[0029] Figure 2 This is another flowchart illustrating the method for large-scale fine surface reconstruction of UAVs that integrates implicit neural representation and surface elements in the embodiments of this application;

[0030] Figure 3 This is a schematic diagram of the physical device structure of a large-scale fine surface reconstruction system for UAVs in the embodiments of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] To facilitate understanding, the application scenarios of the embodiments of this application are described below.

[0034] The embodiments of this application can be applied to, but are not limited to, large-scale 3D reconstruction scenarios based on aerial imagery. Specifically, in scenarios such as city-level real-scene 3D modeling, digital twin city construction, smart city visualization, geographic information system data updates, land surveying, natural resource investigation, real estate registration, digital protection of cultural heritage, emergency rescue situation assessment, power line inspection, mine surveying, forestry resource monitoring, agricultural precision management, transportation infrastructure inspection, and digital archiving of scenic spots, it is often necessary to use cameras carried by low-altitude UAVs or manned aircraft to perform oblique or orthophotos of the ground area to obtain aerial images, and then use the aerial images to invert and obtain a high-fidelity 3D model corresponding to the ground area. The 3D model is used to support subsequent visualization browsing, spatial measurement, planning and design, change detection, and business system integration applications.

[0035] In the aforementioned application scenarios, the process of performing 3D reconstruction from aerial images typically faces the following technical challenges: First, the initial camera pose values ​​calculated by the motion reconstruction structure are subject to non-negligible pose deviations due to factors such as image matching errors, weak texture regions, repetitive texture regions, dynamic target interference, and sensor noise. These pose deviations are transmitted and accumulated through the geometric reconstruction process, leading to geometric defects such as misalignment, ghosting, breakage, or delamination on the reconstructed surface. Second, traditional reconstruction schemes based on multi-view stereo matching and explicit meshes often exhibit holes and noise in weak texture, reflection, shadow, and occlusion areas, making it difficult to balance geometric integrity and appearance fidelity in large scenes. Third, existing neural implicit reconstruction schemes mostly perform dense sampling of the radiation field using pure volume rendering, resulting in huge computational overhead and difficulty in adapting to the sparse viewpoints and wide coverage of large aerial scenes. Furthermore, they are sensitive to camera pose errors and lack an effective mechanism for jointly correcting pose parameters and geometric appearance attributes.

[0036] To address the aforementioned technical problems, this application provides a 3D reconstruction method based on aerial imagery. The method involves performing feature extraction and multi-view matching on the aerial imagery to obtain initial camera pose values ​​and sparse point clouds. Based on the sparse point clouds, surface elements are deployed within the target area, and a deformation map is constructed according to the spatial distribution of these surface elements. A learnable neural descriptor is bound to each surface element, and learnable pose parameters are configured for each deformation map node. Deformation is driven by the position and normal of the surface elements according to the pose parameters of the deformation map nodes. The neural descriptors are then aggregated to the query point based on the spatial relationship between the deformed surface elements and the query point, and the result is obtained through neural network regression. The symbolic distance value to the query point is used to construct a symbolic distance field by traversing the query point within the target region. Spherical tracing is performed along the ray on the symbolic distance field to obtain surface intersections. The pixel colors at the surface intersections are obtained by aggregating the surface elements in the neighborhood of the surface intersections and backfilling them to form a predicted image. The reconstruction loss is constructed based on the color difference between the predicted image and the aerial image. Joint gradient descent is performed on the pose parameters of the deformable map nodes and the neural descriptors of the surface elements to update them, resulting in corrected pose parameters and target neural descriptors. Finally, implicit surface extraction and texture generation are performed on the target region based on the corrected pose parameters and target neural descriptors to obtain the 3D reconstruction result corresponding to the target region.

[0037] To facilitate understanding, the method provided in this implementation will be described in detail below, using the above scenario as an example. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a method for large-scale fine surface reconstruction of UAVs that integrates implicit neural representation and surface elements, as described in this application.

[0038] S101. Acquire aerial images of the target area taken by the UAV from multiple perspectives, restore the initial surface geometry based on the aerial images and discretize it into a set of surface elements.

[0039] Among them, "UAV" refers to an unmanned aerial vehicle with aerial photography capabilities, used to collect images by flying along a preset route over the target area; "target area" refers to the geographic space covered by this reconstruction mission, including urban blocks, industrial parks and mines, and continuous linear infrastructure corridors; "multiple views" refers to the set of viewpoints corresponding to multiple shots of the same surface of the target area from different positions and orientations; "aerial image" refers to two-dimensional raster images captured and stored by the imaging sensor carried by the UAV during flight, with each aerial image carrying a corresponding exposure timestamp and initial camera pose; "initial surface geometry" refers to an approximate description of the surface position of the target area obtained by using motion reconstruction structure and multi-view stereo matching process based on multi-view aerial images, which can be carried by sparse feature point cloud, dense matching point cloud, or coarse triangular mesh; "surface element" refers to the local surface unit obtained after discretizing the initial surface geometry, with each surface element represented by a spatial point and accompanied by an orientation vector; "spatial position" refers to the three-dimensional coordinates of the representative point of the surface element in the world coordinate system; "surface normal" refers to the unit vector perpendicular to the local tangent plane of the initial surface geometry at the representative point of the surface element.

[0040] Specifically, an UAV aerial photography mission is performed on the target area according to a preset flight path, resulting in a multi-view aerial image set containing several flight strips. The initial camera pose and interior orientation elements at the time of capture are recorded for each aerial image. The aerial image set is then input into a motion reconstruction structure (MCR) workflow to extract and match feature points, calculating the relative pose and sparse 3D points between each aerial image. Under the constraint of sparse 3D points, multi-view stereo matching is performed to obtain a dense 3D point cloud as the initial surface geometry. Noise reduction and outlier removal are performed on the dense 3D point cloud, and the point cloud is homogenized and resampled according to a preset density threshold to obtain a set of candidate points to be discretized. Each candidate point is then searched. The set of spatial nearest points is calculated using principal component analysis. The eigenvector corresponding to the smallest eigenvalue of the covariance matrix is ​​used as the surface normal of the candidate point. The corresponding normal direction is adjusted outward according to the camera's optical center orientation. The spatial coordinates of the candidate point are used as the spatial position of the surface element, and the outward-adjusted normal vector is used as the surface normal of the surface element. The surface element set P = {p_i = (x_i, n_i) | i = 1, 2, ..., N} is obtained by registering the surface elements according to a unified numbering system. Here, x_i represents the spatial position of the i-th surface element, n_i represents the surface normal of the i-th surface element, and N represents the total number of surface elements.

[0041] In some embodiments, the initial surface geometry restoration and surface element discretization can be achieved in several ways: Optionally, a motion reconstruction structure is first performed on multi-view aerial images to solve the camera pose. Then, a depth map is generated based on the solved camera pose using a multi-view stereo matching method. Each depth map is back-projected according to the camera pose to obtain the corresponding local point cloud. The local point clouds are then fused and deduplicated. The fused point cloud is then downsampled according to a preset density, and the surface normal of each sampled point is estimated using principal component analysis to obtain a set of surface elements. Optionally, a sparse point cloud and camera pose are first obtained using a motion reconstruction structure. Then, an initial triangular mesh is constructed using Poisson reconstruction or signed distance field truncation fusion. Points are taken on the mesh surface using equal-area resampling. The location of the taken points is used as the spatial location of the surface elements, and the face normal of the triangular facet containing the taken points is used as the surface normal of the surface elements to obtain a set of surface elements. It is understood that the initial surface geometry restoration and surface element discretization can also be achieved by directly regressing depth using a learning-based multi-view stereo network; this is not limited here.

[0042] S102. Bind learnable neural descriptors to surface elements, arrange deformation graph nodes in the target area, establish the association relationship between surface elements and adjacent deformation graph nodes, so that the spatial position and surface normal of the surface elements are updated synchronously with the pose parameters of the associated deformation graph nodes.

[0043] Here, a neural descriptor is a low-dimensional feature vector bound to a single surface element, whose parameters can be updated during training, used to carry the geometric and appearance information of the neighborhood of that surface element; a deformable graph node is a control unit arranged sparsely in the target region, with each node having local rigidity transformation parameters; an association relationship refers to the subordinate correspondence formed between a surface element and several neighboring deformable graph nodes; pose parameters refer to the rotational and translational components describing the local rigidity transformation of the deformable graph node, which can be represented by an Euclidean transformation matrix or a Lie algebra vector; synchronous update means that after the pose parameters of the deformable graph node change, the spatial position and surface normal of the associated surface element change together in a skin-weighted manner; skin weight is a normalized scalar of the degree to which the surface element is affected by the pose parameters of each associated deformable graph node.

[0044] Specifically, a learnable neural descriptor f_i∈R^d of dimension d is assigned to each surface element p_i, and f_i, along with its spatial position x_i and surface normal n_i, are registered as binding attributes of that surface element. The spatial positions of the surface element set are uniformly downsampled according to a preset voxel size v_g, and the mean of the spatial positions of surface elements within each non-empty voxel is used as the initial position g_k of the deformable graph node, resulting in a deformable graph node set G={g_k|k=1,2,...,M}. For each deformable graph node, pose parameters {R_k, t_k} containing the rotation matrix R_k and translation vector t_k are initialized, initially R_k is the identity matrix I, and t_k is the zero vector. For each surface element p_i, the K nearest deformable graph nodes are retrieved to form an associated deformable graph node set N( i), calculate the skin weight w_{i,k} according to the formula w_{i,k}=exp(-||x_i-g_k||^2 / sigma^2) / sum_{jinN(i)}exp(-||x_i-g_j||^2 / sigma^2), where sigma represents the distance attenuation bandwidth; perform a weighted composite update on the spatial position of the surface element according to the formula x_i'=sum_{kinN(i)}w_{i,k}(R_k(x_i-g_k)+g_k+t_k), and perform a weighted composite update on the surface normal of the surface element according to the formula n_i'=normalize(sum_{kinN(i)}w_{i,k}R_kn_i), while keeping the index relationship between the neural descriptor f_i and the surface element number unchanged.

[0045] In some embodiments, the binding and pose-driven operation of surface elements and deformation map nodes can be achieved in several ways: Optionally, the spatial positions of surface elements are hierarchically divided according to an octree structure, and the center of the leaf node reaching the target voxel size is taken as the initial position of the deformation map node. For each surface element, the associated deformation map nodes are determined by octree neighborhood query. Skin weights are constructed according to the exponential distance kernel, and the pose parameters of the associated nodes are weighted and compounded and applied to the surface element. Optionally, the spatial positions of surface elements are clustered according to K-means clustering, and each cluster center is taken as the initial position of the deformation map node. For each surface element, the nearest cluster centers are retrieved according to Euclidean distance as associated deformation map nodes. Skin weights are constructed according to the inverse distance weighting method and the pose parameters of the associated nodes are fused in a dual quaternion mixing method before being applied to the surface element. It is understood that the binding and pose-driven operation of surface elements and deformation map nodes can also be achieved by obtaining the node position based on uniform grid sampling, which is not limited here.

[0046] In some embodiments, this step specifically includes:

[0047] S1021. Assign a learnable neural descriptor to each surface element, and register the neural descriptor together with the spatial position and surface normal of the surface element as the binding attribute of the surface element.

[0048] Within the target area, the spatial positions of surface elements are downsampled according to a preset voxel size. The spatial positions retained after downsampling are used as the initial positions of the deformation map nodes, and pose parameters for characterizing local rigidity transformation are initialized for each deformation map node.

[0049] S1022. Within the target area, retrieve the nearest preset number of deformation map nodes for each surface element as associated deformation map nodes of the surface element, and construct skin weights based on the spatial distance between the surface element and the associated deformation map nodes.

[0050] S1023. The pose parameters of the associated deformation graph nodes are weighted and composited based on the skin weight, and the weighted composite pose parameters are applied to the spatial position and surface normal of the surface element, so that the spatial position and surface normal of the surface element are updated synchronously with the pose parameters of the associated deformation graph nodes, and the neural descriptor bound to the surface element keeps the index consistent with the updated spatial position.

[0051] S103. Retrieve surface elements in the neighborhood of the query point within the target area, aggregate the neural descriptors bound to the retrieved surface elements and input them into the neural network, and regress to obtain the symbolic distance value of the query point.

[0052] Here, the query point refers to a 3D spatial sampling point within the target area from which the symbolic distance value is to be calculated, which may come from a voxel grid, ray step position, or random sampling point; the neighborhood refers to a local spatial region centered on the query point and defined by a preset search radius or a preset number of nearest neighbors; the spatial index structure refers to a hierarchical data structure established for the spatial positions of surface elements to accelerate nearest neighbor retrieval, including KD trees, ball trees, and voxel hashes; aggregation refers to the computational process of binding the neural descriptions of multiple surface elements within the neighborhood into a single vector according to preset weights; the neural network refers to a multilayer perceptron composed of several fully connected layers and nonlinear activation layers connected in series, used to map aggregated features into scalar outputs; regression refers to the computational process of the neural network predicting the target quantity in a continuous value manner; the symbolic distance value refers to a directed scalar of the nearest distance from a spatial point to the surface of the target area, with positive values ​​on the outside of the surface, negative values ​​on the inside of the surface, and zero values ​​on the surface; the symbolic distance field refers to the scalar field formed by the continuous change of symbolic distance values ​​with spatial coordinates within the target area.

[0053] Specifically, a spatial index structure T is established for the surface element set P based on its spatial location. The index structure T can be carried in the form of a KD tree, a ball tree, or a voxel hash. For a query point q∈R^3, the K_q nearest surface elements to q are retrieved according to the index structure T to form a neighborhood surface element set M(q). For each neighborhood surface element p_j∈M(q), the spatial distance is calculated according to the formula d_j=||q-x_j||, and the orientation deviation measure of the surface normal relative to the query point is calculated according to the formula c_j=(q-x_j)·n_j / ||q-x_j||. The aggregation weight of the neighborhood surface elements to the query point is calculated according to the formula a_j=exp(-d_j^2 / sigma_d^2)exp(-(1-c_j) / sigma_c^2). Normalization is performed according to the formula a_j_norm=a_j / sum_{linM(q)}a_l, where sigma_d and sigma_c represent the distance bandwidth and orientation bandwidth, respectively. The aggregate feature F(q) of the query point is obtained according to the formula F(q)=sum_{jinM(q)}a_j_normf_j. For each neighborhood surface element, a relative position vector delta_j=q-x_j is constructed. The aggregate feature F(q) and each relative position vector are concatenated according to the channel dimension to obtain the input tensor h(q). h(q) is then input into the neural network Phi_sdf regression to obtain the symbolic distance value s(q)=Phi_sdf(h(q)). The symbolic distance field S is obtained by traversing the query point in the target region.

[0054] In some embodiments, neighborhood aggregation and symbolic distance regression can be implemented in several ways: Optionally, a KD-tree is constructed as a spatial index structure for the spatial locations of surface elements. The nearest surface elements to the query point are retrieved using a K-nearest neighbor approach. Aggregation weights are calculated using a Gaussian kernel combining distance and orientation, and the bound neural descriptors are weighted and summed. The aggregated features are concatenated with the relative position vectors and input into a symbolic distance regression network composed of several fully connected layers to obtain the symbolic distance value. Optionally, the spatial locations of surface elements are mapped to a voxel grid of a preset resolution using a voxel hashing method. Candidate neighborhoods are formed by the voxel where the query point is located and the surface elements within its neighboring voxels. The neural descriptors bound to the surface elements within the candidate neighborhoods are aggregated using an inverse distance weighting method. The aggregated features are concatenated with the relative position vectors from the query point to each candidate neighborhood surface element and input into a symbolic distance regression network to obtain the symbolic distance value. It is understood that neighborhood aggregation and symbolic distance regression can also be implemented using a GPU-accelerated nearest neighbor search based on sparse point clouds; this is not limited here.

[0055] In some embodiments, this step specifically includes:

[0056] S1031. Establish a spatial index structure based on the spatial location of surface elements within the target area, and retrieve a preset number of surface elements closest to the query point as neighboring surface elements based on the spatial index structure.

[0057] Here, the target region refers to the spatial extent covered by aerial imagery and used for 3D reconstruction; surface element refers to the local surface unit obtained from previous steps and bound with attributes of position, surface normal, and neural descriptor; spatial position refers to the 3D coordinate vector of the surface element in the world coordinate system; spatial index structure refers to the hierarchical data organization structure that supports nearest neighbor retrieval based on spatial position; query point refers to the 3D spatial point within the target region whose symbol distance value is to be calculated; preset number of surface elements refers to the subset of surface elements with the smallest spatial distance to the query point obtained by retrieval based on a preset number of nearest neighbors K_n; and neighborhood surface elements refers to the set of surface elements surrounding the query point obtained by retrieval using the spatial index structure.

[0058] Specifically, for the extended surface element set P_ext={p_j=(x_j, n_j, f_j)|j=1, 2, ..., J}, a KD-tree spatial index structure T_kd is constructed according to the surface element position x_j, where n_j represents the surface normal and f_j represents the neural descriptor; for each query point q, the K-nearest neighbor search operation KNN(q, K_n, T_kd) is performed according to the spatial index structure T_kd to obtain the K_n surface element index set N(q)={j_1, j_2, ..., J} with the smallest spatial distance to the query point q. ..., j_{K_n}} and arranged from nearest to farthest according to the formula ||q-x_{j_1}||≤||q-x_{j_2}||≤...≤||q-x_{j_{K_n}}||; the corresponding position, surface normal and neural descriptor attributes are read from the extended surface element set P_ext according to the index set N(q) to obtain the neighborhood surface element set {p_{j_i}=(x_{j_i},n_{j_i},f_{j_i})|i=1,2,...,K_n} corresponding to the query point q.

[0059] S1032. Based on the spatial distance between the neighboring surface element and the query point and the orientation deviation of the surface normal of the neighboring surface element relative to the query point, calculate the aggregation weight of the neighboring surface element to the query point. The aggregation weight increases as the spatial distance and orientation deviation decrease.

[0060] Among them, spatial distance refers to the Euclidean distance between the position of the neighboring surface element and the query point; orientation deviation refers to the degree of deviation between the surface normal of the neighboring surface element and the direction vector pointing from the position of the neighboring surface element to the query point; aggregation weight refers to the scalar value assigned to the neighboring surface element and participating in the subsequent feature weighted summation; preset mapping relationship refers to the composite monotonic function relationship in which the smaller the spatial distance and the smaller the orientation deviation, the larger the aggregation weight.

[0061] Specifically, for each query point q and each neighboring surface element p_{j_i} in its neighboring surface element set {p_{j_i}|i=1,2,...,K_n}, the spatial distance is calculated according to the formula d_{q,j_i}=||q-x_{j_i}||; the spatial distance weight is calculated according to the formula w_{q,j_i}^{dist}=exp(-d_{q,j_i}^2 / (2sigma_d^2)), where sigma_d represents the preset distance attenuation bandwidth; the unit direction vector from the position of the neighboring surface element to the query point is calculated according to the formula e_{q,j_i}=(q-x_{j_i}) / ||q-x_{j_i}||; and the surface normal of the neighboring surface element is calculated according to the formula cos_alpha_{q,j_i}=n_{j_i}·e_{q,j_i}. The cosine value between the unit direction vectors of the neighboring surface element positions pointing to the query point is used; the cosine value is truncated on one side according to the formula c_tilde_{q,j_i}=max(cos_alpha_{q,j_i},0), so that the neighboring surface elements whose surface normals are away from the query point are given zero orientation correlation; the orientation deviation weight is calculated according to the formula w_{q,j_i}^orient=exp(-(1-c_tilde_{q,j_i})² / (2×sigma_o²)), so that the orientation deviation weight of the neighboring surface element whose surface normals are on the same side as the unit direction vector and whose included angle is smaller is larger, so that the orientation deviation weight of the neighboring surface element whose surface normals are away from the query point converges to a preset small amount below exp(-1 / (2×sigma_o²)). The single-sided truncation process ensures that the surface elements corresponding to the opposite curved surfaces on both sides of the thin-walled structure contribute symbolic distance values ​​to the query points located on the same side of their normal direction only. This avoids the surface elements on both sides of the thin-walled structure from being aggregated due to reverse normal matching during the symbolic distance field regression process, thereby avoiding the problem of abnormal zero iso-surface thickness or adhesion of adjacent opposite-side components in the reconstruction results of the thin-walled structure.

[0062] S1033. Based on the aggregation weight, the neural descriptors bound to the neighboring surface elements are weighted and summed to obtain the aggregated features corresponding to the query point.

[0063] Here, aggregation weight refers to the normalized aggregation weight obtained by S1032; neural descriptor refers to a learnable high-dimensional feature vector f_{j_i}∈R^{D_f} that is bound to surface elements and used to encode local geometric and appearance information, where D_f represents the dimension of the neural descriptor; weighted summation refers to the process of performing a linear combination operation on the neural descriptors of the neighboring surface elements according to the aggregation weight; and aggregation feature refers to a high-dimensional feature vector obtained by weighted summation operation that comprehensively represents the geometric information around the query point.

[0064] Specifically, for each query point q, the neural descriptors of the neighborhood surface elements are weighted and summed according to the formula F_q=sum_{i=1}^{K_n}w_{q,j_i}norm*f{j_i} to obtain the aggregate feature F_q∈R^{D_f} corresponding to the query point q.

[0065] S1034. The aggregated features and the relative position vectors of the query point with respect to the spatial position of the neighboring surface elements are concatenated and input into the neural network to obtain the symbolic distance value of the query point through regression. This symbolic distance value constitutes the symbolic distance field as the query point traverses the target region.

[0066] Among them, the aggregated feature refers to the high-dimensional feature vector F_q corresponding to the query point obtained by S1033; the relative position vector refers to the three-dimensional vector obtained by subtracting the position of the neighboring surface element from the position of the query point and representing the geometric offset of the query point relative to the neighboring surface element; concatenation refers to the operation process of connecting the aggregated feature and the relative position vector end to end in a preset dimensional order to obtain the joint feature vector; the neural network refers to the regression neural network Phi_sdf composed of a multi-layer fully connected structure, with the joint feature vector as input and the scalar signed distance value as output; the signed distance value refers to the signed shortest distance from the query point to the nearest surface, with positive values ​​on the outside of the surface and negative values ​​on the inside of the surface; the signed distance field refers to the continuous scalar field composed of the signed distance values ​​obtained by traversing the target region by the query point.

[0067] Specifically, for each query point q and its neighborhood surface element set {p_{j_i}|i=1,2,...,K_n}, the relative position vector is calculated according to the formula delta_{q,j_i}=q-x_{j_i}; the relative position vector is concatenated according to the nearest neighbor order according to the formula delta_q=concat(delta_{q,j_1}, delta_{q,j_2},..., delta_{q,j_{K_n}}) to obtain the merged relative position vector delta_q∈R^{3K_n}; the aggregation feature F_q is combined with the merged relative position vector according to the formula z_q=concat(F_q,delta_q). The position vector delta_q is concatenated to obtain the joint feature vector z_q∈R^{D_f+3K_n}; the joint feature vector z_q is input into the symbolic distance regression neural network Phi_sdf and the symbolic distance value s_q of the query point q is calculated according to the formula s_q=Phi_sdf(z_q;Theta_sdf), where Theta_sdf represents the set of learnable parameters of the neural network; the symbolic distance value set {s_q|q∈target region} is obtained by performing a preset traversal operation on the query point q in the target region and arranging it according to the query point position to form the symbolic distance field SDF(q)=Phi_sdf(z_q;Theta_sdf).

[0068] S104. Based on the camera pose corresponding to the aerial image, a predicted image is generated from the symbolic distance field through differentiable rendering. The gradient is backpropagated based on the reconstruction loss between the predicted image and the aerial image to update the pose parameters and neural descriptors of the deformable graph nodes, resulting in corrected pose parameters and a target neural descriptor. The camera optical center is determined based on the camera pose corresponding to the aerial image, and rays are projected from the camera optical center along the pixel directions of the aerial image to the target region.

[0069] Among them, camera pose refers to the position and orientation parameters of the imaging sensor carried by the UAV in the world coordinate system during aerial image capture, including 3D translation vectors and 3D rotation components; differentiable rendering refers to the imaging process in which each computational step in the rendering process maintains differentiability of parameters, used to construct a loss function that can be optimized by gradient descent at the image level; predicted image refers to a 2D raster image with the same resolution as the aerial image generated by the differentiable rendering process based on the current symbolic distance field and neural descriptor; reconstruction loss refers to a scalar function that measures the degree of deviation between the predicted image and the aerial image in pixel color; gradient refers to the partial derivative of the reconstruction loss with respect to the parameters to be optimized; corrected pose parameters refer to the pose parameters of the deformable graph nodes after iterative convergence; target neural descriptor refers to the neural descriptor bound to the surface elements after iterative convergence; camera optical center refers to the 3D position of the camera projection center in the world coordinate system; pixel direction refers to the line-of-sight vector of the pixel center relative to the camera optical center, determined by the camera's internal orientation elements; ray refers to a semi-infinite straight line that starts from the camera optical center and extends along the pixel direction; spherical tracing refers to a ray-based traversal method that uses the distance value of each point in the symbolic distance field to determine the step length of the next movement.

[0070] Specifically, for each aerial image I, the corresponding initial camera pose {R_cam, t_cam} is obtained and the camera optical center position o = -R_cam^Tt_cam is calculated; for each pixel p, the line-of-sight direction is calculated according to the camera intrinsic parameter matrix K^{-1} and rotated to the world coordinate system via R_cam^T to obtain the pixel direction unit vector v_p, and a ray r_p(t) = o + tv_p is constructed, t ≥ 0; spherical tracing is performed along the ray on the symbolic distance field, initializing t_0 = 0, and iteratively following the formula t_{m+1} = t_m + The process proceeds by moving s(r_p(t_m)). The iteration terminates when |s(r_p(t_m))| is less than the preset convergence threshold eps_st or the number of iterations reaches the upper limit M_st. The final position r_p(t_m) is taken as the surface intersection point x_p^ corresponding to pixel p. The surface elements within the neighborhood of x_p^ are obtained according to the neighborhood retrieval method described in S103, and the bound neural descriptors are weighted and summed according to the aggregation weight to obtain the aggregated appearance feature F_c(x_p^). The surface normal n(x_p^) at the intersection point is then compared with the surface normal n(x_p^). The concatenation of x_p^) and the viewing direction v_p is input into the color decoding network Phi_rgb. Regression yields the pixel color c_p = Phi_rgb(F_c(x_p^), n(x_p^), v_p) at the surface intersection. c_p is then backfilled into the prediction image according to the pixel direction to obtain the prediction image I_pred. The reconstruction loss L is constructed according to the formula L = sum_p||I(p) - I_pred(p)||_1. Following the chain rule, L is applied to the pose parameters {R_k, t_k} of the deformed image nodes and the table... The partial derivatives of the neural descriptor f_i bound to the face elements are calculated, and iterative updates are performed according to the gradient descent formula {R_k, t_k}←{R_k, t_k}-eta_gdL / d{R_k, t_k} and f_i←f_i-eta_fdL / df_i, where eta_g and eta_f represent the learning rate of the pose parameters and the learning rate of the neural descriptor, respectively. The iteration continues until the difference between two adjacent values ​​of L is less than the preset convergence threshold eps_L, thus obtaining the corrected pose parameters {R_k^, t_k^} and the target neural descriptor f_i^.

[0071] In some embodiments, differentiable rendering and parameter updating can be implemented in multiple ways: optionally, only one ray is sampled for each pixel, a surface intersection point is obtained through sphere tracing, neural descriptors bound to neighborhood surface elements at the surface intersection point are aggregated and then passed through a color decoding network to obtain the pixel color, a reconstruction loss is constructed according to the L1 norm, and the Adam optimizer is used to perform synchronous gradient updating on the pose parameters of the deformation graph nodes and the neural descriptors; optionally, several sample points are sampled along the ray for each pixel near the estimated surface intersection point according to a Gaussian distribution, the volume density of the sample points is calculated according to the conversion relationship from the signed distance value to the volume density, and the pixel color is obtained through volume rendering integration, a reconstruction loss is constructed in the combined form of the L1 norm and the structural similarity index, gradient updating is performed on the pose parameters of the deformation graph nodes using Lie algebra representation and right perturbation, and gradient updating is performed on the neural descriptors using the Adam optimizer. It can be understood that differentiable rendering and parameter updating can also be implemented by a coarse-to-fine phased optimization manner according to an image pyramid, which is not limited herein.

[0072] In the above sphere tracing process, the surface intersection point x_p* solved from the iteration termination condition |s(r_p(t_m))|<eps_st is obtained through a discrete iteration process, and the step size selection and termination judgment in the discrete iteration process do not directly participate in gradient backpropagation. To enable the geometry and appearance parameters at the surface intersection point to receive gradients along the direction of the photometric loss, the derivative of the surface intersection point x_p* with respect to the pose parameters {R_k, t_k} of the deformation graph nodes and the neural descriptor f_i is solved in the following manner according to the implicit function theorem: the sphere tracing iteration trajectory is frozen at the convergence position, the surface intersection point is regarded as an implicit solution of the signed distance field constraint Phi_sdf(x_p*; theta)=0, according to the formula

[0073] ∂x_p* / ∂theta=-[1 / (grad_Phi_sdf(x_p*)·v_p)]×v_p×∂Phi_sdf(x_p*; theta) / ∂theta

[0074] The partial derivative of the surface intersection point with respect to the network parameters theta={R_k, t_k, f_i} is solved, where v_p denotes the unit vector in the pixel direction, grad_Phi_sdf(x_p*) denotes the spatial gradient of the signed distance field at the surface intersection point, and grad_Phi_sdf(x_p*)·v_p takes a non-zero value when the normal direction at the surface intersection point is not orthogonal to the line-of-sight direction, thereby ensuring that there exists a definite partial derivative solution of the surface intersection point with respect to the pose parameters of the deformation graph nodes and the neural descriptors.

[0075] For rays that do not hit the zero isosurface, according to the preset occupancy probability density function

[0076] sigma(s)=alpha×sigmoid(-s / beta)×sigmoid'(-s / beta)

[0077] The symbolic distance value s is converted into volume density and rendered integrally according to volume rendering.

[0078] c_hat_p=∫t n ear, t f ar,×sigma(r_p(t))×c(r_p(t))dt

[0079] Where T(t) = exp(-∫t n `ear`, `tsigma(r_p(tau))`, and `dtau` represent the transmittance, and `alpha` and `beta` are preset occupancy-density transformation parameters. The pixel color is obtained by integrating along the ray using uniform sampling, so that background pixels and occluded areas also participate in the reconstruction loss gradient backpropagation. Either the spherical tracing path or the volume rendering integration path can be chosen as the implementation method for the differentiable rendering. Alternatively, the spherical tracing can be used for accelerated hit pixels, and the volume rendering integration can be used for supervision of missed pixels, performing a hybrid scheduling of the two.

[0080] In some embodiments, this step specifically includes:

[0081] S1041. Perform spherical tracing along the ray to trace the symbolic distance field, and take the intersection of the ray and the zero isosurface of the symbolic distance field as the surface intersection point corresponding to the pixel direction.

[0082] Here, a ray is a three-dimensional parametric straight line that originates from the camera's optical center and is projected into the scene space through the pixel center; the pixel direction is a unit vector determined by the pixel coordinates within the camera and indicating the direction of ray propagation; the symbolic distance field is a continuous scalar field SDF(q) obtained from S1034 and whose symbolic distance values ​​are calculated using the query point as input; spherical tracing is an iterative intersection method that advances along the ray direction with the symbolic distance value at the current ray point as a safe step size until the symbolic distance value converges to zero; the zero isosurface is the object surface corresponding to the implicit surface with a symbolic distance field of zero; and the surface intersection point is a three-dimensional spatial point where the ray intersects the zero isosurface.

[0083] Specifically, the camera optical center position is calculated for the intrinsic parameter matrix K_k and the extrinsic parameter matrix [R_k|t_k] of the camera corresponding to the current aerial image I_k according to the formula o_k=-R_k^Tt_k; for each pixel coordinate (u, v), the unit vector in the pixel direction is calculated according to the formula d_{u,v}=R_k^TK_k^{-1}[u,v,1]^T / ||K_k^{-1}[u,v,1]^T||; a ray starting from the camera optical center and propagating along the pixel direction is constructed according to the formula r_{u,v}(t)=o_k+td_{u,v}, where t represents the signed distance parameter along the ray; t_0=t_near is initialized, and t_near represents the preset near clipping plane distance; for each iteration step m, the current ray point position is solved according to the formula q_m=r_{u,v}(t_m), the signed distance value of the current ray point is read according to the formula s_m=SDF(q_m), and spherical tracing advancement is performed on the distance parameter according to the formula t_{m+1}=t_m+s_m; convergence is determined when |s_m|<epsilon_s, and the current ray point is taken as the surface intersection point according to the formula q_{u,v}^*=r_{u,v}(t_m), where epsilon_s represents the preset zero isosurface convergence threshold; a miss is determined when t_m>t_far or the number of iteration steps exceeds the preset maximum iteration number N_max, and the pixel direction is marked as background, where t_far represents the preset far clipping plane distance; spherical tracing operation is performed for all pixel directions in a traversal manner to obtain a surface intersection point set {q_{u,v}^*|(u,v)∈hit pixel set}.

[0084] S1042. Retrieve surface elements in the neighborhood of the surface intersection point, aggregate the neural descriptors bound to the retrieved surface elements, input the aggregation result into a color decoding network for regression to obtain the pixel color at the surface intersection point, and backfill the pixel color according to the pixel direction to form a predicted image.

[0085] The surface intersection point refers to the intersection point q_{u,v}^* between the ray obtained in S1041 and the zero isosurface; the neighborhood refers to the distribution range of neighboring surface elements retrieved by a spatial index structure with the surface intersection point as the center and according to the preset neighbor number K_c; the surface element refers to a local surface unit with attributes of position, surface normal and neural descriptor; the neural descriptor refers to a learnable high-dimensional feature vector bound to the surface element and used for encoding local appearance information; the aggregation refers to an operation process of performing weighted summation on the neural descriptors of the neighboring surface elements according to the distance weight and the orientation weight; the color decoding network refers to a regression-type neural network Phi_color composed of a multi-layer fully connected structure, which takes the aggregated feature and the line of sight direction as inputs and outputs RGB color; the pixel color refers to a three-dimensional color vector obtained by regression at the surface intersection point via the color decoding network; the predicted image refers to a synthesized two-dimensional image obtained by backfilling the pixel colors of all hit pixels according to the pixel coordinates.

[0086] Specifically, for each surface intersection point q_{u, v}^, the K-nearest neighbor retrieval operation KNN(q_{u, v}^, K_c, T_kd) is performed according to the KD tree spatial index structure T_kd of S1031 to obtain the nearest neighbor surface element index set N_c(q_{u, v}^)={j_1, j_2, ..., j_{K_c}}; for each nearest neighbor surface element p_{j_i}=(x_{j_i}, n_{j_i}, f_{j_i}) the spatial distance is calculated according to the formula d_i=||q_{u, v}^-x_{j_i}|| and the unit direction vector is calculated according to the formula e_i=(q_{u, v}^-x_{j_i}) / d_i; according to the formula w_ i^{dist}=exp(-d_i^2 / (2sigma_d^2)) and c_tilde_i=max(n_{j_i}·e_i,0); w_i^orient=exp(-(1-c_tilde_i)² / (2×sigma_o²)); calculate distance weight and orientation weight respectively; obtain normalized aggregate weights according to the formulas w_i=w_i^{dist}w_i^{orient} and w_i_norm=w_i / sum_{i=1}^{K_c}w_i; and obtain normalized aggregate weights according to the formulas F_{q_{u,v}^}=sum_{i=1}^{K_c}w_i_normf_{j _i} performs a weighted summation of the neural descriptors of the nearest surface elements to obtain the aggregated feature F_{q_{u,v}^}∈R^{D_f} at the surface intersection; calculates the unit vector of the gaze direction from the surface intersection to the camera optical center according to the formula v_{u,v}=-d_{u,v}; concatenates the aggregated feature, gaze direction, and surface normal according to the formula z_{u,v}^{color}=concat(F_{q_{u,v}^}, v_{u,v}, n_{q_{u,v}^}) to obtain the joint input vector of the color decoding network, where n_{q_{u,v}^} represents the normal at the surface intersection obtained by weighted aggregation according to the nearest surface normal; the joint input vector z_ {u, v}^{color} is input into the color decoding network Phi_color, and the pixel colors c_{u, v}∈R^3 at the surface intersections are calculated according to the formula c_{u, v}=Phi_color(z_{u, v}^{color};Theta_color), where Theta_color represents the set of learnable parameters of the color decoding network. For all hit pixels, the pixel colors c_{u, v} are backfilled into the corresponding positions of the predicted image according to the pixel coordinates (u, v). For the missing pixels, the preset background color c_bg is filled in, and the predicted image I_k_pred={c_{u, v}|(u, v)∈the entire set of image pixels} is obtained.

[0087] S1043. Construct a reconstruction loss based on the difference in pixel color between the predicted image and the aerial image, and backpropagate the gradient of the reconstruction loss to the pose parameters and neural descriptors of the deformed image nodes respectively. Iterate and update the pose parameters and neural descriptors of the deformed image nodes through gradient descent until the reconstruction loss converges, and obtain the corrected pose parameters and target neural descriptors.

[0088] Here, the predicted image refers to the two-dimensional image I_k_pred synthesized from the current viewpoint obtained by S1042 through neural rendering; the aerial image refers to the two-dimensional observation image I_k acquired by the camera from the real scene and used as a supervision signal; the difference in pixel color refers to the degree of deviation between the color vectors of the predicted image and the aerial image at the same pixel position; the reconstruction loss refers to the scalar loss function L_rec constructed according to the pixel color difference and used to reflect the overall deviation between the predicted image and the aerial image; and the pose parameters of the deformable graph nodes refer to the parameters obtained from the previous steps and used to drive the position of surface elements. The node rotation matrix R_g and translation vector t_g are related to the normal transformation; the neural descriptor is a learnable high-dimensional feature vector f_j obtained from the preceding steps and bound to the surface elements; the gradient is the partial derivative of the reconstruction loss with respect to the pose parameters and the neural descriptor according to the chain rule; gradient descent is the optimization process of iteratively updating the pose parameters and the neural descriptor according to the gradient direction and a preset learning rate; the corrected pose parameters are the pose parameters of the deformed graph nodes obtained after iterative optimization convergence; the target neural descriptor is the neural descriptor obtained after iterative optimization convergence and bound to the surface elements.

[0089] Specifically, construct the L1 pixel color difference loss on the hit pixel set Omega_k for the predicted image and the aerial image according to the formula L_color=sum_{k=1}^{N_view}sum_{(u,v)∈Omega_k}||I_k_pred(u,v)-I_k(u,v)||1, where N_view represents the number of views participating in optimization; construct the Eikonal regularization term according to the formula L_eik=E_q[(||gradSDF(q)||2-1)^2] to constrain the signed distance field to satisfy the unit gradient property; construct the smooth regularization term for the pose parameters between adjacent nodes (g, g') of the deformation graph according to the formula L_reg^{pose}=sum{(g,g')∈E_g}||R_g-R{g'}||F^2+lambda_t*||t_g-t{g'}||2^2, where E_g represents the set of edges of the deformation graph, and lambda_t represents the weight of the translation term; perform weighted summation on each loss sub-item according to the formula L_rec=L_color+lambda_eikL_eik+lambda_poseL_reg^{pose} to obtain the total reconstruction loss, where lambda_eik and lambda_pose represent the weight coefficients of the Eikonal regularization term and the pose smooth regularization term respectively; back-propagate the gradient for the pose parameters of the deformation graph nodes according to the chain rule by the formula grad{R_g}L_rec=∂L_rec / ∂R_g and grad_{t_g}L_rec=∂L_rec / ∂t_g, and back-propagate the gradient for the surface element neural descriptor according to the chain rule by the formula grad_{f_j}L_rec=∂L_rec / ∂f_j; perform gradient descent update with manifold retraction on the rotation matrix on the SO(3) manifold according to the formula R_g^{(l+1)}=Retr_{SO(3)}(R_g^{(l)}-eta_Rgrad_{R_g}L_rec), where Retr_{SO(3)} represents the SO(3) manifold retraction operator, and eta_R represents the learning rate of the rotation term; perform gradient descent update on the translation vector and the neural descriptor respectively according to the formulas t_g^{(l+1)}=t_g^{(l)}-eta_tgrad_{t_g}L_rec and f_j^{(l+1)}=f_j^{(l)}-eta_fgrad_{f_j}L_rec, where eta_t and eta_f represent the learning rates of the translation term and the neural descriptor term respectively; it is determined to be convergent when the reconstruction loss difference |L_rec^{(l+1)}-L_rec^{(l)}|<epsilon_rec between adjacent iteration rounds or the number of iteration rounds exceeds the preset maximum optimization round number L_max, where epsilon_rec represents the preset loss convergence threshold;Output the deformation graph node pose parameters {(R_g^, t_g^)|g∈full set of deformation graph nodes} at the convergence moment as corrected pose parameters, and output the surface element neural descriptors {f_j^|j=1, 2, ..., J} at the convergence moment as target neural descriptors.;

[0090] During the gradient backpropagation process of the Eikonal regularization term L_eik, the spatial gradient grad_Phi_sdf(q) of the signed distance field Phi_sdf must pass through the neighborhood search operator KNN(q, K_n, T_kd) based on the spatial index structure; the neighborhood search operator returns the neighborhood surface element numbers according to the integer index set, and there exists a non-differentiable jump when deriving the query point q with respect to position. To keep the gradient backpropagation of the Eikonal regularization term stable, the processing is performed in one of the following ways:

[0091] First, after the signed distance value is obtained by forward regression, the neighborhood surface element index set N(q) corresponding to the query point is frozen, and gradient paths are reserved only for the position x_j, surface normal n_j and neural descriptor f_j of the surface elements in the neighborhood, according to the formula

[0092] grad_qPhi_sdf(q)=(∂Phi_sdf / ∂F(q))×grad_qF(q)+Σj∈N(q)j∈N(q)j∈N(q)(∂Phi_sdf / ∂delta_j)×grad_qdelta_j

[0093] The spatial gradient is solved on the premise that the neighborhood index set is regarded as a constant;

[0094] Second, the hard nearest neighbor search is replaced with soft neighborhood aggregation, according to the Gaussian kernel

[0095] w_tilde_{q,j}=exp(-||q-x_j||² / sigma_d²)

[0096] Continuous weights are applied to the surface elements in the extended candidate set, and small weight terms are truncated according to the weight threshold tau_w, so that the aggregated feature F(q) remains continuously differentiable with respect to the position of the query point q at the boundary;

[0097] Third, the Eikonal regularization term is applied only in the narrow band near the zero isosurface (i.e., |Phi_sdf(q)|<delta_band), and the action domain of the regularization term is restricted in such a manner that L_eik is zero outside the narrow band, so as to avoid disturbance to the gradient backpropagation caused by frequent jumps of the neighborhood search operator far from the surface.

[0098] Any of the three processing methods described above can enable the Eikonal regularization term to participate in the gradient update process of the pose parameters of the deformable graph nodes and the neural descriptor in a determined partial derivative form.

[0099] S105. Drive the surface elements to the correction position using the correction pose parameters, generate a dense symbolic distance field based on the surface elements at the correction position and the bound target neural descriptor, and extract a three-dimensional surface model covering the target area from the dense symbolic distance field.

[0100] Among them, the corrected position refers to the updated spatial position of the surface element after the pose parameters of the deformed graph node are updated to the corrected pose parameters, and the surface element is weighted composite by the skin; the dense symbolic distance field refers to the discrete field of symbolic distance values ​​generated in the target area at a more refined sampling interval than that in the training stage, which can be carried by a three-dimensional voxel grid; the voxel grid refers to the set of cubic units obtained by dividing the three-dimensional bounding box of the target area at equal intervals; the zero isosurface refers to the implicit surface formed by points where the symbolic distance field value is zero; the moving cube method refers to the isosurface extraction algorithm that extracts triangular facets by looking up a table based on the combination of eight vertex symbolic distance values ​​of the voxel grid; the three-dimensional surface model refers to the triangular mesh surface representation extracted from the dense symbolic distance field zero isosurface, and its vertex positions and triangular connection relationships are recorded in the mesh data structure.

[0101] Specifically, the spatial position of each surface element p_i is updated according to the formula x_i^=sum_{kinN(i)}w_{i,k}(R_k^**(x_i-g_k)+g_k+t_k^) to obtain the corrected position x_i^, and the surface normal of the surface element is updated according to the formula n_i^=normalize(sum_{kinN(i)}w_{i,k}*R_k^*n_i) to obtain the corrected normal n_i^*; the three-dimensional bounding box of the target region is uniformly divided according to the preset voxel size v_d to obtain a dense voxel grid V_d={v_q|q=1,2,...,Q}, and the center q_q of each voxel is used as the query point. The neighborhood retrieval method described in S103 is used to retrieve the corrected position p_i^. Neighboring surface elements are retrieved from the surface element set, and the target neural descriptor f_i^* is weighted and summed according to the aggregation weight to obtain the aggregated feature. The aggregated feature is concatenated with the relative position vector and input into the neural network Phi_sdf to obtain the symbolic distance value s(q_q). The dense symbolic distance field S_d is obtained by traversing all voxel centers within the voxel grid. Triangular patches are extracted from the dense symbolic distance field S_d near the zero isosurface using the moving cube method. The vertex connection order of the triangular patches is arranged according to the preset normal consistency rule to obtain the three-dimensional surface model M_surf=(V_surf, F_surf) covering the target region, where V_surf represents the set of surface vertices and F_surf represents the set of triangular patch connection relationships.

[0102] In some embodiments, dense symbolic distance field generation and 3D surface model extraction can be achieved in multiple ways: Optionally, the target region is uniformly gridded according to a preset voxel size, and the symbolic distance value of each voxel center is regressed using a neighborhood aggregation method to obtain a dense symbolic distance field. Zero-isosurface triangular patches are extracted from the dense symbolic distance field using the moving cube method, and a 3D surface model is obtained through vertex merging and manifold repair. Optionally, the target region is subdivided according to an octree adaptive structure, and a finer subdivision size is used for voxels near the zero-isosurface. The symbolic distance value of the subdivision unit center is regressed using a neighborhood aggregation method to obtain an adaptive dense symbolic distance field. Surface triangular patches are extracted from the adaptive dense symbolic distance field using a dual contour method to obtain a 3D surface model. It is understood that dense symbolic distance field generation and 3D surface model extraction can also be achieved by extracting isosurfaces using a moving least squares surface fitting method; this is not limited here.

[0103] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the method for large-scale fine surface reconstruction of UAVs that integrates implicit neural representation and surface elements in the embodiments of this application.

[0104] S201. Perform differentiable rendering of the dense symbol distance field according to the camera pose to obtain a rendered image that is pixel-by-pixel aligned with the aerial image.

[0105] Among them, camera pose refers to the position and orientation parameters of the imaging sensor in the world coordinate system at the moment of aerial image capture; dense symbolic distance field refers to the discrete field of symbolic distance values ​​generated in the target area according to the preset fine voxel size; differentiable rendering refers to the imaging process in which each operation in the rendering process maintains differentiability of parameters; pixel-by-pixel alignment refers to the one-to-one correspondence between the rendered image and the aerial image in terms of pixel row and column numbers; and the rendered image refers to the two-dimensional raster image generated according to the current dense symbolic distance field and the target neural descriptor according to the differentiable rendering process.

[0106] Specifically, for each aerial image I, the corresponding corrected camera pose {R_cam, t_cam} is obtained, and the camera optical center position o = -R_cam^Tt_cam is calculated; for each pixel p in the aerial image, the pixel orientation unit vector v_p is calculated according to the camera intrinsic parameter matrix K^{-1} and rotated to the world coordinate system via R_cam^T; a ray r_p(t) = o + tv_p is constructed, and spherical tracing is performed along the ray on the dense symbolic distance field S_d, according to the iterative formula t_{m+1} = t_m + s The iteration proceeds (r_p(t_m)) and terminates when |s(r_p(t_m))| is less than the preset convergence threshold eps_st, resulting in the surface intersection point x_p^*. The target neural descriptors bound to the surface elements in the neighborhood of x_p^* are weighted and summed according to the combination of distance and orientation to obtain the aggregated features. The aggregated features are concatenated with the surface normal and the viewing direction and then input into the color decoding network Phi_rgb. The pixel color c_p is obtained by regression and backfilled according to the pixel row and column numbers to obtain the rendered image I_render.

[0107] S202. Calculate the absolute value of the color difference between the aerial image and the rendered image at the same pixel position to obtain the residual image.

[0108] Among them, aerial imagery refers to two-dimensional raster images captured by an imaging sensor carried by a UAV during flight; rendered imagery refers to two-dimensional raster images generated by S201 according to a differentiable rendering process; color difference refers to the difference between the red, green, and blue channel values ​​at the same pixel position in aerial imagery and rendered imagery; absolute value refers to the non-negative scalar value after removing the sign from the difference; residual imagery refers to two-dimensional raster images with the same resolution as aerial imagery and using the absolute value of the color difference as the pixel value.

[0109] Specifically, for each pixel position p=(u,v) in both the aerial image I and the rendered image I_render, the absolute value of the red channel difference is calculated using the formula delta_R(p)=|I_R(u,v)-I_render_R(u,v)|, the absolute value of the green channel difference is calculated using the formula delta_G(p)=|I_G(u,v)-I_render_G(u,v)|, and the absolute value of the green channel difference is calculated using the formula delta_B(p)=|I_B(u,v)-I_render_B(u,v)|. The absolute value of the difference in the blue channel; the absolute value of the difference in the three channels is averaged according to the channel dimension using the formula e(p)=(delta_R(p)+delta_G(p)+delta_B(p)) / 3 to obtain the scalar residual e(p) at the pixel position; e(p) is arranged according to the original pixel row and column numbers to obtain the residual image I_res with the same resolution as the aerial image; the above operation is performed on each aerial image to obtain the residual image set {I_res^l|l=1,2,...,L}, where L represents the total number of aerial images.

[0110] S203. Pixels in the residual image whose residuals are greater than the preset residual threshold are back-projected onto the three-dimensional surface model along the imaging direction corresponding to the camera pose to obtain the surface region to be refined on the three-dimensional surface model.

[0111] Among them, the preset residual threshold refers to the scalar boundary value tau_e used to distinguish between high residual pixels and low residual pixels; pixels with residuals greater than the preset residual threshold refer to the set of pixel positions in the residual image where the scalar residual value exceeds tau_e; the imaging direction refers to the ray direction vector that originates from the camera optical center and passes through the center of the corresponding pixel; back projection refers to the operation process of extending from the two-dimensional pixel position through the imaging direction to the three-dimensional space and intersecting with the three-dimensional curved surface model; the curved surface region to be refined refers to the local curved surface subset on the three-dimensional curved surface model composed of the back projection hit point and its connected neighborhood.

[0112] Specifically, for each pixel position p in the residual image I_res, the pixel positions that satisfy e(p)>tau_e are assigned to the high residual pixel set H={p_h|h=1,2,...,H_n}; for each high residual pixel p_h, the imaging direction v_{p_h} is calculated according to the camera intrinsic parameter matrix K^{-1} and the camera pose {R_cam, t_cam} and the back projection ray r_{p_h}(t)=o+t*v_{p_h} is constructed; for the back projection ray... The nearest intersection point y_h between the projected ray and the 3D surface model M_surf is calculated using the ray-triangle intersection method, resulting in the back-projection hit point set Y={y_h}. For each hit point y_h, connected neighborhood vertices are searched on the 3D surface model according to the preset geodesic radius rho_g to form a local vertex subset. The union of the local vertex subsets corresponding to all hit points is taken and constrained into a manifold sub-pattern according to the vertex connection relationship, resulting in the surface region Omega_r to be refined on the 3D surface model.

[0113] S204. Add sampling points to the area of ​​the surface to be refined as new surface elements, and use the distance-weighted interpolation result of the target neural descriptor in the neighborhood as the initial neural descriptor bound to the new surface element.

[0114] Here, sampling points refer to newly generated three-dimensional spatial points within the surface area to be refined according to a preset density; newly added surface elements refer to newly added local surface units composed of sampling points and their surface normals; neighborhood refers to a local spatial region centered on the sampling point and defined by a preset search radius; distance-weighted interpolation refers to the process of performing a weighted summation operation on the corresponding values ​​using the reciprocal of the spatial distance between the neighboring surface elements and the sampling points as the weight; and initial neural descriptor refers to the feature vector initially assigned to the newly added surface elements and participating in subsequent optimization.

[0115] Specifically, within the surface region Omega_r to be refined, uniform resampling is performed on the surface at a preset sampling interval delta_s to obtain a set of new sampling points X_new={x_i^new|i=1,2,...,N_new}; for each new sampling point x_i^new, the surface normal n_i^new of the new surface element is obtained by weighted averaging of the surface normals of the triangular facet it belongs to and the surface normals of the adjacent facets; for each new sampling point x_i^new, the nearest K_n surface elements are retrieved from the original set of surface elements to form a neighborhood surface element set B(i); for each neighborhood surface element p_j∈B(i), the nearest K_n surface elements are retrieved to form a neighborhood surface element set B(i); for each neighborhood surface element p_j∈B(i), the surface normal n_i^new is obtained by weighted averaging of the surface normals of the adjacent facets; for each new sampling point x_i^new, the nearest K_n surface elements are retrieved to form a neighborhood surface element set B(i); for each neighborhood surface element p_j∈B(i), the surface normal n_i^new is obtained by weighted averaging of the surface normals of the adjacent facets and the surface normal n_i^new of the new surface element. The inverse distance weights are calculated using the formula w_{i,j}=1 / (||x_i^new-x_j^*||+eps_w), where eps_w represents a preset small amount to prevent division by zero. The weights are normalized using the formula w_{i,j}norm=w{i,j} / sum_{linB(i)}w_{i,l}. The initial neural descriptor f_i^new for the newly added surface element bindings is obtained using the formula f_i^new=sum_{jinB(i)}w_{i,j}_norm*f_j^*, and {x_i^new, n_i^new, f_i^new} are registered as the binding attributes of the newly added surface element.

[0116] S205. Within the surface region to be refined, re-search the neighborhood surface elements according to the generation method of the dense symbolic distance field, aggregate the bound neural descriptors, and regress the symbolic distance values ​​to obtain the refined symbolic distance field.

[0117] Among them, the surface region to be refined refers to the local surface subset on the three-dimensional surface model determined by S203; the neighborhood surface element refers to the set of nearest neighbor surface elements obtained by retrieving the query point according to the spatial index structure; the neural descriptor refers to the low-dimensional feature vector bound to the surface element and participating in the training, including the target neural descriptor of the original surface element and the initial neural descriptor of the newly added surface element; the regression symbolic distance value refers to the operation process of the neural network predicting the symbolic distance scalar at the query point with aggregated features and relative position vectors as input; the refined symbolic distance field refers to the discrete field of symbolic distance values ​​generated according to the fine voxel size within the spatial bounding box of the surface region to be refined.

[0118] Specifically, the bounding box B_r of the surface region Omega_r to be refined is calculated, and the bounding box B_r is uniformly divided according to the preset fine voxel size v_r to obtain the refined voxel grid V_r={v_q^r|q=1,2,...,Q_r}; the original surface elements and the newly added surface elements are merged to form an extended surface element set P_ext, and a spatial index structure T_ext is established for P_ext according to spatial position; for each refined voxel center q_q^r as the query point, the nearest K_q surface elements are retrieved according to the index structure T_ext to form a neighborhood surface element set M(q_q^r); the neighborhood surface elements are processed according to the formula... The aggregation weights are calculated and normalized using the formula a_j = exp(-||q_q^r-x_j||^2 / sigma_d^2)*exp(-(1-((q_q^r-x_j)·n_j / ||q_q^r-x_j||)) / sigma_c^2). The aggregated features F(q_q^r) are obtained by weighted summation of the bound neural descriptors according to the normalized weights. F(q_q^r) is concatenated with the relative position vectors from the query point to each neighboring surface element and then input into the neural network Phi_sdf to obtain the symbolic distance value s(q_q^r). The refined symbolic distance field S_r is obtained by traversing all voxel centers within the refined voxel grid.

[0119] S206. Extract isosurfaces from the refined symbolic distance field to replace the surface portion of the 3D surface model within the area to be refined, thus obtaining the refined 3D surface model.

[0120] Among them, the refined symbolic distance field refers to the discrete field of symbolic distance values ​​generated by S205 within the bounding box of the surface region to be refined; the isosurface refers to the implicit surface formed by points where the symbolic distance field takes the value of zero; extraction refers to the operation process of generating triangular facets from the discrete symbolic distance field according to the isosurface extraction algorithm; the surface part within the surface region to be refined refers to the subset of triangular facets on the three-dimensional surface model that fall within the surface region to be refined; the refined three-dimensional surface model refers to the triangular mesh surface representation after completing local surface replacement and seam treatment.

[0121] Specifically, the refined symbolic distance field S_r is processed by extracting triangular facets near the zero isosurface using the moving cube method to obtain the locally refined surface M_r = (V_r_surf, F_r_surf); the original 3D surface model M_surf is culled according to the boundary of the surface region Omega_r to be refined, and the boundary vertices after culling are registered by number to obtain the boundary vertex ring E_b; the boundary vertices of the locally refined surface M_r are matched with the boundary vertex ring E_b by Euclidean distance, according to equation y_m `atch=argmin_{yinE_b}||v_r-y||` establishes the vertex correspondence and generates transition triangles F_bridge between the two boundary vertices according to the stitching triangulation method; merges M_surf after removing F_omega with the locally refined surface M_r and the transition triangle set F_bridge according to the unified vertex numbering system, and arranges the connection order of the merged triangles according to the normal consistency rule to obtain the refined 3D surface model M_surf^=(V_surf^,F_surf^*).

[0122] S207. Take points on the refined 3D surface model at preset sampling intervals, and filter out the aerial images that are not self-occluded by the refined 3D surface model according to the camera pose to obtain the visible aerial images corresponding to the sampling points.

[0123] Among them, the refined 3D surface model refers to the triangular mesh surface obtained by merging S206; the preset sampling interval refers to the preset Euclidean distance threshold delta_t between adjacent sampling points; self-occlusion refers to the phenomenon of other parts of the surface blocking the line connecting the sampling point to the camera optical center; visible aerial image refers to aerial image where the line connecting the camera optical center to the sampling point is not blocked by the refined 3D surface model; and screening refers to the operation process of selecting a subset that meets the visibility discrimination conditions from the aerial image set.

[0124] Specifically, uniform surface sampling is performed on the refined 3D surface model M_surf^* at a preset sampling interval delta_t to obtain a sampling point set T={t_k|k=1, 2, ..., K_t}; for each sampling point t_k, the aerial image set {I^l|l=1, 2, ..., L} and the corresponding camera optical centers {o^l} are traversed, and a visibility ray from the camera optical center to the sampling point is constructed as r_{k,l}(s)=o^l+s(t_k-o^l) / ||t_k-o^l||; the closest intersection point z_{k,l} between r_{k,l}(s) and the refined 3D surface model M_surf^* is calculated by a ray-triangle intersection method, and the distance from the ray starting point to the closest intersection point is calculated according to the formula d_{k,l}=||z_{k,l}-o^l||; visibility is determined according to the formula |d_{k,l}-||t_k-o^l|||<eps_v, where eps_v represents a preset visibility tolerance; the numbers of aerial images satisfying the visibility condition are classified into the visible aerial image set V(k)={I^l|the visibility condition is satisfied} corresponding to the sampling point t_k.

[0125] S208: Project the sampling points onto the visible aerial images according to the camera poses, and sample colors at the projected pixel positions to obtain the candidate texture colors of the sampling points under each visible aerial image.

[0126] Herein, the sampling points refer to 3D spatial points obtained on the refined 3D surface model at the preset sampling interval in S207; the projection refers to the operation process of mapping a 3D spatial point to a 2D pixel position according to the camera pose and camera intrinsic parameters; the visible aerial images refer to aerial images determined in S207 that have no self-occlusion with the sampling points; the projected pixel position refers to the pixel row and column number corresponding to the sampling point on the visible aerial image after projection; the candidate texture color refers to a red, green and blue three-channel color vector obtained by pixel sampling at the projected pixel position.

[0127] Specifically, for each sampling point t_k and each visible aerial image I^l in its set of visible aerial images V(k), the position vector t_k^cam = R_cam^l * t_k + t_cam^l in the camera coordinate system is calculated according to the camera extrinsic parameters {R_cam^l, t_cam^l}; the homogeneous pixel coordinates [u_{k,l} * z_{k,l}^cam, v_{k,l} z_{k,l}^cam, z_{k,l}^cam]^T = K^lt_k^cam are calculated according to the camera intrinsic parameter matrix K^l, and u_{k,l} = K^l_{11} * x_{k,l}^cam / z_{k,l}^cam + K^ l_{13} and v_{k,l}=K^l_{22}*y_{k,l}^cam / z_{k,l}^cam+K^l_{23} calculate the projected pixel positions (u_{k,l}, v_{k,l}); sample (u_{k,l}, v_{k,l}) in the red, green, and blue channels of the visible aerial image I^l using bilinear interpolation to obtain candidate texture colors c_{k,l}=(c_{k,l}^R, c_{k,l}^G, c_{k,l}^B); perform the above operation on the sampling point t_k and all visible aerial images to obtain the candidate texture color set C(k)={c_{k,l}|I^l∈V(k)}.

[0128] S209. Retrieve surface elements in the neighborhood of the sampling point within the target area, and use the distance-weighted average of the retrieved surface normals as the surface normal of the sampling point.

[0129] Among them, the target region refers to the spatial range covered by the three-dimensional surface model; the sampling point neighborhood refers to the local spatial region defined by the sampling point and the preset search radius; the surface element refers to the local surface unit registered by S204 and the original process, which includes the location and surface normal binding attributes; the distance-weighted average value refers to the result of performing a weighted summation and normalization operation on the surface normal according to the inverse ratio of the spatial distance between the neighboring surface element and the sampling point; the surface normal refers to the unit vector at the sampling point used to represent the orientation of the local surface.

[0130] Specifically, a spatial index structure T_ext is established for the extended surface element set P_ext according to spatial positions; for each sampling point t_k, the neighborhood surface element set N(k)={p_j|||t_k-x_j||<rho_n} is retrieved according to the index structure T_ext within a preset search radius rho_n; for each neighborhood surface element p_j, the inverse distance weight is calculated according to the formula w_{k,j}=1 / (||t_k-x_j||+eps_w), where eps_w denotes a preset small quantity for preventing division by zero; normalization processing is performed on the weights according to the formula w_{k,j}_norm=w_{k,j} / sum_{i∈N(k)}w_{k,i}; weighted summation is performed on the surface normal vectors bound to the neighborhood surface elements according to the formula n_k_raw=sum_{j∈N(k)}w_{k,j}_norm*n_j; unitization processing is performed on the weighted result according to the formula n_k=n_k_raw / ||n_k_raw|| to obtain the surface normal n_k at the sampling point t_k.

[0131] S210. According to a preset mapping relationship that increases as the included angle between the surface normal and the viewing direction of a visible aerial image decreases, the included angle is converted into a viewing weight corresponding to the candidate texture color.

[0132] Wherein, the surface normal refers to the unit normal vector at the sampling point obtained in S209; the viewing direction refers to the unit direction vector pointing from the sampling point to the optical center of the camera; the included angle refers to the spatial geometric angle theta between the surface normal and the viewing direction; the preset mapping relationship refers to a monotonically decreasing functional relationship where the smaller the included angle, the larger the mapping output; the viewing weight refers to a scalar value assigned to the corresponding candidate texture color and involved in subsequent weighted fusion.

[0133] Specifically, for each sampling point t_k and each visible aerial image I^l in its set of visible aerial images V(k), the unit vector of the view direction is calculated according to the formula d_{k,l}view=(o^l-t_k) / ||o^l-t_k||; the cosine value of the surface normal and the view direction is calculated according to the formula cos_theta{k,l}=n_k·d_{k,l}view, and values ​​of cos_theta{k,l} less than zero are truncated according to max(cos_theta_{k,l},0); and w_{k,l}vie is calculated according to the formula. The viewpoint weights are calculated using w = (cos_theta{k, l})^gamma_v, where gamma_v represents a preset index controlling the sensitivity of the weights to the included angle. The viewpoint weights are normalized using the formula w_{k, l}view_norm = w{k, l}view / sum{linV(k)}w_{k, l}view. The normalized viewpoint weights are obtained by performing the above operation on the sampled point t_k and all visible aerial images. The viewpoint weight set W(k) = {w{k, l}_view_norm|I^l∈V(k)} is obtained.

[0134] S211. The candidate texture colors are weighted and summed according to the viewpoint weights and arranged according to the sampling point positions to obtain the texture map bound to the refined 3D surface model.

[0135] Among them, the viewpoint weight refers to the normalized scalar value obtained by S210; the candidate texture color refers to the red, green and blue three-channel color vector obtained by S208; the weighted summation refers to the process of performing a linear combination operation on the candidate texture colors according to the viewpoint weight; the sampling point position arrangement refers to the process of performing a raster arrangement on the fused color values ​​according to the two-dimensional coordinates of the sampling points in the texture coordinate system; and the texture map refers to the two-dimensional raster color image bound to the vertex texture coordinates of the refined three-dimensional surface model.

[0136] Specifically, for each sampling point t_k, the candidate texture colors are weighted and summed according to the formula c_k=sum_{linV(k)}w_{k,l}view_norm*c{k,l} to obtain the blended color c_k=(c_k^R,c_k^G,c_k^B); the refined 3D surface model M_surf^ is expanded to the 2D texture coordinate plane using the minimum stretching parameterization method to obtain the vertex texture coordinates {uv_k}, and rasterized according to the preset texture map resolution W_t×H_t to obtain the texture pixel position (u_k_t) of the sampling point. ,v_k_t)=(round(uv_k.xW_t),round(uv_k.y*H_t)); For each sampling point t_k, backfill the texture pixel position (u_k_t, v_k_t) according to the blending color c_k to obtain a sparse texture map T_sparse; For the pixel positions in T_sparse that have not been backfilled, perform inverse distance weighted interpolation filling according to the neighboring backfilled pixels, and perform boundary expansion processing on the out-of-bounds pixels according to the boundary of the triangular facet obtained by surface parameterization to obtain a texture map T_final that is bound to the refined 3D surface model.

[0137] In some embodiments, the step of adding sampling points as new surface elements within the area of ​​the surface to be refined specifically includes:

[0138] The residual values ​​of the residual images corresponding to each pixel position in the surface region to be refined are back-projected onto the surface of the 3D surface model along the imaging direction to obtain the surface residual values ​​at each surface point in the surface region to be refined. Based on the surface residual values, the surface region to be refined is divided into multiple residual level sub-regions, where the larger the surface residual value, the higher the supplementary sampling density. Supplementary sampling is performed on the surface of the 3D surface model according to the supplementary sampling density corresponding to each residual level sub-region to obtain new surface elements. At the boundary of two adjacent residual level sub-regions with different residual levels, the transition sampling density is determined by distance-weighted interpolation of the supplementary sampling densities on both sides, and transition surface elements are supplemented according to the transition sampling density to keep the refined symbolic distance field continuous between different residual level sub-regions.

[0139] Specifically, this step upgrades the "supplementary sampling" process in the surface area to be refined in the aforementioned refinement process from uniform supplementation to hierarchical supplementation driven by differentiable rendering residuals. The core of this process is to use the unique differentiable rendering residual signal (rather than curvature, texture gradient, or any other general geometric quantity) as the sole driving source for sampling density, and then transfer this signal from the two-dimensional image space to the three-dimensional surface space to perform hierarchical density control and continuous transition processing.

[0140] For each aerial image within the refined surface region Omega_r, the residual image I_res is traversed to find the pixel position p_h within the corresponding pixel range of the surface region to be refined, and the scalar residual value e(p_h) at that pixel position is read. This residual value is not generated by external geometric analysis tools (such as curvature estimation or texture gradient calculation), but is directly measured by the pixel color difference between the predicted image generated by differentiable rendering of the symbolic distance field within this framework and the real aerial image. It only exists after the complete pipeline of "surface element binding neural descriptor + deformation map-driven pose update + symbolic distance field regression + differentiable rendering" converges. The imaging direction unit vector v_{p_h} is calculated based on the camera pose {R_cam, t_cam} corresponding to the pixel position and the camera intrinsic parameter matrix K. A back-projection ray r_{p_h}(t) = o + t * v_{p_h} is constructed and its intersection point y_h is calculated with the 3D surface model M_surf using the ray-triangle intersection method. The scalar residual value e(p_h) is assigned to the intersection point y_h as the surface residual value e_s(y_h) at that surface point. This back-projection operation transfers the residual signal from the 2D pixel grid to the 3D surface manifold, ensuring that subsequent sampling density is controlled in the surface's intrinsic coordinate system rather than the image coordinate system. This eliminates the problem of inconsistent residual measurement caused by projection scaling differences in the same surface region in images from different viewpoints. When the same surface point is hit by the pixel back projection of multiple aerial images, the maximum value of the residual value of each hit is taken as the final surface residual value of the surface point, e_s(y_h)=max_l{e^l(p_h^l)}. The above operation is performed on all hit surface points in the surface area to be refined to obtain the set of surface residual values ​​{e_s(y_h)|y_h∈Omega_r}.

[0141] Performing hierarchical division on the to-be-refined surface region according to the surface residual values: the number of preset residual levels is L_level, and the preset residual level threshold sequence tau_1<tau_2<...<tau_{L_level-1} divides the value range of the surface residual values into L_level intervals. The division basis here is the distribution of the differentiable rendering residual on the surface, rather than geometric curvature or texture complexity — a high residual means that the density of surface elements of the current signed distance field in this local region is insufficient to support the neural network to fit the detailed geometry. This criterion directly corresponds to the mechanism of "the neural descriptor aggregated by neighborhood surface elements regresses the signed distance value via the neural network" in this framework, and is a quantitative diagnosis of the insufficient local representation capability of the signed distance field, rather than a generalized estimation of the complexity of the surface itself. For each surface point y_h in the to-be-refined surface region, the residual level l_h∈{1, 2, ..., L_level} to which the surface point belongs is determined according to the number of the interval in which its surface residual value e_s(y_h) falls, and the set of surface points belonging to the same residual level and their connected triangular patches form a residual level sub-region Omega_r^l, so that a plurality of residual level sub-regions {Omega_r^1, Omega_r^2, ..., Omega_r^{L_level}} are obtained; a supplementary sampling density sequence rho_1<rho_2<...<rho_{L_level} corresponding to each residual level is preset, wherein a larger level number indicates a larger surface residual value, and the corresponding supplementary sampling density is higher, that is, the number of supplementary sampling points per unit surface area is larger. This hierarchical density strategy concentrates computing resources on local surfaces with severe geometric distortion in a residual-driven manner, and avoids investing redundant surface elements in flat regions where the residuals have converged.

[0142] Supplementary sampling is performed on the surface of the 3D surface model according to the supplementary sampling density corresponding to each residual level sub-region: For each residual level sub-region Omega_r^l, the area A_f of each triangular facet within the sub-region is calculated, and the number of supplementary sampling points N_f^l within each triangular facet is determined according to the formula N_f^l=round(rho_lA_f); for each triangular facet, N_f^l supplementary sampling points are generated by random sampling of the centroid coordinates, specifically by generating the spatial position of the sampling points according to the formula x_new=alphav_1+beta*v_2+(1-alpha-beta)*v_3, where v_1, v_2, and v_3 are the triangular facets. The vertex coordinates of the facet, alpha and beta, are randomly sampled within a triangular region that is uniformly distributed and satisfies alpha≥0, beta≥0 and alpha+beta≤1. For each supplementary sampling point, the surface normal n_new is obtained by weighting the area of ​​the normal of the triangular facet and the normal of the adjacent facets. The spatial position and surface normal of the supplementary sampling point are registered as the binding attributes of the new surface element. The above operation is performed on all residual level sub-regions to obtain the set of new surface elements P_new={p_i^new=(x_i^new,n_i^new)|i=1,2,...,N_new_total}.

[0143] For the oversampling processing at the boundary between two adjacent residual level sub-regions with different residual levels: Since the regression of the symbolic distance field in this framework depends on the aggregation of neural descriptors bound to the surface elements in the neighborhood of the query point, if the surface element density of adjacent sub-regions changes abruptly, the query point located near the boundary will cause a directional bias in the aggregation feature when performing neighborhood retrieval due to the sparse surface elements on one side and the dense surface elements on the other side. This will cause the zero isosurface of the regressed symbolic distance field to show a step-like discontinuity at the boundary. To eliminate this problem, the boundaries between residual level sub-regions within the surface area to be refined are detected. For each pair of adjacent residual level sub-regions Omega_r^{l_a} and Omega_r^{l_b} with residual levels l_a and l_b (l_a≠l_b), the shared boundary edge set E_{a, b} is extracted. For each boundary edge e_bnd in the boundary edge set E_{a, b} and the surface area within the preset transition bandwidth delta_trans on both sides, the geodesic distances d_a and d_b from each position point within the transition band to the interior of the two sub-regions are calculated. The distances are then calculated according to the formula rho_trans=rho_{l_a}*d_b / (d_a+d_b)+rho_{l_b}* d_a / (d_a+d_b) performs distance-weighted interpolation on the supplementary sampling densities on both sides to obtain the transition sampling density rho_trans at that location; calculate the number of supplementary sampling points for the triangular facets within the transition zone according to the transition sampling density rho_trans and generate transition surface elements by random sampling according to the centroid coordinates; register the transition surface elements as binding attributes of the new surface elements according to their spatial position and surface normal and incorporate them into the new surface element set P_new, so that the refined symbolic distance field obtained by regression in the surface region to be refined remains continuous between different residual level sub-regions due to the smooth transition of surface element distribution density, avoiding geometric discontinuities caused by abrupt changes in sampling density at the boundaries of the sub-regions in the isosurface extraction results.

[0144] The above four steps constitute a technical link deeply coupled with this framework: differentiable rendering residuals (generated only in the surface elements + symbolic distance field + differentiable rendering pipeline of this framework) → backprojection to the surface to obtain surface residual values ​​(transferring the signal from the image domain to the surface domain) → hierarchical control of surface element supplementation density according to surface residual values ​​(using residuals as a quantitative diagnosis of insufficient local expressive power of the symbolic distance field) → boundary transition to ensure the continuity of the refined symbolic distance field (eliminating neighborhood aggregation bias caused by density abrupt changes). The input of each step in this link depends on the output of the previous step and on the unique expression structure of this framework, which is fundamentally different from adaptive mesh refinement methods driven by general signals such as curvature or texture gradients in terms of driving source, scope, and continuity guarantee mechanism.

[0145] The following describes the UAV large-scale fine surface reconstruction system in the embodiments of this invention from the perspective of hardware processing. Please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of a physical device structure of a large-scale fine surface reconstruction system for UAVs in this application embodiment.

[0146] It should be noted that, Figure 3 The structure of the UAV large-scale fine surface reconstruction system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0147] like Figure 3 As shown, the UAV large-scale fine surface reconstruction system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 308 into random access memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0148] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including hard disks, etc.; and communication section 309 including network interface cards such as LAN (Local Area Network) cards, modems, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0149] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0150] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0152] Specifically, the UAV large-scale fine surface reconstruction system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the UAV large-scale fine surface reconstruction method that integrates implicit neural representation and surface elements provided in the above embodiment.

[0153] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the large-scale fine surface reconstruction system for UAVs described in the above embodiments; or it may exist independently and not assembled into the large-scale fine surface reconstruction system for UAVs. The storage medium carries one or more computer programs that, when executed by a processor of the large-scale fine surface reconstruction system for UAVs, cause the system to implement the large-scale fine surface reconstruction method for UAVs that integrates implicit neural representations and surface elements provided in the above embodiments.

[0154] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0155] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0156] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for large-scale fine-scale surface reconstruction of unmanned aerial vehicles (UAVs) that integrates implicit neural representation and surface elements, characterized in that, The method, applied to a large-scale fine-surface reconstruction system for unmanned aerial vehicles, includes: Aerial images of the target area taken by a drone from multiple perspectives are acquired, and the initial surface geometry is recovered based on the aerial images and discretized into a set of surface elements, wherein the surface elements have spatial position and surface normal. Learnable neural descriptors are bound to the surface elements, deformation graph nodes are arranged in the target area, and the association between the surface elements and the associated deformation graph nodes adjacent to the surface elements is established, so that the spatial position and surface normal of the surface elements are updated synchronously with the pose parameters of the associated deformation graph nodes. The surface elements in the neighborhood of the query point within the target area are retrieved, the neural descriptors bound to the retrieved surface elements are aggregated and input into the neural network, and the symbolic distance value of the query point is obtained by regression. The symbolic distance value constitutes a symbolic distance field within the target area. Based on the camera pose corresponding to the aerial image, a predicted image is generated from the symbolic distance field through differentiable rendering. The gradient is backpropagated based on the reconstruction loss of the predicted image and the aerial image to update the pose parameters of the deformable graph node and the neural descriptor, thereby obtaining the corrected pose parameters and the target neural descriptor. The surface element is driven to the correction position using the correction pose parameters. A dense symbolic distance field is generated based on the surface element at the correction position and the bound target neural descriptor. A three-dimensional surface model covering the target region is extracted from the dense symbolic distance field.

2. The method according to claim 1, characterized in that, The steps of binding learnable neural descriptors to the surface elements, arranging deformation graph nodes within the target region, and establishing associations between the surface elements and their neighboring deformation graph nodes specifically include: A learnable neural descriptor is assigned to each surface element, and the neural descriptor, together with the spatial position and surface normal of the surface element, is registered as a binding attribute of the surface element. Within the target area, the spatial positions of the surface elements are downsampled according to a preset voxel size. The spatial positions retained after downsampling are used as the initial positions of the deformation map nodes, and pose parameters for characterizing local rigidity transformations are initialized for each deformation map node. Within the target area, a predetermined number of the nearest deformation map nodes are retrieved for each surface element as associated deformation map nodes of the surface element, and skin weights are constructed based on the spatial distance between the surface element and the associated deformation map nodes. The pose parameters of the associated deformation graph nodes are weighted and composited based on the skin weights, and the weighted and composited pose parameters are applied to the spatial position and surface normal of the surface element, so that the spatial position and surface normal of the surface element are updated synchronously with the pose parameters of the associated deformation graph nodes, and the neural descriptor bound to the surface element keeps the index consistent with the updated spatial position.

3. The method according to claim 1, characterized in that, The steps of retrieving the surface elements in the neighborhood of the query point within the target region, aggregating the neural descriptors bound to the retrieved surface elements and inputting them into a neural network to regress and obtain the symbolic distance value of the query point specifically include: A spatial index structure is established within the target area based on the spatial location of the surface elements, and a preset number of the surface elements closest to the query point are retrieved based on the spatial index structure as neighboring surface elements; Based on the spatial distance between the neighboring surface element and the query point and the orientation deviation of the surface normal of the neighboring surface element relative to the query point, the aggregation weight of the neighboring surface element to the query point is calculated. The aggregation weight increases as the spatial distance and the orientation deviation decrease. Based on the aggregation weight, the neural descriptors bound to the neighborhood surface elements are weighted and summed to obtain the aggregated features corresponding to the query point; The aggregated features are concatenated with the relative position vector of the query point relative to the spatial position of the neighboring surface elements and then input into the neural network to regress and obtain the symbolic distance value of the query point. The symbolic distance value constitutes the symbolic distance field as the query point traverses the target region.

4. The method according to claim 1, characterized in that, The steps of generating a predicted image from the symbolic distance field based on the camera pose corresponding to the aerial image through differentiable rendering, backpropagating the gradient based on the reconstruction loss of the predicted image and the aerial image, and updating the pose parameters of the deformable graph nodes and the neural descriptor specifically include: The camera optical center is determined based on the camera pose corresponding to the aerial image, and rays are projected from the camera optical center along the pixel direction of the aerial image to the target area; Spherical tracing is performed along the ray on the symbolic distance field, and the intersection of the ray and the zero isosurface of the symbolic distance field is taken as the surface intersection point corresponding to the pixel direction; The surface elements in the neighborhood of the surface intersection are retrieved, the neural descriptors bound to the retrieved surface elements are aggregated, and the aggregation result is input into a color decoding network to regress and obtain the pixel color at the surface intersection. The pixel color is backfilled according to the pixel direction to form the predicted image. The reconstruction loss is constructed based on the difference in pixel color between the predicted image and the aerial image. The reconstruction loss is then used to backpropagate gradients to the pose parameters of the deformed image nodes and the neural descriptor. The pose parameters of the deformed image nodes and the neural descriptor are iteratively updated through gradient descent until the reconstruction loss converges, thereby obtaining the corrected pose parameters and the target neural descriptor.

5. The method according to claim 1, characterized in that, After the step of extracting a three-dimensional surface model covering the target region from the dense symbolic distance field, the method further includes: The dense symbol distance field is rendered differently according to the camera pose to obtain a rendered image that is pixel-by-pixel aligned with the aerial image. The absolute value of the color difference between the aerial image and the rendered image at the same pixel position is calculated to obtain the residual image; Pixels in the residual image whose residuals are greater than a preset residual threshold are back-projected onto the three-dimensional surface model along the imaging direction corresponding to the camera pose to obtain the surface region to be refined on the three-dimensional surface model. In the area of ​​the surface to be refined, additional sampling points are added as new surface elements, and the distance-weighted interpolation result of the target neural descriptor in the neighborhood is used as the initial neural descriptor bound to the new surface element. Within the surface region to be refined, neighborhood surface elements are re-searched according to the generation method of the dense symbolic distance field, the bound neural descriptors are aggregated, and the symbolic distance values ​​are regressed to obtain the refined symbolic distance field; The isosurface is extracted from the refined symbol distance field to replace the surface portion of the three-dimensional surface model in the area to be refined, thus obtaining the refined three-dimensional surface model.

6. The method according to claim 5, characterized in that, After the step of extracting a three-dimensional surface model covering the target region from the dense symbolic distance field, the method further includes: Points are taken on the refined 3D surface model at preset sampling intervals, and the aerial images that are not self-occluded by the refined 3D surface model are screened out according to the camera pose to obtain the visible aerial images corresponding to the sampling points. The sampling points are projected onto the visible aerial image according to the camera pose, and the color is sampled at the projected pixel position to obtain the candidate texture color of the sampling points in each visible aerial image. Within the target area, the surface elements in the neighborhood of the sampling point are retrieved, and the distance-weighted average of the retrieved surface normals is used as the surface normal of the sampling point. According to a preset mapping relationship that increases as the angle between the surface normal and the viewing angle direction of the visible aerial image decreases, the angle is converted into the viewing angle weight corresponding to the candidate texture color; The candidate texture colors are weighted and summed according to the viewpoint weights and arranged according to the sampling point positions to obtain a texture map bound to the refined 3D surface model.

7. The method according to claim 5, characterized in that, The step of adding sampling points as new surface elements within the area of ​​the surface to be refined specifically includes: The residual values ​​of the residual image corresponding to each pixel position in the surface region to be refined are back-projected onto the surface of the three-dimensional surface model along the imaging direction to obtain the surface residual values ​​at each surface point in the surface region to be refined. The surface region to be refined is divided into multiple residual level sub-regions based on the surface residual value, wherein the residual level sub-region with a larger surface residual value corresponds to a higher supplementary sampling density; Supplementary sampling is performed on the surface of the three-dimensional curved surface model according to the supplementary sampling density corresponding to each residual level sub-region to obtain the newly added surface elements; At the boundary of two adjacent residual level sub-regions with different residual levels, the transition sampling density is determined by distance-weighted interpolation based on the supplementary sampling densities on both sides, and the transition surface elements are supplemented by sampling according to the transition sampling density, so that the refined symbol distance field remains continuous between different residual level sub-regions.

8. A large-scale fine surface reconstruction system for unmanned aerial vehicles, characterized in that, The UAV large-scale fine surface reconstruction system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the UAV large-scale fine surface reconstruction system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the UAV large-scale fine surface reconstruction system, the UAV large-scale fine surface reconstruction system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the UAV large-scale fine surface reconstruction system, the UAV large-scale fine surface reconstruction system performs the method as described in any one of claims 1-7.

Images