Three-dimensional modeling processing method based on unmanned aerial vehicle oblique photography

Abstract

The invention provides a three-dimensional modeling processing method based on unmanned aerial vehicle oblique photography. The method is applied to the technical field of three-dimensional modeling, and comprises the following steps: obtaining a serialized image set containing geographical coordinate information according to original image data collected by an oblique photography camera carried by a multi-rotor unmanned aerial vehicle; according to time-space synchronization parameters of the serialized image set, determining a multi-view image matching relation matrix with an overlapping degree quantitative index; determining mixed three-dimensional point cloud data fusing sparse point cloud and dense point cloud according to geometric constraint conditions of the multi-view image matching relation matrix; determining an initial three-dimensional grid model with multi-level details according to the topological connection relationship of the mixed three-dimensional point cloud data; and determining an optimized three-dimensional model based on adaptive texture mapping according to the surface curvature distribution characteristics of the initial three-dimensional mesh model. In this way, the efficiency of three-dimensional modeling can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of three-dimensional modeling, and in particular to a three-dimensional modeling processing method based on oblique photography of unmanned aerial vehicles. Background Art

[0002] 3D modeling based on drone-mounted oblique photography combines drone technology with oblique photogrammetry to achieve 3D reconstruction. It is playing an increasingly important role in urban planning, land surveying, environmental monitoring, disaster assessment, and other fields. By using drones equipped with oblique cameras to acquire high-precision image data of ground objects and combining it with advanced 3D modeling and processing methods, accurate modeling and 3D visualization of ground objects and environments can be achieved. 3D modeling based on drone-mounted oblique photography enables the acquisition of high-precision imagery over large areas in a short period of time. Drones can flexibly navigate complex terrain, even in inaccessible areas, significantly improving data acquisition efficiency and accuracy. 3D reconstruction of oblique image data from drones uses precise geometric correction and incorporates spatial relationships between images to generate high-precision 3D point clouds. These point clouds form the foundation of 3D models. Processing and optimization of these point clouds ultimately yields precise 3D models that not only reflect the external form of the ground but also enable more accurate display and analysis in virtual environments, avoiding the potential errors associated with traditional methods. Drones, operated remotely or autonomously, can acquire high-precision image data without touching the ground, significantly reducing labor, material, and time costs. Therefore, three-dimensional modeling based on UAV oblique photography is of great significance.

[0003] At present, conventional methods use multiple flights or multiple sensors to collect data in time, resulting in inconsistent spatiotemporal benchmarks. Subsequent manual intervention is required for geographic coordinate alignment, which not only increases the complexity of data processing, but also creates invalid overlapping images, seriously restricting the timeliness of modeling. In the image matching and point cloud generation links, existing technologies mostly use a single sparse point cloud or dense point cloud processing process. Due to the lack of dynamic control of overlapping quantification indicators, the feature matching calculation amount grows exponentially. Especially when dealing with densely populated urban areas, topological fractures are prone to occur in the point cloud fusion stage, and multiple iterative optimizations are required to ensure continuity, which significantly prolongs the processing cycle. Existing methods will generate a large number of redundant triangular facets when processing complex surfaces, resulting in a surge in model data volume. At the same time, fixed-resolution texture mapping strategies are difficult to adapt to the expression requirements of different levels of detail. High-configuration hardware platforms still face low rendering efficiency, which in turn leads to low efficiency of three-dimensional modeling. Summary of the Invention

[0004] The present disclosure provides a method for 3D modeling based on drone oblique photography. The method includes:

[0005] S1, based on the original image data collected by the oblique photography camera carried by the multi-rotor drone, a serialized image set containing geographic coordinate information is obtained;

[0006] S2, determining a multi-view image matching relationship matrix with overlap quantification indicators according to the spatiotemporal synchronization parameters of the serialized image set;

[0007] S3, determining hybrid three-dimensional point cloud data by fusing the sparse point cloud and the dense point cloud according to the geometric constraints of the multi-view image matching relationship matrix;

[0008] S4, determining an initial three-dimensional mesh model with multiple levels of details based on the topological connection relationship of the hybrid three-dimensional point cloud data;

[0009] S5, determining an optimized three-dimensional model based on adaptive texture mapping according to the surface curvature distribution characteristics of the initial three-dimensional mesh model, wherein the optimized three-dimensional model includes an LOD dynamic scheduling mechanism and a multi-platform compatible format conversion function.

[0010] Furthermore, S3 includes:

[0011] S31, determining a set of matching pairs of point clouds with the same name according to a baseline length parameter of the multi-view image matching relationship matrix;

[0012] S32, determining a point cloud registration parameter matrix according to the spatial distribution density of the set of matching pairs of point clouds with the same name;

[0013] S33, determining a joint adjustment equation for fusing plane constraints and surface constraints according to the residual distribution characteristics of the point cloud registration parameter matrix;

[0014] S34, determining a dense point cloud reconstruction data set containing normal vector optimization information according to a least squares solution result of the joint adjustment equation;

[0015] S35, determining hybrid three-dimensional point cloud data with multi-source data complementary characteristics according to the fusion weight coefficient of the dense point cloud reconstruction data set and the lidar point cloud.

[0016] Further:

[0017] If the local point cloud density of the hybrid three-dimensional point cloud data is lower than a density threshold, the area is determined to be a data missing area and a make-up flight instruction is sent to the multi-rotor UAV;

[0018] If the global matching error of the multi-view image matching relationship matrix exceeds the error threshold, it is determined that the image acquisition posture is abnormal;

[0019] If the texture complexity of the dense point cloud reconstruction data set is higher than a complexity threshold, feature enhancement is performed using a patch segmentation algorithm.

[0020] Further:

[0021] If the rate of change of the boundary curvature of the data missing area exceeds a curvature threshold, multi-scale image data is collected by using a multi-altitude stacked flight mode of the multi-rotor UAV;

[0022] If the light intensity difference in the data missing area exceeds a light threshold, the exposure parameters of the oblique photography camera and the power of the fill light device are adjusted.

[0023] Furthermore, S4 includes:

[0024] S41, determining an initial facet set for semantic segmentation based on a normal vector consistency index of the mixed three-dimensional point cloud data;

[0025] S42, generating a topological connection relationship graph according to the geometric continuity features of the initial semantic segmentation facets;

[0026] S43, determining implicit surface generation parameters of the fused Poisson surface reconstruction equation according to the edge weight distribution of the topological connection relationship graph;

[0027] S44, determining a multi-resolution mesh model with adaptive subdivision capability based on the normal field optimization result of the implicit surface generation parameters;

[0028] S45 , determining a multi-level detail three-dimensional mesh model optimized by a feature-preserving filtering algorithm according to a Hausdorff distance detection result of the multi-resolution mesh model.

[0029] Further:

[0030] If the local geometric accuracy of the multi-resolution grid model is lower than the accuracy threshold, performing surface fitting optimization by a moving least squares method;

[0031] If the crack length ratio is detected to exceed the crack threshold during the isosurface extraction process of the implicit surface generation parameters, topology repair is performed using a non-rigid deformation compensation algorithm;

[0032] If the noise suppression strength of the feature-preserving filtering algorithm exceeds a suppression threshold, the geometric detail retention rate and the model smoothness index are automatically balanced.

[0033] Furthermore, the surface fitting optimization by the moving least squares method includes:

[0034] If the angle between the local geometric accuracy deviation direction and the main curvature direction exceeds the angle threshold, the mode is switched to anisotropic fitting weight distribution;

[0035] If the number of surface fitting iterations exceeds the threshold and the residuals do not converge, the parameters are adaptively adjusted through particle swarm optimization;

[0036] If the Gaussian curvature mutation value of the fitted surface exceeds the mutation threshold, control vertices are inserted to reconstruct the parametric expression of the NURBS surface.

[0037] Furthermore, S5 includes:

[0038] S51, determining a multi-view texture mapping weight matrix based on projection deformation correction according to UV unfolding parameters of the initial three-dimensional mesh model;

[0039] S52, determining a texture fusion coefficient generated by an illumination estimation model according to a color consistency index of the multi-view texture mapping weight matrix;

[0040] S53, determining a texture seam elimination solution based on a Markov random field according to the spatial continuity characteristics of the texture fusion coefficient;

[0041] S54: Determine a dynamic LOD scheduling strategy that supports both WebGL and ARCore engines based on a visual quality evaluation result after texture seam elimination.

[0042] S55 , determining to output a multi-platform compatible three-dimensional model in OSGB, GLTF and point cloud PCD formats according to the rendering performance parameters of the dynamic LOD scheduling strategy.

[0043] Further:

[0044] If the distortion rate of the multi-view texture mapping weight matrix exceeds a distortion threshold, optimizing the texture coordinates by resampling the control points;

[0045] If the ambient light occlusion coefficient of the illumination estimation model is lower than the occlusion threshold, the real illumination data collected by the ambient light probe is superimposed for compensation;

[0046] If the frame rate fluctuation amplitude of the dynamic LOD scheduling strategy exceeds the frame rate threshold, the level division granularity of the tile pyramid is adjusted.

[0047] Furthermore, the optimization of texture coordinates obtained by resampling control points includes:

[0048] If the texture stretching rate after control point resampling exceeds the stretching threshold, a secondary optimization is performed using a texture relaxation algorithm based on differential coordinates;

[0049] If the resolution difference of the maps after texture coordinate optimization exceeds the resolution threshold, a seamless map set is generated through multi-scale texture synthesis technology.

[0050] The present invention establishes precise matching priority and spatiotemporal parameter constraints through overlapping quantitative indicators to reduce the amount of redundant feature matching calculations; combines sparse matching speed with dense reconstruction accuracy through a hybrid point cloud strategy, and optimizes GPU parallel computing resource allocation through geometric constraints; automatically generates topological connection relationships to reduce manual repair, and implements detailed hierarchical construction through multi-level LOD; reduces the amount of curvature-adaptive non-uniform texture data, and dynamically schedules LOD to reduce real-time rendering load, thereby improving the efficiency of 3D modeling.

[0051] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0053] Figure 1 A flowchart of a three-dimensional modeling processing method based on drone oblique photography according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0055] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0056] Figure 1 A flowchart of a method for processing three-dimensional modeling based on drone oblique photography according to an embodiment of the present disclosure is shown, and the method includes:

[0057] S1, based on the original image data collected by the oblique photography camera carried by the multi-rotor drone, a serialized image set containing geographic coordinate information is obtained;

[0058] S2, determining a multi-view image matching relationship matrix with overlap quantification indicators according to the spatiotemporal synchronization parameters of the serialized image set;

[0059] S3, determining hybrid three-dimensional point cloud data by fusing the sparse point cloud and the dense point cloud according to the geometric constraints of the multi-view image matching relationship matrix;

[0060] S4, determining an initial three-dimensional mesh model with multiple levels of details based on the topological connection relationship of the hybrid three-dimensional point cloud data;

[0061] S5, determining an optimized three-dimensional model based on adaptive texture mapping according to the surface curvature distribution characteristics of the initial three-dimensional mesh model, wherein the optimized three-dimensional model includes an LOD (Level of Detail) dynamic scheduling mechanism and a multi-platform compatible format conversion function.

[0062] According to the embodiments of the present disclosure, precise matching priorities are established through overlap quantification indicators, spatiotemporal parameter constraints are used to reduce the amount of redundant feature matching calculations; a hybrid point cloud strategy is used to combine sparse matching speed with dense reconstruction accuracy, and geometric constraints are used to optimize GPU parallel computing resource allocation; topological connection relationships are automatically generated to reduce manual repair, and multi-level LODs are used to achieve detailed hierarchical construction; the amount of curvature-adaptive non-uniform texture data is reduced, and LOD dynamic scheduling reduces real-time rendering load, thereby improving the efficiency of three-dimensional modeling.

[0063] In some embodiments, S2 includes: in a drone oblique photography mission, the drone flies along a preset route, equipped with a five-lens oblique camera (front, back, left, right, and downward), and collects a total of 10 images (numbered IMG1-IMG10). The spatiotemporal parameters (latitude and longitude, altitude, timestamp, and camera attitude) of each image are recorded; the GPS coordinates (longitude, latitude, altitude), timestamp, and camera attitude (pitch angle, yaw angle, and roll angle) of each image are read from the POS system, for example: IMG1: longitude 116.3°E, latitude 39.9°N, altitude 120m, time 10:00:00, downward-looking camera; IMG2: Longitude 116.3001°E, latitude 39.9°N, altitude 120m, time 10:00:02, forward-looking camera (-30° pitch angle); calculate the ground coverage of each image based on camera parameters (focal length, sensor size) and attitude. For example, the ground coverage area of ​​IMG1 (downward-looking) is rectangular A, and the inclined area of ​​IMG2 (forward-looking) is trapezoidal B; quantify overlap indicators, including horizontal overlap: the proportion of overlapping area of ​​the ground projections of the two images (for example, IMG1 and IMG3 overlap 70%) and vertical overlap: the complementarity of the perspectives of images at different heights / angles (for example, IMG2 (forward-looking) and IMG5 (right-looking) overlap 60% on the side of the building); create a 10×10 matrix, with rows and columns representing image numbers and element values ​​being the overlap (horizontal + vertical); an example matrix fragment is:

[0064] IMG1 IMG2 IMG3 IMG4 ... IMG1 - 20% 70% 5% ... IMG1 20% - 65% 60% ... IMG1 70% 65% - 15% ... ... ... ... ... ... ...

[0065] In some embodiments, S3 includes: S31, determining a set of point cloud matching pairs with the same name based on the baseline length parameter of the multi-view image matching relationship matrix; S32, determining a point cloud registration parameter matrix based on the spatial distribution density of the point cloud matching pair set with the same name; S33, determining a joint adjustment equation for fusing plane constraints and surface constraints based on the residual distribution characteristics of the point cloud registration parameter matrix; S34, determining a dense point cloud reconstruction data set containing normal vector optimization information based on the least squares solution result of the joint adjustment equation; S35, determining mixed three-dimensional point cloud data based on the fusion weight coefficient of the dense point cloud reconstruction data set and the lidar point cloud. According to the embodiments of the present disclosure, effective matching pairs are screened by using baseline length parameters, redundant calculations and mismatches are reduced, and the initial matching efficiency is improved; the registration parameters are optimized by using spatial distribution density, high confidence areas are prioritized, and the number of registration iterations is reduced; adjustment constraints are dynamically fused through residual distribution, plane and surface fitting are balanced, and error convergence is accelerated; normal vector optimization is used to enhance the geometric consistency of point clouds, reduce post-processing corrections, and accelerate dense reconstruction; the overall accuracy and integrity are improved through the complementary fusion of multi-source data, and the time-consuming correction of single data defects is reduced, thereby improving the efficiency of three-dimensional modeling.

[0066] For example, S31, based on the matching relationship matrix of the image, a geometric matching algorithm (such as a basic matrix or a homography matrix) is used to determine the pairs of points with the same name. Each pair of matching points should meet spatial consistency. Based on the selection of the baseline length, the mismatched point pairs can be filtered out, and the point pairs with a longer distance and higher matching degree are given priority. S32, based on the spatial distribution density of the point cloud matching pairs, the least squares method (Levenberg-Marquardt or other optimization algorithms) is used to solve the transformation matrix between the point clouds, including the rotation matrix and the translation vector. The registration parameter matrix may include rotation and translation information for subsequent precise registration of the point clouds. S33, the residual distribution characteristics can be obtained through the error analysis of the point cloud registration. If the residual is large, there may be inaccurate registration or data noise. Plane constraints usually refer to the plane structure in the point cloud data. The normal vector information of the point cloud is used to constrain the point cloud registration. The surface constraints are constrained by the geometric characteristics of the curved surface area on the point cloud. The joint adjustment equation will integrate these constraints to optimize the registration accuracy while minimizing the residual and taking into account the constraints of the plane and surface. S34, the least squares solution of the joint adjustment equation will provide optimized point cloud registration parameters and normal vector information. The optimized normal vector can more accurately reflect the surface morphology of the point cloud and improve reconstruction accuracy. Dense point cloud reconstruction can generate dense 3D point cloud data through techniques such as structured light and stereo matching. The optimized normal vector helps improve reconstruction quality, especially in areas with rich details. By combining the normal vector of each point with its position, a complete dense point cloud dataset is formed, supporting subsequent high-precision 3D reconstruction. S35, using weighted averaging or other multi-source data fusion techniques (such as weighted least squares method, Kalman filtering, etc.), the fusion weight coefficient is determined based on the quality, density and resolution of each point cloud data. The final generated hybrid 3D point cloud dataset can combine the high precision of the lidar point cloud with the details of the dense point cloud to form a 3D reconstruction result with higher accuracy and richer information.

[0067] In some embodiments, if the local point cloud density of the mixed three-dimensional point cloud data is lower than a density threshold, the area is determined to be a data missing area and a make-up flight instruction is sent to the multi-rotor drone; if the global matching error of the multi-view image matching relationship matrix exceeds the error threshold, it is determined that the image acquisition posture is abnormal; if the texture complexity of the dense point cloud reconstruction data set is higher than the complexity threshold, feature enhancement is performed through a patch segmentation algorithm. According to the embodiment of the present disclosure, by detecting the local point cloud density to determine whether the area is a data missing area and sending a make-up flight instruction to the drone, the reduction in reconstruction accuracy caused by data missing is effectively reduced, ensuring that the final generated three-dimensional model or map is more complete and accurate; by analyzing the global matching error of the multi-view image matching relationship matrix, if the error exceeds the threshold, the system can identify the image acquisition posture abnormality. Posture anomalies may be caused by drone control problems or sensor failures. Timely detection and correction can avoid inconsistencies in subsequent image data or low-quality reconstruction results. By analyzing the texture complexity in the dense point cloud reconstruction dataset, if the complexity exceeds the set threshold, the patch segmentation algorithm will be used to enhance the features of the area, which helps to better capture details, thereby improving the quality of 3D reconstruction and further improving the efficiency of 3D modeling.

[0068] For example, when the number of valid points in a 10cm×10cm grid in mixed 3D point cloud data is less than 20 / m 2 When the POS data is used to determine the pitch angle deviation of the current route, it is determined that the pitch angle deviation of the current route exceeds 5°, triggering an abnormal flight attitude alarm.

[0069] In some embodiments, if the rate of change of the boundary curvature of the data missing area exceeds a curvature threshold, multi-scale image data is collected through the multi-altitude stacked flight mode of the multi-rotor drone; if the light intensity difference in the data missing area exceeds a light threshold, the exposure parameters of the oblique photography camera and the power of the fill light device are adjusted. According to the embodiments of the present disclosure, through the multi-altitude stacked flight mode of the multi-rotor drone, image data can be collected at different heights and viewing angles, which helps to provide a more comprehensive and detailed view, especially suitable for complex terrain and scenes; when the boundary curvature change rate of the data missing area exceeds a threshold, the flight mode is automatically adjusted to collect images, reducing data loss caused by terrain complexity or changes, and ensuring data continuity and integrity; when the light intensity difference in the data missing area exceeds a threshold, the exposure parameters of the oblique photography camera and the power of the fill light device can be automatically adjusted to ensure that clear and stable image data can still be obtained under different lighting conditions, avoiding the image quality being affected by lighting problems; the collection strategy is automatically adjusted according to environmental changes to ensure the efficiency and accuracy of data collection, reduce manual intervention, and improve work efficiency; by adjusting the exposure parameters and the power of the fill light device, more details can be captured in low-light or high-contrast environments, improving the quality and usability of the image, and thereby improving the efficiency of three-dimensional modeling.

[0070] For example, if a data missing area of ​​0.8m is detected -1 The instantaneous curvature change (the preset threshold is 0.5m -1 ), the flight control system immediately switches to the preset vertical layered flight program, completing a round-trip flight at each of the three altitude layers of 30m, 15m, and 5m, acquiring aerial images with ground resolutions of 0.2m, 0.1m, and 0.05m, respectively; if the light intensity in the data-missing area shows a sudden change of 20,000 lux (the preset threshold is 15,000 lux), the oblique camera automatically adjusts the exposure time from 1 / 1000s to 1 / 500s, and at the same time increases the output power of the onboard LED array fill light from 200W to 300W.

[0071] In some embodiments, S4 includes: S41, determining an initial facet set for semantic segmentation based on a normal vector consistency index of the mixed three-dimensional point cloud data; S42, generating a topological connection relationship graph based on the geometric continuity features of the initial facets for semantic segmentation; S43, determining implicit surface generation parameters of a fused Poisson surface reconstruction equation based on the edge weight distribution of the topological connection relationship graph; S44, determining a multi-resolution mesh model with adaptive subdivision capability based on the normal field optimization results of the implicit surface generation parameters; S45, determining a multi-level detail three-dimensional mesh model optimized by a feature-preserving filtering algorithm based on the Hausdorff distance detection results of the multi-resolution mesh model. According to the embodiments of the present disclosure, semantic patches are automatically divided through the normal vector consistency index to reduce manual intervention and redundant calculations; a topological connection relationship diagram is constructed through geometric continuity features to avoid iterative attempts of invalid patch combinations; the parameters of the Poisson equation are adaptively optimized through edge weight distribution to reduce the computational complexity of surface fitting; normal field optimization combined with adaptive subdivision is used to reduce uniform grid redundancy and retain key geometric features; Hausdorff distance detection and feature retention filtering are used to quickly balance detail accuracy and computational complexity, thereby improving the efficiency of three-dimensional modeling.

[0072] For example, S41 performs denoising on the input mixed 3D point cloud data, including smoothing and removing abnormal points; calculates the normal vector of each point in the point cloud, which is usually obtained by fitting a local plane using the least squares method; measures the consistency of the normal vectors by the angle between the normal vectors, and uses cosine similarity to calculate the consistency of the normal vectors of each pair of points; uses this consistency measure to divide the point cloud into multiple initial facet sets, that is, points with similar normal vectors are classified as the same facet; based on the normal vector consistency, obtains a preliminary facet set; S42 calculates the geometric features of each facet, For example, boundary smoothness, curvature change, edge connection of facets, etc.; evaluate the geometric continuity between facets to determine whether they are connected by shared edges or vertices, or whether there are obvious geometric breaks between them; construct a topological connection relationship graph based on the connection relationship between facets (such as adjacency, shared edges or vertices); each node in the graph represents a facet, and the edge represents the adjacent facets. The weight of the edge can be weighted based on information such as geometric similarity, distance, and normal vector difference between the facets; S43, analyze the weight of each edge in the topological graph to determine which facets The connection strength between them is high and which is weak; based on the edge weight distribution of the topological connection relationship graph, an implicit surface is generated through the Poisson surface reconstruction method; S44, after the implicit surface is generated, the normal field is optimized to ensure that the normal vector of the reconstructed surface transitions smoothly between the facets; adaptive subdivision is performed on the surface, and the subdivision strategy is adjusted according to the local geometric complexity of the surface (such as curvature), using fewer subdivisions in flatter areas and more subdivisions in areas with larger curvature; a multi-level mesh model is generated to represent surfaces at different resolutions; through multi-resolution meshes, appropriate accuracy and efficiency can be selected in different application scenarios; S45, the Hausdorff distance between the multi-resolution mesh model and the original point cloud data is calculated to evaluate the difference in the distance between the farthest points in space between the two. The smaller the Hausdorff distance, the closer the reconstructed mesh is to the original data; the feature-preserving filtering algorithm is used to further optimize the multi-resolution mesh to ensure that important geometric features and details are not lost during the filtering process; after optimization through feature-preserving filtering, a multi-level 3D mesh model with richer details and more accurate shape is obtained.

[0073] In some embodiments, if the local geometric accuracy of the multi-resolution mesh model is lower than the accuracy threshold, surface fitting optimization is performed by the moving least squares method; if the crack length ratio is detected to exceed the crack threshold during the isosurface extraction of the implicit surface generation parameters, topology repair is performed by the non-rigid deformation compensation algorithm; if the noise suppression strength of the feature-preserving filtering algorithm exceeds the suppression threshold, the geometric detail retention rate and the model smoothness index are automatically balanced. According to the embodiment of the present disclosure, adaptive surface fitting is performed only for low-precision areas by the moving least squares method, avoiding global recalculation and reducing redundant calculations; non-rigid deformation compensation based on the crack length ratio is performed only when topological defects are detected, preventing time consumption in iterative reconstruction of the entire model; the detail retention rate and smoothness are automatically adjusted according to the noise suppression strength, avoiding efficiency loss caused by repeated manual adjustment of parameters, thereby improving the efficiency of three-dimensional modeling.

[0074] For example, when the Hausdorff distance detection value in the multi-resolution grid model reaches 7mm (the preset accuracy threshold is 5mm), a moving least squares surface fitting is performed in this area with a spherical neighborhood with a radius of 50mm, and the control point spacing is set to 10mm and the polynomial order is 2. After 3 iterations, the Hausdorff distance of this area is reduced to 3mm, while keeping the curvature change rate of the window frame edge ≤0.05mm. -1 ; During the isosurface extraction process, it was detected that the length of continuous cracks accounted for 15% (the crack threshold was 10%). A non-rigid deformation compensation algorithm was used, and the control grid resolution was set to 0.1m and the deformation energy weight λ=0.7. After 4 iterations, the crack ratio was reduced to 2%. After the repair, the Laplace coordinate offset of the grid vertices was controlled within ±2mm; when the Gaussian kernel standard deviation of the feature-preserving filter reached 0.4mm (the suppression threshold was 0.3mm), the system gradually reduced the noise suppression parameter from σ=0.4mm to 0.2mm, and at the same time increased the curvature sensitivity coefficient from 0.8 to 1.2. After optimization, the root mean square roughness of the model surface was improved from 0.15mm to 0.08mm, and the curvature extreme value retention rate of the eaves feature line was maintained at more than 98%.

[0075] In some embodiments, the surface fitting optimization by moving least squares method includes: if the angle between the local geometric accuracy deviation direction and the main curvature direction exceeds an angle threshold, switching to an anisotropic fitting weight distribution mode; if the number of surface fitting iterations exceeds the number threshold and the residual has not converged, adaptive adjustment of the parameters through particle swarm optimization; if the Gaussian curvature mutation value of the fitted surface exceeds the mutation threshold, inserting control vertices to reconstruct the parametric expression of the NURBS surface.

[0076] For example, when it is detected that the angle between the geometric fitting deviation direction (vector direction angle θ=65°) and the main curvature direction (direction angle θ=25°) reaches 40° (preset threshold 30°), the weight distribution mode is switched to anisotropic fitting, the kernel function weight coefficient along the main curvature direction (long axis) is set to 0.8, the transverse (short axis) weight coefficient is set to 0.3, and the Gaussian kernel function is used (long axis bandwidth 15mm, short axis bandwidth 5mm); if the residual of the moving least squares fitting still fluctuates between 0.12mm-0.15mm (convergence threshold 0.05mm) after 15 iterations (preset number threshold 10 times), the particle swarm size is set to 50, the iterations are 20 times, and the parameter search range is: regularization coefficient λ∈[0.01,0.1], kernel radius r∈[5mm,30mm]; if the Gaussian curvature is from 0.6m -2 Sudden increase to 2.3m -2 (Mutation threshold 1.5m -2 ), the system inserts three control vertices at the parameter domain coordinates (u=0.42,v=0.78), adjusts the original NURBS node vector from uniform distribution U={0,0.25,0.5,0.75,1} to non-uniform U={0,0.2,0.4,0.6,0.8,1}, and increases the number of control vertices from 24 to 27.

[0077] In some embodiments, S5 includes: S51, determining a multi-perspective texture mapping weight matrix based on projection deformation correction according to the UV unfolding parameters of the initial three-dimensional mesh model; S52, determining a texture fusion coefficient generated by an illumination estimation model according to the color consistency index of the multi-perspective texture mapping weight matrix; S53, determining a texture seam elimination scheme based on a Markov random field according to the spatial continuity characteristics of the texture fusion coefficient; S54, determining a dynamic LOD scheduling strategy that supports WebGL and ARCore dual engines according to the visual quality evaluation results after texture seam elimination; S55, determining to output a multi-platform compatible three-dimensional model in OSGB, GLTF and point cloud PCD formats according to the rendering performance parameters of the dynamic LOD scheduling strategy. According to the embodiments of the present disclosure, multi-perspective texture mapping weights are automatically allocated based on projection deformation correction, thereby avoiding manual alignment of perspectives and repeated projection calculations; texture fusion coefficients are generated through an illumination estimation model, thereby reducing the time cost of manually adjusting illumination consistency; Markov random fields automatically eliminate texture seams, replacing manual patching and improving texture fusion efficiency; a dynamic LOD scheduling strategy optimizes rendering loads in real time based on visual quality assessment results, thereby avoiding redundant detail calculations; and multi-platform format compatible output reduces the secondary processing overhead of model conversion and adaptation, thereby improving the efficiency of three-dimensional modeling.

[0078] For example, in S51, when UV unfolding the initial 3D mesh model, a conformal mapping algorithm is used to generate UV coordinates. When the projection distortion rate reaches 0.15 pixels / cm², a 50×50 grid is divided in the UV parameter domain, and a quadratic polynomial fitting is used to correct the projection deformation. The weight of the front view image in the weight matrix is ​​set to 0.6 and the weight of the side view image is set to 0.4. After three iterations, the projection error is reduced to 0.02 pixels / cm². In S52, for the color difference (ΔE=8.5) of multi-view images in the wall area, a spherical harmonic illumination estimation model is used to decompose the texture fusion coefficient into a diffuse reflection component α=0.7 and a specular reflection component β=0.3. The color difference ΔE≤2.3 is made through chromaticity histogram matching, and a spatial continuity constraint radius of 15 cm is set in the shadow transition area. In S53, for the seam area, a Markov random field energy function E=λ1*color difference+λ2*gradient difference (λ1=0.6,λ2= 0.4), the graph cut algorithm was used for five iterative optimizations, which increased the SSIM index of the seam area from 0.82 to 0.96 and reduced the proportion of visual discontinuity length from 3.2% to 0.7%; S54, after the visual quality score VMAF = 92 points was obtained through quality assessment, the LOD dynamic scheduling parameters were set: a 500,000-facet model was loaded when the viewpoint distance was 0-20m, a simplified version of 100,000 facets was loaded when the viewpoint distance was 20-50m, and a low-poly model of 20,000 facets was loaded when the viewpoint distance was above 50m; S55, when outputting, the OSGB format was set to use DEFLATE compression (compression rate 75%), the GLTF format was set to merge vertex spacing ≤ 0.5mm, and the PCD format was set to downsample voxels by 2cm. A three-platform compatible data package containing the Forbidden City Hall of Supreme Harmony model (center coordinates 116.3975°E, 39.9085°N) was generated, with an OSGB file size of 328MB, a GLTF file size of 178MB, and a PCD point cloud with 120 million vertices.

[0079] In some embodiments, if the distortion rate of the multi-view texture mapping weight matrix exceeds the distortion threshold, the texture coordinates are optimized by resampling the control points; if the ambient occlusion coefficient of the illumination estimation model is lower than the occlusion threshold, the real illumination data collected by the ambient light probe is superimposed for compensation; if the frame rate fluctuation amplitude of the dynamic LOD scheduling strategy exceeds the frame rate threshold, the hierarchical division granularity of the tile pyramid is adjusted. According to the embodiment of the present disclosure, by resampling the control points only for the distortion exceeding limit area, the global texture coordinate recalculation is avoided and computational redundancy is reduced; based on the ambient occlusion coefficient, the real illumination data is superimposed, and the missing illumination information is supplemented as needed, reducing the resource consumption of manual adjustment and global rendering; by dynamically adjusting the tile pyramid hierarchical granularity, the rendering load and detail accuracy are balanced, and the performance waste caused by fixed LOD is avoided, thereby improving the efficiency of three-dimensional modeling.

[0080] For example, when the texture mapping distortion rate of the detected area (coordinates 116.3976°E, 39.9153°N) reaches 0.12 pixels / cm² (the preset distortion threshold is 0.1 pixels / cm²), the 2m×2m area is divided into an 8×8 control point grid in the UV parameter domain, and the texture coordinates are resampled using the thin plate spline interpolation algorithm, with the control point spacing set to 25cm and the regularization parameter λ=0.05; the ambient light occlusion coefficient is measured to be 0.4 (occlusion threshold 0.6) in the area (coordinates 116.3975°E, 39.9085°N), and 12 ambient light probes (radius 1.2m / probe, 512×512 HDR sampling), the collected global illumination data was fused with spherical harmonic coefficients (3rd order, weight α=0.85); when the model (5.2 million facets) was running in the Unity engine, the frame rate fluctuation in the viewing distance range of 30-50m was detected to be ±8fps (threshold ±5fps), and the original tile pyramid level division was adjusted from 3 layers (50m / 100m / 200m) to 5 layers (30m / 60m / 120m / 240m / 480m), and the gradient of the number of facets in adjacent layers was set to ≤50% (for example, the number of facets in the 50m layer was adjusted from 120,000 to 80,000→40,000→20,000→10,000).

[0081] In some embodiments, the optimization of texture coordinates obtained by resampling control points includes: if the texture stretch rate after control point resampling exceeds a stretch threshold, performing secondary optimization using a texture relaxation algorithm based on differential coordinates; if the difference in map resolution after texture coordinate optimization exceeds a resolution threshold, generating a seamless map set using multi-scale texture synthesis technology. According to the disclosed embodiments, by applying differential coordinate optimization to areas where the texture stretch rate exceeds the limit, iterative calculation of global texture coordinates is avoided; and by using multi-scale texture synthesis to automatically adapt to the map resolution requirements of different areas, the time cost of manual stitching and interpolation optimization is eliminated, thereby improving the efficiency of 3D modeling.

[0082] For example, when a texture stretching rate of 18% (preset stretching threshold of 15%) was detected in the resampled region (UV coordinates u = 0.15-0.85, v = 0.2-0.75), a relaxation algorithm based on differential coordinates was used: the UV space was divided into a 256×256 grid, the diagonal elements of the Laplacian operator weight matrix were set to 0.6, the adjacent edge weights were set to -0.1, and the relaxation factor β was set to 0.35. After 8 iterations, the stretching rate was reduced to 13.5%, and the maximum mesh vertex offset Δuv = 0. 02; When the texture resolution of the optimized wall area (texture coordinates u=0.3-0.6) suddenly changes from 2048×2048 to 1024×1024 (resolution difference ratio 2:1, exceeding the threshold of 1.5:1), wavelet pyramid decomposition is used to generate a 5-layer multi-scale texture (scaling factor 0.5). After frequency domain fusion, a 4096×4096 seamless tile set is output. The cross-scale blending width is set to 12 pixels and the anisotropic filtering level is set to 8x. The final PSNR value between adjacent tiles is ≥42dB.

[0083] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0084] It should be understood that the various forms of the above-mentioned processes can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved. This is not limited herein.

[0085] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A 3D modeling processing method based on UAV oblique photography, characterized in that: include: S1, based on the original image data collected by the oblique photography camera carried by the multi-rotor drone, a serialized image set containing geographic coordinate information is obtained; S2, determining a multi-view image matching relationship matrix with overlap quantification indicators according to the spatiotemporal synchronization parameters of the serialized image set; S3, determining hybrid three-dimensional point cloud data by fusing the sparse point cloud and the dense point cloud according to the geometric constraints of the multi-view image matching relationship matrix; S4, determining an initial three-dimensional mesh model with multiple levels of details based on the topological connection relationship of the hybrid three-dimensional point cloud data; S5, determining an optimized three-dimensional model based on adaptive texture mapping according to the surface curvature distribution characteristics of the initial three-dimensional mesh model, wherein the optimized three-dimensional model includes an LOD dynamic scheduling mechanism and a multi-platform compatible format conversion function.

2. The 3D modeling processing method based on UAV oblique photography according to claim 1, characterized in that S3 include: S31, determining a set of matching pairs of point clouds with the same name according to a baseline length parameter of the multi-view image matching relationship matrix; S32, determining a point cloud registration parameter matrix according to the spatial distribution density of the set of matching pairs of point clouds with the same name; S33, determining a joint adjustment equation for fusing plane constraints and surface constraints according to the residual distribution characteristics of the point cloud registration parameter matrix; S34, determining a dense point cloud reconstruction data set containing normal vector optimization information according to a least squares solution result of the joint adjustment equation; S35, determining hybrid three-dimensional point cloud data with multi-source data complementary characteristics according to the fusion weight coefficient of the dense point cloud reconstruction data set and the lidar point cloud.

3. The 3D modeling method based on drone oblique photography according to claim 2, characterized in that: If the local point cloud density of the hybrid three-dimensional point cloud data is lower than a density threshold, the area is determined to be a data missing area and a make-up flight instruction is sent to the multi-rotor UAV; If the global matching error of the multi-view image matching relationship matrix exceeds the error threshold, it is determined that the image acquisition posture is abnormal; If the texture complexity of the dense point cloud reconstruction data set is higher than a complexity threshold, feature enhancement is performed using a patch segmentation algorithm.

4. The 3D modeling method based on drone oblique photography according to claim 3, characterized in that: If the rate of change of the boundary curvature of the data missing area exceeds a curvature threshold, multi-scale image data is collected by using a multi-altitude stacked flight mode of the multi-rotor UAV; If the light intensity difference in the data missing area exceeds a light threshold, the exposure parameters of the oblique photography camera and the power of the fill light device are adjusted.

5. The 3D modeling processing method based on UAV oblique photography according to claim 1, characterized in that S4 include: S41, determining an initial facet set for semantic segmentation based on a normal vector consistency index of the mixed three-dimensional point cloud data; S42, generating a topological connection relationship graph according to the geometric continuity features of the initial semantic segmentation facets; S43, determining implicit surface generation parameters of the fused Poisson surface reconstruction equation according to the edge weight distribution of the topological connection relationship graph; S44, determining a multi-resolution mesh model with adaptive subdivision capability based on the normal field optimization result of the implicit surface generation parameters; S45 , determining a multi-level detail three-dimensional mesh model optimized by a feature-preserving filtering algorithm according to a Hausdorff distance detection result of the multi-resolution mesh model.

6. The 3D modeling method based on drone oblique photography according to claim 5, characterized in that: If the local geometric accuracy of the multi-resolution grid model is lower than the accuracy threshold, performing surface fitting optimization by a moving least squares method; If the crack length ratio is detected to exceed the crack threshold during the isosurface extraction process of the implicit surface generation parameters, topology repair is performed using a non-rigid deformation compensation algorithm; If the noise suppression strength of the feature-preserving filtering algorithm exceeds a suppression threshold, the geometric detail retention rate and the model smoothness index are automatically balanced.

7. The 3D modeling method based on UAV oblique photography according to claim 6, characterized in that: The surface fitting optimization by the moving least squares method includes: If the angle between the local geometric accuracy deviation direction and the main curvature direction exceeds the angle threshold, the mode is switched to anisotropic fitting weight distribution; If the number of surface fitting iterations exceeds the threshold and the residuals do not converge, the parameters are adaptively adjusted through particle swarm optimization; If the Gaussian curvature mutation value of the fitted surface exceeds the mutation threshold, control vertices are inserted to reconstruct the parametric expression of the NURBS surface.

8. The 3D modeling processing method based on UAV oblique photography according to claim 1, characterized in that S5 include: S51, determining a multi-view texture mapping weight matrix based on projection deformation correction according to UV unfolding parameters of the initial three-dimensional mesh model; S52, determining a texture fusion coefficient generated by an illumination estimation model according to a color consistency index of the multi-view texture mapping weight matrix; S53, determining a texture seam elimination solution based on a Markov random field according to the spatial continuity characteristics of the texture fusion coefficient; S54: Determine a dynamic LOD scheduling strategy that supports both WebGL and ARCore engines based on a visual quality evaluation result after texture seam elimination. S55 , determining to output a multi-platform compatible three-dimensional model in OSGB, GLTF and point cloud PCD formats according to the rendering performance parameters of the dynamic LOD scheduling strategy.

9. The 3D modeling method based on drone oblique photography according to claim 8, characterized in that: If the distortion rate of the multi-view texture mapping weight matrix exceeds a distortion threshold, optimizing the texture coordinates by resampling the control points; If the ambient light occlusion coefficient of the illumination estimation model is lower than the occlusion threshold, the real illumination data collected by the ambient light probe is superimposed for compensation; If the frame rate fluctuation amplitude of the dynamic LOD scheduling strategy exceeds the frame rate threshold, the level division granularity of the tile pyramid is adjusted.

10. The 3D modeling processing method based on UAV oblique photography according to claim 9, characterized in that: The optimization of texture coordinates by resampling control points includes: If the texture stretching rate after control point resampling exceeds the stretching threshold, a secondary optimization is performed using a texture relaxation algorithm based on differential coordinates; If the resolution difference of the maps after texture coordinate optimization exceeds the resolution threshold, a seamless map set is generated through multi-scale texture synthesis technology.

Images