A high-precision urban real scene three-dimensional model generation method, system and storage medium

Abstract

The application provides a high-precision urban real scene three-dimensional model generation method, system and storage medium, and the method comprises the following steps: arranging image control points in a target area and measuring; obtaining oblique image data of the target area by using a UAV; obtaining original point cloud data of the target area by using a laser scanner; performing aerial triangulation calculation on the oblique image data in combination with ground control point data to obtain air triangulation encryption data; performing registration on the original point cloud data in combination with the ground control point data and then performing preprocessing to obtain laser point cloud data; fusing and reconstructing the air triangulation encryption data and the laser point cloud data to generate an initial three-dimensional model; and performing fine processing on the initial three-dimensional model to generate a high-precision three-dimensional model; the application can automatically repair water surface cavities and realize efficient and intelligent color uniformity when constructing a real scene three-dimensional model, thereby effectively improving the geometric precision and visual quality of the model and meeting the engineering application requirements.

Description

Technical Field

[0001] This invention relates to the field of surveying and mapping technology, and more specifically, to a method, system, and storage medium for generating high-precision 3D models of urban real-world scenes. Background Technology

[0002] With the rapid development of smart cities, digital twins, and other fields, 3D models of urban real-world scenes based on oblique photogrammetry have become an important carrier for spatial information analysis. However, existing 3D model construction schemes still have significant technical bottlenecks, especially in water surface modeling and texture color balancing, with specific problems as follows:

[0003] 1) Hollow Area Problem in Water Surface: Existing technologies mainly use multi-view images obtained from aerial photography to generate point clouds through dense matching, and then construct triangular mesh models. Due to the high reflectivity, low texture features, and dynamic fluctuations of the water surface, the matching algorithm cannot generate effective point cloud data, resulting in large-area holes in the water surface area of ​​the model. Traditional solutions usually use manual repair or introduce digital elevation models (DEMs) to fill the water surface, but there are still obvious defects. On the one hand, manual repair is inefficient and it is difficult to guarantee the authenticity of the water surface geometry. On the other hand, the water surface generated by the DEM filling method is a plane or a simple curved surface, which cannot reflect the fluctuation details of real water bodies (such as ripples and reflections), resulting in visual distortion of the model and affecting its application accuracy in scenarios such as flood simulation and ship navigation.

[0004] 2) Low efficiency of texture color balancing: City-level real-scene models need to integrate massive amounts of images collected under various time periods and lighting conditions. Existing color balancing techniques mainly rely on the following two methods: one is manual color adjustment, where operators manually adjust color levels / curves based on experience, which is time-consuming and highly subjective, making it difficult to guarantee color consistency across a large range of models; the other is globally automated color balancing algorithms, such as histogram matching and color transfer, which reduce manual intervention but are prone to producing color block breaks or overly smoothing phenomena in areas with large lighting differences (such as the sunny / shaded sides of buildings); in addition, the computational complexity of color balancing algorithms increases exponentially when processing ultra-large-scale images, and color balancing of tens of thousands of images can take tens of hours, severely restricting modeling efficiency.

[0005] Therefore, the aforementioned problems make it difficult for existing 3D models of urban real-world scenes to meet the stringent requirements of geometric integrity and visual realism for applications such as smart city management and high-precision maps for autonomous driving. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies in constructing realistic 3D models, such as the difficulty in repairing voids in water surfaces and the low efficiency of texture color balancing, this invention provides a high-precision method, system, and storage medium for generating realistic 3D urban models. This method can automatically repair voids in water surfaces and achieve efficient and intelligent color balancing during the construction of realistic 3D models, thereby effectively improving the geometric accuracy and visual quality of the models and meeting the needs of engineering applications.

[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0008] A method for generating high-precision 3D models of urban real scenes includes the following steps:

[0009] S1: Deploy and measure ground control points in the target area to obtain ground control point data; acquire oblique image data of the target area using a drone; acquire raw point cloud data of the target area using a laser scanner;

[0010] S2: Combine the ground control point data with aerial triangulation to obtain aerial triangulation densified data; combine the ground control point data with the original point cloud data for registration and preprocessing to obtain laser point cloud data;

[0011] S3: The aerial triangulation encrypted data and the laser point cloud data are fused and reconstructed to generate an initial three-dimensional model;

[0012] S4: Refine the initial 3D model to generate a high-precision 3D model; the refinement process includes at least one of the following: detecting and repairing water surface voids in the initial 3D model and efficient color balancing operation.

[0013] Preferably, in step S1, inclined flight design is performed in the target area to determine flight altitude, overlap, flight path and control points, and to obtain design information;

[0014] Based on the design information, an unmanned aerial vehicle (UAV) is used to perform oblique photography to obtain aerial image photos;

[0015] POS data is extracted from the aerial imagery; using the ground control point data and the POS data, geometric correction and absolute orientation are performed on the aerial imagery to obtain the oblique imagery data.

[0016] Preferably, in step S2, the original point cloud data is registered using the ground control point data.

[0017] The registered raw point cloud data is sequentially denoised, segmented, and simplified to obtain the laser point cloud data.

[0018] Preferably, in step S4, detecting and repairing water surface voids in the initial three-dimensional model includes:

[0019] S401: In the initial three-dimensional model, the water surface cavity boundary is automatically identified and delineated using a preset algorithm to preliminarily determine the range of the water surface cavity; the preset algorithm includes at least one of the following: threshold segmentation algorithm, morphological analysis algorithm, Canny edge detection algorithm, and machine learning algorithm;

[0020] S402: Perform topology optimization on the initially determined water surface cavity range, search and delete edge noise and isolated points to obtain the optimized water surface cavity range;

[0021] S403: Based on the size of the water surface cavity area, the optimized water surface cavity area is divided into small cavities and large cavities; for small cavities, Delaunay triangulation or Poisson surface reconstruction algorithm is used for surface reconstruction; for large cavities, tilted Gaussian splashing technology is used to construct a continuous surface, thereby maintaining the continuity of water flow while filling the cavity; the water surface cavity after surface reconstruction is obtained.

[0022] S404: Perform texture repair and optimization on the water surface voids after the surface reconstruction to complete the repair of the water surface voids in the initial three-dimensional model.

[0023] Preferably, in step S404, texture repair and optimization are performed on the water surface voids after the surface reconstruction, including:

[0024] Texture restoration of water surface voids after surface reconstruction is performed using the Criminisi image inpainting algorithm or the Poisson fusion algorithm. For voids in dynamic water surfaces, spatiotemporal data is further constructed by combining multi-frame tilted image data, and water surface movement is tracked using optical flow to achieve continuous repair of voids in dynamic water surfaces. For water surface void areas with weak texture, texture restoration is further performed by simulating the reflection and refraction effects of the water surface using a generative adversarial network.

[0025] During the texture restoration process, surface tension and gravity constraints are applied to the water surface based on a fluid physics engine to simulate the natural fluctuations of the water surface;

[0026] The texture optimization includes:

[0027] For areas that have undergone texture restoration, shadows and highlights are calculated using ray tracing or ambient occlusion algorithms to ensure consistency with the lighting of the surrounding environment. At the same time, collisions between the curved surfaces of the corresponding areas and surrounding objects are detected, and the surface shapes are fine-tuned to ensure topological consistency.

[0028] Preferably, in step S4, the efficient color-balancing operation includes:

[0029] S411: Convert the initial 3D model from the original RGB color space to the CIELAB color space to obtain the 3D model after color conversion;

[0030] S412: The color-converted 3D model is divided into multiple regions with similar color and texture features using a superpixel segmentation algorithm. The average color value of each segmented region in the CIELAB color space is calculated as the color feature of the corresponding region. Furthermore, the color feature differences between different regions are calculated based on Euclidean distance or histogram.

[0031] S413: Select a reference area, use the histogram matching algorithm to adjust the color distribution of other areas to be consistent with the reference area, and complete the overall color uniformity; use the K-means clustering algorithm to cluster the color features of the model after overall color uniformity, obtain several color clusters, and make targeted adjustments to the areas within each color cluster to keep the color features within the same cluster consistent, and complete the color uniformity of local areas where the color feature difference is greater than the preset threshold.

[0032] S414: The Poisson fusion algorithm is used to fuse the boundaries of regions after overall and local color homogenization, so as to achieve a smooth color transition.

[0033] Preferably, in step S413, for each color cluster, a color standard deviation threshold within the cluster is set, regions whose color features exceed the color standard deviation threshold within the cluster are detected, and color correction is performed; during the color correction process, different color parameter adjustment priorities are set according to different physical objects.

[0034] The color parameters include at least: brightness, contrast, intensity, hue, and saturation.

[0035] Preferably, after step S4, the method further includes: checking and deleting suspended objects in the high-precision 3D model to ensure that the model is clean and aesthetically pleasing.

[0036] This invention also provides a high-precision urban real-scene 3D model generation system, which applies the above-mentioned method and includes:

[0037] Data acquisition unit: used to deploy and measure ground control points in the target area to obtain ground control point data; to acquire oblique image data of the target area using a UAV; and to acquire raw point cloud data of the target area using a laser scanner.

[0038] Data processing unit: used to perform aerial triangulation calculations on the oblique image data in combination with the ground control point data to obtain aerial triangulation densified data; and to preprocess the original point cloud data after registration in combination with the ground control point data to obtain laser point cloud data.

[0039] Initial modeling unit: used to fuse and reconstruct the aerial triangulation data and the laser point cloud data to generate an initial 3D model;

[0040] Model refinement unit: used to refine the initial 3D model to generate a high-precision 3D model; the refinement process includes at least one of the following: detecting and repairing water surface voids in the initial 3D model and efficient color balancing operation.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0042] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0043] This invention provides a method, system, and storage medium for generating a high-precision 3D model of a city scene. First, ground control points are deployed and measured in the target area to obtain ground control point data. Then, oblique image data of the target area is acquired using a drone. Next, raw point cloud data of the target area is acquired using a laser scanner. Aerial triangulation calculations are performed on the oblique image data in conjunction with the ground control point data to obtain aerial triangulation data. After registering the raw point cloud data with the ground control point data, preprocessing is performed to obtain laser point cloud data. The aerial triangulation data and laser point cloud data are fused and reconstructed to generate an initial 3D model. Finally, the initial 3D model is refined to generate a high-precision 3D model. The refinement process includes at least one of the following: detecting and repairing water surface voids in the initial 3D model and efficient color balancing.

[0044] This invention introduces multiple algorithms to repair water surface voids in different scenarios, effectively improving the efficiency and accuracy of void repair. Simultaneously, based on algorithms such as the CIELAB color space and superpixel segmentation, it reduces the time required for traditional color balancing schemes from hours to minutes, significantly improving the efficiency of model color balancing. This invention can automatically repair water surface voids and achieve efficient and intelligent color balancing when constructing realistic 3D models, overcoming the two major technical bottlenecks of water surface voids and inefficient color balancing in realistic 3D modeling. This effectively improves the geometric accuracy and visual quality of the model, enabling refined realistic models to leap from "verification-level usability" to "engineering-level reliability." Attached Figure Description

[0045] Figure 1 This is a flowchart of a high-precision urban real-scene 3D model generation method provided in Example 1.

[0046] Figure 2This is an overall framework diagram of a high-precision urban real-scene 3D model generation method provided in Example 2.

[0047] Figure 3 The image shown is of the water surface cavity before repair, as provided in Example 2.

[0048] Figure 4 The image shows the repaired water surface cavity provided in Example 2.

[0049] Figure 5 This is a model image before color balancing provided in Example 2.

[0050] Figure 6 This is the model image after color balancing provided in Example 2.

[0051] Figure 7 This is a structural diagram of a high-precision urban real-scene 3D model generation system provided in Example 3. Detailed Implementation

[0052] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application.

[0053] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0054] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0055] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0056] Example 1

[0057] like Figure 1 As shown, this embodiment provides a method for generating a high-precision 3D model of a real urban scene, including the following steps:

[0058] S1: Deploy and measure ground control points in the target area to obtain ground control point data; acquire oblique image data of the target area using a drone; acquire raw point cloud data of the target area using a laser scanner;

[0059] S2: Combine the ground control point data with aerial triangulation to obtain aerial triangulation densified data; combine the ground control point data with the original point cloud data for registration and preprocessing to obtain laser point cloud data;

[0060] S3: The aerial triangulation encrypted data and the laser point cloud data are fused and reconstructed to generate an initial three-dimensional model;

[0061] S4: Refine the initial 3D model to generate a high-precision 3D model; the refinement process includes at least one of the following: detecting and repairing water surface voids in the initial 3D model and efficient color balancing operation.

[0062] In the specific implementation process, firstly, ground control points are set up and measured in the target area to obtain ground control point data; at the same time, UAVs are used to acquire oblique image data of the target area; and laser scanners are used to acquire raw point cloud data of the target area.

[0063] Next, aerial triangulation calculations were performed on the oblique image data in conjunction with ground control point data to obtain aerial triangulation densified data; and the original point cloud data was registered in conjunction with ground control point data and then preprocessed to obtain laser point cloud data.

[0064] The aerial triangulation encrypted data and laser point cloud data are then fused and reconstructed to generate an initial 3D model;

[0065] Finally, the initial 3D model is refined to generate a high-precision 3D model. In this embodiment, the refinement process includes detecting and repairing water surface voids in the initial 3D model and efficient color balancing.

[0066] This method, by fusing oblique photogrammetry data and laser point cloud data, helps to further overcome the problems of water surface voids and inefficient color uniformity in traditional methods, and improves the geometric accuracy and visual quality of the model. If only oblique photogrammetry data is used, the point cloud generation will be insufficient due to problems such as the high reflectivity of the water surface, and it will be impossible to achieve void repair and model optimization through fine processing.

[0067] This method can automatically repair water surface voids and achieve efficient and intelligent color balancing when constructing realistic 3D models, thereby effectively improving the geometric accuracy and visual quality of the models and meeting the needs of engineering applications.

[0068] Example 2

[0069] This embodiment provides a method for generating a high-precision 3D model of a real urban scene, including the following steps:

[0070] S1: Deploy and measure ground control points in the target area to obtain ground control point data; acquire oblique image data of the target area using a drone; acquire raw point cloud data of the target area using a laser scanner;

[0071] S2: Combine the ground control point data with aerial triangulation to obtain aerial triangulation densified data; combine the ground control point data with the original point cloud data for registration and preprocessing to obtain laser point cloud data;

[0072] S3: The aerial triangulation encrypted data and the laser point cloud data are fused and reconstructed to generate an initial three-dimensional model;

[0073] S4: Refine the initial 3D model to generate a high-precision 3D model; the refinement process includes at least one of: detecting and repairing water surface voids in the initial 3D model and efficient color balancing operation;

[0074] In step S1, inclined flight design is performed in the target area to determine flight altitude, overlap, flight path and control points, and to obtain design information.

[0075] Based on the design information, an unmanned aerial vehicle (UAV) is used to perform oblique photography to obtain aerial image photos;

[0076] POS data is extracted from the aerial imagery; using the ground control point data and the POS data, geometric correction and absolute orientation are performed on the aerial imagery to obtain the oblique imagery data;

[0077] In step S2, the original point cloud data is registered using the ground control point data.

[0078] The registered raw point cloud data is sequentially denoised, segmented, and simplified to obtain the laser point cloud data.

[0079] In step S4, detecting and repairing water surface voids in the initial 3D model includes:

[0080] S401: In the initial three-dimensional model, the water surface cavity boundary is automatically identified and delineated using a preset algorithm to preliminarily determine the range of the water surface cavity; the preset algorithm includes at least one of the following: threshold segmentation algorithm, morphological analysis algorithm, Canny edge detection algorithm, and machine learning algorithm;

[0081] S402: Perform topology optimization on the initially determined water surface cavity range, search and delete edge noise and isolated points to obtain the optimized water surface cavity range;

[0082] S403: Based on the size of the water surface cavity area, the optimized water surface cavity area is divided into small cavities and large cavities; for small cavities, Delaunay triangulation or Poisson surface reconstruction algorithm is used for surface reconstruction; for large cavities, tilted Gaussian splashing technology is used to construct a continuous surface, thereby maintaining the continuity of water flow while filling the cavity; the water surface cavity after surface reconstruction is obtained.

[0083] S404: Perform texture repair and optimization on the water surface voids after the surface reconstruction to complete the repair of the water surface voids in the initial three-dimensional model;

[0084] In step S404, the texture repair and optimization of the water surface voids after the surface reconstruction includes:

[0085] Texture restoration of water surface voids after surface reconstruction is performed using the Criminisi image inpainting algorithm or the Poisson fusion algorithm. For voids in dynamic water surfaces, spatiotemporal data is further constructed by combining multi-frame tilted image data, and water surface movement is tracked using optical flow to achieve continuous repair of voids in dynamic water surfaces. For water surface void areas with weak texture, texture restoration is further performed by simulating the reflection and refraction effects of the water surface using a generative adversarial network.

[0086] During the texture restoration process, surface tension and gravity constraints are applied to the water surface based on a fluid physics engine to simulate the natural fluctuations of the water surface;

[0087] The texture optimization includes:

[0088] For areas that have undergone texture restoration, shadows and highlights are calculated using ray tracing or ambient occlusion algorithms to ensure consistency with the lighting of the surrounding environment. At the same time, collisions between the curved surfaces of the corresponding areas and surrounding objects are detected, and the surface shapes are fine-tuned to ensure topological consistency.

[0089] In step S4, the efficient color balancing operation includes:

[0090] S411: Convert the initial 3D model from the original RGB color space to the CIELAB color space to obtain the 3D model after color conversion;

[0091] S412: The color-converted 3D model is divided into multiple regions with similar color and texture features using a superpixel segmentation algorithm. The average color value of each segmented region in the CIELAB color space is calculated as the color feature of the corresponding region. Furthermore, the color feature differences between different regions are calculated based on Euclidean distance or histogram.

[0092] S413: Select a reference area, use the histogram matching algorithm to adjust the color distribution of other areas to be consistent with the reference area, and complete the overall color uniformity; use the K-means clustering algorithm to cluster the color features of the model after overall color uniformity, obtain several color clusters, and make targeted adjustments to the areas within each color cluster to keep the color features within the same cluster consistent, and complete the color uniformity of local areas where the color feature difference is greater than the preset threshold.

[0093] S414: The Poisson fusion algorithm is used to fuse the boundaries of the regions after overall and local color balancing, so as to achieve a smooth color transition;

[0094] In step S413, for each color cluster, a color standard deviation threshold within the cluster is set, regions whose color features exceed the color standard deviation threshold within the cluster are detected, and color correction is performed; during the color correction process, different color parameter adjustment priorities are set according to different physical objects.

[0095] The color parameters include at least: brightness, contrast, hue, and saturation;

[0096] After step S4, the method further includes: checking and deleting suspended objects in the high-precision 3D model to ensure that the model is clean and aesthetically pleasing.

[0097] In the specific implementation process, such as Figure 2 As shown, firstly, ground control points are deployed and measured in the target area to obtain ground control point data; simultaneously, oblique image data of the target area is acquired using a drone; and raw point cloud data of the target area is acquired using a laser scanner.

[0098] Specifically, select an area to be surveyed (in this example, an urban area), and according to the surveying accuracy requirements, evenly distribute a certain number of ground control points within the area; these control points should have clear markings, such as signs or ground markings, so that drones and laser scanners can accurately identify and locate them; collect the precise coordinates (including longitude, latitude, and elevation) of each control point, for example, using a high-precision GNSS receiver or other measuring equipment, and these coordinate data will serve as the reference for subsequent processing;

[0099] For oblique photogrammetry data, oblique flight design is first performed within the target city area to determine flight altitude, overlap, flight path, and ground control points (GCPs) to obtain design information. Based on the design information, oblique photogrammetry is performed using a UAV to acquire aerial images. In this embodiment, according to the designed flight plan, the UAV can be equipped with a multi-lens oblique camera for flight photography. The UAV flies along a preset flight path, while the multi-lens camera takes photos from multiple angles (such as front, back, left, right, and down) to obtain rich three-dimensional information. Then, POS (Position and Orientation System) data is extracted from the aerial images, including the shooting position (longitude, latitude, altitude) and attitude (pitch angle, roll angle, yaw angle) information of each aerial image. Using ground control point data, professional photogrammetry software (such as Pix4Dmapper, Agisoft Metashape, etc.) is used in conjunction with the POS data to perform geometric correction and absolute orientation of the aerial images, completing the acquisition of oblique image data, thereby eliminating errors such as camera distortion and atmospheric effects, and accurately positioning the images to the actual geographic coordinate system.

[0100] Simultaneously, laser scanning is performed within the target area to acquire a large number of three-dimensional coordinate points on the object's surface, forming point cloud data. This data will contain detailed information such as the object's shape, size, and location. A suitable stationary or mobile laser scanner is selected to ensure high precision and efficiency.

[0101] Next, aerial triangulation (AT) calculations were performed on the oblique image data in combination with ground control point data to obtain aerial triangulation densified data. These data constituted the three-dimensional point cloud data of the target area, reflecting the surface morphology and building structure. At the same time, the data was combined with ground control point data, so that the aerial triangulation densified point cloud data and the laser point cloud data were in the same coordinate system.

[0102] Simultaneously, the original point cloud data is registered with ground control point data. Since the point cloud data obtained from each scan is based on its own coordinate system when scanning at multiple locations or angles, it is necessary to combine the ground control point data and transform these point cloud data into a unified coordinate system through registration to achieve seamless data stitching. After that, the registered original point cloud data is segmented, denoised, and simplified in sequence to obtain laser point cloud data, thereby removing redundant information, improving the speed and efficiency of subsequent processing, and preserving the geometric features and details of the data.

[0103] The aerial triangulation encrypted data and laser point cloud data are then fused and reconstructed to generate an initial 3D model;

[0104] Finally, the initial 3D model is refined to generate a high-precision 3D model. In this embodiment, the refinement process includes detecting and repairing water surface voids in the initial 3D model and efficient color balancing.

[0105] Because urban areas contain a large amount of water surface and feature points are not obvious, it is difficult to extract feature points, resulting in holes in the reconstructed 3D model. Holes in water surfaces in real-world 3D measurements are mostly caused by water reflection, occlusion, and measurement blind spots. This embodiment provides an intelligent algorithm framework that combines data-driven approaches with physical models, focusing on data acquisition optimization, hole-filling logic, and result verification. In this embodiment, detecting and repairing water surface holes in the initial 3D model includes the following steps:

[0106] S401: In the initial 3D model, the water surface cavity boundary is automatically identified and delineated using a preset algorithm to preliminarily determine the range of the water surface cavity; the preset algorithm includes at least one of the following: threshold segmentation algorithm, morphological analysis algorithm, Canny edge detection algorithm and machine learning algorithm;

[0107] Specifically, the first step is to define the extent of the void. Based on a 3D point cloud or mesh model, algorithms such as threshold segmentation and morphological analysis are used to automatically identify or delineate the boundaries of the water surface voids to determine the void extent. Note that islands should be delineated or excavated, and the spatial coordinates of the void area and its topological relationship with the original data are recorded. For complex scenes, cross-validation can be performed by combining multi-source data (such as laser point clouds and oblique photogrammetry) to ensure the accuracy of boundary extraction. Alternatively, content-aware algorithms can be used to automatically distinguish the water surface from surrounding features (such as vegetation and buildings) to achieve precise location of void areas.

[0108] S402: Perform topology optimization on the initially determined water surface cavity range, search and delete edge noise and isolated points to obtain the optimized water surface cavity range;

[0109] Specifically, topology repair is performed on the extracted boundaries, and abnormal edge points (such as isolated points used by only one edge) are deleted to ensure that the boundaries form closed loops. Isolated points are searched and deleted until all edge points are connected by at least two edges to avoid self-intersection or structural anomalies during subsequent filling. At the same time, the integrity and accuracy of the data around the holes are checked, and noise and outliers are removed to ensure that the data used for repair is true and reliable.

[0110] S403: Based on the size of the water surface cavity area, the optimized water surface cavity area is divided into small cavities and large cavities; for small cavities, Delaunay triangulation or Poisson surface reconstruction algorithm is used for surface reconstruction; for large cavities, tilted Gaussian splashing technology is used to construct a continuous surface, thereby maintaining the continuity of water flow while filling the cavity; the water surface cavity after surface reconstruction is obtained.

[0111] Specifically, this embodiment uses surface fitting as a cavity filling strategy, treating the water surface as a continuous surface and using a mathematical model to fit the cavity region, generating a 3D model that conforms to the overall shape of the water surface. For small cavities, Delaunay triangulation or minimum area surface algorithms (such as Poisson reconstruction) are used to generate smooth triangular meshes based on the cavity boundary points. For large water areas, fluid dynamics models (such as shallow water wave equations) are combined to simulate water flow, filling cavities while maintaining water flow continuity. During implementation, feature lines or contours around the cavity are first extracted, then a suitable surface model (selected according to the size of the cavity) is selected for fitting, and parameters are adjusted to ensure a smooth transition between the surface and the surrounding data. The fitted surface is then merged with the original model to complete the geometric repair.

[0112] For water surfaces with complex textures or details (such as ripples and waves), multi-resolution analysis (such as wavelet transform) can be used to separate high-frequency details from low-frequency structures. Generally, the low-frequency part of the void area is filled by surface fitting, while the high-frequency details are transferred through neighborhood texture migration. When repairing small voids, the slope and curvature of the surrounding water surface can be automatically identified by a program or script, and a surface can be constructed by linear interpolation or polynomial fitting. When dealing with large water bodies, the Oblique Photo Grammetry Gaussian Splatting (OPGS) method can be used to achieve natural water replenishment without boundary constraints.

[0113] S404: Perform texture repair and optimization on the water surface voids after surface reconstruction to complete the repair of water surface voids in the initial 3D model;

[0114] Specifically, texture restoration includes: extending the texture of adjacent areas to the hole area through image deformation and texture mapping algorithms, adjusting parameters such as brightness and contrast to ensure a natural texture transition; in this embodiment, sample-based image restoration techniques (such as the Criminisi algorithm) can be used to copy sample blocks from texture-rich areas to fill the hole area and restore texture details; alternatively, texture features around the hole can be extracted from the original image and mapped to the repaired area through a seamless stitching algorithm (such as Poisson fusion);

[0115] During the repair process, there may be some special water surfaces that require further targeted optimization. For example, for holes in dynamic water surfaces (such as waves), spatiotemporal data can be constructed by combining multi-frame tilted image data, and the water surface movement can be tracked using optical flow to achieve continuous repair of holes in dynamic water surfaces. For weak texture areas (such as mirror-like water surfaces), a texture synthesis script based on deep learning can be developed to simulate the reflection and refraction effects of the water surface using generative adversarial networks (GANs).

[0116] Meanwhile, during the texture restoration process, a fluid physics engine is introduced. Based on the fluid physics engine, surface tension and gravity constraints are applied to the water surface to simulate the natural fluctuations of the water surface. For example, when there is water flow around the cavity, the physics engine will adjust the shape of the filled surface according to the flow rate and direction to avoid the appearance of abrupt static planes.

[0117] In this embodiment, texture optimization includes:

[0118] For areas that have undergone texture restoration, shadows and highlights are calculated using ray tracing or ambient occlusion algorithms to ensure consistency with the lighting of the surrounding environment, thereby achieving lighting and shadow matching; for example, when dealing with holes in the water surface under the shade of trees, the projection effect of leaves is simulated.

[0119] Simultaneously, it detects collisions between the curved surfaces in the corresponding areas and surrounding objects (such as bridge piers and ships), and fine-tunes the surface shape to ensure topological consistency, thus achieving collision detection and topology maintenance. For example, when dealing with water surface voids under bridges, it identifies the bridge structure manually or automatically and generates a curved surface that fits the bottom of the bridge to prevent the model from penetrating.

[0120] After the water surface cavities are repaired, a quality control mechanism can be further introduced. On the one hand, geometric accuracy verification is carried out by using control points, measured data, or high-precision auxiliary data, or point cloud registration and error analysis, to compare the error between the repaired water surface and the actual shape, ensuring that the plane and elevation accuracy meet the measurement specifications. On the other hand, visual effect evaluation is carried out by using multi-angle visualization inspection, or by using perceptual hashing algorithms (such as pHash) or structural similarity index (SSIM) to quantitatively evaluate the fusion effect of texture and lighting, evaluate texture continuity and geometric smoothness, and make local adjustments and optimizations for areas with obvious defects.

[0121] like Figure 3 and 4 The images show water surface cavities before and after repair. The water surface cavity repair scheme provided in this embodiment is based on a combination of data-driven and physical models, and achieves efficient repair of water surface cavities through the synergy of multiple technologies.

[0122] It is worth mentioning that the water surface is a typical "difficult area" in modeling. Due to the strong reflection and wave interference of the water surface during drone photography, "holes" (areas with missing data) are easily formed. Traditional restoration methods have obvious limitations, such as restoring only from the perspective of "mathematical fitting" without considering the physical properties of the water surface. When there are "interference elements" such as islands on the water surface, traditional methods are prone to mistakenly bringing surrounding non-water surface features into the water surface restoration, resulting in distorted results. For large-area holes, the restoration results are prone to "over-smoothing" or "pseudo-texture". The ripples and waves of natural water surfaces are dynamically changing, and traditional static restoration methods cannot simulate the dynamic texture of real water surfaces. The restored water surface is prone to a "rigid" effect (such as looking like a "glass surface" rather than a real water body).

[0123] This method incorporates various factors, including the physical constraints of the water surface, and obtains the precise elevation of the shoreline using laser point clouds. Using this as a boundary condition, a continuous curved surface model of the "water surface-shoreline" is constructed, avoiding a "discontinuity" between the repaired voids and the shoreline. Secondly, this method can target dynamic features such as ripples. It can train deep learning to understand the ripple distribution patterns of real water surfaces. For example, for void areas, it generates ripple textures with similar patterns based on the direction and density of neighboring ripples, enhancing visual realism. For water surfaces containing interfering elements such as islands, it first distinguishes between "pure water surface areas" and "areas surrounding obstacles" through semantic segmentation, and then performs targeted repairs, thereby improving the realism of void repairs.

[0124] Besides the voids in the water surface, the 3D model also suffers from color difference issues. The colors in the aerial triangulation are uneven or distorted due to the influence of sunny or rainy weather and different lenses, requiring adjustment to the correct colors using software. Considering that the workload of color balancing the original image using Photoshop is too large, this embodiment uses MeshMaster in conjunction with DasViewer to complete the color balancing of the model. That is, first, the color adjustment function is used in the model browser to balance the color of the model display, and then the color balancing adjustment scheme is exported to MeshMaster. MeshMaster is then used to perform overall color balancing of the model in batches. This process of working backward from the result to find the color balancing scheme ensures that the overall effect of the model is more harmonious.

[0125] For color scheme generation, this embodiment transforms the problem of uneven color in the 3D real-world model into the analysis and unified adjustment of different color data; through color space conversion, color elements such as brightness, hue and saturation are separated, feature extraction technology is used to identify color differences in different areas, and then a specific algorithm is used to achieve color matching and adjustment, so that the color of the entire model transitions naturally and is coordinated and unified; a combination of automated processing and manual fine adjustment can be used to meet the needs of different scenarios;

[0126] In this embodiment, the efficient color-balancing operation includes:

[0127] S411: Import the initial 3D model into professional modeling software (such as ContextCapture, Metashape) for format conversion to ensure data integrity; then convert the original RGB color space to the more suitable CIELAB color space for processing.

[0128] The CIELAB color space divides color into three independent channels: brightness (L), green-red axis (a), and blue-yellow axis (b), making it easy to adjust brightness, hue, and saturation separately. In Python, the cvtColor function from the OpenCV library can be used to convert RGB to CIELAB. (Partial code example follows.)

[0129] import cv2;

[0130] import numpy as np;

[0131] image = cv2.imread('input_image.jpg');

[0132] lab_image = cv2.cvtColor(image, cv2.COLOR_BGR2LAB);

[0133] In addition, in this step, principal component analysis (PCA) can be used to reduce the dimensionality of color distribution and identify abnormal color patch clusters caused by weather / lens effects, such as cool tones in rainy areas and uneven brightness in backlit areas.

[0134] S412: This algorithm uses superpixel segmentation to divide the color-converted 3D model into multiple regions with similar color and texture features. In Python, the skimage.segmentation library can be used to implement SLIC segmentation. Partial code example:

[0135] from skimage.segmentation import slic;

[0136] from skimage.io import imread;

[0137] import matplotlib.pyplot as plt;

[0138] image = imread('input_image.jpg');

[0139] segments = slic(image, n_segments=100, compactness=10);

[0140] plt.imshow(segments);

[0141] plt.show();

[0142] Next, color feature extraction is performed, and the average color value of each segmented region in the CIELAB color space (mean values ​​of L, a, and b in the CIELAB color space) is calculated as the color feature of the corresponding region; and the color feature differences of different regions are further calculated based on Euclidean distance or histogram.

[0143] S413: Select a reference area (usually a representative area with relatively uniform lighting and color), and use a histogram matching algorithm to adjust the color distribution of other areas to match the reference area, thus achieving overall color uniformity. In Python, the `skimage.exposure.match_histograms` function can be used. (Partial code example follows.)

[0144] fromskimage.exposureimportmatch_histograms;

[0145] import cv2;

[0146] source_image=cv2.imread('source_image.jpg');

[0147] reference_image=cv2.imread('reference_image.jpg');

[0148] matched_image=match_histograms(source_image,reference_image,multichannel=True);

[0149] For local areas with significant color differences, a clustering-based color adjustment algorithm is used. First, K-means clustering is used to cluster the color features of the entire model. Based on the clustering results, targeted adjustments are made to the regions within each cluster to make the colors within the same cluster more consistent. In Python, K-means clustering can be implemented using sklearn.cluster.KMeans. (Partial code example follows.)

[0150] from sklearn.cluster import Kmeans;

[0151] import numpy as np;

[0152] # Assume that lab_features are the CIELAB color features of all regions;

[0153] kmeans = KMeans(n_clusters=5, random_state=0).fit(lab_features);

[0154] labels = kmeans.labels_;

[0155] In this embodiment, for each color cluster, a standard deviation threshold for color within the cluster is set, and regions where the color features exceed the standard deviation threshold for color within the cluster are detected and color correction is performed.

[0156] During color correction, different color parameters (brightness, contrast, hue, and saturation) are adjusted with different priorities based on different physical objects (such as buildings, water bodies, and vegetation). For example, saturation is adjusted first in water bodies, while color consistency is emphasized in building areas. The adjustment intensity of each area is controlled by weighting coefficients.

[0157] S414: The Poisson blending algorithm is used to blend the boundaries of the regions after overall and local color homogenization, avoiding obvious splicing marks and achieving a smooth color transition. In Python, this can be achieved using the `cv2.seamlessClone` function from the `opencv-python` library. (Partial code example follows.)

[0158] import cv2;

[0159] import numpyasnp;

[0160] src=cv2.imread('src.jpg');

[0161] dst = cv2.imread('dst.jpg');

[0162] mask=np.zeros(src.shape[:2],dtype=np.uint8);

[0163] center=(dst.shape[1] / / 2,dst.shape[0] / / 2);

[0164] output=cv2.seamlessClone(src,dst,mask,center,cv2.NORMAL_CLONE);

[0165] The above steps generate standardized scheme files (such as JSON / XML format), containing information such as adjustment parameter values, applicable scenario labels, and adjustment area ranges. Then, GridMaster is used to perform batch color balancing operations on the same scene. GridMaster possesses powerful cluster processing and transformation capabilities, supporting cluster computation across multiple computers or services. This cluster computation mode significantly improves processing efficiency when facing large-scale real-scene 3D data processing tasks. For example, in tasks such as data aggregation, format conversion, coordinate transformation, and tile re-division, multiple computers working collaboratively can quickly and efficiently transform and integrate data that would otherwise require a significant amount of time to process, meeting the demand for rapid processing of large-scale data in practical applications. This advantage is particularly evident in the increasing demand for data aggregation of real-scene 3D construction results during the nationwide urbanization and smart city construction process.

[0166] After completing the color balancing operation, an effect evaluation and optimization mechanism can be introduced. On the one hand, a subjective evaluation can be conducted, observing the adjusted 3D reality model from different perspectives to check whether the colors are natural and harmonious, and whether there are obvious color unevenness or distortion problems. On the other hand, an objective evaluation can be conducted, calculating the mean square error (MSE) of the model's color features before and after adjustment, using the following formula:

[0167]

[0168] in, x i and y i These are the color feature values ​​of the corresponding areas before and after adjustment; n Total number of regions;

[0169] Based on the evaluation results, unsatisfactory areas can be further adjusted and optimized.

[0170] like Figure 5 and 6 The images shown are model images before and after color balancing. It can be seen that the color balancing scheme of this method ensures that the overall effect of the model is more harmonious.

[0171] In addition, after the above-mentioned repair of water surface voids and color balancing of the model, further inspection and removal of suspended objects in the high-precision 3D model are carried out to ensure that the model is clean and beautiful. Due to image matching, obstruction by surrounding buildings, or the presence of attachments, 3D debris (suspended objects) may form in the air or underground. At this time, switch to free view mode to view the suspended objects in the model, select them in batches and delete them. For particularly small debris or debris connected to buildings, it is necessary to adjust different perspectives to observe, select them locally and delete them. Removing suspended objects can make the model more beautiful.

[0172] The final model can be further output as DSM and DOM data for the entire survey area. The Digital Orthophoto Map (DOM) generates distortion-free surface images by orthorectifying the image data, which are used for further analysis and display. The Digital Surface Model (DSM) is a surface elevation model generated based on the image data, which can realistically reflect the topographic relief of Conghua District. These data will provide basic topographic information for disaster analysis and early warning.

[0173] In this embodiment, the addition of LiDAR point clouds significantly compensates for the inherent shortcomings of oblique photogrammetry. Through multi-source data fusion of "active measurement + passive imaging," the accuracy, detail integrity, and scene adaptability of the 3D model are improved. The core advantages of adding LiDAR point cloud data include: 1) Improved modeling accuracy, especially enhanced elevation and geometric accuracy; the elevation accuracy of oblique photogrammetry is easily affected by image matching errors, terrain undulations, and camera distortion, while the elevation accuracy of LiDAR point clouds can typically reach 5~10cm, providing a precise elevation benchmark for the model and correcting elevation deviations in oblique photogrammetry; simultaneously, since the 3D coordinates (X,Y,Z) of LiDAR point clouds are directly obtained through physical ranging, there is no cumulative error from image matching, which can constrain the oblique photogrammetry density. 1) Set matching results (especially in areas with weak texture, such as the water surface encountered in the project), reducing model "drift" or "holes"; 2) Enhance the detail integrity of complex scenes; oblique photography is prone to matching failure in areas such as water surfaces due to the lack of texture features. By adding laser point cloud, since it is not affected by texture, it can accurately capture the geometry of these areas, ensuring the continuity and integrity of the model; 3) Laser point cloud data is not affected by light and illumination intensity. When fused with oblique photography data, it reduces the void rate and color difference of the water surface; 4) Laser point cloud data has high point cloud density, which can improve the detail of the 3D model. Especially for the complex buildings, fusing point cloud data with oblique photography data can improve the fineness of the 3D building model and reduce the scraggly appearance and blind spots.

[0174] In summary, this method can automatically repair water surface voids and achieve efficient and intelligent color balancing when constructing realistic 3D models, thereby effectively improving the geometric accuracy and visual quality of the models and meeting the needs of engineering applications.

[0175] Example 3

[0176] like Figure 7 As shown, this embodiment provides a high-precision urban real-scene 3D model generation system, which applies the method described in embodiment 1 or 2, including:

[0177] Data acquisition unit 301: used to set up image control points in the target area and perform measurements to obtain ground control point data; use a UAV to acquire oblique image data of the target area; and use a laser scanner to acquire raw point cloud data of the target area;

[0178] Data processing unit 302: used to perform aerial triangulation calculation on the oblique image data in combination with the ground control point data to obtain aerial triangulation densified data; and to preprocess the original point cloud data after registration in combination with the ground control point data to obtain laser point cloud data;

[0179] Initial modeling unit 303: used to fuse and reconstruct the aerial triangulation data and the laser point cloud data to generate an initial three-dimensional model;

[0180] Model refinement unit 304: used to refine the initial three-dimensional model to generate a high-precision three-dimensional model; the refinement process includes at least one of: detecting and repairing water surface voids in the initial three-dimensional model and efficient color balancing operation.

[0181] In the specific implementation process, the data acquisition unit 301 first sets up image control points in the target area and performs measurements to obtain ground control point data; at the same time, it uses a drone to acquire oblique image data of the target area; and it uses a laser scanner to acquire raw point cloud data of the target area.

[0182] Next, the data processing unit 302 combines the ground control point data to perform aerial triangulation calculations on the oblique image data to obtain aerial triangulation densified data; and after registering the original point cloud data with the ground control point data, it performs preprocessing to obtain laser point cloud data.

[0183] Then, the initial modeling unit 303 fuses and reconstructs the aerial triangulation encrypted data and the laser point cloud data to generate the initial 3D model;

[0184] Finally, the initial modeling unit 303 performs fine-tuning on the initial 3D model to generate a high-precision 3D model. In this embodiment, the fine-tuning includes detecting and repairing water surface voids in the initial 3D model and performing efficient color balancing.

[0185] This system can automatically repair water surface voids and achieve efficient and intelligent color balancing when constructing realistic 3D models, thereby effectively improving the geometric accuracy and visual quality of the models and meeting the needs of engineering applications.

[0186] The same or similar labels correspond to the same or similar parts;

[0187] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this application.

[0188] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for generating a high-precision 3D model of a real urban scene, characterized in that, Includes the following steps: S1: Deploy ground control points in the target area and perform measurements to obtain ground control point data; acquire oblique image data of the target area using a drone; acquire raw point cloud data of the target area using a laser scanner; S2: Combine the ground control point data with aerial triangulation to obtain aerial triangulation densified data; combine the ground control point data with the original point cloud data for registration and preprocessing to obtain laser point cloud data; S3: The aerial triangulation encrypted data and the laser point cloud data are fused and reconstructed to generate an initial three-dimensional model; S4: Refine the initial 3D model to generate a high-precision 3D model; the refinement process includes at least one of: detecting and repairing water surface voids in the initial 3D model and efficient color balancing operation; The detection and repair of water surface voids in the initial 3D model includes: S401: In the initial three-dimensional model, the water surface cavity boundary is automatically identified and delineated using a preset algorithm to preliminarily determine the range of the water surface cavity; the preset algorithm includes at least one of the following: threshold segmentation algorithm, morphological analysis algorithm, Canny edge detection algorithm and machine learning algorithm; S402: Perform topology optimization on the initially determined water surface cavity range, search and delete edge noise and isolated points to obtain the optimized water surface cavity range; S403: Based on the size of the water surface cavity area, the optimized water surface cavity area is divided into small cavities and large cavities; for small cavities, Delaunay triangulation or Poisson surface reconstruction algorithm is used for surface reconstruction; for large cavities, tilted Gaussian splashing technology is used to construct a continuous surface, thereby maintaining the continuity of water flow while filling the cavity; the water surface cavity after surface reconstruction is obtained. S404: Perform texture repair and optimization on the water surface voids after the surface reconstruction to complete the repair of the water surface voids in the initial three-dimensional model; In step S404, texture repair and optimization are performed on the water surface voids after the surface reconstruction, including: Texture restoration of water surface voids after surface reconstruction is performed using the Criminisi image inpainting algorithm or the Poisson fusion algorithm. For voids in dynamic water surfaces, spatiotemporal data is further constructed by combining multi-frame tilted image data, and water surface movement is tracked using optical flow to achieve continuous repair of voids in dynamic water surfaces. For water surface void areas with weak texture, texture restoration is further performed by simulating the reflection and refraction effects of the water surface using a generative adversarial network. During the texture restoration process, surface tension and gravity constraints are applied to the water surface based on a fluid physics engine to simulate the natural fluctuations of the water surface; The texture optimization includes: For areas that have undergone texture restoration, shadows and highlights are calculated using ray tracing or ambient occlusion algorithms to ensure consistency with the lighting of the surrounding environment. At the same time, collisions between the curved surfaces of the corresponding areas and surrounding objects are detected, and the surface shapes are fine-tuned to ensure topological consistency. The efficient color balancing operation includes: S411: Convert the initial 3D model from the original RGB color space to the CIELAB color space to obtain the 3D model after color conversion; S412: The color-converted 3D model is divided into multiple regions with similar color and texture features using a superpixel segmentation algorithm. The average color value of each segmented region in the CIELAB color space is calculated as the color feature of the corresponding region. Furthermore, the color feature differences between different regions are calculated based on Euclidean distance or histogram. S413: Select a reference area, use the histogram matching algorithm to adjust the color distribution of other areas to be consistent with the reference area, and complete the overall color uniformity; use the K-means clustering algorithm to cluster the color features of the model after overall color uniformity, obtain several color clusters, and make targeted adjustments to the areas within each color cluster to keep the color features within the same cluster consistent, and complete the color uniformity of local areas where the color feature difference is greater than the preset threshold. S414: The Poisson fusion algorithm is used to fuse the boundaries of the regions after overall and local color balancing, so as to achieve a smooth color transition; In step S413, for each color cluster, a color standard deviation threshold within the cluster is set, regions whose color features exceed the color standard deviation threshold within the cluster are detected, and color correction is performed; during the color correction process, different color parameter adjustment priorities are set according to different physical objects. The color parameters include at least: brightness, contrast, intensity, hue, and saturation.

2. The method for generating a high-precision 3D model of a real urban scene according to claim 1, characterized in that, In step S1, inclined flight design is performed in the target area to determine flight altitude, overlap, flight path and control points, and to obtain design information. Based on the design information, an unmanned aerial vehicle (UAV) is used to perform oblique photography to obtain aerial image photos; POS data is extracted from the aerial imagery; using the ground control point data and the POS data, geometric correction and absolute orientation are performed on the aerial imagery to obtain the oblique imagery data.

3. The method for generating a high-precision 3D model of a real urban scene according to claim 1, characterized in that, In step S2, the original point cloud data is registered using the ground control point data. The registered raw point cloud data is sequentially denoised, segmented, and simplified to obtain the laser point cloud data.

4. A method for generating a high-precision 3D model of a city scene according to any one of claims 1 to 3, characterized in that, After step S4, the method further includes: checking and deleting suspended objects in the high-precision 3D model to ensure that the model is clean and aesthetically pleasing.

5. A high-precision urban real-scene 3D model generation system, using the method described in any one of claims 1 to 4, characterized in that, include: Data acquisition unit: used to deploy and measure ground control points in the target area to obtain ground control point data; to acquire oblique image data of the target area using a UAV; and to acquire raw point cloud data of the target area using a laser scanner. Data processing unit: used to perform aerial triangulation calculations on the oblique image data in combination with the ground control point data to obtain aerial triangulation densified data; and to preprocess the original point cloud data after registration in combination with the ground control point data to obtain laser point cloud data. Initial modeling unit: used to fuse and reconstruct the aerial triangulation data and the laser point cloud data to generate an initial 3D model; Model refinement unit: used to refine the initial 3D model to generate a high-precision 3D model; the refinement process includes at least one of the following: detecting and repairing water surface voids in the initial 3D model and efficient color balancing operation.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 4.

Images