Air-ground integrated three-dimensional scene automatic modeling method, system and device and storage medium

Abstract

The invention relates to an air-ground integrated three-dimensional scene automatic modeling method, system and device and a storage medium, and the method comprises the steps: carrying out the point cloud fusion and coordinate registration of a preprocessed air original point cloud and a preprocessed ground original point cloud, and obtaining an air-ground fusion point cloud image; performing semantic segmentation on the air-ground fusion point cloud image through a pre-trained point cloud semantic segmentation network to obtain a composite point cloud cluster containing multiple semantic categories; extracting a building outer contour of the building type point cloud cluster, simplifying the building outer contour according to a preset simplification rule, and constructing a building three-dimensional simplified model according to the simplified outer contour; extracting category feature parameters of the non-building type point cloud clusters, searching a standard model conforming to the category feature parameters from the preset model library, and replacing the corresponding point cloud clusters with the standard model to obtain a non-building three-dimensional simplified model; and performing archiving according to the semantic layer to form a structured three-dimensional model of the to-be-modeled urban area.

Description

Technical Field

[0001] This application relates to the field of urban modeling technology, and in particular to automatic modeling methods, systems, devices and storage media for integrated air-ground 3D scenes. Background Technology

[0002] Current city-level 3D modeling generally adopts the technical approach of "oblique photogrammetry + structure-of-motion (SfM) + dense matching," relying solely on UAVs equipped with five-lens cameras to collect multi-view oblique images along a preset flight path. Existing city-level 3D modeling methods generally suffer from the following problems: First, relying on image pixel matching to generate point clouds, the matching error in vegetation areas is large due to leaf light transmission and motion blur. Therefore, fine structural areas such as vegetation, power lines, and railings are prone to distortion and other problems, resulting in visual distortion and making them unusable for quantitative analysis. Second, the UAV's perspective cannot penetrate obstructions (such as canopies and overpasses), leading to image gaps and point cloud voids. This further results in geometrical defects or distortions in the model in obstructed areas (such as inward collapse of building bases), blurred textures, or blank areas. Finally, existing 3D modeling algorithms output continuous triangular meshes, where all objects are glued together, leading to an explosion in the model's face count. This results in low rendering efficiency, GPUs cannot load city-level scenes in real time, and individual objects cannot be separated or their attributes modified.

[0003] In summary, existing urban scene 3D modeling solutions suffer from problems such as large-scale twin details being damaged, models being indivisible and difficult to edit, and excessive number of triangles in large-scale urban model construction due to hardware limitations (single drone) and algorithm path dependence. Summary of the Invention

[0004] This application provides an automatic modeling method, system, device, and storage medium for integrated air-ground 3D scenes, which at least solves the problems of low accuracy and low efficiency in existing city-level 3D modeling methods in related technologies.

[0005] Firstly, this application provides an automatic modeling method for an integrated air-ground 3D scene, including: The original aerial point cloud and the original ground point cloud of the urban area to be modeled are acquired and preprocessed. The preprocessed original aerial point cloud and the original ground point cloud are then fused and coordinate registered to obtain the air-ground fused point cloud map. The air-ground fused point cloud map is semantically segmented by a pre-trained point cloud semantic segmentation network to obtain a composite point cloud cluster containing multiple semantic categories, including building point cloud clusters and non-building point cloud clusters. Extract the building outline of the building point cloud cluster, simplify the building outline according to the preset simplification rules, and construct a simplified 3D model of the building based on the simplified outline; Obtain a pre-set model library, extract the category feature parameters of the non-building point cloud clusters, find a standard model that matches the category feature parameters from the pre-set model library, and replace the corresponding point cloud clusters with the standard model to obtain a simplified 3D model of non-building. The simplified 3D model of the building and the simplified 3D model of the non-building are archived according to semantic layers to form a structured 3D model of the urban area to be modeled.

[0006] Specifically, the method further includes: The original aerial and ground images of the urban area to be modeled are acquired, and the original aerial and ground images are processed by a sparse reconstruction algorithm to obtain the camera pose and sparse point cloud of each frame. The camera pose, the sparse point cloud, the original aerial image, and the original ground image are processed using a dense reconstruction algorithm to generate a three-dimensional mesh model. Texture blocks are extracted from the three-dimensional mesh model, and all texture blocks are stitched together to obtain a two-dimensional texture image; Associating each point of the three-dimensional mesh model with the corresponding position in the two-dimensional texture image, a texture mapping index between the image and the point cloud surface is established; The two-dimensional texture image is mapped onto the surface of the structured three-dimensional model according to the texture mapping index, generating a twin three-dimensional scene map of the urban area to be modeled.

[0007] Specifically, the step of performing point cloud fusion and coordinate registration processing on the original aerial point cloud and the original ground point cloud to obtain an air-to-ground fused point cloud map includes: The time axis of the original aerial point cloud and the original ground point cloud are unified through the time synchronization mechanism of Kalman filtering; The ICP algorithm was used to spatially register the time-unified aerial point cloud and the ground point cloud to obtain the aerial registration transformation matrix and the ground registration transformation matrix. The time-unified original aerial point cloud and the original ground point cloud are fused according to the aerial registration transformation matrix and the ground registration transformation matrix to generate the air-ground fused point cloud map.

[0008] Specifically, the step of extracting the building outline of the building-type point cloud cluster, simplifying the building outline according to a preset simplification rule, and constructing a simplified 3D building model based on the simplified outline includes: The building-type point cloud clusters are preprocessed, and the main building structure surfaces are extracted from the preprocessed building-type point cloud clusters using the RANSAC algorithm. The points belonging to each of the main structural surfaces of the building are projected onto the corresponding plane to generate a two-dimensional point set, and the outline boundary is extracted from the two-dimensional point set to obtain the outer outline of the building. The Douglas-Peucker algorithm is used to simplify the points in the building's outer contour, retaining the points that reflect the original shape features of the building, resulting in a simplified two-dimensional contour. The bottom and top contour points are obtained from the simplified two-dimensional contour, and the bottom and top contour points are stretched according to the preset LOD level and the inverse projection process to generate a three-dimensional contour point cloud. The QEM algorithm is used to simplify the 3D contour point cloud, reducing the number of triangular patches, and thus obtaining the simplified 3D model of the building.

[0009] Specifically, the step of extracting the category feature parameters of the non-building point cloud clusters, searching for a standard model that matches the category feature parameters from the pre-set model library, and replacing the corresponding point cloud clusters with the standard model to obtain a simplified 3D non-building model includes: Obtain the ground reference elevation and point cloud cluster elevation, project the points of the non-building point cloud cluster onto the horizontal plane and extract the entity outline boundary, and calculate the geometric parameters of the entity corresponding to the non-building point cloud cluster based on the ground reference elevation, point cloud cluster elevation and entity outline boundary. The geometric parameters are discretized to obtain discretized parameters corresponding to the geometric parameters, and multiple discretized parameters are combined to generate a hash key; The corresponding standard model is matched from the preset model library according to the hash key, and the standard model is positioned at the center of the non-building point cloud cluster. The orientation and scale of the standard model are adjusted to align with the distribution direction of the non-building point cloud cluster. At the same time, the triangular facets of the non-building point cloud cluster are hidden to obtain the non-building 3D simplified model.

[0010] Specifically, the step of archiving the simplified 3D building model and the simplified 3D non-building model according to semantic layers to form a structured 3D model of the urban area to be modeled includes: The simplified 3D building model and the simplified 3D non-building model are divided into multiple base layers according to entity type, and each simplified model in the base layer is divided into multiple sub-layers according to functional attributes. A hierarchical structured model library is established based on the logical relationship and control parameters of the base layer and the sub-layers, and the structured model library is managed according to a graph structure. The structured 3D model is generated by fusing the various simplified models according to the structured model library.

[0011] Specifically, the step of establishing a hierarchical, structured model library based on the logical relationship and control parameters of the base layer and the sub-layers includes: A unique node is created for each simplified model, basic attributes are assigned, and the connection relationships between the simplified models are automatically analyzed, including spatial relationships and semantic relationships. The nodes and their corresponding connections are stored using a graph database, and spatial indexes are created based on the spatial relationships, while attribute indexes are created based on the semantic relationships. The structured model library is generated based on the nodes, the connections between nodes, and the corresponding indexes. The location layout of each simplified model library is managed through the connections, and classification retrieval is achieved through the indexes.

[0012] Secondly, this application provides an integrated system for automatic modeling of air-ground integrated 3D scenes, used to implement the automatic modeling method for air-ground integrated 3D scenes as described in any of the preceding claims, including: The aerial acquisition terminal is used to scan and photograph the urban area to be modeled, and obtain the original aerial point cloud and original aerial image. Ground acquisition terminal is used to scan and photograph the urban area to be modeled to obtain the original ground point cloud and the original ground image; Edge computing device is used to acquire and preprocess the original aerial and ground point clouds of the urban area to be modeled. A central modeling server is used to perform point cloud fusion and coordinate registration on the pre-processed aerial and ground point clouds to obtain an air-ground fused point cloud map. A pre-trained point cloud semantic segmentation network is used to perform semantic segmentation on the air-ground fused point cloud map, resulting in composite point cloud clusters containing multiple semantic categories, including building-type and non-building-type point cloud clusters. The building outlines of the building-type point cloud clusters are extracted and simplified according to preset simplification rules. A simplified 3D model of the building is constructed based on the simplified outlines. A pre-set model library is obtained, and the category feature parameters of the non-building-type point cloud clusters are extracted. A standard model matching the category feature parameters is found in the pre-set model library, and the corresponding point cloud clusters are replaced with the standard model to obtain a simplified 3D model of the non-building type. The simplified 3D model of the building type and the simplified 3D model of the non-building type are archived according to semantic layers to form a structured 3D model of the urban area to be modeled. A 3D visualization platform is used to display and interactively edit the structured 3D model.

[0013] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic modeling method for integrated air-ground 3D scenes as described in the first aspect above.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic modeling method for an integrated air-ground 3D scene as described in the first aspect above.

[0015] The air-ground integrated 3D scene automatic modeling method, system, equipment, and storage medium provided in this application have at least the following technical effects: This application achieves a breakthrough in both accuracy and efficiency for digital twin 3D scenes through air-ground collaborative data fusion and semantic-driven lightweight modeling. Specifically, by combining air-ground and point cloud data, occluded areas can be completely reconstructed; and through semantic segmentation, the types of entities corresponding to the point cloud are accurately identified, ensuring high-precision restoration of building outlines, vegetation morphology, etc. In addition, model simplification and replacement achieve a lightweight model effect, thereby reducing the total number of faces in the model, compressing the data volume, and improving the rendering efficiency of modeling.

[0016] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an embodiment of the automatic modeling method for an integrated air-ground 3D scene in this application; Figure 2 This is a system architecture diagram of an integrated air-ground three-dimensional scene automatic modeling system according to one embodiment of this application; Figure 3 This is a flowchart of point cloud semantic segmentation in one embodiment of this application; Figure 4 This is a flowchart illustrating a simplified outline of a building in one embodiment of this application; Figure 5 These are comparison images of the effects of an integrated air-ground 3D scene automatic modeling method in one embodiment of this application; Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0019] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0020] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] Current city-level 3D modeling generally adopts the technical approach of "oblique photogrammetry + structure-of-motion (SfM) + dense matching", which has the following technical problems: 1. "Patterned" phenomenon in vegetation and fine-structure areas Because the MVS algorithm relies on image pixel matching to generate point clouds, the matching error in vegetation areas is large due to leaf light transmission and motion blur. Furthermore, the forced connection of discontinuous points by the triangular mesh will result in the generation of broken triangular patches (such as "cloth strip" deformed surfaces formed between branches). For example, fine structural areas such as vegetation, power lines, and railings will appear distorted and patterned, resulting in visual distortion and making them unusable for quantitative analysis.

[0023] 2. Incomplete modeling of the building's base and obstructed areas. Because the drone's viewpoint cannot penetrate obstructions (such as canopies or overpasses), resulting in image gaps, and SfM requires at least three overlapping images, the occlusion area cannot meet this requirement, leading to point cloud voids (e.g., point cloud density at the base of a building <10 points / m²). Furthermore, because the void areas are not covered by images, it affects texture interpolation and filling, resulting in blurry color blocks. For example, the model geometry in the occluded area may be missing or distorted (e.g., the base of a building may collapse inwards), and the texture may be blurry or blank.

[0024] 3. The model has high redundancy and cannot be edited as a single unit. Because the output is a continuous triangular mesh, all objects (buildings, trees, ground) are stuck together, resulting in insufficient lightweighting. If the triangular mesh retains all details (such as a leaf containing dozens of triangular faces), the number of faces in the model will explode (1 billion faces per km²), further affecting rendering efficiency. The GPU cannot load city-level scenes in real time (frame rate <25fps); and it is impossible to separate independent objects (such as manually cutting the mesh to delete a tree), and attributes cannot be modified.

[0025] Existing technologies suffer from hardware limitations (single drone) and algorithm path dependence (SfM+MVS), resulting in large-scale digital twin details exhibiting defects such as broken patterns, models being treated as a single unit making secondary processing and editing difficult, and excessive triangular faces in large-scale city model construction. This application addresses these issues by leveraging technological advancements in air-ground collaborative data acquisition, point cloud semantic understanding, model classification and reconstruction, and model replacement to fundamentally sever the causal chain of these problems, achieving a breakthrough in the "high precision, low redundancy, and editability" of city-level digital twin models. Based on the above, embodiments of this application provide an integrated air-ground 3D scene automatic modeling method, system, device, and storage medium.

[0026] Firstly, embodiments of this application provide an automatic modeling method for an integrated air-ground 3D scene. Figure 1 This is a flowchart of the method; for the specific implementation process, please refer to the following steps.

[0027] Example 1

[0028] This application's integrated air-ground 3D scene automatic modeling method relies on an integrated system consisting of an aerial acquisition terminal, a ground acquisition terminal, an edge computing device, a central modeling server, and a 3D visualization platform. Through semantic segmentation of point clouds, the point cloud data is categorized according to geographic entities into buildings, roads, greenery, water bodies, bridges, and ground information. The RANSAC algorithm is used to extract the main structural surfaces (facades and roofs) of building point clouds, and complex objects are abstracted into cubes and prisms according to LoD1 or LoD2 levels. Some high-face-count entities (such as trees and garbage dumps) are replaced with existing models using the system's built-in model library, reducing the face count of model resources. This also ensures the preservation of the spatial characteristics of key entities (such as height, area, and elevation information), enabling modeling of large-scale urban areas. In a preferred embodiment, the ground mobile scanning vehicle can be replaced by a backpack-mounted laser scanning system, with a lightweight LiDAR manually carried into narrow areas to collect data, covering drone blind spots (alleys / indoor spaces).

[0029] Step S1: Obtain the original aerial point cloud and the original ground point cloud of the urban area to be modeled and preprocess them. Perform point cloud fusion and coordinate registration on the preprocessed original aerial point cloud and the original ground point cloud to obtain an air-ground fused point cloud map. The original aerial point cloud and the original ground point cloud are obtained by scanning the urban area to be modeled through an aerial acquisition terminal and a ground acquisition terminal, respectively.

[0030] In the point cloud fusion and coordinate registration process, the time axes of the original aerial point cloud and the original ground point cloud are unified through the time synchronization mechanism of Kalman filtering; the ICP algorithm is used to perform spatial registration on the time-unified original aerial point cloud and the original ground point cloud to obtain the aerial registration transformation matrix and the ground registration transformation matrix; the time-unified original aerial point cloud and the original ground point cloud are fused according to the aerial registration transformation matrix and the ground registration transformation matrix to generate the air-ground fused point cloud map.

[0031] Step S2: The air-ground fused point cloud map is semantically segmented using a pre-trained point cloud semantic segmentation network to obtain a composite point cloud cluster containing multiple semantic categories, including building-type point cloud clusters and non-building-type point cloud clusters.

[0032] Specifically, the multimodal point cloud semantic segmentation engine reference Figure 3 This method effectively fuses point cloud coordinates, image color, LiDAR intensity, normal vectors, and local curvature. The first step is to align the data and perform normalization preprocessing, followed by selecting an appropriate fusion strategy based on task requirements. To improve performance, an attention mechanism is introduced to dynamically weight feature importance, or a graph neural network is used to aggregate multimodal information within a local neighborhood. The ultimate goal is to significantly improve the accuracy of urban scene segmentation and detection tasks by fusing complementary features (such as geometric structure + optical properties). Improvements to the fusion of geometric / optical / reflection features are based on the RandLA-Net+ optimization rules for urban scenes, referencing [reference needed]. Figure 1 Five feature dimensions are defined to classify the physical meaning of the collected data. In a preferred embodiment, the feature input can be fused through a multimodal Transformer, inputting point cloud and image data into the ViT network to learn features across modalities.

[0033] Table 1: Feature Dimension Physical meaning effect XYZ Point cloud coordinates Geometric positioning RGB Image color values Differentiate materials Intensity LiDAR reflectivity Identify metal / glass Normal Normal vector direction Distinguishing between planes and curved surfaces Curvature Local curvature Identifying complex building structures Step S3: Extract the building outline of the building point cloud cluster, simplify the building outline according to preset simplification rules, and construct a simplified 3D building model based on the simplified outline. Specifically, this is achieved through a building model simplification engine. Input the classified building point cloud cluster, calculate the minimum bounding box (OBB) of the point cloud cluster to generate a cube base model. If the point cloud curvature is greater than a threshold (indicating complex structures), extract key contour lines to generate simplified polygons (face count < 50), and output a lightweight building model (.glb format).

[0034] Specifically, the building-type point cloud clusters are preprocessed, and the main structural surfaces of the buildings are extracted from the preprocessed building-type point cloud clusters using the RANSAC algorithm. The points belonging to each of the main structural surfaces of the buildings are projected onto the corresponding planes to generate a two-dimensional point set, and the contour boundary is extracted from the two-dimensional point set to obtain the outer contour of the building. The Douglas-Peucker algorithm is used to simplify the points in the outer contour of the building, retaining the points that reflect the original shape features of the building, to obtain a simplified two-dimensional contour. The bottom contour points and top contour points are obtained from the simplified two-dimensional contour, and the bottom contour points and top contour points are stretched according to a preset LOD level and inverse projection process to generate a three-dimensional contour point cloud. The QEM algorithm is used to simplify the three-dimensional contour point cloud, reducing the number of triangular patches, to obtain the simplified three-dimensional model of the building.

[0035] Preferably, manually setting the curvature threshold to 0.05 allows for automatic determination of complexity through deep learning, training a ResNet classifier, inputting a building point cloud rendering image, and outputting "simple / complex" labels.

[0036] Step S4: Obtain a pre-set model library, extract the category feature parameters of the non-building point cloud clusters, find a standard model that matches the category feature parameters from the pre-set model library, and replace the corresponding point cloud clusters with the standard model to obtain a simplified non-building 3D model. This application can employ a semantic-driven model replacement engine. "Semantic-driven" refers to a technical paradigm that triggers intelligent model replacement decisions based on the semantic information of objects (rather than geometric features). Its essence is to achieve accurate modeling optimization by understanding the entity meaning of a 3D scene. Input the set of triangular facets (including texture coordinates) of the original tilted model, and the point cloud semantic segmentation results (each triangular facet is associated with a semantic label). Identify the point cloud data of different physical entities and perform individual replacements respectively. Preferably, for vegetation modeling, the tree height-crown width matching model library can be generated in real time through Neural Radiation Field (NeRF), and the NeRF model is trained using vegetation point clouds to render a lightweight 3D model.

[0037] Specifically, the ground reference elevation and point cloud cluster elevation are obtained. The points of the non-building point cloud cluster are projected onto a horizontal plane and the entity outline boundary is extracted. Based on the ground reference elevation, point cloud cluster elevation, and entity outline boundary, the geometric parameters of the entity corresponding to the non-building point cloud cluster are calculated. The geometric parameters are discretized to obtain the discretized parameters corresponding to the geometric parameters. Multiple discretized parameters are combined to generate a hash key. Based on the hash key, the corresponding standard model is matched from a pre-set model library, and the standard model is positioned at the center of the non-building point cloud cluster. The orientation and scale of the standard model are adjusted to align with the distribution direction of the non-building point cloud cluster. At the same time, the triangular facets of the non-building point cloud cluster are hidden to obtain the simplified 3D model of the non-building.

[0038] Step S5: Archive the simplified 3D building model and the simplified 3D non-building model according to the semantic layers to form a structured 3D model of the urban area to be modeled.

[0039] Specifically, multiple base layers are created for the simplified 3D building model and the simplified non-building 3D model according to entity type, and each simplified model in the base layer is divided into multiple sub-layers according to functional attributes; a hierarchical structured model library is established based on the logical relationship and control parameters of the base layers and the sub-layers, and the structured model library is managed according to a graph structure; the simplified models are merged according to the structured model library to generate the structured 3D model.

[0040] More specifically, a unique node is created for each simplified model, basic attributes are assigned, and the connection relationships between the simplified models are automatically analyzed, including spatial relationships and semantic relationships. The nodes and their corresponding connection relationships are stored using a graph database, and a spatial index is created based on the spatial relationships and an attribute index is created based on the semantic relationships. The structured model library is generated based on the nodes, the connection relationships between nodes, and the corresponding indexes. The location layout of each simplified model library is managed through the connection relationships, and classification retrieval is achieved through the indexes.

[0041] After obtaining the structured 3D model, texture mapping is required. Specifically, aerial and ground images of the city area to be modeled are acquired. These images are then processed using a sparse reconstruction algorithm to obtain the camera pose and sparse point cloud for each frame. The aerial and ground images are obtained by scanning the city area using aerial and ground acquisition terminals, respectively. A dense reconstruction algorithm is then used to process the camera pose, sparse point cloud, aerial images, and ground images to generate a 3D mesh model. Texture blocks are extracted from the 3D mesh model and stitched together to obtain a 2D texture image. Each point in the 3D mesh model is associated with its corresponding position in the 2D texture image, establishing a texture mapping index between the image and the point cloud surface. Based on this texture mapping index, the 2D texture image is mapped onto the surface of the structured 3D model, generating a twin 3D scene map of the city area to be modeled.

[0042] Example 2

[0043] Based on Example 1, this application provides a more detailed method for automatic modeling of an integrated air-ground 3D scene, as detailed in the following steps.

[0044] Step 1: Integrated Air-Ground Data Acquisition The drone performs multi-angle oblique aerial photography and LiDAR scanning according to a set flight path, while ground equipment (data collection vehicle or personnel handheld visible light + laser point cloud data acquisition) collects street point cloud data; both record IMU + GPS information for subsequent registration. The equipment used includes multi-rotor drones and ground vehicles, both equipped with laser scanners, SLAM equipment, and visible light imaging equipment. Data types: Dense point cloud (PCD), high-resolution images (RGB). Spatial reference system: Unified WGS-84 or projected coordinate system (such as CGCS2000).

[0045] Step 2: Point cloud fusion and coordinate registration A time synchronization mechanism based on Kalman filtering is used to unify the time axis of the airborne point cloud. An improved ICP algorithm is used for spatial registration, including coarse registration (GPS guidance) and fine registration (point cloud matching and normal consistency constraints). Point clouds acquired by the UAV are generated using Photogrammetry or LiDAR, and ground point clouds are fused using ground-based SLAM methods (such as LOAM). The fusion process is based on the following formula:

[0046] Among them, P ground P is the raw point cloud collected by ground equipment. airRaw point cloud collected by drone, T ground T air Each device employs a registration transformation matrix. The transformation matrix is ​​obtained by inversely operating on the original point cloud data, converting local coordinates to global coordinates and mapping the data used by the devices to a describable world coordinate system. This ensures that data points from multiple devices can be registered in the same coordinate system. Specifically, the input includes UAV GPS / RTK positioning data (WGS84 coordinates), ground equipment SLAM odometry and GPS-assisted positioning, and IMU data (three-axis acceleration / angular velocity). The following formula is used to determine the coordinate system:

[0047] The transformation matrix (usually a 4×4 homogeneous transformation matrix) can achieve coordinate transformation because it perfectly describes rigid transformations (rotation + translation) in three-dimensional space through linear algebra operations.

[0048] In three-dimensional space, the homogeneous coordinates of a point p are represented as:

[0049] The first three elements are spatial coordinates, and the fourth element is 1 (to ensure that translation operations can be represented).

[0050] The 4×4 transformation matrix contains all the transformation information, as shown in the following formula:

[0051] The top-left 3×3 submatrix is ​​a rotation matrix used to control the rotation of the coordinate system (attitude alignment), the top-right 3×1 vector is a translation vector used to control the offset of the coordinate system origin (position alignment), and the last row of homogeneous coordinates maintains the consistency of homogeneous coordinates.

[0052] The coordinate transformation process is shown in the following formula:

[0053] The resulting components are:

[0054] Step 3: Image Pose Determination and Texture Reconstruction SfM (Structure from Motion) and OpenMVS (Open Multiple View Stereo) are two commonly used tools in 3D reconstruction, and they are usually used in combination. The entire process is divided into two main stages: sparse reconstruction (SfM) and dense reconstruction (MVS).

[0055] (1) SfM (Sparse Reconstruction) This paper reconstructs dense point clouds corresponding to images using SfM and OpenMVS methods; and performs unified texture stitching by combining UAV images and SLAM images to extract high-fidelity texture blocks. Input is multi-view images (from UAV and ground equipment). Output is camera pose (position and orientation) and sparse point cloud.

[0056] 1) Feature extraction and matching: Extract feature points (such as SIFT, SURF, ORB, etc.) from each image, and then match feature points between different images.

[0057] 2) Incremental reconstruction: Select an initial image pair, calculate the fundamental matrix or essential matrix, and triangulate the initial 3D points.

[0058] - Add new images incrementally and solve the pose of the new camera using the PnP (Perspective-n-Point) algorithm (using known 3D points and their corresponding points on the image).

[0059] - Perform bundle adjustment (BA) optimization, while adjusting camera parameters (focal length, distortion, etc.), camera pose, and 3D point coordinates to minimize reprojection error.

[0060] 3) Obtain the optimized camera pose (rotation matrix R and translation vector t) and sparse point cloud.

[0061] (2) SfM (Sparse Reconstruction) The dense point cloud is obtained by processing the camera pose and sparse point cloud output from the previous SfM step, along with the original image.

[0062] 1) Dense point cloud generation: The multi-view stereo matching (MVS) algorithm is used to calculate the 3D position of each pixel. The MVS algorithm in OpenMVS takes into account the viewpoints of neighboring images and generates a dense point cloud by matching pixel patches.

[0063] 2) Point cloud filtering: Removes outliers and noise.

[0064] 3) Surface reconstruction: Convert dense point clouds into mesh models (such as Poisson surface reconstruction).

[0065] (3) Texture splicing and high-fidelity texture block extraction 1) Viewpoint selection. For each triangular facet of the mesh, select the image with the best quality that allows you to see that facet (usually the image with the most frontal view and the highest resolution). The main considerations are the viewpoint angle (the angle between the normal direction and the camera's optical axis), image resolution, and occlusion. 2) Texture patch extraction. For each triangular facet, extract the corresponding texture patch from the selected image based on the texture coordinates (UV coordinates) of its three vertices.

[0066] 3) Local texture optimization. Since adjacent patches may come from different images, direct stitching will result in seams and color inconsistencies. Therefore, global optimization is needed to adjust the brightness, contrast, and color balance between different images to make them consistent. During the texture map generation process, techniques such as graph cut or Poisson blending are used to eliminate seams.

[0067] 4) High-fidelity texture block extraction. Before texture mapping, the image is preprocessed, including denoising, sharpening, and contrast stretching, to retain more details. For important areas (such as building facades), information from multiple images can be fused to generate higher-fidelity textures through super-resolution technology.

[0068] (4) Pose determination using SfM / SLAM. SfM solves camera pose through feature matching and incremental reconstruction. For UAV images, since it is an aerial viewpoint, SfM is usually used. Ground SLAM images: Pose determination is done using SLAM (Simultaneous Localization and Mapping). For consecutive frames, camera motion (pose transformation between adjacent frames) is estimated through feature matching or direct methods, and pose graph optimization (Pose Graph Optimization) or BA is used to jointly optimize the pose of all keyframes. It is then determined whether the position has returned to the previous position to eliminate accumulated errors.

[0069] SfM / SLAM is used to solve the pose of the image, and texture detail patches are extracted from the image:

[0070] Establish a texture mapping index between the image and the point cloud surface, associating each point (or triangular facet) on the surface of the 3D model with the corresponding position in the 2D texture image, as follows.

[0071] (1) Texture coordinate (UV coordinate) calculation. For each vertex of the 3D mesh, calculate its coordinates (U,V) on the texture image, ranging from [0,1]. Project the 3D points onto the image plane using a projection function, with the formula: (μ,ν)=Π camera (P model ) (2) Indexing. Store the UV coordinates of each vertex. Each triangle consists of three vertices, so its texture is determined by the triangle region defined by the UV coordinates of the three vertices. For efficient rendering, multiple small texture blocks are usually packaged into a large texture atlas. Each facet has a corresponding region in the texture atlas.

[0072] (3) Texture mapping function. For a point P on the mesh, it is transformed to the camera coordinate system through model transformation. The image coordinates (u,v) are normalized to texture coordinates (U,V) by projecting the camera intrinsic parameter matrix onto the image plane.

[0073] (4) Indexed storage. The texture coordinates of each vertex are stored in the 3D model file (such as OBJ or GLTF format). During rendering, the UV coordinates are passed to the fragment shader through the vertex shader, and colors are sampled from the texture image.

[0074] Step 4: Point Cloud Semantic Segmentation The RandLA-Net network was trained on the SensatUrban and Toronto-3D datasets to segment point clouds into categories such as buildings, roads, vegetation, and water. RandLA-Net is a key deep learning architecture for achieving semantic segmentation of point clouds. Its role is to accurately classify the fused empty-point cloud into semantic categories such as buildings, roads, vegetation, and water, laying the foundation for subsequent differential modeling.

[0075] Where C is the set of categories, and fpoint is the point feature vector.

[0076] Step 4a: Building point cloud cluster -> Geometric simplification / abstraction algorithm (such as bounding box calculation, key point extraction).

[0077] Sub-step 4b: Vegetation point cloud clusters -> Match rules from the parametric model library (based on point cloud geometric properties).

[0078] Sub-step 4c: Modeling methods for other categories (ground, roads, etc.).

[0079] Traditional point clouds only use XYZ coordinates, while this solution uses 10-dimensional feature vectors.

[0080] Step 5: Building structure extraction and simplified modeling refer to Figure 4 The RANSAC algorithm is used to extract the main structural surfaces (facade and roof) of the building point cloud, which are then abstracted into cubes and prisms according to LoD1 or LoD2 levels. QEM is combined to reduce the number of facets and improve real-time rendering efficiency. This step mainly involves extracting the external contours of the building (usually the bottom and roof contours) from the segmented building point cloud, and then constructing a simplified 3D model (such as a cube or prism) based on these contours.

[0081] (1) Point cloud preprocessing. Outlier removal: Use statistical filtering or radius filtering to remove noise points. Normal estimation: Calculate the normal vector of each point in the point cloud for subsequent plane segmentation.

[0082] (2) Principal Structure Surface Extraction (RANSAC Algorithm). The RANSAC (Random Sample Consensus) algorithm is used to fit a plane from a point cloud. Its basic idea is to randomly select three points to determine a plane, then calculate the distances from other points to this plane, and consider points with distances less than a threshold as interior points. This process is repeated multiple times, and the plane with the most interior points is selected as the optimal plane. The plane equation is: , where (a,b,c) are the normal vectors of the plane, and d is a constant term.

[0083] For three points P_1(x_1,y_1,z_1), P_2(x_2,y_2,z_2), and P_3(x_3,y_3,z_3), calculate the plane parameters. Set a threshold (e.g., 0.1 meters), and consider points within a distance of this threshold as interior points. Repeat this iteration multiple times, selecting the plane with the most interior points.

[0084] For architecture, we typically need to extract multiple master plans (such as the four facades and the roof plan). Therefore, after extracting one plan, its interior points are removed from the point cloud, and other plans are extracted from the remaining point cloud.

[0085] Then, calculate the distance from each point Pi to the plane:

[0086] (3) Contour projection. For each extracted principal plane (such as the bottom or roof plane), we project the points belonging to that plane onto the plane to obtain a two-dimensional point set. Then, we extract the contour from the two-dimensional point set.

[0087] (4) Contour extraction (from a set of 2D projected points). On a 2D plane, contours are extracted using convex hull or concave hull algorithms, such as Alpha Shapes. Alpha Shapes is a method for extracting non-convex shapes. It generates a concave hull by setting a parameter alpha (which controls the tightness of the contour).

[0088] 1) Construct the Delaunay triangulation of the point set.

[0089] 2) Traverse all triangles and calculate their circumcircle radius R.

[0090] 3) If the circumradius R of a triangle is less than 1 / alpha (alpha is a preset parameter), then the sides of the triangle are retained; otherwise, they are deleted.

[0091] 4) Of the edges that are retained, only the outermost boundary (i.e. the edges that form the contour) is taken.

[0092] (5) Contour simplification. The extracted contour may have a large number of points, so simplification is necessary. A commonly used algorithm is the Douglas-Peucker algorithm: Given a polygonal curve, find the point farthest from the line connecting the beginning and end. If the distance is greater than a threshold, keep the point and recursively divide the curve into two segments using that point as the boundary; otherwise, connect the beginning and end points with a straight line.

[0093] The simplified outline consists of fewer points, but still retains the main features of the original shape.

[0094] (5) 3D reconstruction. Based on the simplified 2D outline, we can construct a 3D model: 1) Bottom contour: Convert the two-dimensional contour points back to three-dimensional coordinates (on the plane) according to the reverse process of projection.

[0095] 2) Top profile: If it is a flat-roofed building, the top profile is the same as the bottom profile but the height is different; if it is a pitched roof, there may be multiple top profiles (extracted from multiple planes).

[0096] 3) Side view: Connect the corresponding points of the bottom and top contours to form the side view.

[0097] For the simplified model, it is constructed according to LOD levels: LOD1: The building is extruded into a cube or prism using a single base profile (polygon) and height (usually the highest point of the building). LOD2: The building may have multiple base profiles (such as the main building and the podium) and multiple top profiles (such as a pitched roof), which are then extruded separately. Finally, the QEM algorithm is used to simplify the generated 3D mesh, reducing the number of triangular facets.

[0098] Simplify to a cube model (simplified model) and establish a LOD hierarchy to reduce the number of triangles and improve real-time rendering efficiency.

[0099] Step 6: Complex Model Replacement By determining the ground reference elevation and measuring the elevation of the tree crown apex (the highest point in the point cloud), the actual tree height is calculated as: tree crown apex elevation - ground reference elevation. The tree crown outline and height parameters are analyzed, and the tree crown point cloud is projected onto a horizontal plane (XY plane). The Alpha Shape algorithm is used to extract the precise outline boundary, and the convex hull is calculated to determine the maximum crown diameter. Ellipse fitting is then used to quantify the outline shape features (eccentricity, compactness). Finally, a standard vegetation model with animation effects from the engine library is automatically replaced, improving the scene's realism and dynamism. Model library matching is achieved through hash indexing or graphical feature matching.

[0100] (1) Divide continuous eigenvalues ​​into discrete intervals, for example -Tree height: Classified by 2 meters (0-2m, 2-4m, ..., >30m) - Crown diameter: graded in 1-meter increments (0-1m, 1-2m, ..., >15m) - Profile eccentricity: in increments of 0.1 (0-0.1, 0.1-0.2, ..., >1.0) (2) Hash key generation. Combine discretization parameters to generate a unique string key, instance format: height grade_crown grade_eccentricity grade.

[0101] -Tree height 12.3m → Grade "12" (12-14m range) - Crown width 6.8m → Graded "6" (6-7m range) -Eccentricity 0.73 → Grade "7" (0.7-0.8 range) Generate hash key: "12_6_7" (3) List of matching model IDs.

[0102] (4) Model base construction scene building. Position the standard model at the center of the original vegetation area. Rotate the model so that its main axis is aligned with the point cloud distribution direction, scale the model proportionally according to the height parameter, blend the original texture and the model texture to achieve a transition, hide the corresponding triangular facets of the original tilted model, and add the new model to the scene.

[0103] Step 7: Model Blending and Layer Generation Simplified models of various categories such as buildings, green spaces, roads, and water bodies are archived according to semantic layers to form a structured 3D model; it supports export in GeoJSON or CityGML format, which facilitates subsequent platform integration and analysis.

[0104] Semantic layer archiving refers to organizing 3D models hierarchically according to real-world entity categories and functional attributes, establishing a logically related spatial data management structure. Its essence is to achieve intelligent structuring of scenes through semantic information (rather than geometric features), as illustrated in Table 2.

[0105] Table 2:

[0106] Step 8: Texture Mapping and Enhancement Processing Lighting is balanced according to the building's orientation; combined with image enhancement methods (such as contrast stretching and edge sharpening), high-resolution texture blocks are mapped onto the surface of the simplified model to achieve a balance between high realism and face count control.

[0107] Detail compensation is applied to the abstract model, and the texture mapping is calculated as follows:

[0108] in This represents the projection function from a 3D point onto the image plane.

[0109] Step 9: Multi-category model fusion and independent management Each model has its own independent layers and control parameters, which can be managed using a graph structure:

[0110] Step 10: Dynamic Model Updates and Interactive Editing Users can select a region of the model to trigger local reconstruction, architectural style replacement, or material editing. The system automatically calls the original point cloud and image data for local updates. We use a graph structure to manage multi-category models. Its core principle is to treat each independent model as a node in the graph, and the various relationships between models as edges connecting the nodes. In this way, we can not only efficiently organize and manage massive amounts of model data, but also express complex spatial, semantic, and logical relationships.

[0111] (1) Constructing nodes - Create a unique node for each model and assign it basic attributes (type, position, size, etc.).

[0112] - For example: building nodes include height, area, and ownership information; tree nodes include tree species, age, and maintenance records.

[0113] (2) Establishing relationships - Automatically analyze spatial relationships between models: determine "adjacent" relationships through spatial calculations (such as bounding box intersection detection).

[0114] - Define semantic relationships manually or through business rules: such as specifying that a certain substation is "connected" to a certain power grid.

[0115] - Record temporal relationships: When the model is updated, establish a "replacement" relationship chain between the old and new versions.

[0116] (3) Graph storage and indexing - Use a graph database (such as Neo4j) to store nodes and relationships.

[0117] - Create spatial indexes (R-trees) to accelerate spatial queries, and create attribute indexes to accelerate semantic queries.

[0118] (4) Hierarchical management mechanism - Geometric layer: Manages the layout of the model's location through spatial relationship edges.

[0119] - Semantic layer: Enables categorized retrieval through tags and attributes (e.g., finding all commercial buildings).

[0120] All objects support semantic attribute editing and lifecycle maintenance. User interaction is supported in selecting specific areas and replacing elements including vegetation type, building style library, building facade material, and texture.

[0121] In summary, the automatic modeling method for integrated air-ground 3D scenes provided in this application has the following beneficial effects: 1. Solve the problem of poor modeling effect in occluded areas This application employs a combined air-ground data acquisition method (UAV + mobile ground scanning vehicle + fixed scanner), with the ground equipment specifically designed to supplement data in low-altitude and obstructed areas (such as the bottom 0-3m of buildings). The ground scanning vehicle, equipped with a LiDAR, acquires high-density point clouds (≥500 points / ㎡) at a frequency of 20Hz, and fuses them with the UAV point clouds using an ICP algorithm (spatiotemporal synchronization accuracy at the μs level). Through this combined air-ground acquisition, the point cloud coverage rate in obstructed areas (such as gaps between buildings and under overpasses) reaches over 98% (compared to only 40% using traditional methods), filling data gaps.

[0122] Meanwhile, during the texture mapping stage, the texture of the building's base is preferentially fused using a perspective projection from a panoramic ground image (8K resolution), rather than relying on a single aerial image. As a result, the modeling completeness of occluded areas is improved from the traditional ≤40% to ≥95%, solving the problem of model incompleteness or distortion caused by missing viewpoints in oblique photogrammetry.

[0123] 2. Eliminate detailed decorative patterns (especially in vegetated areas). refer to Figure 5 This application uses point cloud semantic segmentation (an improved RandLA-Net network) to accurately classify vegetation point clouds and replaces them with instances from a parametric model library, instead of directly reconstructing oblique photogrammetric triangulation. The segmentation network takes 10-dimensional features (coordinates, RGB, intensity, and normal vector) as input and outputs vegetation point cloud clusters. The matching rules call a pre-defined LOD model (such as a 300-facet sycamore tree model) based on height and crown diameter. This bypasses the dense matching process of oblique photogrammetry in vegetation modeling, avoiding motion blur and triangulation errors. Simultaneously, the model library instances come with optimized textures and dynamic shaders (such as wind field animation), eliminating reconstruction noise; the number of facets in the model is simplified (the number of facets in the vegetation area is reduced by more than 90%), thus completely eliminating the "patchy" effect in the vegetation area and improving the visual realism of the model.

[0124] 3. Significantly reduce the number of polygons in the model and improve the rendering efficiency of city-level areas. refer to Figure 5This application implements differentiated modeling, simplifying buildings into OBB bounding boxes (simple buildings) or outline models (complex buildings) based on complexity, and replacing vegetation with low-face-count model library instances. When the curvature of the building point cloud is <0.05, an OBB cube (12 faces) is generated; when the curvature is ≥0.05, key outlines are preserved (face count <50). The vegetation model library includes LOD levels (e.g., shrubs with 50 faces, pine trees with 800 faces), thus significantly reducing the geometric complexity of the model and minimizing redundant triangular faces. Simultaneously, by combining texture compression (BC7 algorithm compresses 4K textures to 512KB) and virtual texture technology, GPU memory usage is reduced by 70%. Therefore, the total number of faces in the model is reduced by 88% (from 1.12 billion faces to 130 million faces), and the GPU rendering frame rate is increased from 24fps to 60fps, meeting real-time interactive requirements (such as real-time simulation of digital twin cities).

[0125] Secondly, embodiments of this application provide an integrated system for automatic modeling of air-ground integrated 3D scenes, used to implement the automatic modeling method for air-ground integrated 3D scenes as described in any of the preceding embodiments, including: The aerial acquisition terminal is used to scan and photograph the urban area to be modeled, and obtain the original aerial point cloud and original aerial image. Ground acquisition terminal is used to scan and photograph the urban area to be modeled to obtain the original ground point cloud and the original ground image; Edge computing device is used to acquire and preprocess the original aerial and ground point clouds of the urban area to be modeled. A central modeling server is used to perform point cloud fusion and coordinate registration on the pre-processed aerial and ground point clouds to obtain an air-ground fused point cloud map. A pre-trained point cloud semantic segmentation network is used to perform semantic segmentation on the air-ground fused point cloud map, resulting in composite point cloud clusters containing multiple semantic categories, including building-type and non-building-type point cloud clusters. The building outlines of the building-type point cloud clusters are extracted and simplified according to preset simplification rules. A simplified 3D model of the building is constructed based on the simplified outlines. A pre-set model library is obtained, and the category feature parameters of the non-building-type point cloud clusters are extracted. A standard model matching the category feature parameters is found in the pre-set model library, and the corresponding point cloud clusters are replaced with the standard model to obtain a simplified 3D model of the non-building type. The simplified 3D model of the building type and the simplified 3D model of the non-building type are archived according to semantic layers to form a structured 3D model of the urban area to be modeled. A 3D visualization platform is used to display and interactively edit the structured 3D model.

[0126] It should be noted that the integrated system provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, terms such as "module," "unit," and "subunit" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0127] Thirdly, embodiments of this application provide an electronic device, Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 6 As shown, the electronic device may include a processor 11 and a memory 12 storing computer program instructions.

[0128] Specifically, the processor 11 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0129] The memory 12 may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory 12 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 12 may include removable or non-removable (or fixed) media. Where appropriate, the memory 12 may be internal or external to a data processing device. In a particular embodiment, the memory 12 is non-volatile memory. In a particular embodiment, the memory 12 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0130] The memory 12 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 11.

[0131] The processor 11 reads and executes computer program instructions stored in the memory 12 to implement any of the automatic modeling methods for integrated air-ground 3D scenes in the above embodiments.

[0132] In one embodiment, the electronic device may further include a communication interface 13 and a bus 10. Wherein, as... Figure 6 As shown, the processor 11, memory 12, and communication interface 13 are connected through bus 10 and communicate with each other.

[0133] The communication interface 13 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 13 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0134] Bus 10 includes hardware, software, or both, that couples components of an electronic device together. Bus 10 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 10 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 10 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0135] Fourthly, embodiments of this application provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the automatic modeling method for an integrated air-ground 3D scene provided in the first aspect.

[0136] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0137] In a possible implementation, this application can also be implemented as a program product comprising program code, wherein when the program product is run on a terminal device, the program code is used to cause the terminal device to perform the steps of implementing the air-ground integrated three-dimensional scene automatic modeling method provided in the first aspect.

[0138] The program code for executing this application can be written in any combination of one or more programming languages, and the program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An automatic modeling method for an integrated air-ground 3D scene, characterized in that, include: The original aerial point cloud and the original ground point cloud of the urban area to be modeled are acquired and preprocessed. The preprocessed original aerial point cloud and the original ground point cloud are then fused and coordinate registered to obtain the air-ground fused point cloud map. The air-ground fused point cloud map is semantically segmented by a pre-trained point cloud semantic segmentation network to obtain a composite point cloud cluster containing multiple semantic categories, including building point cloud clusters and non-building point cloud clusters. Extract the building outline of the building point cloud cluster, simplify the building outline according to the preset simplification rules, and construct a simplified 3D model of the building based on the simplified outline; Obtain a pre-set model library, extract the category feature parameters of the non-building point cloud clusters, find a standard model that matches the category feature parameters from the pre-set model library, and replace the corresponding point cloud clusters with the standard model to obtain a simplified 3D model of non-building. The simplified 3D model of the building and the simplified 3D model of the non-building are archived according to semantic layers to form a structured 3D model of the urban area to be modeled.

2. The automatic modeling method for integrated air-ground 3D scenes according to claim 1, characterized in that, The method further includes: The original aerial and ground images of the urban area to be modeled are acquired, and the original aerial and ground images are processed by a sparse reconstruction algorithm to obtain the camera pose and sparse point cloud of each frame. The camera pose, the sparse point cloud, the original aerial image, and the original ground image are processed using a dense reconstruction algorithm to generate a three-dimensional mesh model. Texture blocks are extracted from the three-dimensional mesh model, and all texture blocks are stitched together to obtain a two-dimensional texture image; Associating each point of the three-dimensional mesh model with the corresponding position in the two-dimensional texture image, a texture mapping index between the image and the point cloud surface is established; The two-dimensional texture image is mapped onto the surface of the structured three-dimensional model according to the texture mapping index, generating a twin three-dimensional scene map of the urban area to be modeled.

3. The automatic modeling method for integrated air-ground 3D scenes according to claim 1, characterized in that, The step of performing point cloud fusion and coordinate registration on the original aerial point cloud and the original ground point cloud to obtain an air-ground fused point cloud map includes: The time axis of the original aerial point cloud and the original ground point cloud are unified through the time synchronization mechanism of Kalman filtering; The ICP algorithm was used to spatially register the time-unified aerial point cloud and the ground point cloud to obtain the aerial registration transformation matrix and the ground registration transformation matrix. The time-unified original aerial point cloud and the original ground point cloud are fused according to the aerial registration transformation matrix and the ground registration transformation matrix to generate the air-ground fused point cloud map.

4. The automatic modeling method for integrated air-ground 3D scenes according to claim 1, characterized in that, The process of extracting the building outline from the building-type point cloud cluster, simplifying the building outline according to a preset simplification rule, and constructing a simplified 3D building model based on the simplified outline includes: The building-type point cloud clusters are preprocessed, and the main building structure surfaces are extracted from the preprocessed building-type point cloud clusters using the RANSAC algorithm. The points belonging to each of the main structural surfaces of the building are projected onto the corresponding plane to generate a two-dimensional point set, and the outline boundary is extracted from the two-dimensional point set to obtain the outer outline of the building. The Douglas-Peucker algorithm is used to simplify the points in the building's outer contour, retaining the points that reflect the original shape features of the building, resulting in a simplified two-dimensional contour. The bottom and top contour points are obtained from the simplified two-dimensional contour, and the bottom and top contour points are stretched according to the preset LOD level and the inverse projection process to generate a three-dimensional contour point cloud. The QEM algorithm is used to simplify the 3D contour point cloud, reducing the number of triangular patches, and thus obtaining the simplified 3D model of the building.

5. The automatic modeling method for integrated air-ground 3D scenes according to claim 1, characterized in that, The process involves extracting the category feature parameters of the non-building point cloud clusters, searching for a standard model that matches the category feature parameters from the pre-set model library, and replacing the corresponding point cloud clusters with the standard model to obtain a simplified 3D non-building model, including: Obtain the ground reference elevation and point cloud cluster elevation, project the points of the non-building point cloud cluster onto the horizontal plane and extract the entity outline boundary, and calculate the geometric parameters of the entity corresponding to the non-building point cloud cluster based on the ground reference elevation, point cloud cluster elevation and entity outline boundary. The geometric parameters are discretized to obtain discretized parameters corresponding to the geometric parameters, and multiple discretized parameters are combined to generate a hash key; The corresponding standard model is matched from the preset model library according to the hash key, and the standard model is positioned at the center of the non-building point cloud cluster. The orientation and scale of the standard model are adjusted to align with the distribution direction of the non-building point cloud cluster. At the same time, the triangular facets of the non-building point cloud cluster are hidden to obtain the non-building 3D simplified model.

6. The automatic modeling method for integrated air-ground 3D scenes according to claim 1, characterized in that, The process of archiving the simplified 3D building model and the simplified 3D non-building model according to semantic layers to form a structured 3D model of the urban area to be modeled includes: The simplified 3D building model and the simplified 3D non-building model are divided into multiple base layers according to entity type, and each simplified model in the base layer is divided into multiple sub-layers according to functional attributes. A hierarchical structured model library is established based on the logical relationship and control parameters of the base layer and the sub-layers, and the structured model library is managed according to a graph structure. The structured 3D model is generated by fusing the various simplified models according to the structured model library.

7. The automatic modeling method for integrated air-ground 3D scenes according to claim 6, characterized in that, The step of establishing a hierarchical, structured model library based on the logical relationship and control parameters of the base layer and the sub-layers includes: A unique node is created for each simplified model, basic attributes are assigned, and the connection relationships between the simplified models are automatically analyzed, including spatial relationships and semantic relationships. The nodes and their corresponding connections are stored using a graph database, and spatial indexes are created based on the spatial relationships, while attribute indexes are created based on the semantic relationships. The structured model library is generated based on the nodes, the connections between nodes, and the corresponding indexes. The location layout of each simplified model library is managed through the connections, and classification retrieval is achieved through the indexes.

8. An integrated system for automatic modeling of three-dimensional scenes combining air and ground, characterized in that, The method for automatically modeling an integrated air-ground 3D scene as described in any one of claims 1 to 7 includes: The aerial acquisition terminal is used to scan and photograph the urban area to be modeled, and obtain the original aerial point cloud and original aerial image. Ground acquisition terminal is used to scan and photograph the urban area to be modeled to obtain the original ground point cloud and the original ground image; Edge computing device is used to acquire and preprocess the original aerial and ground point clouds of the urban area to be modeled. A central modeling server is used to perform point cloud fusion and coordinate registration on the pre-processed aerial and ground point clouds to obtain an air-ground fused point cloud map. A pre-trained point cloud semantic segmentation network is used to perform semantic segmentation on the air-ground fused point cloud map, resulting in composite point cloud clusters containing multiple semantic categories, including building-type and non-building-type point cloud clusters. The building outlines of the building-type point cloud clusters are extracted and simplified according to preset simplification rules. A simplified 3D model of the building is constructed based on the simplified outlines. A pre-set model library is obtained, and the category feature parameters of the non-building-type point cloud clusters are extracted. A standard model matching the category feature parameters is found in the pre-set model library, and the corresponding point cloud clusters are replaced with the standard model to obtain a simplified 3D model of the non-building type. The simplified 3D model of the building type and the simplified 3D model of the non-building type are archived according to semantic layers to form a structured 3D model of the urban area to be modeled. A 3D visualization platform is used to display and interactively edit the structured 3D model.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic modeling method for an integrated air-ground 3D scene as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the automatic modeling method for an integrated air-ground 3D scene as described in any one of claims 1 to 7.

Images