Automatic extraction method of real scene three-dimensional semantic information based on multi-source data fusion

Abstract

The application discloses a kind of based on multi-source data fusion real scene three-dimensional semantic information automation extraction method, belong to information processing technical field.The method includes: to oblique photography image, laser point cloud, multispectral image and auxiliary text are collected and registered;Through cross-modal feature fusion network, geometry, texture and spectral feature are extracted and fused;Utilize multi-task collaborative network to realize semantic segmentation, instance segmentation and relationship reasoning simultaneously;With the aid of visual and language big model, component identification and attribute association are carried out;Constructing spatiotemporal knowledge graph and driving generation fine modeling, finally output queryable, computable, support natural language interaction semantic three-dimensional model.The application realizes the whole process automation of semantic extraction, significantly improves the semantic consistency, structural integrity and practical value of three-dimensional model.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and specifically to a method for automatically extracting real-scene 3D semantic information based on multi-source data fusion. Background Technology

[0002] Currently, real-scene 3D modeling has become the spatial digital foundation for fields such as digital cities and smart parks; existing semantic information extraction methods mainly rely on a single data source (such as oblique photogrammetry models) and classify and label 3D models through manual intervention or semi-automatic algorithms.

[0003] However, existing semantic information extraction methods usually treat semantic classification, instance segmentation, and spatial relationship analysis as independent tasks, ignoring their inherent connections, which sometimes leads to inconsistent results and missing relationships. In addition, existing methods can only identify a limited number of predefined categories, making it difficult to adapt to new object types or learn to expand semantics from unstructured data.

[0004] Furthermore, the mesh model generated by oblique photography is prone to defects in areas with weak texture, which affects the accuracy of subsequent semantic analysis. Summary of the Invention

[0005] To address this issue, the present invention provides an automated method for extracting real-world 3D semantic information based on multi-source data fusion, thereby solving the problems in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An automated method for extracting real-world 3D semantic information based on multi-source data fusion includes the following steps:

[0008] Step 1. Standardize and preprocess the multi-source data to obtain oblique photogrammetric images, laser scanning point clouds, multispectral images, and structured or unstructured auxiliary data covering the same target area, and perform spatiotemporal benchmark unification and registration.

[0009] Step 2. Perform cross-modal feature deep fusion encoding on the multi-source data processed in Step 1. Extract point cloud geometric features, image texture features and material spectral features through a deep feature learning network, and use an adaptive attention mechanism for weighted fusion to generate a unified fusion feature tensor.

[0010] Step 3. Input the fused feature tensor into the multi-task collaborative learning network, and simultaneously execute the three sub-tasks of semantic segmentation, instance segmentation and spatial relationship reasoning. The attention interaction module is used to realize information interaction and result correction between tasks.

[0011] Step 4. Based on the instance segmentation results and the corresponding semantic category information, use the visual big model to perform refined component recognition of key targets, and call the big language model to parse unstructured text, and associate the extracted attribute information with the corresponding three-dimensional entity instance;

[0012] Step 5. Construct a spatiotemporal knowledge graph using 3D entity instances as nodes and semantic and spatial relationships as edges. Based on the semantic information and geometric attributes of entity nodes in the spatiotemporal knowledge graph, and combined with the global and local fusion features obtained in Step 2, drive generative fine modeling to generate a 3D scene model that couples semantic and geometric structures.

[0013] Step 6. Output a semantic real-world 3D model with semantic tags that is queryable and reasonable.

[0014] Furthermore, the specific process of cross-modal feature deep fusion encoding in step 2 includes:

[0015] Geometric features of point clouds are extracted using a 3D convolutional neural network, and texture features of RGB images and material features of multispectral images are extracted using a 2D convolutional neural network. An adaptive attention mechanism is then used to perform weighted fusion of the multimodal features.

[0016] The adaptive attention-weighted fusion calculation formula is as follows:

[0017] ;

[0018] ;

[0019] in, For the unified feature tensor after fusion; Let k = p, r, m correspond to point cloud, RGB, and multispectral data, respectively, satisfying the following conditions: ; This is a learnable parameter matrix used to calculate the attention score; To splice multimodal features along the channel dimension; For the geometric feature tensor of the point cloud; RGB image texture features; This refers to the material characteristics of multispectral images.

[0020] Furthermore: In step 3, the multi-task collaborative learning network is trained using the following loss function:

[0021] ;

[0022] This represents the total loss value. , and All are task balance coefficients; For semantic segmentation loss; The instance segmentation loss; Loss due to spatial relation inference.

[0023] Furthermore: In step 4, when segmenting components using a large visual model, visual-text similarity matching is employed to achieve zero-sample or small-sample guided recognition. The similarity calculation formula is as follows:

[0024] ;

[0025] in, Visual-text similarity score; The CLIP model is used for the visual embedding of the target component image; Text embedding of part names for CLIP models.

[0026] Furthermore: In step 4, when parsing unstructured text using a large language model, entity and attribute information is extracted through a semantic coding model, and this information is spatially semantically matched and associated with three-dimensional entity instances.

[0027] Furthermore: In step 5, when constructing the spatiotemporal knowledge graph, a graph convolutional network is used to update node features. The node feature update formula is:

[0028] ;

[0029] in, Let i be the embedding vector of node i in the l-th layer graph; Let i be the set of neighboring nodes of node i; This is an aggregation operation; The trainable weight matrix for the graph convolutional layer; A nonlinearity is introduced for the activation function.

[0030] Furthermore, in step 5, the generative fine modeling is driven by a generative method based on physical rendering, which combines knowledge graphs and fusion features to generate a visually faithful, lightweight, and semantically complete 3D scene model through 3D Gaussian splashing technology.

[0031] Furthermore, the semantic real-world 3D model output in step 6 supports natural language interaction, enabling users to perform semantic queries, scene analysis, and decision support through the integrated large language model intelligent agent.

[0032] Furthermore, the multi-source data also includes existing GIS vector data, which is rasterized into a spatial mask in step 2 to provide contour constraints and participate in feature fusion.

[0033] The present invention has the following advantages: It realizes full-process automation from multi-source data acquisition to semantic 3D model generation, which greatly reduces the manpower cost and time cycle of professional annotation; through multi-task collaboration and knowledge graph construction, it ensures the high consistency of semantic information in logical and spatial relationships, effectively overcoming the drawbacks of scattered results and missing relationships in traditional methods; it introduces a large visual and linguistic model, enabling the system to continuously absorb new knowledge from open domain data and expand the scope of semantic recognition.

[0034] Other features and advantages of the present invention will be set forth in the following description. Attached Figure Description

[0035] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0036] Figure 1 The flowchart illustrates the implementation of the automated extraction method for real-scene 3D semantic information based on multi-source data fusion provided in this application embodiment. Detailed Implementation

[0037] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these embodiments are merely for further explanation of the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described content.

[0038] Please see Figure 1 An automated method for extracting real-world 3D semantic information based on multi-source data fusion includes the following steps:

[0039] Step 1. Standardize the collection of multi-source data and perform preprocessing;

[0040] Acquire multiple complementary spatial data covering the same target area and perform rigorous preprocessing to ensure data consistency on a spatiotemporal reference.

[0041] The specific processing content is as follows: High-resolution, multi-view aerial image sequences are acquired through oblique photogrammetry technology to restore the macroscopic texture and geometry of the scene;

[0042] High-precision 3D point cloud data can be obtained through airborne or ground-based laser scanning technology, providing accurate spatial structure information that is unaffected by lighting and texture.

[0043] By equipping multispectral or hyperspectral sensors, spectral image data reflecting the composition of ground materials are collected;

[0044] Collect existing structured geographic information data (such as GIS maps and building outlines) and unstructured business text data (such as facility archives and planning reports) related to the area.

[0045] Accurate joint aerial triangulation adjustment and coordinate unification registration are performed on all imagery, point cloud, and vector data to achieve strict spatial alignment of multimodal data.

[0046] Step 2. Perform deep fusion encoding on cross-modal features;

[0047] The multi-source data preprocessed in step 1 is input into a deep feature learning network for hierarchical feature fusion; this deep feature learning network can achieve cross-modal feature complementarity and enhancement.

[0048] Among them, point cloud data is used to extract multi-level geometric structure features through a three-dimensional convolutional neural network; RGB images and multispectral images are used to extract fine-grained texture features and material spectral features of the surface through a two-dimensional convolutional neural network, respectively.

[0049] The formula for the geometric feature tensor of the point cloud is:

[0050] ;

[0051] The formula for inputting a point cloud set is:

[0052] ;

[0053] in, The geometric feature tensor of the point cloud (shape [C, N], where C is the number of channels and N is the number of points); The input is a point cloud set, including N 3D coordinate points. ; It is a 3D point cloud processing network used to extract local geometric features.

[0054] The formula for extracting texture features from RGB images is:

[0055] ;

[0056] The formula for extracting material features from multispectral images is:

[0057] ;

[0058] in, For RGB image texture features (shape [H, W, Cr], where H and W are spatial dimensions, and Cr is the number of channels); The material features of the multispectral image (shape [H, W, Cm], where Cm is the number of spectral channels); High-resolution RGB image; This is a multispectral / hyperspectral image.

[0059] The aforementioned geometric structural features, fine-grained texture features, and material spectral features are weighted and fused through an adaptive attention mechanism to generate a unified feature tensor with highly dense information.

[0060] The formula for attention-weighted fusion is: ;

[0061] ;

[0062] in, For the unified feature tensor after fusion; Let k = p, r, m correspond to point cloud, RGB, and multispectral data, respectively, satisfying the following conditions: ; This is a learnable parameter matrix used to calculate the attention score; To splice multimodal features along the channel dimension; For the geometric feature tensor of the point cloud; RGB image texture features; This refers to the material characteristics of multispectral images.

[0063] Vector data such as GIS data are rasterized and used as a spatial mask for fusion, providing contour constraints.

[0064] Step 3. Multi-task collaborative semantic interpretation and relation inference;

[0065] The fused unified feature tensor is input into an interactive multi-task learning network; three sub-tasks, semantic segmentation, instance segmentation, and spatial relationship reasoning, are executed simultaneously, with the three sub-tasks sharing the underlying fused features.

[0066] The semantic segmentation subtask determines the category of each 3D unit (such as a point or voxel); the instance segmentation subtask further distinguishes different independent individuals within the same category; and the spatial relationship reasoning subtask analyzes the spatial topological relationships (such as adjacency and coverage) and semantic logical relationships between individuals.

[0067] In addition, the three sub-tasks exchange information in real time through an attention interaction module.

[0068] The instance segmentation subtask provides precise object boundary information and feeds it back to the semantic segmentation subtask in real time, helping the semantic segmentation subtask to correct the judgment of ambiguous areas of category boundaries. At the same time, the preliminary semantic and relation judgments are sent to the spatial relation reasoning subtask for logical consistency verification and completion. This closed-loop interaction ensures that the final output semantic map maintains global consistency and rationality in target individual identification, category classification and relation network.

[0069] In this embodiment, the loss functions for the three sub-tasks are:

[0070] ;

[0071] This represents the total loss value. , and This is a task balancing factor, typically adjusted through grid search or adaptive adjustment; For semantic segmentation loss; The instance segmentation loss; Loss due to spatial relation inference.

[0072] The revised semantic segmentation feature formula is as follows:

[0073] ;

[0074] ;

[0075] in, These are the corrected semantic segmentation features; Instance segmentation features; The gating factor controls the injection intensity of instance segmentation information; Use the Sigmoid activation function; These are learnable parameters used to map instance features to gated signals.

[0076] Step 4. Inject open-domain knowledge and perform attribute association; for key targets in instance segmentation output (such as buildings), leverage the powerful zero-shot generalization capability of the visual large model to perform refined component segmentation (such as identifying doors, windows, and roofs), requiring only a very small number of examples for guidance.

[0077] The formula for component segmentation similarity matching is:

[0078] ;

[0079] in, Visual-text similarity score; The CLIP model is used for the visual embedding of the target component image; Text embedding of part names for CLIP models;

[0080] By using a large language model as an intelligent information extraction agent, relevant unstructured text data can be automatically parsed. The large language model can identify entity names, attribute descriptions (such as "built in 20xx", "reinforced concrete structure") and event relationships from the data. Through semantic matching and spatial positioning, this information is automatically linked to the corresponding three-dimensional entity instances, realizing the automatic association and fusion of knowledge.

[0081] The formula for the probability of attribute association is:

[0082] ;

[0083] in, Let e ​​be the probability that entity e is extracted from text d. The classification header parameters map the BERT output to the entity category space; This is the semantic encoding of unstructured text d by the BERT model.

[0084] Step 5. Construct a spatiotemporal knowledge graph to drive generative fine-grained modeling;

[0085] All the information generated in steps 1-4 is structured and integrated, and the expression of the 3D model is optimized.

[0086] The construction of the spatiotemporal knowledge graph involves using each three-dimensional entity instance as a graph node, the relationships obtained in steps 3 and 4 as "edges," and categories and attributes as node features to build a machine-readable and reasonable spatiotemporal knowledge graph containing spatial topology and semantic logic. This spatiotemporal knowledge graph fully describes "who is where, what, what attributes, and what relationship with whom" in the scene.

[0087] The node features are then updated during the construction of the spatiotemporal knowledge graph as follows:

[0088] ;

[0089] in, Let i be the embedding vector of node i in the l-th layer graph; Let i be the set of neighboring nodes of node i; This is an aggregation operation; is the trainable weight matrix of the graph convolutional layer; ReLU is the activation function, introducing non-linearity.

[0090] The formula for the probability score of the existence of an edge is:

[0091] ;

[0092] in, For the boundary Existence probability score; For MLP networks, the input is the concatenated node embeddings. .

[0093] It drives generative fine modeling and uses a physically based generative method for final modeling. Specifically, driven by knowledge graphs and fusion features, it uses 3D Gaussian splashing technology with language embedding to generate a 3D scene model with high visual fidelity, lightweight and tightly coupled semantic and geometric structures. It can effectively repair model holes and supports independent editing and manipulation of entities with semantic tags.

[0094] Step 6. Output a computable model

[0095] The final output is a semantic real-world 3D model that is computable, queryable, and reasonable. Visually, the semantic real-world 3D model is a detailed 3D scene with multi-level semantic tags. Its underlying structure is a spatiotemporal knowledge graph that supports complex spatial semantic queries (such as "find all office buildings with solar panels on their roofs and located near main roads"). By integrating an LLM agent, users can directly interact with the 3D model using natural language to conduct intelligent question answering, analysis, and decision support.

[0096] Example 1

[0097] In this embodiment, the goal is to construct a full-element semantic 3D model of a new urban area with centimeter-level precision; the specific execution is as follows:

[0098] First, perform step 1 to simultaneously acquire oblique photogrammetry images, airborne LiDAR point clouds, hyperspectral images, and urban planning GIS databases for the area.

[0099] Through steps 2 and 3, the system automatically and accurately segments each building, each road, each street lamp, and each green space from the fused data, and establishes a relationship network such as "roads connecting buildings" and "vehicles parked in parking spaces".

[0100] In step 4, the system calls the LLM agent to automatically parse the project approval documents and associate attribute information such as "construction unit", "planned use" and "number of floors" for each building entity;

[0101] Finally, through steps 5 and 6, a computable 3D smart city foundation is generated that can be used for applications such as urban planning simulation and urban management.

[0102] This invention significantly reduces the cost and time required for professional manual annotation; the multi-task collaboration mechanism ensures a high degree of consistency between the interpretation results in logic and space; it can continuously absorb new knowledge from business data and expand the semantic system; the output 3D model has higher quality and usability, achieving a unity of visual realism, geometric accuracy and semantic computability.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automated method for extracting real-scene 3D semantic information based on multi-source data fusion, characterized in that, Includes the following steps: Step 1. Standardize and preprocess the multi-source data to obtain oblique photogrammetric images, laser scanning point clouds, multispectral images, and structured or unstructured auxiliary data covering the same target area, and perform spatiotemporal benchmark unification and registration. Step 2. Perform cross-modal feature deep fusion encoding on the multi-source data processed in Step 1. Extract point cloud geometric features, image texture features and material spectral features through a deep feature learning network, and use an adaptive attention mechanism for weighted fusion to generate a unified fusion feature tensor. Step 3. Input the fused feature tensor into the multi-task collaborative learning network, and simultaneously execute the three sub-tasks of semantic segmentation, instance segmentation and spatial relationship reasoning. The attention interaction module is used to realize information interaction and result correction between tasks. Step 4. Based on the instance segmentation results and the corresponding semantic category information, use the visual big model to perform refined component recognition of key targets, and call the big language model to parse unstructured text, and associate the extracted attribute information with the corresponding three-dimensional entity instance; Step 5. Construct a spatiotemporal knowledge graph using 3D entity instances as nodes and semantic and spatial relationships as edges. Based on the semantic information and geometric attributes of entity nodes in the spatiotemporal knowledge graph, and combined with the global and local fusion features obtained in Step 2, drive generative fine modeling to generate a 3D scene model that couples semantic and geometric structures. Step 6. Output a semantic real-world 3D model with semantic tags that is queryable and reasonable.

2. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, The specific process of cross-modal feature deep fusion encoding in step 2 includes: Geometric features of point clouds are extracted using a 3D convolutional neural network, and texture features of RGB images and material features of multispectral images are extracted using a 2D convolutional neural network. An adaptive attention mechanism is then used to perform weighted fusion of the multimodal features. The adaptive attention-weighted fusion calculation formula is as follows: ； ； in, For the unified feature tensor after fusion; Let k = p, r, m correspond to point cloud, RGB, and multispectral data, respectively, satisfying the following conditions: ; This is a learnable parameter matrix used to calculate the attention score; To splice multimodal features along the channel dimension; For the geometric feature tensor of the point cloud; RGB image texture features; This refers to the material characteristics of multispectral images.

3. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, In step 3, the multi-task collaborative learning network is trained using the following loss function: ； This represents the total loss value. , and All are task balance coefficients; For semantic segmentation loss; The instance segmentation loss; Loss due to spatial relation inference.

4. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, In step 4, when segmenting components using a large visual model, visual-text similarity matching is employed to achieve zero-sample or small-sample guided recognition. The similarity calculation formula is as follows: ； in, Visual-text similarity score; The CLIP model is used for the visual embedding of the target component image; Text embedding of part names for CLIP models.

5. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, In step 4, when parsing unstructured text using a large language model, entity and attribute information is extracted through a semantic coding model, and this information is spatially semantically matched and associated with three-dimensional entity instances.

6. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, In step 5, when constructing the spatiotemporal knowledge graph, a graph convolutional network is used to update node features. The node feature update formula is as follows: ； in, Let i be the embedding vector of node i in the l-th layer graph; Let i be the set of neighboring nodes of node i; This is an aggregation operation; The trainable weight matrix for the graph convolutional layer; A nonlinearity is introduced for the activation function.

7. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, In step 5, the generative fine modeling is driven by a generative method based on physical rendering, which combines knowledge graphs and fusion features to generate a visually faithful, lightweight, and semantically complete 3D scene model through 3D Gaussian splashing technology.

8. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, The semantic real-world 3D model output in step 6 supports natural language interaction, and users can perform semantic queries, scene analysis, and decision support through the integrated large language model intelligent agent.

9. The method for automated extraction of real-scene 3D semantic information based on multi-source data fusion according to claim 1, characterized in that, The multi-source data also includes existing GIS vector data, which is rasterized into a spatial mask in step 2 to provide contour constraints and participate in feature fusion.

Images