Multi-modal remote sensing data driven refined city three-dimensional reconstruction method and system thereof

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to a multi-mode remote sensing data driven refined city three-dimensional reconstruction method and system, and the method comprises the steps: obtaining an original satellite optical image for providing city surface information; synthetic aperture radar (SAR) data is obtained and used for assisting reconstruction of a point cloud data missing region; the method comprises the following steps: acquiring laser radar data carried by a flight aerial carrier for acquiring high-precision three-dimensional point cloud; performing data preprocessing; carrying out fusion on the preprocessed multi-modal data; on the basis of the fused features, complementing space detail information; on the basis of the refined features, progressive reconstruction is carried out, vertexes and edges are gradually added from the initial vertexes, and a grid model is constructed; in the reconstruction process, performing semantic segmentation on the grid model by using a multi-category segmentation network, and judging the category of each triangular patch; time sequence reconstruction is completed through fusion of multi-temporal data, and a dynamic city change model is generated; and complementation of advantages of different modal data is realized.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and more specifically, to a multi-modal remote sensing data-driven refined urban three-dimensional reconstruction method and system. Background Art

[0002] With the rapid development of modern transportation infrastructure, the maintenance and management of road surfaces have become particularly important. With the rapid development of remote sensing technology and geographic information systems, urban 3D reconstruction has become one of the key technologies in the fields of smart city construction, disaster assessment, urban planning, etc. The current technical path mainly relies on a single data source or a limited multimodal fusion method, and has a significant lack of ability to express the geometric structure, semantic information, and temporal changes of complex urban scenes. In areas with high building density or diverse terrain conditions, the reconstruction results of traditional methods are often difficult to achieve high accuracy and high consistency requirements, which seriously limits its practical application effect.

[0003] In the existing technology, satellite optical images are widely used to extract urban surface features, with the advantages of large coverage and rich texture details. However, due to the sensitivity of optical images to weather conditions and shadows, this single-modality data is difficult to overcome the problem of missing point cloud data. At the same time, synthetic aperture radar (SAR), as a microwave remote sensing technology, can penetrate clouds and part of the surface, but its feature expression ability is weak, and there is ambiguity in the description of the geometric characteristics of complex urban terrain. In addition, although lidar point cloud data can provide high-precision spatial structure information, there is unevenness in the processing of data-dense areas and sparse areas, especially when aligning multi-source data, it is easy to introduce errors.

[0004] In recent years, multimodal data fusion technology has begun to be applied to the field of 3D reconstruction. This method attempts to combine the advantages of different modal data to improve the accuracy and consistency of the model. However, due to the significant differences in resolution, scale, geometric structure, etc. between different modal data, existing methods have significant technical bottlenecks in the fusion process. For example, there are problems such as error accumulation during multimodal data registration, imperfect processing of information redundancy and complementarity during the fusion process, and insufficient expression of sparse data area features. The existence of these problems directly leads to deficiencies in the reconstruction results in terms of geometric accuracy, texture consistency, and dynamic expression.

[0005] On the other hand, existing 3D reconstruction methods lack sufficient exploration of the synergistic effects between data processing steps in algorithm design. For example, the quality of data preprocessing directly affects the accuracy of subsequent feature extraction and fusion, but existing methods fail to achieve adaptive optimization in the connection between preprocessing and feature refinement. In addition, the real-time update of vertex density distribution and semantic segmentation in progressive reconstruction often leads to local incoherence of the results due to the lack of an effective dynamic feedback mechanism.

[0006] In dynamic urban change scenarios, existing methods usually rely on static point cloud data and find it difficult to effectively model temporal changes. In particular, in terms of multi-temporal data integration and dynamic 3D reconstruction, existing technologies lack a consistent temporal registration mechanism and efficient temporal feature expression, resulting in insufficient temporal and spatial consistency of the model. Summary of the invention

[0007] In view of the above problems, the present invention proposes a multimodal remote sensing data-driven refined urban 3D reconstruction method and system, and strives to solve the deficiencies in the prior art through deep fusion of multimodal data, dynamic feature refinement and progressive modeling.

[0008] The present invention provides a multi-modal remote sensing data driven refined urban three-dimensional reconstruction method, comprising:

[0009] The acquisition steps include:

[0010] Acquire original satellite optical images to provide urban surface information;

[0011] Acquire synthetic aperture radar (SAR) data to assist in the reconstruction of areas where point cloud data is missing;

[0012] Obtain the laser radar data carried by the flying aircraft to obtain high-precision three-dimensional point clouds;

[0013] Processing steps include:

[0014] Based on the acquired satellite optical images, SAR data and lidar point cloud data, data preprocessing is performed, wherein the data preprocessing includes format conversion, data filtering, registration and feature extraction;

[0015] fusing the preprocessed multimodal data, wherein the fusion includes data splicing, cross-modal feature fusion and data enhancement;

[0016] Based on the fused features, graph neural networks are used to refine the data features and complete the spatial detail information;

[0017] Reconstruction steps include:

[0018] Based on the refined features, progressive reconstruction is performed, gradually adding vertices and edges from the initial vertices to build a mesh model;

[0019] During the reconstruction process, a multi-category segmentation network is used to perform semantic segmentation on the mesh model to determine the category of each triangle.

[0020] Through the fusion of multi-temporal data, time series reconstruction is completed to generate a dynamic urban change model;

[0021] Output steps include:

[0022] Use physically based rendering technology to render the reconstructed 3D mesh model to generate a visual 3D city model;

[0023] Output the rendered 3D reconstruction of the city.

[0024] Preferably, the obtaining step specifically includes:

[0025] The acquired satellite optical images include infrared, visible light and multispectral images, which are used to provide texture and color information;

[0026] SAR data were collected in dual-polarization mode to enhance spatial resolution;

[0027] LiDAR data acquisition includes multi-view point cloud data collected from different angles to improve the integrity of stereo information.

[0028] Preferably, the data preprocessing specifically includes:

[0029] The noise reduction algorithm based on deep learning is used to filter the noise of point cloud data;

[0030] Use feature point-based registration technology to align data from different modalities;

[0031] Extract geometric features, semantic information and material features to form structured point cloud data.

[0032] Preferably, the cross-modal feature fusion specifically includes:

[0033] Use cross-modal attention mechanism to strengthen the complementary information between different modal features;

[0034] The fused features are enhanced at multiple scales to improve the robustness and consistency of the features.

[0035] Preferably, the feature refinement specifically includes:

[0036] Utilize node relationship modeling in graph neural networks to optimize point cloud feature expression;

[0037] The point cloud data in sparse areas is weighted with specific weights to complete the spatial detail information.

[0038] Preferably, the progressive reconstruction specifically includes:

[0039] Dynamically adjust vertex density through region growing algorithm;

[0040] The grid structure is optimized based on the smoothness constraint of neighborhood features.

[0041] Preferably, the multi-category segmentation network specifically includes:

[0042] Using the multi-scale feature fusion module, the features generated from multiple angles are concatenated and fused through a 1×1 convolutional layer;

[0043] Multi-category classification algorithm based on deep learning, updating classification information in real time.

[0044] Preferably, the timing reconstruction specifically includes:

[0045] The point cloud data collected at different time phases are registered to generate a disparity map with multi-view consistency;

[0046] The depth map is calculated based on the disparity map and mapped to the three-dimensional space through the triangulation method.

[0047] Preferably, the physically based rendering specifically includes:

[0048] Calculate the light intensity, reflectivity and material color of each mesh node;

[0049] Generate high-precision visualization images based on rendering equations optimized by deep learning.

[0050] The multimodal remote sensing data-driven refined urban 3D reconstruction system based on the method includes:

[0051] Multimodal data acquisition module, used to acquire satellite optical images, SAR data and lidar data;

[0052] Data preprocessing module, used for filtering, registering and feature extraction of acquired data;

[0053] Cross-modal feature fusion module, used to fuse and enhance multimodal data;

[0054] Feature refinement module, used to refine the fused features;

[0055] A progressive reconstruction module is used to generate a 3D mesh model based on the refined features;

[0056] Multi-class segmentation network module, used for semantic segmentation of 3D mesh models;

[0057] A time series reconstruction module is used to generate a city dynamic model based on multi-temporal data;

[0058] A rendering module, used to generate a visual 3D model based on physical rendering technology;

[0059] The output module is used to output the reconstructed three-dimensional city model.

[0060] Specifically, the present invention realizes the complementary advantages of different modal data through the organic combination of satellite optical images, SAR data and lidar point cloud data. Data preprocessing ensures the high quality of input data through denoising, registration and feature extraction. Based on the collaborative application of cross-modal feature fusion and graph neural network, the method of the present invention can refine data features and complete the geometric details of sparse areas. Furthermore, progressive reconstruction combined with a multi-category semantic segmentation network realizes the dynamic construction and real-time optimization of the grid model. Especially in the modeling of dynamic urban changes, the present invention provides an efficient time series reconstruction solution through the fusion of multi-phase point cloud data.

[0061] The beneficial effect of the present invention is that, through the complementarity and synergy between the steps, the geometric accuracy, semantic consistency and temporal dynamic expression of the three-dimensional reconstruction results are significantly improved. For example, cross-modal feature fusion can effectively utilize the texture information of optical images and the spatial characteristics of SAR data to achieve a high degree of complementarity between modal information; the introduction of graph neural networks has demonstrated superior feature expression capabilities in the reconstruction of sparse point cloud data; progressive modeling combined with dynamic vertex density adjustment and semantic segmentation optimization makes the mesh model more natural and smooth in complex urban scenes.

[0062] Therefore, the present invention provides an innovative method and system, which not only overcomes the defects of the prior art, but also further expands the application scope of urban three-dimensional reconstruction in dynamic scenes, and provides reliable technical support for fields such as smart city construction, disaster assessment and urban planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a logic block diagram of the method of the present invention.

[0064] Figure 2 This is a logic block diagram of the data preprocessing module of the present invention.

[0065] Figure 3 This is a logical block diagram of the cross-modal feature fusion module of the present invention.

[0066] Figure 4 This is a logic block diagram of the feature refinement module of the present invention.

[0067] Figure 5It is a logic block diagram of the progressive reconstruction module of the present invention.

[0068] Figure 6 It is a logic block diagram of the multi-category segmentation network module of the present invention.

[0069] Figure 7 It is a logic block diagram of the timing reconstruction module of the present invention. DETAILED DESCRIPTION

[0070] Please refer to Figure 1-7 The present invention discloses a method for fine-grained urban three-dimensional reconstruction driven by multimodal remote sensing data, comprising: an acquisition step, including acquiring original satellite optical images for providing urban surface information; acquiring synthetic aperture radar (SAR) data for assisting in the reconstruction of areas where point cloud data is missing; and acquiring laser radar data carried by a flying aircraft for obtaining high-precision three-dimensional point clouds.

[0071] The processing steps include: performing data preprocessing based on the acquired satellite optical images, SAR data and lidar point cloud data, wherein the data preprocessing includes format conversion, data filtering, alignment and feature extraction; fusing the preprocessed multimodal data, wherein the fusion includes data splicing, cross-modal feature fusion and data enhancement; and based on the fused features, using graph neural networks to refine the data features and complete the spatial detail information.

[0072] The reconstruction step includes: based on the refined features, progressive reconstruction is performed, and vertices and edges are gradually added from the initial vertices to construct a mesh model;

[0073] During the reconstruction process, a multi-category segmentation network is used to perform semantic segmentation on the mesh model to determine the category of each triangle.

[0074] By integrating multi-temporal data, time series reconstruction is completed and a dynamic urban change model is generated.

[0075] The output step includes: rendering the reconstructed three-dimensional grid model using a physically based rendering technique to generate a visualized three-dimensional city model; and outputting the rendered three-dimensional city reconstruction result.

[0076] The present invention relates to a refined urban three-dimensional reconstruction method based on multimodal remote sensing data, especially in scenes with complex urban terrain, dense buildings and diverse data sources, by integrating satellite optical images, SAR data and lidar point cloud data, high-precision and dynamic reconstruction of urban three-dimensional scenes can be achieved.

[0077] Preferably, in one embodiment of the present invention, the acquisition step includes using high-resolution satellite optical images with a resolution of 0.3 meters; Synthetic Aperture Radar (SAR) data uses dual polarization mode, and the resolution is preferably 1 meter, which is used to enhance surface texture and three-dimensional perception; laser radar data is acquired through a LiDAR system mounted on a drone, and the scanning frequency is preferably 50Hz, and the point cloud accuracy is ±1 cm. Through this multi-modal data acquisition method, it is ensured that the input data of the three-dimensional reconstruction model has high resolution and high quality.

[0078] In the processing step, data preprocessing uses a denoising algorithm (such as a three-dimensional Gaussian filter) to remove noise in the point cloud; data registration is based on a multi-feature point registration method using the formula

[0079]

[0080] Among them, E is the registration error, R is the rotation matrix, t is the translation vector, and p i and q i They represent the corresponding matching point pairs in the point cloud.

[0081] Multimodal data fusion is preferably achieved through a cross-attention mechanism. The optical image feature is represented as F optical , SAR feature is expressed as F SAR ,The fused features are expressed as:

[0082] F fused =αF optical +βF SAR

[0083] Among them, α and β are weight coefficients, which are dynamically adjusted according to data distribution. Through this fusion method, the consistency and integrity of the data are significantly improved.

[0084] In one embodiment of the present invention, the obtaining step specifically includes:

[0085] The acquired satellite optical images include infrared, visible light and multispectral images, which are used to provide texture and color information;

[0086] SAR data were collected in dual-polarization mode to enhance spatial resolution;

[0087] LiDAR data acquisition includes multi-view point cloud data collected from different angles to improve the integrity of stereo information.

[0088] In a preferred embodiment of the present invention, the acquisition step is further refined into the joint acquisition of multimodal information, including but not limited to the combined application of infrared spectral data and multispectral optical data. Infrared images are used to perceive the thermal radiation characteristics of the urban surface, and multispectral images are used to identify vegetation coverage and building material characteristics. The acquisition resolution is preferably 0.5 meters.

[0089] SAR data collection uses dual polarization mode (such as HH and VV polarization), which enhances the reflection intensity of building facade features on the one hand, and reduces ground noise interference on the other hand. The collection of lidar data preferably sets multi-viewpoint scanning angles, such as collecting from the north and west at 30 degrees respectively, and the coverage rate of the generated point cloud data is increased to more than 95%.

[0090] The advantage of this multimodal information acquisition method is that it can not only generate a complete urban spatial structure, but also provide more detailed geometric and material information in complex scenes (such as areas with dense high-rise buildings).

[0091] In a preferred embodiment of the present invention, the data preprocessing specifically includes:

[0092] The noise reduction algorithm based on deep learning is used to filter the noise of point cloud data;

[0093] Use feature point-based registration technology to align data from different modalities;

[0094] Extract geometric features, semantic information and material features to form structured point cloud data.

[0095] Preferably, in one embodiment of the present invention, noise filtering of point cloud data is implemented by a deep learning network, such as a PointNet++ network. The input point cloud data is represented as:

[0096] P={p1,p2,…,p n},p i =(x i ,y i ,z i ,f i )

[0097] Among them, p i is a point cloud data point, x i ,y i ,z i is the three-dimensional coordinate, f i is the feature of the point (such as reflectance or color value).

[0098] The registration process is optimized using the iterative closest point algorithm (ICP), and the objective function is:

[0099]

[0100] Where E is the error function, T(p i ) is the point after transformation, q i is the target point.

[0101] Geometric features (such as edge points and plane features), semantic features (such as buildings and roads) and material features (such as reflectivity and color) are extracted through feature extraction module 3. The extracted semantic features are further represented as high-dimensional vectors for subsequent cross-modal fusion processing.

[0102] The advantage of this preprocessing step is that the combination of noise filtering and feature extraction significantly improves the completeness and accuracy of the data, especially in densely built urban environments, ensuring high accuracy of the final reconstructed model.

[0103] In a preferred embodiment of the present invention, the cross-modal feature fusion specifically includes:

[0104] Use cross-modal attention mechanism to strengthen the complementary information between different modal features;

[0105] The fused features are enhanced at multiple scales to improve the robustness and consistency of the features.

[0106] In the process of cross-modal feature fusion, the method of the present invention significantly enhances the utilization efficiency of complementary information of different modal data by introducing a cross-modal attention mechanism. Preferably, in one embodiment of the present invention, the cross-modal attention mechanism is implemented as follows: first, the input optical image feature F is calculated. optical and SAR feature F SAR The correlation matrix A between them is calculated as follows:

[0107]

[0108] Among them, Q i is the query vector of optical features, K j is the key vector of SAR features, d k is the normalization parameter of the feature dimension. The final fusion feature is expressed as:

[0109] F fused =A·V

[0110] Among them, V is the value vector of SAR features, and the weighted features are generated after matrix multiplication.

[0111] The fused features are preferably optimized through a multi-scale enhancement module. In this method, a three-layer convolutional network is selected, and the sizes of the convolution kernels in each layer are 1x1, 3x3, and 5x5, respectively, to capture spatial information of different scales. This module significantly improves the robustness of the fused features, especially in scenes with complex building boundaries and dense road intersections, which can ensure clear edges and accurate classification of the reconstruction results.

[0112] Through the above design, the method of the present invention realizes efficient interaction and enhancement of information between modalities, overcomes the limitations of single modality data in feature expression, and provides high-quality input for subsequent feature refinement and three-dimensional reconstruction.

[0113] In a preferred embodiment of the present invention, the feature refinement specifically includes:

[0114] Utilize node relationship modeling in graph neural networks to optimize point cloud feature expression;

[0115] The point cloud data in sparse areas is weighted with specific weights to complete the spatial detail information.

[0116] In the embodiment of the present invention, feature refinement is implemented based on a graph neural network (GNN), which is particularly suitable for processing feature completion in sparse areas of point clouds. Preferably, the method of the present invention represents the point cloud as an undirected graph G = (V, E), where V is a point set, E is an edge set, and the feature h of the point is v It is expressed as:

[0117]

[0118] in, is the neighbor set of point v, σ is the activation function, W (k) is the weight matrix of the kth layer. Through multi-layer propagation, the point cloud features are gradually completed, especially the geometric details of the sparse areas are restored. Preferably, the method of the present invention weights the sparse points with specific weights, and the weights are dynamically adjusted by the point cloud density control function:

[0119]

[0120] Among them, w v is the weighting coefficient of point v, d v is the neighbor density of the point, μ is the density threshold, and λ is the adjustment coefficient. This method can effectively complete the spatial details of the sparse area and make the overall reconstruction result more coherent and complete.

[0121] In an embodiment of the present invention, the progressive reconstruction specifically includes:

[0122] Dynamically adjust vertex density through region growing algorithm;

[0123] The grid structure is optimized based on the smoothness constraint of neighborhood features.

[0124] The progressive reconstruction method of the present invention adopts a region growing algorithm, starting from the initial vertex, and generates a mesh model by dynamically adding vertices and edges. Preferably, the method of the present invention selects a point cloud dense area as the initial growing point, and the vertex selection rules of the growing process are as follows:

[0125]

[0126] Among them, C is the current unselected vertex set, p current is the current vertex, and ∥·∥ represents the Euclidean distance.

[0127] Preferably, to ensure the smoothness of the grid structure, the method of the present invention performs a ij Adding smoothness constraints, the optimization objective function is:

[0128]

[0129] Among them, E smooth is the smoothness error, p i ,p j are the coordinates of vertices i and j respectively. This constraint can avoid excessive distortion of the mesh and ensure that the generated 3D model still has good geometric continuity in complex structural areas.

[0130] The multi-category segmentation network specifically includes:

[0131] Using the multi-scale feature fusion module, the features generated from multiple angles are concatenated and fused through a 1×1 convolutional layer;

[0132] Multi-category classification algorithm based on deep learning, updating classification information in real time.

[0133] In a preferred embodiment of the present invention, the multi-class segmentation network significantly improves the classification accuracy by introducing a multi-scale feature fusion module. Preferably, the module splices features from multiple angles, and the spliced ​​features are represented as:

[0134] F concat =[F1,F2,…,F k ]

[0135] Among them, F i The feature generated for the i-th angle has a concatenated feature dimension that is k times the original feature dimension. After fusion, the feature is reduced in dimension through a 1×1 convolutional layer, and the output is:

[0136] F output =W·F concat +b

[0137] Among them, W is the convolution kernel and b is the bias.

[0138] Real-time classification updates are achieved through a deep learning multi-class classification algorithm using a cross entropy loss function:

[0139]

[0140] Among them, C is the number of categories, y c is the true label, By continuously optimizing the loss function, the classification boundary can be adjusted in real time to ensure the accuracy and consistency of the classification results.

[0141] The invented method shows significant advantages in cross-modal fusion, feature refinement, progressive reconstruction and multi-category segmentation.

[0142] In an embodiment of the present invention, the timing reconstruction specifically includes:

[0143] The point cloud data collected at different time phases are registered to generate a disparity map with multi-view consistency;

[0144] The depth map is calculated based on the disparity map and mapped to the three-dimensional space through the triangulation method.

[0145] In the preferred embodiment of the present invention, in order to achieve the high-precision requirements of urban dynamic 3D reconstruction, the time series reconstruction adopts a method based on time phase point cloud registration and multi-view consistency. First, the point cloud data of different time phases are input into the registration module, which aligns the data through an optimization algorithm based on feature points. The error function of the registration is expressed as:

[0146]

[0147] Among them, T(p i ) is the transformed point cloud data, q i is the point cloud matching point of the reference phase, E align is the registration error. Through this optimization function, a unified data set of multi-phase point clouds is generated. The registered point cloud data is used to generate a disparity map. Preferably, this method combines the multimodal information of optical images and point cloud data, and the disparity map calculation formula is as follows:

[0148]

[0149] Where D(x,y) is the disparity value of the pixel point (x,y), f is the focal length of the camera, B is the baseline length, and Z is the depth value of the pixel point. Based on the disparity map, the depth value is mapped to the three-dimensional space using triangulation to generate the three-dimensional point cloud data of the corresponding phase.

[0150] Through the method of the present invention, point cloud data of different time phases can be integrated and reconstructed, especially in scenarios that require dynamic models such as urban development or post-disaster assessment, which significantly improves the completeness and accuracy of the model's time dimension information.

[0151] In a preferred embodiment of the present invention, the physically based rendering specifically includes:

[0152] Calculate the light intensity, reflectivity and material color of each mesh node;

[0153] Generate high-precision visualization images based on rendering equations optimized by deep learning.

[0154] Preferably, the method of the present invention adopts physically based rendering technology in the rendering process to enhance the authenticity and visualization effect of the 3D reconstructed model. The core of the rendering process lies in the accurate calculation of the light intensity, reflectivity and material color of each grid node.

[0155] In a preferred embodiment of the present invention, the illumination intensity of a grid node is calculated by the following formula:

[0156] I=I a +I d (N·L)+I s ·(R·V) n

[0157] Where I is the total light intensity of the node, I a is the ambient light intensity, I d is the diffuse reflected light intensity, I s is the intensity of specular light, N is the surface normal vector, L is the light direction, R is the reflection direction, V is the viewing direction, and n is the smoothness index.

[0158] The material color is obtained through the texture mapping of the node, and the lighting information is combined to generate a high-precision node material representation. The rendering equation based on deep learning optimization further improves the rendering efficiency and visual effects. The rendering equation is:

[0159]

[0160] Among them, C is the pixel color value, K is the rendering function, T is the number of sampling times, and I t is the light intensity, ρ t is the reflectivity.

[0161] Through the above rendering technology, the three-dimensional reconstruction model of the present invention is more realistic in visual effect and is suitable for scenes such as virtual reality, urban planning and disaster assessment.

[0162] The present invention also discloses a multi-modal remote sensing data-driven refined urban three-dimensional reconstruction system corresponding to the method, comprising:

[0163] Multimodal data acquisition module 1, used to acquire satellite optical images, SAR data and lidar data;

[0164] Data preprocessing module 2, used for filtering, registering and feature extraction of acquired data;

[0165] Cross-modal feature fusion module 3, used to fuse and enhance multimodal data;

[0166] Feature refinement module 4, used for refining the fused features;

[0167] A progressive reconstruction module 5, for generating a three-dimensional mesh model based on the refined features;

[0168] A multi-class segmentation network module 6, used for semantic segmentation of the three-dimensional mesh model;

[0169] A time series reconstruction module 7, used to generate a city dynamic model based on multi-temporal data;

[0170] A rendering module 8, used to generate a visual three-dimensional model based on physical rendering technology;

[0171] The output module 9 is used to output the reconstructed three-dimensional city model.

[0172] The system design of the present invention fully considers the technical requirements of multimodal data processing and three-dimensional reconstruction. Preferably, the multimodal data acquisition module 1 is connected to a satellite imaging system, a SAR data collector and a laser radar carried by an unmanned aerial vehicle to collect high-resolution surface data.

[0173] The data preprocessing module 2 combines denoising, registration and feature extraction algorithms to provide high-quality point cloud and image data input for subsequent modules. The cross-modal feature fusion module 3 uses a cross-attention mechanism to achieve efficient fusion of optical image and SAR data features, providing a unified feature expression for the feature refinement module 4.

[0174] Preferably, the feature refinement module 4 enhances the spatial details of the sparse area point cloud data through the graph neural network; the progressive reconstruction module 5 realizes the gradual generation and optimization of the mesh model by dynamically adjusting the vertex density; the multi-category segmentation network module 6 further performs semantic segmentation on the mesh model to ensure that the category labeling of the reconstruction results is accurate.

[0175] The time reconstruction module 7 supports the analysis and reconstruction of dynamic urban changes. The rendering module 8 outputs a high-precision visual three-dimensional model based on the optimized physical rendering equation, and finally generates results suitable for application scenarios such as planning and simulation through the output module 9.

[0176] Through the collaboration of the above modules, the system of the present invention performs excellently in data processing and reconstruction accuracy, and has wide applicability and high innovation.

[0177] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A refined urban 3D reconstruction method driven by multimodal remote sensing data, characterized in that: include: The acquisition steps include: Acquire original satellite optical images to provide urban surface information; Acquire synthetic aperture radar (SAR) data to assist in the reconstruction of areas where point cloud data is missing; Obtain the laser radar data carried by the flying aircraft to obtain high-precision three-dimensional point clouds; Processing steps include: Based on the acquired satellite optical images, SAR data and lidar point cloud data, data preprocessing is performed, wherein the data preprocessing includes format conversion, data filtering, registration and feature extraction; fusing the preprocessed multimodal data, wherein the fusion includes data splicing, cross-modal feature fusion and data enhancement; Based on the fused features, graph neural networks are used to refine the data features and complete the spatial detail information; Reconstruction steps include: Based on the refined features, progressive reconstruction is performed, gradually adding vertices and edges from the initial vertices to build a mesh model; During the reconstruction process, a multi-category segmentation network is used to perform semantic segmentation on the mesh model to determine the category of each triangle. Through the fusion of multi-temporal data, time series reconstruction is completed to generate a dynamic urban change model; Output steps include: Use physically based rendering technology to render the reconstructed 3D mesh model to generate a visual 3D city model; Output the rendered 3D reconstruction of the city.

2. The method according to claim 1, characterized in that The acquisition step specifically includes: The acquired satellite optical images include infrared, visible light and multispectral images, which are used to provide texture and color information; SAR data were collected in dual-polarization mode to enhance spatial resolution; LiDAR data acquisition includes multi-view point cloud data collected from different angles to improve the integrity of stereo information.

3. The method according to claim 1, characterized in that The data preprocessing specifically includes: The noise reduction algorithm based on deep learning is used to filter the noise of point cloud data; Use feature point-based registration technology to align data from different modalities; Extract geometric features, semantic information and material features to form structured point cloud data.

4. The method according to claim 1, characterized in that: The cross-modal feature fusion specifically includes: Use cross-modal attention mechanism to strengthen the complementary information between different modal features; The fused features are enhanced at multiple scales to improve the robustness and consistency of the features.

5. The method according to claim 1, characterized in that The feature refinement specifically includes: Utilize node relationship modeling in graph neural networks to optimize point cloud feature expression; The point cloud data in sparse areas is weighted with specific weights to complete the spatial detail information.

6. The method according to claim 1, characterized in that The progressive reconstruction specifically includes: Dynamically adjust vertex density through region growing algorithm; The grid structure is optimized based on the smoothness constraint of neighborhood features.

7. The method according to claim 1, characterized in that The multi-category segmentation network specifically includes: Using the multi-scale feature fusion module, the features generated from multiple angles are concatenated and fused through a 1×1 convolutional layer; Multi-category classification algorithm based on deep learning, updating classification information in real time.

8. The method according to claim 1, characterized in that The timing reconstruction specifically includes: The point cloud data collected at different time phases are registered to generate a disparity map with multi-view consistency; The depth map is calculated based on the disparity map and mapped to the three-dimensional space through the triangulation method.

9. The method according to claim 1, characterized in that: The physically based rendering specifically includes: Calculate the light intensity, reflectivity and material color of each mesh node; Generate high-precision visualization images based on rendering equations optimized by deep learning.

10. A refined urban 3D reconstruction system driven by multimodal remote sensing data based on the method according to any one of claims 1 to 9, characterized in that: include: Multimodal data acquisition module, used to acquire satellite optical images, SAR data and lidar data; Data preprocessing module, used for filtering, registering and feature extraction of acquired data; Cross-modal feature fusion module, used to fuse and enhance multimodal data; Feature refinement module, used to refine the fused features; A progressive reconstruction module is used to generate a 3D mesh model based on the refined features; Multi-class segmentation network module, used for semantic segmentation of 3D mesh models; A time series reconstruction module is used to generate a city dynamic model based on multi-temporal data; A rendering module, used to generate a visual 3D model based on physical rendering technology; The output module is used to output the reconstructed three-dimensional city model.

Images