Multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method

Abstract

The application discloses a multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method, relates to the technical field of satellite image three-dimensional modeling, and comprises the following steps: obtaining multi-source input data for three-dimensional modeling, performing object-level feature extraction, and outputting a unified object set; performing registration on the unified object set based on a generative model to obtain fusion data; performing change detection on the fusion data, constructing an initial three-dimensional model according to a detection result by using a local incremental modeling algorithm, and performing global consistency optimization; performing defect identification on the optimized initial three-dimensional model and performing correction to output a final three-dimensional model; and the application solves the problems of obvious three-dimensional model boundary seams, discontinuous texture and untimely change area updating in the prior art through object-level multi-source data fusion, high-precision registration of the generative model, change adaptive incremental modeling and local geometry and texture iterative correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of satellite image three-dimensional modeling, more particularly, the present application relates to a multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method. BACKGROUND

[0002] With the improvement of high-resolution satellite image acquisition capability, the demand for high-precision three-dimensional models in the fields of three-dimensional city modeling, land surface change monitoring and remote sensing data analysis is increasing. Traditional three-dimensional modeling methods mainly rely on single sensor data, such as optical images or radar images, to construct three-dimensional models through stereo image matching, point cloud generation and surface reconstruction. However, single data source has limitations in spatial coverage, resolution and observation angle, making it difficult to ensure both geometric accuracy and texture continuity of the model.

[0003] To solve the problem of insufficient modeling accuracy of single-source data, multi-source data fusion methods have been proposed in recent years, including optical image and laser radar point cloud fusion, optical image and multi-temporal remote sensing image fusion. However, the existing technology still has the following main defects in practical application:

[0004] Existing methods mostly rely on pixel-level or point cloud-level features for fusion, and object-level information is not fully utilized. There is a lack of unified modeling processing of complete structure information of buildings or ground objects, resulting in geometric deviations or texture discontinuities in the three-dimensional model at the boundary or local details. And multi-source data registration usually uses rigid transformation or feature point-based optimization method, which is difficult to fully consider global structure constraints and local geometric relationships, resulting in problems such as local misplacement, obvious joints or topological discontinuity in the generated initial three-dimensional model. Moreover, even after global optimization, there may still be small joints or texture discontinuities at the model boundary, and the existing technology lacks a high-precision correction mechanism for these local defects, making it difficult to meet the visual and structural requirements of high-precision three-dimensional modeling.

[0005] To solve the above problems, the present application provides a solution. SUMMARY

[0006] To overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method, which solves the problems of obvious three-dimensional model boundary joints, texture discontinuity and delayed change area update in the prior art through object-level multi-source data fusion, high-precision model registration, change adaptive incremental modeling and local geometric and texture iterative correction.

[0007] To achieve the above-mentioned purposes, the present application provides the following technical solutions:

[0008] In a first aspect, the application provides a multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method, which comprises: acquiring multi-source input data for three-dimensional modeling, performing object-level feature extraction, and outputting a unified object set; registering the unified object set based on a generative model to obtain fusion data; performing change detection on the fusion data, constructing an initial three-dimensional model according to the detection result using a local incremental modeling algorithm, and performing global consistency optimization; identifying defects in the optimized initial three-dimensional model and correcting the defects to output a final three-dimensional model.

[0009] In one embodiment, multi-source input data for three-dimensional modeling is acquired, object-level feature extraction is performed, and a unified object set is output. Specifically, the multi-source input data is preprocessed, the processed multi-source input data is respectively constructed into independent branch networks, and multi-branch feature extraction is performed; the multi-branch feature extraction results are fused in multiple modalities to generate preliminary segmentation results; the preliminary segmentation results are segmented at the object level based on a conditional random field optimization segmentation method to output the unified object set.

[0010] In one embodiment, the preliminary segmentation results are segmented at the object level based on a conditional random field optimization segmentation method to output the unified object set. Specifically, each pixel is regarded as a CRF node, the nodes are connected according to the spatial neighborhood relationship in the preset radius range to generate CRF edges; node features are obtained based on the class probability, spectral and texture features, and geometric features in the preliminary segmentation results; a CRF model is constructed according to the nodes, edges, and node features, and the node class distribution of the CRF model is optimized using an iterative method and is normalized after each iteration; the iteration is stopped when a preset number of iterations is reached or the class probability change is lower than a preset change threshold; the optimized class distribution of each node is output, mapped to the original space, and each object is subjected to boundary refinement and object consistency processing to obtain a refined object segmentation result; the refined object segmentation result is combined with the preliminary segmentation result to generate a unified object set.

[0011] In one embodiment, the unified object set is registered based on a generative model to obtain fusion data. Specifically, feature representation learning is performed on the unified object set to extract multi-scale feature vectors, which are encoded into latent space vectors as inputs of the generative model; a generative model is constructed, which includes a generator and a discriminator; the unified object set is subjected to adversarial training based on the generative model to obtain an initial registration transformation model; based on the initial registration transformation model, the latent space vectors are used to output a registered object set; the registered object set is subjected to graph optimization and post-processing correction to generate a final optimized registration object set; and the final optimized registration object set is fused by weighted averaging to output fusion data with spatial consistency and structural integrity.

[0012] In one of the embodiments, based on the registered object set, a final optimized registered object set is generated through graph optimization and post-processing correction, specifically: the registered object set is constructed as a graph model, and a least square graph optimization objective function is constructed, which is a weighted sum of geometric consistency constraints and topological consistency constraints; the graph model is taken as input, and the node position is adjusted using the Gauss-Newton method to minimize the objective function, and iterative optimization is performed; the optimized graph model is subjected to anomaly detection, and based on the iterative optimization and the adjusted final optimized graph, the final optimized registered object set is output.

[0013] In one of the embodiments, change detection is performed on the fused data, and based on the detection result, an initial three-dimensional model is constructed using a local incremental modeling algorithm, and global consistency optimization is performed, specifically: the fused data is divided into a plurality of spatial units, each spatial unit containing a corresponding object point set; for each spatial unit, the difference values of geometric position and attribute features are obtained; based on a preset change threshold, change regions and stable regions are set; the spatial units of the change regions are divided into local modeling units; for each local modeling unit, an incremental modeling algorithm based on point cloud surface reconstruction is used to generate a local three-dimensional structure; the adjacent local modeling units are spliced and topologically connected to output an initial three-dimensional model; the initial three-dimensional model is subjected to global consistency optimization based on a hierarchical multi-scale graph optimization strategy.

[0014] In one of the embodiments, the initial three-dimensional model is subjected to global consistency optimization based on a hierarchical multi-scale graph optimization strategy, specifically: the initial three-dimensional model is represented as a graph model; the graph model is divided into a coarse-grained layer and a fine-grained layer according to a node importance index; the nodes of the coarse-grained layer are subjected to global optimization, and the node positions are adjusted through least square iterative optimization to minimize geometric consistency error and topological consistency constraint error, forming a coarse-grained global framework; the nodes and edges of the fine-grained layer are added to the coarse-grained global framework to establish a local neighborhood graph; a graph optimization objective function is established for the local neighborhood graph, and multi-scale iterative optimization is performed to update the target node positions until all fine-grained nodes are greater than a preset graph optimization objective threshold, and the optimized initial three-dimensional model is output.

[0015] In one of the embodiments, the optimized initial three-dimensional model is subjected to defect identification and correction to output a final three-dimensional model, specifically: the normal vector difference between the boundary points in the optimized initial three-dimensional model is obtained, and the boundary point pairs greater than a preset normal vector difference threshold are marked as potential joint regions; the texture features corresponding to the boundary points are extracted, and the similarity index of the adjacent unit boundary texture features is calculated based on cosine similarity; the boundary points with a similarity index lower than a preset threshold are marked as texture discontinuous regions; the potential joint regions and the texture discontinuous regions marked in the optimized initial three-dimensional model are corrected.

[0016] In one of the embodiments, the marked potential seam region and texture discontinuous region in the optimized initial three-dimensional model are corrected, specifically: a local correction sub-region is generated with the marked boundary point as the center, the boundary points of the local correction sub-region are subjected to surface fitting, the boundary points are projected onto the fitted surface, and their positions are adjusted through iterative optimization to realize spatial continuity of the seam region and the adjacent unit boundary, and the potential seam region is corrected; the local boundary points in the local correction sub-region and their neighborhood are subjected to texture fusion, and the fused texture is mapped back to the point set of the optimized initial three-dimensional model to correct the texture discontinuous region; all the local correction sub-regions are sequentially sorted, after completing a group of local correction, the three-dimensional model is updated, and whether the seam and the texture discontinuous region meet the requirements are re-judged; if not, the correction step is repeated until the overall seam and texture continuity of the model meets the preset threshold; and the final three-dimensional model after iterative optimization of local geometry and texture correction is output.

[0017] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0018] 1. Through unified object-level feature extraction of multi-source and multi-modal data, conditional random field optimization segmentation, generation model registration, graph optimization and post-processing correction, high-precision, fine and spatially consistent three-dimensional data fusion is realized. The advantages are: on the one hand, the macro coverage of satellite images, the local high-resolution information of unmanned aerial images and the accurate three-dimensional geometric features of laser radar point clouds are fully integrated to realize unified expression of two-dimensional texture and three-dimensional geometry, improve object segmentation accuracy and boundary fine effect; on the other hand, the combination of generation model and graph optimization automatically learns complex nonlinear registration transformation, ensures geometric and topological consistency, and ensures data integrity through anomaly detection and local repair, thereby generating fusion data with spatial continuity, structural integrity and high reliability, providing high-quality basis for subsequent three-dimensional modeling, change detection and geographic information analysis.

[0019] 2. Through fine change detection, the fusion data is divided into stable regions and change regions, local incremental modeling is used in the change regions to generate an initial three-dimensional model, and hierarchical multi-scale graph optimization is used to realize global consistency; then, potential seam and texture discontinuous region are locally corrected and iteratively optimized to realize high-precision continuity of geometry and texture. This scheme not only ensures the accuracy of local details, but also improves the modeling efficiency and global structural stability, can quickly respond to changes in dynamic or multi-temporal environment, effectively eliminates minor defects, ensures the spatial continuity and visual quality of the three-dimensional model, and has accuracy, robustness and computational efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The flowchart of the high-resolution satellite image three-dimensional modeling integrated processing method of multi-source data fusion provided by the embodiments of the present application is shown. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figure 1 As shown in the diagram, the integrated processing method for 3D modeling of high-resolution satellite imagery based on multi-source data fusion provided by this invention includes the following steps:

[0023] S1 acquires multi-source input data for 3D modeling, performs object-level feature extraction, and outputs a unified object set.

[0024] In this embodiment, the multi-source input data includes high-resolution satellite imagery, UAV imagery, and lidar point cloud data. High-resolution satellite imagery is used to provide large-scale surface information and the distribution of ground features; UAV imagery is used to provide local high-precision view information, supplementing the insufficient resolution of satellite imagery; and lidar point cloud data is used to provide accurate three-dimensional geometric information, including surface elevation and building outlines.

[0025] Specifically, this involves acquiring multi-source input data for 3D modeling, extracting object-level features, and outputting a unified object set.

[0026] Preprocessing is performed on multi-source input data, which includes high-resolution satellite imagery, UAV imagery, and lidar point cloud data.

[0027] Independent branch networks are constructed for the processed multi-source input data, and multi-branch feature extraction is performed. The independent branch networks include satellite imagery branch which uses convolutional neural networks to extract spatial texture and spectral features, UAV imagery branch which uses lightweight convolutional networks to extract local high-resolution texture features, and point cloud branch which uses graph convolutional networks to extract three-dimensional geometric structure features.

[0028] The multi-branch feature extraction results are fused in a multimodal manner to generate preliminary segmentation results. The preliminary segmentation results include the class probability of each pixel, the spatial coordinate information of each pixel, the preliminary boundary contour of each object, spectral features, and texture features.

[0029] Among them, multimodal fusion maps the features extracted from satellite imagery, UAV imagery, and lidar point cloud branch networks onto a unified three-dimensional spatial grid, performs multimodal fusion on the mapped features, and unifies the texture and spectral features of two-dimensional images with the geometric and topological features of three-dimensional point clouds into an object-level feature vector;

[0030] The segmentation method based on conditional random field optimization performs object-level segmentation on the preliminary segmentation result and outputs a unified object set.

[0031] The preprocessing includes geometric correction and radiation correction of the high-resolution satellite image and the unmanned aerial vehicle image, ensuring that the images can be compared under the same spatial reference system; filtering of the laser radar point cloud to remove noise points and non-ground points and improve the quality of the point cloud; and uniform projection or resampling of data of different resolutions to ensure that the multi-source data are aligned in space.

[0032] Further, the segmentation method based on conditional random field optimization performs object-level segmentation on the preliminary segmentation result and outputs a unified object set, specifically as follows.

[0033] Each pixel is regarded as a CRF node, and the nodes are connected according to the spatial neighborhood relationship in the preset radius range to generate a CRF edge;

[0034] The node features are obtained based on the class probability in the preliminary segmentation result, the spectral and texture features and the geometric features;

[0035] The CRF model is constructed according to the nodes, edges and node features, and the node class distribution of the CRF model is optimized by using an iterative method, and is normalized after each iteration, wherein each iteration includes: updating the node class according to the class probability and the feature similarity (obtained by cosine similarity) in the node neighborhood; checking the node class probability change to determine whether the convergence condition is reached;

[0036] The iteration is stopped when the preset number of iterations is reached or the class probability change is lower than a preset change threshold;

[0037] The optimized node class distribution of each node is output, mapped to the original space, and the boundary refinement and object consistency processing are performed on each object to obtain a refined object segmentation result;

[0038] The original space refers to the spatial position of the input data in its original coordinate system. During the CRF (conditional random field optimization) optimization process, the data are often converted into a graph structure or a matrix form (nodes, edges and feature vectors). This representation is not intuitive for users to understand as “image” or “point cloud”. Therefore, after the optimization is completed, the result needs to be corresponded to the pixel or point coordinates in the original space, so that the intuitive and usable object boundary and three-dimensional spatial position can be obtained.

[0039] The refined object segmentation result is combined with the preliminary segmentation result to generate a unified object set, and the object set includes an object class label, a refined boundary contour, a multi-modal feature vector (texture, spectrum, geometry), and spatial coordinate information.

[0040] In the boundary refinement and object consistency processing of each object, the boundary is adjusted using a boundary smoothing method such as neighborhood averaging, morphological operation or curve fitting; the boundary closure and self-intersection are checked and corrected; and the overlapping area of adjacent objects is separated or re-labeled to ensure spatial continuity.

[0041] It should be noted that through multi-source data fusion, object-level feature extraction and conditional random field optimization, local high-precision information of unmanned aerial vehicle images and three-dimensional geometric information of laser radar point clouds can be introduced while ensuring wide coverage of satellite images, and unified expression of two-dimensional and three-dimensional features is realized. The advantages of this are: on the one hand, the accuracy of object segmentation and the effect of boundary refinement can be significantly improved, and blurring and deviation caused by a single data source can be avoided; on the other hand, through CRF optimization and mapping back to the original space, the consistency and continuity of the segmentation result in the spatial coordinates are ensured, thereby providing high-quality, refined and structurally complete object set input for subsequent three-dimensional modeling.

[0042] S2, registering the unified object set based on the generative model to obtain fused data.

[0043] The generative model can learn the complex feature distribution of multi-source data objects in the latent space, thereby automatically inferring a high-precision registration transformation, reducing the dependence on artificial feature design or coarse registration, and simultaneously being capable of processing nonlinear deformation and noise interference to realize unified alignment of multi-scale and multi-modal data.

[0044] In this embodiment, the unified object set is registered based on the generative model to obtain fused data, specifically as follows:

[0045] The unified object set is subjected to feature representation learning, multi-scale feature vectors of the objects in geometric morphology, local texture and global structure are extracted, and the feature vectors are encoded into latent space vectors as inputs of the generative model;

[0046] The generative model is constructed, and the generative model includes a generator and a discriminator; wherein the generator adopts a convolution-deconvolution structure to map the latent space vectors to registration transformation parameters (rotation matrix, translation vector or nonlinear deformation field), and the discriminator adopts a similar convolution structure to discriminate the spatial consistency and structural consistency of the generated data and the reference data;

[0047] The unified object set is subjected to adversarial training based on the generative model, the generator gradually learns the registration transformation from the source object to the target object, and the discriminator constrains the generated result to approximate the target object set through a discrimination loss; the generator parameters are optimized through iterative training to obtain an initial registration transformation model;

[0048] Based on the initial registration transformation model, the object set after registration is output according to the latent space vectors.

[0049] The registered object set is corrected based on graph optimization and post-processing to generate a final optimized registered object set;

[0050] Based on the final optimized registered object set, fusion is performed by weighted average to output fusion data with spatial consistency and structural integrity.

[0051] Further, the registered object set is corrected based on graph optimization and post-processing to generate a final optimized registered object set, specifically:

[0052] The registered object set is constructed into a graph model, where the objects are graph nodes, and the spatial adjacency between nodes is a graph edge, and each graph edge contains geometric constraint information, including the distance, relative position and direction between nodes;

[0053] A least squares graph optimization objective function is constructed, which is a weighted sum of geometric consistency constraints and topological consistency constraints;

[0054] Among them, the geometric consistency constraint is used to minimize the error between the actual position and the expected position of each node in the graph, which can be obtained by node position error, and the topological consistency constraint is used to maintain the connectivity of nodes, the distribution of edge weights and the integrity of key topological structure, which can be obtained by comparing node connectivity, ring structure and edge weight distribution;

[0055] Using the graph model as input, the node position is adjusted using the Gauss-Newton method to minimize the objective function, and iterative optimization is performed;

[0056] The optimized graph model is subjected to anomaly detection, and local adjustment is performed for broken and isolated nodes, based on the final optimized graph after iterative optimization and adjustment, to output the final optimized registered object set, wherein the local adjustment includes finding the nearest neighbor node or matching node, establishing a new edge between the isolated node and the node at a reasonable position in the graph to solve the abnormal situation of isolated nodes; in the broken area, intermediate nodes or adjusted node positions are generated by interpolation or local optimization to restore the edge to a reasonable length and direction while ensuring topological consistency with adjacent nodes, to solve the problem of broken edges.

[0057] Among them, the isolated node detection is to traverse all nodes and count the number of neighbors of each node; if the number of neighbors of a node is equal to 0 or lower than a preset threshold, it is determined to be an isolated node; the broken anomaly detection is to obtain the actual distance between nodes, and if the actual distance exceeds a preset reasonable range, it is determined to be a broken edge.

[0058] The fused data refers to a high-precision three-dimensional data set obtained by unified registration, adversarial generation modeling, graph optimization and post-processing correction on a multi-source, multi-modal or multi-temporal object set. It is consistent in space, maintains integrity in structure, and integrates the geometric shape, local texture, global structure and other attribute features of each source data, can reflect the coordination between different data sources, and becomes a reliable basis for subsequent three-dimensional modeling, change detection or analysis.

[0059] It should be noted that the generation model registers the unified object set, combines the graph optimization and post-processing correction, and realizes the unified alignment and fusion of high-precision, multi-modal and multi-scale data. Specifically, the feature representation learning can comprehensively extract multi-scale information of geometric shape, local texture and global structure, providing rich potential space representation for the generation model; the generation model and its adversarial training enable the registration transformation to automatically learn complex nonlinear mapping, reduce the dependence on artificial feature design and coarse registration, and effectively resist noise interference; the graph optimization models the registered object set as a node-edge graph, ensures the node position accuracy through geometric consistency constraint, and maintains the connectivity and key structure integrity through topological consistency constraint, thereby improving the registration stability and global consistency; the anomaly detection and local adjustment finely repair isolated nodes and broken edges, ensuring the topological structure integrity and data reliability; finally, the optimized registration object set is weighted and fused to realize the fusion data output of spatial consistency and structural integrity, providing a high-quality, unified multi-source data basis for subsequent analysis, modeling or decision-making.

[0060] S3, change detection is performed on the fused data, and an initial three-dimensional model is constructed using a local incremental modeling algorithm according to the detection result, and global consistency optimization is performed.

[0061] In this embodiment, change detection is performed on the fused data, an initial three-dimensional model is constructed using a local incremental modeling algorithm according to the detection result, and global consistency optimization is performed, specifically:

[0062] The fused data is divided into a plurality of spatial units, each spatial unit containing a corresponding object point set;

[0063] For each spatial unit, the difference value in geometric position and attribute feature between the spatial unit and the preset reference time data is obtained;

[0064] According to a preset change threshold, spatial units greater than the change threshold are set as change regions, and vice versa as stable regions;

[0065] The spatial units of the change region are divided into local modeling units, each unit containing an object point set and its attribute information;

[0066] For each local modeling unit, a local three-dimensional structure is generated by using an incremental modeling algorithm based on point cloud surface reconstruction;

[0067] The boundary splicing and topological connection of adjacent local modeling units are performed by nearest neighbor point matching, and an initial three-dimensional model formed by the integration of each local unit is output;

[0068] The initial three-dimensional model is globally optimized based on a hierarchical multi-scale graph optimization strategy.

[0069] The difference value of the geometric position is used to measure the degree of deviation of each spatial unit object point set in the fusion data and the corresponding object point set in the reference time data in the spatial position. Usually represented by the Euclidean distance of the spatial coordinates or the deviation of the center of gravity of the point set; the centroid of each spatial unit object point set and the centroid of the corresponding spatial unit at the reference time are obtained, and the difference value of the geometric position is obtained; the difference value of the attribute feature is used to measure the change of the non-geometric information of the object in the spatial unit, including color, texture, density, material type, etc., by extracting the attribute vector of each spatial unit object point set and the corresponding attribute vector of the spatial unit at the reference time, the cosine similarity is obtained by calculation.

[0070] Further, the initial three-dimensional model is globally optimized based on a hierarchical multi-scale graph optimization strategy, specifically:

[0071] The initial three-dimensional model is represented as a graph model, where the nodes represent the local modeling units, and the edges represent the spatial constraints and geometric relationships between the nodes. Each graph edge contains the Euclidean distance, relative position and direction between nodes;

[0072] The graph model is divided into a coarse-grained layer and a fine-grained layer according to the node importance index. The coarse-grained layer retains key nodes and representative edges, which are used to quickly establish global structural constraints. The fine-grained layer contains the remaining nodes and detailed edges, which provide fine adjustment objects for subsequent local optimization;

[0073] The nodes in the coarse-grained layer are globally optimized by least squares iterative optimization to adjust the node positions, so as to minimize the geometric consistency error and topological consistency constraint error, and form a stable coarse-grained global framework;

[0074] The nodes and edges of the fine-grained layer are added to the coarse-grained global framework, and a local neighborhood graph is established for each added fine-grained node. The neighborhood graph nodes include the target node and its spatial adjacent nodes, and the edges contain geometric constraint information;

[0075] A graph optimization objective function is established for the local neighborhood graph, and a sparse least squares method is used for multi-scale iterative optimization to update the position of the target node until all fine-grained nodes are greater than a preset graph optimization objective threshold, and the optimized initial three-dimensional model is output.

[0076] The node importance index is used to determine which nodes are most critical to global consistency and local geometric preservation in hierarchical / multi-scale graph optimization. Generally, the local geometric variation can be quantified. The local geometric variation measures the degree of change of the geometric features of the region where the node is located. The greater the variation, the more important the node is in modeling and optimization. The specific acquisition method is as follows: for each node, define its local neighborhood, which can be determined by a fixed number of neighbors. The geometric difference between the node and the neighborhood points is calculated by the mean square distance error method to obtain the local geometric variation. The local geometric variation of all nodes is normalized as the node importance index.

[0077] It should be noted that by performing fine change detection on the fused data, the data is divided into stable regions and change regions, and local incremental modeling is used in the change regions, which not only ensures the accuracy of modeling, but also improves the calculation efficiency; then, global consistency is achieved through hierarchical multi-scale graph optimization, which takes into account the coordination of local details and overall structure.

[0078] It can ensure the accuracy of local details while achieving global structural consistency, improve modeling efficiency and accuracy, and is particularly suitable for three-dimensional reconstruction in dynamic or multi-temporal environments, can quickly respond to updates in change regions, and through hierarchical optimization, reduces the computational complexity, and takes into account fine modeling and global stability.

[0079] S4, defect recognition and correction of the optimized initial three-dimensional model, output the final three-dimensional model.

[0080] In this embodiment, the optimized initial three-dimensional model is subjected to defect recognition and correction, and the final three-dimensional model is output, specifically:

[0081] Obtain the normal vector difference between the boundary points in the optimized initial three-dimensional model, and mark the boundary point pairs greater than the preset normal vector difference threshold as potential seam regions;

[0082] The normal vector difference acquisition step is: for each boundary point in the optimized three-dimensional model, the normal vector of the boundary point is determined using the weighted average method or principal component analysis (PCA), and for adjacent boundary point pairs, the included angle between the corresponding normal vectors is obtained, and the normal vector difference is obtained by the cosine difference of the included angle;

[0083] Extract the texture features corresponding to the boundary points, and calculate the similarity index of the adjacent unit boundary texture features based on the cosine similarity;

[0084] Mark the boundary points with a similarity index lower than a preset threshold as texture discontinuous regions;

[0085] Correct the potential seam regions and texture discontinuous regions marked in the optimized initial three-dimensional model.

[0086] Further, the potential seam area and the texture discontinuous area marked in the optimized initial three-dimensional model are corrected, specifically:

[0087] A local correction sub-area is generated with the marked boundary point as the center, the boundary points of the local correction sub-area are subjected to surface fitting, the boundary points are projected onto the fitted surface, and their positions are adjusted through iterative optimization, so that the seam area and the adjacent unit boundary are spatially continuous, and the potential seam area is corrected;

[0088] The local boundary points in the local correction sub-area and their neighborhood are subjected to texture fusion, and the texture fusion includes multi-scale texture interpolation, local texture blending or weight-based attribute mapping;

[0089] The fused texture is mapped back to the point set of the optimized initial three-dimensional model, and the texture discontinuous area is corrected;

[0090] All local correction sub-areas are sequentially sorted, and after completing a group of local corrections, the three-dimensional model is updated, and it is determined again whether the seam and the texture discontinuous area meet the requirements;

[0091] If not, repeat the correction step until the overall seam and texture continuity of the model meet the preset threshold;

[0092] Output the final three-dimensional model after iterative optimization of local geometry and texture correction.

[0093] It should be noted that by analyzing the normal vector difference of the boundary points and the texture similarity of the optimized initial three-dimensional model, the potential seam and the texture discontinuous area are accurately marked, and combined with surface fitting of the local correction sub-area, iterative geometry optimization and multi-scale texture fusion, the local defects are corrected. The advantages of this scheme are that it can effectively eliminate the problem of micro-seam and texture discontinuity while maintaining the consistency of the overall structure and texture, improve the spatial continuity and visual quality of the three-dimensional model, and at the same time, the iterative updating mechanism ensures that the correction result gradually converges to the preset accuracy standard, and has both accuracy and robustness.

[0094] The above embodiments can be realized all or partially by software, hardware, firmware or any combination thereof. When realized by software, the above embodiments can be realized in the form of a computer program product all or partially.

[0095] Those skilled in the art can understand that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0096] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0097] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any skilled in the art within the technical scope disclosed in the present application can easily think of changes or replacements, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0098] Finally: the above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for integrated processing of high-resolution satellite image three-dimensional modeling of multi-source data fusion, characterized in that, The application relates to a three-dimensional modeling method and device. The method comprises the following steps: Preprocessing multi-source input data, constructing independent branch networks for the processed multi-source input data, and performing multi-branch feature extraction; Fusing the multi-branch feature extraction results to generate a preliminary segmentation result; Performing object-level segmentation on the preliminary segmentation result based on a conditional random field optimization method to output a unified object set, specifically as follows: Connecting nodes according to the spatial neighborhood relationship in a preset radius range to generate CRF edges; Obtaining node features based on the class probability, spectrum and texture features and geometric features in the preliminary segmentation result; Constructing a CRF model according to the nodes, edges and node features, and optimizing the node class distribution of the CRF model by using an iterative method, and normalizing after each iteration; Stopping iteration when a preset iteration number is reached or the class probability change is lower than a preset change threshold; Outputting the optimized node class distribution, mapping to the original space, and performing boundary refinement and object consistency processing on each object to obtain a refined object segmentation result; Combining the refined object segmentation result with the preliminary segmentation result to generate a unified object set; Registering the unified object set based on a generative model to obtain fused data; Performing change detection on the fused data, constructing an initial three-dimensional model by using a local incremental modeling algorithm according to the detection result, and performing global consistency optimization. The method comprises the following steps:

2. The method according to claim 1, wherein, Performing feature representation learning on the unified object set, extracting a multi-scale feature vector, and encoding the multi-scale feature vector into a latent space vector as a generative model input; Constructing a generative model, wherein the generative model comprises a generator and a discriminator; Performing adversarial training on the unified object set based on the generative model to obtain an initial registration transformation model; Outputting a registered object set based on the initial registration transformation model according to the latent space vector; Performing graph optimization and post-processing correction on the registered object set to generate a final optimized registration object set; Fusing the final optimized registration object set by using a weighted average to output fused data with spatial consistency and structural integrity. The method comprises the following steps:

3. The method according to claim 2, wherein, Constructing a graph model based on the registered object set, and constructing a least squares graph optimization objective function, wherein the graph optimization objective function is a weighted sum of geometric consistency constraints and topological consistency constraints; Taking the graph model as an input, adjusting the node position by using a Gauss-Newton method, minimizing the objective function, and performing iterative optimization; Performing anomaly detection on the optimized graph model, and outputting the final optimized registration object set based on the iterative optimization and the adjusted final optimized graph. The method comprises the following steps:

4. The method according to claim 1, wherein, ​ The fusion data is divided into a plurality of spatial units, and each spatial unit contains a corresponding object point set; For each spatial unit, a difference value of geometric position and attribute feature is obtained; According to a preset change threshold, a change region and a stable region are set; The spatial units in the change region are divided into local modeling units; For each local modeling unit, an incremental modeling algorithm based on point cloud surface reconstruction is used to generate a local three-dimensional structure; Adjacent local modeling units are spliced and topologically connected to output an initial three-dimensional model; The initial three-dimensional model is globally optimized based on a hierarchical multi-scale graph optimization strategy.

5. The method according to claim 4, wherein, The hierarchical multi-scale graph optimization strategy is used to globally optimize the initial three-dimensional model, specifically as follows: The initial three-dimensional model is represented as a graph model; The graph model is divided into a coarse-grained layer and a fine-grained layer according to a node importance index; The nodes in the coarse-grained layer are globally optimized, and the node positions are iteratively optimized and adjusted by a least squares method to minimize geometric consistency error and topological consistency constraint error, thereby forming a coarse-grained global framework; The nodes and edges in the fine-grained layer are added to the coarse-grained global framework to establish a local neighborhood graph; A graph optimization objective function is established for the local neighborhood graph, and multi-scale iterative optimization is performed to update the target node positions until all fine-grained nodes are greater than a preset graph optimization objective threshold, thereby outputting an optimized initial three-dimensional model.

6. The method according to claim 5, wherein, The optimized initial three-dimensional model is subjected to defect identification and correction to output a final three-dimensional model, specifically as follows: The normal vector difference between the boundary points in the optimized initial three-dimensional model is obtained, and boundary point pairs with a normal vector difference greater than a preset normal vector difference threshold are marked as potential seam regions; Texture features corresponding to the boundary points are extracted, and a similarity index of the boundary texture features of adjacent units is calculated based on cosine similarity; Boundary points with a similarity index lower than a preset threshold are marked as texture discontinuous regions; The potential seam regions and the texture discontinuous regions in the optimized initial three-dimensional model are corrected.

7. The method according to claim 6, wherein, The potential seam regions and the texture discontinuous regions in the optimized initial three-dimensional model are corrected, specifically as follows: A local correction sub-region is generated around the marked boundary points, the boundary points in the local correction sub-region are subjected to surface fitting, the boundary points are projected onto the fitted surface, and the positions of the boundary points are iteratively optimized and adjusted to achieve spatial continuity between the seam region and the adjacent unit boundary, thereby correcting the potential seam region; The local boundary points in the local correction sub-region and their neighborhoods are subjected to texture fusion, and the fused texture is mapped back to the point set of the optimized initial three-dimensional model, thereby correcting the texture discontinuous region; All local correction sub-regions are sequentially sorted, and after completing a group of local corrections, the three-dimensional model is updated, and it is determined whether the seam and the texture discontinuous region meet the requirements; If not, the correction step is repeated until the overall seam and texture continuity of the model meet a preset threshold; The final three-dimensional model after iterative optimization of local geometry and texture correction is output.

Images