Multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method

Abstract

The invention discloses a high-resolution satellite image three-dimensional modeling integrated processing method based on multi-source data fusion, and relates to the technical field of satellite image three-dimensional modeling, and the method comprises the following steps: obtaining multi-source input data of three-dimensional modeling, carrying out object-level feature extraction, and outputting a unified object set; registering the unified object set based on the generative model to obtain fusion data; performing change detection on the fused data, constructing an initial three-dimensional model by adopting a local incremental modeling algorithm according to a detection result, and performing global consistency optimization; performing defect identification and correction on the optimized initial three-dimensional model, and outputting a final three-dimensional model; according to the method, through object-level multi-source data fusion, high-precision registration of the generated model, change adaptive incremental modeling and local geometry and texture iterative correction, the problems that in the prior art, boundary seams of a three-dimensional model are obvious, textures are discontinuous, and change areas are not updated in time are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of satellite image three-dimensional modeling, more particularly, the present application relates to a multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method. BACKGROUND

[0002] With the improvement of high-resolution satellite image acquisition capability, the demand for high-precision three-dimensional models in the fields of three-dimensional city modeling, land surface change monitoring and remote sensing data analysis is increasing. Traditional three-dimensional modeling methods mainly rely on single sensor data, such as optical images or radar images, to construct three-dimensional models through stereo image matching, point cloud generation and surface reconstruction. However, single data source has limitations in spatial coverage, resolution and observation angle, making it difficult to ensure both geometric accuracy and texture continuity of the model.

[0003] To solve the problem of insufficient modeling accuracy of single-source data, multi-source data fusion methods have been proposed in recent years, including optical image and laser radar point cloud fusion, optical image and multi-temporal remote sensing image fusion. However, the existing technology still has the following main defects in practical application: Existing methods rely mainly on pixel-level or point cloud-level features for fusion, and object-level information is not fully utilized. There is a lack of unified modeling processing of complete structural information of buildings or ground objects, resulting in geometric deviations or texture discontinuities in the three-dimensional model at the boundaries or local details. Moreover, multi-source data registration usually uses rigid transformation or feature point-based optimization methods, which are difficult to fully consider global structural constraints and local geometric relationships, resulting in problems such as local misplacement, obvious seams or topological discontinuity in the generated initial three-dimensional model. Furthermore, even after global optimization, there may still be small seams or texture discontinuities at the boundaries of the model, and existing technology lacks a high-precision correction mechanism for these local defects, making it difficult to meet the visual and structural requirements of high-precision three-dimensional modeling.

[0004] To solve the above problems, the present application provides a solution. SUMMARY

[0005] To overcome the above-mentioned defects of the prior art, embodiments of the present application provide a multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method, which solves the problems of obvious boundary seams, texture discontinuity and delayed updating of changed areas in the prior art through object-level multi-source data fusion, high-precision model registration, change adaptive incremental modeling and local geometric and texture iterative correction.

[0006] To achieve the above-mentioned purposes, the present application provides the following technical solutions: In a first aspect, the application provides a multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method, which comprises: acquiring multi-source input data for three-dimensional modeling, performing object-level feature extraction, and outputting a unified object set; registering the unified object set based on a generative model to obtain fusion data; performing change detection on the fusion data, constructing an initial three-dimensional model according to the detection result using a local incremental modeling algorithm, and performing global consistency optimization; identifying defects in the optimized initial three-dimensional model and correcting the defects to output a final three-dimensional model.

[0007] In one embodiment, multi-source input data for three-dimensional modeling is acquired, object-level feature extraction is performed, and a unified object set is output. Specifically, the multi-source input data is preprocessed, the processed multi-source input data is respectively constructed into independent branch networks, and multi-branch feature extraction is performed; the multi-branch feature extraction results are fused in multiple modalities to generate preliminary segmentation results; the preliminary segmentation results are segmented at the object level based on a conditional random field optimization segmentation method to output the unified object set.

[0008] In one embodiment, the preliminary segmentation results are segmented at the object level based on a conditional random field optimization segmentation method to output the unified object set. Specifically, each pixel is regarded as a CRF node, the nodes are connected according to the spatial neighborhood relationship in the preset radius range to generate CRF edges; node features are obtained based on the class probability, spectral and texture features, and geometric features in the preliminary segmentation results; a CRF model is constructed according to the nodes, edges, and node features, and the node class distribution of the CRF model is optimized using an iterative method and is normalized after each iteration; the iteration is stopped when a preset number of iterations is reached or the class probability change is lower than a preset change threshold; the optimized class distribution of each node is output, mapped to the original space, and each object is subjected to boundary refinement and object consistency processing to obtain a refined object segmentation result; the refined object segmentation result is combined with the preliminary segmentation result to generate a unified object set.

[0009] In one embodiment, the unified object set is registered based on a generative model to obtain fusion data. Specifically, feature representation learning is performed on the unified object set to extract multi-scale feature vectors, which are encoded into latent space vectors as inputs of the generative model; a generative model is constructed, which includes a generator and a discriminator; the unified object set is subjected to adversarial training based on the generative model to obtain an initial registration transformation model; based on the initial registration transformation model, the latent space vectors are used to output a registered object set; the registered object set is subjected to graph optimization and post-processing correction to generate a final optimized registration object set; and the final optimized registration object set is fused by weighted averaging to output fusion data with spatial consistency and structural integrity.

[0010] In one of the embodiments, based on the registered object set, a final optimized registered object set is generated through graph optimization and post-processing correction, specifically: the registered object set is constructed as a graph model, and a least square graph optimization objective function is constructed, which is a weighted sum of geometric consistency constraints and topological consistency constraints; the graph model is taken as input, and the node position is adjusted using the Gauss-Newton method to minimize the objective function, and iterative optimization is performed; the optimized graph model is subjected to anomaly detection, and based on the iterative optimization and the adjusted final optimized graph, the final optimized registered object set is output.

[0011] In one of the embodiments, change detection is performed on the fused data, and based on the detection result, an initial three-dimensional model is constructed using a local incremental modeling algorithm, and global consistency optimization is performed, specifically: the fused data is divided into a plurality of spatial units, each spatial unit containing a corresponding object point set; for each spatial unit, the difference values of geometric position and attribute features are obtained; based on a preset change threshold, change regions and stable regions are set; the spatial units of the change regions are divided into local modeling units; for each local modeling unit, an incremental modeling algorithm based on point cloud surface reconstruction is used to generate a local three-dimensional structure; the adjacent local modeling units are spliced and topologically connected to output an initial three-dimensional model; the initial three-dimensional model is subjected to global consistency optimization based on a hierarchical multi-scale graph optimization strategy.

[0012] In one of the embodiments, the initial three-dimensional model is subjected to global consistency optimization based on a hierarchical multi-scale graph optimization strategy, specifically: the initial three-dimensional model is represented as a graph model; the graph model is divided into a coarse-grained layer and a fine-grained layer according to a node importance index; the nodes of the coarse-grained layer are subjected to global optimization, and the node positions are adjusted through least square iterative optimization to minimize geometric consistency error and topological consistency constraint error, forming a coarse-grained global framework; the nodes and edges of the fine-grained layer are added to the coarse-grained global framework to establish a local neighborhood graph; a graph optimization objective function is established for the local neighborhood graph, and multi-scale iterative optimization is performed to update the target node positions until all fine-grained nodes are greater than a preset graph optimization objective threshold, and the optimized initial three-dimensional model is output.

[0013] In one of the embodiments, the optimized initial three-dimensional model is subjected to defect identification and correction to output a final three-dimensional model, specifically: the normal vector difference between the boundary points in the optimized initial three-dimensional model is obtained, and the boundary point pairs greater than a preset normal vector difference threshold are marked as potential joint regions; the texture features corresponding to the boundary points are extracted, and the similarity index of the adjacent unit boundary texture features is calculated based on cosine similarity; the boundary points with a similarity index lower than a preset threshold are marked as texture discontinuous regions; the potential joint regions and the texture discontinuous regions marked in the optimized initial three-dimensional model are corrected.

[0014] In one of the embodiments, the potential seam region and the texture discontinuous region marked in the optimized initial three-dimensional model are corrected, specifically: a local correction sub-region is generated with the marked boundary point as the center, the boundary points of the local correction sub-region are subjected to surface fitting, the boundary points are projected onto the fitted surface, and their positions are adjusted through iterative optimization to realize spatial continuity of the seam region and the adjacent unit boundary, and the potential seam region is corrected; the local boundary points in the local correction sub-region and their neighborhood are subjected to texture fusion, and the fused texture is mapped back to the point set of the optimized initial three-dimensional model to correct the texture discontinuous region; all the local correction sub-regions are sequentially sorted, after each group of local correction is completed, the three-dimensional model is updated, and whether the seam and the texture discontinuous region meet the requirements is re-judged; if not, the correction step is repeated until the overall seam and the texture continuity of the model meet the preset threshold; and the final three-dimensional model after the local geometry and texture correction iterative optimization is output.

[0015] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages: 1. Through unified object-level feature extraction of multi-source and multi-modal data, conditional random field optimization segmentation, generation model registration, graph optimization and post-processing correction, high-precision, fine and spatially consistent three-dimensional data fusion is realized. The advantages are: on the one hand, the macro coverage of satellite images, the local high-resolution information of unmanned aerial images and the accurate three-dimensional geometric features of laser radar point clouds are fully integrated to realize unified expression of two-dimensional texture and three-dimensional geometry, improve object segmentation accuracy and boundary fine effect; on the other hand, the combination of generation model and graph optimization automatically learns complex nonlinear registration transformation, ensures geometric and topological consistency, and ensures data integrity through anomaly detection and local repair, thereby generating fusion data with spatial continuity, structural integrity and high reliability, providing high-quality basis for subsequent three-dimensional modeling, change detection and geographic information analysis.

[0016] 2. Through fine change detection, the fusion data is divided into stable regions and change regions, local incremental modeling is used to generate an initial three-dimensional model in the change region, and hierarchical multi-scale graph optimization is used to realize global consistency; then, the potential seam and the texture discontinuous region are locally corrected and iteratively optimized to realize high-precision continuity of geometry and texture. This scheme not only ensures the accuracy of local details, but also improves the modeling efficiency and global structural stability, can quickly respond to changes in dynamic or multi-temporal environment, effectively eliminates minor defects, ensures the spatial continuity and visual quality of the three-dimensional model, and has accuracy, robustness and computing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the high-resolution satellite image three-dimensional modeling integrated processing method of multi-source data fusion provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0019] Referring to Figure 1 The multi-source data fusion high-resolution satellite image three-dimensional modeling integrated processing method provided by the present application is shown in the flowchart, which comprises the following steps: S1, obtaining multi-source input data for three-dimensional modeling, and performing object-level feature extraction to output a unified object set.

[0020] In this embodiment, the multi-source input data comprises high-resolution satellite images, unmanned aerial vehicle images, and laser radar point cloud data. The high-resolution satellite images are used to provide large-scale ground surface information and ground object distribution; the unmanned aerial vehicle images are used to provide local high-precision perspective information, supplementing the problem of insufficient resolution of satellite images; and the laser radar point cloud data is used to provide accurate three-dimensional geometric information, including ground surface elevation and building contour.

[0021] The multi-source input data for three-dimensional modeling is obtained, and object-level feature extraction is performed to output a unified object set, specifically as follows: The multi-source input data is preprocessed, and the multi-source input data comprises high-resolution satellite images, unmanned aerial vehicle images, and laser radar point cloud data. The processed multi-source input data is respectively constructed into independent branch networks, and multi-branch feature extraction is performed, wherein the independent branch networks comprise a satellite image branch using a convolutional neural network to extract spatial texture and spectral features, an unmanned aerial vehicle image branch using a lightweight convolutional network to extract local high-resolution texture features, and a point cloud branch using a graph convolutional network to extract three-dimensional geometric structure features. The multi-branch feature extraction results are subjected to multi-modal fusion to generate preliminary segmentation results, and the preliminary segmentation results comprise class probability of each pixel, spatial coordinate information of each pixel, preliminary boundary contour of each object, spectral features, and texture features. The multi-modal fusion is to map the features extracted by the satellite image branch, the unmanned aerial vehicle image branch, and the laser radar point cloud branch network to a unified three-dimensional space grid, and to perform multi-modal fusion on the mapped features, so as to uniformly represent the texture and spectral features of the two-dimensional image and the geometric and topological features of the three-dimensional point cloud as object-level feature vectors. The preliminary segmentation results are subjected to object-level segmentation based on a conditional random field optimization segmentation method to output a unified object set.

[0022] Among them, the preprocessing includes geometric correction and radiation correction of high-resolution satellite images and drone images to ensure that the images can be compared in the same spatial reference system; filtering of lidar point clouds to eliminate noise points and non-ground object points to improve the quality of point clouds; and unified projection or resampling of data of different resolutions to ensure spatial alignment of multi-source data.

[0023] Furthermore, the segmentation method based on conditional random field optimization performs object-level segmentation on the preliminary segmentation results and outputs a unified object set, specifically: Treat each pixel as a CRF node, connect the nodes according to the spatial neighborhood relationship within a preset radius, and generate CRF edges; Node features are obtained based on the category probability, spectral and texture features, and geometric features in the preliminary segmentation results; A CRF model is constructed based on nodes, edges, and node features. The node category distribution is optimized using an iterative method and normalized after each iteration. Each iteration includes: updating the node category based on the category probability and feature similarity (obtained through cosine similarity) within the node neighborhood; checking the change in node category probability to determine whether the convergence condition has been met; The iteration is stopped when the preset number of iterations is reached or the class probability change is lower than the preset change threshold; Output the optimized category distribution of each node, map it to the original space, and perform boundary refinement and object consistency processing on each object to obtain the refined object segmentation result; The original space refers to the spatial position of the input data in its original coordinate system. During the CRF (conditional random field) optimization process, the data is often converted into a graph structure or matrix form (nodes, edges, eigenvectors). This representation is not intuitively understood as an "image" or "point cloud" by users. Therefore, after the optimization is completed, the results need to be re-mapped to the pixel or point coordinates of the original space to obtain intuitive and usable object boundaries and three-dimensional spatial positions. The refined object segmentation results are combined with the preliminary segmentation results to generate a unified object set, which includes object category labels, refined boundary contours, multimodal feature vectors (texture, spectrum, geometry), and spatial coordinate information.

[0024] Among them, boundary refinement and object consistency processing for each object include adjusting the boundary using boundary smoothing methods (such as neighborhood averaging, morphological operations or curve fitting); checking boundary closure and self-intersection problems and making corrections; separating or re-marking the overlapping areas of adjacent objects to ensure spatial continuity.

[0025] It should be noted that by means of multi-source data fusion, object-level feature extraction and conditional random field optimization, local high-precision information of unmanned aerial vehicle images and three-dimensional geometric information of laser radar point clouds can be introduced while ensuring wide coverage of satellite images, so as to realize unified expression of two-dimensional and three-dimensional features. The advantages of this are: on the one hand, the accuracy of object segmentation and the boundary refinement effect can be significantly improved, and blurring and deviation caused by a single data source can be avoided; on the other hand, through CRF optimization and mapping back to the original space, the consistency and continuity of the segmentation results in the spatial coordinates are ensured, thereby providing high-quality, refined and structurally complete object set input for subsequent three-dimensional modeling.

[0026] S2, registering the unified object set based on the generative model to obtain fused data.

[0027] The generative model can learn the complex feature distribution of the multi-source data objects in the latent space, thereby automatically inferring a high-precision registration transformation, reducing the dependence on artificial feature design or coarse registration, and being capable of processing nonlinear deformation and noise interference to realize unified alignment of multi-scale and multi-modal data.

[0028] In this embodiment, the unified object set is registered based on the generative model to obtain fused data, specifically as follows: feature representation learning is performed on the unified object set to extract multi-scale feature vectors of the objects in terms of geometric shape, local texture and global structure, and the multi-scale feature vectors are encoded into latent space vectors as inputs of the generative model; The generative model includes a generator and a discriminator. The generator adopts a convolution-deconvolution structure to map the latent space vectors to registration transformation parameters (rotation matrix, translation vector or nonlinear deformation field), and the discriminator adopts a convolution structure to discriminate the spatial consistency and structural consistency of the generated data and the reference data. The generative model is used for adversarial training of the unified object set. The generator gradually learns the registration transformation from the source objects to the target objects, and the discriminator constrains the generated results to approximate the target object set through a discrimination loss. The generator parameters are optimized through iterative training to obtain an initial registration transformation model. Based on the initial registration transformation model, the object set after registration is output according to the latent space vectors; The object set after registration is corrected based on graph optimization and post-processing to generate a final optimized registration object set. Based on the final optimized registration object set, the fused data with spatial consistency and structural integrity are output through weighted averaging.

[0029] Further, the object set after registration is corrected based on graph optimization and post-processing to generate a final optimized registration object set, specifically as follows: The registered object set is constructed as a graph model, where the objects are graph nodes, the spatial adjacency between nodes are graph edges, and each graph edge contains geometric constraint information including the distance, relative position and direction between nodes; A least square graph optimization objective function is constructed, which is a weighted sum of geometric consistency constraints and topological consistency constraints; The geometric consistency constraints are used to minimize the error between the actual position and the expected position of each node in the graph, which can be obtained by node position error acquisition, and the topological consistency constraints are used to maintain the connectivity of nodes, the distribution of edge weights and the integrity of key topological structures, which can be obtained by comparing node connectivity, ring structure and edge weight distribution; The graph model is taken as input, and the Gauss-Newton method is used to adjust the node position to minimize the objective function as the goal, and iterative optimization is performed; The optimized graph model is subjected to anomaly detection, local adjustment is performed for broken and isolated nodes, and the final optimized registration object set is output based on the iterative optimization and the adjusted final optimized graph, the local adjustment includes finding the nearest neighbor node or matching node, establishing a new edge between the isolated node and the node with reasonable position in the graph to solve the abnormal situation of isolated nodes; interpolation or local optimization is performed in the broken area to generate intermediate nodes or adjust the node position, so that the edge restores to reasonable length and direction, while ensuring topological consistency with adjacent nodes, to solve the problem of broken edges.

[0030] The isolated node detection is to traverse all nodes and count the number of neighbors of each node; if the number of neighbors of a node is equal to 0 or lower than a preset threshold, it is determined as an isolated node; the broken anomaly detection is to obtain the actual distance between nodes, and if the actual distance exceeds a preset reasonable range, it is determined as a broken edge.

[0031] The fused data refers to a high-precision three-dimensional data set obtained by unified registration, adversarial generation modeling, graph optimization and post-processing correction of multi-source, multi-modal or multi-temporal object sets. It has consistency in space and integrity in structure, while integrating the geometric shape, local texture, global structure and other attribute features of each source data, and can reflect the coordination results between different data sources, becoming a reliable basis for subsequent three-dimensional modeling, change detection or analysis.

[0032] It should be noted that the generation model is registered to the unified object set, combined with graph optimization and post-processing correction, to realize high-precision, multi-modal and multi-scale data unified alignment and fusion. Specifically, feature representation learning can comprehensively extract multi-scale information of geometric shape, local texture and global structure, providing rich potential space representation for the generation model; the generation model and its adversarial training enable the registration transformation to automatically learn complex nonlinear mapping, reducing the dependence on artificial feature design and coarse registration, and effectively resisting noise interference; graph optimization models the registered object set as a node-edge graph, ensures the node position accuracy through geometric consistency constraint, maintains connectivity and key structure integrity through topological consistency constraint, thereby improving registration stability and global consistency; abnormality detection and local adjustment fine-tune isolated nodes and broken edges, ensuring the topological structure integrity and data reliability; finally, the optimized registration object set is weighted and fused to realize spatial consistency and structure integrity of the fusion data output, providing high-quality, unified multi-source data basis for subsequent analysis, modeling or decision-making.

[0033] S3, change detection is performed on the fusion data, and an initial three-dimensional model is constructed based on a local incremental modeling algorithm according to the detection result, and global consistency optimization is performed.

[0034] In this embodiment, change detection is performed on the fusion data, and an initial three-dimensional model is constructed based on a local incremental modeling algorithm according to the detection result, and global consistency optimization is performed, specifically as follows: The fusion data is divided into a plurality of spatial units, each spatial unit containing a corresponding object point set; For each spatial unit, the difference value in geometric position and attribute feature between the spatial unit and the preset reference time data is obtained; According to the preset change threshold, the spatial unit greater than the change threshold is set as a change region, and vice versa as a stable region; The spatial unit of the change region is divided into a local modeling unit, each unit containing an object point set and its attribute information; For each local modeling unit, an incremental modeling algorithm based on point cloud surface reconstruction is used to generate a local three-dimensional structure; The adjacent local modeling units are spliced and topologically connected through nearest neighbor point matching, and an initial three-dimensional model formed by integration of each local unit is output; The initial three-dimensional model is globally optimized based on a hierarchical multi-scale graph optimization strategy.

[0035] The difference value of the geometric position is used to measure the deviation degree of each spatial unit object point set in the fusion data and the corresponding object point set in the reference time data in the spatial position. Generally, the Euclidean distance of the spatial coordinates or the deviation of the center of gravity of the point set is used to represent it; the geometric position difference value of each spatial unit object point set and the center of gravity of the corresponding spatial unit in the reference time is obtained, and the geometric position difference value of the two is obtained; the difference value of the attribute feature is used to measure the change of the non-geometric information of the object in the spatial unit, including color, texture, density, material type, etc. The attribute vector of each spatial unit object point set and the corresponding attribute vector of the spatial unit in the reference time are extracted, and the cosine similarity is calculated to obtain.

[0036] Further, the initial three-dimensional model is globally optimized based on a hierarchical multi-scale graph optimization strategy, specifically: The initial three-dimensional model is represented as a graph model, wherein the nodes represent local modeling units, and the edges represent the spatial constraints and geometric relationships between the nodes. Each graph edge contains the Euclidean distance, relative position and direction between the nodes. The graph model is divided into a coarse-grained layer and a fine-grained layer according to the node importance index. The coarse-grained layer retains key nodes and representative edges, which are used to quickly establish global structural constraints. The fine-grained layer contains the remaining nodes and detail edges, which provide fine adjustment objects for subsequent local optimization. The nodes in the coarse-grained layer are globally optimized. The node positions are iteratively optimized and adjusted by the least squares method to minimize the geometric consistency error and the topological consistency constraint error, forming a stable coarse-grained global framework. The nodes and edges of the fine-grained layer are added to the coarse-grained global framework. A local neighborhood graph is established for each added fine-grained node. The neighborhood graph nodes include the target node and its spatial neighboring nodes, and the edges contain geometric constraint information. A graph optimization objective function is established for the local neighborhood graph, and a sparse least squares method is used for multi-scale iterative optimization to update the position of the target node. The optimization of the initial three-dimensional model is output until all fine-grained nodes are greater than a preset graph optimization objective threshold.

[0037] The node importance index is used to determine which nodes are most critical to global consistency and local geometry in hierarchical / multi-scale graph optimization. Generally, the local geometric change amount can be quantified. The local geometric change amount measures the degree of change of the geometric features of the region where the node is located. The larger the change amount, the more important the node is in modeling and optimization. The specific acquisition method is as follows: for each node, define its local neighborhood, which can be determined by a fixed number of neighbors. The local geometric change amount is obtained by calculating the geometric difference between the node and the neighborhood points by the mean square distance error method. The local geometric change amount of all nodes is normalized as the node importance index.

[0038] It should be noted that by fine change detection on the fused data, the data is divided into stable regions and change regions, and local incremental modeling is used in the change regions, which not only ensures the accuracy of modeling, but also improves the calculation efficiency; then, the hierarchical multi-scale graph optimization is used to realize global consistency, which takes into account the coordination of local details and overall structure.

[0039] The global structure consistency can be realized while ensuring the accuracy of local details, the modeling efficiency and accuracy are improved, and it is particularly suitable for three-dimensional reconstruction in dynamic or multi-temporal environment, can quickly respond to the update of the change region, and through hierarchical optimization, the calculation complexity is reduced, and the fine modeling and global stability are taken into account.

[0040] S4, defect recognition and correction are performed on the optimized initial three-dimensional model, and the final three-dimensional model is output.

[0041] In this embodiment, defect recognition and correction are performed on the optimized initial three-dimensional model, and the final three-dimensional model is output, specifically: The normal vector difference between the boundary points in the optimized initial three-dimensional model is obtained, and the boundary point pairs with a normal vector difference greater than a preset normal vector difference threshold are marked as potential seam regions; The normal vector difference is obtained by: for each boundary point in the optimized three-dimensional model, the normal vector of the boundary point is determined by using weighted average method or principal component analysis (PCA), and for adjacent boundary point pairs, the included angle between the corresponding normal vectors is obtained, and the normal vector difference is obtained by the cosine difference of the included angle; The texture features corresponding to the boundary points are extracted, and the similarity index of the adjacent unit boundary texture features is calculated based on the cosine similarity; The boundary points with a similarity index lower than a preset threshold are marked as texture discontinuous regions; The marked potential seam regions and texture discontinuous regions in the optimized initial three-dimensional model are corrected.

[0042] Further, the marked potential seam regions and texture discontinuous regions in the optimized initial three-dimensional model are corrected, specifically: A local correction sub-region is generated with the marked boundary point as the center, the boundary points in the local correction sub-region are surface fitted, the boundary points are projected onto the fitted surface, and their positions are adjusted through iterative optimization, so that the seam region and the adjacent unit boundary realize spatial continuity, and the potential seam region is corrected; The local boundary points in the local correction sub-region and their neighborhood are texture fused, and the texture fusion includes multi-scale texture interpolation, local texture mixing or weight-based attribute mapping; The fused texture is mapped back to the point set of the optimized initial three-dimensional model, and the texture discontinuous region is corrected; sequentially order all local correction sub-regions, update the three-dimensional model after completing a group of local corrections, and re-determine whether the joint and texture discontinuous regions meet the requirements; If not, repeat the correction step until the overall joint and texture continuity of the model meets the preset threshold; Output the final three-dimensional model after iterative optimization of local geometry and texture correction.

[0043] It should be noted that by analyzing the boundary point normal vector difference and texture similarity of the optimized initial three-dimensional model, the potential joint and texture discontinuous regions are accurately marked, and combined with surface fitting, iterative geometry optimization and multi-scale texture fusion of local correction sub-regions, the local defects are corrected. The advantages of this scheme are: it can effectively eliminate the problem of small joints and texture discontinuity while maintaining the consistency of the overall structure and texture, improve the spatial continuity and visual quality of the three-dimensional model, and at the same time, use the iterative update mechanism to ensure that the correction result gradually converges to the preset precision standard, and has accuracy and robustness.

[0044] The above embodiments can be realized all or partially by software, hardware, firmware or any combination thereof. When realized by software, the above embodiments can be realized all or partially in the form of a computer program product.

[0045] Those skilled in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0046] In addition, the functional modules in each embodiment of the present application can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0047] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0048] Finally: the above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An integrated processing method for 3D modeling of high-resolution satellite images based on multi-source data fusion, characterized by: The steps include: Obtain multi-source input data for 3D modeling, perform object-level feature extraction, and output a unified object set; Register the unified object set based on the generative model to obtain fused data; Perform change detection on the fused data, build the initial 3D model using a local incremental modeling algorithm based on the detection results, and perform global consistency optimization; Defects are identified and corrected on the optimized initial 3D model, and the final 3D model is output.

2. The method for integrated processing of high-resolution satellite image 3D modeling based on multi-source data fusion according to claim 1, characterized in that: The method of obtaining multi-source input data for 3D modeling, performing object-level feature extraction, and outputting a unified object set is as follows: Preprocess the multi-source input data, build independent branch networks for the processed multi-source input data, and perform multi-branch feature extraction; Perform multi-modal fusion on the multi-branch feature extraction results to generate preliminary segmentation results; The segmentation method based on conditional random field optimization performs object-level segmentation on the preliminary segmentation results and outputs a unified object set.

3. The method for integrated processing of high-resolution satellite image 3D modeling based on multi-source data fusion according to claim 2 is characterized in that: The segmentation method based on conditional random field optimization performs object-level segmentation on the preliminary segmentation results and outputs a unified object set, specifically: Treat each pixel as a CRF node, connect the nodes according to the spatial neighborhood relationship within a preset radius, and generate CRF edges; Node features are obtained based on the category probability, spectral and texture features, and geometric features in the preliminary segmentation results; Construct a CRF model based on nodes, edges, and node features, and use an iterative method to optimize the node category distribution of the CRF model and normalize it after each iteration; The iteration is stopped when the preset number of iterations is reached or the class probability change is lower than the preset change threshold; Output the optimized category distribution of each node, map it to the original space, and perform boundary refinement and object consistency processing on each object to obtain the refined object segmentation result; The refined object segmentation results are combined with the preliminary segmentation results to generate a unified object set.

4. The method for integrated processing of high-resolution satellite imagery 3D modeling based on multi-source data fusion according to claim 3, characterized in that: The unified object set is registered based on the generative model to obtain fused data, specifically: Performing feature representation learning on the unified object set, extracting multi-scale feature vectors, and encoding them into latent space vectors as input to a generative model; Constructing a generative model, wherein the generative model includes a generator and a discriminator; Performing adversarial training on a unified object set based on the generative model to obtain an initial registration transformation model; Based on the initial registration transformation model, the registered object set is output according to the latent space vector; The registered object set is optimized and corrected based on the graph, generating the final optimized registered object set; Based on the final optimized set of registered objects, fusion is performed through weighted averaging to output fused data with spatial consistency and structural integrity.

5. The method for integrated processing of high-resolution satellite image 3D modeling based on multi-source data fusion according to claim 4 is characterized in that: The registered object set is subjected to graph optimization and post-processing correction to generate a final optimized registered object set, specifically: Constructing the registered object set into a graph model and constructing a least squares graph optimization objective function, wherein the graph optimization objective function is a weighted sum of geometric consistency constraints and topological consistency constraints; Taking the graph model as input, the Gauss-Newton method is used to adjust the node positions and iterative optimization is performed with the goal of minimizing the objective function. Anomaly detection is performed on the optimized graph model, and based on the final optimized graph after iterative optimization and adjustment, the final optimized registration object set is output.

6. The method for integrated processing of high-resolution satellite imagery 3D modeling based on multi-source data fusion according to claim 5, characterized in that: The fused data is subjected to change detection, and an initial 3D model is constructed using a local incremental modeling algorithm based on the detection results, and global consistency optimization is performed, specifically: Divide the fused data into several spatial units, each of which contains a corresponding set of object points; For each spatial unit, obtain the difference value of geometric position and attribute characteristics; According to the preset change threshold, the change area and the stable area are set; Divide the spatial units of the change area into local modeling units; For each local modeling unit, an incremental modeling algorithm based on point cloud surface reconstruction is used to generate the local three-dimensional structure; Perform boundary splicing and topological connection on adjacent local modeling units to output the initial 3D model; The initial 3D model is globally optimized based on a hierarchical multi-scale graph optimization strategy.

7. The method for integrated processing of high-resolution satellite imagery 3D modeling based on multi-source data fusion according to claim 6, characterized in that: The global consistency optimization of the initial 3D model based on the hierarchical multi-scale graph optimization strategy is specifically as follows: Representing the initial three-dimensional model as a graphical model; The graph model is divided into coarse-grained layer and fine-grained layer according to the node importance index; Perform global optimization on the nodes in the coarse-grained layer and adjust the node positions through iterative optimization using the least squares method to minimize the geometric consistency error and the topological consistency constraint error, thus forming a coarse-grained global framework. Add fine-grained layer nodes and edges to the coarse-grained global framework to build a local neighborhood graph; A graph optimization objective function is established for the local neighborhood graph, and multi-scale iterative optimization is performed to update the target node position until all fine-grained nodes are larger than the preset graph optimization target threshold, and the optimized initial three-dimensional model is output.

8. The method for integrated processing of high-resolution satellite imagery 3D modeling based on multi-source data fusion according to claim 7, characterized in that: The process of identifying and correcting defects in the optimized initial 3D model and outputting the final 3D model is as follows: Obtaining the normal vector differences between boundary points in the optimized initial 3D model, and marking boundary point pairs with normal vector differences greater than a preset threshold as potential seam areas; Extract the texture features corresponding to the boundary points and calculate the similarity index of the boundary texture features of adjacent units based on cosine similarity; The boundary points whose similarity index is lower than the preset threshold are marked as texture discontinuity areas; Correct the potential seam areas and texture discontinuity areas marked in the optimized initial 3D model.

9. The method for integrated processing of high-resolution satellite imagery 3D modeling based on multi-source data fusion according to claim 8, characterized in that: The potential seam areas and texture discontinuity areas marked in the optimized initial 3D model are corrected, specifically: Generate a local correction sub-region with the marked boundary point as the center, perform surface fitting on the boundary points of the local correction sub-region, project the boundary points onto the fitting surface, and adjust their positions through iterative optimization to make the seam area spatially continuous with the adjacent unit boundaries and correct the potential seam area; Texture fusion is performed on the local boundary points and their neighborhoods in the local correction sub-region, and the fused texture is mapped back to the point set of the optimized initial 3D model to correct the texture discontinuity area; All local correction sub-regions are sorted sequentially. After each set of local corrections is completed, the 3D model is updated and the seams and texture discontinuity areas are re-evaluated to see if they meet the requirements. If not, repeat the correction steps until the overall seam and texture continuity of the model meet the preset threshold; Output the final 3D model after iterative optimization of local geometry and texture correction.

Images