Urban component automatic three-dimensional reconstruction method of sub-meter high-resolution remote sensing image

Abstract

The invention, which relates to the technical field of image processing, discloses a city component automatic three-dimensional reconstruction method based on a sub-meter high-resolution remote sensing image, comprising the following steps: acquiring a sub-meter high-resolution remote sensing image of a target area, and establishing an initial ground-image projection mapping relation based on a rational polynomial coefficient model; acquiring a ground control point, correcting the mapping relation, and outputting a registration image; performing multi-view geometric matching on the registered image to generate dense earth surface point cloud, and inputting the dense earth surface point cloud into a semantic neural radiation field network to generate dense point cloud; performing cross-modal feature alignment and fusion on the dense point cloud and the vector data, and outputting a fused multi-source dense point cloud; according to the method, through fine correction of the multi-source remote sensing image, the problems of large registration error, more point cloud sparse noise and semantic deficiency are solved, and high-precision and automatic three-dimensional reconstruction of urban components is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method for automatically reconstructing urban components from sub-meter high-resolution remote sensing images. Background Art

[0002] In the process of urban digital transformation and smart city construction, the construction of high-precision three-dimensional models has become an important support for applications such as urban planning, infrastructure management, disaster monitoring, traffic simulation, and digital twin cities. However, traditional urban three-dimensional reconstruction methods mostly rely on manual mapping, low-resolution aerial imagery, or single-source LiDAR point clouds. These methods have problems such as high cost, limited coverage, long update cycles, and lack of semantic information. They are unable to meet the current urgent needs for refined urban modeling and intelligent cognition. With the rapid development of remote sensing technology, sub-meter high-resolution remote sensing imagery has gradually become an important data source for urban three-dimensional reconstruction due to its high resolution, wide acquisition range, and strong timeliness. However, in the process of using sub-meter high-resolution remote sensing imagery for automated three-dimensional reconstruction of urban components, several technical challenges still exist.

[0003] For example, the invention patent with publication number CN113343346B discloses a method for rapid modeling of three-dimensional traffic scenes based on high-precision maps, including the following steps: S1: geometrically correcting the remote sensing image and correcting it using a two-dimensional polynomial correction model; S2: extracting the building boundaries and building roof feature lines from the geometrically corrected remote sensing image to generate a feature point chain; setting a threshold to eliminate adjacent point chains, generate building feature lines, and output building vector data; S3: fusing the building vector data into the high-precision map vector data through a one-dimensional polynomial correction model; S4: constructing a data encoding mapping dictionary table to integrate classification information into the high-precision map vector data; S5: using a rule-based modeling method to batch construct three-dimensional models of various traffic elements and buildings along the line; S6: using a refined modeling method to build a refined traffic element model; S7: constructing a three-dimensional traffic scene with monomeric characteristics based on the modeling results of steps S5 and S6.

[0004] For example, the invention patent with publication number CN120471962A discloses a method and device for three-dimensional reconstruction of urban buildings. The method includes: obtaining multi-view satellite orthophotos from the same scene, satellite-borne laser altimetry data of the target area, a digital terrain model, and two-dimensional building vector data. The multi-view satellite orthophotos are accurately geometrically registered and densely matched to generate an initial disparity point cloud. Combining the azimuth information of the multi-view orthophotos, a disparity map with accurate planar positioning is constructed. The elevation value above the ground is calculated using satellite-borne laser altimetry data and a digital terrain model. The elevation value and the disparity value are combined to estimate the disparity elevation scale coefficient to convert the disparity map into a height map. Combined with the two-dimensional building vector data, three-dimensional building vector data of the target area is generated. This solves the problem in the related art that the cost of surveying-level remote sensing data is high and data acquisition is limited, making it impossible to meet the needs of three-dimensional reconstruction of cities on a global scale.

[0005] The above disclosed technical solutions have at least the following technical problems: Traditional automated 3D reconstruction methods for urban components from sub-meter high-resolution remote sensing images usually only perform one-time global optimization for RPC-based image correction, which is prone to local distortion in complex terrain or multi-temporal imaging conditions. In addition, traditional stereo matching methods have poor robustness to occluded areas and texture-poor areas, and point clouds contain many holes and outliers.

[0006] In view of the above problems, the present invention proposes a solution. Summary of the Invention

[0007] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an automated three-dimensional reconstruction method for urban components from sub-meter high-resolution remote sensing images. Through precise correction of multi-source remote sensing images, dense point cloud generation, semantic NeRF enhancement and vector data fusion, the problems of large registration errors, sparse point cloud noise and semantic loss in traditional methods are solved, thereby achieving high-precision and automated three-dimensional reconstruction of urban components.

[0008] To achieve the above object, the present invention provides the following technical solutions: An automated three-dimensional reconstruction method for urban components from sub-meter high-resolution remote sensing images comprises: obtaining sub-meter high-resolution remote sensing images of a target area, establishing an initial ground-image projection mapping relationship based on a rational polynomial coefficient model; obtaining ground control points, correcting the mapping relationship, and outputting a registered image; performing multi-view geometric matching on the registered image to generate a dense surface point cloud, which is input into a semantic neural radiation field network to generate a dense point cloud; performing cross-modal feature alignment and fusion of the dense point cloud and vector data, and outputting a fused multi-source dense point cloud.

[0009] In a preferred embodiment, the sub-meter high-resolution remote sensing image of the target area is obtained, and an initial ground-image projection mapping relationship is established based on a rational polynomial coefficient model, specifically as follows: multiple sub-meter high-resolution remote sensing images of the target area and the accompanying rational polynomial coefficient files are obtained, and the projection coordinate system of the image is unified with the geographic coordinate system and elevation benchmark of the elevation reference to form a sub-meter high-resolution remote sensing image and a corresponding initial RPC function set; data features in the sub-meter high-resolution remote sensing image are extracted, screened, and the optimal stereo combination is selected based on the information gain criterion; the RPC function set is parsed, a projection function is established, and the Jacobian is output. Matrix and differentiable projection operator; divide the image area into sub-blocks according to the terrain slope of the target area and the projection linearization error, and generate a multi-scale pyramid image in each sub-block; sample the ground point coordinates in each sub-block, project the ground point coordinates to each pixel plane of the optimal stereo combination through the differentiable projection operator, obtain the corresponding pixel coordinates, and calculate the error covariance of each pixel point through the prior elevation variance and Jacobian matrix; construct the covariance matrix based on the pixel domain covariance, and derive the corresponding confidence ellipse parameters; store the ground point coordinates, pixel coordinates and confidence ellipse parameters as table entries, and organize them into a perception lookup table according to the spatial position index.

[0010] In a preferred embodiment, the data features in the sub-meter high-resolution remote sensing images are extracted, screened, and the optimal stereo combination is selected based on the information gain criterion, specifically as follows: the sub-meter high-resolution remote sensing images are eliminated according to a preset threshold to obtain a set of candidate images; features such as imaging time, off-axis angle, overlap and cloud cover are used as input variables to construct a comprehensive evaluation function, and mapped to the quality score of the candidate images; the contribution of each feature to the final quality score is calculated according to the quality score using the information gain criterion; each feature is sorted according to the size of the information gain, and the sorting results are normalized to obtain the weight value of each feature within a fixed interval; according to the determined feature weights, the comprehensive evaluation function of all candidate images is weighted and solved to obtain the corresponding weighted comprehensive score; the candidate images are sorted from high to low according to the weighted comprehensive score, and the image pair with the highest comprehensive score is selected as the optimal stereo combination.

[0011] In a preferred embodiment, the ground control points are obtained, the mapping relationship is corrected, and the registration image is output, specifically as follows: the known three-dimensional coordinates of the ground control points are obtained in the geographic coordinate system, and the predicted pixel coordinates of the three-dimensional coordinates in the optimal stereo combination are calculated by a microprojection operator; the observed pixel coordinates of the ground control points in the optimal stereo combination are extracted and the predicted pixel coordinates are calculated, and the ground control point projection residuals are output; external multi-source data are obtained, the corresponding three-dimensional geographic coordinates are projected to the optimal stereo combination by a microprojection operator, and the difference calculation is performed in combination with the observed pixel coordinates to form an observation constraint residual; the ground control point projection residuals are combined with the observation constraint residuals to generate a projection residual set; the deviation field of the RPC model is parameterized as a variable node to be optimized, and the projection residual is used as a factor constraint to construct a perceptual map optimization energy function; with the perceptual map optimization energy function as input, a hierarchical optimization solution is performed based on the projection residual set and the RPC deviation field parameterized nodes; according to the RPC after the hierarchical optimization solution, the optimal stereo combination is reprojected to a unified geographic coordinate system, and the error-compensated registration image is output.

[0012] In a preferred embodiment, the perceptual map optimization energy function is used as input, and a hierarchical optimization solution is performed based on the projection residual set and the RPC bias field parameterized nodes, specifically as follows: the perceptual map optimization energy function is used as the optimization target, and the variables to be optimized are divided according to the scope of action to form global parameter nodes, local parameter nodes and time series parameter nodes; the projection residual set is used as a factor constraint, and the parameter nodes are introduced to form a hierarchical factor graph; in the hierarchical factor graph, only the global parameter nodes are activated, and the global parameters of the RPC are roughly corrected by the sparse nonlinear least squares optimization method; after the global layer optimization is completed, the local parameter nodes are activated, and the local bias field within the sub-block is finely corrected by the factor graph optimization method, and local discontinuities are eliminated by the sub-block boundary consistency factor; under the condition of multi-temporal images, the time series parameter nodes are activated, and a time smoothing constraint term is introduced; in the layer graph optimization process, the residual factor weights are dynamically adjusted by the confidence information in the perceptual lookup table, and the bias field parameters of the RPC are updated by the iterative reweighted least squares method until the energy function converges.

[0013] In a preferred embodiment, the multi-view geometric matching of the registered images is performed to generate a dense surface point cloud, specifically as follows: LiDAR point cloud data is obtained and projected to the registered image coordinate system to generate the pixel position and depth value of each projected point; a convolutional neural network is applied to each registered image to extract deep convolution features; a depth feature vector of the corresponding pixel is generated for the projected LiDAR point cloud; the image features and the LiDAR depth features are aligned according to the pixel position, and a cross-modal feature descriptor is formed through feature splicing and weighted fusion; a multi-resolution image pyramid is constructed to match images of different scales; in each scale layer, a differentiable stereo matching method is used to calculate the matching error, and the error is used as a differentiable loss to iteratively optimize the depth value; the optimized depth value is projected to the geographic coordinate system to generate an initial dense surface point cloud; the initial dense surface point cloud is subjected to noise filtering by applying confidence weights and neighborhood consistency constraints to eliminate abnormal points to generate a high-precision dense surface point cloud.

[0014] In a preferred embodiment, the input is input to the semantic neural radiation field network to generate a dense point cloud, specifically as follows: the dense surface point cloud is locally voxelized, the dense surface point cloud is divided into several spatial voxel units, each voxel contains several points, and each point is feature encoded; the features of each point are normalized to generate a point cloud enhanced feature vector; the multi-view image pixels and the point cloud enhanced feature vector are input into the differentiable rendering module of the semantic NeRF network, the volume density and color contribution of each pixel ray are calculated through ray sampling, and the network is trained through volume rendering error and self-supervised consistency constraints; the three-dimensional coordinates, color and semantic probability distribution of the points of each voxel unit are predicted through the trained network to form a preliminary dense point cloud; low confidence points and isolated points are preprocessed, and the processed points are globally coordinate mapped and formatted to generate a final high-precision dense point cloud.

[0015] In a preferred embodiment, the dense point cloud and vector data are cross-modally aligned and fused to output a fused multi-source dense point cloud, specifically as follows: vector data is obtained, and the coordinate system of the vector data is converted so that it is in a unified geographic coordinate system with the dense point cloud; geometric feature encoding of the vector data is performed to form vector features; local neighborhood feature encoding of the dense point cloud is performed to form point cloud features; point cloud features and vector features are mapped to the same high-dimensional feature space, and based on the nearest neighbor search or graph matching method, the local features of the dense point cloud are matched with the features of the vector data; cross-modal feature data is extracted based on the matching results, and a joint optimization energy function is constructed; the three-dimensional coordinates and local feature vectors of the dense point cloud are iteratively optimized with the joint optimization energy function as the target; in each iteration, the point cloud position is adjusted according to the gradient descent method, and it is iterated until the energy function converges; after the iterative optimization converges, the updated three-dimensional coordinates of each point are obtained, and the updated results are stored as the preliminarily optimized dense point cloud; the optimized dense point cloud is fused with the vector data to generate a multi-source dense point cloud.

[0016] In a preferred embodiment, the optimized dense point cloud is fused with the vector data to generate a multi-source dense point cloud, specifically as follows: the optimized dense point cloud and the corresponding vector data are obtained, a corresponding constraint point set is generated for the key urban structures in the vector data, and a preliminary spatial matching is performed on each point in the dense point cloud to determine its nearest neighbor relationship with the vector constraint point; for each point in each dense point cloud, the spatial distance between it and the nearest neighbor point in the constraint point set is calculated as a geometric error indicator, and the point position is adjusted according to the error so that the dense point cloud is geometrically aligned with the vector data; the semantic probability of each point is weighted and updated so that its semantic category distribution is consistent with the vector data to achieve semantic fusion; the fused dense point cloud is subjected to neighborhood consistency filtering to eliminate local noise and isolated points, and abnormal points are removed to form the final dense point cloud.

[0017] The technical effects and advantages of the method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images of the present invention are as follows: 1. This invention establishes an initial ground-image projection relationship based on a rational polynomial coefficient (RPC) model. Combining ground control points with external multi-source data, it employs a hierarchical factor graph optimization method to progressively correct projection residuals. This method not only enables dynamic adjustment of global and local parameters but also introduces temporal smoothing constraints for multi-temporal imagery, thereby ensuring spatial and temporal continuity and consistency of the registered images. This hierarchical optimization mechanism effectively suppresses the local distortion and projection discontinuity issues that arise in traditional single-parameter correction methods, significantly improving the accuracy and stability of image registration.

[0018] 2. This invention achieves high-precision alignment of image and depth features by cross-modal feature fusion of the registered image and the LiDAR point cloud, and by splicing deep convolution features with depth projection features. Supported by a multi-scale image pyramid, a differentiable stereo matching method is used to iteratively optimize depth values, resulting in a highly confident, dense surface point cloud. Compared to traditional stereo matching or single-modality point cloud reconstruction methods, this invention effectively reduces the proportion of noise and outliers while maintaining point cloud density, ensuring that the point cloud results have higher geometric fidelity and continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a method for automated three-dimensional reconstruction of urban components from sub-meter high-resolution remote sensing images according to the present invention. DETAILED DESCRIPTION

[0020] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0021] Example 1, Figure 1 The present invention provides a method for automatically reconstructing urban components from sub-meter high-resolution remote sensing images, comprising the following steps: S1, obtain sub-meter high-resolution remote sensing images of the target area and establish the initial ground-image projection mapping relationship based on the rational polynomial coefficient model; In this embodiment, a sub-meter high-resolution remote sensing image of the target area is obtained, and an initial ground-image projection mapping relationship is established based on a rational polynomial coefficient model, as follows: Obtain multiple sub-meter high-resolution remote sensing images of the target area and their accompanying rational polynomial coefficient (RPC) files, and unify the image's projected coordinate system with the elevation reference's geographic coordinate system (such as WGS84 / UTM) and elevation datum to form a sub-meter high-resolution remote sensing image and the corresponding initial RPC function set. Extract data features from sub-meter high-resolution remote sensing images, screen them, and select the optimal stereo combination based on the information gain criterion. The data features include imaging time, off-axis angle, overlap, and cloud cover. The imaging time is obtained by extracting the shooting timestamp of each image and calculating the imaging time difference between candidate images. The off-axis angle is obtained by calculating the angle between the image center and the normal of the target area based on the satellite attitude and imaging geometric parameters. The overlap is obtained by calculating the overlap ratio between the candidate image coverage area and the target area after geometric registration. The cloud cover is obtained by extracting the image cloud mask using a cloud detection algorithm (such as threshold-based spectral discrimination or deep learning segmentation model) and calculating the cloud coverage rate. Parse the RPC function set, establish forward (ground → image pixel) and reverse (image pixel → ground) projection functions, and output the Jacobian matrix and differentiable projection operator; The image area is divided into sub-blocks according to the terrain slope and projection linearization error of the target area, and a multi-scale pyramid image is generated in each sub-block; Sample the ground point coordinates in each sub-block, project them to each pixel plane of the optimal stereo combination through the differentiable projection operator, obtain the corresponding pixel coordinates, and calculate the error covariance of each pixel point through the prior elevation variance and Jacobian matrix; Construct a covariance matrix based on the pixel domain covariance and derive the corresponding confidence ellipse parameters (major axis, minor axis and direction angle); The ground point coordinates, pixel coordinates, and confidence ellipse parameters are stored as table entries and organized into a perception lookup table by spatial position index.

[0022] The extraction of data features from sub-meter high-resolution remote sensing images, screening, and selection of the optimal stereo combination based on the information gain criterion are as follows: Eliminate image pairs whose imaging time difference is greater than a preset threshold to ensure the temporal consistency of the image combination; Eliminate images with excessive off-axis angles (exceeding the preset angle range) to avoid severe geometric distortion; Eliminate image pairs with overlap below a threshold to ensure the effectiveness of subsequent stereo matching; Eliminate images with cloud cover exceeding a set ratio to ensure the effective information ratio; According to the above elimination rules, the sub-meter high-resolution remote sensing images are eliminated to obtain the candidate image set; Taking imaging time, off-axis angle, overlap, and cloud cover as input variables, a comprehensive evaluation function is constructed, which is mapped into the quality score of the candidate image. The contribution of each feature to the final quality score is calculated using the information gain criterion. The features are sorted according to the information gain, and the sorting results are normalized to obtain the weight value of each feature in the interval [0,1] to ensure the comparability of the weight distribution. The information gain refers to the contribution degree. According to the determined feature weights, the comprehensive evaluation function of all candidate images is weighted and solved to obtain the corresponding weighted comprehensive score; The candidate images are sorted from high to low according to the weighted comprehensive scores to ensure that high-quality image pairs are given priority, and the image pair with the highest comprehensive score is selected as the optimal stereo combination.

[0023] S2, obtains ground control points, corrects the mapping relationship, and outputs the registered image; In this embodiment, the ground control points are obtained, the mapping relationship is corrected, and the registered image is output as follows: Obtain the known three-dimensional coordinates of the ground control points in the geographic coordinate system, and calculate the predicted pixel coordinates of the three-dimensional coordinates in the optimal stereo combination through the differentiable projection operator; Extract the observed pixel coordinates of the ground control points in the optimal stereo combination and perform difference calculation with the predicted pixel coordinates, and output the ground control point projection residual; Obtain external multi-source data, project the corresponding 3D geographic coordinates to the optimal stereo combination using a microprojection operator, and perform difference calculations with the observed pixel coordinates to form observation-constrained residuals. The external multi-source data includes LiDAR elevation points, SAR strong scattering points, and vector road intersections. Combine the ground control point projection residuals with the observation constraint residuals to generate a set of projection residuals; The bias field of the RPC model is parameterized as a variable node to be optimized, and the projection residual is used as a factor constraint to construct a perceptual map optimization energy function. The perceptual map optimization energy function parameterizes the system bias field existing in the RPC and sets each component of the bias field as a variable node to be optimized, so that the model can dynamically correct the deviations of different regions and different features during the optimization process. Then, the projection residual is used as a factor constraint. Specifically, for the projection difference of the same geographical location in the image domain and the LiDAR domain, it is defined as a residual vector, and a corresponding covariance matrix is ​​assigned to each residual through a perceptual lookup table to characterize the confidence level and uncertainty level of the residual, thereby constructing a perceptual map optimization energy function; Taking the perceptual graph optimization energy function as input, a hierarchical optimization solution is performed based on the parameterized nodes of the projection residual set and the RPC bias field; According to the RPC solved by hierarchical optimization, the optimal stereo combination is reprojected to a unified geographic coordinate system. During the reprojection process, the confidence ellipse information updated by the perception lookup table is integrated, and the occluded areas and low-confidence areas are masked to ensure the geometric consistency and reliability of the generated image, and the error-compensated registration image is output.

[0024] In this embodiment, the perceptual map optimization energy function is used as input, and a hierarchical optimization solution is performed based on the projection residual set and the RPC bias field parameterized node, as follows: The perceptual map optimization energy function is used as the optimization objective. The variables to be optimized are divided according to their scope of application, forming global parameter nodes (used to describe the bias and scale distortion of the RPC model as a whole), local parameter nodes (used to characterize the local geometric deviation within each sub-block area), and temporal parameter nodes (used to model the continuity between multi-temporal images). The projection residual set is used as a factor constraint and introduced into the parameter node to form a hierarchical factor graph, in which the residual factors of the ground control points and LiDAR points constrain the global parameter nodes, the residual factors of the road intersections and building corners constrain the local parameter nodes, and the time series parameter nodes are constrained by the time smoothing factor; In the hierarchical factor graph, only the global parameter nodes are activated, and a sparse nonlinear least squares optimization method is used to perform a coarse correction on the global parameters of the RPC model to constrain the long baseline distortion. After the global layer optimization is completed, the local parameter nodes are activated, the local deviation field within the sub-block is finely corrected using the factor graph optimization method, and the local discontinuity is eliminated through the sub-block boundary consistency factor; Under the condition of multi-temporal images, the time series parameter node is activated and the time smoothing constraint term is introduced; During the layer graph optimization process, the residual factor weights are dynamically adjusted by sensing the confidence information in the lookup table, and the bias field parameters of the RPC are updated through the iterative reweighted least squares method until the energy function converges.

[0025] The hierarchical optimization solution includes global layer, local layer and time series layer, as follows: Global layer: A sparse nonlinear least squares optimization method is used to roughly calibrate the global parameters of the RPC model to constrain the long baseline distortion; Local layer: Based on the sub-block division, the factor graph optimization method is used to fine-tune the local deviation field to constrain the local geometric error; Temporal layer: Under multi-temporal data, a time smoothing constraint is introduced to ensure the continuity of the RPC deviation field over time.

[0026] S3, performs multi-view geometric matching on the registered images to generate a dense surface point cloud, which is then input into the semantic neural radiance field network to generate a dense point cloud; In this embodiment, multi-view geometric matching is performed on the registered images to generate a dense surface point cloud, as follows: Obtain LiDAR point cloud data and project it into the registration image coordinate system to generate the pixel position and depth value of each projected point; Applying a convolutional neural network (CNN) to each registered image to extract deep convolution features, including texture, edge, and corner information; Generate a depth feature vector for each pixel of the projected LiDAR point cloud, where the depth feature vector includes three-dimensional coordinates, normal vector, and local curvature information; Align image features with LiDAR depth features by pixel position, and form a cross-modal feature descriptor through feature splicing and weighted fusion. Each descriptor encodes both image texture information and spatial depth information. Construct a multi-resolution image pyramid to match images of different scales to enhance the adaptability to geometric structures of different scales; At each scale layer, a differentiable stereo matching method is used to calculate the matching error, and the error is used as a differentiable loss to iteratively optimize the depth value. Project the optimized depth value to the geographic coordinate system to generate the initial dense surface point cloud, where each point contains three-dimensional coordinates, image texture and depth information; Confidence weights and neighborhood consistency constraints are applied to the initial dense surface point cloud to perform noise filtering, remove outliers, and generate a high-precision dense surface point cloud.

[0027] In this embodiment, the input is fed into the semantic neural radiance field network to generate a dense point cloud, as follows: Perform local voxelization on dense surface point clouds, divide the dense surface point clouds into several spatial voxel units, each voxel contains several points, and perform feature encoding on each point, including: a. Color features, which are obtained by multi-view image projection sampling to obtain the corresponding pixel color; b. Geometric features, including 3D coordinates, normal vectors, and local curvature; c. Semantic features: extracting the probabilities of categories such as buildings, roads, and vegetation through a shallow semantic segmentation network; Normalize the features of each point to generate a unified representation of the point cloud enhanced feature vector; The multi-view image pixels and point cloud enhanced feature vectors are fed into the differentiable rendering module of the semantic NeRF network. The volume density and color contribution of each pixel ray are calculated through ray sampling. The network is trained using volume rendering errors and self-supervised consistency constraints, including view consistency, depth consistency, and semantic consistency constraints. In the training process, point cloud confidence weighting is combined to increase the loss contribution of high-confidence areas and reduce the loss weight of low-confidence areas to enhance the accuracy of key structure areas; Perform local sampling optimization on each voxel unit in the training network to improve rendering efficiency and reduce repeated calculations; The trained network predicts the three-dimensional coordinates, color, and semantic probability distribution of each voxel unit to form a preliminary dense point cloud; Perform neighborhood smoothing and elimination on low-confidence points or isolated points to reduce noise and outliers, thereby improving the overall reliability and accuracy of the dense point cloud; The processed points are mapped to global coordinates and formatted to generate a final high-precision dense point cloud. The formatting includes the standardization of point attributes (coordinates, color, semantic probability) and the generation of voxel indexes.

[0028] S4, performs cross-modal feature alignment and fusion on the dense point cloud and vector data, and outputs the fused multi-source dense point cloud; In this embodiment, the dense point cloud and vector data are cross-modal feature aligned and fused, and the fused multi-source dense point cloud is output as follows: Acquire vector data and perform coordinate conversion on the vector data so that it is in a unified geographic coordinate system with the dense point cloud. The vector data includes vector geometric information of road centerlines, building outlines, bridges, and other urban components. Performing geometric feature encoding on the vector data to form vector features, wherein the geometric feature encoding includes line segment direction, curvature, length, topological relationship and semantic category; Perform local neighborhood feature encoding on the dense point cloud to form point cloud features. The local neighborhood feature encoding includes the three-dimensional coordinates, normal vector, color, semantic probability and point cloud confidence of the point; Map point cloud features and vector features to the same high-dimensional feature space, and match dense point cloud local features with vector data features based on nearest neighbor search or graph matching methods; Extracting cross-modal feature data based on the matching results and constructing a joint optimization energy function, wherein the cross-modal feature data includes geometric errors, semantic differences, and local smoothness constraints of the vector data; Taking the joint optimization energy function as the goal, the three-dimensional coordinates and local feature vectors of dense point cloud points are iteratively optimized; At each iteration, the point cloud position is adjusted according to gradient descent or high-order optimization methods to minimize the geometric error, semantic error and smoothness constraint, and the iteration is carried out until the energy function converges; After the iterative optimization converges, the updated three-dimensional coordinates of each point are obtained, and the updated results are stored as a dense point cloud after preliminary optimization, where the points are each three-dimensional point in the dense point cloud; The optimized dense point cloud is fused with vector data to generate a multi-source dense point cloud, where each point contains three-dimensional coordinates, color, semantic probability and vector constraint information.

[0029] In this embodiment, the optimized dense point cloud is fused with vector data to generate a multi-source dense point cloud, as follows: Obtain the optimized dense point cloud and corresponding vector data, generate corresponding constraint point sets for key urban structures in the vector data (including but not limited to building outlines, road centerlines, and intersections), and perform preliminary spatial matching on each point in the dense point cloud to determine its nearest neighbor relationship with the vector constraint points; For each point in each dense point cloud, the spatial distance between it and the nearest neighbor point in the constraint point set is calculated as the geometric error indicator, and the point position is fine-tuned according to the error to make the dense point cloud geometrically aligned with the vector data; The semantic probability of each point is weighted and updated to make its semantic category distribution consistent with the vector data, thus achieving semantic-level fusion. The fused dense point cloud is subjected to neighborhood consistency filtering to eliminate local noise and isolated points, and to remove abnormal points to form a smooth, continuous and highly accurate final dense point cloud.

[0030] The generation of the corresponding constraint point set is as follows: Building outline: discretize the vector polygon boundary into equally spaced sampling points as constraint points; Road centerline: Fixed-interval sampling is performed along the vector centerline to obtain road constraint points; Road intersections and building corners: Extract vector topological nodes (intersections, corners) and use them directly as high-weight constraint points; Other linear or regional features: Uniformly sample or grid them according to their geometric properties to form corresponding constraint points.

[0031] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0032] The above embodiments may be implemented in whole or in part through software, hardware, firmware or any other combination. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0033] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0034] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0035] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0036] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automated 3D reconstruction of urban components from sub-meter high-resolution remote sensing images, characterized by: include: Obtain sub-meter high-resolution remote sensing images of the target area and establish the initial ground-image projection mapping relationship based on the rational polynomial coefficient model; Obtain ground control points, correct the mapping relationship, and output the registered image; Perform multi-view geometric matching on the registered images to generate a dense surface point cloud, which is then input into the semantic neural radiance field network to generate a dense point cloud. Perform cross-modal feature alignment and fusion of dense point clouds and vector data, and output the fused multi-source dense point cloud.

2. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 1, characterized in that: The sub-meter high-resolution remote sensing image of the target area is obtained, and the initial ground-image projection mapping relationship is established based on the rational polynomial coefficient model, as follows: Obtain multiple sub-meter high-resolution remote sensing images of the target area and the accompanying rational polynomial coefficient files, and unify the image's projection coordinate system with the geographic coordinate system and elevation datum of the elevation reference to form a sub-meter high-resolution remote sensing image and the corresponding initial RPC function set; Extract data features from sub-meter high-resolution remote sensing images, screen them, and select the optimal stereo combination based on the information gain criterion; Parse the RPC function set, establish the projection function, and output the Jacobian matrix and differentiable projection operator; The image area is divided into sub-blocks according to the terrain slope and projection linearization error of the target area, and a multi-scale pyramid image is generated in each sub-block; Sample the ground point coordinates in each sub-block, project them to each pixel plane of the optimal stereo combination through the differentiable projection operator, obtain the corresponding pixel coordinates, and calculate the error covariance of each pixel point through the prior elevation variance and Jacobian matrix; Construct a covariance matrix based on pixel domain covariance and derive the corresponding confidence ellipse parameters; The ground point coordinates, pixel coordinates, and confidence ellipse parameters are stored as table entries and organized into a perception lookup table by spatial position index.

3. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 2, characterized in that: The extraction of data features from sub-meter high-resolution remote sensing images, screening, and selection of the optimal stereo combination based on the information gain criterion are as follows: Eliminate sub-meter high-resolution remote sensing images according to a preset threshold to obtain a candidate image set; Taking imaging time, off-axis angle, overlap, and cloud cover as input variables, a comprehensive evaluation function is constructed and mapped into the quality score of the candidate image. According to the quality score, the contribution of each feature to the final quality score is calculated using the information gain criterion; Sort each feature according to the information gain, normalize the sorting results, and obtain the weight value of each feature within a fixed interval; According to the determined feature weights, the comprehensive evaluation function of all candidate images is weighted and solved to obtain the corresponding weighted comprehensive score; The candidate images are sorted from high to low according to the weighted comprehensive scores, and the image pair with the highest comprehensive score is selected as the optimal stereo combination.

4. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 3, characterized in that: The method of obtaining ground control points, correcting the mapping relationship, and outputting the registered image is as follows: Obtain the known three-dimensional coordinates of the ground control points in the geographic coordinate system, and calculate the predicted pixel coordinates of the three-dimensional coordinates in the optimal stereo combination through the differentiable projection operator; Extract the observed pixel coordinates of the ground control points in the optimal stereo combination and perform difference calculation with the predicted pixel coordinates, and output the ground control point projection residual; Obtain external multi-source data, project the corresponding three-dimensional geographic coordinates to the optimal stereo combination through the microprojection operator, and perform difference calculation based on the observed pixel coordinates to form the observation constraint residual; Combine the ground control point projection residuals with the observation constraint residuals to generate a set of projection residuals; The bias field of the RPC model is parameterized as the variable node to be optimized, and the projection residual is used as a factor constraint to construct the perception graph optimization energy function; Taking the perceptual graph optimization energy function as input, a hierarchical optimization solution is performed based on the parameterized nodes of the projection residual set and the RPC bias field; According to the RPC solved by hierarchical optimization, the optimal stereo combination is reprojected to a unified geographic coordinate system, and the error-compensated registration image is output.

5. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 4, characterized in that: The perceptual graph optimization energy function is used as input, and a hierarchical optimization solution is performed based on the projection residual set and the RPC bias field parameterized node, as follows: The perceptual graph optimization energy function is used as the optimization target, and the variables to be optimized are divided according to their scope of action to form global parameter nodes, local parameter nodes, and timing parameter nodes. The projection residual set is used as a factor constraint, and the parameter nodes are introduced to form a hierarchical factor graph; In the hierarchical factor graph, only global parameter nodes are activated, and the global parameters of RPC are roughly corrected by sparse nonlinear least squares optimization method; After the global layer optimization is completed, the local parameter nodes are activated, the local deviation field within the sub-block is finely corrected using the factor graph optimization method, and the local discontinuity is eliminated through the sub-block boundary consistency factor; Under the condition of multi-temporal images, the time series parameter node is activated and the time smoothing constraint term is introduced; During the layer graph optimization process, the residual factor weights are dynamically adjusted by sensing the confidence information in the lookup table, and the bias field parameters of the RPC are updated through the iterative reweighted least squares method until the energy function converges.

6. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 5, characterized in that: The multi-view geometric matching of the registered images is performed to generate a dense surface point cloud, as follows: Obtain LiDAR point cloud data and project it into the registration image coordinate system to generate the pixel position and depth value of each projected point; Apply convolutional neural network to each registered image to extract deep convolution features; Generate the depth feature vector of the corresponding pixel for the projected LiDAR point cloud; Align image features with LiDAR depth features by pixel position, and form a cross-modal feature descriptor through feature splicing and weighted fusion; Construct a multi-resolution image pyramid to match images of different scales; At each scale layer, a differentiable stereo matching method is used to calculate the matching error, and the error is used as a differentiable loss to iteratively optimize the depth value. Project the optimized depth value to the geographic coordinate system to generate the initial dense surface point cloud; Confidence weights and neighborhood consistency constraints are applied to the initial dense surface point cloud to perform noise filtering, remove outliers, and generate a high-precision dense surface point cloud.

7. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 6, characterized in that: The input is fed into the semantic neural radiance field network to generate a dense point cloud, as follows: Perform local voxelization on dense surface point clouds, divide the dense surface point clouds into several spatial voxel units, each voxel contains several points, and perform feature encoding on each point; Normalize the features of each point to generate a point cloud enhanced feature vector; The multi-view image pixels and point cloud enhanced feature vectors are input into the differentiable rendering module of the semantic NeRF network. The volume density and color contribution of each pixel ray are calculated through ray sampling, and the network is trained using volume rendering errors and self-supervised consistency constraints. The trained network predicts the three-dimensional coordinates, color, and semantic probability distribution of each voxel unit to form a preliminary dense point cloud; Low-confidence points and isolated points are preprocessed, and the processed points are mapped and formatted in global coordinates to generate the final high-precision dense point cloud.

8. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 7, characterized in that: The dense point cloud and vector data are aligned and fused across modal features to output a fused multi-source dense point cloud, as follows: Obtain vector data and perform coordinate system conversion on the vector data so that it is in the same geographic coordinate system as the dense point cloud; Encode the geometric features of vector data to form vector features; Encode local neighborhood features of dense point clouds to form point cloud features; Map point cloud features and vector features to the same high-dimensional feature space, and match dense point cloud local features with vector data features based on nearest neighbor search or graph matching methods; Extract cross-modal feature data based on the matching results and construct a joint optimization energy function; Taking the joint optimization energy function as the goal, the three-dimensional coordinates and local feature vectors of the dense point cloud are iteratively optimized; At each iteration, the point cloud position is adjusted according to the gradient descent method, and the iteration is carried out until the energy function converges; After the iterative optimization converges, the updated three-dimensional coordinates of each point are obtained, and the updated results are stored as a dense point cloud after preliminary optimization; The optimized dense point cloud is fused with vector data to generate a multi-source dense point cloud.

9. The method for automatic 3D reconstruction of urban components from sub-meter high-resolution remote sensing images according to claim 8, characterized in that: The optimized dense point cloud is fused with the vector data to generate a multi-source dense point cloud, as follows: Obtain the optimized dense point cloud and corresponding vector data, generate a set of corresponding constraint points for the key urban structures in the vector data, and perform preliminary spatial matching on each point in the dense point cloud to determine its nearest neighbor relationship with the vector constraint points; For each point in each dense point cloud, the spatial distance between it and the nearest neighbor point in the constraint point set is calculated as the geometric error indicator, and the point position is adjusted according to the error so that the dense point cloud is geometrically aligned with the vector data; The semantic probability of each point is weighted and updated to make its semantic category distribution consistent with the vector data, thus achieving semantic fusion; The fused dense point cloud is subjected to neighborhood consistency filtering to eliminate local noise and isolated points, and outliers are removed to form the final dense point cloud.

Images