Low-cost urban-level three-dimensional scene reconstruction method and system

Abstract

The embodiment of the application discloses a low-cost urban three-dimensional scene reconstruction method and system, comprising the following steps: for image data to be reconstructed, based on the three-dimensional space distribution of camera poses, the overall scene in the image data is divided into multiple sub-regions with spatial continuity, each sub-region corresponding to a data block; based on the camera poses and corresponding image data in each sub-region, a three-dimensional model corresponding to each sub-region is constructed by a 3DGS algorithm according to the data block of each sub-region; all three-dimensional models of the sub-regions are fused to obtain a complete overall scene three-dimensional model; the overall scene three-dimensional model is rendered and optimized to obtain a high-quality overall three-dimensional model meeting the requirements. The application reduces the hardware resource demand and overall cost, improves the processing efficiency and reconstruction accuracy, and can realize high-quality urban three-dimensional scene rendering in a low-cost hardware environment.

Description

Technical Field

[0001] This invention belongs to the technical field of 3D scene reconstruction, specifically relating to a low-cost city-level 3D scene reconstruction method and system. Background Technology

[0002] With the continuous advancement of technology, emerging fields such as smart cities, autonomous driving, virtual reality, and augmented reality are booming. As research deepens, the demand for high-precision 3D information of urban scenes is increasing. Therefore, there is an urgent need for a technological means to quickly and efficiently obtain 3D information of urban scenes. Thus, large-scale image set urban scene reconstruction technology has emerged, and the demand is growing. In March 2020, the Neural Radiation Field (NeRF) technology was proposed, providing a new solution for urban scene reconstruction. However, due to the large-scale neural network computation required during rendering, its rendering speed is slow, and its application cost is high, preventing its widespread use in large-scale urban scene reconstruction. In 2023, 3DGS was proposed, gaining popularity due to its superior rendering effects and faster training speed than NeRF, providing a new solution for large-scale urban scene reconstruction. However, due to the vastness of the scenes to be reconstructed, 3DGS still requires long training times and significant memory usage. Training NeRF on the Cityscapes dataset took 72 hours (GPU: NVIDIA V100) and consumed over 40GB of memory; while 3DGS only took 8 hours to train on the same dataset and consumed 12GB of memory. However, when directly applied to city-level scenes, further optimization of the block segmentation strategy is still needed. Therefore, we need to find a 3D reconstruction method and device that can significantly reduce costs and efficiently process city-level data while ensuring reconstruction quality. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a low-cost method for reconstructing city-level 3D scenes. This method reduces hardware resource requirements and overall costs while ensuring reconstruction quality, and improves processing efficiency and reconstruction accuracy. It can achieve high-quality city-level 3D scene rendering in a low-cost hardware environment.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] A low-cost method for reconstructing city-level 3D scenes includes the following steps:

[0006] Step 1: For the image data to be reconstructed, based on the three-dimensional spatial distribution of the camera pose, the overall scene in the image data is divided into multiple spatially continuous sub-regions, each sub-region corresponding to a data block.

[0007] Step 2: Based on the camera pose and corresponding image data in each sub-region, construct the corresponding 3D model of each sub-region using the 3DGS algorithm according to the data block of each sub-region;

[0008] Step 3: Merge the 3D models of all sub-regions to obtain a complete overall scene 3D model;

[0009] Step 4: Render and optimize the overall scene 3D model to obtain a high-quality overall 3D model that meets the requirements.

[0010] Furthermore, the method for partitioning the overall scene in step 1 is as follows:

[0011] First, using global geographic coordinates as the root node, recursively divide and generate spatially continuous cubic sub-blocks;

[0012] Secondly, semantic features are introduced to optimize the block granularity. A semantic segmentation model is used to perform semantic parsing on the image data, associate pose data with semantic labels, and perform secondary processing on the initial spatial sub-blocks, that is, merge spatially adjacent and homogeneous semantic sub-blocks.

[0013] Furthermore, when partitioning the overall scene, the methods for setting the size of the sub-partitions include:

[0014] First, the overall scene is divided into urban and mountainous areas based on the DEM elevation. Areas with DEM elevation greater than the elevation difference threshold are classified as mountainous areas, while areas with DEM elevation not greater than the elevation difference threshold are classified as urban areas.

[0015] For cities, they are further divided into blocks, where the area of ​​each block is not less than an area threshold, and each block is treated as an independent data block.

[0016] Furthermore, the method for constructing the 3D model of each sub-region using the 3DGS algorithm in step 2 is as follows:

[0017] The formula for constructing a model using the 3DG algorithm is as follows:

[0018]

[0019] in, It has an average value Covariance Gaussian function, Indicates the color associated with the view. This represents a traversal of all Gaussian points. The final projection color;

[0020] After the initial three-dimensional reconstruction of each sub-block is completed, the reconstruction result is compared and analyzed with the real scene reference data of the region, and the error index of the model is calculated. If the error exceeds the preset threshold, the parameters of the 3DGS algorithm are adjusted again according to the type and distribution of the error, and then the three-dimensional reconstruction is performed again based on the adjusted parameters, and the iteration is continuously performed until the error of the reconstructed model meets the accuracy requirement.

[0021] Further, the method of fusing the three-dimensional models of all sub-regions in step 3 comprises:

[0022] A unified coordinate system is established, and the three-dimensional models of all sub-regions are converted to the unified coordinate system for alignment and splicing. The conversion method between coordinate systems is:

[0023]

[0024] wherein, is the coordinate in the original coordinate system, is the converted coordinate, K is the scaling, R is the rotation, and dX is the translation.

[0025] After splicing, for the boundary regions between the three-dimensional models of all sub-regions, the feature matching is performed on the overlapping parts by using the camera pose and image information of the boundary regions, so that seamless transition is realized between the adjacent sub-block models. The feature matching algorithm includes one or more of scale invariant feature transform (SIFT) based feature matching, ORB feature matching, deep learning based feature matching, and point cloud normal vector based feature matching.

[0026] Further, in the three-dimensional model fusion process in step 3, the fusion quality of the overall three-dimensional model is evaluated by a quantitative index, and the quantitative index includes geometric flatness. The evaluation method of geometric flatness comprises:

[0027] A curvature distribution model is constructed for the adjacent triangular facets between the fused sub-regional three-dimensional model boundaries, and an abnormal region is identified through statistical difference test. At the same time, the included angle distribution of the facet normal vectors is calculated, and when the angle deviation exceeds the tolerance range, it is determined as geometric discontinuity, at which time the combination of the feature matching algorithm is adjusted. When the fusion defects are detected, an adaptive optimization method based on Gaussian kernel function is used to iteratively adjust the problem area. In the adjustment process, the distribution of the feature matching algorithm is adjusted again, at the same time, the combination of different feature matching algorithm methods is used to increase the number of sample points in the overlapping area, and then the Gaussian point parameters of the overlapping part are optimized, and then the model fusion is performed again. This iterative process is repeated until the quality of the overall scene three-dimensional model meets the requirements.

[0028] Further, in step 4, only the data of the visible part is processed when rendering optimization is performed, assuming that the parameters of the view frustum are:

[0029] ;

[0030] wherein V is a set attribute of a view frustum, is a vertex position of the view frustum, is a main axis direction of the view frustum, representing the orientation of the camera, is a field of view range of the view frustum;

[0031] The geometric information of the scene object is ;

[0032] wherein O is the geometric information of the scene object, is a vertex set of the object, describing the geometric shape of the object, is a face sheet set of the object;

[0033] By judging whether the vertex in the view frustum is within the space range defined by the view frustum, data outside the view frustum is removed, and only data within the space range of the view frustum is processed. O V The present application also provides a system for implementing the above-mentioned low-cost urban-level three-dimensional scene reconstruction method, comprising:

[0034] A sub-region division module is configured to divide the overall scene in the image data into a plurality of sub-regions with spatial continuity based on the three-dimensional spatial distribution of the camera poses, each sub-region corresponding to a data block.

[0035] A sub-region three-dimensional model construction module is configured to construct a three-dimensional model corresponding to each sub-region based on the camera poses and the corresponding image data in each sub-region according to the data block of each sub-region by using the 3DGS algorithm.

[0036] A model fusion module is configured to fuse the three-dimensional models of all sub-regions to obtain a complete overall scene three-dimensional model.

[0037] A model optimization module is configured to perform rendering optimization on the overall scene three-dimensional model to obtain a high-quality overall three-dimensional model meeting the requirements.

[0038] A model optimization module is configured to perform rendering optimization on the overall scene three-dimensional model to obtain a high-quality overall three-dimensional model meeting the requirements.

[0039] An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0040] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0041] Compared with the prior art, the present application has the following advantages: ​

[0042] 1. The present application reduces the demand for hardware resources by reasonable data blocking and parallel computing; the optimization algorithm of each link reduces the amount of calculation, completes the task under low hardware configuration, and reduces the overall cost.

[0043] 2. The data blocking of the present application facilitates parallel computing and improves overall processing efficiency; the optimization of each stage algorithm quickly completes the work of block reconstruction, model fusion, etc., and accelerates the process of city-level three-dimensional scene reconstruction.

[0044] 3. The present application adopts a special modeling method for complex ground objects, accurately splices and optimizes during model fusion, ensures the accuracy and continuity of the reconstructed model, and improves the accuracy of three-dimensional scene reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the low-cost city-level three-dimensional scene reconstruction method of the embodiment of the present application is shown in the figure.

[0046] Figure 2 The data blocking flowchart of the embodiment of the present application is shown in the figure.

[0047] Figure 3 The flowchart of the three-dimensional reconstruction of the embodiment of the present application is shown in the figure.

[0048] Figure 4 The flowchart of the three-dimensional model fusion and optimization of the sub-region of the embodiment of the present application is shown in the figure.

[0049] Figure 5 The flowchart of the rendering optimization of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0052] The present application will be further described below in conjunction with specific embodiments, but is not limited by the present application.

[0053] As shown in the figure, the embodiment of the present application discloses a low-cost city-level three-dimensional scene reconstruction method, which comprises the following steps: Figure 1

[0054] ​Step 1, for the image data to be reconstructed, based on the three-dimensional spatial distribution of camera poses, the overall scene in the image data is divided into multiple sub-regions with spatial continuity, each sub-region corresponds to a data block;

[0055] In this step, in order to solve the problem of large amount of data and complex calculation in large-scale city scene reconstruction, the data set composed of calculated camera poses is first processed by block. According to the geographical position, semantic features and other factors of the scene, the whole scene is divided into multiple sub-regions with spatial continuity, each sub-region corresponds to a data block. Specifically:

[0056] Firstly, the camera pose is obtained by the Structure from Motion (SFM) algorithm, and then based on the three-dimensional spatial distribution of the camera pose, the scene is divided in a coarse-grained manner using octree structure: taking global geographic coordinates as root node, covering the whole target area, recursively dividing to generate spatially continuous cubic sub-blocks (sub-block size is dynamically adjusted according to scene density), using density adaptive strategy to select point cloud density, feature point number, texture complexity, etc. according to the applicable scene, to ensure that each sub-block has spatial locality and supports parallel reconstruction calculation. Secondly, the semantic feature optimization block granularity is introduced, and the semantic segmentation model is used to analyze the image data semantically, and the pose data is associated with semantic labels (such as buildings, roads, vegetation), and the initial spatial sub-block is processed again - for spatially adjacent and homogeneous semantic (such as continuous building facade or large area of vegetation) sub-blocks, the region growing algorithm is used to merge them into larger units, which significantly reduces redundant calculation and maintains semantic consistency. Merging adjacent homogeneous semantic blocks to optimize computing efficiency and splitting cross-semantic boundary blocks to enhance reconstruction accuracy.

[0057] As Figure 2 shown, by reasonably setting the size of the sub-region, the complexity of processing a single sub-region can be reduced, parallel computing can be facilitated, the overall processing efficiency can be improved, the demand for hardware resources can be reduced, and thus low-cost processing of city-level data can be realized. For example, according to the division of city blocks, the camera pose data of each block is taken as an independent data block, which ensures that the data amount in each data block is within the processable range, while reducing the overlap and redundancy between different data blocks, and improving the processing efficiency. Different block rules are used for mountainous areas with large terrain undulations and relatively flat urban center areas to fully adapt to the data characteristics of different areas. The pseudo code example is as follows:

[0058] Region Type == "Urban Block"

[0059] Block Size = Road Network Segmentation (Area Threshold = 0.5 );

[0060] Region type == "Mountainous area"

[0061] Block size = DEM elevation segmentation Elevation difference threshold = 100 meters ;

[0062] After the initial data partitioning, the processing time and computational resource usage of each data block are statistically analyzed. If some data blocks are found to have excessively long processing times or high resource usage, the partitioning rules are fine-tuned according to the specific circumstances. For example, if a data block in a mountainous area is difficult to process due to its complex terrain, it can be further subdivided; if a data block in a city block has too small a data volume, adjacent data blocks are merged, and then the data is partitioned and processed again. This process is iterated until a better balance is achieved between the processing efficiency and resource usage of each data block.

[0063] Step 2: Based on the camera pose and corresponding image data in each sub-region, construct the corresponding 3D model of each sub-region using the 3DGS algorithm according to the data block of each sub-region;

[0064] For each data segment, a 3DGS-based 3D reconstruction is performed independently. For example... Figure 3 As shown, using the camera pose and corresponding image data within each block, a 3D model of each sub-region is constructed using the 3DGS algorithm. The formula for the 3DGS algorithm is as follows:

[0065]

[0066] in, It has an average value Covariance Gaussian function, Indicates the color associated with the view. This represents a traversal of all Gaussian points. This is the final projected color.

[0067] For high-rise buildings, overpasses, and other structures, a special modeling method is used to reduce the amount of calculation while ensuring the accuracy of the model. In this process, the parameters of the 3DGS algorithm are adjusted adaptively according to the data characteristics of each sub-block, such as the number and distribution density of Gaussian points. For sub-blocks with dense buildings and rich details, the number and accuracy of Gaussian points are appropriately increased to better capture complex geometric structures and appearance information. For relatively open area sub-blocks, the number of Gaussian points is reduced to improve reconstruction speed and achieve a balance between reconstruction accuracy and efficiency. Compared to traditional methods, this sub-block reconstruction method based on 3DGS technology can quickly complete the three-dimensional reconstruction of each data block with low hardware configuration requirements, reducing costs. After the initial three-dimensional reconstruction of each sub-block is completed, the reconstruction results are compared and analyzed with the reference data of the actual scene in that area, such as high-precision map data, field measurement data, etc. The error indicators of the reconstructed sub-area model are calculated, such as geometric shape error and texture matching error. If the error exceeds the preset threshold, the 3DGS algorithm parameters are adjusted again according to the type and distribution of the error. For example, if the geometric shape error is large, the number of Gaussian points or their distribution density is appropriately increased. If the texture matching error is large, the parameters related to color are optimized ci . Then the three-dimensional reconstruction is performed again based on the adjusted parameters, and the iteration is continued until the error of the reconstructed model meets the accuracy requirements.

[0068] Step 3, fuse all the three-dimensional models of the sub-areas to obtain a complete three-dimensional model of the overall scene;

[0069] After completing the three-dimensional reconstruction of each sub-block, the three-dimensional models of these sub-areas are fused to form a complete three-dimensional model of the large scene. Specifically, as shown in Figure 4 , the three-dimensional models of the sub-areas are converted to a unified coordinate system for alignment and splicing. The coordinate conversion formula is:

[0070]

[0071] wherein, is the coordinate in the original coordinate system, is the converted coordinate, K is the scaling, R is the rotation, and dX is the translation.

[0072] During the fusion process, for the boundary areas between sub-block models, a method based on feature matching and optimization of overlapping areas is used to eliminate gaps and inconsistencies in splicing. Using the camera pose and image information of the boundary area, the Gaussian points in the overlapping part are adjusted and optimized to achieve seamless transition between adjacent sub-block models, ensuring the integrity and consistency of the entire large scene three-dimensional model. For the overlapping areas of model fusion, there are several feature matching methods:

[0073] a、SIFT-based feature matching: The SIFT algorithm detects image features by finding extreme points in different scale spaces, which are invariant to image rotation, scaling, and brightness changes. In the boundary region of the block model, the SIFT feature points of each block boundary image are extracted, and the descriptor is generated. The Euclidean distance between the descriptors is used to match the feature points on different block boundaries, and the point pairs with higher matching degrees are selected. According to the coordinate information of the matching point pairs, the relative transformation relationship between the block models is calculated, which is used for subsequent adjustment of the boundary region Gaussian points.

[0074] In the boundary processing of the block terrain model, the embodiment is based on the SIFT implementation of OpenCV 4.5, and a feature enhancement process is designed for the block boundary region. In specific implementation, first, the block boundary images are preprocessed: the bilateral filter (σ_d=3, σ_r=75) is used to eliminate noise interference while preserving edge features, and then the limited contrast adaptive histogram equalization (CLAHE, clipLimit=2.0) is used to enhance texture details.

[0075] In the feature extraction stage, the SIFT key point detection parameters are set as follows: contrast threshold 0.04, edge threshold 10, to ensure that effective feature points can be detected in the transition area with low contrast. For the generated 128-dimensional descriptor, we use the RootSIFT improvement scheme: first, perform L1 normalization on the original descriptor, then perform square root operation, and finally perform L2 normalization. This method has been proven to improve matching robustness in the SIFT improvement research of CVPR 2012.

[0076] Taking a DEM stitching project as an example, adjacent blocks A (UTM 49N) and B (UTM 50N) have a 3.2-meter height offset at the projection zone boundary. By extracting the SIFT features of the 500-meter buffer zone of the boundary (A detects 1,528 key points, and B detects 1,702 key points), 217 effective matching point pairs are obtained through the above process. The affine transformation matrix is calculated using the M-estimate sample consistency (MSAC) algorithm:

[0077]

[0078] The matrix shows a rotation deviation of 0.03° and a translation deviation of about 3 meters between the blocks. Based on this transformation relationship, the Gaussian point cloud in the overlapping area is adjusted by bidirectional weighting: using the boundary line as the reference, a Sigmoid function (steepness coefficient k=0.05) is used to dynamically allocate the fusion weight, ensuring smooth interpolation of elevation values within the ±20 meter transition zone. Field verification shows that this method reduces the root mean square error (RMSE) at the splice from 4.7 meters to 1.2 meters, significantly better than the 2.8 meters achieved by the traditional ICP algorithm. (Note: The parameters in this example are referenced from the ISPRS Journal 2021 publication "Multi-scale DEM Registration Using Enhanced SIFT Features", and the experimental data is from the USGS 3DEP public data set)

[0079] b. ORB (Oriented FAST and Rotated BRIEF) feature matching: ORB features combine FAST corner detection and BRIEF descriptors, with high computational efficiency, and certain rotation invariance and noise resistance. In the boundary area of the block model, ORB algorithm is used to quickly extract feature points and generate binary descriptors, rBRIEF (Rotation-aware BRIEF) is used to generate 256-bit binary descriptors, and a greedy algorithm is used to select binary test points with high variance and low correlation. Hamming distance is used to match descriptors at different block boundaries to determine matching point pairs. Based on these matching point pairs, the geometric transformation between block models is estimated. Due to the rigidity of the terrain model, a similarity transform model is used, which includes rotation (θ), scaling (s), translation (t), and four degrees of freedom. For parameter solving, a least squares method is used to optimize the objective function: t x , t y ) four degrees of freedom. For parameter solving, a least squares method is used to optimize the objective function:

[0080]

[0081] where, represents the coordinates of the corresponding points in the matching image, represents the initial coordinates of the matched points, R is the rotation matrix, and t is the translation vector.

[0082] Compared with SIFT, ORB is faster in operation and suitable for large-scale model fusion scenarios with certain real-time requirements. For example, when dealing with a large number of patch models in a city-level three-dimensional scene, using ORB feature matching can complete feature matching in a short time, providing fast and effective data support for subsequent model fusion optimization.

[0083] c. Deep learning-based feature matching (e.g., SuperGlue model): SuperGlue is a deep learning-based feature matching model that can learn the matching relationship between image features through an end-to-end network structure. By inputting the image data of the boundary region of the patch model into the trained SuperGlue model, the model outputs the matched feature point pairs and their confidence. Compared with traditional manual feature matching, this method can better handle matching problems in complex scenes and texture similar areas. For example, in areas with similar building styles in the city, the SuperGlue model can accurately find the corresponding relationship between the boundaries of the patch models using its powerful feature learning ability, providing more accurate matching results for model fusion and reducing the number of splicing defects and discontinuous areas.

[0084] d. Feature matching based on point cloud normal vector: For three-dimensional models, feature matching can be performed using the normal vector information of the point cloud data. In the boundary region processing of the block model, the principal component analysis (PCA) algorithm is first used to calculate the normal vector of the discrete point cloud. For each sampling point, select its k neighboring points in the neighborhood, and obtain the eigenvector corresponding to the smallest eigenvalue through eigenvalue decomposition of the covariance matrix as the normal vector direction. The moving least squares method (MLS) is used to smooth and optimize the normal vector direction, eliminating local noise interference. Then, based on the similarity of the normal vector, potential matching point pairs are constructed: by setting the normal vector angle threshold, combined with the bidirectional matching strategy to ensure the geometric consistency of the candidate point pairs, and to exclude the wrong matches with opposite normal vector directions. In the spatial relationship screening, first calculate the local curvature difference of the matching point pairs, and eliminate the candidate pairs with incompatible geometric features. Then, construct a KD-Tree spatial index to verify the neighborhood topological consistency of the matching point pairs: require the standard deviation of the k-neighbor point distance of the candidate point pairs to be no more than 30% of the average point cloud distance, and the corresponding neighborhood point normal vector distribution needs to meet the similarity constraint. Further exclude outliers through the RANSAC algorithm, and retain the reliable matching point set that meets the rigid transformation constraint. Based on the screened matching point pairs, an improved iterative closest point (ICP) algorithm is used to solve the spatial transformation matrix between the blocks. Specifically, the optimal rotation matrix R and translation vector t are calculated through singular value decomposition (SVD), and a distance-weighted objective function is introduced to minimize the registration error. Finally, the obtained transformation matrix is applied to the coordinate adjustment of the Gaussian integral points in the boundary region, and the weight coefficients are corrected through the Jacobian matrix to ensure that the physical field of the adjacent blocks meets the C0 / C1 continuity condition at the interface. The whole process introduces adaptive step optimization and regularization processing to improve the numerical stability under large deformation. This method can fully utilize the geometric information of three-dimensional models, and is especially suitable for scenes with complex geometric structures and relatively insufficient texture information. For example, in complex terrain areas such as mountainous areas, feature matching based on point cloud normal vectors can effectively solve the matching difficulty problem caused by similar textures and improve the model fusion quality.

[0085] The above four methods also have some limitations in matching. In general, if precision is prioritized, SuperGlue+point cloud normal vector is used; if efficiency is prioritized, ORB+SIFT is used. Combining the two approaches can achieve the best results.

[0086] By using the phased "coarse-fine matching combination + dynamic algorithm switching", specifically, in this embodiment, at the beginning of matching, ORB + SIFT is used to greatly improve the matching efficiency, and the matching completion situation is concerned at all times, when the matching has been carried out to 80%, SuperGlue + point cloud normal vector method is converted to ensure the matching quality, which can not only meet the accuracy requirements of urban scene, but also greatly improve the efficiency. This scheme is especially suitable for the following scenarios: limited hardware resources, need to balance cost and effect; the urban scene contains heterogeneous areas (such as mixed buildings and mountains); applications that require real-time rendering (such as autonomous driving simulation).

[0087] After completing the initial model fusion, the overall quality of the fused large scene three-dimensional model is evaluated. In the quality control of three-dimensional model fusion, the continuity of geometry and texture needs to be evaluated by quantitative indicators. In this embodiment, the geometric flatness is used as a quantitative indicator. The detection of geometric flatness is usually based on local curvature analysis and normal vector consistency calculation: a curvature distribution model is constructed for the adjacent triangular patches between the boundaries of the fused sub-regional three-dimensional model, and abnormal areas are identified through statistical difference test; at the same time, the included angle distribution of the patch normal vector is calculated, and when the angle deviation exceeds the tolerance range, it can be determined that the geometry is discontinuous, at this time, we need to adjust the use time of the initial two methods. When the fusion defects are detected, an adaptive optimization method based on Gaussian kernel function is used: by defining geometric smoothing constraints and data fitting constraints, a numerical optimization algorithm is used to iteratively adjust the problem area, and in the adjustment process, the proportion of the above-mentioned two methods (the initial proportion is set to 8:2) is adjusted, a certain number of traversal calculations are performed, and the best time allocation method is selected according to the matching quality. At the same time, different feature matching methods are combined to increase the number of sample points in the overlapping area, and the Gaussian point parameters of the overlapping part are re-optimized. Then the model fusion is performed again, and the iteration is repeated until the quality of the large scene three-dimensional model reaches the ideal state.

[0088] Step 4, render the overall scene three-dimensional model to obtain a high-quality overall three-dimensional model that meets the requirements;

[0089] In order to achieve high-quality rendering effect in a low-cost hardware environment, a series of rendering optimization strategies are adopted, and the frustum clipping function of 3DGS technology is used to process only the visible part of the data during rendering, reducing unnecessary calculation. Assuming that the parameters of the frustum are where V is the set attribute of the frustum, is the position of the vertex of the frustum, is the main axis direction of the frustum, indicating the orientation of the camera, is the field of view range of the frustum, and the geometric information of the scene object is where O is the geometric information of the scene object, is a vertex set of the object, describing the geometric shape of the object, is a face set of the object, by judging whether the vertex in O is within the space range defined by V , the data outside the view frustum is discarded, avoiding useless data processing, greatly reducing the rendering calculation amount and reducing the cost.

[0090] At the same time, the texture of the three-dimensional model is compressed and optimized, and the texture data amount is reduced without affecting the visual effect. Progressive rendering and multi-resolution rendering technology are adopted. When initializing rendering, only a few block models adjacent to the initial view angle are loaded, and corresponding block models are dynamically loaded and unloaded as the view angle moves. At the same time, according to the distance between the camera and the scene, the resolution of rendering is automatically adjusted. When the camera is far away from the scene, the rendering resolution is reduced, the data transmission and calculation amount are reduced, and the rendering speed is improved; when the camera is close to the scene, the rendering resolution is improved, more details are displayed, and the rendering quality is guaranteed, that is, the smooth and high-quality city-level three-dimensional scene rendering is realized on low-cost graphics hardware. After each rendering is completed, the feedback information of the user (such as rendering lagging, picture detail clarity evaluation, etc.) and the system performance index (such as frame rate, memory occupation, etc.) are collected. If there are problems such as rendering lagging or user feedback that the picture details are insufficient, the rendering strategy is adjusted according to the specific circumstances. For example, if the frame rate is low, the texture compression ratio is appropriately reduced to reduce the decompression time, or the number of blocks loaded each time in the progressive rendering is adjusted; if the picture details are insufficient, the resolution switching threshold of the multi-resolution rendering is optimized. Then the rendering is performed again, and the rendering parameters and strategies are iteratively optimized to adapt to different hardware environments and user needs, and the best rendering effect is provided.

[0091] The embodiment of the application also provides a system for realizing the low-cost city-level three-dimensional scene reconstruction method.

[0092] The sub-region division module is used for dividing the overall scene in the image data into a plurality of sub-regions with spatial continuity based on the three-dimensional space distribution of the camera poses, and each sub-region corresponds to a data block.

[0093] The sub-region three-dimensional model construction module is used for constructing the corresponding three-dimensional model of each sub-region according to the data block of each sub-region through the 3DGS algorithm based on the camera pose and the corresponding image data in each sub-region.

[0094] The model fusion module is used for fusing the three-dimensional models of all sub-regions to obtain a complete overall scene three-dimensional model.

[0095] The model optimization module is used for rendering optimization of the overall scene three-dimensional model to obtain a high-quality overall three-dimensional model meeting the requirements.

[0096] An electronic device comprising a memory storing a computer program and a processor, which, when executing the computer program, implements the steps of the above method.

[0097] A computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the above method.

[0098] The above merely provides the preferred embodiment of the present application, and is not intended to limit the embodiments and protection scope of the present application. It should be noted that, for those skilled in the art, any equivalent replacement and obvious changes made according to the content of the present application should be included in the protection scope of the present application.

Claims

1. A low-cost method for reconstructing city-level 3D scenes, characterized in that, Includes the following steps: Step 1: For the image data to be reconstructed, based on the three-dimensional spatial distribution of the camera pose, the overall scene in the image data is divided into multiple spatially continuous sub-regions, each sub-region corresponding to a data block. Step 2: Based on the camera pose and corresponding image data in each sub-region, construct the corresponding 3D model of each sub-region using the 3DGS algorithm according to the data block of each sub-region; Step 3: Merge the 3D models of all sub-regions to obtain a complete overall scene 3D model; Step 4: Render and optimize the overall scene 3D model to obtain a high-quality overall 3D model that meets the requirements; The method for dividing the overall scene into sections in step 1 is as follows: First, using global geographic coordinates as the root node, covering the entire target area, recursively divide and generate spatially continuous cubic sub-blocks; Secondly, semantic features are introduced to optimize the block granularity. A semantic segmentation model is used to perform semantic parsing on image data, associate pose data with semantic labels, and perform secondary processing on the initial spatial sub-blocks, that is, merge spatially adjacent and semantically homogeneous sub-blocks. When partitioning the overall scene, methods for setting the size of sub-partitions include: First, the overall scene is divided into urban and mountainous areas based on the DEM elevation. Areas with DEM elevation greater than the elevation difference threshold are classified as mountainous areas, while areas with DEM elevation not greater than the elevation difference threshold are classified as urban areas. For cities, they are further divided into blocks, where the area of ​​each block is not less than an area threshold, and each block is treated as an independent data block.

2. The low-cost city-level 3D scene reconstruction method according to claim 1, characterized in that, The method for constructing the 3D model of each sub-region using the 3DGS algorithm in step 2 is as follows: The formula for constructing a model using the 3DG algorithm is as follows: in, It has an average value Covariance Gaussian function, Indicates the color associated with the view. This represents a traversal of all Gaussian points. The final projection color; After the initial 3D reconstruction of each block is completed, the reconstruction results are compared and analyzed with the real scene reference data of the corresponding area, and the error index of the model is calculated. If the error exceeds the preset threshold, the parameters of the 3DGS algorithm are adjusted again according to the type and distribution of the error. Then, the 3D reconstruction is carried out again based on the adjusted parameters, and the process is iterated until the error of the reconstructed model meets the accuracy requirements.

3. The low-cost city-level 3D scene reconstruction method according to claim 1, characterized in that, Step 3 involves fusing the 3D models of all sub-regions, including: Establish a unified coordinate system, transform the 3D models of all sub-regions to this unified coordinate system, and then align and stitch them together. The transformation methods between coordinate systems are as follows: in, These are the coordinates in the original coordinate system. These are the transformed coordinates, where K is the scaling, R is the rotation, and dX is the translation. After stitching, for the boundary regions between the 3D models of all sub-regions, feature matching is performed on the overlapping parts using the camera pose and image information of the boundary regions to achieve seamless transition between adjacent block models. The feature matching algorithm includes one or more of the following: scale-invariant feature transformation based SIFT feature matching, ORB feature matching, deep learning-based feature matching, and point cloud normal vector-based feature matching.

4. The low-cost city-level 3D scene reconstruction method according to claim 3, characterized in that, In step 3, during the 3D model fusion process, the overall 3D model fusion quality is evaluated using quantitative indicators, including geometric flatness. The methods for evaluating geometric flatness include: Curvature distribution models are constructed for adjacent triangular faces between the boundaries of the fused sub-region 3D model. Abnormal regions are identified through statistical difference tests. Simultaneously, the angle distribution of the facet normal vectors is calculated. When the angle deviation exceeds the tolerance range, it is determined to be a geometric discontinuity, and the combination of feature matching algorithms is adjusted. When fusion defects are detected, an adaptive optimization method based on Gaussian kernel function is used to iteratively adjust the problem region. During the adjustment process, the allocation method of the feature matching algorithm is readjusted. At the same time, different feature matching algorithm methods are combined to increase the number of sample points in the overlapping area, and the Gaussian point parameters of the overlapping part are re-optimized. Then, the model is fused again. This process is repeated iteratively until the quality of the overall scene 3D model meets the requirements.

5. The low-cost city-level 3D scene reconstruction method according to claim 1, characterized in that, In step 4, during rendering optimization, only the visible data is processed. Assume the view frustum parameters are: ； Where V is a set property of the view frustum. It is the position of the vertex of the view frustum. It is the direction of the principal axis of the view frustum, indicating the orientation of the camera. It is the visual field range of the visual cone; The geometric information of the scene objects is ; Where O represents the geometric information of the scene objects. It is the set of vertices of an object, describing the object's geometry. It is a collection of facets of an object; By judgment O Are the vertices in the middle? V Within the defined spatial range, data outside the view frustum is discarded, and only data within the view frustum spatial range is processed.

6. A system for implementing the low-cost city-level 3D scene reconstruction method according to any one of claims 1-5, characterized in that, include: The sub-region division module is used to divide the image data to be reconstructed into multiple spatially continuous sub-regions based on the three-dimensional spatial distribution of the camera pose. Each sub-region corresponds to a data block. The sub-region 3D model construction module is used to construct the corresponding 3D model of each sub-region based on the camera pose and corresponding image data within each sub-region, according to the data block of each sub-region using the 3DGS algorithm. The model fusion module is used to fuse the 3D models of all sub-regions to obtain a complete overall scene 3D model; The model optimization module is used to optimize the rendering of the overall scene 3D model to obtain a high-quality overall 3D model that meets the requirements.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images