Smart city planning three-dimensional scene reconstruction optimization system combined with semantic segmentation

Abstract

The invention discloses a smart city planning three-dimensional scene reconstruction optimization system combined with semantic segmentation, and belongs to the technical field of three-dimensional scene reconstruction optimization. The system comprises a data sensing module which collects aerial images and laser point clouds, and complements a sheltered area to obtain city modeling data; the preprocessing module performs denoising, data registration and data fusion on the city modeling data to generate texture point cloud data; the semantic understanding module realizes multi-modal semantic segmentation of the texture point cloud data through a fine-tuned SAM network and an improved RandLA-Net, and semantic tag data is obtained through restoration and optimization; the reconstruction optimization module constructs an initial three-dimensional grid model based on the texture point cloud data, and optimizes a ground feature boundary and a missing region of the model in combination with a semantic tag; and the output application module converts the optimized model into a standard format and outputs the model. Through deep coupling of semantic segmentation and reconstruction optimization, the precision of the three-dimensional scene model is improved, and reliable support is provided for smart city planning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional scene reconstruction optimization, in particular to a three-dimensional scene reconstruction optimization system for smart city planning combined with semantic segmentation. BACKGROUND

[0002] With the advancement of smart city construction, three-dimensional scene reconstruction technology has become the core support for city planning. The mainstream scheme collects aerial images, laser point clouds and other multi-source data through unmanned aerial vehicles, and constructs a three-dimensional model combined with SfM and MVS technologies. However, the existing technology has obvious shortcomings in adapting to high-precision planning requirements. In the data preprocessing link, the missing area completion accuracy of aerial images is low due to occlusion and lighting, the local sparse recovery effect of laser point clouds is poor, and the edge details are easily lost during image denoising and effective feature points are easily removed during point cloud denoising. Cross-modal data registration relies on traditional algorithms and is prone to deviations, affecting the subsequent fusion effect.

[0003] The existing technology also has the problems of insufficient coupling of semantic segmentation and reconstruction, lack of targeted reconstruction optimization, and poor data fusion collaboration. Semantic segmentation is usually performed independently, and the segmentation accuracy of small targets is low. Multi-modal segmentation lacks collaboration, and labels cannot effectively guide reconstruction. Reconstruction is only based on geometric information, and the model has problems such as boundary artifacts and unreasonable occlusion completion. It is difficult to balance macro and micro planning requirements. Image texture and point cloud spatial information are not accurately aligned, and fusion is prone to texture misalignment, ultimately resulting in insufficient model accuracy and reliability, which cannot meet the requirements of smart city site selection, layout simulation and other planning work. Therefore, it is an urgent need in the current smart city planning field to develop a three-dimensional scene reconstruction optimization system that deeply couples semantic segmentation and reconstruction optimization and improves data processing and model construction accuracy throughout the process. SUMMARY

[0004] The present application aims to provide a three-dimensional scene reconstruction optimization system for smart city planning combined with semantic segmentation to solve the problems raised in the background technology.

[0005] To solve the above technical problems, the present application provides the following technical solutions: A three-dimensional scene reconstruction optimization system for smart city planning combined with semantic segmentation, comprising a data perception module, a preprocessing module, a semantic understanding module, a reconstruction optimization module and an output application module.

[0006] The data perception module collects city multi-source modeling data and completes the occluded area of the city multi-source modeling data to obtain city modeling data containing image data and point cloud data; The preprocessing module is used for denoising, data registration and data fusion of the city modeling data in sequence to obtain texture point cloud data; The semantic understanding module performs multi-modal semantic segmentation on the texture point cloud data, generates pixel-level image labels and point-level point cloud labels, and optimizes the pixel-level image labels and point-level point cloud labels to obtain semantic label data. The reconstruction optimization module constructs an initial three-dimensional grid model based on the texture point cloud data, and combines the semantic label data to perform grid optimization on the boundaries and missing areas of the initial three-dimensional grid model, to obtain a city planning three-dimensional scene model. The output application module is configured to convert the city planning three-dimensional scene model into a standard format and output it.

[0007] Preferably, the city multi-source modeling data includes aerial images for representing city scene plane texture and details, and laser point clouds for accurately depicting the three-dimensional spatial form of geographical objects. The city multi-source modeling data is collected by a UAV equipped with a high-resolution aerial camera and a multi-line laser radar device, which flies according to a preset flight route to collect geographical features and landforms in the target area of smart city planning. The UAV is equipped with a GNSS and IMU combined positioning module to ensure the spatio-temporal synchronization of the aerial images and laser point clouds during the collection process. The target area of smart city planning focused by the UAV includes but is not limited to new urban areas, old city reconstruction areas, and key functional land blocks.

[0008] Preferably, the occlusion area completion aims to solve the data missing caused by city scene occlusion and device collection limitations when the UAV collects city multi-source modeling data. Specifically, for the missing areas in the aerial images, which are mostly caused by pixel blanking, a multi-view stereo matching algorithm is used for completion. The multi-view stereo matching algorithm selects adjacent aerial images containing missing areas, the same time period, and high overlap rate, extracts feature points of each adjacent aerial image through the SIFT (Scale-Invariant Feature Transform) algorithm, establishes the corresponding relationship of the feature points between adjacent aerial images using the FLANN (Fast Library for Approximate Nearest Neighbors) matcher, and removes the false matching pairs combined with the epipolar constraint to obtain the feature point matching result. Based on the feature point matching result, a SGM (Semiglobal Matching) algorithm is used for dense pixel matching to calculate the corresponding pixel values of each blank pixel in the missing area in the multi-view adjacent aerial images, generate the pixel-level texture information of the missing area by weighted averaging, and obtain the aerial image with the missing area completed, i.e., image data. The local point density of the laser point cloud is counted by voxelizing a grid, the area with a point density less than 100 pts / m2 is automatically screened out, and the point cloud in the area is extracted in the form of a local point cloud block. The extracted local point cloud block is subjected to standardization processing for normalizing the point cloud coordinates of the point cloud in the local point cloud block to the interval [-1, 1]. The local point cloud block subjected to the standardization processing is input into a pre-trained PointNet++ point cloud completion network for point cloud completion, thereby obtaining the laser point cloud, i.e. point cloud data, of the sparse or blank area of the completed point cloud. The PointNet++ point cloud completion network is trained based on massive urban scene point clouds, can learn the geometric features of urban scene ground objects, and generate supplementary point clouds with true structures conforming to the urban scene, so that the density of the completed point cloud meets the standard and the structure is complete.

[0009] Preferably, the denoising processing comprises: The image data is subjected to denoising processing, and the denoising processing is realized by a hybrid denoising network based on deep learning composed of a DnCNN (deep convolutional neural network) and a non-local attention module. The hybrid denoising network based on deep learning takes the DnCNN as a backbone network, the DnCNN contains 8 layers of 3x3 convolution layers and ReLU activation functions, and a non-local attention module is embedded between the 3rd-5th convolution layers of the DnCNN. The non-local attention module calculates the similarity weight of any two pixels in the image data through a similarity weight formula, strengthens the identification and removal of large-area continuous noise by using the similarity weight, and at the same time, focuses on retaining the ground object details in the high-similarity area, thereby solving the problem of insufficient processing of long-distance noise by the DnCNN. The point cloud data is subjected to denoising processing, and the local density characteristics of the point cloud data are analyzed by SOR statistical filtering to remove outliers. The point cloud data after removing the outliers is subjected to point cloud completion processing by a PointCleanNet network, thereby obtaining the point cloud data subjected to the denoising processing. The similarity weight formula is: ; wherein, is the similarity weight, is the pixel feature of the pixel, is the similarity coefficient.

[0010] Preferably, the data registration is: cross-modal matching points of the denoised image data and point cloud data are extracted by using the LoFTR algorithm, and the RANSAC (Random Sample Consensus) algorithm is used to remove the mismatched points in the cross-modal matching points to obtain an effective matching point set, and the effective matching point set is used to solve an initial spatial transformation matrix by using the least square method, the initial spatial transformation matrix can preliminarily map the two-dimensional pixel coordinates of the image to the three-dimensional coordinate system of the point cloud, and accordingly, the two-dimensional feature points in the denoised image data are mapped and converted into an image feature point three-dimensional point set by using the initial spatial transformation matrix, and then the image feature point three-dimensional point set and the denoised point cloud data are processed by using the ICP algorithm to obtain a precise spatial transformation matrix; the data fusion is: based on the precise spatial transformation matrix, the pixel texture information of each pixel of the denoised image data is re-projected into the corresponding three-dimensional point of the denoised point cloud data, and texture point cloud data with geometric accuracy and texture authenticity are obtained.

[0011] Preferably, the multi-modal semantic segmentation is: the image part of the texture point cloud data is segmented by using the SAM network fine-tuned for urban scene elements to generate pixel-level image labels, and the SAM network fine-tuned for urban scene elements is obtained by: taking the core ground objects concerned in the smart city planning as the annotation categories, the annotation categories are the urban scene elements, a data set containing a plurality of urban scene image samples is constructed, the data set is input into the original SAM network, the classification head parameters and the feature extraction layer weights of the original SAM network are adjusted by using the transfer learning to improve the feature sensitivity of the original SAM network to the ground objects of the urban scene, and the SAM network fine-tuned for urban scene elements is obtained, and the urban scene elements include but are not limited to buildings, roads, green lands and water bodies; The point cloud part of the texture point cloud data is segmented to generate a point-level point cloud label by using an improved RandLA-Net as a prior constraint of a pixel-level image label, the point-level point cloud label is consistent with the label category of the pixel-level image label, and the improved RandLA-Net is obtained by replacing a fixed-step downsampling module of the RandLA-Net with dynamic downsampling based on point cloud density. The point cloud of the urban scene has a significant density difference, and the fixed-step downsampling can cause details of a high-density region to be redundant or lost and calculation of a low-density region to be redundant. The dynamic downsampling can adaptively adjust a sampling interval based on the density to balance detail retention and calculation efficiency. Specifically, the point cloud is voxelized and divided, a 0.3m*0.3m*0.3m voxel grid is used, the number of point clouds in each voxel grid is counted to determine the local point density, double-density thresholds including a high-density threshold and a low-density threshold are set, if the local point density is greater than the high-density threshold, a 0.2m sampling interval is used to ensure that the point cloud of a small three-dimensional structure is retained, if the local point density is less than the low-density threshold, a 0.6m sampling interval is used to reduce redundant calculation, and if the local point density is between the high-density threshold and the low-density threshold, a 0.4m medium sampling interval is used. Meanwhile, a point cloud normal vector feature is introduced into a self-attention layer of the RandLA-Net. The point cloud normal vector feature can directly represent the geometric morphology of a feature, and the existing RandLA-Net only depends on point cloud coordinates and reflectivity features, and it is difficult to distinguish geometrically similar but different feature categories. Specifically, for the point cloud after the dynamic downsampling, a neighborhood three-dimensional point within a 0.2m radius range is searched with each three-dimensional point as a center, a neighborhood point covariance matrix is constructed, eigenvalues and eigenvectors of the neighborhood point covariance matrix are solved by principal component analysis, the eigenvector corresponding to the minimum eigenvalue is the point cloud normal vector feature of the three-dimensional point, the original feature of the point cloud and the point cloud normal vector feature are spliced to form a 7-dimensional feature vector, and the 7-dimensional feature vector is input into the self-attention layer of the RandLA-Net. The original feature includes three-dimensional coordinates X, Y and Z and reflectivity R, and the point cloud normal vector feature includes normal vector components Nx, Ny and Nz.

[0012] Preferably, the repair optimization method is: The pixel-level image label and the point-level point cloud label are respectively filled with a morphological dilation operation to fill label holes and a morphological erosion operation to eliminate label edge redundant noise, to obtain repaired pixel-level image labels and point-level point cloud labels. The pixel-level image label is filled with a 3*3 rectangular structural element through 1-2 times of morphological dilation operation to fill label holes, and the number of times of morphological dilation operation is determined according to the hole area. When the hole area is greater than 10 pixels, the iteration is 2 times, and then 1 time of morphological erosion operation is performed with a 3*3 rectangular structural element to eliminate label edge redundant noise. The point-level point cloud label is filled with a spherical structural element with a radius of 0.2 m through 1 time of morphological dilation operation to generate a weighted average supplementary point to fill point cluster holes in a spherical range with at least 2 effective points of the same class in the blank area. Then, 1 time of morphological erosion operation is performed with a spherical structural element with a radius of 0.3 m to eliminate isolated points with less than 3 effective points of the same class in the spherical range to eliminate label edge redundant noise. Then, the repaired pixel-level image label and the point-level point cloud label are mapped to the same coordinate system through a precise spatial transformation matrix of data registration, and input into a full connection CRF model for boundary optimization, to obtain semantic label data including repaired and optimized pixel-level image labels and point-level point cloud labels.

[0013] Preferably, the construction method of the initial three-dimensional mesh model is: generating a sparse point cloud from texture point cloud data through SfM technology, and encrypting the sparse point cloud to generate a dense point cloud through MVS technology, and then performing surface meshing processing on the dense point cloud by using a greedy projection triangulation algorithm to obtain an initial three-dimensional mesh model; the mesh optimization method is: extracting the repaired and optimized pixel-level image label in the semantic label data as a constraint condition input into a graph cut algorithm, and the graph cut algorithm optimizes the feature boundary of the initial three-dimensional mesh model according to the constraint condition, specifically: extracting the repaired and optimized pixel-level image label in the semantic label data, mapping the pixel-level image label to the texture coordinate space of the initial three-dimensional mesh model through a precise spatial transformation matrix obtained by data registration, establishing an association relationship between the mesh surface and the image-level label category, each mesh surface corresponding to one or more pixel categories in the pixel-level image label, and then inputting the association relationship as a constraint condition into a graph cut algorithm, and the graph cut algorithm realizes boundary optimization by constructing an energy function, wherein the energy function includes a data term and a smoothing term, the formula of the data term is , and the formula of the smoothing term is , for avoiding the abrupt change of the grid surface class at the boundary, the minimum cut algorithm is used to solve the minimum value of the energy function, and the ground object boundary of the initial three-dimensional grid model is optimized; meanwhile, based on the point-level point cloud label of the repair optimization in the semantic label data, the data missing area of the initial three-dimensional grid model is identified through voxel analysis, if the point-level point cloud label density in a voxel is less than 80 pts / m2 and the corresponding number of grid surfaces is less than 5, it is determined that the data missing area, and the data missing area is completed through the GAN network, thereby obtaining the urban planning three-dimensional scene model.

[0014] Preferably, the output application module converts the urban planning three-dimensional scene model into a standard format including OBJ, CityGML, 3DTiles and IFC.

[0015] Due to the adoption of the above technical solutions, the present application has achieved the following technical progress compared with the prior art: 1. The prior art has problems such as low precision of occlusion completion of modeling data, easy loss of details in denoising, large deviation in cross-modal registration, etc., which affect the subsequent reconstruction effect. The present application solves the above defects through a full-process optimization scheme of type-specific completion, differential denoising and precise registration: for missing aerial images, a multi-view stereo matching algorithm is used for completion, and for sparse areas of laser point cloud, a PointNet++ network is used for completion; image denoising uses a hybrid denoising network combining DnCNN and a non-local attention module, and point cloud denoising uses SOR statistical filtering combined with a PointCleanNet network; data registration extracts cross-modal matching points through the LoFTR algorithm, removes false matching points through the RANSAC algorithm, and then solves the precise spatial transformation matrix through the ICP algorithm. This optimization significantly improves the completeness, purity and alignment of the preprocessed data, providing high-quality basic data for subsequent reconstruction.

[0016] 2. In the prior art, semantic segmentation is performed independently, single-modal segmentation accuracy is insufficient, multi-modal lacks coordination, and labels cannot effectively guide reconstruction, resulting in model ground object class confusion and boundary ambiguity. The present application innovatively adopts a coupling scheme of dual-modal collaborative segmentation and semantically constrained reconstruction: the image part generates pixel-level image labels through a SAM network fine-tuned for urban scene elements, the point cloud part uses the labels as prior constraints to generate point-level point cloud labels using an improved RandLA-Net, and the two types of labels form high-precision semantic label data through morphological repair and CRF model boundary optimization; in the reconstruction stage, the semantic labels are used as the core constraints to optimize the ground object boundary through the graph cut algorithm and complete the missing area through the GAN network, realizing the deep linkage of semantic segmentation and reconstruction optimization and solving the related problems caused by the dependence of traditional reconstruction on geometric information.

[0017] 3. The prior art reconstruction is only based on geometric information, and has defects such as boundary artifacts, unreasonable occlusion completion, and difficulty in adapting to different planning requirements. The present application improves the reconstruction quality through the targeted scheme of boundary accurate optimization and missing intelligent completion: based on the pixel-level image label of repair optimization, the energy function containing data items and smoothing items is constructed through the graph cut algorithm to accurately adjust the feature boundary of the initial grid model; based on the point-level point cloud label of repair optimization, the missing area is identified through voxel analysis, and the improved GAN network is used to generate a complete structure continuous with the surrounding geometry; meanwhile, macro and micro models suitable for different planning scenarios are generated, taking into account the use requirements of urban scenarios and plot-level scenarios, so that the final three-dimensional scene model has higher practical value. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings described below are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0019] Figure 1 It is a schematic diagram of the system function module of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0021] Embodiments, such as Figure 1 The wisdom city planning three-dimensional scene reconstruction optimization system combined with semantic segmentation includes a data perception module, a preprocessing module, a semantic understanding module, a reconstruction optimization module and an output application module.

[0022] The data perception module collects city multi-source modeling data, and completes the occlusion area of the city multi-source modeling data to obtain city modeling data containing image data and point cloud data; The preprocessing module is used for sequentially denoising, data registration and data fusion of the city modeling data to obtain texture point cloud data; The semantic understanding module performs multi-modal semantic segmentation on the texture point cloud data to generate pixel-level image labels and point-level point cloud labels, and performs repair optimization on the pixel-level image labels and point-level point cloud labels to obtain semantic label data; The reconstruction optimization module constructs an initial three-dimensional mesh model based on the texture point cloud data, and performs mesh optimization on the boundaries and missing areas of the initial three-dimensional mesh model in combination with the semantic label data, to obtain the urban planning three-dimensional scene model. The output application module is configured to convert the urban planning three-dimensional scene model into a standard format and output.

[0023] Further, the working principle of the present application is illustrated by the following embodiments: In this embodiment, the core planning plot of a new district of a provincial capital city is taken as an application scenario, the plot has an area of about 2 square kilometers, covers residential land, commercial land, green land and municipal roads, and a high-precision three-dimensional scene model needs to be constructed by the system to provide data support for plot planning layout optimization, building height control and pipeline laying planning.

[0024] The data perception module includes two parts of urban multi-source modeling data acquisition and occluded area completion. First, a UAV equipped with a full-format high-resolution aerial camera and a 64-line laser radar device is used for data acquisition. The UAV is also equipped with a GNSS and IMU combined positioning module to ensure the spatio-temporal synchronization of aerial images and laser point clouds during the acquisition process. During acquisition, a pre-set flight route is followed, and the horizontal overlap rate is maintained at 60% and the vertical overlap rate is maintained at 30%. The aerial images and laser point clouds are used to represent the planar texture and details of the urban scene, and the three-dimensional spatial form of the ground objects, respectively. The two together constitute the urban multi-source modeling data. Then, the occluded and missing areas in the urban multi-source modeling data are completed. For the missing areas in the aerial images caused by the occlusion of tall trees, the mutual occlusion of buildings and the limitation of the shooting angle of the device, a multi-view stereo matching algorithm is used. By selecting 3-5 effective adjacent aerial images of the same time period, the same resolution and covering the surrounding area of the missing area, the feature points of each adjacent aerial image are extracted and accurately matched, and the pixel-level texture information of the missing area is generated to complete the image completion. For the areas with low point density in the laser point cloud caused by the limitations of the laser beam penetration, the influence of the acquisition distance and the difference in the reflectivity of the ground object surface, a pre-trained PointNet++ point cloud completion network is used to complete the point cloud completion. After completion, the urban modeling data containing complete image data and point cloud data is obtained.

[0025] The preprocessing module sequentially performs denoising processing, data registration and data fusion on the urban modeling data. Firstly, denoising processing is performed. For the noise characteristics of image data, a hybrid denoising network based on deep learning composed of DnCNN and a non-local attention module is adopted. The network takes DnCNN containing 8 layers of 3x3 convolution layers and ReLU activation function as the backbone network, and embeds a non-local attention module between the 3rd-5th layer convolution layers of DnCNN. The local noise features of the image are extracted by DnCNN, the non-local attention module strengthens the identification and removal of large-area continuous noise, while focusing on preserving the details of the ground objects in the high-similarity area, and realizes the denoising processing of image data. For the noise characteristics of point cloud data, SOR statistical filtering is first used to analyze the local density characteristics of point cloud to remove outliers caused by flying birds and dust reflection in the air, and then PointCleanNet network is used to repair the local sparse area of the point cloud data after removing outliers, to complete the denoising processing of point cloud data. Then, data registration is performed. LoFSTR algorithm is used to extract the cross-modal matching points of the denoised image data and point cloud data. The RANSAC algorithm is used to remove the mismatched points in the cross-modal matching points to obtain an effective matching point set. Based on the effective matching point set, the least squares method is used to solve the initial space transformation matrix that can preliminarily map the two-dimensional pixel coordinates of the image to the three-dimensional coordinate system of the point cloud. According to the initial space transformation matrix, the two-dimensional feature points such as building corner points and road edge points in the denoised image data are mapped and converted into image feature point three-dimensional point sets. Then, the image feature point three-dimensional point sets and the denoised point cloud data are iteratively optimized by ICP algorithm to obtain a precise spatial transformation matrix. Finally, data fusion is performed. Based on the precise spatial transformation matrix obtained by data registration, the pixel texture information of each pixel of the denoised image data is accurately reprojected into the corresponding three-dimensional point of the denoised point cloud data, so that the point cloud has both accurate three-dimensional geometric structure and high-definition texture details, and finally the texture point cloud data is obtained.

[0026] The semantic understanding module includes multi-modal semantic segmentation and label repair optimization. First, multi-modal semantic segmentation is performed. For the image part of the texture point cloud data, a SAM network fine-tuned for urban scene elements is used for segmentation to generate pixel-level image labels. The SAM network fine-tuned for urban scene elements is constructed by using a data set of urban scene image samples with core ground objects in smart city planning as labeled categories. The data set is input into the original SAM network, and the network classification head parameters and feature extraction layer weights are adjusted through transfer learning to improve the sensitivity of the network to urban scene features. For the point cloud part of the texture point cloud data, the generated pixel-level image labels are used as prior constraints, and an improved RandLA-Net is used for segmentation to generate point-level point cloud labels. Subsequently, label repair optimization is performed. Morphological dilation operation is used to fill label holes for the generated pixel-level image labels and point-level point cloud labels, and morphological erosion operation is used to eliminate label edge redundant noise points. For the pixel-level image labels, a 3x3 rectangular structural element is used, the number of morphological dilation operations is determined according to the hole area (2 iterations when the hole area is greater than 10 pixels), and then 1 iteration of morphological erosion operation is performed with the same structural element. For the point-level point cloud labels, 1 iteration of morphological dilation operation is performed with a spherical structural element with a radius of 0.2 m, and then 1 iteration of morphological erosion operation is performed with a spherical structural element with a radius of 0.3 m. The repaired pixel-level image labels and point-level point cloud labels are obtained. Finally, the repaired pixel-level image labels and point-level point cloud labels are mapped to the same coordinate system through the accurate spatial transformation matrix obtained by data registration, and input into the full connection CRF model for boundary optimization. Accordingly, the semantic label data including the repaired and optimized pixel-level image labels and point-level point cloud labels are obtained.

[0027] The reconstruction optimization module includes initial three-dimensional mesh model construction and mesh optimization. First, the initial three-dimensional mesh model is constructed. Sparse point cloud is generated from the texture point cloud data by SfM technology. Dense point cloud is generated by encrypting the generated sparse point cloud by MVS technology. Finally, the surface meshing of the dense point cloud is processed by the greedy projection triangulation algorithm to obtain the initial three-dimensional mesh model. Then, the mesh optimization is performed. The pixel-level image label of repair optimization in the semantic label data is extracted. The pixel-level image label is mapped to the texture coordinate space of the initial three-dimensional mesh model by the accurate spatial transformation matrix obtained by data registration, and the association between the mesh surface and the image-level label category is established. Then, it is input into the graph cut algorithm as a constraint condition. The graph cut algorithm realizes boundary optimization by constructing an energy function containing data items and smoothing items. The data items are used to ensure that the mesh surface category fits the semantic label. When the mesh surface category is consistent with the image-level label category, the energy value is low, and when it is inconsistent, the energy value is high. The smoothing item is used to avoid the sudden change of the mesh surface category at the boundary. When the adjacent mesh surface categories are the same, the energy value is low, and when they are different, the energy value is high. The minimum cut algorithm is used to solve the minimum value of the energy function, and the ground object boundary of the initial three-dimensional mesh model is optimized accordingly. At the same time, based on the repair-optimized point-level point cloud label in the semantic label data, the data missing area of the initial three-dimensional mesh model is identified by voxel analysis with voxel size set to 0.3m x 0.3m x 0.3m. The judgment standard is that the point-level point cloud label density in a voxel is less than 80 pts / m2 and the number of corresponding mesh surfaces is less than 5. The identified data missing area is completed by GAN network. Finally, the urban planning three-dimensional scene model is obtained.

[0028] The output application module mainly converts the urban planning three-dimensional scene model obtained by the reconstruction optimization module into different standard formats and outputs them. According to different use requirements of smart city planning, the urban planning three-dimensional scene model is converted into different standard formats. The OBJ format is used for building single design verification, the CityGML format is used for city-level planning database construction, the 3DTiles format is used for planning scheme visualization display, and the IFC format is used for subsequent building information model integration application. Through multi-format output, the smart city planning multi-scene use requirements are met.

[0029] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A smart city planning 3D scene reconstruction and optimization system combining semantic segmentation, characterized in that, The method comprises the following steps: A data perception module is used to collect city multi-source modeling data and complete the occluded area of the city multi-source modeling data to obtain city modeling data containing image data and point cloud data; A preprocessing module is used to sequentially perform denoising processing, data registration and data fusion on the city modeling data to obtain texture point cloud data; A semantic understanding module is used to perform multi-modal semantic segmentation on the texture point cloud data to generate pixel-level image labels and point-level point cloud labels, and to repair and optimize the pixel-level image labels and the point-level point cloud labels to obtain semantic label data; A reconstruction optimization module is used to construct an initial three-dimensional grid model based on the texture point cloud data, and to perform grid optimization on the boundary and missing area of the initial three-dimensional grid model in combination with the semantic label data to obtain a city planning three-dimensional scene model; An output application module is used to convert the city planning three-dimensional scene model into a standard format and output it. 2.The system of claim 1, wherein, The city multi-source modeling data comprises aerial images and laser point clouds, and is obtained by collecting the ground features and topography of the target area of the smart city planning through a UAV carrying an aerial camera and a laser radar device. 3.The system of claim 2, wherein, The occluded area completion is: the missing area in the aerial image is completed by using a multi-view stereo matching algorithm, and the area with a point density less than 100 pts / m² in the laser point cloud is completed by using a PointNet++ point cloud completion network. 4.The system of claim 3, wherein, The denoising processing comprises: denoising the image data by using a hybrid denoising network based on deep learning, and denoising the point cloud data by using SOR statistical filtering combined with a PointCleanNet network, wherein the hybrid denoising network based on deep learning is a combination network of DnCNN and a non-local attention module. 5.The system of claim 4, wherein, The data registration is: cross-modal matching points of the denoised image data and point cloud data are extracted by using a LoFTR algorithm, and the effective matching point set is obtained by eliminating the mismatched points in the cross-modal matching points by using a RANSAC algorithm, and then the initial spatial transformation matrix is solved based on the effective matching point set, and then the two-dimensional feature points in the denoised image data are mapped and converted into image feature point three-dimensional point sets based on the initial spatial transformation matrix, and then the image feature point three-dimensional point sets and the denoised point cloud data are processed by using an ICP algorithm to obtain a precise spatial transformation matrix; the data fusion is: the pixel texture information of the denoised image data is re-projected to the denoised point cloud data based on the precise spatial transformation matrix to obtain texture point cloud data. 6.The system of claim 5, wherein, The multi-modal semantic segmentation is: the image part of the texture point cloud data is segmented by using a SAM network fine-tuned for urban scene elements to generate pixel-level image labels; the point cloud part of the texture point cloud data is segmented by using an improved RandLA-Net with the pixel-level image labels as prior constraints to generate point-level point cloud labels, wherein the improved RandLA-Net is a dynamic downsampling based on point cloud density instead of a fixed step downsampling of the RandLA-Net, and the point cloud normal vector feature is introduced into the self-attention layer of the RandLA-Net. 7.The system of claim 6, wherein, The repair optimization method is: The pixel-level image label and the point-level point cloud label are respectively filled with morphological dilation operation to fill label holes and eliminated with morphological erosion operation to eliminate label edge redundant noise, to obtain repaired pixel-level image label and point-level point cloud label, and then the repaired pixel-level image label and point-level point cloud label are input into the CRF model for boundary optimization, so as to obtain semantic label data including repaired and optimized pixel-level image label and point-level point cloud label. 8.The system of claim 7, wherein, The construction method of the initial three-dimensional mesh model is: sparse point cloud is generated from texture point cloud data by SfM technology, and dense point cloud is generated from the sparse point cloud by MVS technology, and then the dense point cloud is subjected to surface meshing treatment by using a greedy projection triangulation algorithm to obtain an initial three-dimensional mesh model; the mesh optimization method is: the repaired and optimized pixel-level image label in the semantic label data is extracted and input into a graph cut algorithm as a constraint condition, and the graph cut algorithm optimizes the feature boundary of the initial three-dimensional mesh model accordingly, and meanwhile, based on the repaired and optimized point-level point cloud label in the semantic label data, a data missing area of the initial three-dimensional mesh model is identified, and the data missing area is completed by using a GAN network, so as to obtain the urban planning three-dimensional scene model. 9.The system of claim 8, wherein, The standard format of the urban planning three-dimensional scene model converted by the output application module includes OBJ, CityGML, 3DTiles and IFC.

Images