A city building three-dimensional visual lightweight system based on data fusion

Abstract

The application belongs to the technical field of building three-dimensional visualization, and particularly relates to a city building three-dimensional visual lightweight system based on data fusion, wherein the system comprises a real image acquisition module, a bridging module and a building object updating module. According to the scheme, the point cloud data comparison and vector analysis are used to remove the disappeared building point cloud and add the new building, so that the model of the unchanged part is ensured unchanged, the large-scale point cloud processing calculation amount is reduced, the nearest neighbor search optimization is optimized by adopting the octree block, the point cloud matching calculation amount is further reduced, the verification and model correction mechanism is used, and compared with the traditional static city model, the scheme is more suitable for dynamic scenes such as city planning and disaster monitoring; starting from the unmanned aerial vehicle perspective, the height is gradually reduced, the intermediate visual image is rendered through the initial rough model, the feature gap with the ground perspective is gradually reduced, and the cross-perspective three-dimensional reconstruction quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of building three-dimensional visualization, and particularly relates to a city building three-dimensional visualization lightweight system based on data fusion. BACKGROUND

[0002] The city building three-dimensional visualization lightweight system based on data fusion is a system for model reconstruction of city buildings by means of an airborne laser radar system.

[0003] In the existing approximate scheme, for example, CN115937439B, a city building three-dimensional model construction method, device and electronic equipment, this scheme aims at the technical problem of high labor cost required for constructing a city building three-dimensional model, first generates building attributes based on satellite remote sensing images and AI interpretation results, groups multiple vector outlines of the same building through vector intersection judgment and Hausdorff distance calculation, generates a facade white model through stretching modeling based on the geographical position and height data of the vector outlines, automatically determines the texture scaling ratio and repetition rule by combining the building height and window width, and maps the facade texture data to the white model, which solves the problem of inconsistent attributes of different parts of the building in the prior art, achieves the technical effect of avoiding high cost of manual adjustment of texture mapping, but still has the technical problems of dependence of modeling authenticity and uniformity on random filling, influence on accuracy of texture mapping, and difficulty in dynamic updating of the building model.

[0004] For example, CN118365819A, a city building three-dimensional reconstruction method and device, this scheme aims at the technical problem that the limitation based on the imaging principle of laser radar cannot efficiently collect data in scenes such as rainy days and nights, first uses a millimeter wave radar to replace the laser radar, takes the center of the array radar system as the rotation center, rotates to obtain echo signals at each angle, images the echo signals into a main image and a sub-image through a BP algorithm (BackProjection), calculates three-dimensional coordinates based on an RD equation (RangeDopplerEquation), and outputs three-dimensional data through neighborhood matching, which solves the problem of inconsistent attributes of different parts of the building in the prior art, achieves the technical effect of avoiding high cost of manual adjustment of texture mapping, but still has the technical problems of limited information fusion, room for improvement in stability and integrity of three-dimensional reconstruction, large difference between high-altitude and ground perspectives, and inability to directly fuse data. SUMMARY

[0005] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a lightweight 3D visual system for urban buildings based on data fusion. While addressing the inconsistency of attributes across different parts of a building in existing technologies and achieving the technical effect of avoiding the high cost of manual texture mapping adjustments, it still suffers from the technical problems of relying on random filling for model realism and uniformity, affecting the accuracy of texture mapping, and difficulty in dynamically updating building models. This solution, through point cloud data comparison and vector analysis, removes missing building point clouds and adds new buildings, ensuring that the model of unchanging parts remains unchanged while reducing the computational load of large-scale point cloud processing. Furthermore, it employs octree block optimization and nearest neighbor search optimization to further reduce the computational load of point cloud matching. By utilizing verification and model correction mechanisms, this approach is more suitable for dynamic scenarios such as urban planning and disaster monitoring compared to traditional static city models. Addressing the limitations of existing technologies that primarily use single main images, single secondary images, or a small number of secondary images for matching, resulting in limited information fusion and room for improvement in the stability and completeness of 3D reconstruction, as well as the significant differences between aerial and ground perspectives and the inability to directly fuse data, this solution starts from the UAV's perspective, gradually decreasing the altitude layer by layer. It renders intermediate-perspective virtual images using an initial coarse model, progressively narrowing the feature gap with the ground perspective. By using pixel differences, structural similarity, and semantic matching, it enhances rendering realism, ensuring that the model is consistent with the real scene in terms of geometry, structure, and semantics, thereby improving the quality of cross-perspective 3D reconstruction.

[0006] The technical solution adopted by the present invention is as follows: The present invention provides a lightweight 3D visual system for urban buildings based on data fusion, the lightweight 3D visual system for urban buildings based on data fusion includes a real image acquisition module, a bridging module and a building object update module;

[0007] The real image acquisition module acquires drone images and ground view images by using real photos taken by drones and on the ground, and the drone images and ground view images form an image dataset;

[0008] The bridging module uses a bridging method to bridge the feature gap between the UAV and the ground view, and constructs a three-dimensional model of urban buildings.

[0009] The building object update module uses a building object update method to identify changes in building form, update the three-dimensional model of urban buildings, and track geometric and semantic changes of buildings.

[0010] Furthermore, the bridging method specifically includes the following steps:

[0011] Step A1: Initialization. Specifically, since the differences in perspective between drone images are small, a neural network is used to first construct an initial rough building layout model based on the drone photos from a high-altitude perspective.

[0012] Step A2: iteratively generating intermediate viewpoints, specifically, starting from the height of the UAV, lowering the height layer by layer, rendering a virtual image of the current iteration height using the rough building layout model, adding the virtual image to the image dataset, aligning and unifying the image dataset to a globally consistent three-dimensional spatial coordinate system, and obtaining the current camera pose;

[0013] Step A3: virtual training, specifically, using three-dimensional Gaussian splashing technology, rendering a virtual image of the next lower height based on the image dataset and the camera pose, using a loss function to improve the realism of the virtual image, generating a virtual image of the ground height, using a semantic segmentation network to determine the semantics of the buildings in the virtual image, and training the neural network by comparing the UAV map, ground view map and virtual image, optimizing the rough building layout model, and finally constructing a three-dimensional model of urban buildings, wherein the loss function formula is as follows:

[0014] ;

[0015] wherein, the total loss function is represented by L, the pixel difference score of the color difference of each pixel in the virtual image and the real photo is represented by Lc, the structural similarity score between the virtual image and the real photo is represented by Ls, the matching degree of the semantic judgment result between the virtual image and the real photo is represented by Ls.

[0016] Further, the building object updating method specifically includes the following steps:

[0017] Step B1: predefinition, specifically, converting the image dataset used when establishing the three-dimensional model of urban buildings into point cloud data, and recording it as the first point cloud set, and recording the three-dimensional model of urban buildings corresponding to the first point cloud set as the first building model;

[0018] Step B2: new data acquisition, specifically, using an airborne laser radar system to emit laser pulses in the air over the city to obtain a current three-dimensional spatial point dataset, recorded as the second point cloud set;

[0019] Step B3: preprocessing, specifically, denoising and coordinate registration of the second point cloud set, and using a semantic segmentation algorithm to distinguish the buildings in the second point cloud set;

[0020] Step B4: vector intersection, used to determine the existence of buildings according to the spatial intersection of the first point cloud set and the second point cloud set, specifically, the area where there is no corresponding building of the first point cloud set in the second point cloud set is marked as a disappeared building, and the area where there is no corresponding building of the second point cloud set in the first point cloud set is marked as a new building, the first point cloud set and the second point cloud set are blocked using an octree to reduce the calculation amount of the nearest neighbor search, and the nearest neighbor search algorithm is used to obtain the vector distance of the buildings between the first point cloud set and the second point cloud set, the addition of buildings and the reconstruction of roof structures will directly reflect the changes in roof height and shape, and the change in the vertical distance in the Z-axis direction can be more intuitive, in comparison, the opening and closing of windows, the increase and decrease of decorative components, and other facade changes have less impact on the overall geometric update of the building, therefore, the projection value of the vector distance in the Z-axis direction is calculated, since the roof point of the building is theoretically perpendicular to the ground, the projection value is higher, and the normal vector of the facade point is mostly horizontal, so the projection value is lower, therefore, a first threshold is set, only the point cloud corresponding to the projection value higher than the first threshold is retained, the facade points of the building are filtered, the connected component clustering is performed on the point cloud, the point cloud cluster is obtained and retained, and the building corresponding to the point cloud cluster is marked as a changed building;

[0021] Step B5: verification stage, used to improve the accuracy of the model, and the changed building is constructed in the urban building three-dimensional model.

[0022] In step B5, the verification stage specifically includes the following steps:

[0023] Step B51: constructing the changed building in the urban building three-dimensional model, verifying whether the new building and the disappeared building are constructed in the urban building three-dimensional model according to the point cloud corresponding to the new building in the second point cloud set;

[0024] Step B52: removing the point cloud corresponding to the disappeared building and the new building in the first point cloud set, verifying whether the part of the urban building three-dimensional model that is not the new building and the disappeared building is unchanged according to the first point cloud set;

[0025] Step B53: correcting the urban building three-dimensional model according to the verification result.

[0026] The beneficial effects obtained by the above-mentioned scheme are as follows:

[0027] (1) In view of the inconsistent attributes of different parts of the building in the prior art, the technical effect of avoiding manual adjustment of texture mapping with high cost is realized, but there are still technical problems that the reality and unity of modeling depend on random filling, affecting the accuracy of texture mapping, and the building model is difficult to dynamically update, the scheme compares and analyzes the point cloud data, removes the disappeared building point cloud, adds the new building, ensures that the unchanged model of the unchanged part, reduces the calculation amount of large-scale point cloud processing, uses octree block optimization nearest neighbor search optimization, further reduces the point cloud matching calculation amount, uses the verification and model correction mechanism, compared with the traditional static city model, it is more suitable for dynamic scenes such as city planning and disaster monitoring;

[0028] (2) In view of the technical problems that the prior art mainly uses single main image, single auxiliary image or a small amount of auxiliary image matching, the information fusion is limited, the stability and integrity of three-dimensional reconstruction still have room for improvement, the difference between high-altitude and ground view is large, and the data cannot be directly fused, the scheme starts from the unmanned aerial vehicle view, gradually reduces the height, renders the intermediate view virtual image through the initial rough model, gradually narrows the feature gap with the ground view, improves the rendering reality through pixel difference, structural similarity and semantic matching, ensures that the model is consistent with the real scene in the geometric, structural and semantic level, and improves the cross-view three-dimensional reconstruction quality. BRIEF DESCRIPTION OF DRAWINGS

[0029] Fig. 1 A module connection diagram of a city building three-dimensional visual lightweight system based on data fusion provided by the application is provided.

[0030] Fig. 2 A flowchart of the bridging method is shown.

[0031] Fig. 3 A flowchart of the building object updating method is shown.

[0032] The accompanying drawings are used to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0034] Embodiment one: refer to Figs. 1-3The embodiment provides a city building three-dimensional visual lightweight system based on data fusion, which comprises a real image acquisition module, a bridging module and a building object updating module.

[0035] The real image acquisition module acquires an unmanned aerial vehicle (UAV) image and a ground view image through real photos taken by a UAV and the ground, and forms an image data set from the UAV image and the ground view image.

[0036] The bridging module adopts a bridging method to bridge the feature gap between the UAV and the ground view, and constructs a city building three-dimensional model.

[0037] The building object updating module adopts a building object updating method to identify changes in building forms, update the city building three-dimensional model, and track geometric and semantic changes of buildings.

[0038] Embodiment two: refer to Figs. 1-2 The bridging method comprises the following steps:

[0039] Step A1: initialization, specifically, when the view angle difference between UAV images is small, a neural network is used to first construct an initial rough building layout model according to UAV photos from a high-altitude view angle.

[0040] Step A2: iteratively generate intermediate view angles, specifically, starting from the height of the UAV, the height is gradually lowered, a virtual image of the current iteration height is rendered using the rough building layout model, the virtual image is added to the image data set, the image data set is aligned and unified to a global consistent three-dimensional space coordinate system, and the current camera pose is obtained.

[0041] Step A3: virtual training, specifically, a three-dimensional Gaussian splashing technology is used to render a virtual image of a lower height based on the image data set and the camera pose, a loss function is used to improve the authenticity of the virtual image, a virtual image of the ground height is generated, a semantic segmentation network is used to determine the semantics of the buildings in the virtual image, the neural network is trained by comparing the UAV image, the ground view image and the virtual image, and the rough building layout model is optimized, and finally a city building three-dimensional model is constructed, wherein the loss function formula is as follows:

[0042] ;

[0043] In the formula, Ltotal represents the total loss function, Lpixel represents the pixel difference score of the color difference of each pixel in the virtual image and the real photo, Lstructure represents the structural similarity score between the virtual image and the real photo, a matching degree representing a semantic judgment result between a virtual image and a real photo.

[0044] Embodiment Three Figs. 1-3 This embodiment is based on the above embodiment, and the building object updating method specifically comprises the following steps:

[0045] Step B1: predefinition, specifically, converting the image dataset used when establishing the urban building three-dimensional model into point cloud data, and recording it as a first point cloud set, and recording the urban building three-dimensional model corresponding to the first point cloud set as a first building model;

[0046] Step B2: new data acquisition, specifically, using an airborne laser radar system to emit laser pulses in the air over the city to obtain a current three-dimensional space point dataset, recorded as a second point cloud set;

[0047] Step B3: preprocessing, specifically, denoising and coordinate registration of the second point cloud set, and using a semantic segmentation algorithm to distinguish the buildings in the second point cloud set;

[0048] Step B4: vector intersection, used to determine the existence of buildings according to the spatial intersection of the first point cloud set and the second point cloud set, specifically, the area where there is no building corresponding to the first point cloud set in the second point cloud set is marked as a disappeared building, and the area where there is no building corresponding to the second point cloud set in the first point cloud set is marked as a newly added building, the first point cloud set and the second point cloud set are divided into blocks using an octree to reduce the calculation amount of the nearest neighbor search, and a nearest neighbor search algorithm is used to obtain the vector distance of the buildings between the first point cloud set and the second point cloud set, the addition of buildings and the reconstruction of roof structures will directly reflect as changes in roof height and shape, and the change in the vertical distance in the Z-axis direction can more directly reflect, in comparison, the changes in the facade such as window opening and closing, and the increase and decrease of decorative components have less impact on the overall geometric update of the building, therefore, the projection value of the vector distance in the Z-axis direction is calculated, since the roof point of the building is theoretically perpendicular to the ground, its projection value is higher, and the normal vector of the facade point is mostly horizontal, so the projection value is lower, therefore, a first threshold value is set, only the point cloud corresponding to the projection value higher than the first threshold value is retained, the facade points of the building are filtered, the connected component clustering of the point cloud is performed, the point cloud cluster is obtained and retained, and the building corresponding to the point cloud cluster is marked as a changed building;

[0049] Step B5: verification stage, used to improve the accuracy of the model, and the changed building is constructed in the urban building three-dimensional model.

[0050] Embodiment Four Figs. 1-3 This embodiment is based on the above embodiment, and in step B5, the verification stage specifically comprises the following steps:

[0051] Step B51: constructing the changed building in the urban building three-dimensional model, verifying whether the added building and the disappeared building are constructed in the urban building three-dimensional model according to the point cloud corresponding to the added building in the second point cloud set;

[0052] Step B52: removing the point cloud corresponding to the disappeared building and the added building in the first point cloud set, verifying whether the part of the urban building three-dimensional model other than the added building and the disappeared building is unchanged according to the first point cloud set;

[0053] Step B53: correcting the urban building three-dimensional model according to the verification result.

[0054] Embodiment five: refer to Figs. 1-3 This embodiment is based on the above-mentioned embodiment, and the image data set includes a training set and a test set. The training set contains building images collected from ground view and unmanned aerial high-altitude view. The test set contains real images with a height decreasing by two hundred meters in turn.

[0055] Embodiment six: refer to Figs. 1-3 This embodiment is based on the above-mentioned embodiment, and in step A2, the height is lowered layer by layer, specifically by two hundred meters per layer.

[0056] It should be noted that, in this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0057] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

[0058] The above describes the present application and its embodiments, which are not restrictive, and the drawings only show one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired by it, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solution can be designed and belong to the protection scope of the present application.

Claims

1. A city building three-dimensional visual lightweight system based on data fusion, characterized in that, The real image acquisition module, the bridging module and the building object updating module are included. The real image acquisition module acquires an unmanned aerial vehicle (UAV) image and a ground view image through real photos taken by a UAV and the ground, and generates an image dataset from the UAV image and the ground view image. The bridging module bridges the feature gap between the UAV and the ground view by using a bridging method to construct a three-dimensional (3D) model of urban buildings. The building object updating module identifies changes in building shapes and updates the 3D model of urban buildings by using a building object updating method to track geometric and semantic changes of buildings. The bridging method specifically includes the following steps: Step A1: initialization, specifically, when the view difference between UAV images is small, a neural network is used to first construct a rough building layout model from UAV photos at a high-altitude view; Step A2: iteratively generating an intermediate view, specifically, starting from the height of the UAV, the height is gradually lowered, a virtual image at the current iteration height is rendered using the rough building layout model, the virtual image is added to the image dataset, the image dataset is aligned and unified to a global consistent 3D space coordinate system, and the current camera pose is obtained; Step A3: virtual training, specifically, a 3D Gaussian blur technique is used to render a virtual image at a lower height based on the image dataset and the camera pose, a loss function is used to improve the authenticity of the virtual image, a virtual image at ground height is generated, a semantic segmentation network is used to determine the semantics of buildings in the virtual image, and the neural network is trained by comparing the UAV image, the ground view image and the virtual image to optimize the rough building layout model, and finally a 3D model of urban buildings is constructed.

2. The urban building three-dimensional visual lightweight system based on data fusion according to claim 1, characterized in that, The building object updating method specifically includes the following steps: Step B1: predefinition, specifically, the image dataset used when establishing the 3D model of urban buildings is converted into a point cloud dataset, denoted as a first point cloud set, and the 3D model of urban buildings corresponding to the first point cloud set is denoted as a first building model; Step B2: new data acquisition, specifically, a laser pulse is emitted in the air above the city using an airborne laser radar system to obtain a current 3D space point dataset, denoted as a second point cloud set; Step B3: preprocessing, specifically, the second point cloud set is denoted and coordinate calibrated, and a semantic segmentation algorithm is used to distinguish buildings in the second point cloud set; Step B4: vector intersection, used to determine the existence of buildings according to the spatial overlap of the first point cloud set and the second point cloud set, specifically, a region without a building corresponding to the first point cloud set in the second point cloud set is marked as a disappeared building, a region without a building corresponding to the second point cloud set in the first point cloud set is marked as a newly added building, the first point cloud set and the second point cloud set are divided into blocks using an octree, a nearest neighbor search algorithm is used to obtain the vector distance of buildings between the first point cloud set and the second point cloud set, the projection value of the vector distance in the Z-axis direction is calculated, a first threshold is set, only the point cloud corresponding to the projection value higher than the first threshold is retained, the point cloud is executed for connected component clustering, a point cloud cluster is obtained and retained, and the building corresponding to the point cloud cluster is marked as a changed building. Step B5: verification stage, used to improve the accuracy of the model, and to construct the changed buildings in the urban building three-dimensional model.

3. The urban building three-dimensional visual lightweight system based on data fusion according to claim 2, characterized in that, In step B5, the verification stage, specifically includes the following steps: Step B51: construct the changed buildings in the urban building three-dimensional model, and verify whether the new buildings and the disappeared buildings are constructed in the urban building three-dimensional model according to the point clouds corresponding to the new buildings in the second point cloud set; Step B52: remove the point clouds corresponding to the disappeared buildings and the new buildings in the first point cloud set, and verify whether the part of the urban building three-dimensional model that is not the new buildings and the disappeared buildings is unchanged according to the first point cloud set; Step B53: correct the urban building three-dimensional model according to the verification result.

Images