A building facade three-dimensional model generation method and system based on unmanned aerial vehicle oblique photography

Abstract

The application discloses a kind of based on unmanned plane tilt photography's building facade three-dimensional model generation method and system, belong to unmanned plane measurement technical field.The application solves the problems that building facade three-dimensional modeling method in prior art is complex, cost is high, data acquisition efficiency is low, especially in high-rise or complex environment, it is difficult to safely, comprehensively obtain high-precision data, the application can automatically, efficiently collect full-coverage image by planning multilayer surrounding route and multi-view tilt photography, combined with motion recovery structure and multi-view stereo vision algorithm, can generate high-quality dense point cloud, utilize the machine learning model of fusion random forest and support vector machine, can accurately segment and extract facade point cloud, and finally through surface reconstruction and intelligent texture mapping, automatically generate building facade three-dimensional model with accurate geometric structure and realistic surface texture, significantly improve the automation degree of modeling, processing efficiency and result accuracy.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) measurement technology, specifically to a method and system for generating 3D models of building facades based on UAV oblique photography. Background Technology

[0002] With the continuous advancement of urbanization and the increasing demand for digital management, accurate and efficient 3D modeling of buildings is of great significance in fields such as urban and rural planning, architectural heritage protection, and smart city management. Traditional 3D modeling methods for building facades mainly rely on total stations, 3D laser scanning, or manual photogrammetry. These methods typically suffer from problems such as complex operation, high cost, and low data acquisition efficiency. Especially for high-rise buildings or buildings in complex environments, traditional methods struggle to comprehensively and securely acquire high-precision facade data; therefore, they do not meet current needs. To address this, we propose a method and system for generating 3D building facade models based on UAV oblique photogrammetry. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for generating 3D building facade models based on UAV oblique photography. By combining multi-layered surround flight path planning with multi-view oblique photography for data acquisition, and integrating motion recovery structure, multi-view stereo vision, semantic segmentation of machine learning, and surface reconstruction and texture mapping technologies, the invention achieves the entire process from automated image acquisition to the generation of high-precision 3D facade models with realistic textures, thus solving the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for generating a 3D model of a building facade based on UAV oblique photography, comprising the following steps:

[0005] For the target building, based on the building's geometry and surrounding environmental conditions, plan the multi-route flight path of the UAV and set the shooting parameters of the multi-lens oblique photography system. Control the UAV equipped with the oblique photography system to fly along the preset route and simultaneously collect multi-view oblique image sequences covering the building's facades, roof and adjacent areas.

[0006] The acquired oblique photogrammetric image sequence is preprocessed, and image distortion correction and exposure consistency adjustment are performed sequentially. Based on the structure of motion restoration and multi-view stereo vision algorithm, the preprocessed image sequence is reconstructed in three dimensions to generate dense three-dimensional point cloud data containing buildings and surrounding scenes.

[0007] Based on dense 3D point cloud data, a pre-trained machine learning model is used to perform semantic segmentation on the dense 3D point cloud data, automatically distinguish and extract the point cloud subset belonging to the building facade, and separate the point clouds of the roof, ground and environmental elements.

[0008] The extracted building facade point cloud is optimized and geometrically reconstructed. A two-dimensional surface mesh model representing the geometric structure of the building facade is generated through point cloud filtering and surface reconstruction algorithms.

[0009] Based on the geometric projection relationship between the oblique photogrammetric image and the three-dimensional surface mesh model, texture information is calculated and assigned to the mesh model. Through texture selection and fusion processing, a three-dimensional model of the building facade with a realistic surface appearance is formed.

[0010] The textured 3D building facade model is converted into a universal 3D file format and integrated into a 3D visualization platform to enable real-time rendering and interactive browsing of the 3D building facade model.

[0011] Furthermore, the plan includes multi-route flight paths for the UAV and setting the shooting parameters for the multi-lens tilt photography system, specifically including:

[0012] Based on the three-dimensional outline and facade height of the target building, multiple layered loop routes are designed, including a horizontal loop route along the middle height of the building facade, and a loop route along the top area of ​​the building while maintaining an inclined viewing angle.

[0013] By adjusting the drone's attitude and lens angle, the main optical axis of the tilt camera is made to form an angle within a preset range with the building facade surface;

[0014] Set the synchronous shooting mode and time interval of the multi-lens tilt photography system to ensure that the images captured continuously in the flight path have a high overlap rate and that the preset overlap coverage requirements are met between the side images of adjacent flight paths.

[0015] Furthermore, the acquired oblique photogrammetry image sequence undergoes preprocessing, specifically including:

[0016] By utilizing the camera calibration parameters of the oblique photography system, optical distortion correction is performed on each image to eliminate radial and tangential distortion caused by the lens shape;

[0017] Image enhancement technology is used to perform exposure equalization processing on all images in the sequence, and the image with the best exposure is used as the reference image. The brightness and contrast distribution of images with different exposure states from the reference image are then adjusted.

[0018] Furthermore, based on motion reconstruction structures and multi-view stereo vision algorithms, the preprocessed image sequence is reconstructed in 3D to generate dense 3D point cloud data containing buildings and surrounding scenes, specifically including:

[0019] The correspondence between images is obtained through feature extraction and matching algorithms. The incremental motion recovery structure method is used to solve the shooting position and attitude parameters of each image and generate a sparse three-dimensional point cloud.

[0020] Based on the pose parameters of the images, a multi-view stereo vision process is executed to generate a corresponding depth map for each image. Then, through depth map fusion and point cloud fusion methods, dense 3D point cloud data containing architectural and scene details is formed.

[0021] Furthermore, the machine learning model employs a classification strategy that combines random forests and support vector machines for automatic segmentation and extraction, specifically including:

[0022] Random forest is used to process the original features of 3D points and output a probability feature vector representing the possibility of each category.

[0023] The probabilistic feature vector is used as input, and a support vector machine is used to complete the final semantic label assignment, which is used to achieve the segmentation of the point cloud of the building facade.

[0024] Furthermore, the extracted building facade point cloud is subjected to data optimization and geometric reconstruction, specifically including:

[0025] Noise filtering is applied to the point cloud of the building facade to remove discrete noise points and outlier data.

[0026] Estimate the local normal vector direction of each point in the point cloud;

[0027] Using point cloud and normal vector information, a continuous three-dimensional triangular mesh surface is constructed using a surface reconstruction algorithm to represent the geometric shape of the building facade.

[0028] Furthermore, based on the geometric projection relationship between the oblique photogrammetric image and the 3D surface mesh model, texture information is calculated and assigned to the mesh model, specifically including:

[0029] For each triangular facet of the 3D mesh, based on its 3D coordinates and the external orientation parameters of the image, determine the multiple tilted images that can be projected onto that facet.

[0030] Based on the projection quality evaluation criteria, the most suitable texture source is selected for each patch from the candidate images;

[0031] The selected textures are processed for boundary blending and color consistency to eliminate texture seams and color differences between adjacent patches, achieving a natural transition of the overall texture.

[0032] A system for generating 3D building facade models based on UAV oblique photography, used to execute a method for generating 3D building facade models based on UAV oblique photography, including:

[0033] The data acquisition module is configured to generate an automated flight plan with multiple flight paths in layers based on the geometric features, spatial scale, and on-site environmental factors of the target building. It also sets the synchronous shooting mode, lens angle, and overlap rate parameters of the multi-lens tilt photography system to control the UAV to complete the image data acquisition of the building facade and surrounding area.

[0034] The 3D reconstruction module is configured to perform batch preprocessing on the acquired image data, including distortion correction based on camera intrinsic parameters and exposure normalization based on reference images. It also calculates the image pose and generates dense 3D point cloud data containing buildings and scenes through feature matching, structure-of-motion reconstruction and multi-view stereo vision algorithms.

[0035] The point cloud segmentation module has a built-in machine learning model based on a cascaded random forest and support vector machine. It is configured to extract multi-dimensional geometric and optical features from dense 3D point cloud data and achieve automatic identification and semantic segmentation of building facade point clouds and point clouds of roof, ground, vegetation and ground features through the machine learning model.

[0036] The model generation module is configured to remove outliers and filter data from the segmented building facade point cloud, and to construct a continuous triangular mesh surface model that expresses the geometric structure of the building facade based on normal vector estimation and surface reconstruction algorithms.

[0037] The texture mapping module is configured to establish a projection mapping relationship between triangular facets and multi-view images based on the geometric information of the triangular mesh surface model and the exterior orientation elements of the image. It selects the best texture source for each facet through a weight evaluation mechanism and performs fusion processing on the texture boundaries to generate a three-dimensional texture model.

[0038] The output interaction module is configured to convert textured 3D models into standard 3D format files and integrate them into a 3D visualization platform, providing dynamic loading of 3D models, display of multiple levels of detail, lighting simulation, and user interaction functions.

[0039] Furthermore, a system for generating 3D models of building facades based on UAV oblique photography also includes:

[0040] The model pre-construction module is used to pre-construct the model based on the architectural parameters of the target building, and obtain a pre-constructed model;

[0041] The finite element segmentation module performs finite element segmentation on the pre-built model to obtain multiple finite element elements;

[0042] The finite element mapping module is used to establish a mapping relationship between finite element elements and the position of the UAV based on the position of the finite element elements in the pre-built model, flight path, UAV attitude and camera angle;

[0043] The image segmentation and mapping module is used to segment images acquired by the UAV based on the mapping relationship between finite element elements and the UAV's position, and to determine the image elements of the associated surfaces of the finite element elements.

[0044] The integration module integrates the image elements of the associated surfaces of the finite element elements into the pre-built model to obtain the reference model;

[0045] The fusion module merges the reference model and the textured 3D model to obtain the final model.

[0046] Furthermore, the finite element mapping module performs the following operations:

[0047] Based on the flight path, drone attitude, and camera angle, the regions of the pre-built model are determined to correspond to the images collected by the drone at each point along the flight path.

[0048] When the region contains finite element elements, the corresponding points are stored in the pre-screening group;

[0049] Based on the position of the finite element element in the image, an evaluation value is obtained by evaluating the points in the pre-screened group.

[0050] Based on the drone's position, attitude, and camera angle corresponding to the image, and the surface of the pre-built model corresponding to the image, a secondary evaluation is performed on the points in the pre-screened group to obtain secondary evaluation values.

[0051] By combining the primary and secondary evaluation values, the points mapped to the finite element elements are determined, and the mapping relationship between the finite element elements and the UAV position is established at the determined points.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] This invention overcomes the operational difficulties and safety risks of traditional manual or ground-based equipment in high-rise, complex buildings and hazardous environments by using automated drone flight operations. It can quickly and contactlessly cover all facades and roofs of buildings. By using multi-view images provided by oblique photography, combined with motion reconstruction structure and multi-view stereo vision algorithms, it effectively solves the common problems of occlusion and model voids in single-view photogrammetry. The resulting dense point cloud can more accurately depict facade details and complex structures. On this basis, a semantic segmentation method that integrates machine learning models can automatically and accurately separate the building facade point cloud from the complex scene point cloud containing roofs, ground, vegetation, etc., avoiding the problems of strong subjectivity and low efficiency of traditional manual or rule-based segmentation methods. Furthermore, through intelligent texture selection and fusion technology based on projection relationships, it can endow the geometric model with high-quality, seamless, and realistic textures from the original images, thereby improving the visual realism of the 3D model. Attached Figure Description

[0054] Figure 1 This is a flowchart of a method for generating a 3D model of a building facade based on UAV oblique photography according to the present invention;

[0055] Figure 2 This is a structural diagram of a system for generating 3D models of building facades based on UAV oblique photography, according to the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] To address the technical challenges of existing 3D building facade modeling methods, such as complex operation, high cost, and low data acquisition efficiency, especially in high-rise or complex environments where it is difficult to safely and comprehensively acquire high-precision data, please refer to [the relevant documentation]. Figures 1-2 This embodiment provides the following technical solution:

[0058] A method for generating a 3D model of a building facade based on UAV oblique photography includes the following steps:

[0059] For the target building, based on the building's geometry and surrounding environmental conditions, plan the multi-route flight path of the UAV and set the shooting parameters of the multi-lens oblique photography system. Control the UAV equipped with the oblique photography system to fly along the preset route and simultaneously collect multi-view oblique image sequences covering the building's facades, roof and adjacent areas.

[0060] The acquired oblique photogrammetric image sequence is preprocessed, and image distortion correction and exposure consistency adjustment are performed sequentially. Based on the structure of motion restoration and multi-view stereo vision algorithm, the preprocessed image sequence is reconstructed in three dimensions to generate dense three-dimensional point cloud data containing buildings and surrounding scenes.

[0061] Based on dense 3D point cloud data, a pre-trained machine learning model is used to perform semantic segmentation on the dense 3D point cloud data, automatically distinguish and extract the point cloud subset belonging to the building facade, and separate the point clouds of the roof, ground and environmental elements.

[0062] The extracted building facade point cloud is optimized and geometrically reconstructed. A two-dimensional surface mesh model representing the geometric structure of the building facade is generated through point cloud filtering and surface reconstruction algorithms.

[0063] Based on the geometric projection relationship between the oblique photogrammetric image and the three-dimensional surface mesh model, texture information is calculated and assigned to the mesh model. Through texture selection and fusion processing, a three-dimensional model of the building facade with a realistic surface appearance is formed.

[0064] The textured 3D building facade model is converted into a universal 3D file format and integrated into a 3D visualization platform to enable real-time rendering and interactive browsing of the 3D building facade model.

[0065] The technical effects of the above solution are as follows: Based on motion recovery structure and multi-view stereo image reconstruction technology, high-density and high-precision 3D point cloud data can be recovered from multi-view tilted image sequences, thus providing a reliable data foundation for subsequent 3D modeling. Through machine learning-driven point cloud semantic segmentation, the point cloud of the building facade can be automatically and accurately extracted, thereby effectively separating the roof and environmental interference, and thus realizing refined 3D modeling of the target structure. On this basis, a 3D model of the building facade with a realistic appearance is generated through texture mapping and fusion technology. With the integration of general 3D format and visualization platform, high-quality real-time rendering and interactive browsing are realized, thus fully supporting 3D visualization applications.

[0066] In summary, by comprehensively utilizing UAV oblique photography and multi-view stereo vision algorithms, an efficient process from image acquisition to the generation of a complete 3D model can be achieved, significantly improving the automation and accuracy of 3D modeling of building facades.

[0067] Planning multi-flight paths for drones and setting shooting parameters for a multi-lens tilt photography system, specifically including:

[0068] Based on the three-dimensional outline and facade height of the target building, multiple layered loop routes are designed, including a horizontal loop route along the middle height of the building facade, and a loop route along the top area of ​​the building while maintaining an inclined viewing angle.

[0069] By adjusting the drone's attitude and lens angle, the main optical axis of the tilt camera is made to form an angle within a preset range (such as an angle of 20-45 degrees) with the building facade surface.

[0070] Set the synchronous shooting mode and time interval of the multi-lens tilt photography system to ensure that the images captured continuously in the flight direction have a high overlap rate and that the images in the lateral direction of adjacent flight directions meet the preset overlap coverage requirements, such as the overlap rate of adjacent shooting positions being no less than 80% in the flight direction and no less than 60% in the lateral direction.

[0071] In this embodiment, based on the three-dimensional outline model of the target building, on-site survey, or pre-flight scanning data, its geometric dimensions, facade height, roof shape, and surrounding spatial distribution are accurately obtained. According to the main height of the building facade, at least two layered circling flight paths are set. The first is a main horizontal circling flight path, with its flight height set at one-half to two-thirds of the total height of the building facade to ensure that the lens can vertically cover the central area of ​​the facade. The second is a top-tilted circling flight path, with its flight height set at an appropriate distance above the building's eaves or roof. The drone's roll or pitch angle is preset so that the principal optical axis of at least one lens in the onboard tilting camera system's lens group points downwards at a preset angle (e.g., 30 degrees) towards the top and upper part of the building. Facade; Based on the camera focal length, sensor size, and preset ground sampling distance, the ground coverage of a single image is calculated. Combined with the minimum overlap requirements of 80% in the forward direction and 60% in the lateral direction, the waypoint spacing and route spacing of each route are accurately calculated to plan a closed polygonal route path around the building. At the same time, the flight control software is set to keep the UAV flying at a constant speed on each route and trigger the multi-lens oblique photography system to take synchronous pictures at constant time intervals to ensure that the images of corresponding positions on adjacent routes can also meet the lateral overlap requirements. Finally, all layered routes are integrated into an automated flight mission sequence in order from low to high or from high to low, and a collision risk assessment and safety boundary check are performed before execution.

[0072] The technical effects of the above solution are as follows: By designing a horizontal and top-level double-layered flight path for the building facade, combined with precise control of the UAV attitude and lens angle, it can systematically and without blind spots collect multi-view tilted images of each facade and roof of the building. This avoids facade occlusion and data loss caused by single-flight shooting, ensuring the integrity and applicability of the original image data. Furthermore, by setting a synchronous shooting mode with a high overlap rate, it can achieve high continuous coverage of images in both the flight direction and the side direction. This enhances the reliability and density of feature matching and 3D point cloud computing in the multi-view stereo vision algorithm, thereby directly improving the geometric accuracy and integrity of subsequent 3D point cloud generation and surface model reconstruction.

[0073] The acquired oblique photogrammetry image sequence is preprocessed, specifically including:

[0074] By utilizing the camera calibration parameters of the oblique photography system, optical distortion correction is performed on each image to eliminate radial and tangential distortion caused by the lens shape;

[0075] Image enhancement technology is used to perform exposure equalization processing on all images in the sequence, and the image with the best exposure is used as the reference image. The brightness and contrast distribution of images with different exposure states from the reference image are adjusted to ensure that the lighting performance of the entire image sequence is consistent.

[0076] In this embodiment, the preprocessing flow loads the original image sequence and its corresponding camera calibration file obtained from the multi-lens tilt photography system. This file contains the intrinsic parameter matrix and distortion coefficients of each lens. For each image, a distortion correction algorithm based on the Brown-Conrady model is used. Utilizing the known radial and tangential distortion coefficients, a mapping relationship is established between the pixel coordinates of the original image and the distortion-free ideal coordinates. Inverse transformation and pixel resampling are then performed to generate the corrected image, eliminating image distortion caused by lens optical characteristics. Subsequently, exposure equalization processing is performed by calculating the global brightness histogram, mean, and standard deviation of each image in the entire image sequence. Using statistical methods, the image with moderate exposure, good contrast, and the most balanced histogram distribution was selected as the baseline image, and its mean brightness and variance were used as reference targets. For each other image in the sequence, histogram matching or adaptive gamma correction techniques were used. By establishing a nonlinear mapping function from the current image pixel value to the target pixel value of the baseline image, the brightness and contrast of each pixel were adjusted to make the overall lighting conditions and color distribution of all images consistent with the baseline image. Finally, the processed image sequence was quality checked to ensure that distortion correction was complete and exposure was balanced and effective, providing high-quality input data with geometric accuracy and radiometric consistency for subsequent 3D reconstruction.

[0077] The technical effects of the above solution are as follows: Precise optical distortion correction based on camera calibration parameters can eliminate the influence of lens distortion on the geometric accuracy of images, ensuring that the image point position of each image accurately corresponds to the real spatial coordinates. This guarantees the geometric accuracy and reliability of subsequent motion reconstruction structure and multi-view stereo vision algorithms for feature matching and 3D point cloud computing. At the same time, exposure equalization processing ensures that the lighting performance of the entire image sequence remains consistent, avoiding problems such as image brightness differences, difficulty in feature point matching, and inconsistent texture colors caused by changes in lighting conditions during shooting. This not only improves the integrity and density of 3D point cloud generation, but also ensures seamless, realistic, and visually consistent texture effects when performing texture mapping and fusion on subsequent 3D surface mesh models. Thus, it is possible to generate highly realistic 3D building facade models that can be applied to real-time rendering.

[0078] Based on motion reconstruction structures and multi-view stereo vision algorithms, 3D reconstruction is performed on preprocessed image sequences to generate dense 3D point cloud data containing buildings and surrounding scenes, specifically including:

[0079] The correspondence between images is obtained through feature extraction and matching algorithms. The incremental motion recovery structure method is used to solve the shooting position and attitude parameters of each image and generate a sparse three-dimensional point cloud.

[0080] Based on the pose parameters of the images, a multi-view stereo vision process is executed to generate a corresponding depth map for each image. Then, through depth map fusion and point cloud fusion methods, dense 3D point cloud data containing architectural and scene details is formed.

[0081] In this embodiment, using a preprocessed sequence of tilted images with consistent geometric and radiometric properties, algorithms such as scale-invariant feature transformation or accelerated robust feature extraction are employed to extract highly discriminative local feature points and their descriptors from each image. A feature matching strategy based on nearest neighbor search is used to establish the correspondence between feature points of all image pairs, and a random sampling consensus algorithm is used to eliminate mismatches. An incremental motion recovery structure is adopted, selecting initial image pairs with sufficient common-view features and appropriate baselines. The initial camera pose and sparse 3D points are recovered through epipolar geometry and triangulation. New images are then incrementally added iteratively. The exterior orientation elements of the new images are solved by back intersection using known 3D points and corresponding image feature points. New 3D points are generated by triangulation based on the common-view relationship between the new images and existing points. Bundle adjustment is then used to adjust all camera parameters and 3D point coordinates. A global nonlinear optimization is performed to minimize reprojection errors, ultimately outputting high-precision exterior orientation elements and sparse point clouds covering buildings and scenes for all images. Based on this, a multi-view stereo vision process is executed. For each image, an image set meeting angle and overlap requirements is selected from its neighboring multi-view images based on known poses as a reference. Depth values ​​for each pixel are calculated using patch matching or deep learning methods to generate an initial depth map. Outliers are removed using depth map filtering and consistency checking algorithms, optimizing and generating detailed depth maps for each view. The depth maps of all images are then transformed to a unified world coordinate system through backprojection. A point cloud fusion algorithm integrates depth point clouds from different views and removes redundant points, generating a high-resolution, high-completeness dense 3D point cloud that fully encompasses the geometric details of building facades, roofs, and surrounding scenes.

[0082] The technical effects of the above solution are as follows: By using an incremental motion recovery structure method based on feature matching, high-precision pose parameters of all images are calculated, thereby establishing a stable geometric relationship between images. This provides a foundation for the entire 3D reconstruction process to correctly recover the spatial structure of the scene. Furthermore, the calibrated image poses are used to drive a multi-view stereo vision process. Through multi-view depth estimation, fusion, and optimization, dense 3D point cloud data with details far exceeding those of sparse point clouds is generated. This point cloud completely covers the geometric details and surface undulations of the building facade, roof, and surrounding scene. Its high density and high precision characteristics directly determine the model refinement and geometric realism that subsequent semantic segmentation, surface mesh reconstruction, and texture mapping can achieve.

[0083] The machine learning model employs a classification strategy that combines random forests and support vector machines for automatic segmentation and extraction, specifically including:

[0084] Random forest is used to process the original features of 3D points and output a probability feature vector representing the possibility of each category.

[0085] The probabilistic feature vector is used as input, and a support vector machine is used to complete the final semantic label assignment, which is used to achieve the segmentation of the point cloud of the building facade.

[0086] In this embodiment, feature calculations are performed on the dense point cloud data obtained from 3D reconstruction to extract a set of original feature vectors describing the geometric and spatial distribution characteristics of each 3D point. A random forest model is trained using a point cloud sample dataset with labeled categories (such as building facades, roofs, ground, vegetation, etc.). After training, the entire point cloud is input into the model, and the random forest model outputs a probability distribution vector for each 3D point, containing the possibility of belonging to each preset category. These probability feature vectors generated by the random forest are used as new input features, replacing the original features, and input into a support vector machine model for training. During the support vector machine model training phase, labeled point cloud sample data is also used to learn how to more accurately distinguish different categories based on probability feature vectors. The complete point cloud data to be segmented is input into this cascaded model system, and the trained support vector machine model assigns a final semantic category label to each point in the point cloud, thereby achieving accurate and automatic segmentation and extraction of building facade point clouds from other category point clouds.

[0087] The technical effects of the above solution are as follows: By utilizing random forests to process the rich original features of 3D point clouds and outputting probability distributions for each category, it can effectively integrate multi-dimensional feature information and provide a higher level of discriminative input. Then, the probability feature vector is input into the support vector machine for final decision-making, thereby leveraging the characteristics of support vector machines in clear classification boundaries for small samples in high-dimensional space. The two complement each other and together achieve more accurate and stable semantic differentiation of point clouds of building facades, roofs, ground, and environmental elements. High-precision automatic segmentation effectively avoids manual intervention, ensuring that the subset of point clouds belonging to building facades is completely and cleanly extracted. This lays a clean data foundation for the subsequent generation of geometrically accurate and detailed building facade surface mesh models, and ultimately supports the realization of high-quality 3D modeling and realistic real-time rendering.

[0088] The extracted building facade point cloud is subjected to data optimization and geometric reconstruction, specifically including:

[0089] Noise filtering is applied to the point cloud of the building facade to remove discrete noise points and outlier data.

[0090] Among them, noise filtering is performed on the extracted building facade point cloud. A statistical outlier removal algorithm is used to calculate the average distance from each point to all its K nearest neighbors and analyze its distribution in the entire point cloud. Discrete points whose average distance exceeds the preset standard deviation range are identified as noise points and removed.

[0091] Estimate the local normal vector direction of each point in the point cloud;

[0092] For the denoised building facade point cloud data, a spatial index structure, such as a KD-Tree, is established to efficiently perform neighborhood search. For each point to be calculated in the point cloud, its spatial neighbors are queried through the index. Typically, a fixed-radius spherical domain search or selection of the K nearest neighbors is used to define the local neighborhood of the point. Principal component analysis is performed on all points within this local neighborhood to calculate the three-dimensional coordinate covariance matrix of these points. Eigenvalue decomposition is then performed on the covariance matrix to obtain its eigenvalues ​​and corresponding eigenvectors. The eigenvector corresponding to the minimum eigenvalue represents the direction in which the distribution of the local neighborhood point set is most discrete. This direction is theoretically perpendicular to the local fitting plane, and therefore can be initially determined as the normal vector of the queried point. Furthermore, since principal component analysis can only determine the line where the normal is located but cannot determine its direction (i.e., positive or negative direction), it is necessary to set a globally consistent viewpoint or use the known approximate orientation of the building facade to redirect the direction of all calculated normal vectors to ensure that the normal vectors of all points on the same flat surface point to the same side, thereby completing the estimation of the normal vectors of the entire point cloud.

[0093] Using point cloud and normal vector information, a continuous three-dimensional triangular mesh surface is constructed using a surface reconstruction algorithm to represent the geometric shape of the building facade.

[0094] The technical effects of the above solution are as follows: By filtering noise, discrete noise and abnormal data in the original point cloud can be removed, thereby improving data purity and avoiding geometric distortion in subsequent modeling. By accurately estimating local normal vectors, key local geometric features and directional information can be provided for the surface reconstruction algorithm, thereby ensuring the directional consistency and smoothness of the reconstructed surface. The surface reconstruction algorithm based on point cloud and normal vectors can intelligently connect discrete points and construct a continuous, sealed and topologically correct three-dimensional triangular mesh surface. This mesh model accurately expresses the overall shape and detailed structure of the building facade, thereby ensuring the accuracy and integrity of the final three-dimensional model of the building facade.

[0095] Based on the geometric projection relationship between the oblique photogrammetric image and the 3D surface mesh model, texture information is calculated and assigned to the mesh model, specifically including:

[0096] For each triangular facet of the 3D mesh, based on its 3D coordinates and the external orientation parameters of the image, determine the multiple tilted images that can be projected onto that facet.

[0097] Based on the projection quality evaluation criteria, the most suitable texture source is selected for each patch from the candidate images;

[0098] The selected textures are processed for boundary blending and color consistency to eliminate texture seams and color differences between adjacent patches, achieving a natural transition of the overall texture.

[0099] The technical effects of the above solution are as follows: Based on a rigorous geometric projection relationship, the most suitable texture source is selected for each triangular facet, ensuring the precise correspondence between texture pixels and 3D geometry, thus guaranteeing the accuracy of texture mapping. Furthermore, through the optimized selection of projection quality evaluation criteria, the image texture with the highest clarity, least occlusion, and most correct viewing angle can be assigned to each facet, greatly improving the visual details and realism of the model surface. By performing intelligent boundary fusion and color consistency processing on the selected texture, the texture seams and color differences caused by differences in original image lighting and viewing angle can be eliminated, achieving seamless, smooth, and natural transition of the texture on the entire model surface. This generates a visually highly unified and realistic textured 3D model, thereby providing support for high-quality real-time rendering and immersive interactive browsing on a 3D visualization platform.

[0100] A system for generating 3D building facade models based on UAV oblique photography, used to execute a method for generating 3D building facade models based on UAV oblique photography, including:

[0101] The data acquisition module is configured to generate an automated flight plan with multiple flight paths in layers based on the geometric features, spatial scale, and on-site environmental factors of the target building. It also sets the synchronous shooting mode, lens angle, and overlap rate parameters of the multi-lens tilt photography system to control the UAV to complete the image data acquisition of the building facade and surrounding area.

[0102] The 3D reconstruction module is configured to perform batch preprocessing on the acquired image data, including distortion correction based on camera intrinsic parameters and exposure normalization based on reference images. It also calculates the image pose and generates dense 3D point cloud data containing buildings and scenes through feature matching, structure-of-motion reconstruction and multi-view stereo vision algorithms.

[0103] The point cloud segmentation module has a built-in machine learning model based on a cascaded random forest and support vector machine. It is configured to extract multi-dimensional geometric and optical features from dense 3D point cloud data and achieve automatic identification and semantic segmentation of building facade point clouds and point clouds of roof, ground, vegetation and ground features through the machine learning model.

[0104] The model generation module is configured to remove outliers and filter data from the segmented building facade point cloud, and to construct a continuous triangular mesh surface model that expresses the geometric structure of the building facade based on normal vector estimation and surface reconstruction algorithms.

[0105] The texture mapping module is configured to establish a projection mapping relationship between triangular facets and multi-view images based on the geometric information of the triangular mesh surface model and the exterior orientation elements of the image. It selects the best texture source for each facet through a weight evaluation mechanism and performs fusion processing on the texture boundaries to generate a visually consistent 3D texture model.

[0106] The output interaction module is configured to convert textured 3D models into standard 3D format files and integrate them into a 3D visualization platform, providing dynamic loading of 3D models, display of multiple levels of detail, lighting simulation, and user interaction functions.

[0107] The technical effects of the above solution are as follows: The data acquisition module, through intelligent hierarchical multi-path planning and shooting parameter setting, ensures optimal configuration of the original image data in terms of coverage, overlap rate, and viewing angle, thus providing complete and high-quality input for subsequent processing. The 3D reconstruction module, through preprocessing and advanced multi-view stereo vision algorithms, can achieve high-precision image reconstruction from image to pose calculation to the generation of dense 3D point clouds, thus providing the geometric foundation for the entire 3D model generation. The point cloud segmentation module, with its cascaded machine learning model, can automatically and accurately separate building facade point clouds from complex scene point clouds, thereby improving the automation level of modeling. Based on the target audience, the model generation module optimizes point clouds and reconstructs surfaces to transform discrete point clouds into continuous and accurate triangular mesh surfaces, thus completing the core geometric construction of 3D modeling. The texture mapping module, through intelligent projection selection and fusion processing, endows the geometric model with seamless, realistic, and high-quality textures, thereby enhancing the visual realism of the model. On this basis, the output interaction module, through standardized output and integration with the 3D visualization platform, enables efficient management, high-quality real-time rendering, and flexible interaction of 3D models, thus forming a complete, efficient, and practical 3D modeling solution for building facades from data acquisition to visualization applications.

[0108] Working Principle: By planning the UAV's layered orbital flight path and oblique photography parameters, a multi-view, highly overlapping image sequence of the target building and its surrounding area can be acquired. After distortion correction and exposure equalization preprocessing, the precise pose of the images is calculated using a motion recovery structure algorithm to generate a sparse point cloud. Combined with a multi-view stereo vision algorithm for fusion, a dense 3D point cloud can be generated. On this basis, a machine learning model using a cascaded random forest and support vector machine is employed to perform semantic segmentation on the point cloud, which can automatically and accurately extract a subset of the building facade point cloud. After filtering and optimization of the extracted facade point cloud, a continuous triangular mesh surface model is constructed using a surface reconstruction algorithm. Then, based on the geometric projection relationship, the best texture image is intelligently selected for the mesh patch. After fusion processing, a 3D model with realistic texture is generated. This invention achieves full automation and intelligence from data acquisition to model generation, significantly improving the security, integrity, and efficiency of acquiring data on high-rise and complex building facades. At the same time, it ensures the geometric accuracy and visual realism of the 3D model of the building facade, effectively overcoming the limitations of traditional methods such as high cost, cumbersome operation, and low efficiency.

[0109] A system for generating 3D models of building facades based on UAV oblique photography also includes:

[0110] The model pre-construction module is used to pre-construct the model based on the architectural parameters of the target building, resulting in a pre-constructed model. The pre-constructed model is an initial model built based on the architectural parameters (usually geometric parameters) of the target building, reflecting the approximate shape of the target building. To reduce the amount of pre-construction processing, the initial model can be built by only constructing the building facade.

[0111] The finite element segmentation module performs finite element segmentation on the pre-built model to obtain multiple finite element elements. The finite element segmentation is mainly based on the plane of the building facade. It uses a pre-configured mesh to segment from a direction perpendicular to the plane. The resulting finite element elements have at least one face that is a segment of the building facade.

[0112] The finite element mapping module is used to establish a mapping relationship between finite element elements and the position of the UAV based on the position of the finite element elements in the pre-built model, the flight path, the UAV attitude, and the camera angle; by constructing the mapping relationship, a mapping path is provided for subsequently mapping image elements to the initial model;

[0113] The image segmentation and mapping module is used to segment images acquired by the UAV based on the mapping relationship between finite element elements and the UAV's position, and to determine the image elements of the associated surfaces of the finite element elements.

[0114] The integration module integrates the image elements of the associated surfaces of the finite element elements into the pre-built model to obtain the reference model; the integration module maps the image elements of the associated surfaces of each finite element element determined by the image segmentation and mapping module to the pre-built module to form the reference model.

[0115] The fusion module merges the reference model and the textured 3D model to obtain the final model. It performs a weighted average of data from points at the same locations in both the reference model and the textured 3D model to arrive at the final model. This weighted average uses pre-configured weighting coefficients corresponding to the reference model and the textured 3D model. To further improve the fusion effect, different weighting coefficients are used for different locations of the target building. The specific fusion formula is as follows:

[0116] ;

[0117] In the formula, The spatial coordinates of the final model after fusion are: The data values ​​of the points; The spatial coordinates of the textured 3D model are: The data values ​​of the points; The spatial coordinates of the reference model are: The data values ​​of the points; The spatial coordinates corresponding to the textured 3D model are The weighting coefficients of the points; The spatial coordinates corresponding to the reference model are The weighting coefficients of the points; express The local curvature of the point; express Whether the point is located on the edge or outline; if yes, take 0, otherwise take 1. The confidence level of texture reconstruction is represented by the analysis of the texture reconstruction process of the textured 3D model. , , These are pre-configured weighting factors corresponding to local curvature, whether it is located at an edge or contour line, and texture reconstruction confidence. These factors are analyzed and configured by professionals. For example, they can be configured to 0.3, 0.3, and 0.4.

[0118] This embodiment combines the generation of a comprehensive 3D model of the building facade with the surface image mapping processing of a pre-built model, taking into account both the accuracy of the building facade construction and the visual effect.

[0119] The finite element mapping module performs the following operations:

[0120] Based on the flight path, drone attitude, and camera angle, the regions of the pre-built model corresponding to the images captured by the drone at each point on the flight path are determined. Specifically, this can be achieved through simulation in the model space. The pre-built model is placed in the simulation space, and then drone shooting simulation is performed based on the flight path, drone attitude, and camera angle. During the simulation, the points on the flight path corresponding to the points on the pre-built model are determined by constructing rays in the shooting direction triggered by the drone's shooting device. Then, the region corresponding to the image is determined with that point as the center.

[0121] When a region contains finite element elements, the corresponding points are stored in the pre-screening group; that is, it is determined whether the region corresponding to the image contains finite element elements that need to be mapped.

[0122] Based on the position of the finite element element in the image, an evaluation is performed on the points in the pre-screened group to obtain an evaluation value. The evaluation value is the highest when the finite element element is located at the center of the image, and the score is the lowest when it is located at the four edge corners. The specific evaluation method is to determine the horizontal and vertical distances from the center of the image, query the pre-configured weight coefficient table to determine the weight coefficients, and multiply them by the pre-configured reference evaluation value to obtain an evaluation value.

[0123] Based on the drone's position, attitude, and camera angle corresponding to the image, and the surface of the pre-built model corresponding to the image, a secondary evaluation is performed on the points in the pre-selected group to obtain secondary evaluation values. The shooting direction is determined by the drone's position, attitude, and camera angle corresponding to the image. The secondary evaluation is performed using the minimum angle between the direction vector representing the shooting direction and the plane containing the surface, and the distance between the drone's position and the surface pointed to by the shooting direction. The formula for calculating the secondary evaluation value is as follows:

[0124] ;

[0125] In the formula, This indicates a secondary evaluation value; This represents a dimensionless constant corresponding to the smallest included angle; This represents the quantized value obtained by querying a pre-configured quantization table corresponding to the dimensionless constant (in SI units) of the distance. , Pre-configured weighting coefficients (set by professionals through analysis of the importance of the two assessments); for example: , It can be configured to 5 or 500 respectively. In the quantization table, X is 1 when the distance is 2000mm, and X decreases as the distance increases or decreases.

[0126] By combining the primary and secondary evaluation values, the points mapped to the finite element units (FEMs) are determined, and a mapping relationship between the FEMs and the UAV positions is established at these determined points. Points where the sum of the primary and secondary evaluation values ​​is greater than or equal to a preset threshold are mapped to the FEMs. The primary evaluation focuses on the image capture effect, while the secondary evaluation focuses on the relative positions of the FEMs and the UAV in the shooting scene. Combining the two evaluations allows for the indirect reference to determine the image unit on the associated surface of the FEM that most closely resembles the actual situation, ensuring overall visual quality.

[0127] During the integration module integration, the sum of the primary and secondary evaluation values ​​is first used as a reference parameter. The image element corresponding to the finite element with the largest reference parameter is placed close to the finite element's associated surface and designated as the reference element. Then, radiating outwards from this element, the integration effect between the reference parameter, the reference element, and the image element is evaluated to determine the corresponding image element to fill the associated surface of the finite element. The evaluation can be performed using a quantized weighted method. A pre-configured quantization library is used to quantize the reference parameter, the reference element, and the image element's associated effect, obtaining quantized values. These values ​​are then weighted and calculated to obtain the evaluation parameters. The image element with the largest evaluation parameter is selected as the image element to be integrated from the surrounding finite element elements.

[0128] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0129] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

Claims

1. A method for generating a 3D model of a building facade based on UAV oblique photography, characterized in that, Includes the following steps: For the target building, based on the building's geometry and surrounding environmental conditions, plan the multi-route flight path of the UAV and set the shooting parameters of the multi-lens oblique photography system. Control the UAV equipped with the oblique photography system to fly along the preset route and simultaneously collect multi-view oblique image sequences covering the building's facades, roof and adjacent areas. The acquired oblique photogrammetric image sequence is preprocessed, and image distortion correction and exposure consistency adjustment are performed sequentially. Based on the structure of motion restoration and multi-view stereo vision algorithm, the preprocessed image sequence is reconstructed in three dimensions to generate dense three-dimensional point cloud data containing buildings and surrounding scenes. Based on dense 3D point cloud data, a pre-trained machine learning model is used to perform semantic segmentation on the dense 3D point cloud data, automatically distinguish and extract the point cloud subset belonging to the building facade, and separate the point clouds of the roof, ground and environmental elements. The extracted point cloud of the building facade is optimized and geometrically reconstructed. A two-dimensional surface mesh model expressing the geometric structure of the building facade is generated through point cloud filtering and surface reconstruction algorithms. Based on the geometric projection relationship between the oblique photogrammetric image and the three-dimensional surface mesh model, texture information is calculated and assigned to the mesh model. Through texture selection and fusion processing, a three-dimensional model of the building facade with a realistic surface appearance is formed. The textured 3D building facade model is converted into a universal 3D file format and integrated into a 3D visualization platform to enable real-time rendering and interactive browsing of the 3D building facade model.

2. The method for generating a 3D model of a building facade based on UAV oblique photography according to claim 1, characterized in that, Planning multi-flight paths for drones and setting shooting parameters for a multi-lens tilt photography system, specifically including: Based on the three-dimensional outline and facade height of the target building, multiple layered loop routes are designed, including a horizontal loop route along the middle height of the building facade, and a loop route along the top area of ​​the building while maintaining an inclined viewing angle. By adjusting the drone's attitude and lens angle, the main optical axis of the tilt camera is made to form an angle within a preset range with the building facade surface; Set the synchronous shooting mode and time interval of the multi-lens tilt photography system to ensure that the images captured continuously in the flight path have a high overlap rate and that the preset overlap coverage requirements are met between the side images of adjacent flight paths.

3. The method for generating a 3D model of a building facade based on UAV oblique photography according to claim 1, characterized in that, The acquired oblique photogrammetry image sequence is preprocessed, specifically including: By utilizing the camera calibration parameters of the oblique photography system, optical distortion correction is performed on each image to eliminate radial and tangential distortion caused by the lens shape; Image enhancement technology is used to perform exposure equalization processing on all images in the sequence, and the image with the best exposure is used as the reference image. The brightness and contrast distribution of images with different exposure states from the reference image are then adjusted.

4. The method for generating a 3D model of a building facade based on UAV oblique photography according to claim 1, characterized in that, Based on motion reconstruction structures and multi-view stereo vision algorithms, 3D reconstruction is performed on preprocessed image sequences to generate dense 3D point cloud data containing buildings and surrounding scenes, specifically including: The correspondence between images is obtained through feature extraction and matching algorithms. The incremental motion recovery structure method is used to solve the shooting position and attitude parameters of each image and generate sparse three-dimensional point clouds. Based on the pose parameters of the images, a multi-view stereo vision process is executed to generate a corresponding depth map for each image. Then, through depth map fusion and point cloud fusion methods, dense 3D point cloud data containing architectural and scene details is formed.

5. The method for generating a 3D model of a building facade based on UAV oblique photography according to claim 1, characterized in that, The machine learning model employs a classification strategy that combines random forests and support vector machines for automatic segmentation and extraction, specifically including: Random forest is used to process the original features of 3D points and output a probability feature vector representing the possibility of each category. The probabilistic feature vector is used as input, and a support vector machine is used to complete the final semantic label assignment, which is used to achieve the segmentation of the point cloud of the building facade.

6. The method for generating a 3D model of a building facade based on UAV oblique photography according to claim 1, characterized in that, The extracted building facade point cloud is subjected to data optimization and geometric reconstruction, specifically including: Noise filtering is applied to the point cloud of the building facade to remove discrete noise points and outlier data. Estimate the local normal vector direction of each point in the point cloud; Using point cloud and normal vector information, a continuous three-dimensional triangular mesh surface is constructed using a surface reconstruction algorithm to represent the geometric shape of the building facade.

7. The method for generating a 3D model of a building facade based on UAV oblique photography according to claim 1, characterized in that, Based on the geometric projection relationship between the oblique photogrammetric image and the 3D surface mesh model, texture information is calculated and assigned to the mesh model, specifically including: For each triangular facet of the 3D mesh, based on its 3D coordinates and the external orientation parameters of the image, determine the multiple tilted images that can be projected onto that facet. Based on the projection quality evaluation criteria, the most suitable texture source is selected for each patch from the candidate images; The selected textures are processed for boundary blending and color consistency to eliminate texture seams and color differences between adjacent patches, achieving a natural transition of the overall texture.

8. A system for generating a 3D model of a building facade based on UAV oblique photography, used to execute the method for generating a 3D model of a building facade based on UAV oblique photography as described in any one of claims 1-7, characterized in that, include: The data acquisition module is configured to generate an automated flight plan with multiple flight paths in layers based on the geometric features, spatial scale, and on-site environmental factors of the target building. It also sets the synchronous shooting mode, lens angle, and overlap rate parameters of the multi-lens tilt photography system to control the UAV to complete the image data acquisition of the building facade and surrounding area. The 3D reconstruction module is configured to perform batch preprocessing on the acquired image data, including distortion correction based on camera intrinsic parameters and exposure normalization based on reference images. It also calculates the image pose and generates dense 3D point cloud data containing buildings and scenes through feature matching, structure-of-motion reconstruction and multi-view stereo vision algorithms. The point cloud segmentation module has a built-in machine learning model based on a cascaded random forest and support vector machine. It is configured to extract multi-dimensional geometric and optical features from dense 3D point cloud data and achieve automatic identification and semantic segmentation of building facade point clouds and point clouds of roof, ground, vegetation and ground features through the machine learning model. The model generation module is configured to remove outliers and filter data from the segmented building facade point cloud, and to construct a continuous triangular mesh surface model that expresses the geometric structure of the building facade based on normal vector estimation and surface reconstruction algorithms. The texture mapping module is configured to establish a projection mapping relationship between triangular facets and multi-view images based on the geometric information of the triangular mesh surface model and the exterior orientation elements of the image. It selects the best texture source for each facet through a weight evaluation mechanism and performs fusion processing on the texture boundaries to generate a three-dimensional texture model. The output interaction module is configured to convert textured 3D models into standard 3D format files and integrate them into a 3D visualization platform, providing dynamic loading of 3D models, display of multiple levels of detail, lighting simulation, and user interaction functions.

9. The system for generating 3D building facade models based on UAV oblique photography according to claim 8, characterized in that, Also includes: The model pre-construction module is used to pre-construct the model based on the architectural parameters of the target building, and obtain a pre-constructed model; The finite element segmentation module performs finite element segmentation on the pre-built model to obtain multiple finite element elements; The finite element mapping module is used to establish a mapping relationship between finite element elements and the position of the UAV based on the position of the finite element elements in the pre-built model, flight path, UAV attitude and camera angle; The image segmentation and mapping module is used to segment images acquired by the UAV based on the mapping relationship between finite element elements and the UAV's position, and to determine the image elements of the associated surfaces of the finite element elements. The integration module integrates the image elements of the associated surfaces of the finite element elements into the pre-built model to obtain the reference model; The fusion module merges the reference model and the textured 3D model to obtain the final model.

10. The system for generating 3D building facade models based on UAV oblique photography according to claim 9, characterized in that, The finite element mapping module performs the following operations: Based on the flight path, drone attitude, and camera angle, the regions of the pre-built model are determined to correspond to the images collected by the drone at each point along the flight path. When the region contains finite element elements, the corresponding points are stored in the pre-screening group; Based on the position of the finite element element in the image, an evaluation value is obtained by evaluating the points in the pre-screened group. Based on the drone's position, attitude, and camera angle corresponding to the image, and the surface of the pre-built model corresponding to the image, a secondary evaluation is performed on the points in the pre-screened group to obtain secondary evaluation values. By combining the primary and secondary evaluation values, the points mapped to the finite element elements are determined, and the mapping relationship between the finite element elements and the UAV position is established at the determined points.

Images