A building facade image extraction method based on unmanned aerial vehicle multi-camera tilt photography

Abstract

This invention provides a method for extracting building facade images based on multi-camera oblique photography from unmanned aerial vehicles (UAVs), comprising the following steps: S1 extracting image feature points from UAV aerial data; S2 performing 3D reconstruction of the image feature points to obtain the spatial geographic coordinates of each feature point; S3 matching the pixel coordinates of the feature points with their spatial geographic coordinates; S4 classifying feature points belonging to the same building spatially based on a building cadastral map; S5 performing outlier analysis among similar feature points and removing error points that may occur during 3D modeling calculations; S6 identifying the minimum bounding rectangle for feature points of different classifications and cropping the aerial image using this rectangle as the boundary to obtain building facade images corresponding to different building numbers. This invention achieves batch extraction of building facade texture information from UAV photographic images in a fully automated manner, providing a new and feasible path for subsequent building information extraction, 3D building modeling, and other related applications.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing measurement technology, and in particular relates to a method for extracting building facade images based on multi-camera oblique photography from unmanned aerial vehicles. Background Technology

[0002] Texture images of building facades play a crucial role in practical applications such as 3D city modeling, urban search and rescue, disaster loss assessment, and 3D navigation system design. Multi-camera oblique photography technology using unmanned aerial vehicles (UAVs) provides an efficient and convenient method for acquiring building facade information and rapidly creating 3D models. Traditional methods of manually extracting building facades from 3D models are clearly insufficient to meet the demand for extracting large amounts of building facades obtained from aerial photography over a wide area. Currently, the common automatic extraction method involves first creating a 3D model from the acquired oblique aerial photographs and then using spatial algorithms to extract the building facades composed of 3D point clouds. For example, Chinese invention patent specification CN115713592A discloses a method for generating orthophotos of building facades. This method utilizes a UAV to first perform close-up photography of the building facade, then uses aerial triangulation to iteratively generate 3D point cloud data, and finally uses this point cloud model to generate an orthophoto of the building facade.

[0003] However, when generating 3D building point cloud data from UAV photographic images, errors in the 3D modeling process can lead to problems such as blurred, missing, or distorted building facade textures. Furthermore, occlusion between buildings and natural vegetation makes close-up UAV flight photography difficult, and there is a lack of simple, practical, and widely applicable data acquisition methods. In addition, conventional building facade extraction methods based on UAV multi-camera oblique photography cannot fully utilize aerial images taken from different angles and locations of the same building, which limits the application of UAV multi-camera oblique photography. Summary of the Invention

[0004] The purpose of this invention is to provide an automatic identification and extraction method for building facade images based on multi-camera oblique photography from unmanned aerial vehicles (UAVs). This method utilizes a UAV equipped with an oblique camera to collect information about a study area, and combines this with cadastral map data of buildings within the area to automatically extract aerial images of building facades. This method significantly simplifies the time and labor costs required to acquire and create facade maps, and achieves batch extraction of building facade texture information from UAV images in a fully automated manner, providing a new and feasible path for subsequent building information extraction, 3D building modeling, and other related applications.

[0005] The technical solution adopted in this invention is:

[0006] A method for extracting building facade images based on multi-camera oblique photography from unmanned aerial vehicles (UAVs), characterized by the following steps:

[0007] Step S1: The drone carrying the tilt camera acquires aerial images of the study area and POS data of the flight, and uses aerial image processing tools to extract image feature points from these data based on SIFT, SURF or ORB algorithms;

[0008] Step S2: Use the SFM (Structure from Motion) algorithm to perform 3D reconstruction of the feature points in the image to obtain the spatial geographical coordinates of each feature point;

[0009] Step S3: Analyze the obtained feature point coordinate file, normalize the pixel coordinates of the feature points with the spatial geographic coordinates, establish a pixel coordinate system for the feature points, and match them one by one according to the same number of the feature points;

[0010] Step S4: Use the building cadastral map as a reference map and feature points to perform spatial topological relationship analysis, thereby classifying feature points that belong to the same building in space;

[0011] Step S5: Calculate the Euclidean distance between similar feature points in any image based on pixel coordinates, and use this as a variable to perform outlier analysis to eliminate error points that may occur during the 3D modeling calculation process;

[0012] Step S6: For feature points of different categories, identify the minimum bounding rectangle and use it as the boundary to crop the aerial image to obtain the building facade image corresponding to different building numbers.

[0013] Furthermore, the aerial image processing tool mentioned in step S1 can be a commercial or open-source aerial image processing tool, such as OpenDroneMap, UAVmapper, or other open-source tools. Such tools can not only extract feature points from UAV images and perform 3D reconstruction, but also output the results of intermediate processing steps in the output file, thereby obtaining the image feature point numbers, pixel coordinates, and the geographic coordinates after 3D reconstruction.

[0014] Further, in step S1, based on the texture content and coverage of the image, a minimum number of feature points to be extracted is set, and a feature point extraction algorithm and a minimum limit threshold for feature points are determined. The image feature point extraction includes the feature point number contained in each image and the position coordinate information after scaling according to the image pixel specification size.

[0015] Furthermore, in step S2, the feature points and aerial data extracted in step S1 are used to perform calculations through a preset feature point extraction algorithm or a three-dimensional reconstruction algorithm, or the feature point number and corresponding geospatial coordinates are obtained directly by using an aerial image processing tool and setting a specific algorithm and output type. The preset algorithm includes the SFM algorithm.

[0016] Furthermore, step S3 includes the following steps:

[0017] Step S301: Extract the relevant parameters from the feature point pixel coordinate file obtained in step S2, including: the name of the photo taken by the drone, the number of the feature point, the horizontal and vertical pixel length of each image, and the coordinates (x,y) of the feature point in the image.

[0018] Step S302: Transform the coordinates (x, y) of the feature points to obtain the key fields, namely the image name ID, feature point ID, and pixel coordinates x and y. pixel y pixel The conversion formula for the data in Table 1 is as follows:

[0019] S = nmax(w, h)

[0020]

[0021]

[0022] Where w and h are the horizontal and vertical pixel lengths of the image, respectively;

[0023] Step S303: Calculate the spatial geographic coordinates (x, y) of the feature points. geo ,y geo This allows us to obtain data containing key fields, namely feature point numbers and geospatial coordinates x and y. geo y geo The calculation formula is as follows: (Data table 2)

[0024] x geo =x core +x off

[0025] y geo =y core +y off (2)

[0026] Where (x) core ,y core ) represents the reconstruction center, x off and y off These represent the offset distances from the reconstruction center in the x and y directions, respectively. The spatial geographic coordinates of the feature point include the UTM coordinate system or the unprojected latitude and longitude coordinate system.

[0027] Step S304: Match the pixel coordinates and spatial geographic coordinates of the feature points one by one according to the same number of the feature points to establish the pixel coordinate system of the feature points.

[0028] Further, step S4 includes the following steps:

[0029] Step S401: Convert the feature points into a vector point layer containing geographic coordinate information;

[0030] Step S402: Perform topological relationship analysis between the point layer and the vector surface layer of the cadastral map using GIS analysis to construct a feature point layer;

[0031] Step S403: Use the building outline polygons contained in the building cadastral layer to perform spatial filtering on the constructed feature point layer, thereby quickly eliminating feature points in non-building areas.

[0032] Step S404: Classify the remaining feature points into different buildings through overlay analysis, obtaining a result containing building number, feature point number, and feature point position x. geo y geo Data table 3 with fields.

[0033] Further, in step S5, the method for calculating the Euclidean distance based on pixel coordinates between similar feature points in any image includes the following steps:

[0034] Step S501: Using the feature point number as the key field, merge data table 1, data table 2 and data table 3 to obtain data table 4 containing key fields such as building number, image name, feature point number, feature point pixel coordinates, and feature point spatial coordinates;

[0035] Step S502: Determine the different building numbers contained in the image and the feature points classified by building number based on the image name;

[0036] Step S503: Calculate the Euclidean distance based on pixel coordinates for a certain type of building feature points contained in the image.

[0037] Furthermore, in step S5, outlier analysis and error point removal for similar feature points include the following steps:

[0038] Step S504: Calculate the distance between any point and its four nearest neighbors using the nearest neighbor algorithm in machine learning.

[0039] Step S505: Standardize the distance array to obtain the variance, and determine whether the point belongs to the outlier based on the variance. The variance threshold is adjusted according to different situations to ensure that similar data points can be retained as much as possible.

[0040] Step S506: Remove outliers that deviate significantly to obtain a data table 5 that is relatively complete in terms of the distribution of feature points according to the building classification number. The fields contained in this data table are the same as those in data table 4.

[0041] Further, step S6 includes the following steps:

[0042] Step S601: Open all images by traversing the list of image names, and find the data sub-fragment 1 corresponding to the image name from data table 5;

[0043] Step S602: Traverse the building numbers in data sub-fragment 1 to obtain data sub-fragment 2 corresponding to different buildings;

[0044] Step S603: Create a folder named after the traversed building number, identify the minimum bounding rectangle of the feature points contained in the sub-segment 2 corresponding to the building number, and use image processing techniques to crop from the corresponding image using this boundary.

[0045] Step S604: Save the cropped sub-images to a folder named after the building number, and obtain multiple building elevation images taken from different angles and positions for each building, and store them in the corresponding folders according to the building number.

[0046] The beneficial effects of this invention are as follows:

[0047] 1. This invention fully utilizes the advantages of UAV low-altitude remote sensing, such as ease of operation, wide coverage, and clear image texture, to automatically process image data from multi-camera oblique photography to extract building facades, providing a new approach for acquiring regional building facade information. This greatly simplifies the work steps for collecting building facade texture information and improves upon the defects of existing methods for extracting building facades from 3D building models, such as texture blurring, missing textures, and distortion.

[0048] 2. The multi-camera oblique photography data from UAVs targeted in this invention generally employs an automatic cruise mode for data acquisition, eliminating the need for unique shooting for specific buildings. This greatly simplifies the workload required for collecting building facade texture information over a wide area and fully utilizes images of the same building from different angles and positions by the UAV, reducing the impact of occlusion around the building on the extraction results. Attached Figure Description

[0049] Figure 1 This is a technical roadmap for the present invention.

[0050] Figure 2 This is a schematic diagram showing the distribution of image feature points in the image.

[0051] Figure 3 A schematic diagram showing the distribution of feature points in an image after geospatial topological analysis.

[0052] Figure 4 This is the final architectural elevation drawing obtained by this invention. Detailed Implementation

[0053] The specific embodiments of the present invention are described in detail below with reference to the technical solutions and accompanying drawings.

[0054] like Figure 1 As shown, the present invention provides a method for extracting building facade images based on multi-camera oblique photography from a UAV, comprising the following steps:

[0055] Step S1: The UAV carrying a tilt camera acquires aerial images of the study area and POS data of the flight, and uses aerial image processing tools to extract image feature points from these data based on SIFT, SURF or ORB algorithms.

[0056] The aforementioned aerial image processing tool refers to an image 3D reconstruction tool capable of extracting image feature points and providing the position of each feature point in the image's pixel coordinates. This example uses the open-source aerial image processing tool OpenDroneMap (ODM), employing the SIFT (Scale Invariant Feature Transform) algorithm to extract feature points. A minimum number of feature points to extract is set based on the image's texture content and coverage area, typically between 8000 and 12000. By importing aerial images and POS data into ODM and setting the feature point extraction algorithm and minimum threshold, the feature point numbers and their scaled-down coordinates according to the image's pixel dimensions can be obtained for each image.

[0057] Step S2: Use the SFM (Structure from Motion) algorithm to perform 3D reconstruction of the feature points in the image to obtain the spatial geographical coordinates of each feature point;

[0058] The aforementioned 3D reconstruction of image feature points can be calculated using the feature points extracted in step S1 and aerial data through the SFM algorithm or other 3D reconstruction algorithms. Alternatively, it can be obtained directly using aerial image processing tools by setting specific algorithms (feature point extraction algorithm, 3D reconstruction algorithm) and output types (feature point pixel coordinate file, 3D reconstruction geographic coordinate file) to obtain feature point numbers and corresponding geographic spatial coordinates.

[0059] Step S3: Analyze the obtained feature point coordinate file, establish a pixel coordinate system for the feature points, and perform normalization processing on the pixel coordinates of the feature points and spatial geographic coordinates, matching them one by one according to the same number of the feature points.

[0060] The purpose of establishing the pixel coordinate system for feature points is to standardize the pixel coordinate file of feature points obtained in step S2 to facilitate subsequent cropping of different building facade images. The coordinate file obtained in step S2 may be a coordinate system based on the image center as the origin or coordinates normalized to the image size. Therefore, analyzing the obtained coordinate file to establish a standardized and unified coordinate system is a necessary step. Taking the ODM-processed file as an example, its output pixel coordinate file path is: . / opensfm / tracks.csv, and its content includes: shot_id, track_id, feature_id, x, y, s, R, G, B, SEG, INS. The parameters relevant to this method include: "shot_id" is the name of the photo taken by the drone, "track_id" is the feature point number, and "x, y" are the horizontal (width, w) and vertical (height, h) coordinates of the feature point in the image, respectively, and the values ​​are scaled after image size adjustment. The final pixel coordinates of the feature point (x...y...) are... pixel ,y pixel The image name, feature point number, and pixel coordinates (x) can be obtained through formula (1). This yields the image name, feature point number, and pixel coordinates (x). pixel y pixel Table 1 shows the data for key fields. The distribution of feature points in the image is as follows: Figure 2 As shown.

[0061]

[0062] The spatial geographic coordinates of the feature points refer to the coordinate files of the feature points obtained through 3D reconstruction in step S2, which are then analyzed to form a standardized UTM coordinate system or an unprojected latitude and longitude coordinate system. The purpose is to ensure consistency with the coordinate system of the subsequent building cadastral map, thereby enabling spatial topological relationship analysis between the feature points and the building outline vector surface. Taking the ODM-processed file as an example, the output file path for the 3D reconstructed feature points is: . / opensfm / reconstruction.json. This file contains the following key fields: "cameras" contains camera parameters, "shots" contains the POS data of the drone when each image was taken, and "points" contains the feature point number "track_id" and the color and spatial position (x, y, x) of the reconstructed point. off ,y off ,elev). Among them, the spatial location information x off ,y off These represent the offset distances from the reconstruction center in the x and y directions, respectively. The reconstruction center location can be found in the second line of the ODM output file . / odm_georeferencing / cords.txt (x core ,ycore The final spatial geographic coordinates (x) of the feature point geo ,y geo The feature point number and its geographic spatial coordinates (x) can be calculated using formula (2). geo y geo Table 2 contains key fields such as those in the data table.

[0063]

[0064] Step S4: Use the building cadastral map as a reference map and feature points to perform spatial topological relationship analysis, thereby classifying feature points that belong to the same building in space;

[0065] The spatial topological relationship analysis between the cadastral map and feature points first requires converting the feature points into a vector point layer containing geographic coordinate information. Then, GIS analysis is used to perform topological relationship analysis between the point layer and the vector polygon layer of the cadastral map. This invention first uses Python's osgeo library to automatically create a vector layer with the same spatial attributes but no elements based on the spatial reference information contained in the cadastral map (*.shp) file. Then, the feature point location data (x... geo ,y geo The feature point layer will be automatically imported.

[0066] After constructing the feature point layer, spatial filtering is first performed using the building outline polygons contained in the cadastral layer. This quickly eliminates feature points from non-building areas. The remaining feature points are then classified into different buildings through overlay analysis to obtain building numbers, feature point numbers, and feature point positions (x-coordinates). geo y geo Table 3 contains data with these five fields. The categorized feature points are shown in the image as follows: Figure 3 As shown.

[0067] Step S5: Calculate the Euclidean distance between similar feature points in any image based on pixel coordinates, and use this as a variable to perform outlier analysis to remove error points that may occur during the 3D modeling calculation process.

[0068] The calculation of the Euclidean distance based on pixel coordinates between similar feature points in any image refers to the following steps: First, using the feature point number as the key field, data tables 1, 2, and 3 are merged to obtain data table 4, which contains key fields such as building number, image name, feature point number, feature point pixel coordinates, and feature point spatial coordinates. Then, based on the image name, the different building numbers and feature points classified by building number are determined. Finally, the Euclidean distance based on pixel coordinates is calculated for a certain type of building feature points in the image.

[0069] Outlier analysis of similar feature points involves: first, using the nearest neighbor algorithm in machine learning to calculate the distance between any point and its four nearest neighbors; then, standardizing this distance array to obtain the variance; and finally, determining whether a point is an outlier based on the magnitude of the variance. This variance threshold can be adjusted according to different situations, with the principle being to retain similar data points as much as possible while removing other obviously deviating points. This results in Table 5, a relatively complete data table showing the distribution of feature points according to building classification numbers. The fields in this table are consistent with those in Table 4.

[0070] Step S6: For feature points of different categories, identify the minimum bounding rectangle and use it as the boundary to crop the aerial image to obtain the building facade image corresponding to different building numbers.

[0071] Open all images by traversing the image name list and find the corresponding data segment 1 from data table 5. Then, traverse the building numbers in data segment 1 to obtain data segment 2 corresponding to different buildings. Create a folder named after the traversed building number, perform minimum bounding rectangle identification on the feature points contained in the sub-segment 2 corresponding to that building number, and use image processing techniques to crop the corresponding image using this boundary. Save the cropped sub-images to the folder named after the building number. By traversing all images according to the above path, multiple building elevation images taken from different angles and positions within the area can be obtained, and stored in corresponding folders according to the building number. The cropped building elevation images are shown below. Figure 4 As shown.

[0072] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting building facade images based on multi-camera oblique photography from unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Step S1: The drone carrying the tilt camera acquires aerial images of the study area and POS data of the flight, and uses aerial image processing tools to extract image feature points from these data based on SIFT, SURF or ORB algorithms; Step S2: Use the SFM (Structure from Motion) algorithm to perform 3D reconstruction of the feature points in the image to obtain the spatial geographic coordinates of each feature point; Step S3: Analyze the obtained feature point coordinate file, normalize the pixel coordinates of the feature points with the spatial geographic coordinates, establish a pixel coordinate system for the feature points, and match them one by one according to the same number of the feature points; Step S4: Use the building cadastral map as a reference map and feature points to perform spatial topological relationship analysis, thereby classifying feature points that belong to the same building in space; Step S5: Calculate the Euclidean distance between similar feature points in any image based on pixel coordinates, and use this as a variable to perform outlier analysis to eliminate error points that may occur during the 3D modeling calculation process; Step S6: For feature points of different categories, identify the minimum bounding rectangle and use it as the boundary to crop the aerial image to obtain the building facade image corresponding to different building numbers. Step S3 includes the following steps: Step S301: Extract the relevant parameters from the feature point pixel coordinate file obtained in step S2, including: the name of the photo taken by the drone, the number of the feature point, the horizontal and vertical pixel length of each image, and the coordinates (x, y) of the feature point in the image. Step S302: Transform the coordinates (x, y) of the feature points to obtain a data table 1 containing key fields, i.e. picture name number, feature point number, and pixel coordinates x pixel , y pixel , the conversion formula is as follows: （1） Where w and h are the horizontal and vertical pixel lengths of the image, respectively; Step S303: Calculate the spatial geographic coordinates (x, y, z) of the feature points. geo , y geo This allows us to obtain information containing key fields, namely feature point numbers and geospatial coordinates x. geo y geo The calculation formula is as follows: (Data table 2) （2） Where (x) core , y core ) represents the reconstruction center, x off and y off These represent the offset distances from the reconstruction center in the x and y directions, respectively. The spatial geographic coordinates of the feature point include the UTM coordinate system or the unprojected latitude and longitude coordinate system. Step S304: Match the pixel coordinates and spatial geographic coordinates of the feature points one by one according to the same number of the feature points to establish the pixel coordinate system of the feature points; Step S4 includes the following steps: Step S401: Convert the feature points into a vector point layer containing geographic coordinate information; Step S402: Perform topological relationship analysis between the point layer and the vector surface layer of the cadastral map using GIS analysis to construct a feature point layer; Step S403: Use the building outline polygons contained in the building cadastral layer to perform spatial filtering on the constructed feature point layer, thereby quickly eliminating feature points in non-building areas. Step S404: Classify the remaining feature points into different buildings through overlay analysis, obtaining a result containing building number, feature point number, and feature point position x. geo y geo Data table 3 with fields.

2. The method for extracting building facade images based on UAV multi-camera oblique photography according to claim 1, characterized in that, The aerial image processing tools mentioned in step S1 include commercial or open-source aerial image processing tools, which are used to extract feature points from UAV images and perform 3D reconstruction, and obtain the results of the intermediate processing in the output file, thereby obtaining the image feature point number, pixel coordinates and 3D reconstructed geographic coordinate information.

3. The method for extracting building facade images based on UAV multi-camera oblique photography according to claim 1, characterized in that, In step S1, based on the texture content and coverage of the image, the minimum number of feature points to be extracted is set, and the feature point extraction algorithm and the minimum limit threshold for feature points are determined. The image feature point extraction includes the feature point number contained in each image and the position coordinate information after scaling according to the image pixel specification size.

4. The method for extracting building facade images based on UAV multi-camera oblique photography according to claim 1, characterized in that, In step S2, the feature points and aerial data extracted in step S1 are used to perform calculations through a preset feature point extraction algorithm or a 3D reconstruction algorithm, or the aerial image processing tool is used directly to obtain the feature point number and corresponding geospatial coordinates by setting a specific algorithm and output type. The preset algorithm includes the SFM algorithm.

5. The method for extracting building facade images based on UAV multi-camera oblique photography according to claim 1, characterized in that, In step S5, the method for calculating the Euclidean distance based on pixel coordinates between similar feature points in any image includes the following steps: Step S501: Using the feature point number as the key field, merge data table 1, data table 2 and data table 3 to obtain data table 4 containing key fields such as building number, image name, feature point number, feature point pixel coordinates, and feature point spatial coordinates; Step S502: Determine the different building numbers contained in the image and the feature points classified by building number based on the image name; Step S503: Calculate the Euclidean distance based on pixel coordinates for a certain type of building feature points contained in the image.

6. The method for extracting building facade images based on UAV multi-camera oblique photography according to claim 5, characterized in that, In step S5, outlier analysis and error point removal for similar feature points include the following steps: Step S504: Calculate the distance between any point and its four nearest neighbors using the nearest neighbor algorithm in machine learning, thereby forming a distance array; Step S505: Standardize the distance array to obtain the variance, and determine whether the point belongs to the outlier based on the variance. The variance threshold is adjusted according to different situations to ensure that similar data points can be retained as much as possible. Step S506: Remove outliers that deviate significantly to obtain a data table 5 that is relatively complete in terms of the distribution of feature points according to the building classification number. The fields contained in this data table are the same as those in data table 4.

7. The method for extracting building facade images based on UAV multi-camera oblique photography according to claim 6, characterized in that, Step S6 includes the following steps: Step S601: Open all images by traversing the list of image names, and find the data sub-fragment 1 corresponding to the image name from data table 5; Step S602: Traverse the building numbers in data sub-fragment 1 to obtain data sub-fragment 2 corresponding to different buildings; Step S603: Create a folder named after the traversed building number, identify the minimum bounding rectangle of the feature points contained in the sub-segment 2 corresponding to the building number, and use image processing techniques to crop from the corresponding image using this boundary. Step S604: Save the cropped sub-images to a folder named after the building number, and obtain multiple building elevation images taken from different angles and positions for each building, and store them in the corresponding folders according to the building number.

Images