Building facade distortionless panoramic image seamless splicing system and method based on unmanned aerial vehicle

Abstract

The invention discloses a building facade distortionless panoramic image seamless splicing system and method based on an unmanned aerial vehicle, and the method comprises the steps: obtaining a preliminary image of a building surface through the unmanned aerial vehicle, and carrying out the modeling according to the preliminary image, and obtaining a rough three-dimensional model of the building; based on the rough three-dimensional model of the building, planning a flight path of the unmanned aerial vehicle approaching flight, collecting a fine image of the surface of the building again, and performing modeling according to the fine image to obtain a fine three-dimensional model of the building; obtaining building point cloud data based on the building fine three-dimensional model, and performing re-projection on each vertical face of the building according to the building point cloud data; and splicing the re-projection images of different facades of the building to form a complete building facade panoramic image. According to the method, the high maneuverability of the unmanned aerial vehicle and the three-dimensional modeling technology are combined, the image of the building surface can be efficiently obtained, the fine three-dimensional model is constructed, the distortionless building facade panoramic image can be generated through the re-projection and splicing technology, and a reliable data basis is provided for related application.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a seamless stitching system and method for distortion-free panoramic images of building facades based on an unmanned aerial vehicle (UAV). Background Art

[0002] In the information era of the continuous development of digital cities, refined building models play an increasingly important role in urban scenes, and it is particularly important to show the realism of the texture information of building facades. The traditional method for obtaining images of building facades is to manually hold a camera to shoot the building surface and then process the captured images. The facade images obtained by the traditional method are often projected images with distortion, and usually it is difficult to meet the requirements of digital city refinement. Summary of the Invention

[0003] The purpose of the present invention is to solve the deficiencies existing in the above background art, and provide a seamless stitching system and method for distortion-free panoramic images of building facades based on an unmanned aerial vehicle (UAV).

[0004] The technical solution adopted by the present invention is: A seamless stitching system for distortion-free panoramic images of building facades based on an unmanned aerial vehicle (UAV), comprising:

[0005] An image acquisition module, which is a camera installed on the unmanned aerial vehicle, and is used to obtain preliminary images of the building surface, and is used to fly according to the flight path to obtain fine images of the building surface;

[0006] A three-dimensional modeling module, which is used to perform modeling based on the preliminary image data to obtain a rough three-dimensional model of the building, and is used to perform modeling according to the fine images to obtain a fine three-dimensional model of the building;

[0007] A flight path planning module, which is used to plan the flight path of the unmanned aerial vehicle flying close to the building based on the rough three-dimensional model of the building;

[0008] A reprojection module, which is used to obtain the building point cloud data based on the fine three-dimensional model of the building, and perform reprojection on each facade of the building according to the building point cloud data;

[0009] An image stitching module, which is used to stitch the reprojection images of different facades of the building to form a complete panoramic image of the building facade.

[0010] Further, the performing reprojection on each facade of the building according to the building point cloud data includes:

[0011] Counting the normal vectors of each point within a set area;

[0012] Selecting the corresponding set area according to the building facade to be reprojected, and determining the average point cloud normal vector of some or all points within the set area;

[0013] Reproject the building facade based on the average point cloud normal vector to form a reprojected image.

[0014] Furthermore, it also includes an image fusion module, which is used to perform color homogenization processing on the reprojected images of the building facade.

[0015] Furthermore, the color homogenization processing of the reprojected images of the building facade includes:

[0016] Select two reprojected images of adjacent building facades, label them as the source image and the target image respectively, and perform color adjustment on the target image based on the source image until the color difference between the target image and the source image is within the set range;

[0017] Repeat the above process until the color adjustment processing of all the reprojected images of the building facade is completed.

[0018] Even further, the stitching of the reprojected images of different building facades includes:

[0019] Crop all the reprojected images of the building facade until all the image sizes are the same;

[0020] Stitch the reprojected images of all the building facades after cropping according to the relative position relationship between the building facades, and label the orientation of the reprojected image of each building facade to form a complete panoramic image of the building facade.

[0021] A method for seamless stitching of distortion-free panoramic images of building facades based on drones,

[0022] Obtain the preliminary image of the building surface through a drone, and build a rough 3D model of the building based on the preliminary image;

[0023] Based on the rough 3D model of the building, plan the flight path of the drone flying close to the building, control the drone to fly according to the flight path to collect the fine images of the building surface again, and build a fine 3D model of the building based on the fine images;

[0024] Obtain the building point cloud data based on the fine 3D model of the building, and reproject each facade of the building according to the building point cloud data;

[0025] Stitch the reprojected images of different building facades to form a complete panoramic image of the building facade.

[0026] Furthermore, the reprojecting of each facade of the building according to the building point cloud data includes:

[0027] Obtain the normal vector of each point in the corresponding area in the point cloud data;

[0028] Select the area to be reprojected according to the building facade to be reprojected, and determine the average point cloud normal vector of all points within the area;

[0029] Based on the average point cloud normal vector, reproject the corresponding building facade to form a reprojected image.

[0030] Furthermore, before stitching the reprojected images of different building facades, it also includes:

[0031] Perform color homogenization processing on the reprojected images of the building facade.

[0032] Furthermore, the color homogenization processing of the reprojected images of the building facade includes:

[0033] Select two reprojected images of adjacent building facades, label them as the source image and the target image respectively, and perform color adjustment on the target image based on the source image until the color difference between the target image and the source image is within the set range;

[0034] Repeat the above process until the color adjustment processing of all reprojected images of the building facade is completed.

[0035] Even further, the stitching of the reprojected images of different building facades includes:

[0036] Crop all the reprojected images of the building facade until all the image sizes are the same;

[0037] Stitch all the cropped reprojected images of the building facade according to the relative position relationship between the building facades, and label the orientation of each reprojected image of the building facade to form a complete panoramic image of the building facade.

[0038] The beneficial effects of the present invention are:

[0039] The present invention combines the high mobility of the drone and 3D modeling technology, and can efficiently acquire images of the building surface and construct a fine 3D model; through reprojecting and stitching technologies, it can generate a distortion-free panoramic image of the building facade, which can provide strong support for fields such as building monitoring, construction, and design evaluation; at the same time, the generated panoramic image of the building facade has the characteristics of high precision and no distortion, can truly reflect the actual situation of the building facade, and provides a reliable data basis for related applications.

[0040] The present invention can reproject more accurately by obtaining the normal vector of each point in the point cloud data, selecting an area according to the building facade to be reprojected and calculating the average normal vector. This method takes into account the geometric characteristics of the building surface and improves the accuracy of reprojecting; moreover, the generated reprojected image can more truly reflect the geometric shape and texture characteristics of the building facade, providing a high-quality basis for subsequent image stitching.

[0041] The present invention performs color homogenization processing on the re-projected images of building facades. The color homogenization processing can eliminate the color difference between the re-projected images of adjacent building facades, making the spliced panoramic image more natural and coherent, and improving the visual quality of the panoramic image; moreover, the re-projected images of building facades after color homogenization processing can present consistent colors and brightness after splicing, enhancing the overall aesthetics and readability of the panoramic image.

[0042] The present invention selects two re-projected images of adjacent building facades and performs color adjustment processing on the target image based on the source image, which can gradually eliminate the color difference, and has the characteristics of simple operation and obvious effect; after gradual color adjustment processing, the color difference between the re-projected images of adjacent building facades gradually decreases until it reaches the set range, ensuring the color consistency of the spliced panoramic image.

[0043] The present invention generates a complete panoramic image of the building facade by cropping and splicing the re-projected images of all building facades and making annotations according to the relative position relationship between the building facades. This method takes into account the spatial relationship of the building facades, improves the accuracy and reliability of splicing, and the generated panoramic image of the building facade not only has the characteristics of high precision and no distortion, but also can clearly display the spatial layout and relative position relationship of the building facade, providing more intuitive and comprehensive data support for fields such as building monitoring and design evaluation. Brief Description of the Drawings

[0044] Figure 1 It is the system schematic diagram of the present invention.

[0045] Figure 2 It is the method flow chart of the present invention.

[0046] Figure 3 It is a panoramic image of a certain house spliced by using the method of the present invention. Detailed Embodiments

[0047] The following further describes the detailed embodiments of the present invention with reference to the drawings. It should be noted here that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation to the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Such as Figure 1As shown in the figure, a seamless stitching system for distortion-free panoramic images of building facades based on drones includes an image acquisition module, a 3D modeling module, a flight path planning module, a reprojection module, and an image stitching module. The image acquisition module, 3D modeling module, reprojection module, and image stitching module are connected in sequence. The output end of the 3D modeling module is also connected to the input end of the flight path planning module, and the output end of the flight path planning module is also connected to the input end of the image acquisition module. The functions implemented by each module are as follows:

[0049] The image acquisition module is used to obtain preliminary images of the building surface by flying along a preliminary flight trajectory and to obtain fine images of the building surface by flying along the flight path. Among them, the image acquisition module is preferably a drone, and a camera with high resolution and corresponding focal length is installed on the drone.

[0050] The 3D modeling module is used to build a rough 3D model of the building based on the preliminary image data, to build a fine 3D model of the building according to the fine images, and to obtain the 3D model data of the entire building. The 3D modeling module is preferably 3D modeling software, such as Context Capture, NAP, or DJI Terra.

[0051] The flight path planning module is used to plan the preliminary flight trajectory of the drone according to the preliminary landform of the area where the building is located, and to plan the flight path for the drone to fly close to the building based on the rough 3D model of the building. The flight path planning module is preferably route planning software, such as DJIPilot, NAP, etc. The preliminary landform of the corresponding area where the building is located can be obtained through external software (such as Google Earth, LSV, Ovi Map, etc.).

[0052] The reprojection module is used to obtain the building point cloud data based on the fine 3D model of the building, and to reproject each facade of the building according to the building point cloud data to obtain the best building observation perspective. The reprojection module is preferably visualization software.

[0053] The image stitching module is used to stitch the reprojected images of different facades of the building to form a complete panoramic image of the building facade. The image stitching module is preferably professional image processing software.

[0054] In some embodiments, the above system may further include an image fusion module. The image fusion module is connected between the reprojection module and the image stitching module. The image fusion module is used to perform color homogenization processing on the reprojected images of the building facade, that is, to process problems such as color differences caused by different lighting and acquisition devices for building facades with different orientations.

[0055] It should be noted that the above-mentioned and subsequent building facades refer to the surfaces of the building that are perpendicular or nearly perpendicular to the horizontal ground. The above-mentioned and subsequent buildings generally refer to houses.

[0056] Based on the above seamless stitching system for distortion-free panoramic images of building facades, the present invention also provides a method for seamless stitching of distortion-free panoramic images of building facades based on an unmanned aerial vehicle (UAV), as Figure 2 shown, which includes the following steps:

[0057] Step 1: Obtain preliminary images of the building surface through a UAV, and perform modeling based on the preliminary images to obtain a rough three-dimensional model of the building;

[0058] Step 2: Plan the flight path of the UAV flying close to the building based on the rough three-dimensional model of the building, control the UAV to fly according to the flight path to collect fine images of the building surface again, and perform modeling based on the fine images to obtain a fine three-dimensional model of the building;

[0059] Step 3: Obtain the building point cloud data based on the fine three-dimensional model of the building, and perform reprojection on each facade of the building according to the building point cloud data;

[0060] Step 4: Perform color homogenization processing on the reprojection images of the building facades.

[0061] Step 5: Stitch the reprojection images of different facades of the building after color homogenization processing to form a complete panoramic image of the building facade.

[0062] In some embodiments, the step of obtaining preliminary images of the building surface through a UAV and performing modeling based on the preliminary images to obtain a rough three-dimensional model of the building in Step 1 includes the following steps:

[0063] Step 1.1: According to the surrounding environment of the building, select a reasonable model of the low-altitude UAV photography equipment, and select suitable weather for flight and shooting (such as noon, less shadow, no wind, no cloud, high visibility, etc.).

[0064] Step 1.2: Use an external map software to understand the topography of the survey area, clarify the task area, whether there are flight restrictions, and perform reasonable division of flight sorties. Consider obstacle avoidance factors and optimize the preliminary flight plan.

[0065] Step 1.3: In the flight path planning software (such as DJIPilot), determine the flight altitude, determine the ground resolution, flight speed, set the overlap degree, yaw angle, etc., and use a high-resolution image UAV and a corresponding focal length camera to perform preliminary flight shooting on the building surface to obtain preliminary images of the building surface.

[0066] Step 1.4: Select a professional UAV image three-dimensional modeling software, import the aerial photography images in Step 1.3 for aerial triangulation reconstruction, set the output coordinate system (such as CGCS2000), complete the preliminary three-dimensional modeling, and obtain a rough three-dimensional model of the building.

[0067] Step 1.5: For areas with complex terrain or high requirements for modeling accuracy, an easily identifiable location can be found in the reconstruction result, and the reconstruction effect can be judged based on the coordinate values of the UAV. If the requirements are not met, image control points can be imported during the aerial triangulation modeling process, and the steps of point piercing, aerial triangulation re-optimization, and then reconstruction can be completed.

[0068] In some embodiments, step 2 is based on the rough three-dimensional model of the building to plan the flight path of the UAV close to the building, control the UAV to fly according to the flight path to collect fine images of the building surface again, and perform modeling based on the fine images to obtain the fine three-dimensional model of the building, including the following steps:

[0069] Step 2.1: Use the rough three-dimensional model of the building generated in step 1.4 and use professional trajectory planning software (NAP) to plan the flight route. According to the structural complexity of the building, select area targets, volume targets, or line targets to determine the type of flight path object.

[0070] Step 2.2: Select the UAV device model and camera focal length (for example, DJI Mavic3 with RTK module and 1 / 1.3-inch CMOS medium telephoto camera). Set shooting parameters such as shooting resolution, shooting distance, forward overlap, side overlap intersection angle, etc., and set parameters such as minimum takeoff height, minimum obstacle avoidance distance, obstacle avoidance strategy, and minimum flight spacing.

[0071] Step 2.3: After setting the parameters in step 2.2, output the flight route, simulate the aerial photography process in the NAP software, observe the flight trajectory of the UAV, check whether the flight route is safe, delete redundant transition points, and export the point passing mode using coordinated turning (no hovering during shooting, higher flight efficiency).

[0072] Step 2.4: According to the flight path generated in step 2.3, fly close to the building again to collect building surface images, and use commercial or open-source three-dimensional modeling software to perform fine modeling of the building to obtain the fine three-dimensional model of the building.

[0073] Step 2.5: After the reconstruction is completed, check whether the model is completely modeled. For areas with poor reconstruction quality, re-plan the flight route, supplement the shooting in a timely manner, and the reconstruction steps are similar to 1.5.

[0074] In some embodiments, step 3 is based on the fine three-dimensional model of the building to obtain the building point cloud data, and re-project each elevation of the building according to the building point cloud data, including the following steps:

[0075] Step 3.1: Input the finely modeled building fine 3D model in Step 2.4 into 3D model visualization software to obtain its point cloud data, and calculate the normal vector of each point within a set area. The set area is an area on the building surface that is relatively flat, has fewer interferences, and has a good reconstruction model. Each building elevation may have one or more corresponding set areas.

[0076] Step 3.2: According to a certain building elevation that needs to be reprojected, select a certain set area corresponding to this building elevation, and calculate the average point cloud normal vector of some or all points within the selected set area.

[0077] After completing the solution of the average point cloud normal vector of the set area, select any point within the set area to locate the projection plane, thereby obtaining all the parameters of the projection plane.

[0078] Project the point cloud of the building elevation along the direction of the average point cloud normal vector onto the determined projection plane. For example, it can be projected using a common point cloud library, Point Cloud Library (PCL), and finally form the reprojected building elevation image.

[0079] In some embodiments, due to the influence of light and shooting time, there are color differences in different building elevations. Therefore, color equalization processing is required to make the surface colors of the entire building as coordinated as possible. Step 4 performs color equalization processing on the reprojected images of the building elevations, including the following steps:

[0080] Step 4.1: Select two reprojected images of adjacent building elevations (i.e., the above-mentioned reprojected building elevation images), input them into professional image processing software, and mark the source image and the target image. Use the source image as a reference to perform color adjustment on the target image until the color difference between the target image and the source image is within the set range. The color adjustment processing includes adjusting parameters such as the color intensity and brightness of the image to make their colors close.

[0081] Step 4.2: Keep the source image unchanged, and select other building elevation images to repeat Step 4.1 for color equalization processing.

[0082] Step 4.3: After performing color equalization processing according to Step 4.1, if the color difference between adjacent images is still obvious, these adjacent images can be separately color-equalized.

[0083] In some embodiments, Step 5 stitches the reprojected images of different building elevations to form a complete panoramic image of the building elevation, including the following steps:

[0084] Step 5.1: On the premise of minimizing the loss of key information of the building elevation images, use the smallest image as a reference to crop all the reprojected images of the building elevations until all the images are of the same size.

[0085] Step 5.2: Select adjacent reprojection images, input one of the images into professional image processing software, and complete accurate stitching according to their relative positional relationships.

[0086] Step 5.3: Repeat Step 5.2 to stitch the reprojection images of all the cropped building facades, then unfold the four surfaces of the building facades, and mark their orientations at the top or bottom of the reprojection images of each building facade to obtain a complete panoramic image of the building facade, as Figure 3 shown. The orientation herein refers to the direction towards which the building facade faces, such as east, southeast, west, etc.

[0087] It should be noted that although the method of the present invention has described the specific implementation steps of the panoramic image of the building facade, the method of the present invention is not limited to some details in the steps. The objects processed in the method of the present invention are not limited to building facades, and can also be applied to polyhedral buildings, as well as special-shaped buildings such as the "hui" character shape and the "gong" character shape, and even can be applied to buildings with an arc-shaped outer facade such as cylinders; the modeling software and flight path planning software used are not limited to software such as DJI Terra, Context Capture, pix4D, smart3D, and NAP; the present invention proposes a process method for constructing a panoramic image of a building facade. Image stitching and color homogenization are not limited to the algorithms and software mentioned in the present invention, and can also be automatic stitching. Color homogenization can also calculate the mean and standard deviation between images and bring them into the Wallis algorithm for color adjustment, etc.; the means of the building image acquisition module are not limited to unmanned aerial vehicles. Because of flight restrictions of unmanned aerial vehicles, in some special areas and narrow sections, it can also be a handheld camera, a vehicle-mounted camera, or a combination of these image acquisition devices, and thus fine-grained modeling can also be completed.

[0088] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any change or replacement that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

Claims

1. A drone-based building facade distortion-free panoramic image seamless splicing system, characterized by: include The image acquisition module is a camera installed on the drone, which is used to obtain a preliminary image of the building surface and to obtain a detailed image of the building surface according to the flight path; A three-dimensional modeling module, used for modeling based on preliminary image data to obtain a rough three-dimensional model of the building, and used for modeling based on fine images to obtain a fine three-dimensional model of the building; The flight trajectory planning module is used to plan the flight trajectory of the drone based on the rough 3D model of the building; A reprojection module is used to obtain building point cloud data based on a fine three-dimensional model of the building, and reproject each facade of the building according to the building point cloud data; The image stitching module is used to stitch the re-projected images of different building facades to form a complete panoramic image of the building facade.

2. The drone-based building facade distortion-free panoramic image seamless stitching system according to claim 1 is characterized in that: The reprojecting of each facade of the building according to the building point cloud data includes: Get the normal vector of each point in the point cloud data in the corresponding area; Select the area to be reprojected according to the building facade to be reprojected, and determine the average point cloud normal vector of all points in the area; The corresponding building facade is reprojected based on the average point cloud normal vector to form a reprojected image.

3. The drone-based building facade distortion-free panoramic image seamless splicing system according to claim 1 is characterized by: It also includes an image fusion module, which is used to perform color uniformity processing on the reprojected image of the building facade.

4. The drone-based building facade distortion-free panoramic image seamless stitching system according to claim 3 is characterized in that: The color uniformity processing of the reprojected image of the building facade comprises: Select two reprojected images of adjacent building facades, mark them as source image and target image respectively, and adjust the color of the target image based on the source image until the color difference between the target image and the source image is within the set range; The above process is repeated until the color adjustment of the reprojected images of all building facades is completed.

5. The drone-based building facade distortion-free panoramic image seamless stitching system according to claim 1, characterized in that: The step of stitching the reprojected images of different facades of the building comprises: Crop the reprojected images of all building facades until all images are of the same size; The reprojected images of all the cropped building facades are spliced ​​according to the relative position relationship between the building facades, and the orientation of the reprojected image of each building facade is marked to form a complete panoramic image of the building facade.

6. A method for seamlessly stitching undistorted panoramic images of building facades based on drones, characterized in that: Obtain preliminary images of the building surface through drones, and perform modeling based on the preliminary images to obtain a rough three-dimensional model of the building; Based on the rough 3D model of the building, the flight path of the UAV is planned, the UAV is controlled to fly along the flight path to collect fine images of the building surface again, and modeling is performed based on the fine images to obtain a fine 3D model of the building; Obtain building point cloud data based on the detailed 3D model of the building, and reproject each facade of the building according to the building point cloud data; The reprojected images of different building facades are stitched together to form a complete panoramic image of the building facade.

7. The method for seamlessly stitching undistorted panoramic images of building facades based on drones according to claim 6 is characterized in that: The reprojecting of each facade of the building according to the building point cloud data includes: Count the normal vectors of each point in the set area; Select a corresponding set area according to the building facade that needs to be reprojected, and determine the average point cloud normal vector of some or all points in the set area; The building facade is reprojected based on the average point cloud normal vector to form a reprojected image.

8. The method for seamlessly stitching undistorted panoramic images of building facades based on drones according to claim 6, characterized in that: Before stitching the reprojected images of different facades of the building, the method further includes: Perform color homogenization on the reprojected image of the building facade.

9. The method for seamlessly stitching undistorted panoramic images of building facades based on drones according to claim 8, characterized in that: The color uniformity processing of the reprojected image of the building facade comprises: Select two reprojected images of adjacent building facades, mark them as source image and target image respectively, and adjust the color of the target image based on the source image until the color difference between the target image and the source image is within the set range; The above process is repeated until the color adjustment of the reprojected images of all building facades is completed.

10. The method for seamlessly stitching undistorted panoramic images of building facades based on drones according to claim 6, characterized in that: The step of stitching the reprojected images of different facades of the building comprises: Crop the reprojected images of all building facades until all images are of the same size; The reprojected images of all the cropped building facades are spliced ​​according to the relative position relationship between the building facades, and the orientation of the reprojected image of each building facade is marked to form a complete panoramic image of the building facade.

Images