A method and system for 3D reality modeling based on unmanned aerial vehicles (UAVs)

Abstract

This application relates to the technical field of 3D modeling, and in particular to a method and system for 3D reality modeling based on unmanned aerial vehicles (UAVs). The method includes: acquiring aerial view images collected by the UAV and multi-angle ground-based photographic data; then, preprocessing the aerial view images and multi-angle photographic data to obtain aerial view images and multi-angle photographic data of the target; wherein the preprocessing includes image distortion correction and point cloud data stitching. Furthermore, multi-source data fusion is performed based on the aerial view images and multi-angle photographic data of the target to obtain multi-source fused data; and 3D modeling is performed based on the multi-source fused data to obtain a 3D reality model. Performing multi-source data fusion integrates data from different sources, fully utilizing the advantages of both, and achieving data complementarity, thereby improving the accuracy and reliability of the 3D reality model created using multi-source fused data.

Description

Technical Field

[0001] This application relates to the technical field of 3D modeling, and in particular to a method and system for 3D real-scene modeling based on unmanned aerial vehicles (UAVs). Background Technology

[0002] With the advancement of information technology and the development of drone technology, drones have been widely used in various industries. Especially in fields such as geographic information systems and remote sensing, drones have gradually become one of the main tools for acquiring high-resolution imagery due to their flexibility and cost-effectiveness.

[0003] In 3D reality modeling, commonly used methods mainly include traditional photogrammetry, which generates 3D reality models from image data collected by drones. This is of great significance for urban planning and rapid disaster response. However, drones have a top-down perspective, so when buildings or other targets are tall, their bases are easily obscured by other objects. Furthermore, while aerial images can capture the overall structure of the target area, they often fail to capture detailed texture information, resulting in low accuracy of 3D reality models generated solely from aerial images collected by drones.

[0004] Therefore, how to provide a 3D reality modeling method that improves model accuracy is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for 3D reality modeling based on unmanned aerial vehicles (UAVs) to solve at least one of the above-mentioned technical problems.

[0006] The above-mentioned inventive objective of this application is achieved through the following technical solutions:

[0007] Firstly, this application provides a method for 3D reality modeling based on unmanned aerial vehicles (UAVs), employing the following technical solution:

[0008] A method for 3D reality modeling based on unmanned aerial vehicles (UAVs) includes:

[0009] The system acquires aerial view images collected by drones and multi-angle ground photography data collected by ground photography equipment. The aerial view images are used to show the overall structure and layout of the land surface, while the multi-angle ground photography data are used to show the details and textures of ground features.

[0010] Preprocessing is performed on the aerial view image and the multi-angle shooting data to obtain the aerial view image of the target and the multi-angle shooting data of the target. The preprocessing includes: image distortion correction and point cloud data stitching.

[0011] Multi-source data fusion is performed based on the high-altitude top-down image of the target and multi-angle shooting data of the target to obtain multi-source fused data, and three-dimensional modeling is performed based on the multi-source fused data to obtain a three-dimensional real scene model.

[0012] By employing the aforementioned technical solution, high-altitude aerial images collected by UAVs and multi-angle ground-based photographic data are acquired. Then, preprocessing is performed on the high-altitude aerial images and multi-angle photographic data to obtain target high-altitude aerial images and target multi-angle photographic data. This preprocessing includes image distortion correction and point cloud data stitching. Furthermore, multi-source data fusion is performed on the target high-altitude aerial images and target multi-angle photographic data to obtain multi-source fused data. Finally, 3D modeling is performed based on the multi-source fused data to obtain a 3D reality model. Performing multi-source data fusion integrates data from different sources, fully utilizing the advantages of both to achieve data complementarity, thereby improving the accuracy and reliability of the 3D reality model created using multi-source fused data.

[0013] In a preferred embodiment, this application can be further configured as follows: the preprocessing based on the aerial view image and the multi-angle shooting data to obtain the target aerial view image and the target multi-angle shooting data, wherein the preprocessing includes: image distortion correction and point cloud data stitching, including:

[0014] Distortion detection is performed based on the high-altitude overhead image to determine the distortion detection result. When the distortion detection result indicates that distortion exists, image distortion correction is performed on the high-altitude overhead image to obtain the target high-altitude overhead image.

[0015] Data filtering is performed based on the multi-angle shooting data to determine the lidar point cloud data. The point cloud data is then stitched together to obtain stitched point cloud data. Finally, the target multi-angle shooting data is determined based on the ground upward view image in the multi-angle shooting data and the stitched point cloud data.

[0016] In a preferred embodiment, this application can be further configured as follows: the multi-source data fusion based on the high-altitude overhead image of the target and the multi-angle shooting data of the target to obtain multi-source fused data includes:

[0017] Feature points are extracted based on the aerial view and the ground view, resulting in feature points for the aerial view and the ground view.

[0018] Feature point matching is performed based on the feature points of the top-view image and the feature points of the bottom-view image to determine the image feature mapping relationship;

[0019] The laser scanning point cloud data corresponding to the high-altitude overhead image is obtained. Based on the image feature mapping relationship, the laser scanning point cloud data and the stitched point cloud data are fused from multiple sources to obtain multi-source fused data.

[0020] In a preferred embodiment, this application may be further configured such that, prior to acquiring the high-altitude overhead imagery gathered by the UAV, it also includes:

[0021] Obtain a map of the area to be modeled, and perform multi-UAV collaborative operation analysis based on the map to be modeled to determine the image acquisition flight plan for each UAV. The multi-UAV collaborative operation analysis is used to control multiple UAVs to acquire images from different heights, angles and flight routes to form a three-dimensional coverage network.

[0022] The image acquisition flight plan is sent to the corresponding target UAV, and the target UAV is controlled to acquire images according to the image acquisition flight plan;

[0023] The image acquisition data sent by each target UAV is acquired, and the image acquisition data of each UAV is fused to obtain an aerial view.

[0024] In a preferred embodiment, this application can be further configured such that, after obtaining a 3D reality model based on the multi-source fusion data, the method further includes:

[0025] Based on the aforementioned 3D reality model, a multi-dimensional model quality assessment is performed to determine the model quality assessment result. The multi-dimensional model quality assessment includes: integrity quality assessment, accuracy quality assessment, consistency quality assessment, and texture quality assessment.

[0026] When the model quality assessment result is found to be non-compliant with quality requirements, an anomaly warning for the real-world model is generated.

[0027] In a preferred embodiment, this application may be further configured as follows: after performing a multi-dimensional model quality assessment based on the three-dimensional reality model and determining the model quality assessment result, it further includes:

[0028] When the model quality assessment result meets the quality requirements, the lighting simulation conditions are obtained. Based on the holographic projection technology and the lighting simulation conditions, a virtual three-dimensional image is obtained for the three-dimensional real scene model. The virtual three-dimensional image is used to simulate scene conditions under different lighting conditions in virtual space.

[0029] Secondly, this application provides a 3D reality modeling system based on unmanned aerial vehicles (UAVs), which adopts the following technical solution:

[0030] The data acquisition module is used to acquire aerial view images collected by drones and multi-angle ground photography data collected by ground photography equipment. The aerial view images are used to show the overall structure and layout of the ground surface, while the multi-angle ground photography data are used to show the details and textures of ground features.

[0031] The preprocessing module is used to preprocess the aerial view image and the multi-angle shooting data to obtain the aerial view image of the target and the multi-angle shooting data of the target. The preprocessing includes: image distortion correction and point cloud data stitching.

[0032] The 3D modeling module is used to perform multi-source data fusion based on the target's high-altitude top-down image and multi-angle shooting data to obtain multi-source fused data, and to perform 3D modeling based on the multi-source fused data to obtain a 3D reality model.

[0033] Thirdly, this application provides an electronic device that adopts the following technical solution:

[0034] At least one processor;

[0035] Memory;

[0036] At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: execute the above-described UAV-based 3D reality modeling method.

[0037] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0038] A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the aforementioned UAV-based 3D reality modeling method.

[0039] In summary, this application includes at least one of the following beneficial technical effects:

[0040] The process involves acquiring aerial view images from drones and multi-angle ground-based photographic data. Preprocessing of these images and data yields aerial view images and multi-angle ground-based photographic data of the target. Preprocessing includes image distortion correction and point cloud data stitching. Subsequently, multi-source data fusion is performed on these images and data to obtain multi-source fused data. This fused data is then used for 3D modeling to create a 3D reality model. Multi-source data fusion integrates data from different sources, leveraging the strengths of both to achieve data complementarity and improve the accuracy and reliability of the 3D reality model created using multi-source fused data.

[0041] Distortion detection is performed based on high-altitude overhead imagery to determine the results. When distortion is detected, image distortion correction is applied to the overhead imagery to obtain a high-altitude overhead image of the target. Image distortion correction improves the clarity and accuracy of the overhead imagery, providing a reliable foundation for subsequent 3D modeling. Simultaneously, data filtering is performed based on multi-angle captured data to determine LiDAR point cloud data. To improve the comprehensiveness of the point cloud data and the accuracy of the subsequent 3D model, point cloud data is stitched together from the LiDAR data to obtain stitched point cloud data. Furthermore, based on ground-view images from multiple angles and the stitched point cloud data, multi-angle captured data of the target is determined. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a method for 3D real-scene modeling based on a drone, according to one embodiment of this application.

[0043] Figure 2 This is a schematic diagram of the structure of a UAV-based 3D reality modeling system according to one embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device according to one embodiment of this application. Detailed Implementation

[0045] The following combination Figures 1 to 3 This application is described in further detail.

[0046] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that in the optional embodiments of this application, the object information and other related data involved require the permission or consent of the object when the embodiments of this application are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. That is to say, if the embodiments of this application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations, and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0048] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0049] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0050] This application provides a method for 3D reality modeling based on unmanned aerial vehicles (UAVs), executed by an electronic device. This electronic device can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication. This application does not impose any limitations on this. Figure 1 As shown, the method includes steps S101, S102, and S103, wherein:

[0051] Step S101: Acquire aerial view images collected by the UAV and multi-angle ground photography data collected by ground photography equipment. The aerial view images are used to show the overall structure and layout of the ground surface, while the multi-angle ground photography data are used to show the details and textures of ground features.

[0052] In the embodiments of this application, during the process of 3D reality modeling, drones have become one of the main tools for acquiring high-resolution images in building 3D reality models due to their flexibility and cost-effectiveness. That is, high-resolution cameras, LiDAR, and various sensors carried by drones are used to take aerial photos of the target area at appropriate flight altitudes and angles. The drones are wirelessly connected to electronic devices so that the electronic devices can acquire aerial images collected by the drones in real time. The aerial images are used to clearly show the overall structure and layout of the ground surface, such as the road network and the distribution of buildings in a city.

[0053] When drones take aerial photos, they may be obstructed by obstacles such as buildings and trees, resulting in some areas not being captured and information gaps in the aerial images. Furthermore, while aerial images can show the overall structure and layout of the ground, their detail resolution is often low due to limitations in shooting angle and distance, making it difficult to accurately capture and represent the detailed shapes and textures of ground features. Therefore, ground-based photography equipment, such as ground-based LiDAR and panoramic cameras, is deployed in the area to be modeled. This equipment is controlled to capture multi-angle images of the target ground features from different angles and positions. The ground-based photography equipment is wirelessly connected to electronic devices, allowing the electronic devices to acquire the multi-angle ground-based image data in real time. This multi-angle ground-based image data is used to display the details and textures of ground features, such as building walls, roofs, doors, and windows.

[0054] Step S102: Based on the aerial view image and multi-angle shooting data, preprocessing is performed to obtain the aerial view image of the target and the multi-angle shooting data of the target. The preprocessing includes: image distortion correction and point cloud data stitching.

[0055] In this embodiment of the application, when a drone takes pictures at high altitude, various factors such as camera lens, flight attitude, and atmospheric refraction can cause distortion in the captured high-altitude view images. Distorted images can affect the accuracy and precision of subsequent 3D modeling. Therefore, image distortion correction is performed to eliminate the distortion of the high-altitude view images, so that the target high-altitude view images conform to the characteristics of the object, thereby improving the quality and clarity of the image capture. At the same time, the LiDAR point cloud data in the multi-angle shooting data are point cloud data from different perspectives. In order to enable the subsequent 3D reality model to more comprehensively display the geometry and spatial distribution of each target object in the area to be modeled, point cloud data stitching is performed based on the LiDAR point cloud data in the multi-angle shooting data to obtain the target multi-angle shooting data. Performing point cloud data stitching helps to reduce the distortion or deformation of the subsequent 3D reality model caused by errors in single point cloud scanning or single-angle point cloud data, or by data loss.

[0056] There are various specific implementation processes for preprocessing, and the embodiments of this application do not limit them. In one feasible method, distortion detection is performed based on high-altitude top-down images to determine the distortion detection results. When the distortion detection results indicate the presence of distortion, a distortion correction algorithm is used to correct the image distortion of the high-altitude top-down images to obtain the target high-altitude top-down images. Data filtering is performed based on multi-angle shooting data to determine the lidar point cloud data. Point cloud data is stitched based on the lidar point cloud data to obtain stitched point cloud data. Finally, based on the ground-view images and stitched point cloud data from the multi-angle shooting data, the target multi-angle shooting data is determined.

[0057] Step S103: Perform multi-source data fusion based on the target's high-altitude overhead image and multi-angle shooting data to obtain multi-source fused data, and perform 3D modeling based on the multi-source fused data to obtain a 3D real scene model.

[0058] In the embodiments of this application, the high-altitude top-down imagery of the target acquired by the UAV can provide a macroscopic and comprehensive perspective, covering a large area, but may lack detail and accuracy. The multi-angle shooting data of the target acquired by ground photography equipment can capture detailed information from multiple angles. Then, multi-source data fusion is performed to integrate data from different sources, fully utilizing the advantages of both to achieve data complementarity, thereby improving the accuracy and reliability of the 3D reality model created using multi-source fused data. There are various specific implementation methods for multi-source data fusion, which are not limited in the embodiments of this application. In one feasible method, feature points are extracted based on the high-altitude top-down imagery and the ground-based bottom-up imagery to obtain feature points in the top-down imagery and the bottom-up imagery; feature point matching is performed based on the feature points in the top-down imagery and the bottom-up imagery to determine the image feature mapping relationship; laser scanning point cloud data corresponding to the high-altitude top-down imagery is acquired; and multi-source data fusion is performed on the laser scanning point cloud data and the stitched point cloud data based on the image feature mapping relationship to obtain multi-source fused data.

[0059] Furthermore, based on multi-source fusion data, 3D modeling is performed to generate a more realistic and accurate 3D reality model. This 3D reality model can realistically reflect the spatial structure and detailed features of the actual scene in the area to be modeled, providing strong support for research and applications in related fields, and also providing a common data foundation for research and applications in different fields. The specific implementation process of 3D modeling is as follows: An irregular triangular mesh is constructed based on the point cloud data in the multi-source fusion data. This irregular triangular mesh can accurately represent the surface morphology of terrain and buildings and is an important component of 3D modeling. Then, texture images from aerial and ground-level images are mapped onto the 3D model to increase the model's realism and detail. This texture addition step needs to ensure the accuracy and consistency of the texture to avoid problems such as texture misalignment or stretching. Next, the initially constructed 3D model is optimized, including but not limited to: removing redundant data, smoothing surfaces, and fixing defects, to improve the accuracy and visualization effect of the 3D model. Finally, multiple 3D models are integrated into a unified scene to form a complete 3D reality model. In this integration step, it is necessary to ensure seamless connection and consistency between different 3D models.

[0060] As can be seen, in this embodiment, aerial view images collected by a UAV and multi-angle ground-based photographic data collected by ground-based photography equipment are acquired. Then, preprocessing is performed on the aerial view images and multi-angle photographic data to obtain aerial view images and multi-angle photographic data of the target. The preprocessing includes image distortion correction and point cloud data stitching. Furthermore, multi-source data fusion is performed on the aerial view images and multi-angle photographic data of the target to obtain multi-source fused data. Three-dimensional modeling is then performed based on the multi-source fused data to obtain a three-dimensional reality model. Performing multi-source data fusion integrates data from different sources, fully utilizing the advantages of both to achieve data complementarity, thereby improving the accuracy and reliability of the three-dimensional reality model created using multi-source fused data.

[0061] Furthermore, to improve the clarity and accuracy of the aerial view images and enhance the comprehensiveness of the point cloud data and the accuracy of the subsequent 3D model, in this embodiment, preprocessing is performed based on the aerial view images and multi-angle shooting data to obtain the target aerial view images and target multi-angle shooting data. The preprocessing includes image distortion correction and point cloud data stitching, including:

[0062] Distortion detection is performed based on aerial view images to determine the distortion detection results. When the distortion detection results indicate the presence of distortion, image distortion correction is performed on the aerial view images to obtain the target aerial view image.

[0063] Data is filtered based on multi-angle shooting data to determine LiDAR point cloud data. Point cloud data is then stitched together to obtain stitched point cloud data. Finally, based on ground-view images from multi-angle shooting data and stitched point cloud data, multi-angle shooting data of the target is determined.

[0064] In this embodiment of the application, when a drone takes photos at high altitudes, various factors such as camera lens, flight attitude, and atmospheric refraction can cause distortion in the captured aerial view images. This distortion leads to a loss of geometric relationships in the images, affecting the accuracy and precision of subsequent 3D modeling. Therefore, image distortion correction is performed on the distorted aerial view images to eliminate the distortion and restore the true geometric relationships of the images, thereby improving the clarity and accuracy of the aerial view images and providing a reliable foundation for subsequent 3D modeling. Specifically, a suitable distortion detection algorithm is selected to perform distortion detection on the aerial view images, and the distortion detection results are determined. The distortion detection results include: the presence of distortion and the absence of distortion. When distortion exists, the distortion detection algorithm can accurately identify the type (e.g., radial distortion, tangential distortion, and atmospheric refraction) and degree of distortion in the aerial view images. Distortion detection algorithms include, but are not limited to, algorithms based on feature point matching and algorithms based on edge detection. Furthermore, when the distortion detection result indicates the presence of distortion, the camera's internal parameters and the corresponding flight attitude data of the UAV are acquired. Based on the camera's internal parameters and flight attitude data, an image distortion correction model is established. This image distortion correction model can describe the relationship between image distortion and flight attitude and is used to predict and correct distortion. Then, the image distortion correction model is used to correct the distortion of the high-altitude overhead image to obtain the target high-altitude overhead image. This image distortion correction is used to convert the distorted pixel coordinates into correct coordinates.

[0065] Meanwhile, the multi-angle shooting data includes LiDAR point cloud data collected by ground-based LiDAR and ground-view images captured by panoramic cameras. Therefore, data filtering is performed based on the multi-angle shooting data to determine the LiDAR point cloud data. However, the LiDAR point cloud data is from a single point cloud scan or a single angle, and this data may contain errors or missing information. Directly using this LiDAR point cloud data for 3D construction will lead to distortion or deformation of the subsequent 3D reality model. To improve the comprehensiveness of the point cloud data and the accuracy of the subsequent 3D model, point cloud data is stitched together based on the LiDAR point cloud data to obtain stitched point cloud data. Through point cloud data stitching, various local point cloud data can be integrated into a unified 3D coordinate system. Local point cloud data complement and verify each other, reducing errors and omissions, thereby improving the accuracy and precision of the subsequently constructed 3D model. Common methods for point cloud data stitching include, but are not limited to, target stitching, overlapping area stitching, and control point stitching. Users can select the appropriate point cloud data stitching method according to the specific situation of the point cloud data. This application embodiment does not limit this method further. Furthermore, based on the ground-view images and stitched point cloud data from the multi-angle shooting data, the multi-angle shooting data of the target is determined.

[0066] As can be seen, in this embodiment, distortion detection is performed based on high-altitude overhead images to determine the distortion detection results. When the distortion detection results indicate the presence of distortion, image distortion correction is performed on the high-altitude overhead images to obtain the target high-altitude overhead image. Image distortion correction is performed to improve the clarity and accuracy of the high-altitude overhead images, providing a reliable foundation for subsequent 3D modeling. Simultaneously, data filtering is performed based on multi-angle shooting data to determine LiDAR point cloud data. To improve the comprehensiveness of the point cloud data and the accuracy of the subsequent 3D model, point cloud data is stitched together based on the LiDAR point cloud data to obtain stitched point cloud data. Furthermore, based on the ground-view image from the multi-angle shooting data and the stitched point cloud data, the target multi-angle shooting data is determined.

[0067] Furthermore, to improve the accuracy and reliability of the 3D reality model, in this embodiment, multi-source data fusion is performed based on the target's high-altitude overhead image and multi-angle shooting data to obtain multi-source fused data, including:

[0068] Feature points are extracted from aerial top-down images and ground-level bottom-up images to obtain feature points from both images.

[0069] Feature point matching is performed based on feature points in top-view and bottom-view images to determine the image feature mapping relationship;

[0070] Laser scanning point cloud data corresponding to high-altitude overhead images are acquired. Based on the image feature mapping relationship, multi-source data fusion is performed on the laser scanning point cloud data and the stitched point cloud data to obtain multi-source fused data.

[0071] In the embodiments of this application, the high-altitude top-down images of the target collected by the UAV can provide a macroscopic and comprehensive perspective, covering a large area, but may lack in detail and accuracy. The multi-angle shooting data of the target collected by the ground photography equipment can capture detailed information from multiple angles. Then, multi-source data fusion is performed to integrate data from different sources, making full use of the advantages of both and achieving data complementarity, so as to improve the accuracy and reliability of the three-dimensional reality model created using multi-source fused data.

[0072] Specifically, feature point extraction is performed based on aerial and ground-level imagery from above and below, resulting in feature points for both images. This involves using feature point detection algorithms (e.g., SIFT, SURF, ORB) to detect feature points in both images. These algorithms can identify points with significant local characteristics, such as corner points and edge points. Then, for each detected feature point, its descriptor, also known as a feature vector, is calculated. This descriptor typically contains local information about the area around the feature point and is used for subsequent feature point matching. Next, a feature point matching algorithm is used to match the feature points in the aerial and ground-level images, establishing a matching relationship between the two points. Specifically, during feature point matching, the algorithm compares the descriptors of the feature points to find possible matching pairs. Using the selected matching pairs, a homography matrix is ​​calculated between the aerial and ground-level imagery. This homography matrix describes the mapping relationship between two planes and can be used to map points from the aerial imagery to the ground-level imagery, or vice versa.

[0073] Furthermore, laser scanning point cloud data corresponding to high-altitude overhead images is acquired. Based on image feature mapping relationships, multi-source data fusion is performed on the laser scanning point cloud data and the stitched point cloud data to obtain multi-source fused data. For multi-source data fusion, the laser scanning point cloud data and the stitched point cloud data are registered using image feature mapping relationships to achieve precise alignment between the point cloud data. Then, based on the registration, the laser scanning point cloud data and the stitched point cloud data are fused to generate high-quality multi-source fused data. This fusion process involves processing steps such as merging, deduplication, and smoothing of the point cloud data.

[0074] As can be seen, in this embodiment, feature points are extracted based on aerial and ground-level images from above, resulting in feature points for both the aerial and ground-level images. Feature point matching is then performed based on these feature points to determine the image feature mapping relationship. Next, laser scanning point cloud data corresponding to the aerial images is acquired. Based on the image feature mapping relationship, multi-source data fusion is performed on the laser scanning point cloud data and the stitched point cloud data to obtain multi-source fused data. Performing multi-source data fusion integrates data from different sources, fully utilizing the advantages of both to achieve data complementarity, thereby improving the accuracy and reliability of the 3D reality model created using multi-source fused data.

[0075] Furthermore, in order to improve the efficiency of UAV image acquisition and obtain more complete and accurate aerial view images, in this embodiment of the application, before acquiring the aerial view images acquired by the UAV, the following steps are also included:

[0076] Obtain a map of the area to be modeled, and perform multi-UAV collaborative operation analysis based on the map to determine the image acquisition flight plan for each UAV. The multi-UAV collaborative operation analysis is used to control multiple UAVs to acquire images from different altitudes, angles and flight routes to form a three-dimensional coverage network.

[0077] The image acquisition flight plan is sent to the corresponding target UAV, and the target UAV is controlled to acquire images according to the image acquisition flight plan;

[0078] The system acquires image data sent by each target drone and fuses the data to obtain an aerial view.

[0079] In the embodiments of this application, before the UAVs collect high-altitude aerial images, a detailed image acquisition flight plan needs to be formulated to ensure that each UAV flies and takes pictures according to the predetermined route and parameters. In order to improve the efficiency of UAV image acquisition, multiple UAVs work together to greatly shorten the image acquisition time. At the same time, by controlling multiple UAVs to collect images from different heights, different angles and different flight routes, more complete and accurate high-altitude aerial images can be obtained.

[0080] Specifically, a map of the area to be modeled is acquired, which may include satellite remote sensing images, topographic maps, digital elevation models (DEMs), etc. This map is then preprocessed to improve its accuracy and usability. Next, based on the size, shape, and complexity of the area to be modeled, the map is divided into several smaller operational areas. Multiple UAVs acquire images of each operational area from different angles and altitudes, ensuring that the image acquisition area covers the entire area to be modeled. Then, based on the specific map data of each operational area, a detailed flight plan is developed for each UAV, including takeoff point, flight route, altitude, speed, and shooting parameters. The flight plan should consider factors such as terrain, obstacles, and weather to ensure safe flight and high-quality image acquisition. Furthermore, a collaborative strategy between UAVs is designed, including communication protocols, information sharing mechanisms, task allocation, and scheduling, to ensure that multiple UAVs can share information and coordinate actions in real time for efficient collaborative operation. After this, the image acquisition flight plan is sent to the corresponding target UAVs, and the target UAVs are controlled to acquire images according to the flight plan.

[0081] The drone and electronic equipment communicate wirelessly to ensure that the image data collected by the drone is transmitted to the electronic equipment in real time. Therefore, after receiving the image data from each target drone, the electronic equipment fuses the data to obtain an aerial view. In the fusion operation of the image data, factors such as image overlap and matching accuracy should be considered to ensure the data quality of the fused aerial view.

[0082] As can be seen, in this embodiment, to improve the efficiency of UAV image acquisition, a map of the area to be modeled is obtained, and multi-UAV collaborative operation analysis is performed based on the map to determine the image acquisition flight plan for each UAV. The image acquisition flight plan is sent to the corresponding target UAV to control the target UAV to acquire images according to the plan. By controlling multiple UAVs to acquire images from different altitudes, angles, and flight paths, a more complete and accurate high-altitude overhead image is obtained. Then, the image acquisition data sent by each target UAV is acquired, and the image acquisition data is fused to obtain the high-altitude overhead image.

[0083] Furthermore, in order to promptly identify defects in the 3D reality model and improve its reliability and stability, in this embodiment of the application, after obtaining the 3D reality model based on multi-source fusion data for 3D modeling, the following steps are also included:

[0084] Multi-dimensional model quality assessment is performed based on 3D reality models to determine the model quality assessment results. The multi-dimensional model quality assessment includes: integrity quality assessment, accuracy quality assessment, consistency quality assessment and texture quality assessment.

[0085] When the model quality assessment result is found to be non-compliant with quality requirements, an alert for abnormality in the real-world model is generated.

[0086] In the embodiments of this application, the quality of the 3D reality model, as an important data foundation, directly affects the effect and accuracy of subsequent applications. Therefore, by performing a multi-dimensional model quality assessment on the 3D reality model, it is helpful to promptly identify problems in the 3D reality model in terms of integrity, accuracy, consistency, and texture, so as to take timely measures to correct and optimize it, thereby improving the reliability and stability of the 3D reality model.

[0087] Specifically, a multi-dimensional model quality assessment is conducted based on the 3D reality model to determine the model quality assessment results. This multi-dimensional model quality assessment includes: integrity quality assessment, accuracy quality assessment, consistency quality assessment, and texture quality assessment. For integrity quality assessment, it verifies whether the 3D reality model covers all target features within the area to be modeled. This is achieved by comparing information from the original data sources (aerial view images and multi-angle shooting data) with the information in the 3D reality model, ensuring that no important geographical or structural features are omitted. For accuracy quality assessment, the accuracy of the 3D reality model is verified using known feature points (e.g., control points, landmarks, etc.). This involves comparing the position, size, and shape of features in the 3D reality model with those of actual features to assess the accuracy of the 3D reality model. For consistency quality assessment, it checks whether different parts of the 3D reality model (e.g., buildings, roads, vegetation, etc.) are logically consistent and free of contradictions, helping to ensure the realism and credibility of the 3D reality model. For texture quality assessment, for 3D reality models containing texture information, it is necessary to check the clarity, coherence and realism of the texture. That is, the texture should match the appearance of the actual ground features without obvious distortion or splicing errors.

[0088] Furthermore, when the model quality assessment result is found to be non-compliant with quality requirements, an anomaly warning for the real-world model is generated. This anomaly warning is used to prompt relevant personnel to respond promptly and take relevant measures to resolve the problem, so as to prevent adverse consequences caused by inaccurate 3D real-world models.

[0089] As can be seen, in this embodiment, a multi-dimensional model quality assessment is performed based on the 3D reality model to determine the model quality assessment result. When the model quality assessment result does not meet the quality requirements, an anomaly warning for the reality model is generated. By performing a multi-dimensional model quality assessment on the 3D reality model, it is helpful to promptly identify problems in the 3D reality model regarding its integrity, accuracy, consistency, and texture, thereby taking timely measures for correction and optimization, which helps to improve the reliability and stability of the 3D reality model.

[0090] Furthermore, to enhance user immersion and experience, and to make the scenes in the virtual space more vivid and realistic, in this embodiment of the application, a multi-dimensional model quality assessment is performed based on the 3D real-scene model. After determining the model quality assessment result, the following steps are also included:

[0091] When the model quality assessment result meets the quality requirements, the lighting simulation conditions are obtained. Based on holographic projection technology and lighting simulation conditions, virtual 3D images are obtained from the 3D real scene model. The virtual 3D images are used to simulate scene conditions under different lighting conditions in virtual space.

[0092] In this embodiment of the application, by combining holographic projection technology and lighting simulation conditions, scene conditions under different lighting conditions are simulated, making the virtual three-dimensional image closer to the real scene, which helps to enhance the user's immersion and experience, and makes the scene in the virtual space more vivid and realistic.

[0093] Specifically, when the model quality assessment result meets the quality requirements, the lighting simulation conditions are obtained. These lighting simulation conditions are lighting data under different time periods and weather conditions. Then, the lighting simulation conditions and the 3D real scene model are loaded into the holographic projection system. Holographic projection technology is used to visualize and render the 3D real scene model to obtain virtual 3D images simulating scene conditions under different lighting conditions.

[0094] As can be seen, in this embodiment, when the model quality assessment result meets the quality requirements, the lighting simulation conditions are obtained. Based on holographic projection technology and lighting simulation conditions, a virtual 3D image is obtained from the 3D real-world model. By combining holographic projection technology and lighting simulation conditions, scene conditions under different lighting conditions are simulated, making the virtual 3D image closer to the real scene, which helps to enhance the user's immersion and experience, and makes the scene in the virtual space more vivid and realistic.

[0095] The above embodiments introduce a UAV-based 3D reality modeling method from the perspective of process flow. The following embodiments introduce a UAV-based 3D reality modeling system from the perspective of virtual modules or virtual units. For details, please refer to the following embodiments.

[0096] This application provides a UAV-based 3D reality modeling system, such as... Figure 2 As shown, the UAV-based 3D reality modeling system may specifically include:

[0097] The data acquisition module 210 is used to acquire aerial view images collected by drones and multi-angle ground photography data collected by ground photography equipment. The aerial view images are used to show the overall structure and layout of the ground surface, while the multi-angle ground photography data are used to show the details and textures of ground features.

[0098] The preprocessing module 220 is used to preprocess the aerial view image and multi-angle shooting data to obtain the aerial view image and multi-angle shooting data of the target. The preprocessing includes image distortion correction and point cloud data stitching.

[0099] The 3D modeling module 230 is used to perform multi-source data fusion based on the target's high-altitude top-view image and multi-angle shooting data to obtain multi-source fused data, and to perform 3D modeling based on the multi-source fused data to obtain a 3D real scene model.

[0100] In this embodiment, aerial view images collected by a drone and multi-angle ground-based photographic data are acquired. Then, preprocessing is performed on the aerial view images and multi-angle photographic data to obtain aerial view images and multi-angle photographic data of the target. The preprocessing includes image distortion correction and point cloud data stitching. Next, multi-source data fusion is performed on the aerial view images and multi-angle photographic data of the target to obtain multi-source fused data. Three-dimensional modeling is then performed based on the multi-source fused data to obtain a three-dimensional reality model. Performing multi-source data fusion integrates data from different sources, fully utilizing the advantages of both to achieve data complementarity, thereby improving the accuracy and reliability of the three-dimensional reality model created using multi-source fused data.

[0101] In one possible implementation of this application embodiment, the preprocessing module 220, when performing preprocessing based on high-altitude overhead images and multi-angle shooting data to obtain target high-altitude overhead images and target multi-angle shooting data, wherein the preprocessing includes: image distortion correction and point cloud data stitching, is used for:

[0102] Distortion detection is performed based on aerial view images to determine the distortion detection results. When the distortion detection results indicate the presence of distortion, image distortion correction is performed on the aerial view images to obtain the target aerial view image.

[0103] Data is filtered based on multi-angle shooting data to determine LiDAR point cloud data. Point cloud data is then stitched together to obtain stitched point cloud data. Finally, based on ground-view images from multi-angle shooting data and stitched point cloud data, multi-angle shooting data of the target is determined.

[0104] In one possible implementation of this application embodiment, the 3D modeling module 230 performs multi-source data fusion based on the target's high-altitude top-view image and multi-angle shooting data of the target. When obtaining the multi-source fused data, it is used for:

[0105] Feature points are extracted from aerial top-down images and ground-level bottom-up images to obtain feature points from both images.

[0106] Feature point matching is performed based on feature points in top-view and bottom-view images to determine the image feature mapping relationship;

[0107] Laser scanning point cloud data corresponding to high-altitude overhead images are acquired. Based on the image feature mapping relationship, multi-source data fusion is performed on the laser scanning point cloud data and the stitched point cloud data to obtain multi-source fused data.

[0108] One possible implementation of this application's embodiment, a UAV-based 3D reality modeling system, further includes:

[0109] The collaborative operation analysis module is used to acquire a map of the area to be modeled, and to perform multi-UAV collaborative operation analysis based on the map of the area to be modeled to determine the image acquisition flight plan corresponding to each UAV. Among them, the multi-UAV collaborative operation analysis is used to control multiple UAVs to acquire images from different altitudes, different angles and different flight routes to form a three-dimensional coverage network.

[0110] The image acquisition flight plan is sent to the corresponding target UAV, and the target UAV is controlled to acquire images according to the image acquisition flight plan;

[0111] The system acquires image data sent by each target drone and fuses the data to obtain an aerial view.

[0112] One possible implementation of this application's embodiment, a UAV-based 3D reality modeling system, further includes:

[0113] The multidimensional model quality assessment module is used to perform multidimensional model quality assessment based on 3D reality models and determine the model quality assessment results. The multidimensional model quality assessment includes: integrity quality assessment, accuracy quality assessment, consistency quality assessment and texture quality assessment.

[0114] When the model quality assessment result is found to be non-compliant with quality requirements, an alert for abnormality in the real-world model is generated.

[0115] One possible implementation of this application's embodiment, a UAV-based 3D reality modeling system, further includes:

[0116] The holographic projection module is used to obtain lighting simulation conditions when the model quality assessment result meets the quality requirements. Based on holographic projection technology and lighting simulation conditions, virtual 3D images are obtained from the 3D real scene model. The virtual 3D images are used to simulate scene conditions under different lighting conditions in virtual space.

[0117] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the UAV-based 3D real-scene modeling system described above can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0118] This application provides an electronic device, such as... Figure 3 As shown, Figure 3 The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.

[0119] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0120] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0121] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0122] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0123] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0124] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0125] This application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments. Compared with related technologies, this application acquires aerial view images collected by a UAV and multi-angle ground-based photographic data collected by ground-based photography equipment. Then, preprocessing is performed on the aerial view images and multi-angle photographic data to obtain aerial view images and multi-angle photographic data of the target. The preprocessing includes image distortion correction and point cloud data stitching. Furthermore, multi-source data fusion is performed on the aerial view images and multi-angle photographic data of the target to obtain multi-source fused data. Three-dimensional modeling is then performed based on the multi-source fused data to obtain a three-dimensional reality model. Performing multi-source data fusion integrates data from different sources, fully utilizing the advantages of both to achieve data complementarity, thereby improving the accuracy and reliability of the three-dimensional reality model created using multi-source fused data.

[0126] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0127] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for 3D reality modeling based on unmanned aerial vehicles (UAVs), characterized in that, include: The system acquires aerial view images collected by drones and multi-angle ground photography data collected by ground photography equipment. The aerial view images are used to show the overall structure and layout of the Earth's surface. Ground-based multi-angle photographic data is used to showcase the details and textures of ground features; Preprocessing is performed on the aerial view image and the multi-angle shooting data to obtain the aerial view image of the target and the multi-angle shooting data of the target. The preprocessing includes: image distortion correction and point cloud data stitching. Multi-source data fusion is performed based on the high-altitude overhead image of the target and multi-angle shooting data of the target to obtain multi-source fused data, and three-dimensional modeling is performed based on the multi-source fused data to obtain a three-dimensional real scene model. The preprocessing based on the high-altitude overhead image and the multi-angle shooting data to obtain the target high-altitude overhead image and target multi-angle shooting data includes: image distortion correction and point cloud data stitching, including: Distortion detection is performed based on the high-altitude overhead image to determine the distortion detection result. When the distortion detection result indicates that distortion exists, image distortion correction is performed on the high-altitude overhead image to obtain the target high-altitude overhead image. Based on the multi-angle shooting data, data filtering is performed to determine the lidar point cloud data. Based on the lidar point cloud data, point cloud data is stitched together to obtain stitched point cloud data. Based on the ground upward view image in the multi-angle shooting data and the stitched point cloud data, the target multi-angle shooting data is determined. The step of fusing multi-source data based on the high-altitude overhead image of the target and multi-angle shooting data of the target to obtain multi-source fused data includes: Feature points are extracted based on the aerial view and the ground view, resulting in feature points for the aerial view and the ground view. Feature point matching is performed based on the feature points of the top-view image and the feature points of the bottom-view image to determine the image feature mapping relationship; The laser scanning point cloud data corresponding to the high-altitude overhead image is obtained. Based on the image feature mapping relationship, the laser scanning point cloud data and the stitched point cloud data are fused from multiple sources to obtain multi-source fused data.

2. The UAV-based 3D reality modeling method according to claim 1, characterized in that, Before acquiring the high-altitude overhead images collected by the UAV, the process also includes: Obtain a map of the area to be modeled, and perform multi-UAV collaborative operation analysis based on the map to be modeled to determine the image acquisition flight plan for each UAV. The multi-UAV collaborative operation analysis is used to control multiple UAVs to acquire images from different heights, angles and flight routes to form a three-dimensional coverage network. The image acquisition flight plan is sent to the corresponding target UAV, and the target UAV is controlled to acquire images according to the image acquisition flight plan; The image acquisition data sent by each target UAV is acquired, and the image acquisition data of each UAV is fused to obtain an aerial view.

3. The UAV-based 3D reality modeling method according to claim 1, characterized in that, After obtaining a 3D reality model based on the multi-source fusion data, the process further includes: Based on the aforementioned 3D reality model, a multi-dimensional model quality assessment is performed to determine the model quality assessment result. The multi-dimensional model quality assessment includes: integrity quality assessment, accuracy quality assessment, consistency quality assessment, and texture quality assessment. When the model quality assessment result is found to be non-compliant with quality requirements, an anomaly warning for the real-world model is generated.

4. The UAV-based 3D reality modeling method according to claim 3, characterized in that, After determining the model quality assessment result based on the three-dimensional reality model, the process further includes: When the model quality assessment result meets the quality requirements, the lighting simulation conditions are obtained. Based on the holographic projection technology and the lighting simulation conditions, a virtual three-dimensional image is obtained for the three-dimensional real scene model. The virtual three-dimensional image is used to simulate scene conditions under different lighting conditions in virtual space.

5. A 3D reality modeling system based on unmanned aerial vehicles (UAVs), characterized in that, include: The data acquisition module is used to acquire aerial view images collected by drones and multi-angle ground photography data collected by ground photography equipment. The aerial view images are used to show the overall structure and layout of the ground surface, while the multi-angle ground photography data are used to show the details and textures of ground features. The preprocessing module is used to preprocess the aerial view image and the multi-angle shooting data to obtain the aerial view image of the target and the multi-angle shooting data of the target. The preprocessing includes: image distortion correction and point cloud data stitching. The 3D modeling module is used to perform multi-source data fusion based on the high-altitude top-view image of the target and multi-angle shooting data of the target to obtain multi-source fused data, and to perform 3D modeling based on the multi-source fused data to obtain a 3D real scene model. The preprocessing module performs preprocessing based on the aerial view image and the multi-angle shooting data to obtain the target aerial view image and the target multi-angle shooting data. The preprocessing includes image distortion correction and point cloud data stitching, and is used for: Distortion detection is performed based on the high-altitude overhead image to determine the distortion detection result. When the distortion detection result indicates that distortion exists, image distortion correction is performed on the high-altitude overhead image to obtain the target high-altitude overhead image. Based on the multi-angle shooting data, data filtering is performed to determine the lidar point cloud data. Based on the lidar point cloud data, point cloud data is stitched together to obtain stitched point cloud data. Based on the ground upward view image in the multi-angle shooting data and the stitched point cloud data, the target multi-angle shooting data is determined. The 3D modeling module, when performing multi-source data fusion based on the high-altitude overhead image and multi-angle photographs of the target to obtain multi-source fused data, is used for: Feature points are extracted based on the aerial view and the ground view, resulting in feature points for the aerial view and the ground view. Feature point matching is performed based on the feature points of the top-view image and the feature points of the bottom-view image to determine the image feature mapping relationship; The laser scanning point cloud data corresponding to the high-altitude overhead image is obtained. Based on the image feature mapping relationship, the laser scanning point cloud data and the stitched point cloud data are fused from multiple sources to obtain multi-source fused data.

6. An electronic device, characterized in that, include: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, said at least one application being configured to: perform the UAV-based 3D reality modeling method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed in a computer, causes the computer to perform the UAV-based 3D reality modeling method according to any one of claims 1 to 4.

Images