Three-dimensional house modeling system constructed based on unmanned aerial vehicle aerial photography in automatic reconnaissance

Abstract

The invention relates to the technical field of unmanned aerial vehicle photogrammetry, computer vision and three-dimensional modeling, and discloses a three-dimensional house modeling system constructed based on unmanned aerial vehicle aerial photography in automatic reconnaissance, and the system comprises a data collection module which is used for collecting image data and spatial information of terrains, buildings and surroundings through a sensing device carried by an unmanned aerial vehicle; the task planning module is used for generating a flight task plan of the unmanned aerial vehicle according to the characteristics of the target area; the data processing module is used for receiving and processing data acquired by the unmanned aerial vehicle; and the three-dimensional modeling module is used for generating a three-dimensional model of the target area or the target building according to the image data and the spatial information provided by the data processing module. Through multi-sensor fusion acquisition, dynamic path planning, block efficient transmission and deep learning optimization modeling technologies, the problems of insufficient data coverage, low flight efficiency and transmission interruption in the prior art are solved, and a high-precision three-dimensional modeling scheme is realized.

Description

Technical Field

[0001] The present invention relates to the technical fields of unmanned aerial vehicle photogrammetry, computer vision and three-dimensional modeling, and in particular to a three-dimensional house modeling system based on unmanned aerial vehicle aerial photography in automated surveying. Background Art

[0002] Traditional house 3D modeling relies on manual on-site measurement and manual model construction, which is time-consuming and costly. In recent years, the development of drone technology and computer vision has made automated 3D modeling possible. However, existing automated modeling methods are often not accurate and lack efficient model optimization methods; Currently, many existing technologies rely on only a single sensor (such as a camera or lidar) for data collection. This solution can meet basic needs in simple environments, but in complex scenes, a single sensor is difficult to fully collect the spatial information of the target building. This collection method cannot meet the requirements of high-precision 3D modeling; In terms of UAV flight path planning, existing technologies mostly use manually preset paths or simple static algorithms. Although this method can cope with areas with relatively flat terrain, it is less efficient in complex terrain or dynamic environments. Because the path planning algorithm does not consider the comprehensive cost of terrain height changes and obstacle detours, UAVs often make repeated flights or fly too long, which seriously affects flight efficiency; In terms of data transmission, existing technologies show obvious deficiencies when facing network fluctuations and transmission interruptions. Most existing methods rely on transmitting a whole block of data at one time. Once the transmission is interrupted, all data needs to be retransmitted from the beginning. In addition, the transmission data is not processed in blocks and dynamically compressed, resulting in low bandwidth utilization efficiency, which is particularly obvious when transmitting large-scale image and point cloud data. In the 3D modeling process, most existing technologies rely on traditional geometric algorithms for model reconstruction. This method can generate basic 3D models when dealing with simple structures, but in complex buildings or scenes with more details, the generated models are not accurate enough and the surface details are seriously missing. Existing technologies have weak ability to repair missing data in modeling, especially in areas where data is not collected sufficiently, where a large number of holes or discontinuities will appear on the model surface. Traditional geometric algorithms are difficult to achieve high-quality texture mapping, resulting in poor performance of the generated models in terms of realism. Therefore, a three-dimensional house modeling system based on drone aerial photography in automated survey is proposed to solve the above problems. Summary of the invention

[0003] In view of the shortcomings of the existing technology, the present invention provides a three-dimensional house modeling system based on drone aerial photography in automated surveys. Through drone multi-sensor fusion technology, combined with high-resolution images and lidar point clouds, the problem of insufficient data coverage of a single sensor is solved; a flight path is generated through a planning module, path planning is optimized and the problem of low efficiency of manual path preset is solved; transmission efficiency is significantly improved, and the problem of data interruption caused by network fluctuations is solved; by combining a three-dimensional modeling algorithm optimized by deep learning, a high-precision and realistic model is generated, which solves the shortcomings of the existing modeling technology, such as the lack of details and poor applicability.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: a three-dimensional house modeling system is constructed based on drone aerial photography in automated survey, including: The data acquisition module is used to collect image data and spatial information of terrain, buildings and surrounding environment through the sensor equipment carried by the drone; The mission planning module is used to generate the flight mission plan of the UAV according to the characteristics of the target area; A data processing module, used to receive and process data collected by the drone; A three-dimensional modeling module, used to generate a three-dimensional model of a target area or a target building according to the image data and spatial information provided by the data processing module, wherein the three-dimensional model is constructed based on multi-angle image feature matching, multi-view geometry algorithm and modeling optimization technology; The system integration module is used to optimize the storage and result output of modeled data, and supports users to access, interact and apply the generated 3D models through multiple devices.

[0005] Preferably, the data acquisition module includes an image acquisition device, a multi-axis gimbal, a GPS, an inertial measurement unit and an auxiliary sensing device, and the multi-axis gimbal controls the image acquisition device to obtain image data from the east, south, west, north and directly above the target building.

[0006] Preferably, the mission planning module calculates the flight path of the UAV by combining geographic elevation data and obstacle distribution information of the target area using a global planning algorithm, and the planning result is based on the principle of minimizing path cost.

[0007] Preferably, the minimization of path cost includes the following factors: The flight distance between the starting point and the target node; The change in terrain height between the current node and the target node; The estimated cost from the target node to the final end point is calculated by weighting the flight distance and altitude change.

[0008] Preferably, the data processing module processes the image data in segments through a block transmission strategy, sets verification information for each segment of data, and supports breakpoint resumption and failure retransmission during the transmission process.

[0009] Preferably, the three-dimensional modeling module includes: A feature point extraction unit, used to extract and match key feature points from multi-angle image data; A point cloud generation unit is used to generate a sparse point cloud model based on the feature points through a multi-view geometry algorithm and gradually densify it; an optimization unit is used to repair missing parts in the sparse point cloud model through a deep learning model and optimize the surface texture and structural details of the model.

[0010] Preferably, the optimization unit performs local area completion and global texture enhancement on the input point cloud data based on a deep learning model, and the deep learning model optimizes its modeling capability through multi-view training of building samples.

[0011] Preferably, the system integration module utilizes a cloud platform for data storage, processing and distribution, supports multiple users to access the generated three-dimensional model simultaneously, and allows users to perform dynamic annotation, measurement and result export operations.

[0012] Preferably, the result export operation includes the following steps: receiving user-selected export options, the options including a target file format, a model resolution, and an output path; Performing format conversion processing on the generated 3D model, converting the data structure of the 3D model from the internal storage format to a universal file format, wherein the file format includes an OBJ file and an STL file, and retaining the geometric topology information and texture mapping information of the model during the conversion process; Calculate the vertex data, patch data and normal vector data of the 3D model and rearrange the data in a standardized form to ensure the integrity of the model structure, where the vertex data is recorded in the form of point coordinates, the patch data is recorded in the form of polygon definition, and the normal vector data is recorded in the form of a vector in the normal direction; Compress and integrate the texture data, embed the image file of the model surface texture mapping into the export file, and use lossless compression to ensure that the exported texture quality is consistent with that when the model is generated; Calculate the file size and generate a checksum for the file, which is used to verify the integrity of the file during transmission or storage; save the exported file in a path specified by the user, or use the system default path if no path is specified; After the export is completed, an export log is generated, which includes the file name, export time, file format and checksum, and provides users with download or viewing functions.

[0013] The present invention also provides a three-dimensional house modeling method based on an unmanned aerial vehicle, comprising the following steps: The sensor equipment carried by the drone collects multi-angle image data and spatial information of the target building and its surrounding environment; Generate the flight path of the drone based on the terrain and obstacle information of the target area and set the shooting mission; The collected image data and spatial information are transmitted to the ground station or cloud platform, and data compression, segment verification and breakpoint resumption are supported during the transmission process; Use 3D modeling algorithms to extract and match feature points from image data to generate a sparse point cloud model, and use geometric algorithms and optimization techniques to generate a dense point cloud model; Combined with deep learning models to repair missing parts in point cloud data and optimize surface texture and structural details; The generated 3D model is stored through the cloud platform, and provides users with multi-terminal access and interaction functions.

[0014] The present invention provides a three-dimensional house modeling system based on drone aerial photography in automated surveying. It has the following beneficial effects: 1. The present invention adopts a data acquisition technology solution based on multi-sensor fusion of drones, and achieves the technical effect of all-round and multi-view acquisition of target buildings and environmental data through the combination of high-resolution images and lidar point clouds. Compared with the technical solution in the prior art that relies on only a single sensor and leads to insufficient data coverage, it solves the problems of missing image information and low point cloud accuracy in complex scenes, and significantly improves the integrity and accuracy of the collected data.

[0015] 2. The present invention uses an automated mission planning module to dynamically generate a UAV flight path in combination with terrain elevation data and obstacle distribution, achieving the technical effect of path planning optimization and real-time adjustment. Compared with the technical solution in the prior art that relies on manually preset paths, resulting in low flight efficiency and poor mission flexibility, it solves the problems of repeated UAV paths and long flight time under complex terrain conditions, while improving mission execution efficiency.

[0016] 3. The present invention effectively improves the transmission efficiency of image data and spatial information through block transmission, breakpoint resume transmission and dynamic compression strategies in the data processing module, achieving the technical effect of high-speed and stable transmission. Compared with the technical solution in the prior art that requires data transmission to start from the beginning after interruption, which is time-consuming and labor-intensive, the problem of transmission interruption caused by network fluctuations or insufficient bandwidth is solved, and the failure rate of data transmission is greatly reduced.

[0017] 4. The present invention adopts a 3D modeling algorithm combined with deep learning optimization, and generates a high-precision and realistic 3D model by densifying sparse point clouds and enhancing the texture mapping of surface details. Compared with the technical solutions in the prior art where the models lack details and the surface texture is not realistic enough, the shortcomings of rough modeling results and unsuitability for complex building structure expression are solved, and the precision and scope of application of 3D modeling are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the system framework of the present invention; Figure 2 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Please refer to the attached Figure 1 The embodiment of the present invention provides a three-dimensional house modeling system based on drone aerial photography in automated survey, including: The data acquisition module is used to collect image data and spatial information of terrain, buildings and surrounding environment through the sensor equipment carried by the drone; The data acquisition module is one of the basic modules of the entire technical solution. Its function is to obtain the image data and spatial information required to build a three-dimensional model. This module mainly uses the sensor equipment carried by the drone to achieve all-round acquisition of the target building and its surrounding environment, providing raw data support for subsequent data processing and three-dimensional modeling. In general, the data acquisition module needs to ensure the integrity and accuracy of the collected data, especially in terms of multi-angle and multi-view image acquisition, which has significant technical characteristics.

[0021] In this embodiment, the data acquisition module includes an image acquisition device, a multi-axis gimbal, a GPS, an inertial measurement unit, and auxiliary sensing equipment. These components work together to efficiently complete the image and measurement data acquisition of the target building and its surrounding environment.

[0022] The image acquisition device uses a high-resolution camera, such as a 4K resolution or higher imaging device, which can capture detailed information of the target building. Specifically, the image acquisition device shoots at a fixed interval per second and can improve the image quality through the automatic exposure function in low light conditions. In order to achieve complete multi-angle acquisition, the image acquisition device is combined with a multi-axis gimbal, which can shoot from the top, east, south, west, and north of the target building. The angle adjustment accuracy of the gimbal can reach 0.1°, ensuring the continuity and consistency of multi-view images.

[0023] GPS provides accurate geographic location information, and the inertial measurement unit (IMU) provides motion data such as acceleration and angular velocity. The two generate the attitude information of the drone during flight through data fusion technology. In general, the output data of the positioning and measurement device includes: The three-dimensional spatial position of the drone P(x,y,z), where x, y, and z are geographic coordinates; The attitude parameters of the UAV include the yaw angle θ, the pitch angle φ, and the roll angle ψ.

[0024] The control method of the multi-axis gimbal adopts a closed-loop feedback control algorithm, which achieves accurate positioning of the camera by adjusting the pitch and yaw angles of the gimbal in real time. Specifically, the control logic of the gimbal includes the following steps: Get the initial direction vector D(x,y,z) of the target shooting position; Calculate the angle α between the current gimbal attitude vector and the target direction vector; Dynamically adjust the pitch and yaw angles of the gimbal according to the angle α to align the camera direction vector with the target direction vector; The final output angle of the gimbal after adjustment (θ c ,φ c ) to guide the camera to complete the shot.

[0025] In one possible implementation, the system is combined with a LiDAR sensor to further improve the collection efficiency. LiDAR is used to capture the three-dimensional point cloud data around the building and synchronize it with the image data to ensure the accuracy of subsequent modeling. Specifically, the data output by LiDAR contains the spatial position Q(x i ,y i ,z i ), where i represents the point number in the point cloud, x i ,y i ,z i is the three-dimensional coordinate of the point.

[0026] As an option, in order to ensure the accuracy of the collected data, the data collection module also includes a real-time verification function. The real-time verification function analyzes the resolution, brightness distribution and texture continuity of the image data to determine whether re-collection is required. For example, when the image quality is lower than the set threshold, the system will automatically issue a re-shooting instruction and complete the supplementary collection after adjusting the shooting parameters.

[0027] In another embodiment, the system also supports the synchronization of timestamps of collected point cloud and image data. In complex environments, timestamp synchronization can be used to associate image data with point cloud data to ensure that the spatial matching relationship between the two is not affected by aircraft vibration or delay. The specific synchronization method is as follows: Image Data I t and point cloud data P t Perform time stamp recording; Comparison I t With P t If |ΔT| is less than the set threshold, the data is considered synchronized, otherwise interpolation compensation is performed; Finally, the matching image and point cloud combination (I t ,P t ) for subsequent modeling.

[0028] To meet the needs of data collection in different environments, the components of the data collection module can be flexibly configured. For example, in an environment with more obstructions, a higher-precision camera and LiDAR combination can be preferred. In areas with complex terrain changes, the system can optimize the flight trajectory of the drone through the joint positioning function of GPS and IMU to reduce the error of data collection.

[0029] The mission planning module is used to generate the flight mission plan of the UAV according to the characteristics of the target area; The mission planning module is an important part of the present invention. Its main function is to dynamically generate the flight mission plan of the UAV according to the geographical characteristics, building distribution and environmental obstacle information of the target area. This module directly affects the flight efficiency and acquisition accuracy of the UAV, ensuring the efficient execution of the mission. In general, this module combines multiple algorithms to generate the optimal flight path and mission parameters by comprehensively analyzing the path cost, terrain undulations and obstacle distribution. The mission planning module plays a connecting role in the entire technical solution and provides necessary support for subsequent data collection.

[0030] In this embodiment, the mission planning module realizes comprehensive planning of the UAV flight mission based on the global path planning algorithm and dynamic adjustment strategy. The core goal of the global path planning algorithm is to minimize the total cost of the flight path. The total cost is composed of the flight distance, terrain height change, and obstacle detour cost. The specific formula is expressed as: Ctotal =C dist +C height +C obs C height =w h ·|h i+1 -h i | Among them, C total is the total cost of the flight path, which is the optimization goal of path planning and represents the sum of the costs of the UAV from the starting point to the end point; C dist is the flight distance cost, which reflects the cost of the flight path length of the UAV from the starting point to the end point; C height is the terrain height change cost, which indicates the additional cost of adjusting the UAV's flight altitude due to terrain fluctuations; C obs is the obstacle detour cost, which indicates the additional cost due to the need to detour to avoid obstacles; w h is the weight coefficient of height change, which is used to adjust the influence of height change on the total cost; h i is the height value of the current node i, indicating the vertical height coordinate of the terrain; h i+1 It is the height value of the target node i+1, indicating the terrain height of the next node in the path.

[0031] The mission planning module establishes a path search space by rasterizing the target area. Rasterization is to divide the target area into equidistant two-dimensional grid cells, and the height value of each cell is determined by terrain data. The search priority of the target point is calculated based on the following formula: P i =G i +H i Among them, P i is the path priority of the current node i, indicating the comprehensive cost from the starting point to the end point passing through node i; G i is the actual cost from the starting point to node i, including all the cumulative costs of known paths; H i is the estimated cost from node i to the target point, which is used to predict the potential cost from the current node to the end point.

[0032] The mission parameters generated by the mission planning module include: The three-dimensional coordinates (x, y, z) of the starting and ending points of the drone; The spatial position P of each node in the flight path i (x i ,y i ,z i ), where i is the path node number; The flight speed of the drone v iAnd the height change Δh are used to adjust the flight attitude.

[0033] Specifically, the mission planning module not only generates the initial path, but also supports dynamic adjustment of the flight mission. For example, when the drone detects a new obstacle during flight, the module immediately updates the obstacle information and recalculates the optimal path to bypass the obstacle. The dynamic adjustment process is based on an incremental update strategy, which only replans the affected path segments, thereby reducing the workload of repeated calculations.

[0034] In general, to ensure the effectiveness of mission planning in complex terrain, the mission planning module combines the optimization processing technology of elevation map data. The elevation map represents the terrain characteristics of the target area in the form of a three-dimensional matrix: H(x,y)={z ij},i=1,…,N,j=1,…,M Among them, H(x,y) is the elevation map of the target area, which represents the three-dimensional distribution of terrain height; ij Represents the height value of position (i, j), N and M are the number of rows and columns after rasterization.

[0035] The module calculates the gradient change, analyzes the complexity of the terrain, and adjusts the weight in path planning. The specific method for calculating the gradient change is as follows: Among them, Grad(x,y) is the gradient value of the position (x,y), which indicates the slope of the terrain at that position; is the rate of change of the elevation map in the x direction (horizontal direction); is the rate of change of the elevation map in the y direction (vertical direction); Areas with larger gradient values ​​are marked as high-cost paths, thus guiding the drone to choose a gentle path.

[0036] As an improvement, the module also supports multi-objective mission planning. In a multi-objective mission scenario, the mission planning module sorts multiple target points and generates the shortest flight path covering the whole world according to the priority. The specific algorithm is based on the optimization model of the traveling salesman problem (TSP), combined with the dynamic programming method, to calculate the optimal path between each target point in turn.

[0037] After the mission planning is completed, the module generates a 3D map containing the flight path, terrain height and obstacle distribution. The operator can view the path details through the system interface and adjust the target point or flight parameters according to the mission requirements.

[0038] A data processing module, used to receive and process data collected by the drone; The data processing module is one of the core technologies of the present invention. Its function is to store, verify, compress and transmit the image data and spatial information collected by the drone to ensure the integrity and efficiency of the data in the subsequent modeling process. Generally, the data that the data processing module needs to process includes multi-angle image data, point cloud information and geographic positioning data. This module takes over the output of the data acquisition module in the entire system and provides structured and optimized input data for the 3D modeling module.

[0039] In this embodiment, the data processing module is mainly composed of a block storage unit, a data verification unit, a compression and decompression unit, and a transmission control unit, which work together to complete data reception and processing.

[0040] In some embodiments, the block storage unit performs segmented processing on the collected image and spatial data. Specifically, the data is divided into multiple fixed-size blocks, and the size of each data block can be adjusted according to the task scenario, for example, the default size is 4MB. This block processing method not only facilitates storage management, but also improves the fault tolerance of data transmission. Each data block after segmentation contains the following information: Data content D k , represents the original image or spatial information; The index number k of the data block is used to mark the position of the data block in the entire data set; Checksum C of the data block k , used to verify data integrity.

[0041] The check value is generated by a hash algorithm. One possible implementation is to use the MD5 algorithm to calculate the check value C. k : C k =MD5(D k ) During the transmission process, the verification unit will receive the D k Recalculate C k and compare it with the original checksum. If the checksums are inconsistent, the retransmission mechanism is triggered.

[0042] Specifically, the compression and decompression unit optimizes data transmission. In general, the compression algorithm uses lossless compression to ensure that the transmitted data is consistent with the original data. As an option, the module implements the JPEG-LS compression algorithm for image data compression. The size of the compressed image file S c It is expressed as: S c =S o (1-R) Among them, S ois the original file size, and R is the compression ratio, which is usually controlled within the range of 20%-50%. The image decompression process is completed at the receiving end, and the restored data will be verified for integrity through the verification unit.

[0043] In a possible implementation, the transmission control unit adopts breakpoint resume and failure retransmission strategies to deal with transmission problems in complex network environments. The specific process is as follows: First, record the current transmission progress P t , which is represented by the set of data block numbers that have completed transmission: P t ={k|data block k has been successfully transmitted} Among them, k is the index number of the data block, which is used to mark the position of each data block in the entire data set.

[0044] When the transmission is interrupted, the system automatically switches from P t The transmission is restarted with the missing data block number without resending the completed part.

[0045] After the transmission is completed, the transmission control unit generates a transmission log, which includes the transmission time, completion status and checksum values ​​of all data blocks.

[0046] In another embodiment, the transmission control unit dynamically adjusts the data block size to adapt to changes in network bandwidth. For example, when the network bandwidth decreases, the data block size B is adjusted to α times the original size: B=B o α Among them, B o is the default data block size, α is the bandwidth adjustment coefficient, and the range of α depends on the real-time network status.

[0047] To further improve the transmission efficiency, this module also supports multi-threaded transmission mode. Generally, data blocks are assigned to multiple threads for parallel transmission. For example, if the data block set is {D1, D2, …, D n}, the transmission task assigned to T threads is expressed as: D ij ,i∈{1,…,T},j∈{1,…,n / T} Among them, D ij The i-th thread is responsible for transmitting the j-th data block; i is the thread number, which is used to identify the thread involved in the transmission task; j is the data block number that the thread is responsible for; T is the total number of threads, indicating the parallelism of data transmission; n is the total number of data blocks, indicating the number of data blocks to be transmitted.

[0048] The number of threads T is automatically allocated based on the system hardware resources, and multi-threaded transmission can significantly increase the data upload speed.

[0049] In some embodiments, the data processing module also provides a real-time compression preview function. This function generates a preview image after image compression. p , allowing users to quickly check data quality through thumbnails. Thumbnail size S p It is expressed as: S p =S c β Wherein, β is the thumbnail scale factor, which is usually 0.1-0.2. After the preview image is generated, the user can choose to re-compress or retain the original image. The three-dimensional modeling module is used to generate a three-dimensional model of the target area or target building based on the image data and spatial information provided by the data processing module. The three-dimensional model is constructed based on multi-angle image feature matching, multi-view geometry algorithm and modeling optimization technology; The 3D modeling module is an important part of the present invention. Its core function is to build a 3D model of the target building or area based on the multi-angle images and spatial information collected by the drone. Generally, the 3D modeling module will go through multiple processing steps such as feature point extraction, point cloud generation, model optimization and texture mapping, and finally generate a high-precision and realistic 3D model. This module receives the structured data provided by the data processing module, integrates the image features of multiple perspectives, and generates a complete and robust 3D modeling result.

[0050] In this embodiment, the 3D modeling module first performs multi-view feature point extraction. Specifically, the module uses an improved feature point detection algorithm (such as SIFT or ORB algorithm) to detect key feature points in the image. The feature points in the image are represented by two-dimensional coordinates, which are recorded as: P i ={(x i ,y i )|i=1,2,…,n} Among them, P i is a feature point set, representing all feature points detected in the image; x i and i Represents the pixel coordinates of the feature points in the image, and n is the number of feature points.

[0051] In some embodiments, the module further uses a feature point matching algorithm to establish a point-to-point spatial correspondence between multi-angle images. The matching results are recorded in the form of feature point pairs as follows: M ij ={(P i ,P j )} Among them, M ij is a set of matched feature point pairs, indicating the corresponding feature point matching pairs in the two images, P i and Pj Respectively represent the matching feature points in the two images.

[0052] As an option, the module uses the structure from motion (SfM) method to generate a sparse point cloud model. Specifically, the relative position and posture of the camera are calculated using the feature point matching information between multi-view images. Then, the three-dimensional coordinates of each feature point are determined based on the triangulation principle. The point cloud coordinates are expressed as: Q i ={(X i ,Y i ,Z i )|i=1,2,…,n} Among them, X i ,Y i ,Z i They represent the three-dimensional coordinates of the feature points respectively, and n is the number of points in the three-dimensional point cloud, which is the same as the number of feature points.

[0053] Generally, a sparse point cloud model only contains the spatial locations of key feature points in the image and cannot fully describe the complete geometric structure of the target building. In one possible implementation, the module densifies the sparse point cloud using a multi-view geometry algorithm to generate a dense point cloud model with more details. During the densification process, all pixel information of the image is used to supplement the point cloud data based on the depth map fusion technology.

[0054] The depth map is generated by estimating the distance from the pixel to the camera. The depth value of each pixel is expressed as: Among them, D p Represents the depth value of pixel p, (X p ,Y p ,Z p ) is the three-dimensional coordinate of point p, (X c ,Y c ,Z c ) is the three-dimensional coordinate of the camera center.

[0055] In one possible implementation, the module combines a deep learning network to optimize the point cloud model. The deep learning network is trained to identify and repair missing areas in the point cloud while optimizing surface smoothness and geometric details. The network input includes sparse point clouds and dense point clouds, and the output is an optimized point cloud model.

[0056] The system integration module is used to optimize the storage and result output of modeled data, and supports users to access, interact and apply the generated 3D models through multiple devices.

[0057] The system integration module is a key component of the present invention, which is used to coordinate the data interaction and result management between the aforementioned data acquisition module, data processing module and 3D modeling module. Generally speaking, this module not only needs to support the real-time transmission and processing of data, but also needs to store the generated 3D model in a structured manner, while providing convenient user access and interaction functions. The system integration module can effectively connect all functional modules to ensure the consistency of data flow and the availability of modeling results.

[0058] In this embodiment, the system integration module includes a data storage unit, a user access interface and a multi-terminal interaction module. The data storage unit is mainly used to receive and manage the result data generated by the three-dimensional modeling module.

[0059] The data storage unit adopts a distributed storage architecture. Specifically, the system splits the 3D model data into two parts: geometry data and texture data, which are stored in different nodes. The geometry data is stored in the form of point cloud or mesh, which is specifically represented as: M={(X i ,Y i ,Z i ),F k} Wherein, M is the data set of the 3D model, including point cloud data and mesh patch data; (X i ,Y i ,Z i ) represents the three-dimensional coordinates of the point cloud, F k A mesh patch representing a model.

[0060] Texture data is stored in image format, and each texture image is bound to a mesh patch of geometric data. As an option, the system supports the use of multiple texture formats (such as JPEG, PNG) to store texture maps. After storage is completed, the data storage unit automatically generates a metadata file to record the organizational structure and storage location of the model.

[0061] Specifically, the user access interface provides standardized services through RESTful API. In general, users can access the stored 3D model data through HTTP requests. In some embodiments, the access interface supports block transmission. Specifically, the system divides the 3D model data into several data blocks for transmission according to the size of the requested data. The size of the block can be adjusted dynamically, and the calculation formula is: Among them, B t is the current block size, B0 is the default block size, W n is the current network bandwidth, W max The maximum value of the network bandwidth.

[0062] In one possible implementation, the multi-terminal interaction module is used to realize real-time interaction between users and 3D models. The module supports access from multiple terminal devices, including mobile devices, desktops, and browsers. Interactive functions include model rotation, scaling, sectioning, and measurement. Specifically, the model rotation operation is implemented by combining the rotation angle θ input by the user with the current coordinate system matrix R of the model, and calculating the new coordinate system matrix through the following formula: R′=R·Rot(θ) Among them, Rot(θ) represents the rotation matrix, specifically: As an option, the multi-terminal interaction module also supports the export of 3D model results. When exporting, users can specify the file format and resolution. Generally, the system will generate the corresponding model file according to the specified parameters, and the formats include OBJ and STL. To improve the export efficiency, the module supports parallel processing, which is implemented by splitting the model data into several subsets, each of which is processed by an independent thread.

[0063] In some embodiments, the system integration module also has a real-time analysis function. Specifically, when the user measures the three-dimensional model during the interaction, the system calculates the geometric properties of the model in real time. For example, the formula for measuring volume V is: Where N is the number of model mesh faces; A k , B k , C k They are the three vertices of mesh surface k, and O is the reference origin.

[0064] Please see attached Figure 2 The present invention also provides a three-dimensional house modeling method based on drones, comprising the following steps: S1, collecting multi-angle image data and spatial information of the target building and the surrounding environment through the sensing equipment carried by the drone; Step S1 is to collect image data and spatial information of the target building and the surrounding environment through a variety of sensing equipment carried by the drone. Generally, the sensing equipment includes a high-resolution camera, a laser radar (LiDAR) and a positioning device (such as GPS and IMU). The high-resolution camera is used to capture multi-angle images of the building, and the LiDAR provides three-dimensional point cloud data of the target area to supplement the spatial information.

[0065] In some embodiments, the drone adjusts the camera angle through a preset multi-axis gimbal to obtain image data from the east, south, west, north and directly above the target building. These image data contain the geometric features, surface texture and surrounding environment information of the building. At the same time, LiDAR captures the height and contour information of the surrounding terrain, and the generated point cloud data is synchronized with the image data collected by the camera with a timestamp to ensure the accuracy of spatial matching.

[0066] S2. Generate the flight path of the drone based on the terrain and obstacle information of the target area, and set the shooting task; In step S2, based on the terrain elevation data and obstacle distribution information of the target area, the system generates the flight path of the drone and sets the image shooting task. Generally, path planning is based on a global optimization algorithm, which comprehensively considers the path distance, terrain height change and obstacle detour cost.

[0067] Specifically, the terrain data is rasterized to form a height matrix, and the system generates the optimal path from the starting point with the target point as the end point. After the path is generated, the drone performs the flight mission according to the planned path and collects images at the set shooting points. The shooting parameters (such as camera angle and resolution) of each shooting point are dynamically set according to the characteristics of the building.

[0068] S3, transmit the collected image data and spatial information to the ground station or cloud platform, and support data compression, segment verification and breakpoint continuation during the transmission process; Step S3 is to enable the system to transmit the image data and spatial information to the ground station or cloud platform for further processing after collecting the data. To ensure transmission efficiency and reliability, this step adopts block transmission, data compression and breakpoint resume transmission.

[0069] Generally, image data and spatial information are processed in blocks, and each data block contains the original data content, index number and check value. The check value is generated by a hash algorithm and is used to verify the integrity of the data during transmission. If the transmission is interrupted, the system will continue the transmission from the interruption point to avoid repeated uploading of the completed part.

[0070] S4, using 3D modeling algorithms to extract and match feature points of image data to generate a sparse point cloud model, and generating a dense point cloud model through geometric algorithms and optimization techniques; The purpose of step S4 is that after the data transmission is completed, the system uses a 3D modeling algorithm to process the collected image data. First, feature points in the multi-view images are extracted and matched between the images to generate a preliminary sparse point cloud model. The generation of sparse point clouds relies on structure from motion (SfM) technology, combined with the estimation of the camera position and posture in the image, and the calculation of 3D coordinates through triangulation.

[0071] The sparse point cloud only contains the spatial positions of the significant feature points in the image and cannot fully reflect the geometric structure of the target building. Based on the sparse point cloud, the system generates a dense point cloud model through densification. During the densification process, the depth map information of the image is used to supplement the point cloud data to ensure that the model has richer details.

[0072] S5. Use deep learning models to repair missing parts in point cloud data and optimize surface texture and structural details. For step S5, after generating the dense point cloud model, the system optimizes the point cloud data with the deep learning model, repairs the missing parts, and enhances the surface texture and geometric structure details. Generally, the deep learning model is trained with large-scale building samples, and can automatically identify and complete the missing areas on the building surface due to occlusion or insufficient collection.

[0073] Specifically, this step uses a convolutional neural network (CNN) to complete the local area of ​​the input point cloud data. At the same time, the point cloud surface is smoothed and the geometric discontinuity is corrected through a global optimization algorithm. The optimized point cloud data can more realistically reflect the geometric form of the building.

[0074] S6. The generated 3D model is stored on a cloud platform, and multi-terminal access and interaction functions are provided to users.

[0075] Step S6 is that after the 3D model is generated, the system stores the model through the cloud platform and provides users with multi-terminal access and interactive functions. Generally, the model storage is divided into two parts: geometric data and texture data. The geometric data is stored in the form of point cloud or mesh, and the texture data is saved in the form of image files.

[0076] As an option, users can access the 3D model through the desktop, mobile or browser, and perform interactive operations such as rotation, scaling, sectioning and measurement. The system also supports exporting the model to common file formats (such as OBJ, STL), which is convenient for users to use it in other software.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional house modeling system is constructed based on drone aerial photography in automated survey, characterized in that: include: The data acquisition module is used to collect image data and spatial information of terrain, buildings and surrounding environment through the sensor equipment carried by the drone; The mission planning module is used to generate the flight mission plan of the UAV according to the characteristics of the target area; A data processing module, used to receive and process data collected by the drone; A three-dimensional modeling module, used to generate a three-dimensional model of a target area or a target building according to the image data and spatial information provided by the data processing module, wherein the three-dimensional model is constructed based on multi-angle image feature matching, multi-view geometry algorithm and modeling optimization technology; The system integration module is used to optimize the storage and result output of modeled data, and supports users to access, interact and apply the generated 3D models through multiple devices.

2. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 1, characterized in that: The data acquisition module includes an image acquisition device, a multi-axis gimbal, a GPS, an inertial measurement unit and auxiliary sensing equipment. The multi-axis gimbal controls the image acquisition device to obtain image data of the east, south, west, north and directly above the target building.

3. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 1, characterized in that: The mission planning module calculates the flight path of the UAV by combining the geographic elevation data and obstacle distribution information of the target area using a global planning algorithm, and the planning result is based on the principle of minimizing the path cost.

4. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 3 is characterized in that: The minimization of path cost includes the following factors: The flight distance between the starting point and the target node; The change in terrain height between the current node and the target node; The estimated cost from the target node to the final end point is calculated by weighting the flight distance and altitude change.

5. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 1, characterized in that: The data processing module processes the image data in segments through a block transmission strategy, sets verification information for each segment of data, and supports breakpoint resumption and failure retransmission during the transmission process.

6. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 1, characterized in that: The three-dimensional modeling module includes: A feature point extraction unit, used to extract and match key feature points from multi-angle image data; A point cloud generation unit, used for generating a sparse point cloud model based on the feature points by a multi-view geometry algorithm and gradually densifying it; The optimization unit is used to repair missing parts in sparse point cloud models through deep learning models and optimize model surface texture and structural details.

7. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 6, characterized in that: The optimization unit performs local area completion and global texture enhancement on the input point cloud data based on a deep learning model, and the deep learning model optimizes its modeling capability through multi-view training of building samples.

8. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 1, characterized in that: The system integration module utilizes a cloud platform for data storage, processing and distribution, supports multiple users to access the generated three-dimensional model simultaneously, and allows users to perform dynamic annotation, measurement and result export operations.

9. The system for constructing a three-dimensional house model based on drone aerial photography in automated survey according to claim 8, characterized in that: The export operation includes the following steps: receiving user-selected export options, the options including a target file format, a model resolution, and an output path; Performing format conversion processing on the generated 3D model, converting the data structure of the 3D model from the internal storage format to a universal file format, the file format includes OBJ file and STL file, and retaining the geometric topology information and texture mapping information of the model during the conversion process; Calculate the vertex data, patch data and normal vector data of the 3D model and rearrange the data in a standardized form to ensure the integrity of the model structure, where the vertex data is recorded in the form of point coordinates, the patch data is recorded in the form of polygon definition, and the normal vector data is recorded in the form of a vector in the normal direction; Compress and integrate the texture data, embed the image file of the model surface texture mapping into the export file, and use lossless compression to ensure that the exported texture quality is consistent with that when the model is generated; Calculate the file size and generate a checksum for the file, which is used to verify the integrity of the file during transmission or storage; Save the exported file in the path specified by the user. If the path is not specified, the system default path is used; After the export is completed, an export log is generated, which includes the file name, export time, file format and checksum, and provides users with download or viewing functions.

10. A three-dimensional house modeling method based on drones, applied to a three-dimensional house modeling system based on drone aerial photography in automated surveying as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: The sensor equipment carried by the drone collects multi-angle image data and spatial information of the target building and its surrounding environment; Generate the flight path of the drone based on the terrain and obstacle information of the target area and set the shooting mission; The collected image data and spatial information are transmitted to the ground station or cloud platform, and data compression, segment verification and breakpoint resumption are supported during the transmission process; Use 3D modeling algorithms to extract and match feature points from image data to generate a sparse point cloud model, and use geometric algorithms and optimization techniques to generate a dense point cloud model; Combined with deep learning models to repair missing parts in point cloud data and optimize surface texture and structural details; The generated 3D model is stored through the cloud platform, and provides users with multi-terminal access and interaction functions.

Images