Three-dimensional grid automatic filling method and device for unmanned aerial vehicle point cloud hole repair

Abstract

The application discloses a kind of unmanned aerial vehicle point cloud cavity repair-oriented three-dimensional grid automatic filling method and device, unmanned aerial vehicle point cloud cavity repair-oriented three-dimensional grid automatic filling method includes: obtaining point cloud dataset, and according to preset spatial resolution, point cloud coverage is divided into regular grid array of equal size;Each laser point in point cloud dataset is mapped to corresponding grid cell according to its plane coordinate;Iterative statistics the number of point clouds in each grid cell, and the grid cell with less number of point clouds than the number threshold is marked as a hollow unit, and adjacent hollow units are merged to form a hollow region;For each hollow region, outwardly expand the grid cell of a predetermined number of adjacent layers to form a continuous neighborhood, and the neighborhood characteristics of the neighborhood are counted, which represent the entity condition of the target area corresponding to the neighborhood grid;Based on the area of each hollow region and the corresponding neighborhood characteristics, at least one filling strategy is selected to fill the grid attribute value of the hollow region.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method and apparatus for automatic three-dimensional mesh filling for repairing holes in point clouds of unmanned aerial vehicles (UAVs). Background Technology

[0002] Due to its high efficiency, high precision, and strong penetration, UAV laser point cloud data acquisition technology has been widely applied in fields such as topographic mapping, urban 3D modeling, power line inspection, and geological hazard assessment. Its typical processing flow includes: point cloud acquisition (obtaining spatial coordinates and reflection intensity information through LiDAR sensors), multi-view point cloud registration (resolving attitude and position deviations), point cloud filtering (separating ground points from non-ground points), point cloud classification (identifying vegetation, buildings, roads, and other land cover types), and finally, generating a structured digital surface model (DSM) or a digital elevation model (DEM) through meshing.

[0003] However, in practical applications, meshed point cloud models often exhibit localized sparse regions or even voids. The main reasons include: physical occlusion (e.g., buildings obscuring the ground or dense vegetation blocking lower terrain layers, preventing lasers from reaching the target area); reflection failure (e.g., strong absorptive or specular reflective surfaces like water bodies and glass curtain walls causing missing echo signals); equipment limitations (e.g., improper settings of parameters such as drone flight altitude, scanning angle, and point cloud density thresholds creating data blind spots); and motion errors (e.g., gaps in point cloud stitching caused by flight attitude fluctuations). These voids significantly impact downstream products, for example, reducing DEM / DSM accuracy, causing terrain distortion or faulting due to missing elevation information in void areas, and resulting in inaccurate slope analysis; causing structural breaks in the 3D model, with non-physical interruptions in building facades or terrain contours affecting the reliability of urban planning and simulation; and propagating errors in derivative products, such as contour distortion generated in void areas and deviations in flood inundation analysis results.

[0004] Existing solutions mostly employ single interpolation methods (such as inverse distance weighting and kriging), texture mapping inpainting, and manual interactive inpainting. These solutions all have obvious limitations. For example, the point cloud density and terrain complexity vary greatly in different regions, making it difficult for a single interpolation algorithm to balance accuracy and efficiency. Texture mapping inpainting is only suitable for restoring visual coherence and cannot reconstruct accurate geometric information. Manual interactive inpainting is inefficient and highly subjective, making it difficult to meet the needs of large-scale engineering projects.

[0005] Therefore, there is an urgent need for a point cloud processing method that can adaptively select filling strategies based on the void situation, so as to achieve high-precision and automated repair while ensuring the continuity of terrain and the integrity of ground features, thereby improving the industrial-grade reliability of point cloud derivative products. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a method, apparatus, computing device and storage medium for automatic three-dimensional mesh filling for UAV point cloud hole repair that overcomes or at least partially solves the above problems.

[0007] According to one aspect of the present invention, a method for automatic 3D mesh filling for repairing holes in point clouds of unmanned aerial vehicles (UAVs) is provided, executed in a computing device. The method includes: performing laser scanning on a target area using an airborne laser scanning system of the UAV to generate a point cloud dataset of the target area; dividing the point cloud coverage area into a regular grid array of equal size according to a preset spatial resolution; mapping each laser point in the point cloud dataset to a corresponding grid cell according to its planar coordinates; traversing and counting the number of point clouds in each grid cell, marking grid cells with a number of point clouds less than a threshold as hole cells, and merging adjacent hole cells to form a hole region; for each hole region, expanding outward by a preset number of adjacent grid cells to form a continuous neighborhood, and counting the neighborhood features of the neighborhood, the neighborhood features representing the entity situation of the target area corresponding to the neighborhood grid; and selecting at least one filling strategy based on the area of ​​each hole region and the corresponding neighborhood features to fill the grid attribute values ​​of the hole region.

[0008] Optionally, in the automatic three-dimensional mesh filling method for repairing point cloud voids for UAVs according to the present invention, the entity information includes entity type, elevation standard deviation, and flatness, and the entity type includes natural entities and man-made entities.

[0009] Optionally, in the automatic 3D mesh filling method for repairing holes in UAV point clouds according to the present invention, the filling strategies include planar interpolation filling method, neighborhood weighted interpolation method, feature matching filling method, multi-scale hierarchical filling method, and constraint optimization filling method.

[0010] Optionally, in the automatic three-dimensional mesh filling method for repairing point clouds of unmanned aerial vehicles according to the present invention, based on the area of ​​each hole region and the corresponding neighborhood features, at least one filling strategy is selected to fill the mesh attribute values ​​of the hole region, including: if the area of ​​the hole region is greater than the area threshold, then at least the multi-scale layered filling method and the constraint optimization filling method are activated; if the neighborhood features indicate that the target region contains a first sub-region of artificial entities, then at least the feature matching filling method should be activated to fill the first sub-region; if the neighborhood features indicate that the target region contains a second sub-region with an elevation standard deviation of less than 0.5m and a flatness that meets the slope angle of less than 5°, then at least the planar interpolation filling method is activated to fill the second sub-region; if the neighborhood features indicate that the target region contains a third sub-region with an elevation standard deviation of not less than 0.5m and less than 2m and a flatness that meets the slope angle of not less than 5° and less than 15°, then at least the planar interpolation filling method is activated to fill the third sub-region.

[0011] Optionally, in the automatic three-dimensional mesh filling method for repairing point clouds of unmanned aerial vehicles according to the present invention, the multi-scale layered filling method includes: determining the overall outline and main structure of the hole region based on the geometry of the neighborhood, and updating the hole region; redetermining the neighborhood features based on the updated hole region, and selecting an appropriate filling strategy according to the determined neighborhood features to fill the mesh attribute values ​​of the updated hole region.

[0012] Optionally, in the automatic three-dimensional mesh filling method for repairing point cloud voids for UAVs according to the present invention, the constraint optimization filling method includes: applying geometric continuity and feature consistency constraints during the filling process, and filling the void region through an optimization algorithm.

[0013] Optionally, the automatic 3D mesh filling method for repairing holes in UAV point clouds according to the present invention further includes: preprocessing the point cloud dataset of the target area, wherein the preprocessing includes at least denoising, registration and classification.

[0014] According to another aspect of the present invention, a three-dimensional mesh automatic filling device for repairing holes in point clouds of unmanned aerial vehicles (UAVs) is provided, residing in a computing device. The device includes: a scanning module adapted to perform laser scanning on a target area using an airborne laser scanning system of the UAV to generate a point cloud dataset of the target area; a partitioning module adapted to divide the point cloud coverage area into a regular grid array of equal sizes according to a preset spatial resolution; a mapping module adapted to map each laser point in the point cloud dataset to a corresponding grid cell according to its planar coordinates; a marking module adapted to traverse and count the number of point clouds in each grid cell, marking grid cells with a point cloud count less than a threshold as hole cells, and merging adjacent hole cells to form a hole region; a statistics module adapted to expand outwards by a preset number of adjacent grid cells to form a continuous neighborhood for each hole region, and statistically analyze the neighborhood characteristics of the neighborhood, where the neighborhood characteristics characterize the entity situation of the target area corresponding to the neighboring grid; and a filling module adapted to select at least one filling strategy based on the area of ​​each hole region and the corresponding neighborhood characteristics to fill the grid attribute values ​​of the hole region.

[0015] According to another aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the methods described above.

[0016] According to another aspect of the present invention, a readable storage medium storing program instructions is provided, which, when read and executed by a computing device, causes the computing device to perform the method described above.

[0017] According to the present invention, through multi-scale neighborhood feature analysis, the neighborhood features exhibited by the surrounding area of ​​the cavity at different scales are extracted in depth. This enables the method to intelligently select or weightedly fuse the most suitable filling strategy (e.g., linear interpolation is selected for flat areas to ensure efficiency, while Delaunay triangulation based on feature matching or constraints is used for areas with obvious structural features to maintain sharp edges). This adaptive capability of "tailoring strategies to the cavity" ensures that the filling results maintain a high degree of continuity and consistency with the surrounding real environment in terms of geometric structure, greatly improving the product quality of digital elevation models (DEM) and digital surface models (DSM).

[0018] Furthermore, the entire filling process requires no manual intervention, achieving fully automated pipeline processing. The method can intelligently identify the extent of holes, adaptively determine the causes of holes (such as occlusion or reflection), and autonomously select or fuse the optimal filling strategy (such as plane fitting, feature matching, etc.). This significantly reduces the human and time costs of data processing, solves the problems of low efficiency and strong subjectivity in traditional manual repair methods, and makes the large-scale, engineering application of automated 3D mesh filling projects for UAV point cloud hole repair possible.

[0019] This paper proposes an automatic meshing method for UAV laser point clouds based on multi-scale hole detection and adaptive interpolation fusion. Through hole detection, multi-scale neighborhood analysis, and adaptive interpolation algorithm selection and fusion, high-precision automatic filling of sparse / missing regions is achieved.

[0020] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0022] Figure 1 A block diagram of the physical components (i.e., hardware) of the computing device 100 is shown;

[0023] Figure 2 A flowchart of an automatic 3D mesh filling method 200 for repairing holes in UAV point clouds according to an embodiment of the present invention is shown;

[0024] Figure 3A schematic diagram of the structure of a three-dimensional mesh automatic filling device 300 for repairing point cloud voids of unmanned aerial vehicles according to an embodiment of the present invention is shown. Detailed Implementation

[0025] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0026] UAV laser point cloud data is widely used in fields such as topographic mapping and urban modeling. Common processing steps include point cloud acquisition, registration, filtering, classification, and meshing. After meshing, some areas may have sparse or hollow point clouds due to occlusion, reflection, equipment limitations, etc., which affects the quality of products such as DEM / DSM.

[0027] Existing hole-filling methods include direct grid statistical methods, simple interpolation methods, inverse distance weighted (IDW) interpolation methods, Kriging interpolation methods, and Poisson reconstruction. Different filling schemes are suitable for different hole conditions (see specific examples below), but most existing hole-filling schemes use a single filling method and do not adaptively select the filling method according to the hole conditions.

[0028] A specific example will be provided to illustrate each filling method in detail:

[0029] 1. Direct grid statistical method

[0030] Technical features: The elevation / intensity and other attributes of the point cloud falling into each grid cell are statistically analyzed, and there is no data in the hole area.

[0031] Implementation method: Direct assignment using statistical methods, without interpolation.

[0032] Application scope: Areas with dense point clouds and few holes.

[0033] 2. Simple interpolation method

[0034] Technical characteristics: such as nearest neighbor and linear interpolation, filling the hole area with the surrounding point cloud attributes, with limited accuracy.

[0035] Implementation method: Simple inference based on spatial distance.

[0036] Application scope: Small, sparse areas.

[0037] 3. Inverse Distance Weighted (IDW) Interpolation

[0038] Technical features: Fills holes based on distance-weighted surrounding point cloud attributes, suitable for small, sparse areas.

[0039] Implementation method: The closer the distance, the greater the weight.

[0040] Application: Localized cavity filling.

[0041] 4. Kriging interpolation

[0042] Technical features: Based on spatial correlation modeling, suitable for interpolation of large-scale continuous surfaces, but computationally complex.

[0043] Implementation method: Spatial statistical modeling.

[0044] Application scope: Large-scale continuous ground surface.

[0045] 5. Poisson Reconstruction

[0046] Technical features: Reconstructs continuous surfaces based on point cloud normal vectors, suitable for high-density point clouds, but with limited effectiveness when there are large holes.

[0047] Implementation method: Surface reconstruction algorithm.

[0048] Application scope: High-density point cloud areas.

[0049] To address the problems existing in the prior art, this invention proposes a solution. This application presents an automatic 3D mesh filling method for repairing holes in UAV point clouds. This method, through multi-scale neighborhood feature analysis, deeply extracts the neighborhood features exhibited by the area surrounding the hole at different scales. This allows the method to intelligently select or weightedly fuse the most suitable filling strategy (e.g., linear interpolation for flat areas to ensure efficiency, and Delaunay triangulation based on feature matching or constraints to maintain sharp edges for areas with obvious structural features). This adaptive "hole-specific" capability ensures that the filling result maintains a high degree of continuity and consistency with the surrounding real environment in terms of geometric structure, greatly improving the product quality of digital elevation models (DEMs) and digital surface models (DSMs).

[0050] The automatic 3D mesh filling method for repairing holes in UAV point clouds can be executed in computing device 100. Computing device 100 can be implemented as a server, such as an application server or web server; it can also be implemented as a desktop computer, laptop computer, processor chip, tablet computer, etc., but is not limited to these. Figure 1A block diagram of the physical components (i.e., hardware) of a computing device 100 is shown. In a basic configuration, the computing device 100 includes at least one processing unit 102 and system memory 104. According to one aspect, depending on the configuration and type of the computing device, the system memory 104 includes, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories.

[0051] According to one aspect, system memory 104 includes operating system 105. System memory 104 also includes program module 150. According to one aspect, operating system 105, for example, is adapted to control the operation of computing device 100. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. Figure 1 The basic configuration is illustrated by the components within the dashed lines 108. According to one aspect, the computing device 100 has additional features or functions. For example, according to one aspect, the computing device 100 includes additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 1 The middle part is shown by removable storage device 109 and non-removable storage device 110.

[0052] As stated above, according to one aspect, a program module 150 is stored in system memory 104. According to one aspect, the program module 150 can be implemented as one or more computer program products. This application does not limit the type of computer program product, and may include, for example, email, word processing applications, spreadsheet applications, database applications, slideshow applications, drawing or computer-aided applications, web browsers, etc. In some embodiments according to this application, computer programs / instructions related to the automatic 3D mesh filling method 200 for UAV point cloud hole repair are encapsulated as a computer program product. When these computer programs / instructions are executed by a processor (i.e., processing unit 102), the automatic 3D mesh filling method 200 for UAV point cloud hole repair according to this application is implemented.

[0053] According to one aspect, examples can be practiced on circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or a microprocessor. For example, it can be practiced via wherein... Figure 1Each or many of the components shown can be implemented as an example by integrating a System-on-a-Chip (SOC) on a single integrated circuit. According to one aspect, such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “burned in”) as a single integrated circuit onto a chip substrate. When operating via the SOC, the functions described herein can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 100. Embodiments of the invention can also be implemented using other techniques capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. Additionally, embodiments of the invention can be implemented within a general-purpose computer or in any other circuit or system.

[0054] According to one aspect, the computing device 100 may also have one or more input devices 112, such as a keyboard, mouse, pen, voice input device, touch input device, etc. It may also include output devices 114, such as a monitor, speaker, printer, etc. The foregoing devices are examples and other devices may also be used. The computing device 100 may include one or more communication connections 116 that allow communication with other computing devices 118, such as printing devices, like printers. Examples of suitable communication connections 116 include, but are not limited to: RF transmitters, receivers, and / or transceiver circuitry; Universal Serial Bus (USB); parallel and / or serial ports.

[0055] As used herein, the term computer-readable medium includes computer storage medium. Computer storage medium can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information (e.g., computer-readable instructions, data structures, or program modules). System memory 104, removable storage device 109, and non-removable storage device 110 are examples of computer storage media (i.e., memory storage). Computer storage media can include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital universal disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computer device 100. According to one aspect, any such computer storage medium can be part of computing device 100. Computer storage media does not include carrier waves or other transmitted data signals.

[0056] According to one aspect, a communication medium is implemented by computer-readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transmission mechanism), and includes any information transmission medium. According to one aspect, the term "modulated data signal" describes a signal having one or more sets of characteristics or altered in a manner that encodes information in the signal. By way of example and not limitation, a communication medium includes wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0057] In an embodiment of the invention, computing device 100 is configured to execute a three-dimensional mesh automatic filling method 200 for UAV point cloud hole repair according to the invention. Computing device 100 includes one or more processors and one or more readable storage media storing program instructions that, when executed by the one or more processors, cause the computing device to execute the three-dimensional mesh automatic filling method 200 for UAV point cloud hole repair according to the embodiment of the invention.

[0058] Figure 2 A flowchart of an automatic 3D mesh filling method 200 for repairing holes in UAV point clouds according to an embodiment of the present invention is shown. Method 200 is adapted to be executed in a computing device (e.g., the aforementioned computing device 100).

[0059] like Figure 2 As shown, the purpose of method 200 is to realize a high-precision automatic filling method for void regions through void detection, multi-scale neighborhood analysis, adaptive filling strategy selection and fusion.

[0060] Method 200 begins with step 202, in which the target area is laser-scanned using an airborne laser scanning system on a UAV to generate a point cloud dataset of the target area.

[0061] This embodiment uses an airborne laser scanning (LiDAR) system mounted on an unmanned aerial vehicle (UAV) to perform automated flight scanning of the target area along a preset route.

[0062] In some embodiments, the airborne laser scanning system integration includes:

[0063] LiDAR sensor: emits high-frequency laser pulses and receives the echo signals returned from the ground surface or the surface of objects;

[0064] High-precision positioning and attitude determination system (POS): Integrating a Global Navigation Satellite System (GNSS) receiver and an inertial measurement unit (IMU), it is used to synchronously record the UAV's three-dimensional spatial position (X, Y, Z) and attitude parameters (heading angle, pitch angle, roll angle) at the moment of each laser pulse emission in real time.

[0065] Control and storage unit: coordinates sensor operation and stores raw scan data.

[0066] During flight scanning, the flight path, altitude, speed of the UAV, and the scanning mode (such as reciprocating scanning, spiral scanning), scanning frequency, and scanning angle of the lidar are controlled by a preset flight plan to ensure the integrity and overlap of the point cloud data coverage and meet the preset point cloud density requirements.

[0067] In some embodiments, after scanning is complete, a point cloud dataset of the target region is generated through the following calculation steps:

[0068] Data Calculation: Based on the precise spatiotemporal reference provided by the POS system, the laser ranging data and platform pose data are jointly calculated. The three-dimensional coordinates of each laser footpoint in the geodetic coordinate system are calculated using the following formula:

[0069] X_point = X_drone + R * (laser vector)

[0070] (Where, R is the rotation matrix calculated from the IMU attitude angle, which transforms the measurement vector in the laser sensor coordinate system to the geodetic coordinate system).

[0071] Point cloud formation: All calculated laser foot coordinates, echo intensity information, echo count, and other information are integrated to form an initial point cloud dataset containing a massive number of three-dimensional spatial points and their attributes.

[0072] In some embodiments, step 202 further includes preprocessing the point cloud dataset of the target region, the preprocessing including at least denoising, registration and classification.

[0073] Specifically, point cloud denoising involves removing obvious flight noise and outliers; point cloud registration, if multiple flights and multiple routes are used for scanning, involves high-precision registration of point cloud data from different routes using algorithms such as Iterative Closest Point (ICP) to unify them into the same coordinate system; and point cloud classification is the process of assigning a specific semantic label to each point in the point cloud dataset based on the geometric features, reflection intensity, color, and other information of the point cloud.

[0074] In addition, preprocessing may include point cloud thinning / densification, and uniformizing the point cloud density according to application requirements.

[0075] The final output is a high-precision point cloud dataset of the target area with georeferenced features, providing a data foundation for subsequent processing.

[0076] After obtaining the point cloud dataset, proceed to step 204, where the point cloud coverage area is divided into a regular grid array of equal size according to the preset spatial resolution.

[0077] To transform unordered, unstructured raw point cloud data into a structured geospatial data model to support subsequent hole detection and quantitative analysis, gridding processing is required. The processing steps are as follows:

[0078] First, determine the grid space range and resolution:

[0079] Spatial bounding: Calculate the maximum and minimum coordinate values ​​(X_min, X_max, Y_min, Y_max) of the point cloud dataset on the horizontal plane (XY plane) to determine a rectangular geospatial bounding box that can completely enclose all point clouds.

[0080] Mesh resolution setting: Define the size of a single mesh cell based on the preset spatial resolution or accuracy requirements of the final product (such as DEM / DSM). This resolution is usually expressed in meters per pixel (e.g., 0.1m, 0.5m, 1.0m), and its value needs to be determined comprehensively based on the original point cloud density, application scenario requirements, and computing resources.

[0081] It should be noted that the choice of resolution is crucial: too high (too small a value) will result in too few points in the grid cells and data sparsity; too low (too large a value) will result in loss of terrain details.

[0082] Construct a regular grid array: Based on the above spatial range and resolution, create a regular two-dimensional grid array on the horizontal plane.

[0083] The array consists of M rows and N columns of grid cells of identical size.

[0084] The number of rows (M) and columns (N) can be calculated in the following way:

[0085] M = ceil( (Y_max - Y_min) / Cell_Size )

[0086] N = ceil( (X_max - X_min) / Cell_Size )

[0087] (Where ceil is the floor function to ensure full coverage).

[0088] Subsequently, in step 206, each laser point in the point cloud dataset is mapped to its corresponding grid cell according to its planar coordinates.

[0089] For each laser point in the point cloud dataset, calculate the row index (i) and column index (j) of its corresponding grid cell using linear mapping based on its planar coordinates (X, Y):

[0090] i = floor( (Y - Y_min) / Cell_Size )

[0091] j = floor( (X - X_min) / Cell_Size )

[0092] (where floor is the floor function).

[0093] Each laser point and its attributes (such as Z coordinate, intensity, etc.) are assigned or "dropped" into the corresponding calculated mesh cell.

[0094] Thus, the unstructured point cloud is effectively organized into a structured two-dimensional grid data structure, with each grid cell becoming a container containing zero or one or more laser points.

[0095] After mapping the laser points to the grid cells, proceed to step 208, traverse and count the number of point clouds in each grid cell, mark the grid cells with a number of point clouds less than the threshold as hollow cells, and merge adjacent hollow cells to form a hollow region.

[0096] For example, check each grid cell one by one: traverse each grid cell (e.g., each 1m×1m cell), count the number of point clouds in that cell, and if the number of point clouds is less than a threshold (e.g., less than 4 points), mark it as an "empty cell".

[0097] Merge adjacent void cells: Check the 8 adjacent cells of each void cell. If the adjacent cells are also void cells, merge them into a void region, eventually forming a void region composed of multiple consecutive void cells.

[0098] If none of the eight adjacent cells of a void cell are void cells, then the void cell is considered a void region.

[0099] In step 210, for each void region, a predetermined number of adjacent grid cells are extended outward to form a continuous neighborhood. The neighborhood characteristics of this neighborhood are statistically analyzed, and the neighborhood characteristics represent the entity situation of the target region corresponding to the neighborhood grid. The neighborhood of a void region refers to a complete ring of grid cells surrounding the void boundary. The neighborhood range is N grid cells (usually N=1-3 layers) extending outward from the void boundary.

[0100] The physical characteristics include the type of entity, the standard deviation of elevation, and the flatness. Entity types include natural entities and man-made entities. Natural entities refer to naturally formed entities, such as mountains, lakes, and rivers, while man-made entities refer to entities constructed by humans, such as buildings, roads, and cars.

[0101] In a specific example, to assign a suitable filling strategy to each void region, a quantitative analysis of its surrounding environment is required. This step performs the following operations for each individual void region identified in the previous steps:

[0102] Constructing a multi-scale analysis neighborhood:

[0103] Using the bounding rectangle or minimum convex hull of the void region as a reference, a preset K-layer grid of cells (K is an integer, such as 1, 2, 3) is buffered outwards to form a continuous analysis neighborhood. This "adjacent layer number" K is a key hyperparameter that can be empirically set or adaptively calculated based on the average density of the original point cloud.

[0104] Multi-scale analysis concept: This operation essentially constructs a multi-scale analysis window. Smaller K values ​​(such as layer 1) define neighborhoods that focus on capturing local details and immediate context at the edges of holes; while larger K values ​​can include contextual information over a wider area, helping to understand overall terrain trends or the structure of large features. The system can process neighborhoods at multiple K values ​​in parallel to obtain a more comprehensive feature description.

[0105] Statistical calculation of neighborhood feature vectors:

[0106] The properties of all non-hollow grid cells (i.e., grids containing effective point clouds) in the analysis neighborhood are statistically calculated to generate a multi-dimensional feature vector that can digitally represent the physical entities in the neighborhood.

[0107] Neighborhood features mainly include, but are not limited to, the following categories:

[0108] 1. Geometric features

[0109] Point cloud density: the distribution density of points within a neighborhood.

[0110] Normal vector: The direction of the normal vector of a local surface.

[0111] Curvature: The degree of surface curvature (Gaussian curvature, mean curvature)

[0112] Flatness: The degree of flatness of a local area.

[0113] 2. Spatial characteristics

[0114] Inter-point distance distribution: statistical characteristics of distances between points in a neighborhood

[0115] Spatial distribution pattern: the regularity of point arrangement

[0116] Neighborhood shape: the geometric shape characteristics of the neighborhood.

[0117] 3. Intensity characteristics

[0118] Laser echo intensity: Reflection intensity value

[0119] Intensity gradient: Intensity variation trend

[0120] Strength consistency: uniformity of strength within a neighborhood

[0121] 4. Multi-scale features

[0122] Feature changes in neighborhoods with different radii

[0123] Scale stability: the consistency of features across multiple scales

[0124] Elevation distribution characteristics: Calculate the statistics of the Z-coordinates of all valid points, such as mean elevation, standard deviation of elevation (characterizing surface roughness), difference between maximum and minimum elevation values ​​(characterizing elevation difference), and elevation quantiles (e.g., 25%, 50%, 75th percentiles). These characteristics are used to distinguish between flat land, slopes, and complex terrain.

[0125] Intensity characteristics: Calculating the average value and standard deviation of the echo intensity of all valid points helps to distinguish ground features with different materials (such as asphalt pavement vs. grass, glass curtain wall vs. concrete wall).

[0126] Geometric and normal vector characteristics:

[0127] Planarity / Roughness: By fitting a local plane to the point cloud within its neighborhood, the distance distribution from the point cloud to the fitted plane is calculated, thus assessing the flatness of the region. Flat regions have high planarity, while vegetated regions have high roughness.

[0128] Normal Vector Orientation: Calculates the direction of the average normal vector of the point cloud in the neighborhood (such as the angle with the horizontal plane), which is used to determine whether it is a horizontal plane (such as the ground), a vertical plane (such as the building facade), or an inclined plane (such as the roof).

[0129] Density characteristics: Calculate the number of laser points per unit area within the neighborhood. The density of point clouds is related to the type of land cover and the degree of occlusion.

[0130] Based on distribution characteristics: Analyze the histogram distribution, skewness, and kurtosis of elevation and intensity values ​​to capture more complex data patterns.

[0131] Feature representation and semantic inference:

[0132] The multi-dimensional feature vectors calculated above work together to form a digital representation of the real-world entities corresponding to this neighborhood. By analyzing these features, the system can make high-confidence semantic inferences about the land cover types that should exist in this hollow area.

[0133] For example, a neighborhood with a low elevation standard deviation, high flatness, and a horizontal normal vector strongly indicates that the corresponding entity is a flat surface.

[0134] A neighborhood with a mid-elevation standard deviation, a specific directional normal vector, and a high intensity value may indicate that it corresponds to a building facade or roof.

[0135] A neighborhood with an extremely high elevation standard deviation, low flatness, and low intensity may indicate that it corresponds to dense vegetation.

[0136] Ultimately, the feature vector generated for each hole region will serve as the core input for the next stage of the "adaptive selection and filling strategy".

[0137] Finally, in step 212, based on the area of ​​each void region and the corresponding neighborhood features, at least one filling strategy is selected to fill the grid attribute values ​​of the void regions.

[0138] In some embodiments, the filling strategies include planar interpolation filling, neighborhood weighted interpolation, feature matching filling, multi-scale hierarchical filling, and constrained optimization filling.

[0139] In some embodiments, step 212 includes the following sub-steps:

[0140] If the area of ​​the void region exceeds an area threshold, at least the multi-scale hierarchical filling method and the constraint-optimized filling method are activated. The multi-scale hierarchical filling method involves: determining the overall outline and main structure of the void region based on the geometry of its neighborhood, updating the void region, redetermining the neighborhood features based on the updated void region, and selecting an appropriate filling strategy based on the determined neighborhood features to fill the updated void region's mesh attribute values. The constraint-optimized filling method involves: applying geometric continuity and feature consistency constraints during the filling process, and filling the void region using an optimization algorithm.

[0141] If the neighborhood features indicate that the target region contains a first sub-region of artificial entities, then the feature matching fill method should at least be activated to fill the first sub-region.

[0142] If the neighborhood feature indicates that the target area contains a second sub-region with an elevation standard deviation of less than 0.5m and a flatness that meets the slope angle requirement of less than 5°, then at least the planar interpolation filling method should be activated to fill the second sub-region.

[0143] If the neighborhood feature indicates that the target area contains a third sub-region with an elevation standard deviation of not less than 0.5m and less than 2m, and a flatness that meets the slope angle requirement of not less than 5° and less than 15°, then at least the planar interpolation filling method should be activated to fill the third sub-region.

[0144] In a specific example, feature similarity matching fill: just like a doctor performing a skin graft, first analyze the "texture features" (geometric features, intensity features, etc.) around the missing area, and then find the region with the most similar features in the complete point cloud data and use it as the "donor" to fill the missing part.

[0145] Multi-scale layered filling: Using a pyramid algorithm similar to that in image processing, the overall outline and main structure are first determined with a large neighborhood range, and then the neighborhood range is gradually reduced to fill in the details, just like building the framework of a house first and then decorating the details.

[0146] Weighted interpolation fill: Different weights are assigned based on the reliability of the surrounding data (assessed through feature stability). Data with higher reliability has a greater weight in the interpolation calculation, just as expert opinions have a greater weight in decision-making.

[0147] Constraint-optimized filling: During the filling process, geometric continuity and feature consistency constraints are set, and optimization algorithms are used to ensure that the filling result conforms to physical laws and transitions naturally with the surrounding environment, avoiding the "patch" effect.

[0148] For example, if the area of ​​a certain void region is greater than the area threshold, and the neighborhood features indicate the presence of man-made entities, and the elevation standard deviation of the natural entity portion is small and the terrain is flat, then a multi-scale layered filling method is first used to construct the main structure of the void region. Then, a constraint optimization filling method is used to set geometric continuity and feature consistency constraints. Subsequently, feature similarity matching is used to fill the corresponding parts of the man-made entities, and interpolation is used to fill the natural entity portions.

[0149] In this embodiment, the choice of which filling strategy to use depends mainly on the geometric features of the neighborhood, and then the optimal filling method is selected based on the geometric features.

[0150] Planar interpolation method (small elevation changes within the neighborhood, flat terrain), specifically, the standard deviation of elevation within the neighborhood is <0.5 meters, the slope change is <5 degrees, and the curvature is close to 0.

[0151] The neighborhood weighted interpolation method (the terrain within the neighborhood has some undulations but is relatively regular) specifically has an elevation standard deviation of 0.5-2 meters and a slope variation of 5-15 degrees within the neighborhood.

[0152] Feature matching and filling method (complex terrain with obvious structural features in the neighborhood) specifically refers to the presence of artificial structures such as buildings and roads in the neighborhood, with complex geometric features.

[0153] Multi-scale layered filling method (large area voids with diverse neighborhood characteristics), specifically, voids with an area > 100 grid cells and containing multiple land cover types within the neighborhood.

[0154] The method provided by this invention deeply extracts the neighborhood features exhibited by the area surrounding the cavity at different scales through multi-scale neighborhood feature analysis. This allows the method to intelligently select or weightedly fuse the most suitable filling strategy (e.g., linear interpolation is selected for flat areas to ensure efficiency, while Delaunay triangulation based on feature matching or constraints is used for areas with obvious structural features to maintain sharp edges). This adaptive "cavity-specific" capability ensures that the filling result maintains a high degree of continuity and consistency with the surrounding real environment in terms of geometric structure, greatly improving the product quality of digital elevation models (DEMs) and digital surface models (DSMs).

[0155] Furthermore, the entire filling process requires no manual intervention, achieving fully automated pipeline processing. The method can intelligently identify the extent of holes, adaptively determine the causes of holes (such as occlusion or reflection), and autonomously select or fuse the optimal filling strategy (such as plane fitting, feature matching, etc.). This significantly reduces the human and time costs of data processing, solves the problems of low efficiency and strong subjectivity in traditional manual repair methods, and makes the large-scale, engineering application of automated 3D mesh filling projects for UAV point cloud hole repair possible.

[0156] Figure 3 A schematic diagram of a three-dimensional mesh automatic filling device 300 for repairing holes in point clouds of unmanned aerial vehicles (UAVs) according to an embodiment of the present invention is shown. The device 300 resides in a computing device. The device 300 includes a scanning module 302, a partitioning module 304, a mapping module 306, a marking module 308, a statistics module 310, and a filling module 312 that are coupled to each other.

[0157] The scanning module 302 is adapted to perform laser scanning on the target area using an airborne laser scanning system on a UAV, generating a point cloud dataset of the target area. The partitioning module 304 is adapted to divide the point cloud coverage area into a regular grid array of equal size according to a preset spatial resolution. The mapping module 306 is adapted to map each laser point in the point cloud dataset to its corresponding grid cell according to its planar coordinates. The marking module 308 is adapted to traverse and count the number of point clouds in each grid cell, mark grid cells with a number of point clouds less than a threshold as empty cells, and merge adjacent empty cells to form empty regions. The statistics module 310 is adapted to expand outward by a preset number of adjacent grid cells to form a continuous neighborhood for each empty region, and count the neighborhood characteristics of the neighborhood, which characterize the entity situation of the target area corresponding to the neighborhood grid. The filling module 312 is adapted to select at least one filling strategy based on the area of ​​each empty region and the corresponding neighborhood characteristics to fill the grid attribute values ​​of the empty regions.

[0158] It should be noted that the working principle and process of the device 300 provided in this embodiment are similar to those of the method 200 described above. For relevant details, please refer to the description of the method 200 described above, which will not be repeated here.

[0159] The various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, USB flash drive, floppy disk, CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, the machine becomes an apparatus for practicing the present invention.

[0160] When the program code is executed on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute the method of the present invention according to instructions in the program code stored in the memory.

[0161] By way of example, and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of readable media.

[0162] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of this invention. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of the invention.

[0163] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0164] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in devices as described in this embodiment, or alternatively, can be located in one or more devices different from the device in this example. The modules in the foregoing examples can be combined into a single module or further divided into multiple sub-modules.

[0165] Furthermore, some of the embodiments described herein are methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the functions. Therefore, a processor having the necessary instructions for implementing the methods or method elements forms means for implementing the methods or method elements. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing the functions performed by elements for the purposes of carrying out the invention.

[0166] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0167] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and alterations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims.

Claims

1. An automatic 3D mesh filling method for repairing holes in UAV point clouds, executed on a computing device, the method comprising: The target area is laser-scanned using an airborne laser scanning system on a drone, generating a point cloud dataset of the target area. Based on the preset spatial resolution, the point cloud coverage area is divided into a regular grid array of equal size; Each laser point in the point cloud dataset is mapped to its corresponding grid cell according to its planar coordinates; The number of point clouds in each grid cell is counted. Grid cells with a number of point clouds less than the threshold are marked as hollow cells and adjacent hollow cells are merged to form hollow regions. For each void region, grid cells with a preset number of adjacent layers are expanded outward to form a continuous neighborhood. The neighborhood characteristics of this neighborhood are statistically analyzed. The neighborhood characteristics represent the entity situation of the target region corresponding to the neighborhood grid. The entity situation includes entity type, elevation standard deviation, and flatness. Entity type includes natural entities and man-made entities. Based on the area of ​​each void region and its corresponding neighborhood features, at least one filling strategy is selected to fill the grid attribute values ​​of the void regions. The filling strategies include planar interpolation filling, neighborhood weighted interpolation, feature matching filling, multi-scale hierarchical filling, and constraint optimization filling. If the area of ​​the void region is greater than the area threshold, at least the multi-scale hierarchical filling and constraint optimization filling are activated. If the neighborhood features indicate that the target region contains a first sub-region with artificial entities, at least the feature matching filling should be activated to fill the first sub-region. If the neighborhood features indicate that the target region contains a second sub-region with an elevation standard deviation of less than 0.5m and a flatness that satisfies a slope angle of less than 5°, at least the planar interpolation filling is activated to fill the second sub-region. If the neighborhood features indicate that the target region contains a third sub-region with an elevation standard deviation of not less than 0.5m and less than 2m and a flatness that satisfies a slope angle of not less than 5° and less than 15°, at least the neighborhood weighted interpolation is activated to fill the third sub-region.

2. The method as described in claim 1, wherein, The multi-scale layered filling method includes: The overall outline and main structure of the void region are determined by the geometry of the neighborhood, and the void region is updated. The neighborhood features are redefined based on the updated void regions, and an appropriate filling strategy is selected based on the determined neighborhood features to fill the grid attribute values ​​of the updated void regions.

3. The method as described in claim 1, wherein, The constraint optimization filling method includes: During the filling process, geometric continuity and feature consistency constraints are applied, and the empty areas are filled using an optimization algorithm.

4. The method of claim 1, wherein, Also includes: The point cloud dataset of the target region is preprocessed, and the preprocessing includes at least denoising, registration and classification.

5. An automatic 3D mesh filling device for repairing holes in UAV point clouds, residing in a computing device, the device comprising: The scanning module is adapted to perform laser scanning on a target area using an airborne laser scanning system on a UAV, and generate a point cloud dataset of the target area; The partitioning module is suitable for dividing the point cloud coverage area into a regular grid array of equal size according to a preset spatial resolution. The mapping module is suitable for mapping each laser point in the point cloud dataset to the corresponding grid cell according to its planar coordinates. The marking module is suitable for traversing and counting the number of point clouds in each grid cell, marking grid cells with a number of point clouds less than the number threshold as hollow cells, and merging adjacent hollow cells to form hollow regions. The statistics module is suitable for expanding outwards by a preset number of adjacent grid cells to form a continuous neighborhood for each void area, and statistically analyzing the neighborhood characteristics of the neighborhood. The neighborhood characteristics characterize the entity situation of the target area corresponding to the neighborhood grid. The entity situation includes entity type, elevation standard deviation, and flatness. The entity type includes natural entities and man-made entities. The filling module is adapted to select at least one filling strategy to fill the grid attribute values ​​of the hollow regions based on the area of ​​each hollow region and the corresponding neighborhood features. The filling strategies include planar interpolation filling, neighborhood weighted interpolation, feature matching filling, multi-scale hierarchical filling, and constraint optimization filling. If the area of ​​the hollow region is greater than the area threshold, at least the multi-scale hierarchical filling and constraint optimization filling are activated. If the neighborhood features indicate that the target region contains a first sub-region with artificial entities, at least the feature matching filling should be activated to fill the first sub-region. If the neighborhood features indicate that the target region contains a second sub-region with an elevation standard deviation of less than 0.5m and a flatness that satisfies a slope angle of less than 5°, at least the planar interpolation filling is activated to fill the second sub-region. If the neighborhood features indicate that the target region contains a third sub-region with an elevation standard deviation of not less than 0.5m and less than 2m and a flatness that satisfies a slope angle of not less than 5° and less than 15°, at least the neighborhood weighted interpolation is activated to fill the third sub-region.

6. A computing device, comprising: At least one processor; and A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the method as described in any one of claims 1-4.

7. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-4.

Images