Port local pit rapid detection method based on unmanned aerial vehicle laser point cloud data

Abstract

This invention relates to the fields of lidar point cloud processing and computer vision, and discloses a rapid detection method for localized potholes in ports based on UAV lidar point cloud data. The method includes: outputting a semantic point cloud of the wharf surface using a port point cloud semantic segmentation model; filtering and fusing a reference plane point cloud and a point cloud below the reference plane to obtain a fused analysis point cloud; calculating the concavity depth and curvature features of the fused analysis point cloud; clustering the suspected damaged point cloud set according to the calculation results to extract a set of candidate damaged regions; and applying constraints such as point count, elevation difference, horizontal scale, and extension to the adjacent reference plane to each candidate damaged region to extract complete pothole damaged regions and output them one by one. This invention enables the rapid detection and location of damaged localized potholes in ports using UAV point cloud data, providing accurate basic data support for the formulation of subsequent port emergency repair and reopening plans.

Description

Technical Field

[0001] This invention relates to the fields of lidar point cloud processing and computer vision. Specifically, it relates to a method for rapid detection of local potholes in ports based on UAV lidar point cloud data. More specifically, it is a method that combines semantic segmentation and geometric feature analysis to achieve automatic identification and extraction of damaged pothole areas on the port surface. Background Technology

[0002] With the continuous development of global trade, ports, as important hubs of maritime transportation, play a vital role in economic development and emergency response. In practical applications, port terminals are prone to localized damage, such as dents, bulges, and structural damage, after being subjected to external impacts, explosions, or long-term loads. Such damage not only affects the normal operation of ports but may also pose serious safety hazards.

[0003] Traditional port damage detection mainly relies on manual inspections or image-based methods, which suffer from low efficiency, high subjectivity, and difficulty in operating stably in complex environments. In recent years, the development of UAVs and LiDAR technology has made it possible to acquire high-precision 3D point cloud data, providing a new technical means for port damage detection. Point cloud data can accurately reflect the spatial structure information of the port surface, offering significant advantages in damage detection.

[0004] However, existing point cloud-based damage detection methods still have the following problems: On the one hand, port scenes are complex, containing various targets such as ships, sea surfaces, shorelines, and loading and unloading equipment. Direct analysis of the original point cloud is easily affected by interference, leading to a decrease in detection accuracy. On the other hand, after damage occurs in a port, factors such as structural damage, unstable areas, and safety risks make it difficult to conduct close-range manual inspections. Traditional detection methods are difficult to acquire data and have low efficiency. In contrast, using UAVs equipped with lidar can quickly acquire large-scale, high-precision 3D point cloud data while ensuring a safe distance. It has advantages such as high mobility, wide coverage, and high data acquisition efficiency, but it also brings problems such as large data scale and complex structure. In addition, existing methods mostly rely on single features (such as elevation or curvature) for judgment, making it difficult to simultaneously meet the detection needs of damage at different scales, especially in small-scale pits or boundary areas, which can easily lead to missed or false detections. At the same time, large-scale point cloud data requires a large amount of computation, and traditional methods cannot ensure both detection accuracy and computational efficiency.

[0005] Therefore, there is an urgent need for a method that can achieve high-precision and high-efficiency damage detection in complex port environments. By performing semantic understanding on point clouds and combining it with geometric feature analysis, the method can automatically identify and extract damaged pit areas on the port surface, thereby improving the accuracy and practicality of detection. Summary of the Invention

[0006] The technical problem this invention aims to solve is the danger of close-range measurement when a port is damaged and the low efficiency of multi-source data fusion processing. This invention aims to achieve the function of quickly detecting and locating local potholes in a damaged port using single-point cloud data from UAVs. It proposes a rapid detection method for local potholes in ports based on UAV laser point cloud data, providing accurate basic data support for the formulation of subsequent port repair and reopening plans.

[0007] To address the aforementioned technical problems, this invention proposes a rapid detection method for localized pits in ports based on UAV laser point cloud data. This method includes the following steps:

[0008] S1. A port point cloud semantic segmentation model with a 3D point cloud semantic segmentation network as the backbone network and a feature enhancement module added between the encoder and decoder is used to output the semantic point cloud of the dock surface.

[0009] S2. Extract the reference plane from the semantic point cloud of the wharf surface, and based on the distance of each data point in the semantic point cloud of the wharf surface to the reference plane, filter and fuse the reference plane point cloud and the point cloud below the reference plane to obtain the fused analysis point cloud;

[0010] S3. Calculate the concave depth and curvature features of the fused analysis point cloud, and based on the calculation results, screen the suspected damaged point cloud set from the fused analysis point cloud, and cluster the suspected damaged point cloud set to extract the damaged candidate region set.

[0011] S4. Treat each candidate damage region in the set of candidate damage regions as an independent processing object, and apply point count, elevation difference, horizontal scale and adjacent reference plane expansion constraints to each candidate damage region. Eliminate candidate regions that do not meet the constraints, and extract complete crater damage regions from candidate regions that meet the constraints and output them one by one.

[0012] The beneficial effects of this invention are as follows:

[0013] (1) This invention preprocesses port point clouds by introducing a semantic segmentation model, thereby achieving accurate extraction of the surface area of ​​the wharf, effectively reducing interference from non-target point clouds such as ships, sea surface and loading and unloading equipment, and improving the accuracy of damage and pit detection.

[0014] (2) By integrating geometric features and elevation information, this invention constructs a joint discrimination mechanism based on directional curvature and reference plane depth, which can simultaneously meet the detection needs of damaged structures at different scales, especially improving the ability to identify damage to small-scale pits and boundary areas, and overcoming the problem of easy missed detection and false detection by single feature methods.

[0015] (3) This invention achieves efficient processing of large-scale point cloud data through sliding window block processing and multi-window fusion strategy, which significantly reduces computational complexity while ensuring detection accuracy and improves the engineering application capability of the algorithm.

[0016] This invention addresses the dangers of close-range measurements during port damage and the low efficiency of multi-source data fusion processing. It enables the rapid detection and location of localized craters in damaged ports using single-point cloud data from UAVs, and proposes a reliable emergency port damage detection method, providing accurate basic data support for the formulation of subsequent port repair and reopening plans. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is an overall flowchart of an example of the present invention;

[0019] Figure 2 This is a semantic segmentation category definition diagram for port areas in an example of the present invention;

[0020] Figure 3 This is a schematic diagram of sliding window detection in an example of the present invention;

[0021] Figure 4 This is a schematic diagram of the port simulation damage point cloud and semantic segmentation results as input for an example of the present invention;

[0022] Figure 5 This is a schematic diagram of the semantic segmentation results of the wharf surface in an example of the present invention;

[0023] Figure 6 This is a schematic diagram of the extracted reference plane result in an example of the present invention;

[0024] Figure 7 This is a schematic diagram of the reference plane extraction and fusion result in an example of the present invention;

[0025] Figure 8 This is a schematic diagram of the calculation results of the concave depth anomaly in an example of the present invention;

[0026] Figure 9 This is a schematic diagram of the XY direction curvature anomaly calculation results in an example of the present invention;

[0027] Figure 10 This is a schematic diagram of the damage candidate region extraction results in an example of the present invention;

[0028] Figure 11 This is a schematic diagram showing the comparison results of point-constraint filtering in an example of the present invention.

[0029] Figure 12 This is a schematic diagram showing the comparison results of elevation change constraints in an example of the present invention;

[0030] Figure 13 This is a schematic diagram showing the comparison results of horizontal scale constraints in an example of the present invention;

[0031] Figure 14 This is a schematic diagram of the pit extraction results in an example of the present invention;

[0032] Figure 15 This is a schematic diagram of the pit detection results in an example of the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings, which will make the technical approach and operation steps of the present invention clearer.

[0034] According to an embodiment of the present invention, a rapid detection method for localized pits in ports based on UAV laser point cloud data is provided.

[0035] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, a method for rapid detection of local pits in ports based on UAV laser point cloud data is provided. The method includes the following steps:

[0036] The technical problem this invention aims to solve is the danger of close-range measurement when a port is damaged and the low efficiency of multi-source data fusion processing. It aims to achieve the function of quickly detecting and locating local potholes in a damaged port using single-point cloud data from UAVs, and to propose a reliable emergency port damage detection method, providing accurate basic data support for the formulation of subsequent port repair and reopening plans.

[0037] To address the aforementioned technical challenges, a rapid detection method for localized pits in ports based on UAV laser point cloud data was invented. By employing a sliding window block processing and multi-feature joint discrimination strategy, the method improves the efficiency of large-scale point cloud data processing while ensuring detection accuracy.

[0038] S1. A port point cloud semantic segmentation model is constructed by using a 3D point cloud semantic segmentation network as the backbone network and adding a feature enhancement module between the encoder and decoder to output the semantic point cloud of the wharf surface.

[0039] Among them, the port point cloud semantic segmentation model, which uses a 3D point cloud semantic segmentation network as the backbone network and adds a feature enhancement module between the encoder and decoder, outputs the semantic point cloud of the wharf surface, including:

[0040] S11. Using a preset window size and sliding step, the pre-collected damaged port point cloud data is divided into multiple overlapping sliding window regions by using a sliding window method; data points within each sliding window region are extracted to form a local point cloud subset covering the entire scene.

[0041] S12. Construct a port point cloud semantic segmentation model, and use the port point cloud semantic segmentation model to perform semantic prediction on the local point cloud subset within each sliding window to obtain the semantic category response value corresponding to each data point.

[0042] Among them, the port point cloud semantic segmentation model uses a 3D point cloud semantic segmentation network based on the Transformer architecture as the backbone network. The network structure includes an input embedding layer, an encoder, a decoder, a skip connection layer, and a classification output layer, and a feature enhancement module is set between the encoder and the decoder.

[0043] Specifically, a port point cloud semantic segmentation model is constructed. This model is then used to perform semantic prediction on a local point cloud subset within each sliding window, yielding the semantic category response value for each data point.

[0044] S121. Based on the three-dimensional spatial coordinates and RGB color information of each data point in the local point cloud subset, construct the initial features of the port damage point cloud;

[0045] S122. Input the initial features of the port damage point cloud into the port point cloud semantic segmentation model, and map the initial features of the port damage point cloud into high-dimensional point features through the input embedding layer.

[0046] S123. The encoder adopts a multi-level feature extraction structure, which performs step-by-step sampling and feature dimensionality upscaling operations on high-dimensional point features to obtain the number of point feature channels output by each coding layer. In each coding layer, attention feature aggregation is performed on local neighborhood points through the point attention aggregation module to obtain the deep features output by the encoder.

[0047] S124. Through the feature enhancement module between the encoder and decoder, the deep features output by the encoder are enhanced geometrically and color-coordinated to obtain the enhanced fused features.

[0048] Specifically, the feature enhancement module between the encoder and decoder performs geometric and color co-enhancement on the deep features output by the encoder, resulting in enhanced fused features including:

[0049] S1241. Construct geometric enhancement branches and color enhancement branches respectively through the feature enhancement module between the encoder and decoder;

[0050] S1242. The geometry enhancement branch extracts local geometric features based on the spatial neighborhood relationship of data points, and maps the local geometric features through a multilayer perceptron to obtain geometry enhancement features.

[0051] S1243. The color enhancement branch constructs color features based on the RGB color information of the data points, and maps the color features through a multilayer perceptron to obtain color enhancement features.

[0052] S1244. The deep features, geometric enhancement features and color enhancement features output by the encoder are fused to obtain the fused features enhanced by the feature enhancement module.

[0053] S125. The fused features are restored to the number of encoder output channels through linear mapping to obtain the enhanced encoded features. The enhanced encoded features are input to the decoder and the category response value of each data point is output through the classification output layer.

[0054] S13. For a target data point that is covered by multiple sliding windows at the same time, obtain the semantic category response value of the target data point in all sliding windows containing it and fuse them to obtain the fused response value of the target data point.

[0055] S14. Select the semantic category with the largest fusion response value as the final semantic category label of the target data point by using the maximum response principle. Select the data points with semantic labels as the dock surface from the final semantic category labels to obtain the semantic point cloud of the dock surface.

[0056] It should be noted that, for ease of understanding of the above technical solution of the present invention, the following further explains the port point cloud semantic segmentation model of the present invention, which uses a three-dimensional point cloud semantic segmentation network as the backbone network and adds a feature enhancement module between the encoder and decoder, to output the semantic point cloud of the wharf surface from the perspective of architecture and principle, as follows:

[0057] This invention utilizes an improved port point cloud semantic segmentation model to segment a port UAV point cloud using simulated random damage data to obtain a semantic point cloud of the wharf surface. This semantic point cloud serves as the input for reference plane extraction and fusion. Specifically, it can be divided into the following aspects:

[0058] 1) Initial feature construction of point cloud;

[0059] The initial feature vector of the input port damage (simulated) point cloud is constructed using the following formula. :

[0060] ;

[0061] in, For the first Feature vectors of points, For the first The spatial coordinates of the points For the first RGB color information of each point.

[0062] 2) Point Transformer V3;

[0063] The port point cloud semantic segmentation model uses Point Transformer V3 as its backbone network. Its network structure includes an input embedding layer, encoder, decoder, skip connection layer, and classification output layer. The input embedding layer maps the original six-dimensional features into high-dimensional point features.

[0064] ;

[0065] in, For input embedding mapping functions, This represents the initial number of feature channels.

[0066] The encoder employs a multi-layered feature extraction structure, performing step-by-step sampling and feature dimensionality upscaling on the point cloud. The feature dimensions of each layer are as follows:

[0067] ;

[0068] in, These represent the number of point feature channels output by different coding layers.

[0069] In each coding layer, attention features of local neighborhood points are aggregated using the Point Transformer Block:

[0070]

[0071] in, For the first Layer output features, For neighborhood points In the The input features of the layer For point The set of neighborhood points, For point With neighboring points Attention weights between them This is a spatial relative position encoding function.

[0072] In this invention, the deep features output by the encoder are not directly input into the decoder, but first enter the PASEM port semantic segmentation enhancement module for geometric and color co-enhancement; the enhanced encoded features are then input into the decoder. The decoder recovers point-level semantic features step by step through feature upsampling and skip connections, and fuses the shallow spatial information from the encoding stage with the deep semantic information to finally obtain point-level classification features:

[0073] ;

[0074] in, Point-level features for semantic classification This represents the number of feature channels before classification. This point-level classification feature is obtained by the decoder from the enhanced encoded features output by the PASEM port semantic segmentation enhancement module, and is used as the input to the classification output layer to calculate the category response value of each semantic category corresponding to this data point, thereby determining the final semantic category label.

[0075] 3) PASEM port semantic segmentation enhancement module;

[0076] The PASEM port semantic enhancement module is inserted after the output features of the Point Transformer V3 encoder and before the feature recovery of the decoder, that is, at the deep semantic feature output end of the backbone network. It is used to perform geometric and color co-enhancement on the port point cloud features extracted by the encoder.

[0077] Specifically, let the deep point features output by the encoder be:

[0078]

[0079] in, For the first The deep features obtained after the points are encoded .

[0080] The PASEM module constructs both a geometry enhancement branch and a color enhancement branch. The geometry enhancement branch extracts local geometric descriptions based on the spatial neighborhood relationships of points.

[0081]

[0082] in, For the first Local geometric feature description of each point For the first One point, Let the set of neighborhood points of this point be . This is a geometric feature extraction function.

[0083] Geometric features are mapped using a multilayer perceptron:

[0084] ;

[0085] in, For geometric enhancement features, For geometric feature mapping function, This represents the number of feature channels for geometric enhancement.

[0086] The color enhancement branch uses the RGB information of points to construct color features:

[0087] ;

[0088] Color feature mapping is performed using a multilayer perceptron.

[0089] ;

[0090] in, For the first RGB color characteristics of each point To enhance color features, For color feature mapping function, Increase the number of feature channels for color enhancement.

[0091] Then, the encoder output features, geometric enhancement features, and color enhancement features are fused together:

[0092] ;

[0093] in, This is a fusion feature enhanced by PASEM.

[0094] To ensure that the output dimension of the PASEM module is consistent with the input dimension of the subsequent decoder, the fused features are restored to the number of encoder output channels through linear mapping:

[0095] ;

[0096] in, These are the encoded features after PASEM enhancement. This is the feature fusion mapping function.

[0097] Enhanced features Input to the Point Transformer V3 decoder, and output the class response value of the point through the classification layer:

[0098]

[0099] in, For the first Category response value of each point, This is a classification mapping function.

[0100] Finally, the semantic category of the point is determined using the maximum response principle:

[0101] ;

[0102] in, For the first The final semantic category of each point.

[0103] The PASEM module is not independent of the semantic segmentation process; rather, it serves as a feature enhancement unit between the Point Transformer V3 encoder and decoder. Its output features directly participate in subsequent decoding and classification calculations, allowing geometric and color enhancement features to work together on the final semantic segmentation result. Laser point cloud data from over ten ports in Jiangsu and Zhejiang provinces were collected by drones and processed into samples using CLoudCompare software according to the 3sDIS dataset format. Specific sample annotation methods are as follows: Figure 2 As shown, model training then commenced. During the training phase, the optimized model employed the AdamW optimizer with a learning rate of 0.0006 to ensure stability during training under enhanced feature input conditions. A total of 1000 epochs were trained, and strategies such as memory cache cleanup, gradient pruning, and empty cache reclamation were implemented to prevent memory explosion.

[0104] 4) Sliding window point cloud segmentation and fusion processing;

[0105] Due to the large scale of port point cloud data, directly performing semantic segmentation on the entire point cloud is computationally complex and easily exceeds computational resource limitations. Therefore, a sliding window approach is used to divide the point cloud into blocks, and overlapping regions are fused and predicted. The specific process is as follows:

[0106] First, the original point cloud is spatially partitioned using the following formula:

[0107] ;

[0108] in, This is the original point cloud collection. For the first A subset of point clouds within a window Let be the coordinates of the point, ( ) is the first The coordinates of the bottom left corner of the window This refers to the window size.

[0109] Secondly, slide the window in the following ways:

[0110] ;

[0111] in, This is the sliding step size. For example... Figure 3As shown, in this invention, the window size is 16*16 meters, and the step size is 10 meters (a sliding window method is used to segment the point cloud into blocks, with the window size set to 16 meters and the sliding step size set to 10 meters; when the number of points in the window is less than 64, the semantic prediction result of that window is discarded). For machines with less than 16GB of video memory, a size of 10-20 meters is recommended, which provides good overall segmentation results for ports and avoids video memory exhaustion. For each window point cloud, prediction is performed independently using the PointTransformer V3 and PASEM semantic segmentation models to obtain the point's category response value (logits), and the points covered by multiple windows are fused using the following formula:

[0112]

[0113] in, For point The fusion response value, For point in the th The prediction results in each window Containing point All window collections, Number of windows. Response value. For point The classification response vector for each semantic category has the following dimensions: ,in This indicates the number of semantic categories. In this invention, the semantic categories include five types: Surface, Ship, Sea, Trees, and Gantry Crane. In this example, Ship refers to a cargo ship. Specifically, the response value can be expressed as:

[0114]

[0115] in, Point Belongs to the The response value for the semantic category is output by the classification layer of the semantic segmentation network and is used to characterize the probability distribution trend of a point belonging to each category.

[0116] Finally, the final category is determined using the following formula:

[0117] ;

[0118] in, For point The final semantic category label is obtained by selecting the category with the largest response value as the classification result for that point. The above process shows that the prediction results of a point in multiple sliding windows are fused to obtain the final classification basis, thereby improving the stability and accuracy of the classification results. Figure 4 As shown, the input point cloud of a port drone, after being processed by the semantic segmentation model using simulated random damage data, is segmented into five semantic categories. Among them, the point cloud of the dock surface (Surface) is shown below. Figure 5 As shown, we will proceed to step S2 to continue the calculation, and the number of point clouds is 1227601.

[0119] S2. Extract the reference plane from the semantic point cloud of the wharf surface, and based on the distance of each data point in the semantic point cloud of the wharf surface to the reference plane, filter and merge the reference plane point cloud and the point cloud below the reference plane to obtain the merged analysis point cloud.

[0120] Specifically, a reference plane is extracted from the semantic point cloud of the wharf surface. Based on the distance from each data point in the semantic point cloud of the wharf surface to the reference plane, the reference plane point cloud and the point cloud below the reference plane are filtered and fused to obtain the fused analysis point cloud, which includes:

[0121] S21. Based on the three-dimensional spatial coordinates of any data point in the semantic point cloud of the wharf surface, and combined with the normal vector components and plane offset parameters of the reference plane, fit the reference plane using the random sampling consistency method.

[0122] S22. Calculate the signed distance from each data point in the semantic point cloud of the wharf surface to the reference plane, and compare the signed distance with a preset threshold. Based on the comparison result, determine the spatial position of each data point relative to the reference plane.

[0123] S23. Based on the signed distance from each data point to the reference plane, the robustness scale is calculated using the median absolute deviation method. The reference plane bandwidth is constructed by combining the robustness scale reference plane with the initial distance threshold.

[0124] S24. Combining the spatial position of each data point relative to the reference plane and the bandwidth of the reference plane, the reference plane point cloud and the point cloud below the reference plane are selected from the point cloud on the surface of the dock. The reference plane point cloud and the point cloud below the reference plane are then fused to obtain the fused analysis point cloud.

[0125] It should be noted that, in order to facilitate understanding of the above technical solutions of the present invention, the following explanation will focus on the extraction of a reference plane from the semantic point cloud of the wharf surface, and the further explanation of how the reference plane point cloud and the point cloud below the reference plane are filtered and fused based on the distance of each data point in the semantic point cloud of the wharf surface to the reference plane, to obtain the fused analysis point cloud. The details are as follows:

[0126] Specifically, this invention extracts a reference plane from the semantic point cloud of the dock surface obtained in step S1 and constructs an analysis point cloud for damage detection in step S3. The specific process is as follows:

[0127] 1) Establishment of the reference plane model;

[0128] Based on the point cloud data of the wharf surface, a reference plane is fitted using the random sampling consistency method, and expressed by the following formula:

[0129] ;

[0130] in, These represent the horizontal, vertical, and elevation coordinates of any point in the point cloud in three-dimensional space. The normal vector component of the reference plane is used to represent the spatial orientation of the plane; This is the plane offset parameter, used to represent the position of the plane relative to the origin.

[0131] 2) Calculation of spatial relationships in point clouds;

[0132] The spatial position of a point relative to a reference plane is determined by calculating the signed distance from the point to the reference plane. The calculation method is as follows:

[0133]

[0134] in, For the first Signed distance from each point to the reference plane The first The three-dimensional coordinates of each point; The aforementioned reference plane parameters; The length of the plane normal vector is used to normalize the distance; when When, it indicates that the point is below the reference plane, when When, it indicates that the point is located above the reference plane, when When the time is 0, it indicates that the point is located on the reference plane.

[0135] 3) Determination of the robust reference plane region;

[0136] Based on the distance distribution from points to the reference plane, the robustness scale is calculated using the median absolute deviation method, and the reference plane bandwidth is constructed as follows:

[0137] ;

[0138] ;

[0139] in, This is a robust scaling parameter used to measure the degree of dispersion in the distance from a point to a reference plane; For the first Signed distance from each point to the reference plane The median of the signed distances to all points; For the first The absolute deviation of each point from the median; The median of the absolute deviation; coefficient This is used to convert the absolute deviation of the median into a robust scaling estimate that is close to the standard deviation. The final determined reference plane bandwidth; The initial distance threshold for the reference plane; This is a minimum bandwidth constraint set based on an initial distance threshold; For robust scaling parameters; The bandwidth is adaptively determined based on the distance distribution; This indicates that the larger of the two values ​​is used to avoid the erroneous exclusion of valid reference plane points due to an insufficient reference plane bandwidth. For example... Figure 6 As shown, a total of 609,453 points were extracted as the reference plane.

[0140] 4) Candidate point cloud construction;

[0141] Based on the relative position of points to the reference plane, the point cloud on the reference plane and the point cloud below the reference plane are filtered and then fused in the following way:

[0142] ;

[0143] in, This is the fused collection of detection and analysis point clouds. As a reference plane point cloud, This forms the point cloud below the reference plane, thus obtaining the analytical point cloud for subsequent damage detection. For example... Figure 7 As shown, after fusing the reference plane and the point cloud below it, 925,458 point clouds were obtained as candidate point cloud data for subsequent calculations, while the clean point cloud of the dock surface and its damaged craters was preserved.

[0144] S3. Calculate the concave depth and curvature features of the fused analysis point cloud. Based on the calculation results, select a set of suspected damaged point clouds from the fused analysis point cloud. Then, cluster the set of suspected damaged point clouds and extract a set of candidate damaged regions.

[0145] The process involves calculating the concavity depth and curvature features of the fused point cloud, selecting a set of suspected damaged point clouds based on the calculation results, clustering the suspected damaged point cloud set, and extracting a set of candidate damaged regions, including:

[0146] S31. Project the fused analysis point cloud onto the reference plane obtained by fitting through the random sampling consistency method, and then calculate the concavity depth of each data point based on the signed distance from each data point in the fused analysis point cloud to the reference plane.

[0147] S32. The residual field is spatially discretized using a reference plane, and the comprehensive curvature of each data point is calculated based on the spatial discretization.

[0148] Specifically, the residual field is spatially discretized using a reference plane, and the comprehensive curvature of each data point is calculated based on this spatial discretization, including:

[0149] S321. Using the reference plane fitted by the random sampling consistency method, the residual field is spatially discretized, and the second derivative of the residual function along two orthogonal directions is calculated.

[0150] S322. Based on the second derivative of the residual function along two orthogonal directions, the curvature response of the residual surface in the two orthogonal directions is calculated.

[0151] S323. The composite curvature of each data point is obtained by taking the maximum value of the curvature response of the residual surface in two orthogonal directions.

[0152] S33. Compare the concavity depth and overall curvature of each data point with the concavity depth threshold and curvature screening threshold respectively to determine curvature anomalies and concavity depth anomalies. Based on the joint constraints of curvature anomalies and concavity depth anomalies, screen the set of suspected damaged point clouds from the fused analysis point cloud.

[0153] S34. Cluster the suspected damaged point cloud set using density clustering method to obtain a set of damaged candidate regions after the clustering is completed.

[0154] Specifically, the suspected damage point cloud set is clustered and grouped using a density clustering method. After the clustering and grouping, the set of damage candidate regions is obtained, including:

[0155] S341. Using the preset neighborhood radius and minimum neighborhood point number threshold as criteria, traverse the suspected damaged point cloud set, mark the data points in the neighborhood that meet the threshold condition as core points, and output the core point set.

[0156] S342. Starting from each core point output, recursively search and aggregate the points in its neighborhood along the density-reachable path, and group all density-connected points into the same cluster until the cluster to which the core point belongs no longer expands.

[0157] S343. Repeat step S342 for the remaining unprocessed core points until all core points are assigned to a certain cluster. Finally, output a set of damage candidate regions consisting of all clusters, with each cluster corresponding to an independent damage candidate region.

[0158] It should be noted that, in order to facilitate understanding of the above technical solution of the present invention, the following describes, from the perspective of architecture and principle, the calculation of concave depth and curvature features of the fused analysis point cloud, the selection of a set of suspected damaged point clouds from the fused analysis point cloud based on the calculation results, and the further explanation of the clustering and grouping of the suspected damaged point cloud set to extract a set of damaged candidate regions, as detailed below:

[0159] This invention performs curvature feature calculation and outlier screening on the fused point cloud from step S2, and extracts candidate damage regions using clustering methods. The specific process is as follows:

[0160] 1) Calculation of residual characteristics;

[0161] Based on the reference plane obtained in step S2 by fitting using the random sampling consensus method (the reference plane extraction uses the random sampling consensus algorithm, and the plane fitting distance threshold is set to 0.08 meters), the fused point cloud is projected onto the reference plane:

[0162] ;

[0163] in, These are the horizontal coordinates of the point; As reference plane along The slope coefficient of the direction; As reference plane along The slope coefficient of the direction; The elevation intercept parameter of the reference plane. For the first Elevation residuals at each point; For the first The actual elevation of each point; when When <0, it indicates that the point is below the reference plane. A value greater than 0 indicates that the point is above the reference plane. Then, based on the signed distance from the point to the reference plane, the depth of concavity at the point is calculated:

[0164] ;

[0165] in, For the first The concavity depth of a point relative to the reference plane (the concavity depth is obtained by calculating the signed distance from the point to the reference plane; if the concavity depth is greater than 0.15 meters and the directional curvature of the point is greater than the curvature threshold, the point is identified as a suspected damage point). For the first The signed distance from each point to the reference plane; when hour, The depth at which this point lies below the reference plane; when hour, =0 indicates that the point does not belong to the concave region. For example... Figure 8 The result shown is the calculation result of the concave depth anomaly.

[0166] 2) Curvature characteristic calculation;

[0167] Based on the reference plane fitted by the random sampling consensus method in step S2, the curvature response of the point in two orthogonal directions is obtained by spatial discretizing the residual field and calculating the second derivative of the residual surface:

[0168]

[0169]

[0170] in, For residual surfaces in Second-order curvature response in the direction; For residual mesh functions; For the residual function along The second derivative of the direction; For residual surfaces in Second-order curvature response in the direction; For the residual function along The second derivative of the direction is used to obtain the combined curvature of the point by taking the maximum value:

[0171] ;

[0172] ;

[0173] in, The final curvature is selected based on the threshold. Curvature values ​​for all composite directions The percentile; Percentile parameter; This is the preset absolute curvature threshold; The ) indicates a larger value, used to avoid false detections due to an excessively small threshold when the overall curvature is low. For example... Figure 9 The results of curvature anomaly calculations in the X and Y directions are shown, and the curvature quantiles of the current instance are also included. 90%, It is 0.006.

[0174] 3) Screening of suspected damage points;

[0175] Based on the joint constraints of curvature anomaly and concavity depth anomaly, a set of suspected damaged point clouds is selected:

[0176] ;

[0177] in, This is a collection of suspected damage points in the cloud. For the first One point; and The first The point is at Curvature response in the direction; Use curvature as the selection threshold; For the first The depth of the concavity at each point; The depth threshold for the depression is 0.15 meters in this example; "This means that any curvature in any direction can be considered to be abnormal. The point cloud residual mesh size is set to 0.20 meters, the Gaussian smoothing parameter is set to 1.0, and the curvature threshold is determined by a combination of percentile and absolute thresholds. The curvature percentile is set to 90%, and the absolute curvature threshold is set to 0.006."

[0178] 4) Clustering of candidate damage regions;

[0179] The suspected damaged point cloud was grouped using density clustering, and the neighborhood relationships between the points satisfy the following:

[0180] ;

[0181] in, For any two points in the suspected damage point cloud set; The Euclidean distance between the two points; The neighborhood radius parameter for density clustering (this invention uses the DBSCAN density clustering method to group suspected damage points, where the neighborhood radius parameter is set to 0.3 meters to 1.0 meters, and the minimum number of neighborhood points is set to 20 to 50). When point The number of points in the neighborhood is greater than the minimum neighborhood threshold. When this point is defined as the core point, and starting from the core point, the neighboring points are recursively expanded through density reachability relations to classify all density-connected points into the same category.

[0182] Then, repeat the process for all points that meet the above conditions until all points have been clustered.

[0183]

[0184] in, This is the set of candidate damage regions obtained after clustering; For the first One potential area for damage; This represents the number of candidate destruction areas. For example... Figure 10 The damage candidate region extraction results are shown. The neighborhood radius parameter is set to 0.3 meters to 1.0 meters, and the minimum number of neighborhood points is set to 20 to 50. In this example, 90,790 point clouds were finally extracted and entered into step S4 for calculation.

[0185] S4. Treat each candidate damage region in the set of candidate damage regions as an independent processing object, and apply point count, elevation difference, horizontal scale and adjacent reference plane expansion constraints to each candidate damage region. Eliminate candidate regions that do not meet the constraints, and extract complete crater damage regions from candidate regions that meet the constraints and output them one by one.

[0186] Specifically, each candidate damage region in the set of candidate damage regions is treated as an independent processing object. Constraints are applied to each candidate damage region based on the number of points, elevation difference, horizontal scale, and extension to the adjacent reference plane. Candidate regions that do not meet the constraints are eliminated. For candidate regions that meet the constraints, complete crater damage regions are extracted and output one by one, including:

[0187] S41. Traverse each candidate damage region in the set of candidate damage regions, and eliminate noisy candidate regions based on the number of points in each candidate damage region.

[0188] S42. Traverse each damage candidate region in the damage candidate region set, and based on the elevation range of each damage candidate region, eliminate damage candidate regions with insufficient elevation fluctuations.

[0189] This involves traversing each candidate damage region in the set of candidate damage regions, and eliminating candidate damage regions with insufficient elevation variation based on the elevation range of each candidate damage region, including:

[0190] S421. Obtain the elevation range of data points in each damage candidate region in the damage candidate region set, and calculate the elevation change range of the damage candidate region.

[0191] S422. Compare the elevation change range of the candidate damaged area with the minimum elevation change threshold value. If the elevation change range is less than the minimum elevation change threshold value, the candidate damaged area is determined to have insufficient elevation fluctuation and is removed.

[0192] S43. Traverse each candidate damage region in the set of candidate damage regions, calculate the maximum horizontal span of each candidate damage region based on its spatial range on the horizontal plane, and remove data points within the region formed by the merging of dock boundaries, range anomalies, or multiple adjacent anomaly regions.

[0193] S44. Traverse each damage candidate region in the damage candidate region set. Based on the horizontal distance relationship between each damage candidate region and the reference plane point cloud, merge the neighboring reference plane point clouds into the damage candidate region to construct a complete pit damage region point cloud set.

[0194] S45. Calculate the center point of the crater based on the spatial coordinates of all data points in the point cloud set of the complete crater damage area, and save each crater as an independent point cloud file.

[0195] It should be noted that, for ease of understanding of the above technical solution of the present invention, the following further explains, from the perspective of architecture and principle, how the present invention treats each candidate damage region in the set of candidate damage regions as an independent processing object, applies constraints on the number of points, elevation difference, horizontal scale, and adjacent reference plane expansion to each candidate damage region, eliminates candidate regions that do not meet the constraints, and extracts complete crater damage regions from the candidate regions that meet the constraints and outputs them one by one, as detailed below:

[0196] This invention applies constraints based on the number of points, elevation difference, horizontal scale, and adjacent reference plane expansion to the candidate damage areas obtained in step S3, extracting complete crater damage areas and outputting them one by one. Specifically, it can be divided into the following aspects:

[0197] 1) Filtering based on point count constraints;

[0198] Based on the number of points within the candidate region, noisy candidate regions are eliminated:

[0199] ;

[0200] ;

[0201] in, For the first The number of points contained in each candidate region; For the first One candidate region; This indicates the number of elements in the set. The minimum number of points threshold; when If the candidate region is too small, it is considered a noise region and is therefore removed. Figure 11 As shown in this example The minimum number of points was set to 300, resulting in the removal of 1163 point clouds.

[0202] 2) Elevation variation constraints;

[0203] Calculate the elevation variation range of the candidate region based on the elevation range of the points within the candidate region:

[0204]

[0205] ;

[0206] in, For the first The elevation variation range of each candidate region; For the first One candidate region; These are points within the candidate region; For point Elevation coordinates; Indicates the elevation of the highest point within the candidate area; This indicates the elevation of the lowest point within the candidate area. The minimum elevation change threshold value (when filtering candidate areas, the minimum number of points is set to 300, and the minimum elevation change threshold is set to 0.15 meters); when If this is the case, it indicates that the elevation variation in the area is insufficient, and the area should be eliminated. For example... Figure 12 As shown in this example The minimum elevation change threshold value was 0.15 meters, and 20,188 points were eliminated.

[0207] 3) Horizontal scale constraints;

[0208] Calculate the maximum horizontal span of the candidate region based on its spatial extent in the horizontal plane:

[0209] ;

[0210] ;

[0211] in, For the first The maximum horizontal span of each candidate region; This indicates that the candidate region is in Span in direction; This indicates that the candidate region is in Span in direction; ) indicates taking the larger value of the span in the two directions. The maximum horizontal span threshold; when In such cases, it indicates that the area may be a dock boundary, a large-scale anomaly, or an area formed by the merger of multiple adjacent anomaly areas, and should be eliminated or further separated. For example... Figure 13 As shown in this example The maximum horizontal span threshold was 5 meters, and 25,481 points were eliminated.

[0212] 4) Regional expansion and integration;

[0213] Based on the horizontal distance relationship between the candidate region and the reference plane point cloud, the adjacent reference plane point clouds are incorporated into the candidate crater region to construct the complete crater damage area:

[0214] ;

[0215] in, For the first A collection of point clouds representing the complete crater damage area; For the first One candidate pit area; The reference plane point cloud set; Points in the reference plane point cloud set; For point To candidate area Minimum distance on the horizontal projection plane; This is the radius of the area expansion. In this example, the expansion radius is set to 4.0 meters.

[0216] 5) Calculation of the center of the pit;

[0217] Calculate the center point of the crater based on the spatial coordinates of all points within the complete crater damage area:

[0218]

[0219] in, For the first The center point of the pit area; For the first A collection of point clouds representing the complete crater damage area; This represents the number of points contained in the pit area; The first in the pit area One point; This represents the summation of the three-dimensional coordinates of all points within the region.

[0220] 6) Extraction of pitting results;

[0221] Based on the final extracted complete crater damage area, each crater is saved as an independent point cloud file:

[0222] ;

[0223] in, For the first A collection of point clouds representing the complete crater damage area; For the first Each pothole corresponds to a LAS format point cloud file; the arrow indicates that the set of pothole point clouds is written to the corresponding result file. For example... Figure 14 The extracted pit point cloud is shown. Figure 15The display shows the location of the potholes in the input port surface semantic point cloud, indicating that all port surface potholes that can be visually interpreted have been completely detected and extracted.

[0224] According to another embodiment of the present invention, a rapid detection system for localized pits in ports based on UAV laser point cloud data is also provided, the system comprising:

[0225] The dock surface point cloud segmentation module is used to output the dock surface semantic point cloud by using a port point cloud semantic segmentation model with a 3D point cloud semantic segmentation network as the backbone network and a feature enhancement module added between the encoder and decoder.

[0226] The point cloud analysis module is used to extract the reference plane from the semantic point cloud of the wharf surface, and based on the distance of each data point in the semantic point cloud of the wharf surface to the reference plane, it filters and merges the reference plane point cloud and the point cloud below the reference plane to obtain the merged analysis point cloud.

[0227] The damage candidate region extraction module is used to calculate the concave depth and curvature features of the fused analysis point cloud, filter the suspected damage point cloud set from the fused analysis point cloud based on the calculation results, and cluster the suspected damage point cloud set to extract the damage candidate region set.

[0228] The complete pothole damage area extraction module is used to process each damage candidate area in the damage candidate area set as an independent processing object, and to impose constraints on each damage candidate area based on the number of points, elevation difference, horizontal scale, and adjacent reference plane expansion. Candidate areas that do not meet the constraints are eliminated, and the complete pothole damage areas of the candidate areas that meet the constraints are extracted and output one by one.

[0229] The method for rapid detection of local pits in ports based on UAV laser point cloud data is not limited to the specific technical solutions and implementation areas of the above embodiments. All technical solutions formed by equivalent substitutions are within the scope of protection claimed by this invention.

Claims

1. A method for rapid detection of local pits in a port based on unmanned aerial vehicle (UAV) laser point cloud data, characterized in that, The method includes the following steps: S1. A port point cloud semantic segmentation model with a 3D point cloud semantic segmentation network as the backbone network and a feature enhancement module added between the encoder and decoder is used to output the semantic point cloud of the dock surface. S2. Extract the reference plane from the semantic point cloud of the wharf surface, and based on the distance of each data point in the semantic point cloud of the wharf surface to the reference plane, filter and fuse the reference plane point cloud and the point cloud below the reference plane to obtain the fused analysis point cloud; S3. Calculate the concave depth and curvature features of the fused analysis point cloud, and based on the calculation results, screen the suspected damaged point cloud set from the fused analysis point cloud, and cluster the suspected damaged point cloud set to extract the damaged candidate region set. S4. Treat each candidate damage region in the set of candidate damage regions as an independent processing object, and impose constraints on the number of points, elevation difference, horizontal scale and adjacent reference plane expansion on each candidate damage region. Eliminate candidate regions that do not meet the constraints, and extract complete crater damage regions from candidate regions that meet the constraints and output them one by one. S3 includes: S31. Project the fused analysis point cloud onto the reference plane obtained by fitting through the random sampling consistency method, and then calculate the concavity depth of each data point based on the signed distance from each data point in the fused analysis point cloud to the reference plane. S32. The residual field is spatially discretized using a reference plane, and the comprehensive curvature of each data point is calculated based on the spatial discretization. S33. Compare the concavity depth and overall curvature of each data point with the concavity depth threshold and curvature screening threshold respectively to determine curvature anomalies and concavity depth anomalies. Based on the joint constraints of curvature anomalies and concavity depth anomalies, screen the set of suspected damaged point clouds from the fused analysis point cloud. S34. Cluster the suspected damaged point cloud set using density clustering method to obtain a set of damaged candidate regions after the clustering is completed.

2. The method for detecting port local pits quickly based on UAV laser point cloud data according to claim 1, characterized in that, S1 includes: S11. Using a preset window size and sliding step, the pre-collected damaged port point cloud data is divided into multiple overlapping sliding window regions by using a sliding window method; data points within each sliding window region are extracted to form a local point cloud subset covering the entire scene. S12. Construct a port point cloud semantic segmentation model, and use the port point cloud semantic segmentation model to perform semantic prediction on the local point cloud subset within each sliding window to obtain the semantic category response value corresponding to each data point. Among them, the port point cloud semantic segmentation model uses a 3D point cloud semantic segmentation network based on the Transformer architecture as the backbone network. The network structure includes an input embedding layer, an encoder, a decoder, a skip connection layer, and a classification output layer, and a feature enhancement module is set between the encoder and the decoder. S13. For a target data point that is covered by multiple sliding windows at the same time, obtain the semantic category response value of the target data point in all sliding windows containing it and fuse them to obtain the fused response value of the target data point. S14. Select the semantic category with the largest fusion response value as the final semantic category label of the target data point by using the maximum response principle. Select the data points with semantic labels as the dock surface from the final semantic category labels to obtain the semantic point cloud of the dock surface.

3. The method for rapid detection of localized pits in ports based on UAV laser point cloud data according to claim 2, characterized in that, S12 includes: S121. Based on the three-dimensional spatial coordinates and RGB color information of each data point in the local point cloud subset, construct the initial features of the port damage point cloud; S122. Input the initial features of the port damage point cloud into the port point cloud semantic segmentation model, and map the initial features of the port damage point cloud into high-dimensional point features through the input embedding layer. S123. The encoder adopts a multi-level feature extraction structure, which performs step-by-step sampling and feature dimensionality upscaling operations on high-dimensional point features to obtain the number of point feature channels output by each coding layer. In each coding layer, attention feature aggregation is performed on local neighborhood points through the point attention aggregation module to obtain the deep features output by the encoder. S124. Through the feature enhancement module between the encoder and decoder, the deep features output by the encoder are enhanced by geometric and color coordination to obtain the enhanced fused features. S125. The fused features are restored to the number of encoder output channels through linear mapping to obtain the enhanced encoded features. The enhanced encoded features are input to the decoder and the category response value of each data point is output through the classification output layer.

4. The method for rapid detection of localized pits in ports based on UAV laser point cloud data according to claim 3, characterized in that, S124 includes: S1241. Construct geometric enhancement branches and color enhancement branches respectively through the feature enhancement module between the encoder and decoder; S1242. The geometry enhancement branch extracts local geometric features based on the spatial neighborhood relationship of data points, and maps the local geometric features through a multilayer perceptron to obtain geometry enhancement features. S1243. The color enhancement branch constructs color features based on the RGB color information of the data points, and maps the color features through a multilayer perceptron to obtain color enhancement features. S1244. The deep features, geometric enhancement features and color enhancement features output by the encoder are fused to obtain the fused features enhanced by the feature enhancement module.

5. The method for rapid detection of localized pits in ports based on UAV laser point cloud data according to claim 1, characterized in that, S2 includes: S21. Based on the three-dimensional spatial coordinates of any data point in the semantic point cloud of the wharf surface, and combined with the normal vector components and plane offset parameters of the reference plane, fit the reference plane using the random sampling consistency method. S22. Calculate the signed distance from each data point in the semantic point cloud of the wharf surface to the reference plane, and compare the signed distance with a preset threshold. Based on the comparison result, determine the spatial position of each data point relative to the reference plane. S23. Based on the signed distance from each data point to the reference plane, the robustness scale is calculated using the median absolute deviation method. The reference plane bandwidth is constructed by combining the robustness scale reference plane with the initial distance threshold. S24. Combining the spatial position of each data point relative to the reference plane and the bandwidth of the reference plane, the reference plane point cloud and the point cloud below the reference plane are selected from the point cloud on the surface of the dock. The reference plane point cloud and the point cloud below the reference plane are then fused to obtain the fused analysis point cloud.

6. The method for rapid detection of localized pits in ports based on UAV laser point cloud data according to claim 1, characterized in that, S32 includes: S321. Using the reference plane fitted by the random sampling consistency method, the residual field is spatially discretized, and the second derivative of the residual function along two orthogonal directions is calculated. S322. Based on the second derivative of the residual function along two orthogonal directions, the curvature response of the residual surface in the two orthogonal directions is calculated. S323. The composite curvature of each data point is obtained by taking the maximum value of the curvature response of the residual surface in two orthogonal directions.

7. The method for rapid detection of localized pits in ports based on UAV laser point cloud data according to claim 6, characterized in that, S34 includes: S341. Using the preset neighborhood radius and minimum neighborhood point number threshold as criteria, traverse the suspected damaged point cloud set, mark the data points in the neighborhood that meet the threshold condition as core points, and output the core point set. S342. Starting from each core point output, recursively search and aggregate the points in its neighborhood along the density-reachable path, and group all density-connected points into the same cluster until the cluster to which the core point belongs no longer expands. S343. Repeat step S342 for the remaining unprocessed core points until all core points are assigned to a certain cluster. Finally, output a set of damage candidate regions consisting of all clusters, with each cluster corresponding to an independent damage candidate region.

8. The method for rapid detection of localized pits in ports based on UAV laser point cloud data according to claim 1, characterized in that, S4 includes: S41. Traverse each candidate damage region in the set of candidate damage regions, and eliminate noisy candidate regions based on the number of points in each candidate damage region. S42. Traverse each damage candidate region in the damage candidate region set, and eliminate damage candidate regions with insufficient elevation fluctuations based on the elevation range of each damage candidate region. S43. Traverse each candidate damage region in the set of candidate damage regions, calculate the maximum horizontal span of each candidate damage region based on its spatial range on the horizontal plane, and remove data points within the region formed by the merging of dock boundaries, range anomalies, or multiple adjacent anomaly regions. S44. Traverse each damage candidate region in the damage candidate region set. Based on the horizontal distance relationship between each damage candidate region and the reference plane point cloud, merge the neighboring reference plane point clouds into the damage candidate region to construct a complete pit damage region point cloud set. S45. Calculate the center point of the crater based on the spatial coordinates of all data points in the point cloud set of the complete crater damage area, and save each crater as an independent point cloud file.

9. The method for rapid detection of localized pits in ports based on UAV laser point cloud data according to claim 8, characterized in that, S42 includes: S421. Obtain the elevation range of data points in each damage candidate region in the damage candidate region set, and calculate the elevation change range of the damage candidate region. S422. Compare the elevation change range of the candidate damaged area with the minimum elevation change threshold value. If the elevation change range is less than the minimum elevation change threshold value, the candidate damaged area is determined to have insufficient elevation fluctuation and is removed.

Images