An unmanned aerial vehicle three-dimensional reconstruction point cloud scene semantic segmentation method based on a boundary point perception network

Abstract

The application discloses a kind of unmanned aerial vehicle three-dimensional reconstruction point cloud scene semantic segmentation methods based on boundary point perception network, it is related to three-dimensional point cloud processing technical field. The original image data of the region to be detected is obtained, the original three-dimensional point cloud data is obtained after three-dimensional reconstruction, input the BANet network trained, and the semantic prediction result is obtained after semantic segmentation;The semantic prediction result is constrained using cross-entropy loss, and the semantic true value of the region to be detected is obtained, and the three-dimensional geographic information of the region to be detected is obtained according to the semantic true value.The application embeds local context aggregator, global feature propagator and boundary point perceiver in BANet, and works collaboratively, solves the problem of the precision of semantic segmentation caused by complex scene content, noise and geometric distortion phenomenon in urban three-dimensional reconstruction point cloud, and promotes the learning expression of semantic features of different objects in large-scale three-dimensional point cloud space obtained by network three-dimensional reconstruction of unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud processing technology, and in particular to a semantic segmentation method for UAV 3D reconstruction point cloud scenes based on boundary point perception networks. Background Technology

[0002] With the rapid development of 3D geospatial data acquisition technology, 3D reconstructed point cloud scenes have become one of the main sources of large-scale urban 3D geographic information, playing a crucial role in urban intelligent modeling, autonomous driving, and urban planning. Accurate semantic information is a prerequisite for applying these point clouds to practical cognitive applications. However, as the scale of the scene increases, the semantic segmentation task of 3D reconstructed point clouds becomes more challenging: on the one hand, the complexity of elements in a 3D point cloud scene increases with changes in the type, shape, and structure of features within the scene. On the other hand, the large scale and high degree of dynamism of the scene make the quality of reconstructed point cloud data more susceptible to interference from noise, occlusion, and other issues, resulting in noise and geometric distortion. These problems are particularly prominent in complex urban scenes, placing higher demands on the expression, integration, and association of spatial semantic relationships in outdoor 3D point cloud scenes.

[0003] In point cloud semantic segmentation methods based on 3D geometric feature learning, current research mainly focuses on exploring feature extraction and local geometric structure learning suitable for the irregular characteristics of point clouds using deep learning methods. However, due to the large computational cost, these methods are difficult to apply to the semantic segmentation of large-scale outdoor point clouds. Some researchers have avoided the huge computational cost of using the entire outdoor point cloud as input by cutting the original point cloud into multiple blocks and randomly sampling 3D point sets for processing. However, this operation easily destroys the geometric structure of the point cloud and is very time-consuming when processing massive point clouds. To directly process large-scale outdoor 3D point clouds, current methods (Landrieu et al., 2018; Huetal., 2020) extract local information of the point cloud by establishing the relationship between adjacent points, thus achieving semantic segmentation of large-scale outdoor point clouds. However, large-scale outdoor reconstructed point clouds exhibit complex scene content, a large amount of noise, and geometric distortion. To obtain accurate semantic information, simply extracting local information is far from sufficient. It is necessary to further improve the reasoning mechanism for complex structural relationships and obtain noise-robust contextual information.

[0004] Therefore, this paper proposes a semantic segmentation technology for large-scale outdoor 3D reconstruction point clouds. This solution can cover the entire process of efficient processing of large-scale point clouds, preservation of complete geometric structure, reasoning of complex spatial structure, extraction of robust contextual information, and accurate semantic segmentation. It solves the technical shortcomings of existing methods, such as high computational cost, easy destruction of structure, weak anti-interference ability, and difficulty in adapting to complex urban scenarios by relying only on local information. This is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a semantic segmentation method for UAV 3D reconstructed point cloud scenes based on boundary point perception network. It aims to construct BANet and enable the local context aggregator, global feature propagator and boundary point perceiver to work together to deal with the semantic segmentation task of outdoor 3D reconstructed point clouds with a lot of noise and geometric distortion, and promote the network to learn and express the semantic features of different objects in the 3D point cloud space.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A semantic segmentation method for UAV 3D reconstruction point cloud scenes based on boundary point awareness networks includes the following steps: S1 acquisition steps: Acquire the original image data of the area to be detected; S2 Reconstruction Steps: Perform 3D reconstruction on the original image data to obtain the original 3D point cloud data of the area to be detected; S3 segmentation steps: Input the original 3D point cloud data into the trained BANet network for semantic segmentation to obtain semantic prediction results; the semantic prediction results represent the 3D model prediction results of the region to be detected at multiple scales; S4 constraint step: Use cross-entropy loss to constrain the semantic prediction results and obtain the semantic truth value of the region to be detected; S5 expression steps: Obtain the three-dimensional geographic information of the area to be detected based on semantic truth values.

[0007] Optionally, in the above method, the raw image data in step S1 includes image data, native image parameters, and native auxiliary data; wherein, the image data is a visible light RGB image acquired by the UAV.

[0008] The above method, optionally, includes the S2 reconstruction step, which specifically includes: S21 Data Verification Steps: Input verification and invalid frame removal are performed on the original image data to obtain standard image data; S22 Feature extraction steps: Extract local features from standard image data, perform bidirectional feature matching and mismatch removal to obtain image-related data; S23 coordinate restoration steps: Based on image association data, the SfM algorithm is used to calculate and restore the 3D coordinates of key points, obtain the corresponding camera pose, and generate sparse color point cloud. S24 point cloud reconstruction steps: Based on camera pose and sparse color point cloud, full pixel depth estimation and reverse derivation are performed using MVS multi-view stereo matching to generate dense 3D point cloud. S25 Filtering and Removal Steps: Filter the dense 3D point cloud to remove distortion noise and error noise, and output the original 3D point cloud data of the area to be detected.

[0009] Optionally, in the above method, the BANet network in S3 is a symmetrical five-layer coding-decoding structure, including a first coding layer, a second coding layer, a third coding layer, a fourth coding layer, a fifth coding layer, a first decoding layer, a second decoding layer, a third decoding layer, a fourth decoding layer, and a fifth decoding layer connected in sequence. In this design, a global feature propagator is embedded between the fifth encoding layer and the first decoding layer, and the fifth encoding layer, the global feature propagator, and the first decoding layer are connected in sequence. Local feature aggregators are embedded after the first, second, third, and fourth encoding layers, and each encoding layer and each local feature aggregator is connected alternately in sequence. Boundary point perceptrons are embedded after the first, second, third, and third decoding layers, and each decoding layer and each boundary point perceptron is connected alternately in sequence.

[0010] The above method, optionally, uses any local feature aggregator in the BANet network to perform the specific process shown below: Data acquisition steps: Obtain feature tensors and 3D point position information, and extract global context information of the 3D point set through feature tensors; wherein, the feature tensor is obtained by encoding the 3D point set by the previous encoding layer at the current local feature aggregator embedding position; Spatial encoding steps: Based on the K nearest neighbor algorithm, local feature encoding is performed on the 3D point position information to obtain local spatial position representation data, and local position encoding is performed on the global context information to obtain enriched context representation data; Adaptive aggregation step: Through a gate mechanism, local spatial location representation data and enriched contextual representation data are guided to undergo context-adaptive aggregation to obtain the initially enhanced 3D point features; Local enhancement step: Perform local enhancement operation and extract more information on the 3D point features after the initial enhancement to obtain the 3D point features after the second enhancement; Residual fusion steps: The global context information of the 3D point set and the 3D point features after secondary enhancement are fused using residual connections to obtain the final locally enhanced features.

[0011] In the above method, optionally, the process expression formula for context-adaptive aggregation in the adaptive aggregation step is as follows: ; In the formula, Point The three-dimensional point features after initial enhancement; point The number of 3D points in the neighborhood, Point Encoding features of nearest neighbors in the neighborhood space Represents the Hadama product. This represents the contextual importance score of nearest neighbors in the local space within a 3D point neighborhood. This represents the position of any point in the 3D point set within the current coding layer. In the neighborhood space any nearest neighbor position.

[0012] Optionally, the global feature propagator in the BANet network can be used to perform the following specific process: Dimension extraction steps: Obtain the encoded features output by the fifth encoding layer, extract the global context information of the encoded features along the channel dimension based on the parallel pooling structure, and obtain the parallel feature vector corresponding to the parallel pooling structure; Feature propagation steps: A weight-sharing multilayer perceptron is used to apply parallel feature vectors to establish internal point feature associations, resulting in associated feature vectors; Fusion mapping steps: Based on the channel dimension, the associated feature vectors are fused using convolutional fusion and then mapped using the sigmoid activation function to obtain the importance weight vector; Global selection step: Selectively enhance or suppress the coding features output by the fifth coding layer according to the importance weight vector to obtain global coding features.

[0013] The above method, optionally, has the following specific formula for calculating the importance weight vector in the fusion mapping step: ; The specific formula for calculating global coding features is as follows: ; in, This represents an importance weight vector, where each value indicates its importance in encoding features. The importance of corresponding point features in the entire three-dimensional space; This represents the sigmoid activation function; This indicates a weight-sharing multilayer perceptron; Represents parallel eigenvectors; This represents the matrix addition operation; Represents global encoding features; This represents the Hadama product.

[0014] Optionally, the boundary point perceptron in the BANet network is used to calculate the semantic labels of the corresponding coding layers, and the specific calculation formula is as follows: ; In the formula, Indicates the first n The first coding layer i The semantic truth value of a three-dimensional point; This represents the average pooling operation; Indicates the first n Three-dimensional points of each coding layer In the n The local neighborhood space of the -1 coding layer, the first n In the coding layer feature, the th i Three-dimensional points It is the first n -1 is obtained by aggregating all three-dimensional point sets in the local space of the coding layer; , Indicates the first n Subsampled point cloud at the scale of each coding layer; n This represents the number of coding layers, and the values ​​correspond one-to-one with each coding layer. n =5.

[0015] Optionally, in the above method, any boundary point perceptron in the BANet network uses a boundary point perceptron loss function to guide the decoding stage of the corresponding decoding layer; the specific calculation formula of the boundary point perceptron loss function is as follows: ; in, Indicates the first n The boundary point sensing loss function value of each decoding layer. n =5; Indicates the first n True values ​​of boundary points in a point cloud within a sub-sampling space; Indicated by point The K-neighborhood space centered on the center; The calculation is of variables a and b Euclidean distance in three-dimensional space , They represent the first n Midpoint of each sub-sampling space and Features; This represents the temperature parameter in the comparative learning; for each boundary point Let its K-neighbor points The data is divided into positive and negative samples, with positive sample points represented as follows: Negative sample points are represented as .

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention provides a semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network, which has the following beneficial effects: (1) Global feature robustness: Based on the outdoor 3D point cloud segmentation network RandLANet, a global feature propagator is designed and embedded to extract anti-interference context information, enhance the network's global scene perception capability, and effectively improve the segmentation accuracy of large-size targets in 3D point cloud scenes. (2) Boundary semantic optimization: To address the boundary semantic confusion caused by geometric distortion, a boundary point perceptron is introduced at the decoding end to construct local point set semantic constraints, mine multi-scale boundary features, and balance boundary distinguishability with semantic consistency of similar targets. (3) Multi-scale collaboration: Multi-scale semantic supervision modules are configured in the multi-layer decoding layer to enhance the cross-scale target recognition capability. The training phase is linked with the boundary perception mechanism, and the overall semantic feature learning quality is optimized by relying on the backpropagation closed loop. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of the 3D point cloud scene semantic segmentation network based on boundary point awareness disclosed in this invention; Figure 2 This is a flowchart illustrating a semantic segmentation method for point cloud scenes in UAV 3D reconstruction based on a boundary point sensing network disclosed in this invention. Figure 3 This is a flowchart illustrating the workflow of the local feature aggregator disclosed in an embodiment of the present invention; Figure 4 This is a detailed schematic diagram illustrating the operation flow of local feature encoding in the local feature aggregator workflow disclosed in this embodiment of the invention; Figure 5 This is a detailed schematic diagram illustrating the operation flow of local position encoding in the local feature aggregator workflow disclosed in this embodiment of the invention; Figure 6This is a detailed operational flowchart illustrating the context-adaptive aggregation disclosed in an embodiment of the present invention; Figure 7 This is a flowchart illustrating the workflow of the global feature propagator disclosed in an embodiment of the present invention. Figure 8 This is a schematic diagram of boundary point feature constraints disclosed in an embodiment of the present invention; Figure 9 This is a diagram showing the overall SUM region data disclosed in an embodiment of the present invention; Figure 10 This is a scene illustration of a 3D reconstructed point cloud using SUM slices, as disclosed in an embodiment of the present invention; wherein, Figure 10 (a) is a schematic diagram of the 3D reconstructed point cloud scene using slices as the training set. Figure 10 (b) Schematic diagram of the 3D reconstructed point cloud scene as a validation set. Figure 10 (c) Schematic diagram of the 3D reconstructed point cloud scene as a test set; Figure 11 This is a semantic truth representation diagram of a SUM slice 3D point cloud disclosed in an embodiment of the present invention; wherein, Figure 11 (a) is a schematic diagram of the semantic ground truth of the sliced ​​3D point cloud used as the training set. Figure 11 (b) A schematic diagram of the semantic truth of the sliced ​​3D point cloud as a validation set. Figure 11 (c) A schematic diagram of the semantic truth of the sliced ​​3D point cloud as a test set; Figure 12 This is a qualitative comparison diagram of the semantic segmentation results of BANet and RandlaNet on the SUM test set disclosed in the embodiments of the present invention; wherein, Figure 12 (a) is a schematic diagram of a 3D point cloud scene for the SUM test set. Figure 12 (b) is a semantic truth diagram of the SUM test set. Figure 12 (c) is a schematic diagram of the segmentation results based on RandaNet. Figure 12 (d) is a schematic diagram of the segmentation results based on BANet; Figure 13 This is a partial comparison diagram of the semantic segmentation results of BANet and RandlaNet on the SUM test set disclosed in the embodiments of the present invention; wherein, Figure 13 (a) is a magnified view of a 3D point cloud scene of the SUM test set. Figure 13 (b) is a locally magnified semantic truth diagram of the SUM test set. Figure 13 (c) is a schematic diagram of the local segmentation results based on RandlaNet. Figure 13 (d) is a schematic diagram of the local segmentation results based on BANet. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0021] See Figure 1 As shown, this invention discloses a semantic segmentation method for UAV 3D reconstruction point cloud scenes based on boundary point perception networks, including the following steps: S1 acquisition steps: Acquire the original image data of the area to be detected; S2 Reconstruction Steps: Perform 3D reconstruction on the original image data to obtain the original 3D point cloud data of the area to be detected; S3 segmentation steps: Input the original 3D point cloud data into the trained BANet network for semantic segmentation to obtain semantic prediction results; the semantic prediction results represent the 3D model prediction results of the region to be detected at multiple scales; S4 constraint step: Use cross-entropy loss to constrain the semantic prediction results and obtain the semantic truth value of the region to be detected; S5 expression steps: Obtain the three-dimensional geographic information of the area to be detected based on semantic truth values.

[0022] Optionally, the raw image data in step S1 includes image data, native image parameters, and native auxiliary data; wherein, the image data is a visible light RGB image acquired by the UAV.

[0023] Raw image data is the underlying input for 3D reconstruction, point cloud semantic segmentation, and scene modeling. It refers to the original image data directly collected by the sensor without undergoing manual / algorithm post-processing such as noise reduction, cropping, enhancement, registration, and fusion. Combined with 3D vision application scenarios, it is completely composed of three major categories: core subject image, sensor metadata, and associated raw auxiliary data.

[0024] The core subject image, or raw image data, is the visible light RGB image acquired by the UAV. Sensor metadata, or native image parameters, are native parameters embedded in the image file or output from the same source. Without this data, camera pose calculation and point cloud registration cannot be completed, and it falls under the broad category of raw image data. Associated native auxiliary data consists of unprocessed image data from the same source as the raw image data, including raw mask images, RAW raw negatives, and raw frame sequences, used for invalid removal and low-quality screening of raw image data.

[0025] Based on raw image data, a structured 3D point cloud can be generated through a visual 3D reconstruction pipeline, following a construction process of data verification, feature association, geometric calculation, dense reconstruction, and output optimization. Optional S2 reconstruction steps specifically include: S21 Data Verification Step: Input verification and invalid frame removal are performed on the raw image data to obtain standard image data. This step integrates the raw image, camera intrinsics, and pose metadata to complete input verification, removes invalid frames such as blurry, overexposed, and occluded frames, ensures the integrity and consistency of the reconstructed input data, provides a compliant benchmark data source for the entire process, and serves as the prerequisite for subsequent feature extraction and geometric calculation.

[0026] S22 Feature Extraction Steps: Extract local features from standard image data, perform bidirectional feature matching and mismatch removal to obtain image association data. Establish geometric relationships between multi-view images based on association constraints to prepare for subsequent camera pose calculation.

[0027] S23 Coordinate Restoration Steps: Based on image association data, the SfM algorithm is used to calculate and restore the 3D coordinates of key points, obtain the corresponding camera pose, and generate a sparse color point cloud. The SfM algorithm is used to solve for the camera's intrinsic and extrinsic parameters and global pose. Triangulation is used to restore the 3D coordinates of key points, generating a sparse color point cloud covering the scene's feature regions. The output camera pose and sparse color point cloud are necessary preparatory steps for dense reconstruction.

[0028] S24 Point Cloud Reconstruction Steps: Based on camera pose and sparse color point cloud, full-pixel depth estimation and reverse derivation are performed using MVS multi-view stereo matching to generate a dense 3D point cloud. Full-pixel depth estimation is completed using MVS multi-view stereo matching, and global spatial coordinates are derived in reverse. Image color, intensity, and other attribute information are fused to generate a noisy initial scene dense 3D point cloud, completing the dimensional upgrade from discrete keypoints to a global point cloud.

[0029] The S25 filtering and removal step involves filtering the dense 3D point cloud to remove distortion and error noise, outputting the original 3D point cloud data of the area to be detected. This step completes the entire process, achieving a closed loop of transformation from raw image data to usable 3D point cloud.

[0030] like Figure 1 As shown, the BANet network in S3 is a symmetrical five-layer coding-decoding structure, including the first coding layer, the second coding layer, the third coding layer, the fourth coding layer, the fifth coding layer, the first decoding layer, the second decoding layer, the third decoding layer, the fourth decoding layer, and the fifth decoding layer connected in sequence. In this design, a global feature propagator is embedded between the fifth encoding layer and the first decoding layer, and the fifth encoding layer, the global feature propagator, and the first decoding layer are connected in sequence. Local feature aggregators are embedded after the first, second, third, and fourth encoding layers, and each encoding layer and each local feature aggregator is connected alternately in sequence. Boundary point perceptrons are embedded after the first, second, third, and third decoding layers, and each decoding layer and each boundary point perceptron is connected alternately in sequence.

[0031] Combination Figure 1 The invention discloses a semantic segmentation method for UAV 3D reconstruction point cloud scenes based on a boundary point perception network. The BANet network used in this method takes a 3D point cloud scene as input and utilizes a symmetrical 5-layer encoder-decoder structure for end-to-end feature learning and semantic prediction. The encoder structure contains 5 encoding stages, and the decoder structure contains 5 decoding stages. Each of the first four encoding stages contains an encoding layer and a local feature aggregator, while the fifth encoding stage contains an encoding layer and a global feature propagator. The decoder contains 5 decoding stages. Each of the first four decoding stages contains a decoding layer and a boundary point perceptron, and the last decoding stage directly performs point-by-point category prediction.

[0032] In this process, each coding layer uses random sampling to quickly extract multi-scale 3D point features. Each coding layer uses a downsampling operation with a sampling rate of 4 to obtain the coding features of the next layer (i.e., the 1st layer). n ( The layer coding feature is the first n (This is four times the number of feature points encoded in layer +1). In each of the first four encoding stages, a Local Feature Aggregator (LFA) is embedded to extract contextual information from the local space, enhancing the network's ability to learn complex local structures while mitigating information loss caused by random sampling. Furthermore, skip connections are used to fuse high-level semantic features from the decoding process and low-level geometric features from the encoding process to preserve detailed information.

[0033] like Figure 2As shown, unlike RandLANet, BANet adds a Global Feature Propagator (GFP) at the top of the encoding layer (the fifth encoding layer). This GFP models long-distance contextual relationships in 3D space by establishing global feature connections between 3D points, thereby enhancing the perception of the overall scene by high-level semantic features. In the feature decoding stage, a Boundary-aware Loss (BAL) is added after each decoding layer. This BAL uses contrastive learning to mine boundary point information of multi-scale features and further strengthens the learning of semantic features during backpropagation, achieving the goal of enhancing the semantic consistency of similar 3D points in local space and the semantic distinguishability of different classes of 3D points. Finally, BANet adds a classifier after the features of the decoding layer at each level, obtaining 3D model prediction results at five different scales, and uses Cross Entropy Loss (CEL) to constrain the multi-scale semantic prediction results. In the BANet network disclosed in this invention, LFA and GFP complement each other, providing stable feature abstraction capabilities for object-level semantic prediction; BAL and CEL in any decoder in the first four decoding stages work together to communicate with LFA and GPF during backpropagation, jointly promoting the learning of semantic features at each stage, thereby improving the robustness of object semantic recognition in 3D point cloud scenes.

[0034] Next, this invention will describe the global feature propagator, local feature aggregator, and boundary point perceptron in the BANet network.

[0035] Reference Figure 3 As shown, the specific process of any local feature aggregator in the BANet network is as follows: Data acquisition steps: Obtain feature tensors and 3D point position information, and extract global context information of the 3D point set through feature tensors; wherein, the feature tensor is obtained by encoding the 3D point set by the previous encoding layer at the current local feature aggregator embedding position; Spatial encoding steps: Based on the K nearest neighbor algorithm, local feature encoding is performed on the 3D point position information to obtain local spatial position representation data, and local position encoding is performed on the global context information to obtain enriched context representation data; Adaptive aggregation step: Through a gate mechanism, local spatial location representation data and enriched contextual representation data are guided to undergo context-adaptive aggregation to obtain the initially enhanced 3D point features; Local enhancement step: Perform local enhancement operation and extract more information on the 3D point features after the initial enhancement to obtain the 3D point features after the second enhancement; Residual fusion steps: The global context information of the 3D point set and the 3D point features after secondary enhancement are fused using residual connections to obtain the final locally enhanced features.

[0036] In semantic segmentation networks, the encoding layer extracts multi-scale contextual information through layer-by-layer downsampling operations; however, downsampling inevitably results in the loss of important information. Combined with... Figure 3 As shown, this invention embeds a local feature aggregator after each coding layer of the BANet network to gradually increase the receptive field of each coding layer to obtain richer local contextual information, thereby enhancing the learning ability of complex local structures of point clouds.

[0037] For any local feature aggregator embedded after the coding layer, it uses a three-dimensional point set Feature tensors obtained after multilayer perceptron encoding As input, and then processed by a point-to-point shared multilayer perceptron to extract global context information of points in three-dimensional space. To enhance the network's ability to learn local features, a local spatial coding structure is used to mine the local contextual information of 3D points in the feature space. During this process, local positional coding information is explicitly embedded to strengthen the relative spatial relationships of 3D point features in the local space. To select contexts with richer information, an adaptive approach is used to aggregate the contextual information of 3D points in the local space, thereby improving the network's ability to recognize complex local structures.

[0038] Spatial context encoding primarily learns the complex structure of a local 3D space by computing mutual positional encodings and feature encodings within that local space. The operational flow of mutual positional encoding in local space is as follows: Figure 4 As shown, for the input 3D point position information First, the K-Nearest Neighbor (KNN) algorithm is used to find the nearest neighbor to the first neighbor. i Three-dimensional points In three-dimensional space, the K nearest neighbors are selected. The KNN algorithm is one of the simplest classification algorithms in machine learning. It selects the K nearest neighbors to represent or enhance the attributes or features of the target sample point by calculating the distance between each sample point and other sample points. In this invention, Euclidean distance is used to obtain the K nearest neighbors. Find the K nearest neighbors and use them as To obtain richer local contextual information from the local spatial content.

[0039] Assumption Centered on, its three-dimensional coordinates are represented as , The K-neighborhood set is represented as , hereN It has no practical representative significance. The character represents the entire point set. Each neighboring point... The three-dimensional coordinates are represented as Based on three-dimensional points in the neighborhood space and center point The relative positions of each neighboring point in the K-neighborhood space. The location code is: ; In the formula, Representing a three-dimensional point and its nearest neighbors The relative distance between positions in three-dimensional space Representing a three-dimensional point and its nearest neighbors Euclidean distance in three-dimensional space; This represents the merging operation along the channel dimension. To learn the spatial relationships between 3D points in the neighborhood space, we further utilize a Multilayer Perceptron (MLP) operation to jointly encode the merged different position representations, ultimately obtaining the neighborhood points. Relative to the center point Position encoding in local space This refers to data representing local spatial locations.

[0040] Similarly, refer to Figure 5 As shown, for a three-dimensional point coding features We used the KNN algorithm to find the features of its neighborhood points. Then, it is combined with the location features of the neighboring points obtained by location encoding. By merging along the channel dimension, we obtain the enhanced feature vectors of the neighborhood points. Ultimately, by constraining the relative positions in the local space, the features of neighboring points in the neighborhood space can yield richer contextual representations (i.e., enriched contextual representation data): ; After obtaining the 3D point features within the neighborhood, a key issue is how to effectively integrate this information for the semantic representation of the 3D point features. Previous methods typically employ max pooling or average pooling to synthesize the information from the neighborhood point set, but these methods often lead to the loss of a significant amount of important information. To effectively integrate useful neighborhood point information in the local space to enrich the feature representation of the center point, we adopt an adaptive pooling approach to fuse the contextual information of the local space, referring to... Figure 6As shown, the specific formula for expressing the context-adaptive aggregation process in the adaptive aggregation step is as follows: ; In the formula, Point The three-dimensional point features after initial enhancement; point The number of 3D points in the neighborhood, Point Encoding features of nearest neighbors in the neighborhood space Represents the Hadama product. This represents the contextual importance score of nearest neighbors in the local space within a 3D point neighborhood. This represents the position of any point in the 3D point set within the current coding layer. In the neighborhood space any nearest neighbor position.

[0041] Combination Figure 6 Adaptive pooling primarily calculates the importance score of the context information of neighboring points to the context of the center point in the local space, and then aggregates the context information of the neighboring points and the center point based on this importance score. Given a set of features of neighboring points... By using a multilayer perceptron to calculate the contextual relevance of 3D points within a neighborhood, the relevance features of all points in the neighborhood space are obtained. Then, the softmax activation function is used to normalize the correlation features of the 3D points in the neighborhood space, mapping the correlation scores of all points to the range [0,1]. It is assumed that... Represents the nearest neighbor in the neighborhood space The encoding features, this process can be expressed by the formula: ; in, N p Point The number of 3D points in the neighborhood; Representing a three-dimensional point Nearest neighbor in the neighborhood Context importance scoring in local space. Using the softmax operation helps to preserve the most important feature response regions in local space, while reducing the impact of noisy or redundant features on the representation of central point features.

[0042] The context importance score of each neighboring point in the neighborhood space can be used as an elastic mask to adaptively select and aggregate the features of the neighboring points that contribute the most to the context of the center point. We guide the aggregation of local spatial features through a gate mechanism, and finally obtain the 3D point features enhanced with local spatial context using a formula for context-adaptive aggregation. (i.e., the three-dimensional point features after the initial enhancement).

[0043] To expand the receptive field of the features, we perform further local enhancement operations on the locally context-enhanced features to extract local contextual information over a larger range (i.e., the 3D point features after secondary enhancement). Finally, residual connections are used to integrate the initial point cloud features... Features after two rounds of local enhancement The fusion yields the final locally enhanced features. By combining two spatial context encoding operations and a context adaptive aggregation operation, BANet can perceive local spatial features within a larger receptive field, thereby improving the model's ability to recognize complex local structures.

[0044] Reference Figure 7 As shown, the global feature propagator in the BANet network performs the following specific process: Dimension extraction steps: Obtain the encoded features output by the fifth encoding layer, extract the global context information of the encoded features along the channel dimension based on the parallel pooling structure, and obtain the parallel feature vector corresponding to the parallel pooling structure; Feature propagation steps: A weight-sharing multilayer perceptron is used to apply parallel feature vectors to establish internal point feature associations, resulting in associated feature vectors; Fusion mapping steps: Based on the channel dimension, the associated feature vectors are fused using convolutional fusion and then mapped using the sigmoid activation function to obtain the importance weight vector; Global selection step: Selectively enhance or suppress the encoded features output by the fifth coding layer according to the importance weight vector to obtain global features.

[0045] Based on the local feature aggregator disclosed in the previous embodiment, local context-enhanced features are obtained. Downsampling operations can acquire encoded features at different scales. In the BANet network structure with its five encoders, the highest-level encoded features obtained after four downsampling operations represent sparse point features that maximally summarize the local features. To enhance the correlation between these local features, BANet introduces a global feature propagator to model long-distance contextual relationships in three-dimensional space.

[0046] Combination Figure 7 The global feature propagator uses the encoded features from the highest layer (i.e., the fifth coding layer). Using global context encoding as input, the association weights between point features are obtained, and these association weights are propagated to the original input feature space to obtain the globally encoded features. Specifically, for the input three-dimensional features First, a parallel pooling structure (maximum pooling layer and average pooling layer) is used to extract global contextual information of different representations along the channel dimension, resulting in parallel feature vectors. Max pooling calculates the maximum value of all point features along the channel dimension, retaining only the most useful information for each point feature across all channels; average pooling calculates the average value of point features along the channel dimension, preserving background information. N Dimensions representing features C Indicates the number of feature channels.

[0047] Optionally, the formula for calculating the importance weight vector in the fusion mapping step is as follows: ; The specific formula for calculating global features is as follows: ; in, This represents an importance weight vector, where each value indicates its importance in encoding features. The importance of corresponding point features in the entire three-dimensional space; This represents the sigmoid activation function; This indicates a weight-sharing multilayer perceptron; Represents parallel eigenvectors; This represents the matrix addition operation; Represents the Hadama product; This represents the global encoding feature.

[0048] Combination Figure 7 In obtaining parallel feature vectors Then, two weight-shared multilayer perceptrons are used to apply the following to... This involves densely propagating the features of each point throughout the feature space, establishing the interrelationships between the point features within the vector, and obtaining the associated feature vector. To represent more comprehensive global context information, we merge the two along the channel dimension, using... The convolutional layer fuses the global context relevance vectors obtained from the two branches and uses a sigmoid activation function to map them to the range (0,1), thus obtaining the encoded features output from the fifth encoding layer. Importance weight vector of midpoint features , Each value in represents its The importance of corresponding point features in the entire 3D space. By inputting features With weight vector Multiply, Point features can be selectively enhanced or suppressed to ultimately obtain globally encoded features with long-range contextual dependencies. .

[0049] Optionally, the boundary point perceptron in the BANet network is used to calculate the semantic labels of the corresponding coding layers. The specific calculation formula is as follows: ; In the formula, Indicates the first n The first coding layer i The semantic truth value of a three-dimensional point; This represents the average pooling operation; Indicates the first n Three-dimensional points of each coding layer In the n The local neighborhood space of the -1 coding layer, the first n In the coding layer feature, the th i Three-dimensional points It is the first n -1 is obtained by aggregating all three-dimensional point sets in the local space of the coding layer; , Indicates the first n Subsampled point cloud at various scales; n This represents the number of coding layers and corresponds to each coding layer. n =5.

[0050] The semantic labels of 3D point cloud semantic segmentation data can clearly represent the region extent of different object classes, a characteristic that provides valuable clues for representing boundary information. For example... Figure 8 As shown, 3D points with the same semantic label within their neighborhood are internal points of an object, while the neighborhood of a boundary point contains 3D points of different semantic categories, as shown by the gray dashed circle in the encoding stage. From this, we can infer that points exhibiting semantic jumps (discontinuities) within a local region are boundary points between objects of different categories. Therefore, we will use the ground truth of the boundary point set of the input 3D point cloud. and predicted value Represented as: ; In the formula, It is a three-dimensional point set One point in the middle, Indicated by The surrounding space centered on, For a point in the neighborhood space, Point The semantic truth value, Point The semantic truth value.

[0051] To better capture the boundary features of point clouds, we explore boundary information during multiple downsampling stages in the encoding process to extract geometric structure information from multi-scale encoded features. However, due to the uncertainty of random sampling, semantic ground truth labels at different sampling scales are difficult to define accurately, thus failing to obtain accurate boundary point information in the subsampling space. Therefore, this paper employs a boundary point extraction method to accurately acquire boundary point information in subsampling spaces at different scales.

[0052] Assume the first n Subsampled point clouds at the scale of each coding layer are represented as follows: The subsampling spaces at different scales originate from different coding stages. For the original input point cloud, the subsampling point cloud of the first coding stage can be represented as... As the coding layer deepens, each subsampling space is obtained. Each three-dimensional point in semantic truth value Perception of the geometric structure within a scene is crucial. Because the first n Each coding layer sub-sampling points From the first n The semantic ground truth of the original input point cloud is generated by aggregating point features from one coding layer. Therefore, downsampling can be used to iteratively generate the semantic ground truth of each sub-sampling space. The result after one-hot processing This represents the semantic label of the first encoding layer, and has Then the first n The semantic labels of each coding layer can be obtained by the calculation formula of the corresponding coding layer in the above embodiments, which will not be repeated here.

[0053] According to the n The semantic label calculation formula for the first coding layer can iteratively calculate the semantic distribution in each sub-sampling space, that is, in each sub-sampling space, the semantic label of the first coding layer... k The value at each position represents its corresponding location in the input point cloud. k The probability distribution of each category. To obtain unique semantic labels for 3D points in the subsampling space, the following method is used... To enhance the expression of dominant features in the local space (such as...) Figure 8 As shown in the histogram of the decoding stage, the category with the highest proportion in the neighborhood is the semantic category of the boundary point. Then, the ground truth of the boundary point at each scale is obtained through the ground truth calculation formula of the boundary point set of the 3D point cloud.

[0054] Optionally, each boundary point perceptron in the BANet network uses a boundary point perceptron loss function to guide the decoding stage of the corresponding decoding layer; the specific calculation formula of the boundary point perceptron loss function is as follows: ; in, Indicates the first n The boundary point sensing loss function value of each decoding layer; Indicates the first n True values ​​of boundary points in a point cloud within a sub-sampling space; Indicated by point The K-neighborhood space centered on the center; The calculation is of variables a and b Euclidean distance in three-dimensional space , They represent the first n Midpoint of each sub-sampling space and Features; This represents the temperature parameter in the comparative learning; for each boundary point Let its K-neighbor points The data is divided into positive and negative samples, with positive sample points represented as follows: Negative sample points are represented as .

[0055] To effectively learn the boundary point features between objects in 3D space, BANet employs a boundary point-aware loss function during the decoding phase. This function guides the feature learning of boundary points by measuring the semantic consistency of point features in the neighborhood space. For example... Figure 1 As shown in the decoding stage, this loss function promotes boundary point learning by encouraging semantic cohesion among 3D points of the same class within the neighborhood and enhancing the semantic difference between boundary points and neighboring points. This is essentially a contrastive learning approach, implicitly supervising the learning of target features by reducing the distance between positive examples and increasing the distance between negative examples. Corresponding to the multi-scale boundary point feature learning in the encoding stage, we add a BAL after the decoding layer at each scale to strengthen the learning of boundary points in each subsampling space.

[0056] Specifically, for points located on the object boundary 3D points similar to local spatial features are classified as belonging to the same category; conversely, 3D points dissimilar to the features of neighboring points in the local space are classified as boundary points. The purpose of BAL is to distinguish boundary points from non-boundary points, that is, to strengthen the constraint between boundary points and the features of points in their neighborhood, thereby separating them and obtaining the specific calculation formula of the boundary point perception loss function as described above.

[0057] By constraining boundary point features in different sub-sampling spaces using BAL, BANet can enhance the network's semantic discrimination ability for boundary point features, while improving the semantic consistency of features of similar points. Furthermore, throughout the network's operation, the boundary point features optimized by BAL will serve as constraints, further optimizing the learning of object semantic features during gradient backpropagation, thereby improving the overall semantic segmentation performance of the 3D scene.

[0058] Furthermore, to supervise the training process of BANet, this paper employs cross-entropy loss, a commonly used method in semantic segmentation tasks, to perform semantic supervision on the semantic prediction results obtained from decoding. For example... Figure 1 As shown, in the decoding process of BANet, we perform segmentation prediction on the semantic features obtained from subsampling spaces of different scales, and then use cross-entropy loss for semantic supervision to enhance the learning ability of semantic features in subsampling spaces of different scales.

[0059] Decoding features obtained from subsampling spaces of different scales We first add two convolutional units after each decoded feature (each convolutional unit includes one...) (A convolutional layer and a block normalization layer) yield multi-scale semantic features with richer contextual information. Then, a classifier is used to... Semantic prediction is performed, and the semantic prediction graph is normalized using the softmax function to obtain semantic prediction results for multiple sub-sampling spaces. Finally, the semantic prediction results of each sub-sampling space are compared with the semantic ground truth at the corresponding scale. The cross-entropy calculation can be expressed by the following formula: ; In the formula, C This represents the total number of semantic categories in the scene, compared to the number of feature channels. C The corresponding values ​​are the same, meaning that the number of feature channels and the total number of categories in the scene have a one-to-one correspondence and the same relationship. The first n Subsampling space of each coding layer; express midpoint The semantic truth value; Point Predicted as category c The probability diagram; This indicates an indicator function.

[0060] During network training, semantic prediction supervision and boundary point awareness constraints in each sub-sampling space work together to learn features at each stage, as expressed by the formula: ; In the formula, This indicates the number of boundary point sensing loss functions inserted in the decoding layer; As a weighting parameter, it is used to balance the importance of semantic supervision and boundary constraints in the overall task. By jointly combining semantic prediction supervision and boundary point-aware constraints, BANet can leverage the geometric structure information of objects to further optimize the learning of semantic features, thereby obtaining more accurate semantic segmentation results.

[0061] To verify the effectiveness of the proposed semantic segmentation method for UAV 3D reconstructed point cloud scenes based on boundary point awareness networks, experiments are conducted using the publicly available urban 3D reconstructed point cloud model semantic segmentation dataset SUM. In the following embodiments, the experimental data and some important parameter settings are first described. Then, the accuracy of BANet's semantic segmentation results on the SUM dataset is evaluated. Comparative experiments are designed to verify the advancement of the proposed 3D reconstructed point cloud scene semantic segmentation method in UAV 3D reconstructed point cloud semantic segmentation tasks.

[0062] The SUM dataset is a benchmark dataset for large-scale urban 3D semantic segmentation derived from 3D reconstruction of oblique aerial imagery by unmanned aerial vehicles (UAVs). The original aerial imagery data for this dataset was collected in the central area of ​​Helsinki, the capital of Finland, with a ground coverage area of ​​approximately 12 square kilometers. In 2021, Gao et al. used the commercial software ContextCapture to perform 3D reconstruction on multi-view aerial imagery, obtaining the original 3D point cloud data for SUM; Figure 9 The effective ground coverage area of ​​the 3D point cloud scene shown is approximately 8 square kilometers (4 square kilometers according to a 2:1 mapping scale). Figure 9 (a) is a schematic diagram of the outdoor 3D reconstructed point cloud scene in the central area of ​​Helsinki. Figure 9 (b) is a schematic diagram of the semantic truth of the outdoor three-dimensional point cloud in the central area of ​​Helsinki.

[0063] For ease of browsing and use, the entire 3D point cloud data is divided into 64 sliced ​​3D point cloud scenes for storage, with each scene covering an area of ​​approximately 250 square meters. For example... Figure 10 and Figure 11 As shown, in the semantic segmentation task, all the slice data is divided into three parts: the training set contains 40 slice data, the validation set contains 12 slice data, and the test set contains 12 slice data. Among them, Figure 10 (a) and Figure 11 (a) shows the 3D reconstructed point cloud scene image and the 3D point cloud semantic ground truth image from the training data, respectively. Figure 10 (b) and Figure 11(b) is the corresponding validation set. Figure 10 (c) and Figure 11 (c) is the corresponding training set.

[0064] After completing the data preparation, this embodiment deploys BANet within the PyTorch framework and trains the model using an NVIDIA GeForce RTX 2080 Ti GPU. BANet employs the Adam optimizer for parameter optimization, with an initial learning rate set to... To guide the network to converge faster, this embodiment employs a learning rate decay strategy of reducing the learning rate by 5% per epoch to adjust the learning rate of the BANet network at each stage, and sets the number of neighborhood points K to 16. Simultaneously, to enable parallel training on multiple datasets, this embodiment samples the 3D point cloud scene for each input, with the number of 3D points being... The 3D point cloud is used as the input to the network.

[0065] Furthermore, the Intersection over Union (IoU) ratio, a commonly used metric in semantic segmentation tasks, is employed to evaluate the accuracy of semantic segmentation results for 3D point cloud scenes. Specifically, the class-wise IoU ratio is used. IoU The semantic prediction of a 3D object is defined as the quotient of the intersection and union of its semantic truth value and the semantic prediction of that 3D object. It can be expressed by the formula: ; in, IoU c Indicates the first c The intersection-union ratio of semantics for three-dimensional objects of each category. P c Represents the semantic prediction value of a 3D object. G c Indicates the corresponding category c The semantic truth value of a three-dimensional object.

[0066] The segmentation precision for all categories is the average intersection-over-union (IoU) ratio for all categories, i.e., the mean intersection-over-union ratio (MUI). mIoU Assume the total number of categories is... A The average intersection-union ratio of semantic segmentation is calculated as follows: ; in, mIoU This represents the average intersection-union ratio (IUU) of semantic segmentation.

[0067] To evaluate the segmentation performance of BANet in the semantic segmentation task of outdoor 3D reconstructed point cloud scenes, this embodiment further compares the semantic segmentation accuracy of BANet with classic 3D point cloud segmentation algorithms PointNet and PointNet++, as well as algorithms for semantic segmentation of large-scale urban 3D point cloud scenes, such as RandaNet and SPG, on the large urban dataset SUM. The comparison results are shown in Table 1.

[0068] Table 1. Quantitative comparison of segmentation results of various comparison methods on the test set of the SUM dataset.

[0069] Referring to Table 1, compared with other methods, the BANet network used in this invention achieved the best segmentation results. mIoU (66.2%, mAcc: 79.2%) mIoU The score was improved by at least 4.8%. Among the methods listed in Table 1, PointNet can extract semantic information from 3D point clouds end-to-end; however, due to the lack of local feature learning, it struggles to capture detailed information in the scene, resulting in poor segmentation results. Building upon PointNet, PointNet++ enhances the network's ability to learn complex local scenes by aggregating point cloud features from local space, achieving better segmentation results than PointNet. mIoU The score improved by 12.6%, but the segmentation performance of this method is still limited due to the lack of an effective local feature aggregation strategy. RF-MRF obtains initial oversegmented blocks with local similarity by oversegmenting the 3D scene, and then uses a random forest classifier to further segment the initial oversegmented blocks to obtain semantic segmentation results. Compared with the classic end-to-end 3D point segmentation method, the segmentation accuracy of this method is further improved, but its cumbersome operation steps make it difficult to be widely used. RandlaNet is a semantic segmentation method designed for large-scale outdoor 3D laser point clouds. This method uses random sampling to greatly improve the efficiency of the network in learning features of large-scale point clouds. At the same time, it uses local spatial encoding to learn the geometric characteristics of 3D point sets in local space, thereby improving the model's ability to recognize local information. However, due to the lack of perception of global context in feature space and constraint of boundary point information, its segmentation results are still limited. In contrast, the BANet network disclosed in this invention improves the perception of the overall structure of the scene by establishing global context connections. At the same time, it implicitly extracts the boundary point features between objects in the 3D point cloud at different feature learning stages, and uses these as constraints to further optimize the learning of semantic features, thereby obtaining mIoU It achieves a good segmentation accuracy of 4.8%–29.4% higher than other methods.

[0070] Due to the complexity and large spatial range of outdoor scenes, multi-view Figure 3 The outdoor 3D model obtained through reconstruction is not only affected by noise and occlusion, but also by the scale differences and category distribution of different object classes in large-scale urban scenes, which can affect the accuracy of semantic segmentation of 3D point cloud scenes. Therefore, based on the content in Table 1, this invention will classify... IoU The scores are shown in Table 1 to evaluate the segmentation performance of different methods on various object categories. As shown in the first row of Table 1, PointNet can hardly recognize vehicles and ships in large 3D scenes. This may be because PointNet only focuses on learning global features in the 3D scene and ignores the extraction of local features. Benefiting from the improved local feature learning ability, PointNet++ improves the recognition performance of vehicles and ships by 0.5% and 1.6% respectively compared to PointNet. RandlaNet improves the network model's recognition performance of small objects, such as vehicles, by aggregating 3D point features in local space layer by layer. mIoU The score was 38.9%, higher than other methods. mIoU More than 1.6% higher; the ship's mIoU The score was 10.1, exceeding that of other methods. mIoU Over 2.7%. Compared to RandlaNet, BANet adds a global feature propagator and a boundary point perceptron, enhancing the network's ability to perceive global context information and improving the intra-class consistency and inter-class discriminability of overall point features in the local space. Therefore, the segmentation accuracy for large and small objects in the scene is further improved; for example, the mIoU score for buildings is improved by 10.7%, and for ships... mIoU The improvement of 2.7% indicates that BANet can effectively improve the semantic segmentation accuracy of different objects in a 3D point cloud scene.

[0071] like Figure 12 The image shown is a partial visualization of the comparison results, where the first column of images is... Figure 12 (a) is a schematic diagram of a 3D point cloud scene for the SUM test set, the second column of images ( Figure 12 (b) is a semantic truth diagram of the SUM test set, and the third column of the image ( Figure 12 (c) is a schematic diagram of the segmentation results based on RandaNet, the fourth column of the image ( Figure 12 (d) is a schematic diagram of the segmentation results based on BANet. The diagram shows the effectiveness of the BANet network in the segmentation results of each category.

[0072] In the current embodiment, the present invention selects RandaNet and BANet proposed in the present invention for a visual comparison of segmentation results. Figure 12The image shows four sets of data (one set per row, for a total of four sets). Each set includes an outdoor 3D point cloud scene directly input into the network, the corresponding semantic ground truth model for the point cloud, the semantic segmentation results obtained by RandlaNet, and the semantic segmentation results predicted by BANet. Figure 12 The four sets of comparison results show that, compared to RandlaNet, BANet can obtain clearer semantic boundaries and more accurately identify different objects in outdoor 3D point cloud scenes. In the first set of data, RandlaNet identified boats in the lake in a fragmented way, even identifying vegetation in the middle of the boats (marked by red circles). At the same time, RandlaNet easily misidentified small protrusions on the ground as buildings (marked by red rectangles), while BANet can identify the sloping boundaries between boats and the lake surface, and between buildings and the ground, and can largely maintain the overall shape of the objects. This is because, compared to RandlaNet, the BANet used in this invention adds a global feature propagator and a boundary point perceptron, establishing semantic connections between global contextual information and local semantic consistency constraints, improving the network's ability to perceive the overall scene content and its ability to semantically distinguish boundary regions, thus obtaining more complete object semantic information and clearer boundary semantic information. In the segmentation results marked by the red circular boxes in the second row, RandlaNet easily confuses the semantics of small objects with similar shapes (vehicles and vegetation), while BANet, utilizing a boundary point perceptron, enhances the semantic distinguishability of different object classes in the local space, thus separating them. In the segmentation results marked by the red rectangular boxes, RandlaNet struggles to completely distinguish between flat building walls and ground objects (vehicles, etc.), exhibiting semantic confusion and semantic adhesion in the segmentation results. In contrast, BANet accurately segments different objects, demonstrating the effectiveness of the semantic segmentation method disclosed in this invention. This phenomenon can also be seen in the segmentation results in the third and fourth rows. Thanks to the perception of global context information and the distinction of boundary point semantic information, BANet can maintain complete semantic information within objects of the same class and generate clearer boundary semantic information between objects of different classes.

[0073] Figure 13 The image shows magnified views of four sets of segmentation results from RandlaNet and BANet, with each row representing one set of data from top to bottom, for a total of four sets. Figure 13 (a) is a magnified view of a 3D point cloud scene of the SUM test set. Figure 13 (b) is a locally magnified semantic truth diagram of the SUM test set. Figure 13 (c) is a schematic diagram of the local segmentation results based on RandlaNet. Figure 13(d) is a schematic diagram of the local segmentation results based on BANet. Combined with the quantitative segmentation results of the two methods in Table 1, the qualitative comparison results are highly consistent with the quantitative data in Table 1.

[0074] Specifically, RandlaNet utilizes multi-scale local feature aggregation to observe complex structures in a scene, but due to its lack of long-range context awareness, it is prone to semantic inconsistencies within objects in the segmentation results. For example... Figure 12 As shown in the first set of data in the first row, RandlaNet's segmentation of buildings from the SUM test set contains semantically incorrect blotches, a phenomenon also observed in the building segmentation results in the second row. In contrast, BANet, with its global context awareness, maintains greater semantic consistency within buildings. In the segmentation results of the third set of data in the third row, RandlaNet semantically confuses similarly shaped vegetation with vehicles. In the fourth set of data in the fourth row, RandlaNet's segmentation results exhibit significant semantic confusion between ships and ground. BANet, on the other hand, provides more accurate semantic labels and clear semantic boundaries between different objects.

[0075] Based on the experimental results of the previous embodiment, it can be seen that designing an end-to-end 3D point cloud scene semantic segmentation neural network using deep learning algorithms is a commonly used method for obtaining 3D semantic point cloud scenes. Current point cloud segmentation methods mainly focus on multi-level point cloud feature extraction and local geometric feature learning, which are insufficient to handle the semantic segmentation task of outdoor 3D reconstructed point clouds with a large amount of noise and geometric distortion. Compared with the commonly used 3D point cloud semantic segmentation methods, the proposed BANet in this invention can achieve better semantic segmentation results on the SUM dataset.

[0076] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic segmentation method for UAV 3D reconstruction point cloud scenes based on boundary point perception networks, characterized in that, Includes the following steps: S1 acquisition steps: Acquire the original image data of the area to be detected; S2 Reconstruction Steps: Perform 3D reconstruction on the original image data to obtain the original 3D point cloud data of the area to be detected; S3 segmentation steps: Input the original 3D point cloud data into the trained BANet network for semantic segmentation to obtain semantic prediction results; the semantic prediction results represent the 3D model prediction results of the region to be detected at multiple scales; S4 constraint step: Use cross-entropy loss to constrain the semantic prediction results and obtain the semantic truth value of the region to be detected; S5 expression steps: Obtain the three-dimensional geographic information of the area to be detected based on semantic truth values.

2. The semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 1, characterized in that, The raw image data acquired in step S1 includes image data, native image parameters, and native auxiliary data; among which, the image data is visible light RGB imagery acquired by the UAV.

3. The semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 1, characterized in that, The S2 reconstruction steps specifically include: S21 Data Verification Steps: Input verification and invalid frame removal are performed on the original image data to obtain standard image data; S22 Feature extraction steps: Extract local features from standard image data, perform bidirectional feature matching and mismatch removal to obtain image-related data; S23 coordinate restoration steps: Based on image association data, the SfM algorithm is used to calculate and restore the 3D coordinates of key points, obtain the corresponding camera pose, and generate sparse color point cloud. S24 point cloud reconstruction steps: Based on camera pose and sparse color point cloud, full pixel depth estimation and reverse derivation are performed using MVS multi-view stereo matching to generate dense 3D point cloud. S25 Filtering and Removal Steps: Filter the dense 3D point cloud to remove distortion noise and error noise, and output the original 3D point cloud data of the area to be detected.

4. The semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 1, characterized in that, The BANet network in S3 is a symmetrical five-layer coding-decoding structure, consisting of a first coding layer, a second coding layer, a third coding layer, a fourth coding layer, a fifth coding layer, a first decoding layer, a second decoding layer, a third decoding layer, a fourth decoding layer, and a fifth decoding layer connected in sequence. In this design, a global feature propagator is embedded between the fifth encoding layer and the first decoding layer, and the fifth encoding layer, the global feature propagator, and the first decoding layer are connected in sequence. Local feature aggregators are embedded after the first, second, third, and fourth encoding layers, and each encoding layer and each local feature aggregator is connected alternately in sequence. Boundary point perceptrons are embedded after the first, second, third, and third decoding layers, and each decoding layer and each boundary point perceptron is connected alternately in sequence.

5. The semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 4, characterized in that, The specific process by which any local feature aggregator in the BANet network is executed is shown below: Data acquisition steps: Obtain feature tensors and 3D point position information, and extract global context information of the 3D point set through feature tensors; wherein, the feature tensor is obtained by encoding the 3D point set by the previous encoding layer at the current local feature aggregator embedding position; Spatial encoding steps: Based on the K nearest neighbor algorithm, local feature encoding is performed on the 3D point position information to obtain local spatial position representation data, and local position encoding is performed on the global context information to obtain enriched context representation data; Adaptive aggregation step: Through a gate mechanism, local spatial location representation data and enriched contextual representation data are guided to undergo context-adaptive aggregation to obtain the initially enhanced 3D point features; Local enhancement step: Perform local enhancement operation and extract more information on the 3D point features after the initial enhancement to obtain the 3D point features after the second enhancement; Residual fusion steps: The global context information of the 3D point set and the 3D point features after secondary enhancement are fused using residual connections to obtain the final locally enhanced features.

6. The semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 5, characterized in that, In the adaptive aggregation step, the specific formula for expressing the context-adaptive aggregation process is as follows: ； In the formula, Point The initial enhanced 3D point features; point The number of 3D points in the neighborhood, Point Encoding features of nearest neighbors in the neighborhood space Represents the Hadama product. This represents the contextual importance score of nearest neighbors in the local space within a 3D point neighborhood. This represents the position of any point in the 3D point set within the current coding layer. In the neighborhood space Any nearest neighbor position.

7. A semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 4, characterized in that, The global feature propagator in the BANet network is used to perform the following specific process: Dimension extraction steps: Obtain the encoded features output by the fifth encoding layer, extract the global context information of the encoded features along the channel dimension based on the parallel pooling structure, and obtain the parallel feature vector corresponding to the parallel pooling structure; Feature propagation steps: A weight-sharing multilayer perceptron is used to apply parallel feature vectors to establish internal point feature associations, resulting in associated feature vectors; Fusion mapping steps: Based on the channel dimension, the associated feature vectors are fused using convolutional fusion and then mapped using the sigmoid activation function to obtain the importance weight vector; Global selection step: Selectively enhance or suppress the coding features output by the fifth coding layer according to the importance weight vector to obtain global coding features.

8. The semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 7, characterized in that, The specific formula for calculating the importance weight vector in the fusion mapping step is as follows: ； The specific formula for calculating global coding features is as follows: ； in, This represents an importance weight vector, where each value indicates its importance in encoding features. The importance of corresponding point features in the entire three-dimensional space; This represents the sigmoid activation function; This indicates a weight-sharing multilayer perceptron; Represents parallel eigenvectors; This represents the matrix addition operation; Represents global encoding features; This represents the Hadama product.

9. A semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 4, characterized in that, The boundary point perceptron in the BANet network is used to calculate the semantic labels of the corresponding coding layers. The specific calculation formula is as follows: ； In the formula, Indicates the first n The first coding layer i The semantic truth value of a three-dimensional point; This represents the average pooling operation; Indicates the first n Three-dimensional points of each coding layer In the n The local neighborhood space of the -1 coding layer, the th n In the coding layer feature, the th i Three-dimensional points It is the first n -1 is obtained by aggregating all three-dimensional point sets in the local space of the coding layer; , Indicates the first n Subsampled point cloud at the scale of each coding layer; n This represents the number of coding layers, and the values ​​correspond one-to-one with each coding layer. n =5.

10. A semantic segmentation method for UAV 3D reconstruction point cloud scene based on boundary point perception network according to claim 4, characterized in that, In the BANet network, each boundary point perceptron uses a boundary point perceptual loss function to guide the decoding stage of the corresponding decoding layer; the specific calculation formula for the boundary point perceptual loss function is as follows: ； in, Indicates the first n The boundary point sensing loss function value of each decoding layer. n =5; Indicates the first n True values ​​of boundary points in a point cloud within a sub-sampling space; Indicated by point The K-neighborhood space centered on the center; The calculation is of variables a and b Euclidean distance in three-dimensional space , They represent the first n Midpoint of each sub-sampling space and Features; This represents the temperature parameter in the comparative learning; for each boundary point Let its K-neighbor points The data is divided into positive and negative samples, with positive sample points represented as follows: Negative sample points are represented as .

Images