Point cloud data processing method for 3D modeling of UAV aerial photography based on multi-dimensional shape adaptive convolution

Abstract

A method for processing point cloud data for drone aerial photography 3D modeling based on multi-dimensional shape adaptive convolution, comprising the following steps: step 1: constructing a point cloud data set; step 2: downsampling the farthest point, in order to alleviate the computational burden brought by the huge number of point clouds, the number of original point clouds is downsampled to a fixed value by the farthest point sampling method; step 3: constructing a neighborhood set; step 4: primary neighborhood features; step 5: encoding local shape features; step 6: constructing multi-dimensional adaptive convolution; step 7: extracting multi-dimensional shape semantic features; step 8: obtaining classification results. The present invention realizes automatic semantic segmentation of 3D models, alleviates the computational burden brought by the huge number of point clouds, enhances the geometric feature integrity of local point clouds, enhances the ability of the method to capture rich context clues, and obtains high-level semantic features with strong expressiveness.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional point cloud data processing, and in particular to a method for processing point cloud data of three-dimensional modeling by unmanned aerial vehicle aerial photography. Background Art

[0002] With the development of 3D data acquisition equipment such as drones, lidars, and depth cameras, 3D point cloud data is increasingly used in scenarios such as unmanned driving, intelligent robots, augmented reality, and smart attractions. Among them, oblique photography plays a key role in photogrammetry, aerial mapping, and remote sensing images at home and abroad. It is usually composed of drones and modeling software. Multi-angle aerial photos are obtained through drone aerial photography, and point cloud data is obtained through modeling software to restore the 3D structure of the scene. Point cloud data is a commonly used 3D model display method, which contains the coordinate information, color information, and spatial topological structure information of each point in the scene. Point cloud semantic segmentation, as a perception technology for fine-grained segmentation of 3D space, is undergoing the test of efficiency, stability, and accuracy. As deep learning technology shows the characteristics of stability, reliability, and automatic feature extraction in image recognition, generation, and segmentation tasks, many works have tried to apply deep learning technology to point cloud semantic segmentation tasks.

[0003] In recent years, inspired by the remarkable success of convolutional neural networks in image tasks, many methods have converted irregular point cloud data into regular data forms such as images and voxels in order to automatically extract features using convolutional neural networks. However, this type of conversion method not only loses the geometric structure information of the original point cloud model, but also generates a lot of computational consumption. PointWeb tightly connects each point in the local neighborhood through an adaptive adjustment module. Feng performs neighborhood queries in multiple directions to obtain rich local context features. Therefore, subsequent static convolution-based methods capture the intrinsic structural information of local areas by constructing neighborhood sets or graph structures to enhance the representation ability of point clouds. However, this type of method cannot capture the deformation of point clouds in Euclidean space, which makes it difficult to distinguish the feature correspondence between point clouds.

[0004] In order to capture the intrinsic relationship between point clouds as efficiently and accurately as possible, some methods have introduced dynamic convolution in point cloud tasks. Some methods dynamically combine multiple weighted static convolution kernels according to the spatial structure of the point cloud to flexibly extract fine-grained latent semantic features. Although dynamic convolution helps to improve the position adaptation ability of the model, it only uses the relative position relationship between point clouds as a local geometric feature, ignoring the overall shape context information. More importantly, the dynamic convolution of this type of method only assigns weights to the number of static convolution kernels, ignoring other dimensions in the kernel space, and does not fully exploit the dynamic characteristics of dynamic convolution, limiting the ability of dynamic convolution to capture rich context clues. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes a method for processing point cloud data for drone aerial photography 3D modeling based on multi-dimensional shape adaptive convolution. Firstly, a point cloud data set is constructed by drone photography, and the farthest point downsampling method is used to alleviate the computational burden caused by the large number of point clouds, and a point cloud domain set is constructed and its domain features are calculated; the local shape feature encoding is used to encode the shape features of the collected point cloud data to capture the shape context features of the point cloud; secondly, the kernel space of the dynamic convolution is modeled in a multi-dimensional and comprehensive manner according to the intrinsic topological structure information of the point cloud, and the spatial structure changes of the geometric body are flexibly simulated to enhance the adaptability of the network model to the essential semantic structure of the point cloud, and the adaptive convolution that adapts to the neighborhood structure is used to extract semantic features of the center point to obtain high-level semantic features with strong expressiveness.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A method for processing point cloud data of unmanned aerial vehicle three-dimensional modeling based on multi-dimensional shape adaptive convolution, the method comprising the following steps:

[0008] Step 1: Build a point cloud dataset. The process is as follows:

[0009] Step 1.1) Collect data with drone

[0010] Use drones to capture images of outdoor scenes, collect relevant image data, use modeling software to restore the three-dimensional geometric structure of the scene based on the flight parameters and longitude and latitude information in the picture, generate point cloud data, and thus restore the three-dimensional structure of the scene;

[0011] Step 1.2) Define the dataset

[0012] Point cloud data consists of a set of dense three-dimensional coordinates. Each coordinate point contains the coordinate information (x, y, z) of the point cloud in three-dimensional space and the appearance information f that reflects the texture characteristics of the point cloud data. The point cloud data set containing M points is denoted as P = {p i |i=1,...,M}, where p i To represent the coordinate information of the i-th point, the texture feature set corresponding to P is denoted as F = {f i |i=1,...,M},f i To represent the color information of the i-th point;

[0013] Step 1.3) Point cloud data preprocessing

[0014] The point cloud dataset P is subjected to random rotation, scaling, and noise preprocessing operations to generate point cloud datasets under different conditions;

[0015] Step 2: Downsampling at the farthest point: downsampling the number of original point clouds to a fixed value through the farthest point sampling method;

[0016] Step 3: Construct a neighborhood set

[0017] Based on the KNN method, the nearest N neighboring points of the point cloud in step 2 are calculated. i The neighborhood set of i ={n i,1 ,...,n i,N}, where n i,k For the i-th point p i The kth neighboring point of , N is the number of neighboring points;

[0018] Step 4: Primary Neighborhood Features

[0019] According to the center point p i and neighborhood set N i The three-dimensional space coordinates of the neighborhood set are expressed as <p i ,n i,k >, <p i ,n i,k > is the neighborhood set N i The basic feature expression in the geometric space extracts the low-level geometric information in the neighborhood space as the input data for extracting high-level semantic features. At the same time, the low-level features and high-level features are fused to enhance the richness of the features in the spatial expression. The expression is:

[0020]

[0021] Among them (p i -n i,k ) is point p i and neighborhood point n i,k The relative coordinates, p i 、n i,k are the three-dimensional coordinates of the center point and the neighborhood point, respectively. Indicates concatenation of features along the same dimension;

[0022] Step 5: Encode local shape features;

[0023] Step 6: Construct multi-dimensional adaptive convolution. The process is as follows:

[0024] Step 6.1) Define multi-dimensional adaptive convolution

[0025] Adaptive convolution is a linear combination of multiple weighted static convolution kernels, where the weighted order of each convolution kernel is based on the neighborhood set N iThe spatial structure of the adaptive convolution is adaptively adjusted, and the calculation formula of the adaptive convolution MD_AdaptConv is as follows:

[0026]

[0027] Where W i Represents the i-th static convolution kernel, each of which has C_W in Input channels, C_W out output channels, input is the input data of the current adaptive convolution, Assigned to the static convolution kernel W i The weight order of , and all static convolution kernels The sum of the coefficients is 1, which satisfies the following formula:

[0028]

[0029] η i,j , i,j are the i-th static convolution kernel W i The ordinal weights of the jth input channel and output channel also satisfy the requirement that the sum of all channel weight coefficients is 1;

[0030]

[0031]

[0032] Step 6.2) Weight learning network

[0033] In order to realize the adaptive convolution to capture the spatial information of the point cloud, the neighborhood area N in step 5 is i Encoded local shape feature X pi As the input data of the weight learning network in the adaptive convolution, at the same time, in order to enhance the richness of the features, X pi and neighborhood area N i The primary geometric information in is fused, which is expressed as:

[0034]

[0035] Where |||| is the Euclidean distance calculation formula, I i,j The center point p i The feature representation between the jth neighboring point;

[0036] The weight learning network simulates the geometric shape of the input data based on self-learning, and models the implicit expression in the neighborhood through the feature extraction ability of the multi-layer convolutional network. Each layer in the multi-layer convolutional network has the same structure, which consists of two-dimensional convolution ψ, normalized BN and Relu activation function. The input data first undergoes feature calculation of two-dimensional convolution, and then uses BN and Relu functions to process the calculated eigenvalues. BN is a data batch normalization operation used to convert data to near 0; Relu function, as an activation function, is used to expand the expression ability of the model in the deep learning model. The weight learning network simultaneously calculates the number of convolution kernels, input channels, and output channel weights through three parallel multi-layer convolutional networks. Its expression is:

[0037]

[0038] n i,j =Ψ2(p i )=Relu(Bn(ψ2(I i,j )))

[0039] λ i,j =Ψ3(p i )=Relu(Bn(ψ3(I i,j )))

[0040] Where Ψ is used to represent the multi-layer convolutional network structure, Ψ i Represents the i-th multi-layer convolutional network structure;

[0041] Step 6.3) Combine multi-dimensional adaptive convolution

[0042] According to the multi-dimensional adaptive convolution formula defined in step 6.1) and the weight ordinal calculated in step 6.2), WN static convolutions are combined in turn to obtain the adaptive neighborhood N i Multi-dimensional adaptive convolution:

[0043]

[0044] Step 7: Extract multi-dimensional shape semantic features

[0045] The MD_AdaptConv constructed in the previous step adaptively adjusts the convolution kernel parameters by capturing the correlation in the spatial structure of the point cloud in the neighborhood set, and converts the adaptive convolution MD_AdaptConv (p i ) i Extract semantic features to obtain high-level semantic features F with strong expressiveness S , whose expression is:

[0046] F S=MD_AdaptConv(p i )·f i

[0047] Step 8: Get classification results

[0048] The semantic features obtained in step 7 are reduced in dimension through the fully connected layer module, and the category label corresponding to each point cloud is output.

[0049] Further, the step 1.3) comprises the following steps:

[0050] Step 1.3.1) Random rotation

[0051] A rotation angle θ is randomly generated, and a three-dimensional coordinate rotation matrix R(θ) is constructed based on the angle θ. Its expression is:

[0052]

[0053] Multiply the point cloud dataset P by the rotation matrix R(θ) to obtain the point cloud dataset P′ after the original point cloud is rotated by an angle θ around the z-axis. The expression is:

[0054]

[0055] Step 1.3.2) Random Scaling

[0056] Randomly generate scaling factor λ D , the scaling ratio λ D Multiplying the rotated point cloud data set P′ generates the scaled data P″, which is expressed as:

[0057] P″=λ D P′

[0058] Step 1.3.3) Add noise

[0059] Randomly generate a set of 3D coordinates J with the same dimension as the point cloud dataset P″, add the scaled dataset P″ and the noise J, and add a tiny noise interference to the original point cloud:

[0060] P″=P″+J.

[0061] Further, the process of step 2) is as follows:

[0062] Step 2.1) Determine the initial sampling point

[0063] Traverse the point cloud data set P″ and calculate the average value of all points in the three coordinate dimensions. Use the calculated average value as the initial point D_P0 for sampling the farthest point. At the same time, add it to the sampling point set D_S = {D_P0};

[0064] Step 2.2) Calculate the distance matrix

[0065] The farthest point sampling is to select the point farthest from D_S from the point cloud data set. In order to obtain the distance between the point cloud data P″ and the sampling point set D_S, the Euclidean distance between each point and the newly added point in D_S is recorded through the distance matrix D_L;

[0066] Step 2.3) Update the sampling point set

[0067] The point corresponding to the maximum value in D_L is denoted as D_P i , and add it to D_S. As D_S updates the sampling points, D_L will recalculate the Euclidean distance between the newly added sampling points and the point cloud dataset, and at the same time update the point cloud whose Euclidean distance is less than the original value. Finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

[0068] Furthermore, the process of step five is as follows:

[0069] Step 5.1) Extract high-dimensional shape features

[0070] The primary neighborhood features in step 4 <p i ,n i,k >Extract high-dimensional shape features S through nonlinear functions i,k , whose expression is:

[0071] S i,k =δ(α(<p i ,n i,k >))

[0072] Among them, α is a multi-layer forward propagation network with a nonlinear activation function, δ is a normalization function, and the nonlinear activation function is a nonlinear function that converts input data into other data. The forward propagation network is composed of a multi-layer network structure, and the input data of the latter layer of the network is the output data of the previous layer of the network;

[0073] Step 5.2) Extract local features

[0074] S i,k N is the neighborhood area i In order to express the integrity of the neighborhood area, the high-dimensional shape features represented by the kth neighborhood point are extracted through the global maximum pooling operation. i The significant features in the holistic feature are expressed as follows:

[0075] S i =MaxPool{S i,1 ,...,S i,K}

[0076] MaxPool is the global maximum pooling operation, which is a method for selecting the maximum eigenvalue from a local area in a convolutional neural network;

[0077] Step 5.3) Extract global feature descriptors

[0078] Center point p i It has its own semantic information in the global three-dimensional space, and also makes an important contribution to the effective identification of point cloud semantic information. Therefore, the center point p is identified by MLP (MultiLayer Perceptron). i The feature descriptor G in the global scope i To extract, the expression is as follows:

[0079] G i =MLP(p i ,f i )

[0080] Among them, MLP is a forward propagation neural network with an input layer, a hidden layer, and an output layer structure, which generates a description of the input data in the global expression space through multiple calculations of the hidden layer; i For point p i Corresponding appearance features;

[0081] Step 5.4) Fusion features

[0082] In order to enhance the center point p i The feature representation power in three-dimensional space is to add the local shape feature S in step 5.2) through element-by-element addition in deep learning i and the global feature descriptor G in step 5.3) i Add them together and input them into the conversion layer to get the weight vector Weight vector is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features, where It shows the local shape feature S i 's contribution, and The importance of the global feature descriptor is explained; finally, the weight vector is respectively combined with the local shape feature S i and the global feature descriptor G i Perform HadamardProduct operation and use the result as the center point p i Fusion features Its expression is:

[0083]

[0084]

[0085] The weight vector in the above formula is It is calculated through the conversion layer, which is composed of the BP neural network τ and the transfer function φ in deep learning. The transfer function is a function that represents the relationship between input data and output results. ⊙ is the Hadamard Product operation in matrix calculation, which is a matrix multiplication operated by element position.

[0086] Preferably, in step eight, the fully connected layer module consists of two two-dimensional convolutions and Dropout, wherein the input channel and output channel of the first convolution are consistent with the dimension of the semantic feature, the number of output channels of the second convolution module is the same as the number of point cloud categories, and Dropout is based on the probability model to discard some connections between convolutional networks during the dimensionality reduction process, thereby preventing overfitting during the learning process.

[0087] The above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0088] (1) The present invention provides a point cloud data processing method for obtaining point cloud data through tilted photography by an unmanned aerial vehicle and performing three-dimensional modeling, and realizes automatic semantic segmentation of the three-dimensional model based on the automatic feature extraction capability of deep learning; the point cloud data processing method provided by the present invention can alleviate the computational burden brought by the large number of point clouds through farthest point downsampling, and construct a point cloud domain set and calculate its domain features.

[0089] (2) The present invention provides a coding method for local shape features of point clouds. By multi-level coding of the geometric space of point clouds, the effective fusion of fine-grained spatial position features and overall feature descriptions is achieved, and the geometric feature integrity of local point clouds is further enhanced. The present invention performs full-dimensional learning on the kernel space of adaptive convolution based on a parallel strategy, thereby enhancing the method's ability to capture rich contextual clues. The adaptive convolution that adapts to the neighborhood structure is used to extract semantic features from the center point, thereby obtaining high-level semantic features with strong expressiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a multi-dimensional adaptive convolution module diagram of the present invention.

[0091] Figure 2 It is a graph of experimental results verifying the method of the present invention on an outdoor data set. DETAILED DESCRIPTION

[0092] The present invention will be further described below in conjunction with the accompanying drawings.

[0093] Reference Figure 1 , a method for processing point cloud data of UAV aerial photography 3D modeling based on multi-dimensional shape adaptive convolution, comprising the following steps:

[0094] Step 1: Build a point cloud dataset. The process is as follows:

[0095] Step 1.1) Collect data with drone

[0096] Oblique photography technology is a method of using drones to take multi-perspective aerial photos of outdoor scenes and restore the coordinates of the object in three-dimensional space based on similar points in multiple views; using drone aerial photography to capture images of outdoor scenes and collect relevant image data; using modeling software such as ContextCapture to restore the three-dimensional geometric structure of the scene based on the flight parameters and longitude and latitude information in the image, generate point cloud data, and thus restore the three-dimensional structure of the scene.

[0097] Step 1.2) Define the dataset

[0098] The oblique photography technology constructs a digital twin model of an outdoor scene through a 3D reconstruction algorithm, thereby using 3D point cloud data to represent a 3D twin model of a real scene. The oblique photography technology uses a drone to take multi-view aerial photos of an outdoor scene, and restores the coordinates of the object in 3D space based on similar points in multiple views. The point cloud data consists of a set of dense 3D coordinates, each of which contains the coordinate information (x, y, z) of the point cloud in 3D space and the appearance information f that reflects the texture features of the point cloud data, such as RGB that represents the color information of the 3D object's appearance. The point cloud data set containing M points is denoted as P = {p i |i=1,…,M}, where p i To represent the coordinate information of the i-th point, the texture feature set corresponding to P is denoted as F = {f i |i=1,…,M},f i To represent the color information of the i-th point;

[0099] Step 1.3) Point cloud data preprocessing

[0100] The point cloud dataset P is subjected to random rotation, scaling, and noise preprocessing operations to generate point cloud datasets under different conditions, thereby enhancing the method's anti-interference ability to data changes, including the following steps:

[0101] Step 1.3.1) Random rotation

[0102] A rotation angle θ is randomly generated, and a three-dimensional coordinate rotation matrix R(θ) is constructed based on the angle θ. Its expression is:

[0103]

[0104] Multiply the point cloud dataset P by the rotation matrix R(θ) to obtain the point cloud dataset P′ after the original point cloud is rotated by an angle θ around the z-axis. The expression is:

[0105]

[0106] Step 1.3.2) Random Scaling

[0107] Randomly generate scaling factor λ D , the scaling ratio λ D Multiplying the rotated point cloud data set P′ generates the scaled data P″, which is expressed as:

[0108] P″=λ D P′

[0109] Step 1.3.3) Add noise

[0110] A set of three-dimensional coordinates J with the same dimension as the point cloud dataset P″ is randomly generated, and the value of J is small. The scaled dataset P″ and the noise J are added to add a small noise interference to the original point cloud;

[0111] P″=P″+J

[0112] Step 2: Downsampling at the farthest point. In order to alleviate the computational burden caused by the huge number of point clouds, the number of original point clouds is downsampled to a fixed value through the farthest point sampling method. The process is as follows:

[0113] Step 2.1) Determine the initial sampling point

[0114] Traverse the point cloud dataset P″ and calculate the average values ​​of all points in the three coordinate dimensions. The calculated average value is used as the initial point D_P0 for sampling the farthest point, and at the same time, it is added to the sampling point set D_S = {D_P0};

[0115] Step 2.2) Calculate the distance matrix

[0116] The farthest point sampling is to select the point farthest from D_S from the point cloud data set. Therefore, in order to obtain the distance between the point cloud data P″ and the sampling point set D_S, the Euclidean distance between each point and the newly added point in D_S is recorded through the distance matrix D_L;

[0117] Step 2.3) Update the sampling point set

[0118] The point corresponding to the maximum value in D_L is denoted as D_P i , and add it to D_S. As D_S updates the sampling points, D_L will recalculate the Euclidean distance between the newly added sampling points and the point cloud dataset, and at the same time update the point cloud whose Euclidean distance is less than the original value. Finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

[0119] Step 3: Construct a neighborhood set

[0120] The shape of a three-dimensional geometry is represented by a set of dense three-dimensional coordinates arranged in order of position. The point cloud data in the same target geometry has a high degree of similarity in spatial feature representation and high-level semantic expression. Therefore, in the point cloud semantic segmentation task, the semantic information extraction of the target geometry depends on the correlation between each point and the surrounding neighborhood points. Based on the KNN method in the traditional search algorithm, the nearest N neighborhood points of the point cloud in step 2 are calculated. Point p i The neighborhood set of i ={n i,1 ,...,n i,N}, where n i,k For the i-th point p i The kth neighboring point of , N is the number of neighboring points;

[0121] Step 4: Primary Neighborhood Features

[0122] According to the center point p i and neighborhood set N i The three-dimensional space coordinates of the neighborhood set are expressed as <p i ,n i,k >, <p i ,n i,k > is the neighborhood set N i The basic feature expression in the geometric space extracts the low-level geometric information in the neighborhood space, which can be used as input data for extracting high-level semantic features. At the same time, the low-level features can be fused with the high-level features to enhance the richness of the features in the spatial expression. The expression is:

[0123]

[0124] Among them (p i -n i,k ) is point p i and neighborhood point n i,k The relative coordinates, p i 、n i,k are the three-dimensional coordinates of the center point and the neighborhood point, respectively. Indicates concatenation of features along the same dimension;

[0125] Step 5: Encode local shape features. The process is as follows:

[0126] Step 5.1) Extract high-dimensional shape features

[0127] The primary neighborhood features in step 4 <p i ,n i,k >Extract high-dimensional shape features S through nonlinear functions i,k , whose expression is:

[0128] Si,k =δ(α(<p i ,n i,k >))

[0129] Among them, α is a multi-layer forward propagation network with a nonlinear activation function, δ is a normalization function, and a nonlinear activation function is a nonlinear function that converts input data into other data, such as the Tanh and Sigmoid functions widely used in deep learning. The forward propagation network is composed of a multi-layer network structure, and the input data of the next layer of the network is the output data of the previous layer of the network;

[0130] Step 5.2) Extract local features

[0131] S i,k N is the neighborhood area i In order to express the integrity of the neighborhood area, the high-dimensional shape features represented by the kth neighborhood point are extracted through the global maximum pooling operation. i The significant features in the holistic feature are expressed as follows:

[0132] S i =MaxPool{S i,1 ,...,S i,K}

[0133] MaxPool is the global maximum pooling operation, which is a method for selecting the maximum eigenvalue from a local area in a convolutional neural network;

[0134] Step 5.3) Extract global feature descriptors

[0135] Center point p i It has its own semantic information in the global three-dimensional space, and also makes an important contribution to the effective identification of point cloud semantic information. Therefore, the center point p is identified by MLP (MultiLayer Perceptron). i The feature descriptor G in the global scope i To extract, the expression is as follows:

[0136] G i =MLP(p i ,f i )

[0137] Among them, MLP is a forward propagation neural network with an input layer, a hidden layer, and an output layer structure, which can generate a description of the input data in the global expression space through multiple calculations of the hidden layer. i For point p i Corresponding appearance features;

[0138] Step 5.4) Fusion features

[0139] In order to enhance the center point p i The feature representation power in three-dimensional space is to add the local shape feature S in step 5.2) through element-by-element addition in deep learning i and the global feature descriptor G in step 5.3) i Add them together and input them into the conversion layer to get the weight vector Weight vector is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features, where It shows the local shape feature S i 's contribution, and The importance of the global feature descriptor is explained; finally, the weight vector is respectively combined with the local shape feature S i and the global feature descriptor G i Perform HadamardProduct operation and use the result as the center point p i Fusion features Its expression is:

[0140]

[0141]

[0142] The weight vector in the above formula is It is calculated through the conversion layer, which is composed of the BP neural network τ and the transfer function φ in deep learning. The transfer function is a function that represents the relationship between input data and output results; ⊙ is the Hadamard Product operation in matrix calculation, which is a matrix multiplication based on element position.

[0143] Step 6: Construct multi-dimensional adaptive convolution

[0144] like Figure 1 As shown in the figure, in order to enhance the adaptability of the method to different target geometries, a multi-dimensional shape adaptive convolution is constructed by multi-dimensionally and comprehensively modeling the convolution kernel space, thereby flexibly simulating the spatial structure changes of the geometric body and enhancing the adaptability of the method to the essential semantic structure of the point cloud;

[0145] The process is as follows:

[0146] Step 6.1) Define multi-dimensional adaptive convolution

[0147] Adaptive convolution is a linear combination of multiple weighted static convolution kernels, where the weighted order of each convolution kernel is based on the neighborhood set N i The spatial structure of the adaptive convolution is adaptively adjusted, and the calculation formula of the adaptive convolution MD_AdaptConv is as follows:

[0148]

[0149] Where W i Represents the i-th static convolution kernel, each of which has C_W in Input channels, C_W out output channels, WN is the number of static convolution kernels, and input is the input data of the current adaptive convolution. Assigned to the static convolution kernel W i The weight order of , and all static convolution kernels The sum of the coefficients is 1, which satisfies the following formula:

[0150]

[0151] η i,j , i,j are the i-th static convolution kernel W i The ordinal weights of the jth input channel and output channel also satisfy the requirement that the sum of all channel weight coefficients is 1;

[0152]

[0153] Step 6.2) Weight learning network

[0154] In order to realize the adaptive convolution to capture the spatial information of the point cloud, the neighborhood area N in step 5 is i Encoded local shape features As the input data of the weight learning network in the adaptive convolution. At the same time, in order to enhance the richness of features, and neighborhood area N i The primary geometric information in the neighborhood is fused, such as the original three-dimensional coordinates and relative position coordinates in the neighborhood area, which is expressed as:

[0155]

[0156] Where |||| is the Euclidean distance calculation formula, I i,j The center point p i The feature representation between the jth neighboring point;

[0157] The weight learning network simulates the geometric shape of the input data based on self-learning, and models the implicit expression in the neighborhood through the feature extraction capability of the multi-layer convolutional network. Each layer in the multi-layer convolutional network has the same structure, consisting of two-dimensional convolution ψ, normalized BN, and Relu activation function. The input data first undergoes feature calculation of two-dimensional convolution, and then the BN and Relu functions are used to process the calculated feature values. BN is a commonly used data batch normalization operation used to convert data to near 0. The Relu function is used as an activation function in deep learning models to expand the expression ability of the model. The weight learning network simultaneously calculates the number of convolution kernels, input channels, and output channel weights through three parallel multi-layer convolutional networks. The expression is:

[0158]

[0159] n i,j =Ψ2(p i )=Relu(Bn(ψ2(I i,j )))

[0160] λ i,j =Ψ3(p i )=Relu(Bn(ψ3(I i,j )))

[0161] Where Ψ is used to represent the multi-layer convolutional network structure, Ψ i Represents the i-th multi-layer convolutional network structure;

[0162] Step 6.3) Combine multi-dimensional adaptive convolution

[0163] According to the multi-dimensional adaptive convolution formula defined in step 6.1) and the weight ordinal calculated in step 6.2), WN static convolutions are combined in turn to obtain the adaptive neighborhood N i Multi-dimensional adaptive convolution:

[0164]

[0165] Step 7: Extract multi-dimensional shape semantic features

[0166] The MD_AdaptConv constructed in the previous step adaptively adjusts the convolution kernel parameters by capturing the correlation in the spatial structure of the point cloud in the neighborhood set, and converts the adaptive convolution MD_AdaptConv (p i ) i Extract semantic features to obtain high-level semantic features F with strong expressiveness S , whose expression is:

[0167] F S=MD_AdaptConv(p i )·f i

[0168] Step 8: Get classification results

[0169] The semantic features obtained in step 7 are reduced in dimension through the fully connected layer module, and the category label corresponding to each point cloud is output. The fully connected layer module consists of two two-dimensional convolutions and Dropout. The input channel and output channel of the first convolution are consistent with the dimension of the semantic features, and the number of output channels of the second convolution module is the same as the number of point cloud categories. Dropout is based on the probability model to discard some connections between convolutional networks during the dimensionality reduction process, thereby preventing overfitting during the learning process. In deep network models, Dropout is a commonly used network structure processing method;

[0170] Reference Figure 2 , the effectiveness of the method of the present invention is verified on an outdoor public data set. The implementation scheme includes the following steps:

[0171] Step 1: Define the method

[0172] SnapNet: SnapNet is a multi-view based method for processing point clouds, which aims to extract point clouds, calculate point features and generate meshes through preprocessing steps;

[0173] Kpconv: Kpconv calculates the weight of the convolution kernel by constructing a linear relationship between the kernel points in Euclidean space and the local neighborhood points;

[0174] SegCloud: SegCloud converts the original point cloud data into a voxel structure, and further uses 3DCNN to extract fine-grained semantic information. Finally, the semantic information is restored to the original point cloud through interpolation and conditional random fields, where voxels are a regular three-dimensional data structure and conditional random fields are conditional probability distribution models of another set of output sequences under a given set of input sequences;

[0175] SPGraph: is a graph-based method for processing large-scale point clouds.

[0176] Step 2: Define evaluation metrics

[0177] In order to quantify the semantic segmentation accuracy of the method of the present invention, the present invention uses the mean intersection over union (mIoU) index and calculates the semantic effect of the method of the present invention on the Semantic3D dataset. The calculation formula of mIoU is:

[0178]

[0179] The number of semantic categories in the point cloud dataset is Cnum, and the prediction results and label results calculated in the method of the present invention are both the number of the i-th type of point cloud, which is recorded as Pr ii , Pr ij It indicates the number of point clouds with label category i and predicted category j;

[0180] Step 3: Verification of outdoor public dataset

[0181] Step 3.1) Introduction to outdoor dataset

[0182] The Semantic3D dataset is a collection of approximately 4 billion point clouds generated by collecting outdoor scenes using a laser scanner. In the Semantic3D dataset, outdoor scenes are mainly churches, streets, squares, and villages, and each scene is divided into eight semantic elements, including artificial terrain, natural terrain, high vegetation, low vegetation, and buildings. Semantic3D contains a total of 30 scenes, and each scene is a separate dataset, with a training set and a test set ratio of 1:1. The data collected by the laser scanner has an intensity attribute, that is, each point has seven-dimensional attributes, including three-dimensional coordinates, RGB color, and intensity. In addition, the data scanned by the lidar is sparse and has widely varying spatial resolutions, so the semantic segmentation of outdoor point clouds is more challenging.

[0183] Step 3.2) Dataset processing

[0184] The semantic labels provided by the Semantic3D dataset include unlabeled point clouds. Since unlabeled point clouds have no real scenes, the present invention removes all point clouds of this category during training. The difference is that the scenes involved in the Semantic3D dataset are all real large-scale and large-range scenes. Therefore, in order to avoid destroying the structural integrity, the present invention sets the sampling block size to 4m×4m;

[0185] Step 3.3) Experimental results evaluation

[0186] Figure 2 The quantitative evaluation results on the Semantic3D dataset are shown. It can be seen from the table that the overall evaluation index of the method of the present invention on the Semantic3D dataset is 77.6%, and mIou achieves the best effect. Compared with SnapNet and SegCloud, the segmentation accuracy of the present invention is improved by nearly 16%. SnapNet and SegCloud need to pre-process the point cloud data into structured data such as multi-views and voxels, so it is inevitable that point cloud information will be lost during the conversion process. Compared with graph-based SPGraph, point convolution-based Kpconv and other methods, the segmentation accuracy of the present invention is improved by 3%.

[0187] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for processing point cloud data of UAV aerial photography 3D modeling based on multi-dimensional shape adaptive convolution, characterized in that: The method comprises the following steps: Step 1: Build a point cloud dataset. The process is as follows: Step 1.1) Collect data with drone Use drones to capture images of outdoor scenes, collect relevant image data, use modeling software to restore the three-dimensional geometric structure of the scene based on the flight parameters and longitude and latitude information in the picture, generate point cloud data, and thus restore the three-dimensional structure of the scene; Step 1.2) Define the dataset Point cloud data consists of a set of dense three-dimensional coordinates. Each coordinate point contains the coordinate information (x, y, z) of the point cloud in three-dimensional space and the appearance information f that reflects the texture characteristics of the point cloud data. The point cloud data set containing M points is denoted as P = {p i |i=1,...,M}, where pi is the coordinate information representing the i-th point, and the texture feature set corresponding to P is denoted as F={f i |i=1,...,M}, fi is the color information representing the i-th point; Step 1.3) Point cloud data preprocessing The point cloud dataset P is subjected to random rotation, scaling, and noise preprocessing operations to generate point cloud datasets under different conditions; Step 2: Downsampling at the farthest point: downsampling the number of original point clouds to a fixed value through the farthest point sampling method; Step 3: Construct a neighborhood set Based on the KNN method, the nearest N neighboring points of the point cloud in step 2 are calculated. i The neighborhood set of i ={n i,1 ,...,n i,N }, where n i,k is the i-th point p i The kth neighboring point of , N is the number of neighboring points; Step 4: Primary Neighborhood Features According to the center point p i and neighborhood set N i The three-dimensional space coordinates of the neighborhood set are expressed as 〈p i ,n i,k 〉, <p i ,n i,k > is the neighborhood set N i The basic feature expression in the geometric space extracts the low-level geometric information in the neighborhood space as the input data for extracting high-level semantic features. At the same time, the low-level features and high-level features are fused to enhance the richness of the features in the spatial expression. The expression is: Among them (p i -n i,k ) is point p i and neighborhood point n i,k The relative coordinates, p i 、n i,k are the three-dimensional coordinates of the center point and the neighborhood point, respectively. Indicates concatenation of features along the same dimension; Step 5: Encode local shape features; Step 6: Construct multi-dimensional adaptive convolution. The process is as follows: Step 6.1) Define multi-dimensional adaptive convolution Adaptive convolution is a linear combination of multiple weighted static convolution kernels, where the weighted order of each convolution kernel is based on the neighborhood set N i The spatial structure of the adaptive convolution is adaptively adjusted, and the calculation formula of the adaptive convolution MD_AdaptConv is as follows: Where W i Represents the i-th static convolution kernel, each of which has C_W in Input channels, C_W out output channels, input is the input data of the current adaptive convolution, Assigned to the static convolution kernel W i The weight order of , and all static convolution kernels The sum of the coefficients is 1, which satisfies the following formula: η i,j , i,j are the i-th static convolution kernel W i The ordinal weights of the jth input channel and output channel also satisfy the requirement that the sum of all channel weight coefficients is 1; Step 6.2) Weight learning network In order to realize the adaptive convolution to capture the spatial information of the point cloud, the neighborhood area N in step 5 is i Encoded local shape features As the input data of the weight learning network in the adaptive convolution, at the same time, in order to enhance the richness of the features, and neighborhood area N i The primary geometric information in is fused, which is expressed as: Where || || is the Euclidean distance calculation formula, I i,j The center point p i Feature representation between the jth neighboring point; The weight learning network simulates the geometric shape of the input data based on self-learning, and models the implicit expression in the neighborhood through the feature extraction ability of the multi-layer convolutional network. Each layer in the multi-layer convolutional network has the same structure, which consists of two-dimensional convolution ψ, normalized BN and Relu activation function. The input data first undergoes feature calculation of two-dimensional convolution, and then uses BN and Relu functions to process the calculated eigenvalues. BN is a data batch normalization operation used to convert data to near 0; Relu function, as an activation function, is used to expand the expression ability of the model in the deep learning model. The weight learning network simultaneously calculates the number of convolution kernels, input channels, and output channel weights through three parallel multi-layer convolutional networks. Its expression is: n i,j =Ψ2(p i )=Relu(Bn(ψ2(I i,j ))) l i,j =Ψ3(p i )=Relu(Bn(ψ3(I i,j ))) Among them Ψ i Represents the i-th multi-layer convolutional network structure; Step 6.3) Combine multi-dimensional adaptive convolution According to the multi-dimensional adaptive convolution formula defined in step 6.1) and the weight ordinal calculated in step 6.2), WN static convolutions are combined in turn to obtain the adaptive neighborhood N i Multi-dimensional adaptive convolution: Step 7: Extract multi-dimensional shape semantic features The MD_AdaptConv constructed in the previous step adaptively adjusts the convolution kernel parameters by capturing the correlation in the spatial structure of the point cloud in the neighborhood set, and converts the adaptive convolution MD_AdaptConv (p i ) i Extract semantic features to obtain high-level semantic features F with strong expressiveness S , whose expression is: F S =MD_AdaptConv(p i )·f i Step 8: Get classification results The semantic features obtained in step 7 are reduced in dimension through the fully connected layer module, and the category label corresponding to each point cloud is output.

2. The method for processing point cloud data of drone aerial photography 3D modeling based on multi-dimensional shape adaptive convolution as claimed in claim 1, characterized in that: The step 1.3) comprises the following steps: Step 1.3.1) Random rotation A rotation angle θ is randomly generated, and a three-dimensional coordinate rotation matrix R(θ) is constructed based on the angle θ. Its expression is: Multiply the point cloud dataset P by the rotation matrix R(θ) to obtain the point cloud dataset P′ after the original point cloud is rotated by an angle θ around the z-axis. The expression is: Step 1.3.2) Random Scaling Randomly generate scaling factor λ D , the scaling ratio λ D Multiplying the rotated point cloud data set P′ generates the scaled data P″, which is expressed as: P″=λ D P′ Step 1.3.3) Add noise Randomly generate a set of 3D coordinates J with the same dimension as the point cloud dataset P″, add the scaled dataset P″ and the noise J, and add a tiny noise interference to the original point cloud: P″=P″+J.

3. The method for processing point cloud data of drone aerial photography 3D modeling based on multi-dimensional shape adaptive convolution as claimed in claim 1 or 2, characterized in that: The process of step 2 is as follows: Step 2.1) Determine the initial sampling point Traverse the point cloud data set P″ and calculate the average value of all points in the three coordinate dimensions. Use the calculated average value as the initial point D_P0 for sampling the farthest point. At the same time, add it to the sampling point set D_S = {D_P0}; Step 2.2) Calculate the distance matrix The farthest point sampling is to select the point farthest from D_S from the point cloud data set. In order to obtain the distance between the point cloud data P″ and the sampling point set D_S, the Euclidean distance between each point and the newly added point in D_S is recorded through the distance matrix D_L; Step 2.3) Update the sampling point set The point corresponding to the maximum value in D_L is denoted as D_P i , and add it to D_S. As D_S updates the sampling points, D_L will recalculate the Euclidean distance between the newly added sampling points and the point cloud dataset, and at the same time update the point cloud whose Euclidean distance is less than the original value. Finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

4. The method for processing point cloud data of unmanned aerial vehicle three-dimensional modeling based on multi-dimensional shape adaptive convolution as claimed in claim 1 or 2, characterized in that: The process of step five is as follows: Step 5.1) Extract high-dimensional shape features The primary neighborhood features in step 4 <p i ,n i,k >Extract high-dimensional shape features S through nonlinear functions i,k , whose expression is: S i,k =δ(α(<p i ,n i,k >)) Among them, α is a multi-layer forward propagation network with a nonlinear activation function, δ is a normalization function, and the nonlinear activation function is a nonlinear function that converts input data into other data. The forward propagation network is composed of a multi-layer network structure, and the input data of the latter layer of the network is the output data of the previous layer of the network; Step 5.2) Extract local features S i,k N is the neighborhood area i In order to express the integrity of the neighborhood area, the high-dimensional shape features represented by the kth neighborhood point are extracted through the global maximum pooling operation. i The significant features in the holistic feature are expressed as follows: S i =MaxPool{S i,1 ,...,S i,K } MaxPool is the global maximum pooling operation, which is a method for selecting the maximum eigenvalue from a local area in a convolutional neural network; Step 5.3) Extract global feature descriptors Center point p i It has its own semantic information in the global three-dimensional space, and also makes an important contribution to the effective identification of point cloud semantic information. Therefore, the center point p is identified by MLP (MultiLayer Perceptron). i The feature descriptor G in the global scope i To extract, the expression is as follows: G i =MLP(p i ,f i ) Among them, MLP is a forward propagation neural network with an input layer, a hidden layer, and an output layer structure, which generates a description of the input data in the global expression space through multiple calculations of the hidden layer; i For point p i Corresponding appearance features; Step 5.4) Fusion features In order to enhance the center point p i The feature representation power in three-dimensional space is achieved through element-by-element addition in deep learning. The local shape feature S in step 5.2) is i and the global feature descriptor G in step 5.3) i Add them together and input them into the conversion layer to get the weight vector Weight vector is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features, where It shows the local shape feature S i 's contribution, and The importance of the global feature descriptor is explained; finally, the weight vector is respectively combined with the local shape feature S i and the global feature descriptor G i Perform Hadamard Product calculation and use the result as the center point p i Fusion features Its expression is: The weight vector in the above formula is It is calculated through the conversion layer, which is composed of the BP neural network τ and the transfer function φ in deep learning. The transfer function is a function that represents the relationship between input data and output results. ⊙ is the Hadamard Product operation in matrix calculation, which is a matrix multiplication operated by element position.

5. The method for processing point cloud data of unmanned aerial vehicle three-dimensional modeling based on multi-dimensional shape adaptive convolution as claimed in claim 1 or 2, characterized in that: In the step eight, the fully connected layer module consists of two two-dimensional convolutions and Dropout, wherein the input channel and output channel of the first convolution are consistent with the dimension of the semantic feature, the number of output channels of the second convolution module is the same as the number of point cloud categories, and Dropout is based on the probability model to discard some connections between convolutional networks during the dimensionality reduction process, thereby preventing overfitting during the learning process.

Images