An automatic semantic modeling method of point cloud for digital twin scenes

Abstract

A point cloud automatic semantic modeling method for digital twin scenes includes the following steps: step 1: constructing a point cloud data set; step 2: farthest point downsampling, downsampling the number of original point clouds to a fixed value by farthest point sampling; step 3: constructing a neighborhood set; step 4: primary neighborhood features; step 5: encoding local shape features; step 6: constructing multi-dimensional adaptive convolution; step 7: extracting multi-dimensional shape semantic features; step 8: adaptive attention mechanism; step 9: encoder module; step 10: decoder module; step 11: jump connection; step 12: calculating classification results; step 13: defining a loss function, the segmentation accuracy of the method can be reflected by comparing the error between the predicted result and the real result, and the error is calculated using the loss function. The present invention improves the efficiency and accuracy of point cloud semantic segmentation.

Description

Technical Field

[0001] The present invention relates to the field of semantic modeling of three-dimensional point clouds, and in particular to an automatic semantic modeling method of point clouds for digital twin scenarios. Background Art

[0002] As an emerging digital tool that covers a variety of new technologies and crosses multiple disciplines, digital twins play an indispensable role in the process of digitalization, informatization, and intelligence in social life. As the basis for building digital twins, digital twin models are a true depiction of physical entities in the dimensions of time and space. They are the premise for realizing various functions and services of digital twins, and are also a necessary link for expanding digital twin technology. In the context of digital twins, modeling methods such as three-dimensional modeling, building information modeling (BIM), and urban intelligent modeling (CIM) can accurately describe physical entities in the real world. In view of the complex geometric structure and wide coverage of physical entities in outdoor scenes, oblique photography modeling technology is usually used to accurately depict and describe outdoor scenes. As a real-life modeling technology suitable for large-scale outdoor scenes, oblique photography modeling technology meets the requirements of digital twins for accurate, usable, and visible models through full coverage, multi-angle data collection methods and flexible model formats.

[0003] The real-life model constructed by oblique photography modeling technology can be represented as three-dimensional point cloud data, which contains the three-dimensional spatial coordinate information and color information of the surface points of the physical entity, which is used to reflect the real geometric structure and texture information of the modeling scene. However, the three-dimensional real-life model constructed by oblique photography modeling technology can only be used for visualization functions, and cannot meet the requirements of digital twins for models to be interactive and fusible. Specifically, the interactivity of the model means that the digital twin model and other elements can complete the mutual interaction of data and instructions through information transmission technology, eliminate the discretely distributed information islands in the system, and thus realize the collaborative use of entity-data-model. Similarly, fusibility is to integrate multiple digital twins into a mechanism model through association relationships, thereby realizing twin modeling of complex systems. Model semanticization is the basic premise for realizing the interactivity and fusibility of digital twin models. It is the link for establishing mapping relationships between models and physical objects and data, and it is the signal for transmitting information between models. Especially in large-scale outdoor scenes, which have a wide coverage, many physical objects, and complex data sources, building an interactive digital twin model requires a lot of data sorting work and consumes too much manpower costs.

[0004] The point cloud semantic segmentation algorithm divides the 3D point cloud data into semantic categories according to the intrinsic topological structure and auxiliary texture information of the point cloud, obtains a real-scene 3D model with semantic information description, and completes the twin semantic modeling of the real-scene model. As a research hotspot in the field of environmental intelligent perception, the point cloud semantic segmentation algorithm is of great significance to the understanding and perception of 3D space. It has been increasingly widely used in scenarios such as unmanned driving, intelligent robots, and augmented reality, especially for the construction of high-precision semantic maps. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes an automatic semantic modeling method for point clouds for digital twin scenarios. First, the shape feature encoding of the collected point cloud data is performed through local shape feature encoding to capture the shape context features of the point cloud. Secondly, the kernel space of the dynamic convolution is modeled in a multi-dimensional and comprehensive manner based on the intrinsic topological structure information of the point cloud to enhance the adaptability of the network model to the essential semantic structure of the point cloud. Finally, the global channel information and spatial information are fused based on the attention mechanism, so that important features are significantly expressed, thereby improving the efficiency and accuracy of point cloud semantic segmentation.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A point cloud automatic semantic modeling method for digital twin scenes, the method comprising the following steps:

[0008] Step 1: Build a point cloud dataset. The process is as follows:

[0009] Step 1.1) Define the dataset

[0010] The oblique photography technology constructs a digital twin model of an outdoor scene through a 3D reconstruction algorithm, thereby using 3D point cloud data to represent the 3D twin model of the real scene; the oblique photography technology uses a drone to take multi-view aerial photos of the outdoor scene, and restores the coordinates of the object in the 3D space based on the similar points in the multiple views; the point cloud data consists of a set of dense 3D coordinates, each coordinate point contains the coordinate information (x, y, z) of the point cloud in the 3D space and the appearance information f reflecting the texture characteristics of the point cloud data; the point cloud data set containing M points is denoted as P = {p i |i=1,...,M}, where p i To represent the coordinate information of the i-th point, the texture feature set corresponding to P is denoted as F = {f i |i=1,...,M},f i To represent the color information of the i-th point;

[0011] Step 1.2) Point cloud data preprocessing

[0012] The point cloud dataset P is subjected to random rotation, scaling, and noise preprocessing operations to generate point cloud datasets under different conditions, thereby enhancing the method's ability to resist interference from data changes;

[0013] Step 2: Downsampling at the farthest point: downsampling the number of original point clouds to a fixed value through the farthest point sampling method;

[0014] Step 3: Construct a neighborhood set

[0015] The shape of a three-dimensional geometry is represented by a set of dense three-dimensional coordinates arranged in order of position. The point cloud data in the same target geometry has a high degree of similarity in spatial feature representation and high-level semantic expression. Therefore, in the task of point cloud semantic segmentation, the semantic information extraction of the target geometry depends on the correlation between each point and its surrounding neighborhood points. Based on the KNN method in the traditional search algorithm, the nearest N neighborhood points of the point cloud in step 2 are calculated. Point p i The neighborhood set of i ={n i,1 ,...,n i,N}, where n i,k is the i-th point p i The kth neighboring point of , N is the number of neighboring points;

[0016] Step 4: Primary Neighborhood Features

[0017] According to the center point p i and the neighborhood set N i The three-dimensional space coordinates of the neighborhood set are expressed as <p i ,n i,k >, <p i ,n i,k > is the neighborhood set N i The basic feature expression in the geometric space extracts the low-level geometric information in the neighborhood space, which can be used as input data for extracting high-level semantic features. At the same time, the low-level features can be fused with the high-level features to enhance the richness of the features in the spatial expression. The expression is:

[0018]

[0019] Among them (p i -n i,k ) is point p i and neighborhood point n i,k The relative coordinates, p i 、n i,k are the three-dimensional coordinates of the center point and the neighborhood point, respectively. Indicates concatenation of features along the same dimension;

[0020] Step 5: Encode local shape features;

[0021] Step 6: Construct multi-dimensional adaptive convolution

[0022] By modeling the convolution kernel space in a multi-dimensional and comprehensive manner, a multi-dimensional shape-adaptive convolution is constructed to flexibly simulate the spatial structure changes of geometric bodies and enhance the method's ability to adapt to the essential semantic structure of point clouds.

[0023] Step 7: Extract multi-dimensional shape semantic features

[0024] The MD_AdaptConv constructed in the previous step adaptively adjusts the convolution kernel parameters by capturing the correlation in the spatial structure of the point cloud in the neighborhood set, and converts the adaptive convolution MD_AdaptConv (p i ) i Extract semantic features to obtain high-level semantic features F with strong expressiveness S , whose expression is:

[0025] F S =MD_AdaptConv(p i )·f i

[0026] Step 8: Adaptive Attention Mechanism

[0027] As the network model becomes more complex and the feature dimension continues to increase, there is information redundancy and similar spatial structures and semantic features between high-dimensional semantic categories. By inferring the feature attention map along the channel direction and spatial direction of the feature map, and adaptively optimizing the input features through the feature attention map, more representative features can be obtained;

[0028] Step 9: Encoder Module

[0029] Steps 2 to 8 are the high-level semantic feature extraction process, which is repeated C 1 Each execution is recorded as a layer of encoding module, and the encoding module of the tth layer is Encode t Indicates that in step 3, the input point cloud is downsampled by the farthest point, so the number of input point clouds in each layer of the encoding structure decreases layer by layer; at the same time, in the feature extraction process of adaptive convolution, the high-level semantic feature dimension of each layer of encoder is increasing;

[0030] Step 10: Decoder Module

[0031] The decoder module Decode consists of a multi-layer structure. Each layer first expands the number of point clouds through point cloud upsampling operations, and restores the number of point clouds layer by layer to the level before downsampling, so as to keep consistent with the number in the decoder; secondly, each layer of the network uses a convolutional structure to extract high-level feature descriptors of the point cloud;

[0032] Step 11: Jump Connection

[0033] The features of the encoder module and the decoder module are concatenated along the same dimension, and the concatenated features are used as the input of the next layer, i.e., the decoder of the tth layer. t The features of the M_C+1-t layer encoder Encode M_C+1-t The features of are concatenated and calculated as the input of the next layer t+1, where M_C is the total number of layers in the encoder module;

[0034] Step 12: Calculate the classification results

[0035] The semantic features obtained in step 11 are reduced in dimension through the fully connected layer module, and the category label corresponding to each point cloud is output. The fully connected layer module consists of two two-dimensional convolutions and Dropout. The input channel and output channel of the first convolution are consistent with the dimension of the semantic features, and the number of output channels of the second convolution module is the same as the number of point cloud categories. Dropout is based on the probability model to discard some connections between convolutional networks during the dimensionality reduction process, thereby preventing overfitting during the learning process;

[0036] Step 13: Define the loss function

[0037] The segmentation accuracy of the method can be reflected by comparing the error between the predicted result and the true result, and the error is calculated using a loss function.

[0038] Further, the step 1.2) comprises the following steps:

[0039] Step 1.2.1) Random rotation

[0040] A rotation angle θ is randomly generated, and a three-dimensional coordinate rotation matrix R(θ) is constructed based on the angle θ. Its expression is:

[0041]

[0042] Multiply the point cloud dataset P by the rotation matrix R(θ) to obtain the point cloud dataset P′ after the original point cloud is rotated by an angle θ around the z-axis. The expression is:

[0043]

[0044] Step 1.2.2) Random Scaling

[0045] Randomly generate scaling factor λ D , the scaling ratio λ D Multiplying the rotated point cloud data set P′ generates the scaled data P″, which is expressed as:

[0046] P″=λ D P′

[0047] Step 1.2.3) Add noise

[0048] Randomly generate a set of 3D coordinates J with the same dimension as the point cloud dataset P″, where the value of J is smaller; add the scaled dataset P″ and the noise J to add a tiny noise interference to the original point cloud;

[0049] P″=P″+J.

[0050] Furthermore, the process of step 2 is as follows:

[0051] Step 2.1) Determine the initial sampling point

[0052] Traverse the point cloud dataset P″ and calculate the average value of all points in the three coordinate dimensions, and use the calculated average value as the initial point D_P for sampling the farthest point 0 At the same time, add it to the sampling point set D_S = {D_P 0};

[0053] Step 2.2) Calculate the distance matrix

[0054] The farthest point sampling is to select the point farthest from D_S from the point cloud data set. Therefore, in order to obtain the distance between the point cloud data P″ and the sampling point set D_S, the Euclidean distance between each point and the newly added point in D_S is recorded through the distance matrix D_L;

[0055] Step 2.3) Update the sampling point set

[0056] The point corresponding to the maximum value in D_L is denoted as D_P i , and add it to D_S. As D_S updates the sampling points, D_L will recalculate the Euclidean distance between the newly added sampling points and the point cloud dataset, and at the same time update the point cloud whose Euclidean distance is less than the original value. Finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

[0057] The process of step five is as follows:

[0058] Step 5.1) Extract high-dimensional shape features

[0059] The primary neighborhood features in step 4 <p i ,ni,k >Extract high-dimensional shape features S through nonlinear functions i,k , whose expression is:

[0060] S i,k =δ(α( <p i ,n i,k >))

[0061] Among them, α is a multi-layer forward propagation network with a nonlinear activation function, δ is a normalization function, and the nonlinear activation function is a nonlinear function that converts input data into other data. The forward propagation network is composed of a multi-layer network structure, and the input data of the latter layer of the network is the output data of the previous layer of the network;

[0062] Step 5.2) Extract local features

[0063] S i,k N is the neighborhood area i In order to express the integrity of the neighborhood area, the high-dimensional shape features represented by the kth neighborhood point are extracted through the global maximum pooling operation. i The significant features in the holistic feature are expressed as follows:

[0064] S i =MaxPool{S i,1 ,...,S i,K}

[0065] MaxPool is the global maximum pooling operation, which is a method for selecting the maximum eigenvalue from a local area in a convolutional neural network;

[0066] Step 5.3) Extract global feature descriptors

[0067] Center point p i It has its own semantic information in the global three-dimensional space, and also makes an important contribution to the effective identification of point cloud semantic information. Therefore, the center point p is identified by MLP (MultiLayer Perceptron). i The feature descriptor G in the global scope i To extract, the expression is as follows:

[0068] G i =MLP(p i ,f i )

[0069] Among them, MLP is a forward propagation neural network with an input layer, a hidden layer, and an output layer structure. It can generate a description of the input data in the global expression space through multiple calculations of the hidden layer. i For point p i Corresponding appearance features;

[0070] Step 5.4) Fusion features

[0071] In order to enhance the center point p i The feature representation power in three-dimensional space is to add the local shape feature S in step 5.2) through element-by-element addition in deep learning i and the global feature descriptor G in step 5.3) i Add them together and input them into the conversion layer to get the weight vector Weight vector is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features, where It shows the local shape feature S i 's contribution, and The importance of the global feature descriptor is explained; finally, the weight vector is respectively combined with the local shape feature S i and the global feature descriptor G i Perform HadamardProduct operation and use the result as the center point p i Fusion features Its expression is:

[0072]

[0073]

[0074] The weight vector in the above formula is It is calculated through the conversion layer, which is composed of the BP neural network τ and the transfer function φ in deep learning. The transfer function is a function that represents the relationship between input data and output results. ⊙ is the Hadamard Product operation in matrix calculation, which is a matrix multiplication operated by element position.

[0075] The process of step six is ​​as follows:

[0076] Step 6.1) Define multi-dimensional adaptive convolution

[0077] Adaptive convolution is a linear combination of multiple weighted static convolution kernels, where the weighted order of each convolution kernel is based on the neighborhood set N i The spatial structure of the adaptive convolution is adaptively adjusted, and the calculation formula of the adaptive convolution MD_AdaptConv is as follows:

[0078]

[0079] Where W i Represents the i-th static convolution kernel, each of which has C_W in Input channels, C_W outoutput channels, WN is the number of static convolution kernels, input is the input data of the current adaptive convolution, Assigned to the static convolution kernel W i The weight order of , and all static convolution kernels The sum of the coefficients is 1, which satisfies the following formula:

[0080]

[0081] η i,j , i,j are the i-th static convolution kernel W i The ordinal weights of the jth input channel and output channel also satisfy the requirement that the sum of all channel weight coefficients is 1;

[0082]

[0083]

[0084] Step 6.2) Weight learning network

[0085] In order to realize the adaptive convolution to capture the spatial information of the point cloud, the neighborhood area N in step 5 is i Encoded local shape features As the input data of the weight learning network in the adaptive convolution, at the same time, in order to enhance the richness of the features, and neighborhood area N i The primary geometric information in is fused, which is expressed as:

[0086]

[0087] Where || || is the Euclidean distance calculation formula, I i,j The center point p i Feature representation between the jth neighboring point;

[0088] The weight learning network simulates the geometric shape of the input data based on self-learning, and models the implicit expression in the neighborhood through the feature extraction capability of the multi-layer convolutional network; each layer in the multi-layer convolutional network has the same structure, consisting of a two-dimensional convolution ψ, a normalized BN, and a Relu activation function. The input data first undergoes feature calculation of the two-dimensional convolution, and then the BN and Relu functions are used to process the calculated feature values. The weight learning network simultaneously calculates the number of convolution kernels, input channels, and output channel weights through three parallel multi-layer convolutional networks. The expression is:

[0089]

[0090]

[0091]

[0092] Where Ψ is used to represent the multi-layer convolutional network structure, Ψ i Represents the i-th multi-layer convolutional network structure;

[0093] Step 6.3) Combine multi-dimensional adaptive convolution

[0094] According to the multi-dimensional adaptive convolution formula defined in step 6.1) and the weight ordinal calculated in step 6.2), WN static convolutions are combined in turn to obtain the adaptive neighborhood N i Multi-dimensional adaptive convolution:

[0095]

[0096] The process of step eight is as follows:

[0097] Step 8.1) Channel Attention

[0098] The high-level semantic features F extracted in step 7 are S Perform global maximum pooling and average maximum pooling along the channel to obtain the global maximum feature F c_max and the average feature F c_avg ;

[0099]

[0100]

[0101] The high-level semantic feature F S With SN spatial numbers, Represents the semantic feature F S The eigenvalue when the number of channels is i and the number of spaces is d;

[0102] In order to obtain the feature map F S The amount of information on the channel, using the shared state transition function F c_max and F c_avg Calculate the amount of information, use the normalization function SoftMax to accept the amount of information of the channel and generate attention weight A for each channel c , A c The calculation formula is

[0103]

[0104] in is a shared state transfer function with a shared learnable parameter W, which is a learning-based feature calculation method in deep learning;

[0105] Attention weight A c The role of F is to measure the input feature S The importance of each channel in F S With channel attention A c By multiplying each channel, we can get the semantic enhancement feature F S1 ;

[0106] F S1 =F S ·A c

[0107] Step 8.2) Spatial Attention Mechanism

[0108] In order to establish the global spatial association and emphasize the feature map F S1 The key points in F are calculated by maximum pooling and average pooling operations. S1 The maximum feature and average feature in the spatial dimension are concatenated into F S2 ;

[0109]

[0110] in Represents the feature map F S1 The eigenvalues ​​of the i-th channel and the d-th space in F S The number of channels,

[0111] Generate feature map F using MD_AdaptConv constructed in step 6.3 S2 Attention on spatial structure s , the expression is:

[0112]

[0113] Step 8.3) Multiply the weights

[0114] The feature F with channel attention S2 With spatial attention A s Multiply them one by one according to the spatial channels to obtain a new feature F with channel attention and spatial attention fuse ;

[0115] F fuse =F S2 ·A s .

[0116] The process of step 10 is as follows:

[0117] Step 10.1) Point cloud upsampling

[0118] In Encodet During the sampling process of the farthest point of the point cloud, the number of input point clouds of each layer of the network decreases layer by layer, where the number of point clouds of the tth layer and the t+1th layer are M and M, respectively. t and M t+1 , and M t The value is greater than M t+1 ; In order to restore the decoded point cloud to its original number layer by layer, the nearest neighbor search algorithm is used to find the M t+1 Find the U_K points closest to the t-th layer of point cloud in the point cloud, and express M based on weighted interpolation. t The semantic features of each point cloud in;

[0119] The nearest neighbor search method is to calculate the Euclidean distance between the point clouds of the tth layer and the t+1th layer by global traversal, and record the U_K points with the closest Euclidean distance and use them as the nearest neighbor matching points; the Euclidean distance between the nearest neighbor matching point in the t+1th layer and the point cloud in the tth layer is recorded as the matching matrix U_D, and the matching weight U_W is assigned to each matching point according to the distance of the Euclidean distance. The closer the distance, the greater the weight of the matching point; the expressions of the matching matrix U_D and the matching weight U_W are:

[0120]

[0121] where p t 、p t+1 Represent the point clouds of layer t and layer t+1 respectively, represents the square of the difference;

[0122]

[0123] According to the matching matrix and matching weight generated by the nearest neighbor search method, M is interpolated by the feature interpolation method. t The point cloud features in are expressed as the sum of the feature weights of the nearest matching points. The feature interpolation method converts M t+1 The matching points in the layer are multiplied by the matching weight U_D, and the calculation results of all matching points are accumulated and summed, and the accumulated sum is divided by the weight sum. The expression is:

[0124]

[0125] Where W i represents the matching weight of the i-th point in the nearest neighbor matching point, Represents the semantic features of the point in the t-th layer decoding module, Represents the semantic features of the t-1th layer;

[0126] Step 10.2) Multi-layer convolution structure

[0127] The multi-layer convolution structure consists of multiple two-dimensional convolutions. The number of channels of each convolution kernel is the same as the number of point clouds in Decode. At the same time, the Relu activation function commonly used in deep learning in two-dimensional convolution is used to enhance the representation ability of features.

[0128] The process of step thirteen is as follows:

[0129] Step 13.1) Define the cross entropy loss function

[0130] Through the cross entropy loss function Loss m To calculate the difference between the point cloud label obtained in step 12 and the true label, so as to quantify the prediction error; in multi-classification tasks, the cross entropy loss function is a common model that describes the similarity between the actual output probability and the expected output probability;

[0131] Step 13.2) Define the weight loss function

[0132] In step six, the adaptive convolution is composed of multiple static convolution kernels, each of which represents a different geometric meaning. In order to ensure the diversity of static convolution kernels, the correlation between different convolution kernels is penalized by weight regularization loss. The formula is as follows:

[0133]

[0134] Step 13.3) Define the model loss function

[0135] Linearly combine the cross entropy loss function and weight loss function defined above and use it as the overall loss of the method;

[0136] Loss=Loss m +λLoss c

[0137] Among them, λ is the ordinal number assigned to the weight loss value, and its value ranges from 0 to 1. In the overall loss, the cross entropy loss function plays a major role and is used to drive the predicted results closer to the true results.

[0138] The beneficial effects of the present invention are mainly manifested in:

[0139] (1) The present invention provides a semantic modeling method for constructing an interactive and fusible digital twin model, and realizes automatic semantic segmentation of three-dimensional models based on the automatic feature extraction capability of deep learning.

[0140] (2) The present invention provides a coding method for local shape features of point clouds. By multi-level coding of the point cloud geometric space, the effective fusion of fine-grained spatial position features and overall feature descriptions is achieved, and the geometric feature integrity of the local point cloud is further enhanced.

[0141] (3) The present invention performs full-dimensional learning on the kernel space of adaptive convolution based on a parallel strategy, thereby enhancing the method's ability to capture rich contextual clues.

[0142] (4) In the process of fusing local features, the present invention adaptively optimizes semantic features through an adaptive attention mechanism to improve the representation power of the features.

[0143] (5) The present invention improves the segmentation accuracy of point cloud semantic segmentation tasks in indoor and outdoor scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] Figure 1 It is a diagram of the network structure of the adaptive attention mechanism.

[0145] Figure 2 It is a result diagram for verifying the effectiveness of the method of the present invention.

[0146] Figure 3 It is a result diagram that verifies the effectiveness of the multi-dimensional adaptive convolution in the method of the present invention.

[0147] Figure 4 This is an experimental result diagram analyzing the effectiveness of the number of kernels on multi-dimensional adaptive convolution.

[0148] Figure 5 It is a result diagram that verifies the effectiveness of local shape coding in the method of the present invention. DETAILED DESCRIPTION

[0149] The method of the present invention is further described below.

[0150] Reference Figure 1 to Figure 5 A point cloud automatic semantic modeling method for digital twin scenes includes the following steps:

[0151] Step 1: Build a point cloud dataset. The process is as follows:

[0152] Step 1.1) Define the dataset

[0153] The oblique photography technology constructs a digital twin model of an outdoor scene through a 3D reconstruction algorithm, thereby using 3D point cloud data to represent a 3D twin model of a real scene. The oblique photography technology uses a drone to take multi-view aerial photos of an outdoor scene, and restores the coordinates of the object in 3D space based on similar points in multiple views. The point cloud data consists of a set of dense 3D coordinates, each of which contains the coordinate information (x, y, z) of the point cloud in 3D space and the appearance information f that reflects the texture features of the point cloud data, such as RGB that represents the color information of the 3D object's appearance. The point cloud data set containing M points is denoted as P = {p i |i=1,...,M}, where p iTo represent the coordinate information of the i-th point, the texture feature set corresponding to P is denoted as F = {f i |i=1,…,M},f i To represent the color information of the i-th point;

[0154] Step 1.2) Point cloud data preprocessing

[0155] The point cloud dataset P is randomly rotated, scaled, and denoised to generate point cloud datasets under different conditions, thereby enhancing the method's ability to resist interference from data changes. The steps are as follows:

[0156] Step 1.2.1) Random rotation

[0157] A rotation angle θ is randomly generated, and a three-dimensional coordinate rotation matrix R(θ) is constructed based on the angle θ. Its expression is:

[0158]

[0159] Multiply the point cloud dataset P by the rotation matrix R(θ) to obtain the point cloud dataset P′ after the original point cloud is rotated by an angle θ around the z-axis. The expression is:

[0160]

[0161] Step 1.2.2) Random Scaling

[0162] Randomly generate scaling factor λ D , the scaling ratio λ D Multiplying the rotated point cloud data set P′ generates the scaled data P″, which is expressed as:

[0163] P″=λ D P′

[0164] Step 1.2.3) Add noise

[0165] Randomly generate a set of 3D coordinates J with the same dimension as the point cloud dataset P″, and the value of J is smaller. Add the scaled dataset P″ and the noise J to add a small noise interference to the original point cloud;

[0166] P″=P″+J

[0167] Step 2: Downsampling at the farthest point

[0168] In order to alleviate the computational burden caused by the huge number of point clouds, the number of original point clouds is downsampled to a fixed value through the farthest point sampling method;

[0169] Step 2.1) Determine the initial sampling point

[0170] Traverse the point cloud dataset P″ and calculate the average value of all points in the three coordinate dimensions, and use the calculated average value as the initial point D_P for sampling the farthest point 0 At the same time, add it to the sampling point set D_S = {D_P 0};

[0171] Step 2.2) Calculate the distance matrix

[0172] The farthest point sampling is to select the point farthest from D_S from the point cloud data set. Therefore, in order to obtain the distance between the point cloud data P″ and the sampling point set D_S, the Euclidean distance between each point and the newly added point in D_S is recorded through the distance matrix D_L.

[0173] Step 2.3) Update the sampling point set

[0174] The point corresponding to the maximum value in D_L is denoted as D_P i , and add it to D_S. As D_S updates the sampling points, D_L will recalculate the Euclidean distance between the newly added sampling points and the point cloud dataset, and at the same time update the point cloud whose Euclidean distance is less than the original value. Finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

[0175] Step 3: Construct a neighborhood set

[0176] The shape of a three-dimensional geometry is represented by a set of dense three-dimensional coordinates arranged in order of position. The point cloud data in the same target geometry has a high degree of similarity in spatial feature representation and high-level semantic expression. Therefore, in the point cloud semantic segmentation task, the semantic information extraction of the target geometry depends on the correlation between each point and the surrounding neighborhood points. Based on the KNN method in the traditional search algorithm, the nearest N neighborhood points of the point cloud in step 2 are calculated. Point p i The neighborhood set of i ={n i,1 ,...,n i,N}, where n i,k is the i-th point p i The kth neighboring point of , N is the number of neighboring points;

[0177] Step 4: Primary Neighborhood Features

[0178] According to the center point p i and the neighborhood set N i The three-dimensional space coordinates of the neighborhood set are expressed as <p i ,n i,k >, <p i ,n i,k > is the neighborhood set N iThe basic feature expression in the geometric space extracts the low-level geometric information in the neighborhood space, which can be used as input data for extracting high-level semantic features. At the same time, the low-level features can be fused with the high-level features to enhance the richness of the features in the spatial expression. The expression is:

[0179]

[0180] Among them (p i -n i,k ) is point p i and neighborhood point n i,k The relative coordinates, p i 、n i,k are the three-dimensional coordinates of the center point and the neighborhood point, respectively. Indicates concatenation of features along the same dimension;

[0181] Step 5: Encode local shape features. The process is as follows:

[0182] Step 5.1) Extract high-dimensional shape features

[0183] The primary neighborhood features in step 4 <p i ,n i,k >Extract high-dimensional shape features S through nonlinear functions i,k , whose expression is:

[0184] S i,k =δ(α( <p i ,n i,k >))

[0185] Where α is a multi-layer forward propagation network with a nonlinear activation function, δ is a normalization function, and a nonlinear activation function is a nonlinear function that converts input data into other data, such as the Tanh and Sigmoid functions widely used in deep learning. The forward propagation network is composed of a multi-layer network structure, and the input data of the latter layer of the network is the output data of the previous layer of the network;

[0186] Step 5.2) Extract local features

[0187] S i,k N is the neighborhood area i In order to express the integrity of the neighborhood area, the high-dimensional shape features represented by the kth neighborhood point are extracted through the global maximum pooling operation. i The significant features in the holistic feature are expressed as follows:

[0188] S i =MaxPool{S i,1 ,...,S i,K}

[0189] MaxPool is the global maximum pooling operation, which is a method for selecting the maximum eigenvalue from a local area in a convolutional neural network;

[0190] Step 5.3) Extract global feature descriptors

[0191] Center point p i It has its own semantic information in the global three-dimensional space, and also makes an important contribution to the effective identification of point cloud semantic information. Therefore, the center point p is identified by MLP (MultiLayer Perceptron). i The feature descriptor G in the global scope i To extract, the expression is as follows:

[0192] G i =MLP(p i ,f i )

[0193] Among them, MLP is a forward propagation neural network with an input layer, a hidden layer, and an output layer structure, which can generate a description of the input data in the global expression space through multiple calculations of the hidden layer. i For point p i Corresponding appearance features;

[0194] Step 5.4) Fusion features

[0195] In order to enhance the center point p i The feature representation power in three-dimensional space is to add the local shape feature S in step 5.2) through element-by-element addition in deep learning i and the global feature descriptor G in step 5.3) i Add them together and input them into the conversion layer to get the weight vector Weight vector is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features, where It shows the local shape feature S i 's contribution, and The importance of the global feature descriptor is explained. Finally, the weight vector is respectively combined with the local shape feature S i and the global feature descriptor G i Perform HadamardProduct operation and use the result as the center point p i Fusion features Its expression is:

[0196]

[0197]

[0198] The weight vector in the above formula is It is calculated through the conversion layer, which is composed of the BP neural network τ and the transfer function φ in deep learning. The transfer function is a function that represents the relationship between input data and output results. ⊙ is the Hadamard Product operation in matrix calculation, which is a matrix multiplication based on element position;

[0199] Step 6: Construct multi-dimensional adaptive convolution

[0200] In order to enhance the adaptability of the method to different target geometries, a multi-dimensional shape adaptive convolution is constructed by multi-dimensionally and comprehensively modeling the convolution kernel space, so as to flexibly simulate the spatial structure changes of the geometric body and enhance the adaptability of the method to the essential semantic structure of the point cloud; the process is as follows:

[0201] Step 6.1) Define multi-dimensional adaptive convolution

[0202] Adaptive convolution is a linear combination of multiple weighted static convolution kernels, where the weighted order of each convolution kernel is based on the neighborhood set N i The spatial structure of the adaptive convolution is adaptively adjusted, and the calculation formula of the adaptive convolution MD_AdaptConv is as follows:

[0203]

[0204] Where W i Represents the i-th static convolution kernel, each of which has C_W in Input channels, C_W out output channels. WN is the number of static convolution kernels, input is the input data of the current adaptive convolution, Assigned to the static convolution kernel W i The weight order of , and all static convolution kernels The sum of the coefficients is 1, which satisfies the following formula:

[0205]

[0206] η i,j , i,j are the i-th static convolution kernel W i The ordinal weights of the j-th input channel and output channel also satisfy the requirement that the sum of all channel weight coefficients is 1.

[0207]

[0208]

[0209] Step 6.2) Weight learning network

[0210] In order to realize the adaptive convolution to capture the spatial information of the point cloud, the neighborhood area N in step 5 is i Encoded local shape features As the input data of the weight learning network in the adaptive convolution, at the same time, in order to enhance the richness of the features, and neighborhood area N i The primary geometric information in the neighborhood is fused, such as the original three-dimensional coordinates and relative position coordinates in the neighborhood area, which is expressed as:

[0211]

[0212] Where || || is the Euclidean distance calculation formula, I i,j The center point p i Feature representation between the jth neighboring point;

[0213] The weight learning network simulates the geometric shape of the input data based on self-learning, and models the implicit expression in the neighborhood through the feature extraction capability of the multi-layer convolutional network. Each layer in the multi-layer convolutional network has the same structure, which is mainly composed of two-dimensional convolution ψ, normalized BN and Relu activation function. The input data is first calculated by the feature of two-dimensional convolution, and then the BN and Relu functions are used to process the calculated feature values. BN is a commonly used data batch normalization operation, which is used to convert data to near 0. The Relu function is used as an activation function in deep learning models to expand the expression ability of the model. The weight learning network calculates the number of convolution kernels, input channels, and output channel weights simultaneously through three parallel multi-layer convolutional networks. Its expression is:

[0214]

[0215] n i,j =Ψ 2 (p i )=Relu(Bn(ψ 2 (I i,j )))

[0216] λ i,j =Ψ 3 (p i )=Relu(Bn(ψ 3 (I i,j )))

[0217] Where Ψ is used to represent the multi-layer convolutional network structure, Ψ i Represents the i-th multi-layer convolutional network structure.

[0218] Step 6.3) Combine multi-dimensional adaptive convolution

[0219] According to the multi-dimensional adaptive convolution formula defined in step 6.1) and the weight ordinal calculated in step 6.2), WN static convolutions are combined in turn to obtain the adaptive neighborhood N i Multi-dimensional adaptive convolution:

[0220]

[0221] Step 7: Extract multi-dimensional shape semantic features

[0222] The MD_AdaptConv constructed in the previous step adaptively adjusts the convolution kernel parameters by capturing the correlation in the spatial structure of the point cloud in the neighborhood set, and converts the adaptive convolution MD_AdaptConv (p i ) i Extract semantic features to obtain high-level semantic features F with strong expressiveness S , whose expression is:

[0223] F S =MD_AdaptConv(p i )·f i

[0224] Step 8: Adaptive Attention Mechanism

[0225] As the network model becomes more complex and the feature dimension continues to increase, there is information redundancy and similar spatial structures and semantic features between high-dimensional semantic categories. By inferring the feature attention map along the channel direction and spatial direction of the feature map, and adaptively optimizing the input features through the feature attention map, more representative features can be obtained; the process is as follows:

[0226] Step 8.1) Channel Attention

[0227] like Figure 1 As shown, the high-level semantic feature F extracted in step 7 is S Perform the global maximum pooling operation MAXPooling and the average maximum pooling operation Avg Pooling along the channel, and then obtain the global maximum feature F c_max and the average feature F c_avg ;

[0228]

[0229]

[0230] The high-level semantic feature F S There are SN spatial numbers. Represents the semantic feature F S The eigenvalue when the number of channels is i and the number of spaces is d;

[0231] In order to obtain the feature map F S The amount of information on the channel, using the shared state transition function F c_max and F c_avg Calculate the amount of information, use the normalization function SoftMax to accept the amount of information of the channel and generate attention weight A for each channel c , A c The calculation formula is

[0232]

[0233] in is a shared state transfer function with a shared learnable parameter W, which is a learning-based feature calculation method in deep learning;

[0234] Attention weight A c The role of F is to measure the input feature S The importance of each channel in F S With channel attention A c By multiplying each channel, we can get the semantic enhancement feature F S1 .

[0235] F S1 =F S ·A c

[0236] Step 8.2) Spatial Attention Mechanism

[0237] In order to establish the global spatial association and emphasize the feature map F S1 The key points in F are calculated by maximum pooling and average pooling operations. S1 The maximum feature and average feature in the spatial dimension are concatenated into F S2 .

[0238]

[0239] in Represents the feature map F S1 The eigenvalues ​​of the i-th channel and the d-th space in F S The number of channels;

[0240] Generate feature map F using MD_AdaptConv constructed in step 6.3 S2 Attention on spatial structure s , the expression is:

[0241]

[0242] Step 8.3) Multiply the weights

[0243] The feature F with channel attention S2 With spatial attention A s Multiply them one by one according to the spatial channels to obtain a new feature F with channel attention and spatial attention fuse ;

[0244] F fuse =F S2 ·A s

[0245] Step 9: Encoder Module

[0246] Steps 2 to 8 are the high-level semantic feature extraction process, which is repeated C 1 The method of the present invention is verified by experiments. 1 The effect is better when the value is 4. Each execution is recorded as a layer of encoding module. The encoding module of the tth layer is Encode t In step 3, the input point cloud is downsampled by the farthest point, so the number of input point clouds in each layer of the encoding structure decreases layer by layer. At the same time, in the feature extraction process of adaptive convolution, the high-level semantic feature dimension of each layer of encoder is increasing.

[0247] Step 10: Decoder Module

[0248] The decoder module Decode consists of a multi-layer structure. Each layer of the structure first expands the number of point clouds through point cloud upsampling operations, and restores the number of point clouds layer by layer to the level before downsampling, so as to keep consistent with the number in the decoder. Secondly, each layer of the network uses a convolutional structure to extract high-level feature descriptors of the point cloud; the process is as follows:

[0249] Step 10.1) Point cloud upsampling

[0250] In Encode t During the sampling process of the farthest point of the point cloud, the number of input point clouds of each layer of the network decreases layer by layer, where the number of point clouds of the tth layer and the t+1th layer are M and M, respectively. t and M t+1 , and M t The value is greater than M t+1 In order to restore the decoded point cloud to its original number layer by layer, the nearest neighbor search algorithm is used to find the M t+1 Find the U_K points closest to the t-th layer of point cloud in the point cloud, and express M based on weighted interpolation. t The semantic features of each point cloud in;

[0251] The nearest neighbor search method is to calculate the Euclidean distance between the point clouds of the tth layer and the t+1th layer by global traversal, and record the U_K points with the closest Euclidean distance and use them as the nearest neighbor matching points. The Euclidean distance between the nearest neighbor matching points in the t+1th layer and the point cloud in the tth layer is recorded as the matching matrix U_D. At the same time, the matching weight U_W is assigned to each matching point based on the distance of the Euclidean distance. The closer the distance, the greater the weight of the matching point. The expressions of the matching matrix U_D and the matching weight U_W are:

[0252]

[0253] where p t 、p t+1 Represent the point clouds of layer t and layer t+1 respectively, Represents the square of the difference.

[0254]

[0255] According to the matching matrix and matching weight generated by the nearest neighbor search method, M is interpolated by the feature interpolation method. t The point cloud features in are represented as the sum of the feature weights of the nearest matching points. Specifically, the feature interpolation method converts M t+1 The matching points in the layer are multiplied by the matching weight U_D, and the calculation results of all matching points are accumulated and summed, and the accumulated sum is divided by the weight sum. The expression is:

[0256]

[0257] Where W i represents the matching weight of the i-th point in the nearest neighbor matching point, Represents the semantic features of the point in the t-th layer decoding module. Represents the semantic features of the t-1th layer;

[0258] Step 10.2) Multi-layer convolution structure

[0259] The multi-layer convolution structure consists of multiple two-dimensional convolutions. The number of channels of each convolution kernel is the same as the number of point clouds in Decode. At the same time, the Relu activation function commonly used in deep learning in two-dimensional convolution is used to enhance the representation ability of features.

[0260] Step 11: Jump Connection

[0261] The features of the encoder module and the decoder module are concatenated along the same dimension, and the concatenated features are used as the input of the next layer, i.e., the decoder of the tth layer. t The features of the M_C+1-t layer encoder Encode M_C+1-tThe features of are concatenated and calculated as the input of the next layer t+1. Where M_C is the total number of layers in the encoder module;

[0262] Step 12: Calculate the classification results

[0263] The semantic features obtained in step 11 are reduced in dimension through the fully connected layer module, and the category label corresponding to each point cloud is output. The fully connected layer module consists of two two-dimensional convolutions and Dropout. The input channel and output channel of the first convolution are consistent with the dimension of the semantic features, and the number of output channels of the second convolution module is the same as the number of point cloud categories. Dropout is based on the probability model to discard some of the connections between convolutional networks during the dimensionality reduction process, thereby preventing overfitting during the learning process. In deep network models, Dropout is a commonly used network structure processing method;

[0264] Step 13: Define the loss function

[0265] The segmentation accuracy of the method can be reflected by comparing the error between the predicted result and the actual result, and this error can be calculated using the loss function; the process is as follows:

[0266] Step 13.1) Define the cross entropy loss function

[0267] Through the cross entropy loss function Loss m To calculate the difference between the point cloud label obtained in step 12 and the true label, so as to quantify the prediction error. In multi-classification tasks, the cross entropy loss function is a common model that describes the similarity between the actual output probability and the expected output probability.

[0268] Step 13.2) Define the weight loss function

[0269] In step six, the adaptive convolution is composed of multiple static convolution kernels, each of which represents a different geometric meaning. In order to ensure the diversity of static convolution kernels, the correlation between different convolution kernels is penalized by weight regularization loss. The specific formula is as follows:

[0270]

[0271] Step 13.3) Define the model loss function

[0272] Linearly combine the cross entropy loss function and weight loss function defined above and use it as the overall loss of the method;

[0273] Loss=Loss m +λLoss c

[0274] Among them, λ is the ordinal number assigned to the weight loss value, and its value ranges from 0 to 1. In the overall loss, the cross entropy loss function plays a major role and is used to drive the predicted results closer to the true results.

[0275] Reference Figure 2 to Figure 4 , the effectiveness of the method of the present invention is verified on an indoor public data set, including the following steps:

[0276] Step 1: Define the public dataset

[0277] The S3DIS dataset (Stanford Large-scale 3D Indoor Spaces Dataset) is an indoor point cloud dataset provided by Stanford University, which is collected by sensors from 6 real indoor scenes. The S3DIS dataset contains a total of 273 million point clouds, each of which consists of 3D spatial coordinates, RGB colors, and normalized coordinates. The indoor scenes collected by the S3DIS dataset mainly consist of ceilings, floors, wall beams, tables and chairs, sofas, beams, columns, windows, doors, bookcases, boards, and other elements.

[0278] In order to enhance the robustness of the method of the present invention and enhance the efficiency of the method in capturing high-level intrinsic structural information, the original data of the present invention is preprocessed by random scaling, rotation, adding disturbance points, etc. At the same time, during the training process, the method of the present invention divides each scene in the S3DIS data set into 1m×1m blocks, and 4096 points randomly selected in the blocks are used as input data of the method of the present invention. In addition, in order to speed up the convergence of the model, the MomentumGradientDescent (MGD) optimizer is used to adjust the parameter learning of the model during the training process. MGD is a gradient optimization algorithm used to optimize deep learning models, which improves the model accuracy by adjusting the model parameters.

[0279] Step 2: Define evaluation metrics

[0280] In order to quantify the semantic segmentation accuracy of the method of the present invention, the present invention uses two indicators, mean Intersection over Union (mIoU) and mean class Accuracy (mAcc), to calculate the semantic effect of the method of the present invention on the S3DIS dataset. The calculation formulas of mIou and mAcc are:

[0281]

[0282]

[0283] The number of semantic categories in the point cloud dataset is Cnum, and the prediction results and label results calculated in the method of the present invention are both the number of the i-th type of point cloud, which is recorded as Pr ii , Pr ij It represents the number of point clouds with label category i and predicted category j. SCN is the number of all point cloud semantic categories in the S3DIS dataset.

[0284] Step 3: Define the point cloud semantic segmentation method

[0285] PointNet++: It uses the farthest point downsampling and KNN neighborhood query method to build a hierarchical network to extract multi-level and multi-scale point cloud features.

[0286] LSACA-Net: This invention proposes an automatic semantic modeling method for point clouds for digital twin scenarios. LSACA-Net consists of three parts. First, the local shape feature module (LSFM) is used to fuse fine-grained and holistic primary geometric features, giving each feature the ability to represent the geometric space. Secondly, a multi-dimensional adaptive convolution module (MD-AdaptConv) is constructed to flexibly simulate the spatial structure of the point cloud and extract more expressive fine-grained features in the local topological structure. Finally, the adaptive convolution attention module (ACAM) is used to effectively capture the potential relationship between high-level semantic features, thereby enhancing the identification ability of semantic features.

[0287] Paconv: Paconv constructs an adaptive convolution kernel based on the basic weight matrix in the dynamic assembly weight library based on the relative position relationship in the neighborhood space, which is used to flexibly simulate complex spatial changes.

[0288] Kpconv: Kpconv calculates the weight of the convolution kernel by constructing a linear relationship between the kernel points in Euclidean space and the local neighborhood points

[0289] Step 4: Verify the effectiveness of LSACA-Net

[0290] In order to prove the effectiveness of the LSACA-Net method of the present invention, the present invention uses PointNet++ as the benchmark model and adds LSFM, MD-AdaptConv, and ACAM modules to the PointNet++ model in turn.

[0291] Step 4.1) Build a comparison model

[0292] like Figure 2 As shown in the figure, the present invention constructs four comparative models, model (a) is the benchmark model PointNet++, model (b) replaces the multi-dimensional adaptive convolution MD-AdaptConv in the method of the present invention with the convolution method in PointNet++, model (c) adds LSFM on the basis of model (b), and model (d) is PointNet++ with three modules at the same time.

[0293] Step 4.2) Calculate the semantic segmentation results

[0294] The present invention uses the Area5 dataset of S3DIS to conduct quantitative evaluation experiments on the above four models. The quantitative analysis results are as follows: Figure 2 As shown. By comparing the experimental results of model (a) and model (b), it can be seen that the model (b) that uses MD-AdaptConv to extract point cloud semantic features is 5.2% higher than the model (a). It can be verified that MD-AdaptConv is more suitable for irregular point cloud data and can effectively improve the network segmentation performance. Comparing model (b) and model (c), the mIou is increased by about 1% after adding the LSFM model. It can be verified that LSFM that integrates local shape features helps to improve the accuracy of semantic segmentation. Similarly, the effectiveness of the ACAM module can be verified by model (c) and model (d). The ACAM module helps model (c) improve the mIoU by nearly 1%. Through the above comparison, it can be seen that each module in LSACA-Net helps to improve the performance of point cloud semantic segmentation.

[0295] Step 5: Verify the effectiveness of multi-dimensional adaptive convolution. The process is as follows:

[0296] Step 5.1) Build a comparison model

[0297] The key to MD-AdaptConv is to consider the impact of three dimensions on adaptive convolution: the number of convolution kernels (Kernel Number), the convolution kernel input channels (Kernel Input), and the convolution kernel output channels (Kernel Output). Therefore, in order to evaluate the impact of different dimensions on convolution performance, the present invention defines and designs the following groups of comparison models: MD-AdaptConv in model (①) only considers the number of convolution kernels, model (②) considers the number of convolution kernels and the number of input channels, model (③) considers the number of convolution kernels and the number of output channels, and model (④) considers the number of convolution kernels, the number of input channels, and the number of output channels at the same time.

[0298] Step 5.2) Calculate the semantic segmentation results

[0299] The four models in step 5.1) are quantitatively evaluated on the S3DIS test dataset Area5. The calculation results are shown in Figure 3 As shown in the figure, the method can achieve optimal performance only when these three dimensions are taken into account at the same time. This proves that the three dimensions are complementary. In addition, the number of convolution kernels in MD-AdaptConv does not need to be a large value to capture local spatial changes.

[0300] Step 5.3) Analyze the performance of the number of convolution kernels

[0301] Figure 4 The effect of the number of convolution kernels on MD-AdaptConv’s ability to extract features is shown. Figure 4 It can be seen that the best effect is achieved when the number of kernels is 4, while Kpconv and Paconv usually require 16 static convolution kernels to capture the deformation of the local space. Therefore, the method of the present invention only needs a small number of static convolution kernels to achieve the effect of Kpconv and Paconv, and more importantly, it can greatly reduce the computational cost.

[0302] Step 6: Verify the validity of the local encoding module. The process is as follows:

[0303] Step 6.1) Build a comparison model

[0304] In order to verify the performance of LSFM in the present invention, the present invention selects several common local spatial feature representation methods from Pointnet++ to replace LSFM in model (d) in step three. Figure 5 The segmentation accuracy of different spatial feature representations on the S3DIS test set is shown. The spatial features in Method (I) are the coordinates of the center point and the neighborhood point in the local area (x i ,y i ,z i ,x j ,y j ,z j ), Method (Ⅱ) is the relative position of the center point coordinates and the neighborhood point coordinates (x i -x j ,y i -y j ,z i -z j ), Method (Ⅲ) is the coordinates of the center point, the relative position of the center point and the neighboring points (x i ,y i ,z i ,x i -x j ,y i-y j ,z i -z j ), Method (Ⅳ) is the coordinates of the center point, the relative position of the center point and the neighboring points, and the Euclidean distance within the neighborhood (x i ,y i ,z i ,x i -x j ,y i -y j ,z i -z j ,d ij ), Method (V) is the LSFM method proposed in the present invention.

[0305] Step 6.2) Quantitative analysis of experimental results

[0306] Depend on Figure 5 The experimental results show that compared with other point cloud spatial feature representation methods, LSFM has favorable advantages, further proving that the fusion of local shape encoding and fine-grained point pair relationship features is more conducive to the expression of semantic information of point clouds.

[0307] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A point cloud automatic semantic modeling method for digital twin scenes, characterized in that: The method comprises the following steps: Step 1: Build a point cloud dataset. The process is as follows: Step 1.1) Define the dataset The oblique photography technology constructs a digital twin model of an outdoor scene through a 3D reconstruction algorithm, thereby using 3D point cloud data to represent the 3D twin model of the real scene; the oblique photography technology uses drones to take multi-view aerial photos of outdoor scenes, and restores the coordinates of objects in 3D space based on similar points in multiple views; the point cloud data consists of a set of dense 3D coordinates, each of which contains the coordinate information (x, y, z) of the point cloud in 3D space and the appearance information f that reflects the texture characteristics of the point cloud data; the point cloud data set containing M points is denoted as P = {p i |i=1,…,M}, where p i To represent the coordinate information of the i-th point, the texture feature set corresponding to P is denoted as F = {f i |i=1,…,M},f i To represent the color information of the i-th point; Step 1.2) Point cloud data preprocessing The point cloud dataset P is subjected to random rotation, scaling, and noise preprocessing operations to generate point cloud datasets under different conditions, thereby enhancing the method's ability to resist interference from data changes; Step 2: Downsampling at the farthest point: downsampling the number of original point clouds to a fixed value through the farthest point sampling method; Step 3: Construct a neighborhood set The shape of a three-dimensional geometry is represented by a set of dense three-dimensional coordinates arranged in order of position. The point cloud data in the same target geometry has a high degree of similarity in spatial feature representation and high-level semantic expression. Therefore, in the task of point cloud semantic segmentation, the semantic information extraction of the target geometry depends on the correlation between each point and its surrounding neighborhood points. Based on the KNN method in the traditional search algorithm, the nearest N neighborhood points of the point cloud in step 2 are calculated. Point p i The neighborhood set of i ={n i,1 ,...,n i, N}, where n i,k is the i-th point p i The kth neighboring point of , N is the number of neighboring points; Step 4: Primary Neighborhood Features According to the center point p i and neighborhood set N i The three-dimensional space coordinates of the neighborhood set are expressed as <p i ,n i,k >, <p i ,n i,k > is the neighborhood set N i The basic feature expression in the geometric space extracts the low-level geometric information in the neighborhood space, which can be used as input data for extracting high-level semantic features. At the same time, the low-level features can be fused with the high-level features to enhance the richness of the features in the spatial expression. The expression is: Among them (p i -n i,k ) is point p i and neighborhood point n i,k The relative coordinates, p i 、n i,k are the three-dimensional coordinates of the center point and the neighborhood point, respectively. Indicates concatenation of features along the same dimension; Step 5: Encode local shape features; Step 6: Construct multi-dimensional adaptive convolution By modeling the convolution kernel space in a multi-dimensional and comprehensive manner, a multi-dimensional shape-adaptive convolution is constructed to flexibly simulate the spatial structure changes of geometric bodies and enhance the method's ability to adapt to the essential semantic structure of point clouds. The process of step six is ​​as follows: Step 6.1) Define multi-dimensional adaptive convolution Adaptive convolution is a linear combination of multiple weighted static convolution kernels, where the weighted order of each convolution kernel is based on the neighborhood set N i The spatial structure of the adaptive convolution is adaptively adjusted, and the calculation formula of the adaptive convolution MD_AdaptConv is as follows: Where W i Represents the i-th static convolution kernel, each of which has C_W in Input channels, C_W out output channels, WN is the number of static convolution kernels, input is the input data of the current adaptive convolution, Assigned to the static convolution kernel W i The weight order of , and all static convolution kernels The sum of the coefficients is 1, which satisfies the following formula: η i,j , i,j are the i-th static convolution kernel W i The ordinal weights of the jth input channel and output channel also satisfy the requirement that the sum of all channel weight coefficients is 1; Step 6.2) Weight learning network In order to realize the adaptive convolution to capture the spatial information of the point cloud, the neighborhood area N in step 5 is i Encoded local shape feature X pi As the input data of the weight learning network in the adaptive convolution, at the same time, in order to enhance the richness of the features, X pi and neighborhood area N i The primary geometric information in is fused, which is expressed as: Where || || is the Euclidean distance calculation formula, I i,j The center point p i Feature representation between the jth neighboring point; The weight learning network simulates the geometric shape of the input data based on self-learning, and models the implicit expression in the neighborhood through the feature extraction capability of the multi-layer convolutional network; each layer in the multi-layer convolutional network has the same structure, consisting of a two-dimensional convolution ψ, a normalized BN, and a Relu activation function. The input data first undergoes feature calculation of the two-dimensional convolution, and then the BN and Relu functions are used to process the calculated feature values. The weight learning network simultaneously calculates the number of convolution kernels, input channels, and output channel weights through three parallel multi-layer convolutional networks. The expression is: n i,j =Ψ2(p i )=Relu(Bn(ψ2(I i,j ))) l i,j =Ψ3(p i )=Relu(Bn(ψ3(I i,j ))) Where Ψ is used to represent the multi-layer convolutional network structure, Ψ i Represents the i-th multi-layer convolutional network structure; Step 6.3) Combine multi-dimensional adaptive convolution According to the multi-dimensional adaptive convolution formula defined in step 6.1) and the weight ordinal calculated in step 6.2), WN static convolutions are combined in turn to obtain the adaptive neighborhood N i Multi-dimensional adaptive convolution: Step 7: Extract multi-dimensional shape semantic features The multi-dimensional adaptive convolution MD_AdaptConv constructed in step 6 adaptively adjusts the convolution kernel parameters by capturing the correlation in the spatial structure of the point cloud in the neighborhood set, and converts the adaptive convolution MD_AdaptConv (p i ) i Extract semantic features to obtain high-level semantic features F with strong expressiveness S , whose expression is: F S =MD_AdaptConv(p i )·f i Step 8: Adaptive Attention Mechanism As the network model becomes more complex and the feature dimension continues to increase, there is information redundancy and similar spatial structures and semantic features between high-dimensional semantic categories. By inferring the feature attention map along the channel direction and spatial direction of the feature map, and adaptively optimizing the input features through the feature attention map, more representative features can be obtained; Step 9: Encoder Module Steps 2 to 8 are the high-level semantic feature extraction process, which is repeated C1 times. Each execution is recorded as a layer of encoding module. The encoding module of the tth layer is Encode t Indicates that in step 3, the input point cloud is downsampled by the farthest point, so the number of input point clouds in each layer of the encoding structure decreases layer by layer; at the same time, in the feature extraction process of adaptive convolution, the high-level semantic feature dimension of each layer of encoder is increasing; Step 10: Decoder Module The decoder module Decode consists of a multi-layer structure. Each layer first expands the number of point clouds through point cloud upsampling operations, and restores the number of point clouds layer by layer to the level before downsampling, so as to keep consistent with the number in the decoder. Secondly, each layer of the network uses a convolutional structure to extract high-level feature descriptors of the point cloud; Step 11: Jump Connection The features of the encoder module and the decoder module are concatenated along the same dimension, and the concatenated features are used as the input of the next layer, i.e., the decoder of the tth layer. t The features and the encoder Encoder of the M_C+1-t layer M_C+1-t The features of are concatenated and calculated as the input of the next layer t+1, where M_C is the total number of layers in the encoder module; Step 12: Calculate the classification results The semantic features obtained in step 11 are reduced in dimension through the fully connected layer module, and the category label corresponding to each point cloud is output. The fully connected layer module consists of two two-dimensional convolutions and Dropout. The input channel and output channel of the first convolution are consistent with the dimension of the semantic features, and the number of output channels of the second convolution module is the same as the number of point cloud categories. Dropout is based on the probability model to discard some connections between convolutional networks during the dimensionality reduction process, thereby preventing overfitting during the learning process; Step 13: Define the loss function The segmentation accuracy of the method can be reflected by comparing the error between the predicted result and the true result, and the error is calculated using a loss function.

2. The method for automatic semantic modeling of point clouds for digital twin scenes according to claim 1, characterized in that: The step 1.2) comprises the following steps: Step 1.2.1) Random rotation A rotation angle θ is randomly generated, and a three-dimensional coordinate rotation matrix R(θ) is constructed based on the angle θ. Its expression is: Multiply the point cloud dataset P by the rotation matrix R(θ) to obtain the point cloud dataset P′ after the original point cloud is rotated by an angle θ around the z-axis. The expression is: Step 1.2.2) Random Scaling Randomly generate scaling factor λ D , the scaling ratio λ D Multiplying the rotated point cloud data set P′ generates the scaled data P″, which is expressed as: P″=λ D P′ Step 1.2.3) Add noise Randomly generate a set of 3D coordinates J with the same dimension as the point cloud dataset P″; add the scaled dataset P″ and the noise J to add a tiny noise interference to the original point cloud; P″=P″+J.

3. The method for automatic semantic modeling of point clouds for digital twin scenes according to claim 1 or 2, characterized in that: The process of step 2 is as follows: Step 2.1) Determine the initial sampling point Traverse the point cloud data set P″ and calculate the average value of all points in the three coordinate dimensions. The calculated average value is used as the initial point D_P0 for sampling the farthest point. At the same time, it is added to the sampling point set D_S = {D_P0}; Step 2.2) Calculate the distance matrix The farthest point sampling is to select the point farthest from D_S from the point cloud data set. Therefore, in order to obtain the distance between the point cloud data P″ and the sampling point set D_S, the Euclidean distance between each point and the newly added point in D_S is recorded through the distance matrix D_L; Step 2.3) Update the sampling point set The point corresponding to the maximum value in D_L is denoted as D_P i , and add it to D_S. As D_S updates the sampling points, D_L will recalculate the Euclidean distance between the newly added sampling points and the point cloud dataset, and at the same time update the point cloud whose Euclidean distance is less than the original value. Finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

4. The method for automatic semantic modeling of point clouds for digital twin scenes according to claim 1 or 2, characterized in that: The process of step five is as follows: Step 5.1) Extract high-dimensional shape features The primary neighborhood features in step 4 <p i ,n i,k >Extract high-dimensional shape features S through nonlinear functions i,k , whose expression is: S i,k =δ(α(<p i ,n i,k >)) Among them, α is a multi-layer forward propagation network with a nonlinear activation function, δ is a normalization function, and the nonlinear activation function is a nonlinear function that converts input data into other data. The forward propagation network is composed of a multi-layer network structure, and the input data of the latter layer of the network is the output data of the previous layer of the network; Step 5.2) Extract local features S i,k N is the neighborhood area i In order to express the integrity of the neighborhood area, the high-dimensional shape features represented by the kth neighborhood point are extracted through the global maximum pooling operation. i The significant features in the holistic feature are expressed as follows: S i =MaxPool{S i,1 ,...,S i,K } MaxPool is the global maximum pooling operation, which is a method for selecting the maximum eigenvalue from a local area in a convolutional neural network; Step 5.3) Extract global feature descriptors Center point p i It has its own semantic information in the global three-dimensional space, and also makes an important contribution to the effective identification of point cloud semantic information. Therefore, the center point p is identified by MLP (MultiLayer Perceptron). i The feature descriptor G in the global scope i To extract, the expression is as follows: G i =MLP(p i ,f i ) Among them, MLP is a forward propagation neural network with an input layer, a hidden layer, and an output layer structure. It can generate a description of the input data in the global expression space through multiple calculations of the hidden layer. i For point p i Corresponding appearance features; Step 5.4) Fusion features In order to enhance the center point p i The feature representation power in three-dimensional space is to add the local shape feature S in step 5.2) through element-by-element addition in deep learning i and the global feature descriptor G in step 5.3) i Add them together and input them into the conversion layer to get the weight vector Weight vector is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features, where It shows the local shape feature S i 's contribution, and The importance of the global feature descriptor is explained; finally, the weight vector is respectively combined with the local shape feature S i and the global feature descriptor G i Perform HadamardProduct operation and use the result as the center point p i Fusion features Its expression is: The weight vector in the above formula is It is calculated through the conversion layer, which is composed of the BP neural network τ and the transfer function φ in deep learning. The transfer function is a function that represents the relationship between input data and output results. ⊙ is the Hadamard Product operation in matrix calculation, which is a matrix multiplication operated by element position.

5. The method for automatic semantic modeling of point clouds for digital twin scenes according to claim 1 or 2, characterized in that: The process of step eight is as follows: Step 8.1) Channel Attention The high-level semantic features F extracted in step 7 are S Perform global maximum pooling and average maximum pooling along the channel to obtain the global maximum feature F c_max and the average feature F c_avg ; The high-level semantic feature F S With SN spatial numbers, Represents the semantic feature F S The eigenvalue when the number of channels is i and the number of spaces is d; In order to obtain the feature map F S The amount of information on the channel, using the shared state transition function F c_max and F c_avg Calculate the amount of information, use the normalization function SoftMax to accept the amount of information of the channel and generate attention weight A for each channel c , A c The calculation formula is in is a shared state transfer function with a shared learnable parameter W, which is a learning-based feature calculation method in deep learning; Attention weight A c The role of F is to measure the input feature S The importance of each channel in F S With channel attention A c By multiplying each channel, we can get the semantic enhancement feature F S1 ; F S1 =F S ·A c Step 8.2) Spatial Attention Mechanism In order to establish the global spatial association and emphasize the feature map F S1 The key points in F are calculated by maximum pooling and average pooling operations. S1 The maximum feature and average feature in the spatial dimension are concatenated into F S2 ; in Represents the feature map F S1 The eigenvalues ​​of the i-th channel and the d-th space in F S The number of channels, Generate feature map F using MD_AdaptConv constructed in step 6.3 S2 Attention on spatial structure s , the expression is: Step 8.3) Multiply the weights The feature F with channel attention S2 With spatial attention A s Multiply them one by one according to the spatial channels to obtain a new feature F with channel attention and spatial attention fuse ; F fuse =F S2 ·A s 。 6. The method for automatic semantic modeling of point clouds for digital twin scenes according to claim 1 or 2, characterized in that: The process of step 10 is as follows: Step 10.1) Point cloud upsampling In Encode t During the sampling process of the farthest point of the point cloud, the number of input point clouds of each layer of the network decreases layer by layer, where the number of point clouds of the tth layer and the t+1th layer are M and M, respectively. t and M t+1 , and M t The value is greater than M t+1 ; In order to restore the decoded point cloud to its original number layer by layer, the nearest neighbor search algorithm is used to find the M t+1 Find the U_K points closest to the t-th layer of point cloud in the point cloud, and express M based on weighted interpolation. t The semantic features of each point cloud in; The nearest neighbor search method is to calculate the Euclidean distance between the point clouds of the tth layer and the t+1th layer by global traversal, and record the U_K points with the closest Euclidean distance and use them as the nearest neighbor matching points; the Euclidean distance between the nearest neighbor matching point in the t+1th layer and the point cloud in the tth layer is recorded as the matching matrix U_D, and the matching weight U_W is assigned to each matching point according to the distance of the Euclidean distance. The closer the distance, the greater the weight of the matching point; the expressions of the matching matrix U_D and the matching weight U_W are: where p t 、p t+1 Represent the point clouds of layer t and layer t+1 respectively, represents the square of the difference; According to the matching matrix and matching weight generated by the nearest neighbor search method, M is interpolated by the feature interpolation method. t The point cloud features in are expressed as the sum of the feature weights of the nearest matching points. The feature interpolation method converts M t+1 The matching points in the layer are multiplied by the matching weight U_D, and the calculation results of all matching points are accumulated and summed, and the accumulated sum is divided by the weight sum. The expression is: Where W i represents the matching weight of the i-th point in the nearest neighbor matching point, Represents the semantic features of the point in the t-th layer decoding module, Represents the semantic features of the t-1th layer; Step 10.2) Multi-layer convolution structure The multi-layer convolution structure consists of multiple two-dimensional convolutions. The number of channels of each convolution kernel is the same as the number of point clouds in Decode. At the same time, the Relu activation function commonly used in deep learning in two-dimensional convolution is used to enhance the representation ability of features.

7. The method for automatic semantic modeling of point clouds for digital twin scenes according to claim 1 or 2, characterized in that: The process of step thirteen is as follows: Step 13.1) Define the cross entropy loss function Through the cross entropy loss function Loss m To calculate the difference between the point cloud label obtained in step 12 and the true label, so as to quantify the prediction error; in multi-classification tasks, the cross entropy loss function is a common model that describes the similarity between the actual output probability and the expected output probability; Step 13.2) Define the weight loss function In step six, the adaptive convolution is composed of multiple static convolution kernels, each of which represents a different geometric meaning. In order to ensure the diversity of static convolution kernels, the correlation between different convolution kernels is penalized by weight regularization loss. The formula is as follows: Step 13.3) Define the model loss function Linearly combine the cross entropy loss function and weight loss function defined above and use it as the overall loss of the method; Loss=Loss m +λLoss c Among them, λ is the ordinal number assigned to the weight loss value, and its value ranges from 0 to 1. In the overall loss, the cross entropy loss function plays a major role and is used to drive the predicted results closer to the true results.

Images