A semantic annotation method for graph convolutional point clouds driven by inter-class representation comparison

Abstract

The invention discloses a graph convolution point cloud semantic labeling method driven by inter-class representation contrast, belongs to the technical field of photogrammetry, and is used for semantic labeling of point clouds. The invention designs a geometric structure encoding module to describe the geometric features within the neighborhood of each point; utilizes a geometric structure and semantic feature fusion graph convolution module, extracts the feature relationship between the center point and each point in the neighborhood by stacked transfer edge convolution between layers, and aggregates the edge feature information into the corresponding center node by relying on a self-attention mechanism; at the same time, the geometric structure encoding information in each encoding layer is transferred between multiple layers, thereby providing a richer spatial basis for mining the semantic relationship of neighboring points; adopts a multi-task optimization strategy driven by inter-class representation contrast and cross entropy loss collaboration, so that the feature distance of points of the same category is closer in the high-dimensional semantic feature space, while the feature distance of points of different categories is farther, thereby obtaining point features with richer semantic expression capabilities.

Description

Technical Field

[0001] The invention discloses a graph convolution point cloud semantic annotation method driven by inter-class representation comparison, and belongs to the technical field of photogrammetry. Background Art

[0002] With the rapid development of 3D data acquisition technology, 3D LiDAR systems have become an indispensable means of 3D data acquisition with their advantages of fast, direct and high-precision acquisition of 3D spatial information on the surface. They have been widely used in cutting-edge fields such as autonomous driving, computer vision, and robotics. Discrete 3D point cloud data is the most widely used data form acquired by 3D LiDAR systems. Since it only contains position information in 3D space but does not carry semantic information, the semantic annotation of discrete 3D point clouds is the basic work for its subsequent applications. The basic principle of semantic annotation is to use computers to quantitatively analyze the different features reflected by different objects in 3D point cloud data, classify each point into one of several categories, and thus establish an association mechanism between discrete 3D point clouds and their semantic labels.

[0003] Since discrete 3D point cloud data has the characteristics of disorder and unstructure, it is difficult to process it using traditional convolutional neural networks. Initially, discrete 3D point cloud data was discretized into regular data formats, such as voxels, multi-views, etc. The voxelization-based method is to perform three-dimensional (3D) convolution operations on the point cloud based on the voxelization expression. Although voxelization solves the problems of disorder and unstructuredness of point clouds, it will cause information loss in the voxelization expression process, and the voxel format will occupy a lot of memory; the multi-view-based method is to project the 3D point cloud into multiple views, perform two-dimensional (2D) convolution on this basis, and finally project the 2D semantic segmentation results back to 3D space. In the process of information conversion, it will also cause serious information loss such as geometric structure, making it difficult to obtain satisfactory semantic annotation results. As the first semantic segmentation model that directly processes points, PointNet pioneered the use of T-Net and maximum pooling structures to solve the rotation invariance and disorder of point clouds respectively. In addition, it also extracts and expresses the 3D coordinate information of point clouds into higher-dimensional semantic information by stacking weight-sharing multi-layer perceptrons (MLP). However, this network only applies symmetric functions independently on each point to extract point-by-point features of the point cloud, ignoring the semantic correlation between 3D points, resulting in low quality of semantic annotation. Summary of the invention

[0004] The purpose of the present invention is to disclose a graph convolution point cloud semantic annotation method driven by inter-class representation comparison, so as to solve the problem of low accuracy of semantic annotation results in the prior art.

[0005] A semantic annotation method for graph convolutional point clouds driven by inter-class representation comparison, comprising:

[0006] S1. Use geometric structure coding to describe the point cloud neighborhood structure;

[0007] S2. The geometric structure encoding information in each encoding layer is transferred between multiple layers, and the geometric structure information under different receptive fields provides a richer spatial basis for mining the semantic relationship of neighboring points;

[0008] S3. Combining structural information with semantic features, the graph convolution module is based on the neighborhood graph structure. In the local neighborhood range, the intra-layer extraction and transmission of neighborhood information is realized through edge convolution and attention pooling. The geometric structure information is used to fully explore the semantic correlation between adjacent points and continuously enrich the semantic feature information.

[0009] S4. Using inter-class representation contrast as a driver, the contrast loss function is used to aggregate high-dimensional semantic features of the same category in the feature space, and to make semantic features of points of different categories as far apart as possible. The inter-class representation contrast loss and cross entropy loss are used to drive the optimization strategy to train the classification results using gradient back propagation.

[0010] S5. Build a point cloud semantic annotation network model and use it to conduct semantic annotation test experiments on 3D point cloud data.

[0011] S1 includes: defining a tensor P = [p1, p2, ..., p i ] T To represent the point cloud in space, where p i Represents the i-th point, establishes a local neighborhood for each center point, constructs a K-nearest neighbor algorithm to search for the K points closest to each center point, uses Euclidean distance as the distance metric, and obtains the local neighborhood of the i-th point [p1, p2, …, p n ], spatial position encoding r i for: in represents the K neighboring points corresponding to each center point, ‖‖ represents the Euclidean distance between the center point and its neighboring points, […] represents the cascade operation on the feature dimension, and the geometric structure information r of the point cloud is obtained. The size of this tensor can be expressed as in Represents a tensor of a specific shape and size, N represents the number of point clouds, 10 represents the feature dimension described by the spatial position code, and the features described are within the neighborhood. The point cloud feature information is aggregated within the neighborhood through the attention pooling operation to obtain the geometric structure code g corresponding to each point. The size of the tensor can be expressed as

[0012] Assume g mThe geometric structure encoding information is obtained in the mth layer. After the geometric structure encoding of the previous layer is transmitted, the geometric structure encoding g containing point clouds with different receptive fields is obtained. m′ :g m′ =[g 1 ,DS(g m-1 ),…,DS(g m )], where DS represents a downsampling operation;

[0013] The semantic feature of point cloud is represented as F = [f1,f2,…,f n ], the mth layer g m′ After being cascaded with the semantic feature D in the feature dimension, the structural information and the semantic feature are input into the graph convolution module, and the result is expressed as: G = [g m′ ,F].

[0014] The graph convolution module for fusing structural information and semantic features includes a transfer edge convolution layer, an attention pooling layer, and a residual stacked graph convolution.

[0015] The process of information transfer by the transfer edge convolution layer includes:

[0016] S2.1. Build a graph model, use KNN to obtain the indexes of the K nearest points to each point, and establish directed edges between the center point and the neighboring points;

[0017] S2.2. Obtain edge vectors. When establishing directed edges, feature vectors with rich semantic information and geometric coding letters with neighborhood structures are used. The edge vector is expressed as: Among them G i represents the geometric structure encoding corresponding to the i-th point, Represents the geometric structure encoding of the jth point in the neighborhood corresponding to the i-th point, It indicates the directed edge information between the i-th point and the j-th point in the corresponding neighborhood;

[0018] S2.3. Extract edge features. The edge vector is passed through three layers of continuously stacked multi-layer perceptrons (MLP) to extract edge attribute features.

[0019] S2.4. Aggregate edge attribute features and use the self-attention mechanism to autonomously learn important neighborhood features to maximize the representation of the neighborhood information contained therein. The final result is expressed as: F i =h Θ (AttPool(E i ), where AttPool represents the attention pooling operation, E i represents the edge attribute features corresponding to the i-th point and its neighboring points, F i It represents the feature vector rich in semantic information of the i-th point after edge feature aggregation.

[0020] The attention pooling layer includes:

[0021] Given the neighborhood edge attribute feature E i , the attention score with normalized characteristics in the neighborhood dimension is calculated by the following formula: score = δ(w,E i ), where w represents a set of weights shared by a group of weights, δ represents the softmax activation function used to normalize the attention score weights and a single-layer MLP;

[0022] The learned attention score is regarded as a mask for automatically selecting important latent features. The attention score is multiplied element by element with the corresponding center point edge attribute feature to generate a set of weighted features. The weighted features are summed to obtain the locally representative center point feature.

[0023] The residual stacked graph convolution includes:

[0024] By stacking continuous transfer edge convolutional layers, the feature aggregation and transfer between neighborhoods are used to continuously expand the point cloud receptive field without changing the number of neighborhood points; and residual connections are used to form residual stacked graph convolution.

[0025] The point cloud semantic annotation network model contains 4 downsampling layers. The number of original point clouds input to the model is 45056, and the corresponding semantic feature dimension is 8. The number of points obtained after each downsampling is 11264, 2816, 704, and 176, respectively, and the corresponding semantic feature dimensions are 16, 64, 128, and 256, respectively.

[0026] During the downsampling process, a part is randomly selected directly from the original point cloud, and the point cloud before the i-th layer sampling is set as ε i , the sampled point cloud is ε i+1 , firstly with ε i+1 The center point is ε i Find the local neighborhood in The neighborhood features are used to characterize the features of the center point, and the maximum pooling is used to aggregate the neighborhood feature information. The feature information of the unsampled points is fully mined and integrated into the sampled points, reducing the information loss caused by the point cloud downsampling process.

[0027] Upsampling uses a relatively light nearest neighbor interpolation method, and uses a skip connection structure to fuse the rich semantic feature information in the high-level layer with the rich spatial structure information in the low-level layer.

[0028] The upsampling also goes through 4 layers, and the number of point clouds corresponding to each layer is 704, 2816, 11264, and 45056, respectively, and the corresponding semantic feature dimensions are 256, 128, 32, and 32, respectively. The semantic feature information of the original point is obtained by upsampling, and then passes through multiple fully connected layers to obtain the score of each point in each category.

[0029] The contrast loss function driven by inter-class representation contrast is:

[0030]

[0031] where μ v represents the high-dimensional semantic feature vector corresponding to a specific category v extracted by the last layer of the network, δ i It represents the continuously updated average feature vector in the i-th category, D(·) here uses the Euclidean distance as the distance measurement standard, Δ represents the similarity between feature vectors; the final loss function is: L total =λ×L cont +L cro ,λ is 3.0, where L cro represents the cross entropy loss function.

[0032] Compared with the prior art, the present invention has the following beneficial effects: a multi-task optimization strategy driven by inter-class representation comparison and cross entropy loss is adopted to realize the convergence and fusion of point cloud spatial context semantic information, so as to generate high-level semantic feature expressions with strong distinguishing ability. At the same time, similar semantic features are closer in high-dimensional space and different semantic features are farther away, thereby improving the descriptive and discriminative capabilities of semantic features. The present invention effectively improves the quality and efficiency of three-dimensional point cloud semantic annotation, and also provides theoretical support for the subsequent in-depth exploration of semantic feature expressions of three-dimensional point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a technical flow chart of the present invention.

[0034] Figure 2 Structure diagram of the point cloud semantic annotation network model.

[0035] Figure 3 The structural diagram of the graph convolution module that fuses geometric structure and semantic features.

[0036] Figure 4 Schematic diagram of increasing the receptive field of the residual-stacked dilated convolution module.

[0037] Figure 5 Schematic diagram of the differences caused by geometric structure encoding information under different receptive fields. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The technical process of the present invention is as follows Figure 1 This paper mainly studies the semantic annotation algorithm in large scenes of unmanned vehicles. In the specific implementation process, the SemanticKITTI dataset collected from highway scenes and rural roads around Karlsruhe, Germany was selected. This dataset provides 23,201 complete 3D scan frames for training and 20,351 frames for testing, and the number of point clouds in each frame exceeds 10 5 , is the largest publicly available dataset. This dataset uses the automotive lidar Velodyne HDL64E to scan and generate a 360-degree field of view to provide a dense point cloud sequence, which is useful for verifying the point cloud semantic annotation network (network such as Figure 2 The performance of the 3D point cloud dataset is significantly better than that of the 3D point cloud dataset (as shown in Figure 2). The dataset is divided into 22 sequences, of which scenes 00-07 and 09-10 are training sets, scene 08 is a validation set, and scenes 11-20 are test sets. The 3D point cloud data includes XYZ three-dimensional coordinates and reflection intensity information, but does not contain RGB information, making the point cloud semantic annotation task for the dataset more challenging.

[0040] A point cloud is a series of unordered points. A point cloud containing N points can be represented by two parts: the inherent coordinates in the three-dimensional space obtained by the 3D sensor. Feature vectors containing semantic information obtained by convolutional networks (CNN) Where C represents the feature dimension. Although the coordinate information P lacks semantic information compared to the semantic feature vector F and cannot provide semantic support for the final classification task, the coordinate information is the basis for extracting semantic information, and as the number of network layers increases, the semantic features will lack point cloud geometric coordinate information due to mapping to high dimensions, resulting in a decrease in classification accuracy. Therefore, making full use of point cloud coordinate information and semantic features to extract point cloud feature maps with richer information is the key to improving semantic segmentation accuracy.

[0041] First, the point cloud coordinate information is passed through a single-layer multi-layer perceptron (MLP) to extract the semantic information, which includes a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function. The neighborhood information of each point is searched through the K-nearest neighbor method. Then, the three-dimensional coordinate information of the point cloud is encoded through spatial position to obtain the spatial position relationship code r between the center point and each neighboring point. The tensor size can be expressed as Where 10 represents the feature dimension described by the spatial position encoding.

[0042] In order to fully capture the fine-grained features of the point cloud, the position encoding information in the neighborhood is aggregated through attention pooling. The position encoding information r is passed through a fully connected layer to autonomously learn the useful potential feature mask, that is, the attention score. Then, the attention score is multiplied by the corresponding elements of the local neighborhood features to obtain the local representative features, thereby avoiding the loss of important geometric information in the neighborhood during the information aggregation process. Finally, the geometric structure code g is obtained, and its tensor size can be expressed as

[0043] At the same time, considering that in the coding layer, due to downsampling, the neighborhood point information of the sampling point will change accordingly (such as Figure 5 As shown in the figure, as the number of sampling layers increases, the receptive field of the corresponding center point continues to increase. In the deep layer, since its receptive field is much larger than the first layer, although it contains higher semantic level and richer feature information, it will inevitably lead to the loss of local details and geometric structures. Therefore, by transferring the spatial position encoding information of different levels between multiple layers, the neighborhood geometric information in the encoding layer is enriched, and the neighborhood geometric structure information under different receptive fields is included, which provides a spatial basis for mining the semantic correlation between adjacent points of discrete 3D point clouds.

[0044] At this time, the geometric information transmitted between multiple layers of geometric structure encoding information is cascaded with the current g and the semantic feature information f of the corresponding point in the feature dimension to obtain G to be input into the GSAGCN module.

[0045] Graph convolution first constructs the data into a graph structure with vertices and edges, and extracts features of node information, edge information or the entire graph structure based on the graph structure data. Since each node is affected by the state of its neighboring nodes, point clouds are more conducive to the extraction and expression of local neighborhood information during the graph convolution process.

[0046] The present invention is based on the neighborhood graph structure. Within the local neighborhood range, it realizes the intra-layer extraction and transmission of neighborhood information through edge convolution and attention pooling, and uses geometric structure information to fully explore the semantic correlation between adjacent points, and continuously enriches semantic feature information. Unlike two-dimensional raster images, three-dimensional point clouds in a specific scene are discrete and disordered, and there is no clear topological relationship between points. For adjacent points in Euclidean space, there is usually an interactive relationship. And for a certain point, the geometric structure formed by its several adjacent points is the basis for mining the semantic information of neighboring points in the point cloud. Based on this, the present invention uses KNN to obtain the indexes of the K points closest to each point, and establishes directed edges between the center point and the neighborhood points.

[0047] The transfer edge convolution layer in GSAGCN is: the cascaded feature information G is input into the geometric structure and semantic feature fusion graph convolution module (structure as shown in Figure 3 ), expand the neighborhood information according to the point cloud neighborhood index, establish the directed edge between the center point and the neighborhood point, so as to complete the construction of the neighborhood graph structure, and select the difference between the characteristics of the center node and each neighborhood node as the edge vector, which can be finally expressed as Among them G i represents the geometric structure code corresponding to each i-th point, Represents the geometric structure encoding of the jth point in the neighborhood corresponding to the i-th point, It represents the directed edge information between the ith point and the jth point in the corresponding neighborhood. Currently common feature extraction modules such as multi-layer perceptron (MLP) can flexibly represent high-dimensional feature vectors, and can fit complex nonlinear problems through a 1×1 convolution layer, a batch normalization layer, and activation functions such as ReLU or Sigmoid. However, due to the limited number of neurons it contains, it is difficult to mine the desired information from the huge point cloud data in a short time, resulting in unsatisfactory fitting effect. The present invention stacks three layers of continuous MLP to extract edge attribute features, thereby obtaining E. In order to aggregate the edge attribute features into the corresponding central node, the self-attention mechanism is also used here to autonomously assign attention weights to each edge vector feature, and finally the edge attribute features are weightedly fused according to the attention weights to obtain node features that are full of fine-grainedness and representative of the neighborhood. Finally, the aggregated information is passed through two layers of stacked MLP to extract the graph node information to obtain the feature Each MLP layer includes a 1×1 1D convolution layer, a batch processing layer, and an activation function ReLU.

[0048] In the field of computer vision, dilated convolution increases the scale of the convolution kernel to expand the receptive field without reducing the resolution. In this paper, we use stacked continuous transfer edge convolution layers to aggregate and transfer features between neighborhoods, and continuously expand the receptive field of the point cloud without changing the number of neighborhood points.

[0049] When the input feature G is convolved for the first time, the receptive field of each point is the number of the corresponding neighborhood range, which is K. When the second convolution is performed, although the number of neighbors of the center point remains unchanged, the points in its field aggregate the features of K points in their respective neighborhoods in the previous convolution layer, so that the receptive field of each point is K. 2 (like Figure 4 shown).

[0050] Theoretically, the more layers are stacked, the larger the receptive field of the point cloud should be, and the corresponding semantic information contained in each point should be richer. However, the continuous increase in the receptive field also means that there are more potential noise points, and the rate of expansion of the receptive field with the stacking of the edge convolutional layers reaches In addition, the computational efficiency and memory issues of semantic segmentation tasks in large scenes are also comprehensively considered. Finally, the present invention stacks two layers of transfer edge convolutional layers and uses residual connections to form a residual-stacked dilated convolutional network.

[0051] Regarding the choice of downsampling method, the farthest point sampling method can provide better coverage of the point cloud, but for 10 5 -10 6 For a point cloud of this series, the sampling time complexity of the farthest point is O(N 2 ), which takes about 20 seconds on a single GPU. Although random sampling can achieve high-speed sampling with a time complexity of O(1), it is easy to cause the loss of geometric structure of point clouds. Therefore, for the task of semantic annotation of point clouds in large scenes, the present invention designs a lightweight inverse density sampling method. The density of each point is characterized by the inverse of the distance between the center of gravity of the point cloud neighborhood and the center point. In addition, after downsampling, the present invention does not directly discard the unsampled points, because these points also contain rich geometric and semantic features. Let the point cloud before the i-th layer sampling be ε i , and the sampled point cloud is ε i+1 , firstly with ε i+1 The center point is ε i The local neighborhood is sought in the , and the neighborhood features are used to characterize the center point features. Finally, the maximum pooling is used to aggregate the neighborhood feature information, so as to fully mine and integrate the feature information of the unsampled points into the sampled points, thereby reducing the information loss caused by the point cloud in the downsampling process.

[0052] The model used in the present invention contains 4 downsampling layers, and the number of original point clouds input to the model is 45056, and its corresponding semantic feature dimension is 8. The number of points obtained after each downsampling is 11264, 2816, 704, and 176, respectively, and the corresponding semantic feature dimensions are 16, 64, 128, and 256, respectively.

[0053] Upsampling uses a relatively light nearest neighbor interpolation method, and uses a skip connection structure to fuse the rich semantic feature information in the high-level layer with the rich spatial structure information in the low-level layer. Upsampling also goes through 4 layers, and the number of point clouds corresponding to each layer is 704, 2816, 11264, and 45056, respectively, and the corresponding semantic feature dimensions are 256, 128, 32, and 32, respectively. Finally, the semantic feature information of the original point obtained by upsampling is then passed through multiple fully connected layers to obtain the score of each point in each category, thereby achieving the semantic segmentation task of the point cloud.

[0054] After the above network structure, the semantic features of the point cloud are first transformed by three-dimensional coordinate extraction, and then the geometric structure information contained in the position encoding under different receptive fields is fused with the semantic features by using the multi-level structure of downsampling. Therefore, the semantic feature vectors of point clouds in the same category should be more similar than those in different categories. Based on this, the contrast loss function is constructed using the feature vectors of the last layer of the network, and the contrast clustering and multi-task cross entropy dual-driven optimization model is established in coordination with the cross entropy loss function. In each training iteration, the points are stored in the corresponding list L according to the category. Among them, each item in the list represents the latest S feature vectors of the category. And after each specific number of iterations, the average feature vector of each category is calculated and updated according to the list content. Therefore, the feature vectors of each point in the last layer of the network are compared with the average feature vector. If the two belong to the same category, the Euclidean distance between the feature spaces of the two should be smaller, otherwise it should be larger. In order to make the Euclidean distance between feature vectors in the same category continue to decrease, and the Euclidean distance between different categories continue to increase.

[0055] For multi-task classification of point clouds, the cross entropy loss function is the most commonly used, and its specific formula is as follows:

[0056]

[0057] where y i represents the predicted value of the network model, Represents the true value of the corresponding point, and V represents the number of categories. The cross entropy function describes the difference between the predicted probability distribution and the true probability distribution. Finally, the cross entropy loss is continuously reduced through gradient back propagation, so that the predicted probability distribution is constantly close to the true probability distribution.

[0058] Cross entropy loss can only narrow the gap between the model prediction results and the actual categories, and cannot effectively utilize the semantic feature connections between points and categories. Points of the same category should be closer in the feature space because they have similar semantic information after the network encoding-decoding structure generates feature vectors with rich semantic information. On the contrary, points of different categories should be farther away.

[0059] Therefore, this problem can be abstracted as a contrast clustering problem. Based on the semantic information feature vector of the last layer of the network, the present invention measures the distance between feature vectors of each category, and constructs a contrast loss function by making the feature distances of points of the same category closer and the feature distances of points of different categories farther in the high-dimensional semantic feature space of the point cloud, that is, using inter-class representation as a driving force.

[0060] In the specific implementation, the present invention first defines a queue L for storing the feature vectors of the corresponding category points, and each category i corresponds to a feature vector β i , whose tensor size can be expressed as D represents the feature dimension of the intermediate layer of the point, and S means that only the latest S points will be stored in a single category. In each iteration, considering the large number of input point clouds, a fixed number of points will be randomly selected to update the queue, and then the feature vectors of each point will be concatenated into the corresponding category β according to the category, and then the last S points of the feature vector β will be intercepted to achieve the update. After each fixed I p After iterations, the average activation feature vector of each category is calculated and updated according to the queue L In order to prevent the center points of the feature vectors of each category from oscillating too quickly in the feature space, the average activation feature vector is updated. When the present invention further sets the momentum m, the result can be expressed as:

[0061]

[0062] The following contrast loss function is defined:

[0063]

[0064] where μ v represents the high-dimensional semantic feature vector corresponding to category v extracted by the last layer of the network, δ i It represents the continuously updated average feature vector in category i, and the l(·,·) function can be expressed as:

[0065]

[0066] D(·) Here, Euclidean distance is selected as the distance measurement standard, and Δ represents the similarity between feature vectors.

[0067] The final loss function can be expressed as: L total =λ×L cont +L cro , here we set λ to 3.0.

[0068] The results predicted by the network model designed by the present invention are very close to their true values, and the network has a high classification accuracy for categories such as roads, buildings, vegetation, and cars. Although small categories of objects such as pedestrians and bicycles are distributed more discretely and the point cloud structure is relatively sparse, due to the geometric structure encoding multi-layer transfer module and the residual stacked graph convolution, the network can still mine relatively complete local neighborhood structure information for relatively sparse and discrete point clouds through the fusion of feature information under different sizes of receptive fields. In summary, the point cloud semantic annotation algorithm proposed by the present invention is suitable for the semantic understanding task of outdoor scenes of unmanned vehicle-mounted laser radar.

[0069] The present invention uses the average intersection-over-union ratio and overall accuracy to evaluate the performance of the network model on the SemanticKITTI dataset. The average intersection-over-union ratio calculation formula is as follows:

[0070]

[0071] Among them, p ij It means predicting category i as category j, p ji It means predicting the j category as the i category, p ii Indicates that category i is predicted as category i, and k represents the number of categories. It has been verified that the average intersection-over-union ratio of the model reached 59.2%, and the overall accuracy reached 91.4%, surpassing the traditional PointNet (14.6%), PointNet++ (20.1%), and the existing point cloud semantic annotation networks for large scenes Point ASNL (46.8%), RandLA-Net (53.9%), PolarNet (54.3%), where the brackets represent the average intersection-over-union ratio. This proves the effectiveness of the graph convolutional neural network designed by the present invention and a collaboratively driven optimization strategy of inter-class representation comparison and cross-entropy loss entropy.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic annotation method for graph convolutional point clouds driven by inter-class representation comparison, characterized in that: include: S1. Use geometric structure coding to describe the point cloud neighborhood structure; S2. The geometric structure encoding information in each encoding layer is transferred between multiple layers, and the geometric structure information under different receptive fields provides a spatial basis for mining the semantic relationship of neighboring points; S3. Combining structural information with semantic features, the graph convolution module is based on the neighborhood graph structure. In the local neighborhood range, the intra-layer extraction and transmission of neighborhood information is realized through edge convolution and attention pooling. The geometric structure information is used to fully explore the semantic correlation between adjacent points and continuously enrich the semantic feature information. S4. Using inter-class representation contrast as a driver, the contrast loss function is used to aggregate high-dimensional semantic features of the same category in the feature space and to keep semantic features of points of different categories away from each other. The inter-class representation contrast loss and cross entropy loss function are used to collaboratively drive the optimization strategy to train the classification results using gradient back propagation. S5. Construct a point cloud semantic annotation network model, and use the point cloud semantic annotation network model to conduct semantic annotation test experiments on 3D point cloud data; S1 includes: defining a tensor P = [p1, p2, ..., p i ] T To represent the point cloud in space, where p i Represents the i-th point, establishes a local neighborhood for each center point, constructs a K-nearest neighbor algorithm to search for the K points closest to each center point, uses Euclidean distance as the distance metric, and obtains the local neighborhood of the i-th point [p1, p2, …, p n ], spatial position encoding r i for: in represents the K neighboring points corresponding to each center point, ‖‖ represents the Euclidean distance between the center point and its neighboring points, […] represents the cascade operation on the feature dimension, and the geometric structure information r of the point cloud is obtained. The size of the tensor is expressed as in Represents a tensor of a specific shape and size, N represents the number of point clouds, 10 represents the feature dimension described by the spatial position encoding, and the features described are within the neighborhood. The point cloud feature information is aggregated within the neighborhood through the attention pooling operation to obtain the geometric structure encoding g corresponding to each point. The size of the tensor is expressed as Assume g m The geometric structure encoding information is obtained in the mth layer. After the geometric structure encoding of the previous layer is transmitted, the geometric structure encoding g containing point clouds with different receptive fields is obtained. m′ :g m′ =[g 1 ,DS(g m-1 ),…,DS(g m )], where DS represents the downsampling operation; the semantic feature of the point cloud is represented by F = [f1,f2,…,f n ], the mth layer g m′ After being cascaded with the semantic feature F in the feature dimension, the structural information and the semantic feature fusion graph convolution module are input, and the result is expressed as: G = [g m′ ,F]; The graph convolution module for fusing structural information and semantic features includes a transfer edge convolution layer, an attention pooling layer, and a residual stacked graph convolution; The process of information transfer by the transfer edge convolution layer includes: S2.

1. Build a graph model, use KNN to obtain the indexes of the K points closest to each point, and establish directed edges between the center point and the neighborhood points; S2.

2. Obtain edge vectors. When establishing directed edges, use feature vectors rich in semantic information and geometric coding information with neighborhood structure. The edge vector is expressed as: Among them G i represents the geometric structure encoding corresponding to the i-th point, Represents the geometric structure encoding of the jth point in the neighborhood corresponding to the i-th point, It represents the directed edge information between the i-th point and the j-th point in the corresponding neighborhood; S2.

3. Extract edge features, extract edge attribute features by passing the edge vector through three layers of continuously stacked multi-layer perceptrons MLP; S2.

4. Aggregate edge attribute features, use the self-attention mechanism to autonomously learn important neighborhood features, and characterize the neighborhood information contained in it to the greatest extent. The final result is expressed as: F i =h Θ (AttPool(E i ), where AttPool represents the attention pooling operation, E i represents the edge attribute features corresponding to the i-th point and its neighboring points, F i Represents the feature vector with semantic information of the i-th point after edge feature aggregation; the attention pooling layer includes: Given the neighborhood edge attribute feature E i , the attention score with normalized characteristics in the neighborhood dimension is calculated by the following formula: score = δ(w,E i ), where w represents a set of weights shared by a group of weights, δ represents the softmax activation function used to normalize the attention score weights and a single-layer MLP; The learned attention score is regarded as a mask for automatically selecting important latent features. The attention score is multiplied element by element with the corresponding center point edge attribute feature to generate a set of weighted features. The weighted features are summed to obtain the locally representative center point feature. By stacking continuous transfer edge convolution layers, the point cloud receptive field is continuously expanded without changing the number of neighborhood points through feature aggregation and transfer between neighborhoods; residual connections are used to form residual stacked graph convolutions; the point cloud semantic annotation network model contains 4 downsampling layers. The number of original point clouds input to the model is 45056, and the corresponding semantic feature dimension is 8. After each downsampling, the number of points obtained is 11264, 2816, 704, and 176, respectively, and the corresponding semantic feature dimensions are 16, 64, 128, and 256, respectively; During the downsampling process, a part is randomly selected directly from the original point cloud, and the point cloud before the i-th layer sampling is set as ε i , the sampled point cloud is ε i+1 , firstly with ε i+1 The center point is ε i Find the local neighborhood in The features of the center point are characterized by neighborhood features, and the maximum pooling is used to aggregate the neighborhood feature information. The feature information of the unsampled points is fully mined and integrated into the sampled points, reducing the information loss caused by the downsampling process of the point cloud. The upsampling uses the nearest neighbor interpolation method, and the skip connection structure is used to fuse the rich semantic feature information in the high-level layer with the rich spatial structure information in the low-level layer. The upsampling also goes through 4 layers, and the number of point clouds corresponding to each layer is 704, 2816, 11264, and 45056 respectively, and the corresponding semantic feature dimensions are 256, 128, 32, and 32 respectively. The semantic feature information of the original point obtained by upsampling is then passed through multiple fully connected layers to obtain the score of each point in each category; the contrast loss function is: where μ v represents the high-dimensional semantic feature vector corresponding to a specific category v extracted by the last layer of the network, δ i It represents the continuously updated average feature vector in the i-th category. D(·) Here, Euclidean distance is used as the distance measurement standard, and Δ represents the similarity between feature vectors. The final loss function is: L total =λ×L con +L cro ,λ is 3.0, where L cro represents the cross entropy loss function.

Images