Point cloud segmentation method and system, medium, equipment and information data processing terminal

Through the combination of dual-branch encoder and cascade attention decoder, the multi-paradigm representation of point clouds is extracted and multi-stage feature fusion is carried out, which solves the problem of failing to fully utilize local and global features in the existing technology, and significantly improves the point cloud The accuracy and robustness of the segmentation.

CN118967698APending Publication Date: 2024-11-15TAIYUAN UNIVERSITY OF TECHNOLOGY

Patent Information

Application Number
CN202410901451.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing technology fails to fully utilize local and global features in point cloud segmentation tasks, resulting in a decrease in model generalization ability and robustness, and the decoding stage cannot effectively aggregate multi-stage information, limiting the accuracy and details of segmentation results.

Method used

Using a dual-branch encoder and a cascade attention decoder, the multi-paradigm representation of point clouds is extracted through graph convolution and Transformer, combining efficient edge convolution and point Transformer feature extraction, multi-scale cross attention fusion is performed, and cascade attention decoding is designed. The device performs upsampling and fusion of multi-stage features.

Benefits of technology

It improves the accuracy of point cloud segmentation, can process large-scale point cloud data more effectively, enhances the generalization ability and robustness of the model, and significantly improves the accuracy and detail retention of segmentation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967698A_ABST
    Figure CN118967698A_ABST
Patent Text Reader

Abstract

The invention provides a point cloud segmentation method and system, a medium, equipment and an information data processing terminal, and belongs to the technical field of point cloud data processing. The problem that local and global features are difficult to effectively extract and fuse in a complex point cloud scene is solved; comprising the following steps that a double-branch encoder and a cascade attention decoder are constructed, the double-branch encoder is composed of graph convolution and Transform and used for obtaining multi-normal-form representation of the point cloud and extracting local and global information of the point cloud, the cascade attention decoder can learn multi-scale and multi-resolution spatial representation, and the multi-normal-form representation of the point cloud is obtained; local and global multi-stage feature representation of the point cloud is utilized; building an efficient point cloud segmentation network based on multi-normal form representation and cascade attention, and training a network model by using a negative logarithm likelihood loss function with a weight; predicting the point cloud in the test set by using the trained network model to obtain a segmentation result of each point in the point cloud; the method is applied to point cloud segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention provides a point cloud segmentation method, system, medium, equipment and information data processing terminal, belonging to the technical field of point cloud data processing. Background Art

[0002] In recent years, the rapid development of a large number of cheap and novel 3D scanning devices such as LiDAR, 3D scanners and RGB-D cameras (such as Intel RealScene, Apple's depth camera and Microsoft Kinect, etc.) has made it easier to obtain 3D point cloud data. 3D point clouds have powerful functions in unfolding three-dimensional objects, and they can ultimately retain the original features of objects, such as shape, size, spatial position, geometric features and surface properties. Efficient processing of 3D point clouds is crucial to advancing many fields, including autonomous driving, aircraft blind spot assembly, etc. How to obtain useful information from a large amount of 3D point cloud data and solve problems in scene understanding is an important part of current computer vision research. Object detection, classification and recognition based on 3D point cloud data are the main technologies for solving scene understanding, and 3D point cloud segmentation is the basis of these technologies.

[0003] Although convolution of regular grid image pixels can effectively learn to express features, it is not directly applicable to 3D point clouds because 3D point clouds are disordered and do not conform to regular grids. After that, researchers have widely explored converting disordered point clouds into regular data formats and performing traditional convolution processing. Different types of methods have been proposed. Multi-view based methods project point clouds from different angles into images and use traditional visual convolutional neural networks to learn the features of the projected images. Voxel-based methods convert point clouds into regular 3D voxels and then perform 3D convolution on the voxels to learn their features. As a pioneer in directly processing unordered point clouds, PointNet uses multi-layer perceptrons and symmetric functions to extract features of point clouds. An important problem of PointNet is that local features cannot be extracted using multi-layer perceptrons. Later, PointNet++ solved the shortcomings of PointNet by performing convolution operations on grouped neighboring points to learn local features. In graph-based methods, a connectivity graph is first constructed on grouped neighboring points. Graph convolution is performed on the local connectivity graph to extract local features, and finally the point cloud is downsampled by graph pooling. PointNet++ and graph-based methods build hierarchical networks by imitating the structure of visual CNNs to extract local features of point clouds layer by layer. Therefore, these methods inherit the shortcomings of visual convolutional neural networks that they cannot effectively extract long-distance global features. The Transformer architecture was originally developed in natural language processing. In the Transformer, the self-attention operation first transforms any pair of input features into query, key, and value spaces. Then, the weight on the aggregated feature value space is calculated based on the similarity of the two transformed features in the query space and the key space. The above operations are performed between any pair of input features regardless of their position distance in the Euclidean space, so that the Transformer structure can extract long-distance global features. In addition, the Transformer architecture processes input data in a parallel and order-independent manner, so they are suitable for processing unordered data. In 3D point cloud research, many deep learning-based methods use Transformers as feature extractors. For example, the point cloud Transformer adopts a transformer architecture based on offset attention to learn global features, and the point cloud Transformer network proposes a vector attention-based operation to capture global features from downsampled point clouds. Although the Transformer architecture can help point clouds extract global features over long distances, it ignores the importance of local features for point cloud segmentation boundary details.

[0004] Point clouds are a set of three-dimensional data consisting of a large number of discrete points. For point cloud scene segmentation tasks, point cloud datasets are very large, so downsampling in the encoder is necessary for processing and analysis. Downsampling can reduce the amount of data, reduce computational complexity, and in some cases increase processing speed. Existing point cloud segmentation networks use the above-mentioned method of directly processing point clouds as encoders, and restore the number of point clouds by upsampling in the decoder. The decoder structure only uses simple upsampling operations to restore the spatial information of the point cloud. Then, features of the same level are fused through jump connection operations. This simple feature mapping method may lead to information loss and spatial blur, especially in high-frequency details and boundaries of complex point cloud scenes.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] (1) For point cloud segmentation tasks, the local and global characteristics of point clouds are not fully utilized in the encoder stage. The simultaneous extraction and fusion of local and global features is the key to point cloud data processing. Although previous studies (such as PointASNL, Pnp-3d, etc.) have successfully combined local features with global features to obtain more expressive features, simply concatenating local features with global features fails to fully utilize local and global information, resulting in a decrease in the generalization ability and robustness of the model.

[0007] (2) For point cloud segmentation tasks, multi-stage information cannot be well aggregated in the decoding stage. The limited expressive power of the decoder makes it unable to fully utilize the fine-grained features in the point cloud, thus limiting the accuracy and detail retention of the segmentation results.

[0008] The difficulty in solving the above problems and defects lies in: how to effectively extract and fuse local and global features in the point cloud and how to design a decoder to process complex point cloud scenes.

[0009] The significance of solving the above problems and defects is: it can improve the accuracy of point cloud segmentation, enable point cloud segmentation to be applied to actual large-scale 3D scenes, and accelerate the development of fields such as autonomous driving, blind spot assembly and augmented reality. Summary of the invention

[0010] In order to overcome the deficiencies in the prior art, the present invention aims to solve the following technical problems: to provide an improvement in a point cloud segmentation method, system, medium, equipment and information data processing terminal.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is: a point cloud segmentation method, comprising the following steps:

[0012] S101: Construct a dual-branch encoder and a cascaded attention decoder. The dual-branch encoder consists of graph convolution and Transformer to obtain a multi-paradigm representation of the point cloud and extract local and global information of the point cloud. The cascaded attention decoder can learn multi-scale and multi-resolution spatial representations and utilizes multi-stage feature representations of the local and global point clouds.

[0013] S102: Build an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention, use a weighted negative log-likelihood loss function, and use the training set to perform supervised training on the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention. Adjust the network parameters according to the loss value in each round of training to obtain a trained network model.

[0014] S103: Use the trained network model to predict the point cloud in the test set to obtain the segmentation result of each point in the point cloud.

[0015] The dual-branch encoder includes n cascaded dual-branch feature extraction modules, each of which includes a cascaded efficient edge convolution feature extraction layer, a point Transformer feature extraction layer and a multi-scale cross-attention fusion layer.

[0016] The construction steps of the dual-branch encoder are as follows:

[0017] 1) Construct an efficient edge convolution feature extraction layer:

[0018] The K-nearest neighbor algorithm is used to construct the point cloud data into a local graph structure. The vertices are found through the farthest point sampling method, and then the neighbor nodes are selected according to the calculated Euclidean distance between the point pairs to form a directed graph structure. After the graph structure is completed, the edge convolution is used to learn the edge features between the point pairs to extract more accurate local feature information. x ;

[0019] Stacking multiple efficient edge convolution feature extraction layers to form graph convolution;

[0020] 2) Constructing the point Transformer feature extraction layer:

[0021] The input coordinates are passed through a multi-layer perceptron to learn a dimensionless embedding feature Fe, and then the embedded feature Fe is fed into four stacked attention modules to learn the semantically rich and discriminative representation of each point, and then the global feature information F is generated through a linear layer. Y ;

[0022] Stack multiple point Transformer feature extraction layers to form a Transformer branch;

[0023] 3) Construct a multi-scale cross-attention fusion layer:

[0024] First, the local feature information F x and global feature information F Y Projection, get X q , X k , X v and Y q , Y k , Y v , and then through the cross self-attention operation, the graph convolution branch and the Transformer branch guide each other in the learning process to obtain X o and Y o , where q, k, v, and o represent query, key, value, and output respectively. o and Y o Addition realizes efficient fusion of information from different branches;

[0025] 4) The efficient edge convolution feature extraction layer, point Transformer feature extraction layer and multi-scale cross-attention fusion layer that are run in parallel are combined into a dual-branch feature extraction module, and n dual-branch feature extraction modules are superimposed in sequence to obtain local feature maps, global feature maps and local-global fused feature maps at different levels.

[0026] The edge convolution function is used to learn the edge features between point pairs. The expression of the edge convolution function is as follows:

[0027] e ij =h θ (P i , P j -P i );

[0028] di = Maxpool{ReLU(e ij )};

[0029] Among them, e ij is the feature of the connecting edge between nodes, d i To obtain the new features, we will use P i The local graph structure of the central node is taken as input, P j is a neighbor node; (P j -P i ) is the vector difference between the neighbor node and the central node, and the association information between the point pairs is obtained; h θ (·) is a nonlinear function with a set of learnable parameters θ, which is used to extract features from the input graph structure and learn θ using a shared multi-layer perceptron; ReLu is a nonlinear activation function;

[0030] Because it is not affected by the order of neighbor nodes, the maximum pooling function is used to aggregate the main features in the edge feature vector to obtain the final local feature information Fx.

[0031] The cascaded attention decoder includes n upsampling modules cascaded in sequence, each upsampling module includes a feature interpolation layer and a feature attention fusion layer cascaded in sequence, and the construction steps are as follows:

[0032] 1) Construct an upsampling module to interpolate the local and global feature maps Fgn obtained at different levels to the same level as Fgn through upsampling. gn-1 The fusion feature map F of the same scale un-1 ∈R (N / 4^n-1)×2^(n-1)d , where R is a real number set, N is the number of initial point clouds, n is the number of levels, and d is the number of channels. gn-1 and F un-1 Perform feature attention fusion, the fusion method is as follows:

[0033] q att (g i-1 ,u i-1 )=σ 1 (BN(C g (g i-1 ))+BN(C u (u i-1 )));

[0034] AG(g i-1 ,u i-1 )=C g (g i-1 )*σ 2 (BN(C(q att (g i-1 ,u i-1 ))));

[0035] F′ i-1 =MLP(concat(u i-1 ,AG(g i-1 ,u i-1 )));

[0036] Among them, σ 1 (·) and σ 2 (·) correspond to ReLU and Sigmoid activation functions, respectively, C g (·), C u (·) and C(·) represent channel convolution operations 1×1, BN(·) is batch normalization operation, g and u are skip connection features and upsampling operations, respectively, qaa t (·) indicates the attention weight calculation, AG(·) indicates the fusion of F gi-1 and F ui-1 The characteristic of F′ i-1 It represents the cascade of fusion features of different levels and scales;

[0037] Then, parallel channel and spatial attention are used to enhance the spatial features and channel features of the point cloud respectively. Then, these features are reduced in dimension through convolution to obtain the feature map F i-1 , the above process is defined as follows:

[0038] W c =σ(MLP(MaxPool(F' i-1 ))+MLP(AvgPool(F' i-1 )));

[0039]

[0040]

[0041] Among them, MaxPool and AvgPool represent Max pooling and Average Pooling respectively, σ represents Sigmoid function, ⊙ represents Hadamard product operation, W c represents the channel attention weight, Represents the spatial position attention weight;

[0042] 2) Superimpose n upsampling modules in sequence, perform feature interpolation and fusion, obtain upsampled multi-level feature information, and finally obtain a feature map that integrates multi-stage features.

[0043] The loss function in step S102 is based on the cross entropy loss. The loss is calculated by multiplying the log probability of the predicted value by the log probability of the target category and taking the negative of the multiplication. The calculation formula of the loss function is as follows:

[0044]

[0045] Among them, total_loss is the calculated total loss, N is the number of samples, and w i is the weight used to adjust the loss contribution of each sample, p i is the probability vector of the i-th sample in the predicted value, which contains the predicted probability of each category, target i is the category index of the i-th sample in the target label.

[0046] A point cloud segmentation system, comprising:

[0047] A feature extraction construction module, used to construct a dual-branch encoder and a cascaded attention decoder, wherein the dual-branch encoder includes n cascaded dual-branch feature extraction modules, each of which includes a cascaded efficient edge convolution feature extraction layer, a point Transformer feature extraction layer, and a multi-scale cross-attention fusion layer; the cascaded attention decoder includes n cascaded upsampling modules containing feature attention; each upsampling module includes a feature interpolation layer and a feature attention fusion layer cascaded in sequence;

[0048] The network model acquisition module is used to build an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention. The weighted negative log-likelihood loss function is used to perform supervised training on the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention using the training set. In each round of training, the network parameters are adjusted according to the loss value to obtain the network model.

[0049] The segmentation result output module is used to use the trained network model to predict the point cloud in the test set and obtain the segmentation result of each point in the point cloud.

[0050] A computer-readable storage medium stores a computer program, wherein the computer program can be used to implement the point cloud segmentation method when loaded and executed by a processor.

[0051] A collection device, comprising: a 3D scanner, an RGB-D camera and a processor, wherein the processor is used to execute the point cloud segmentation method.

[0052] An information data processing terminal is used to implement the point cloud segmentation method.

[0053] The beneficial effects of the present invention compared with the prior art are as follows: the present invention provides an efficient point cloud segmentation network based on multi-paradigm representation and cascaded attention, uses a dual-branch encoder (composed of graph convolution and transformer) to obtain a multi-paradigm representation of the point cloud, and extracts local and global information of the point cloud. The layering enables the network to process a large number of point cloud data. A multi-scale cross-attention fusion module is introduced into the dual-branch encoder, so that the graph convolution branch and the transformer branch guide each other in the learning process, and efficient fusion of information from different branches is achieved; and the present invention proposes a new attention cascade 3D point cloud decoder, which utilizes the local and global multi-stage feature representation of the point cloud, and simultaneously learns multi-scale and multi-resolution spatial representation, thereby improving the accuracy of point cloud segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be further described below in conjunction with the accompanying drawings:

[0055] Figure 1 is a flow chart of a point cloud segmentation method provided by an embodiment of the present invention;

[0056] Figure 2 is a schematic diagram of the structure of a point cloud segmentation system provided by an embodiment of the present invention;

[0057] Figure 3 An efficient point cloud segmentation network based on multi-paradigm representation and cascade attention provided by an embodiment of the present invention;

[0058] Figure 4 is a point cloud segmentation network segmentation result diagram provided by an embodiment of the present invention;

[0059] In the figure: 1. Feature extraction construction module; 2. Network model acquisition module; 3. Segmentation result output module. DETAILED DESCRIPTION

[0060] like Figures 1 to 4 As shown, the present invention provides a point cloud segmentation method, comprising the following steps:

[0061] S101: Construct a dual-branch encoder and a cascaded attention decoder, wherein the dual-branch encoder consists of graph convolution and Transformer, which is used to obtain a multi-paradigm representation of the point cloud and extract local and global information of the point cloud; the dual-branch encoder includes n cascaded dual-branch feature extraction modules, each of which includes an efficient edge convolution feature extraction layer, a point Transformer feature extraction layer and a multi-scale cross-attention fusion layer.

[0062] like Figure 3 As shown, the specific construction process of the dual-branch encoder is as follows.

[0063] 1) Construct an efficient edge convolution feature extraction layer:

[0064] The K-nearest neighbor algorithm is used to construct the point cloud data into a local graph structure. The vertices are found through the farthest point sampling method, and then the neighbor nodes are selected according to the calculated Euclidean distance between the point pairs to form a directed graph structure. The constructed graph structure is G = (V, E), where: V = {P i |i=1,2,…M},E={e i =(e i1 , e i2 ,…,e ik )|i=1,2…,M};

[0065] In the above formula: V is the vertex set, P i is the i-th central node, E is the set of edges between the central node and its neighboring nodes, e i is a set of directed edges with i central nodes, e ik For every i central nodes there are k directed edges, M is the number of vertices, and each vertex has K neighbors.

[0066] After completing the graph structure construction, edge convolution is used to learn the edge features between point pairs and extract more accurate local features. The edge convolution function is:

[0067] e ij =h θ (P i ,P j -P i );

[0068] d i =Maxpool{ReLu(e ij )};

[0069] In the above formula: ei j is the feature of the edge between nodes, P j is the neighbor node, and di is the new feature obtained.

[0070] Among them, P i The local graph structure of the central node is used as input; (P j -P i ) is the vector difference between the neighbor node and the central node, and the association information between the point pairs is obtained; h θ (·) is a nonlinear function with a set of learnable parameters θ, which is used to extract features from the input graph structure, and θ is learned using a shared multi-layer perceptron; ReLu is a nonlinear activation function. Because it is not affected by the order of neighbor nodes, the maximum pooling function is used to aggregate the main features in the edge feature vector to obtain the final local feature information Fx.

[0071] Multiple efficient edge convolution feature extraction layers are stacked to obtain the graph convolution branch.

[0072] 2) Constructing the point Transformer feature extraction layer:

[0073] The input coordinates are embedded into a new feature space (a dimensionless embedding feature Fe is learned through a multi-layer perceptron), and the embedded feature Fe is then fed into four stacked attention modules to learn semantically rich and discriminative representations of each point, and then the output features are generated through a linear layer. The global feature encoding is as follows:

[0074] F 1 =AT 1 (F e );

[0075] F i =AT i (F i-1 ),i=2,3,4;

[0076] F Y =concat(F1 ,F 2 ,F 3 ,F 4 )·W o ;

[0077] In the above formula: F 1 For AT 1 is the first attention module, F i is the feature obtained by self-attention at the i-th layer, AT i represents the i-th attention module. The output dimension of each attention module is the same as its input dimension. o represents the weight of the linear layer, concat represents the connection operation, and F Y Represents the obtained global feature information.

[0078] Stack multiple point Transformer feature extraction layers to get the Transformer branch.

[0079] 3) Construct a multi-scale cross-attention fusion layer

[0080] First, the local feature information F x and global feature information F Y Projection, get X q , X k , X v and Y q , Y k , Y v , and then through the cross self-attention operation, the graph convolution branch and the Transformer branch guide each other in the learning process to obtain X o and Y o , where q, k, v, and o represent query, key, value, and output respectively. o and Y o The addition realizes the efficient fusion of different branch information to obtain g. This process is defined as follows:

[0081]

[0082]

[0083] g=X o +Y o ;

[0084] In the above formula: ⊙ represents element multiplication operation, Represents a matrix multiplication operation, and Sum means adding each row of the matrix multiplication result to the first row.

[0085] 4) Combine the parallel efficient edge convolution feature extraction layer, point Transformer feature extraction layer and multi-scale cross attention fusion layer into a dual-branch feature extraction module, and stack n dual-branch feature extraction modules in sequence to obtain local feature maps of different levels {F X0 ∈R N×d ,F X1 ∈R (N / 4)×2d ,...,F Xn ∈R (N / 4^n)×2^nd}、Global feature map{F Y0 ∈R N×d ,F Y1 ∈ R(N / 4)×2d ,...,F Yn ∈R (N / 4^n)×2^nd} and local-global fusion feature map {F g0 ∈R N×d ,F g1 ∈ R(N / 4)×2d ,...,F gn ∈R (N / 4^n)×2^nd}, where R is a real number set, N is the number of initial point clouds, and d is the number of channels.

[0086] The cascaded attention decoder constructed by the present invention can learn multi-scale and multi-resolution spatial representations while utilizing local and global multi-stage feature representations of point clouds. The cascaded attention decoder includes n upsampling modules cascaded in sequence, and each upsampling module includes a feature interpolation layer and a feature attention fusion layer cascaded in sequence.

[0087] like Figure 3 As shown in Figure 2, the specific process of constructing the cascaded attention decoder is as follows.

[0088] 1) Construct an upsampling module to fuse the local and global feature maps F at different levels gn By upsampling and interpolating to F gn-1 The fusion feature map F of the same scale un-1 ∈R (N / 4^n-1)×2^(n-1)d , F gn-1 and F un-1 Perform feature attention fusion, the fusion method is as follows:

[0089] q att (g i-1 ,u i-1 )=σ 1 (BN(C g (g i-1 ))+BN(C u (u i-1 )));

[0090] AG(g i-1 ,u i-1 )=Cg (g i-1 )*σ 2 (BN(C(q att (g i-1 ,u i-1 ))));

[0091] F′ i-1 =MLP(concat(u i-1 ,AG(g i-1 ,u i-1 )));

[0092] Among them, σ 1 (·) and σ 2 (·) correspond to ReLU and Sigmoid activation functions respectively. C g (·), C u (·) and C(·) represent channel convolution operations 1×1. BN(·) is a batch normalization operation. g and u are skip connection features and upsampling, respectively. qaa t (·) indicates the attention weight calculation, AG(·) indicates the fusion of F gi-1 and F ui-1 The characteristic of F′ i-1 It represents the cascade of fusion features of different levels and scales.

[0093] Then, parallel channel and spatial attention are used to enhance the spatial features and channel features of the point cloud respectively. Then, these features are reduced in dimension through convolution to obtain the feature map F i-1 The fused features retain important spatial and channel information, which helps the decoder to better recover the information. This process is defined as follows:

[0094] W c =σ(MLP(MaxPool(F' i-1 ))+MLP(AvgPool(F' i-1 )));

[0095]

[0096]

[0097] Among them, MaxPool and AvgPool represent Max pooling and Average Pooling respectively, σ represents Sigmoid function, and ⊙ represents Hadamard product operation. c represents the channel attention weight, Represents the spatial position attention weight.

[0098] 2) Superimpose n upsampling modules in sequence, perform feature interpolation and fusion, and the upsampled multi-level feature information is {F n-1 ∈ N / 4^n×D ,...,F 1 ∈ (N / 4)×D ,F 0 ∈ N×D}, and finally obtain the feature map F that integrates multi-stage features 0 ∈ N×D , D represents the number of channels.

[0099] S102: Build an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention, use a weighted negative log-likelihood loss function, and use the training set to perform supervised training on the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention. In each round of training, adjust the network parameters according to the loss value to obtain a trained network model.

[0100] like Figure 3 As shown in the figure, building an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention and training it specifically includes:

[0101] 1) Build a dual-branch feature extraction module, which cascades 5 efficient edge convolution feature extraction layers, point Transformer feature extraction layers and multi-scale cross-attention fusion layers. The local feature maps at different levels are {F X0 ∈R N×64 ,F X1 ∈R (N / 4)×128 ,F X2 ∈R (N / 16)×256 ,F X3 ∈R (N / 64)×512 ,F X4 ∈R (N / 256)×1024}, the global feature map is {F Y0 ∈R N×64 ,F Y1 ∈R (N / 4)×128 ,F Y2 ∈R (N / 16)×256 ,F Y3 ∈R (N / 64)×512 ,F Y4 ∈R (N / 256)×1024}, local-global feature map {F g0 ∈R N×64 ,F g1 ∈R (N / 4)×128 ,F g2 ∈R (N / 16)×256 ,F g3 ∈R (N / 64)×512 ,F g4 ∈R (N / 256)×1024}, feature interpolation and fusion are performed through cascaded attention decoders to obtain a feature map F that combines multi-level features 0 ∈R N×256 , and then the dimension is reduced through MLP{160,64,c}, c is the number of sample labels, and finally the size N×c is obtained through the softmax layer as the final output result.

[0102] 2) First, the original point cloud of the training set is grid-downsampled to obtain an N×6 point cloud with the same number of points as the network input; an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention is built, and a negative log-likelihood loss function with weights is used. The training set is used to perform supervised training on the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention. The network parameters are adjusted according to the loss value in each round of training to obtain the network model. This loss function is based on the cross-entropy loss, which calculates the loss by multiplying the log probability of the predicted value by the log probability of the target category and taking the negative of the log probability. By using the weight w i , samples of different categories can be weighted. The formula is as follows:

[0103]

[0104] Where total_loss is the calculated total loss. N is the number of samples. i is a weight used to adjust the loss contribution of each sample. i is the probability vector of the i-th sample in the predicted value, which contains the predicted probability of each category, target i is the category index of the i-th sample in the target label.

[0105] S103: Use the trained network model to predict the point cloud in the test set to obtain the segmentation result of each point in the point cloud.

[0106] Another object of the present invention is to provide a point cloud segmentation system for implementing the point cloud segmentation method. Figure 2 As shown, the point cloud segmentation system includes:

[0107] Feature extraction building module 1, used to build a dual-branch encoder and a cascaded attention decoder, mainly including a dual-branch feature extraction module and a feature attention upsampling module;

[0108] Network model acquisition module 2 is used to build an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention. A weighted negative log-likelihood loss function is used to perform supervised training on the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention using a training set. In each round of training, the network parameters are adjusted according to the loss value to obtain the network model.

[0109] The segmentation result output module 3 is used to use the trained network model to predict the point cloud in the test set and obtain the segmentation result of each point in the point cloud.

[0110] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps: constructing a dual-branch encoder and a cascaded attention decoder; building an efficient point cloud segmentation network based on multi-paradigm representation and cascaded attention, using a weighted negative log-likelihood loss function, and using a training set to perform supervised training on the point cloud segmentation network obtained by multi-paradigm representation and cascaded attention, adjusting network parameters according to the loss value in each round of training to obtain a network model; using the trained network model to predict the point cloud in the test set to obtain the segmentation result of each point in the point cloud.

[0111] Another object of the present invention is to provide an acquisition device, which includes: a 3D scanner, an RGB-D camera and a computer device, and the computer device is used to execute the point cloud segmentation method.

[0112] Another object of the present invention is to provide an information data processing terminal, which is used to implement the point cloud segmentation method.

[0113] The technical effects of the present invention are described in detail below in conjunction with simulation.

[0114] 1. Simulation conditions

[0115] The present invention completes the simulation experiment of the present invention using Pycharm software with a GPU of Ubuntu 18.04 operating system, a graphics card Nvidia GTX 3090TI (24GB), a Pytorch 1.12.0 deep learning framework, CUDA version 10.0, and an S3DIS data set.

[0116] The S3DIS dataset is the Stanford Large-Scale 3D Indoor Spaces dataset. The S3DIS dataset takes each small room as a scene, with a total of 272 rooms, consisting of 6 indoor areas. Area 5 is used as the test set, and the other 5 areas are used as the training set. There are 204 training sets and 68 test sets, including 13 categories, and the number of point clouds for each sample is 40,960.

[0117] 2. Simulation experiment content and analysis

[0118] Divide the training set and the test set, build an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention, use the training set to train the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention, use the trained network model to predict the point cloud in the test set, and obtain the segmentation result of each point cloud. Compare the segmentation effect of the present invention with the existing PointNet++ and DGCNN methods, use the segmentation accuracy as the evaluation index, and the results are shown in Table 1:

[0119] Evaluation indicators PointNet++ DGCNN The present invention Segmentation accuracy 83.1 84.1 87.3

[0120] Table 1 Segmentation accuracy comparison of S3DIS dataset.

[0121] It can be seen from Table 1 that the segmentation effect of the present invention on the S3DIS dataset is better than that of the prior art PointNet++ and DGCNN.

[0122] Visualize the point segmentation results, such as Figure 4 As shown, it can be seen that the network model designed by the present invention can well segment objects such as tables, chairs, bookcases and walls in indoor scenes. In summary, it can be explained that the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention of the present invention can effectively capture the local-global information of point clouds and effectively improve the segmentation accuracy of point clouds.

[0123] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud segmentation method, characterized in that: The following steps are involved: S101: Construct a dual-branch encoder and a cascaded attention decoder. The dual-branch encoder consists of graph convolution and Transformer to obtain a multi-paradigm representation of the point cloud and extract local and global information of the point cloud. The cascaded attention decoder can learn multi-scale and multi-resolution spatial representations and utilizes multi-stage feature representations of the local and global point clouds. S102: Build an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention, use a weighted negative log-likelihood loss function, and use the training set to perform supervised training on the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention. Adjust the network parameters according to the loss value in each round of training to obtain a trained network model. S103: Use the trained network model to predict the point cloud in the test set to obtain the segmentation result of each point in the point cloud.

2. A point cloud segmentation method according to claim 1, characterized in that: The dual-branch encoder includes n cascaded dual-branch feature extraction modules, each of which includes a cascaded efficient edge convolution feature extraction layer, a point Transformer feature extraction layer and a multi-scale cross-attention fusion layer.

3. A point cloud segmentation method according to claim 1, characterized in that: The construction steps of the dual-branch encoder are as follows: 1) Construct an efficient edge convolution feature extraction layer: The K-nearest neighbor algorithm is used to construct the point cloud data into a local graph structure. The vertices are found through the farthest point sampling method, and then the neighbor nodes are selected according to the calculated Euclidean distance between the point pairs to form a directed graph structure. After the graph structure is completed, the edge convolution is used to learn the edge features between the point pairs to extract more accurate local feature information. x ; Stacking multiple efficient edge convolution feature extraction layers to form graph convolution; 2) Constructing the point Transformer feature extraction layer: The input coordinates are passed through a multi-layer perceptron to learn a dimensionless embedding feature Fe, and then the embedded feature Fe is fed into four stacked attention modules to learn the semantically rich and discriminative representation of each point, and then the global feature information F is generated through a linear layer. Y ; Stack multiple point Transformer feature extraction layers to form a Transformer branch; 3) Construct a multi-scale cross-attention fusion layer: First, the local feature information F x and global feature information F Y Projection, get X q , X k , X v and Y q , Y k , Y v , and then through the cross self-attention operation, the graph convolution branch and the Transformer branch guide each other in the learning process to obtain X o and Y o , where q, k, v, and o represent query, key, value, and output respectively. o and Y o Addition realizes efficient fusion of information from different branches; 4) The efficient edge convolution feature extraction layer, point Transformer feature extraction layer and multi-scale cross-attention fusion layer that are run in parallel are combined into a dual-branch feature extraction module, and n dual-branch feature extraction modules are superimposed in sequence to obtain local feature maps, global feature maps and local-global fused feature maps at different levels.

4. A point cloud segmentation method according to claim 3, characterized in that: The edge convolution function is used to learn the edge features between point pairs. The expression of the edge convolution function is as follows: e ij =h θ (P i ,P j -P i ); d i =Maxpool{ReLu(e ij }; Among them, ei j is the feature of the edge connecting the nodes, d i To obtain the new features, we will use P i The local graph structure of the central node is taken as input, P j is a neighbor node; (P j -P i ) is the vector difference between the neighbor node and the central node, and the association information between the point pairs is obtained; h θ (·) is a nonlinear function with a set of learnable parameters θ, which is used to extract features from the input graph structure and learn θ using a shared multi-layer perceptron; ReLu is a nonlinear activation function; Because it is not affected by the order of neighbor nodes, the maximum pooling function is used to aggregate the main features in the edge feature vector to obtain the final local feature information Fx.

5. A point cloud segmentation method according to claim 1, characterized in that: The cascaded attention decoder includes n upsampling modules cascaded in sequence, each upsampling module includes a feature interpolation layer and a feature attention fusion layer cascaded in sequence, and the construction steps are as follows: 1) Construct an upsampling module to interpolate the local and global feature maps Fgn obtained at different levels to the same level as Fgn through upsampling. gn-1 The fusion feature map F of the same scale un-1 ∈R (N / 4^n-1)×2^(n-1)d , where R is a real number set, N is the number of initial point clouds, n is the number of levels, and d is the number of channels. gn-1 and F un-1 Perform feature attention fusion, the fusion method is as follows: q att (g i-1 ,u i-1 )=σ1(BN(C g (g i-1 ))+BN(C u (u i-1 ))); AG(g i-1 ,u i-1 )=C g (g i-1 )*σ2(BN(C(q att (g i-1 ,u i-1 )))); F′ i-1 =MLP(concat(u i-1 ,AG(g i-1 ,u i-1 ))); Among them, σ1(·) and σ2(·) correspond to ReLU and Sigmoid activation functions respectively, C g (·), C u (·) and C(·) represent channel convolution operations 1×1, BN(·) is batch normalization operation, g and u are skip connection features and upsampling operations respectively, and q aat (·) indicates the attention weight calculation, AG(·) indicates the fusion of F gi-1 and F ui-1 The characteristic of F′ i-1 It represents the cascade of fusion features of different levels and scales; Then, parallel channel and spatial attention are used to enhance the spatial features and channel features of the point cloud respectively. Then, these features are reduced in dimension through convolution to obtain the feature map F i-1 , the above process is defined as follows: W c =σ(MLP(MaxPool(F′ i-1 ))+MLP(AvgPool(F′ i-1 ))); Among them, MaxPool and AvgPool represent Max pooling and Average Pooling respectively, σ represents Sigmoid function, ⊙ represents Hadamard product operation, W c represents the channel attention weight, Represents the spatial position attention weight; 2) Superimpose n upsampling modules in sequence, perform feature interpolation and fusion, obtain upsampled multi-level feature information, and finally obtain a feature map that integrates multi-stage features.

6. A point cloud segmentation method according to claim 1, characterized in that: The loss function in step S102 is based on the cross entropy loss. The loss is calculated by multiplying the log probability of the predicted value by the log probability of the target category and taking the negative of the multiplication. The calculation formula of the loss function is as follows: Among them, total_loss is the calculated total loss, N is the number of samples, and w i is the weight used to adjust the loss contribution of each sample, p i is the probability vector of the i-th sample in the predicted value, which contains the predicted probability of each category, target i is the category index of the i-th sample in the target label.

7. A point cloud segmentation system, characterized in that: include: A feature extraction construction module, used to construct a dual-branch encoder and a cascaded attention decoder, wherein the dual-branch encoder includes n cascaded dual-branch feature extraction modules, each of which includes a cascaded efficient edge convolution feature extraction layer, a point Transformer feature extraction layer, and a multi-scale cross-attention fusion layer; the cascaded attention decoder includes n cascaded upsampling modules containing feature attention; each upsampling module includes a feature interpolation layer and a feature attention fusion layer cascaded in sequence; The network model acquisition module is used to build an efficient point cloud segmentation network based on multi-paradigm representation and cascade attention. The weighted negative log-likelihood loss function is used to perform supervised training on the efficient point cloud segmentation network based on multi-paradigm representation and cascade attention using the training set. In each round of training, the network parameters are adjusted according to the loss value to obtain the network model. The segmentation result output module is used to use the trained network model to predict the point cloud in the test set and obtain the segmentation result of each point in the point cloud.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein the computer program can be used to implement the point cloud segmentation method according to any one of claims 1 to 5 when loaded and executed by a processor.

9. A collection device, characterized in that: The acquisition device includes: a 3D scanner, an RGB-D camera and a processor, wherein the processor is used to execute the point cloud segmentation method according to any one of claims 1 to 5.

10. An information data processing terminal, used for implementing the point cloud segmentation method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Plant organ point cloud segmentation method based on attention mechanism and graph convolution

    CN117036370A

  • Learning attention state evaluation method based on multi-dimensional feature fusion network

    CN117173758A

  • Point cloud data reconstruction method based on GCN-Transform model and electronic equipment

    CN117372617A

Cited By

  • Multi-parameter automobile body flow field prediction method and device and server

    CN119720393A

  • Multi-parameter automotive body flow field prediction method, device and server

    CN119720393B

  • Point cloud classification segmentation method and system of global mask auto-encoder based on fused voxel

    CN119762877A

  • Multi-mode remote sensing intelligent identification method for hidden geological disasters

    CN120107808A

  • Internet of vehicles channel prediction method based on multi-modal fusion and related equipment

    CN120342527A