Three-dimensional point cloud understanding processing method based on asymmetric parallel transformer network

Abstract

The application discloses a point cloud understanding processing method based on an asymmetric parallel Transformer network. Point cloud data is input into a multi-layer perceptron to obtain a point cloud feature vector; after down-sampling operation, the point cloud feature vector is input into a plurality of asymmetric parallel Transformer networks in succession to obtain an optimized point cloud feature vector; in the asymmetric parallel Transformer network, a channel layer is asymmetrically split into two feature vectors, and then the two feature vectors are respectively input into a local branch and a global branch to obtain local features and global features; the local features and the global features are spliced in a channel dimension to be input into a forward feedback network; and the step is repeated for multiple times, and finally, a classification or segmentation head is entered to obtain an output understanding result. The global branch explicitly captures global features, and meanwhile, the asymmetric parallel mode is adopted to effectively combine the features of the global branch and the local branch, global principal component attention is introduced as the global branch to expand an effective receptive field in a point cloud understanding task, local information and global information are effectively integrated, and the precision of a model in a classification and segmentation task is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of three-dimensional point cloud understanding of computer vision, and particularly relates to a three-dimensional point cloud analysis processing method based on an asymmetric parallel Transformer network. BACKGROUND

[0002] Three-dimensional point cloud understanding has become a basic task in various application fields such as autonomous driving, robots and virtual reality, specifically including point cloud classification and segmentation tasks. Because point cloud is different in structure from image, it is essentially a set in continuous three-dimensional space, which is sparse, irregular and unordered. These characteristics make it difficult for mature two-dimensional image understanding deep networks to be directly applied to three-dimensional point cloud understanding. Pointnet: Deep learning on point sets for 3d classification and segmentation published in IEEE International Conference on Computer Vision in 2017 is a pioneering work, which uses multi-layer perceptron and pooling layer to process features. Pointnet++: Deep hierarchical feature learning on point sets in a metric space published in Conference on Neural Information Processing Systems in 2017 is based on PointNet, and adopts a hierarchical sampling strategy to extract local features. After PointNet and PointNet++, there are also many point-based methods, but they mainly focus on proposing new modules to capture local features, ignoring the construction of global features.

[0003] Due to excellent ability to capture global features, Transformer network is very popular in natural language processing and two-dimensional image understanding, and is particularly suitable for three-dimensional point cloud understanding. Because it is invariant to the arrangement and cardinality of input elements, it is essentially a set operator. Based on these advantages, some works try to develop modules based on Transformer network. However, due to the quadratic complexity of attention operator, previous works are still difficult to build global context for three-dimensional point cloud understanding.

[0004] For example, Point Transformer published in 2021 in IEEE International Conference on Computer Vision, an international top computer vision conference, is a pioneering work that uses local sub-attention to aggregate the features of grouped neighbors, but ignores to explicitly capture long-range context features, resulting in limited effective receptive field. Recently, Stratified transformer for 3d point cloud segmentation published in IEEE Conference on Computer Vision and Pattern Recognition in 2022 adopts a window-based sub-attention module and proposes a key-based stratified sampling strategy, which to some extent expands the effective receptive field. However, due to the window-based attention, the global information is lost, and the effective receptive field is still limited. On the other hand, due to the introduction of a special stratified sampling strategy, the computational cost is significantly increased compared with Point Transformer. SUMMARY

[0005] Based on the shortcomings of the prior art, the present application proposes a point cloud understanding analysis method based on an asymmetric parallel Transformer network, which explicitly captures global features through a global branch, and effectively combines the features of the global branch and the local branch in an asymmetric parallel manner, improving the accuracy of the model in classification and segmentation tasks.

[0006] The specific technical solutions of the present application are as follows:

[0007] (1) input the point cloud data into a multi-layer perceptron to obtain a point cloud feature vector;

[0008] (2) then perform down-sampling operation on the point cloud feature vector, and then input it into a plurality of asymmetric parallel Transformer networks for processing to obtain an optimized point cloud feature vector;

[0009] (3) repeat the above step (2) multiple times, and input the last point cloud feature vector into a classification or segmentation head to obtain an output understanding result.

[0010] The final understanding result can be actually used in point cloud classification and segmentation tasks.

[0011] The original point cloud data of the method usually comes from the point cloud data collected by radar and the like.

[0012] For the input point cloud data I N×C , the specific processing of each step is as follows:

[0013] The specific process of step (1) is: first, the point cloud data I ∈ R N×C The feature extraction layer with a multi-layer perception and an asymmetric parallel Transformer network is inputted to obtain a point cloud feature vector X ∈ R N×C , R represents a real number set, N represents the number of points, and C represents the dimension of a channel.

[0014] The feature extraction layer is mainly composed of a multi-layer perception and two asymmetric parallel Transformer networks connected in sequence.

[0015] The single asymmetric parallel Transformer network in steps (1) and (2) is processed in the following manner:

[0016] S1, the point cloud feature vector is asymmetrically split into two feature vectors in the channel layer;

[0017] S2, the two feature vectors are respectively inputted into a local branch and a global branch to obtain respective local features and global features;

[0018] S3, the obtained local features and global features are spliced together in the channel dimension, and then inputted through a forward feedback network to obtain an optimized point cloud feature vector as the output of the asymmetric parallel Transformer network.

[0019] The S3 is specifically: the local feature Y l and the global feature Y g are spliced together in the channel dimension through tensor splicing to obtain spliced features, the spliced features are then residual connected with the point cloud feature vector inputted into the asymmetric parallel Transformer network to obtain fused features, the fused features are then inputted into a forward feedback network to obtain feedback features, and finally the feedback features are residual connected with the fused features to obtain the optimized point cloud feature vector Y.

[0020] The local branch is mainly composed of a local subgroup attention module, the local subgroup attention module is to find the nearest 16 neighboring elements as neighbor elements through K nearest neighbor algorithm for each element in the vector, and the feature value of the traversed element in the vector is taken as a query vector Query, and the feature values of the neighbor elements in the vector are taken as a key vector Key and a value vector Value to input a vector cross attention module to obtain a local feature

[0021] The global branch mainly consists of a global principal element attention module, the global principal element attention module finds the principal element elements uniformly distributed in the global through the farthest point sampling algorithm, and the eigenvalues of the principal element elements in the vector are taken as the key vector Key and the value vector Value, and the eigenvalues of all elements in the vector are taken as the query vector Query to input the scalar cross attention module to obtain the global feature

[0022] In step (2), the point cloud feature X is divided into two parts in the channel dimension N×C Asymmetrically divided into two parts in the channel dimension And X l Indicates the part input to the local branch, X g Indicates the part input to the global branch, C l Indicates the channel dimension of the part input to the local branch, C g Indicates the channel dimension of the part input to the global branch, with the number of times of repeating step (2) in-depth, the channel dimension C g Continuously increases, and the channel dimension C l Continuously decreases, so that the overall model method composed of steps (1) to (3) pays more attention to global features.

[0023] In the first step (2) processing, C g : C is set to 1:8;

[0024] In the second step (2) processing, C g : C is set to 1:8;

[0025] In the third step (2) processing, C g : C is set to 1:4;

[0026] In the fourth step (2) processing, C g : C is set to 1:1.

[0027] In step (3), step (2) is repeated four times, the first time step (2) is repeated through three consecutive asymmetric parallel Transformer network processing, the second time step (2) is repeated through four consecutive asymmetric parallel Transformer network processing, the third time step (2) is repeated through six consecutive asymmetric parallel Transformer network processing, and the fourth time step (2) is repeated through three consecutive asymmetric parallel Transformer network processing.

[0028] In step (3):

[0029] The classification head is mainly composed of a global pooling operation GAP and a multi-layer perception operation MLP connected in sequence.

[0030] The segmentation head comprises a decoder and a multi-layer perception operation (MLP), the decoder comprises a plurality of sampling extraction units and a conversion extraction unit, each sampling extraction unit is sequentially connected by an up-sampling operation and an asymmetric parallel Transformer network, and each conversion extraction unit is sequentially connected by a feature conversion operation (transformation) and an asymmetric parallel Transformer network.

[0031] When the segmentation head is adopted, the number of the sampling extraction units is the same as the number of times of repeating step (2), the optimized point cloud feature vector obtained by processing the last time of repeating step (2) is input into the conversion extraction unit for processing, the output result of the conversion extraction unit and the optimized point cloud feature vector obtained by processing the second last time of repeating step (2) are added together and input into the first sampling extraction unit, the output result of the kth sampling extraction unit and the optimized point cloud feature vector obtained by processing the n-kth time of repeating step (2) are added together and input into the k+1th sampling extraction unit, k is not equal to n and n-1, the output result of the n-1th sampling extraction unit and the point cloud feature vector obtained by processing step (1) are added together and input into the nth sampling extraction unit, and the output result of the nth sampling extraction unit is input into the multi-layer perception operation (MLP) for processing to obtain the optimized point cloud feature vector.

[0032] During training, the above steps (1) to (2) are repeated multiple times until the preset training times are reached, and then the training is ended, and the trained multi-layer perception machine and asymmetric parallel Transformer network are used for processing implementation in the above steps (1) to (3), so that the point cloud classification or segmentation is realized.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1. The present application provides a new point cloud asymmetric parallel Transformer network, which effectively models the short-range and long-range dependency relationship through parallel branches. In addition, the present application also introduces a global principal component attention as a global branch to expand the effective receptive field in the point cloud understanding task.

[0035] 2. The present application designs an asymmetric parallel structure, effectively integrates local information and global information, improves the accuracy in point cloud classification and segmentation tasks, and does not introduce additional computational complexity. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The overall framework of the present application is shown in the following figure:

[0037] Figure 2A flowchart of the asymmetric parallel Transformer network of the present application is shown in the figure.

[0038] Figure 3 A flowchart of the global feature extraction branch is shown in the figure.

[0039] Figure 4 A comparison chart of the effective receptive field of the present application and the prior art is shown in the figure.

[0040] Figure 5 The qualitative segmentation results of the embodiment of the present application on the S3DIS point cloud segmentation dataset are shown in the figure. DETAILED DESCRIPTION

[0041] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments, it should be noted that the following embodiments are intended to facilitate the understanding of the present application and do not limit the present application in any way.

[0042] As shown in the figure, the embodiment of the present application and its implementation process are as follows: Figure 1

[0043] (1) input the point cloud data into the multi-layer perception to obtain the point cloud feature vector;

[0044] For the input point cloud data I∈R N×C , the specific processing of each step is as follows:

[0045] The specific process of step (1) is as follows: first, input the point cloud data I∈R N×C into the feature extraction layer with multi-layer perception and asymmetric parallel Transformer network to obtain the point cloud feature vector X∈R N×C , where R represents the set of real numbers, N represents the number of points, and C represents the dimension of the channel.

[0046] The feature extraction layer is only composed of multi-layer perception and two asymmetric parallel Transformer networks connected in sequence.

[0047] The above asymmetric parallel Transformer network is processed in the following manner:

[0048] S1, asymmetrically split the point cloud feature vector into two feature vectors in the channel layer;

[0049] S2, input the two feature vectors into the local branch and the global branch respectively to obtain their respective local features and global features;

[0050] S3, concatenate the obtained local features and global features in the channel dimension, and then input them through the forward feedback network to obtain the optimized point cloud feature vector as the output of the asymmetric parallel Transformer network. ​

[0051] S3 specifically is: the local feature Y l and the global feature Y g After being spliced together in the channel dimension through tensor splicing, the spliced feature is obtained, and then the spliced feature is connected with the point cloud feature vector of the original input asymmetric parallel Transformer network to obtain a fusion feature, and then the fusion feature is input into a forward feedback network to obtain a feedback feature, and finally the feedback feature and the fusion feature are connected through a residual connection operation to obtain an optimized point cloud feature vector Y.

[0052] As shown in Figure 2 , in the asymmetric parallel Transformer network, the point cloud features are first divided into two sub-features, which are input into the local group attention module (LGA) and the global principal component attention module (GPA), respectively. After that, the outputs of the two branches are merged together, and the result is passed through a forward feedback network layer composed of a multi-layer perceptron.

[0053] The local branch is mainly composed of a local group attention module, which is to find the nearest 16 neighboring elements as neighbor elements through the K nearest neighbor algorithm by traversing each element in the vector, and input the feature value of the traversed element in the vector as the query vector Query, and the feature values of the neighbor elements in the vector as the key vector Key and the value vector Value into the vector cross attention module to obtain the local feature

[0054] The global branch is mainly composed of a global principal component attention module, which is to find the principal component elements uniformly distributed in the global through the farthest point sampling algorithm, and input the feature values of the principal component elements in the vector as the key vector Key and the value vector Value, and the feature values of all elements in the vector as the query vector Query into the scalar cross attention module to obtain the global feature

[0055] As shown in Figure 3 , for the global feature extraction branch, the global principal component attention module (GPA) is adopted. Specifically, the input point cloud is first sampled by the farthest point sampling algorithm to obtain the principal component points uniformly distributed. Then the Key and Value in the attention mechanism are obtained from the principal component points through the multi-layer perceptron, and all point cloud features are used as Query to maintain the number of point clouds in the final output result. After that, the Key and Query are used to generate scalar attention, and the output of the attention mechanism is obtained by multiplying the scalar attention and the Value, and then the final result is obtained through the classification head or the segmentation head.

[0056] As shown in Figure 4As shown, the existing work Point Transformer adopts local group attention within the group neighbors to extract local features. Stratified Transformer uses window-based attention and stratified key sampling strategy to partially expand the effective receptive field. Different from the prior art, the present application uses parallel branches to capture local features and global features respectively. In addition, the global branch directly models long-distance dependencies through the principal point, further expanding the effective receptive field.

[0057] (2) The point cloud feature vector is further down-sampled and input into a plurality of asymmetric parallel Transformer networks for processing to obtain an optimized point cloud feature vector;

[0058] In step (2), the point cloud feature X∈R N×C is asymmetrically divided into two parts in the channel dimension and X l represents the part input to the local branch, X g represents the part input to the global branch, C l represents the channel dimension of the part input to the local branch, C g represents the channel dimension of the part input to the global branch, with the number of times of repeating step (2) deepening, the channel dimension C g continuously increases and the channel dimension C l continuously decreases, so that the overall model method composed of steps (1) to (3) pays more attention to global features.

[0059] (3) Specifically, repeat step (2) four times, when repeating step (2) the first time, pass through three consecutive asymmetric parallel Transformer networks for processing, C g : C is set to 1:8.

[0060] When repeating step (2) the second time, pass through four consecutive asymmetric parallel Transformer networks for processing, C g : C is set to 1:8; when repeating step (2) the third time, pass through six consecutive asymmetric parallel Transformer networks for processing, C g : C is set to 1:3; when repeating step (2) the fourth time, pass through three consecutive asymmetric parallel Transformer networks for processing, C g : C is set to 1:1.

[0061] Finally, input the point cloud feature vector obtained by repeating step (2) the fourth time into a classification or segmentation head to obtain an output understanding result.

[0062] The classification head is mainly composed of a global pooling operation GAP and a multilayer perception operation MLP connected in sequence.

[0063] The segmentation head includes a decoder and a multi-layer perception operation (MLP), the decoder includes a plurality of sampling extraction units and a conversion extraction unit, each sampling extraction unit is sequentially connected by an upsampling operation and an asymmetric parallel Transformer network, and each conversion extraction unit is sequentially connected by a feature conversion operation (transformation) and an asymmetric parallel Transformer network.

[0064] When the segmentation head is used, the number of sampling extraction units is the same as the number of times of repeating step (2), the optimized point cloud feature vector obtained by processing the last time of repeating step (2) is input into the conversion extraction unit for processing, the output result of the conversion extraction unit and the optimized point cloud feature vector obtained by processing the second last time of repeating step (2) are added together and input into the first sampling extraction unit, the output result of the kth sampling extraction unit and the optimized point cloud feature vector obtained by processing the n-kth time of repeating step (2) are added together and input into the k+1th sampling extraction unit, k is not equal to n and n-1, the output result of the n-1th sampling extraction unit and the point cloud feature vector obtained by processing step (1) are added together and input into the nth sampling extraction unit, and the output result of the nth sampling extraction unit is input into the multi-layer perception operation (MLP) for processing to obtain the optimized point cloud feature vector.

[0065] In a specific implementation, steps (1) to (3) constitute an overall model.

[0066] As shown in FIG. 1, for the three-dimensional point cloud shape classification task of the top row, the input features are first embedded into point cloud features by a multi-layer perception, and then passed through an asymmetric parallel Transformer network, and then passed through several stages composed of a down-sampling layer and an asymmetric parallel Transformer network, and the features are transmitted to a classification head composed of a global average pooling and a multi-layer perception. Figure 1

[0067] As shown in FIG. 2, for the three-dimensional point cloud segmentation task of the bottom two rows, the features are input into each stage composed of an up-sampling layer and an asymmetric parallel Transformer network. Then the features pass through a segmentation head composed of a multi-layer perception. The black arrow is the feature flow, and the purple arrow is the merging operation in the up-sampling layer. Figure 1

[0068] ​​During training, the above steps (1) to (2) are repeated multiple times until a preset training number is reached, and the training is ended, obtaining a trained overall model and the multilayer perceptron and asymmetric parallel Transformer network therein, for processing implementation by the overall model constituted by the above steps (1) to (3), to realize point cloud classification or segmentation.

[0069] Figure 5 In the middle, the segmentation effect of the algorithm and the existing working model of the present application is shown. It can be seen that the segmentation result of the algorithm model of the present application has greatly improved the visual perception effect compared with the segmentation result of the existing working model. This qualitatively proves the effectiveness of the algorithm of the present application.

[0070] The method proposed in the present application will be applied to actual examples below, and compared with other methods of the same type to reflect the technical effects and advantages of the present application.

[0071] The present application has been tested on two public data sets, S3DIS point cloud segmentation data set and ModelNet point cloud classification data set.

[0072] The S3DIS data set contains 271 rooms from 6 areas of 3 different buildings. These point clouds are annotated into 13 categories, such as bookshelves, floors and wooden boards. Following the general settings of previous works, the present application reserves area 5 during training and uses it to evaluate the proposed model.

[0073] The present application uses the average class intersection over union (mIoU), the average class accuracy (mAcc) and the overall accuracy (OA) as the evaluation indicators of performance. On the other hand, the present application uses the average GFLOPs of area 5 as the evaluation indicator of efficiency. The results are shown in the following table.

[0074] Table 1

[0075]

[0076] As shown in Table 1, the segmentation model proposed in the present application has better performance than the traditional method, and does not introduce additional computational consumption in terms of efficiency.

[0077] For the task of three-dimensional point cloud shape classification, the present application conducts experiments on the ModelNet40 data set. It consists of 12311 CAD models, including 40 object categories, divided into 9843 models for training and 2468 models for testing. According to previous works, the present application uniformly samples 1024 points from the mesh surface of each model, and rescales these points to adapt to the unit sphere.

[0078] For evaluation metrics, the present application uses the class average accuracy (mAcc) and overall accuracy (OA). The results are shown in the following table.

[0079] Table 2

[0080]

[0081] As shown in Table 2, the algorithm proposed by the present application shows a larger improvement effect than the reference algorithm on the three-dimensional point cloud shape classification task.

[0082] The above-described embodiments describe the technical solutions and beneficial effects of the present application in detail. It should be understood that the above-described is only a specific embodiment of the present application and is not used to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A point cloud understanding and processing method based on asymmetric parallel Transformer networks, characterized in that... The methods include: Step (1) Input the point cloud data into the multilayer perceptron to obtain the point cloud feature vector; Step (2) After downsampling the point cloud feature vector, it is then input into multiple consecutive asymmetric parallel Transformer networks to obtain the optimized point cloud feature vector; Step (3) Repeat step (2) multiple times to input the final point cloud feature vector into the classification or segmentation head to obtain the output understanding result; The specific process of step (1) is as follows: First, the point cloud data I∈R N×C Inputting a feature extraction layer with a multilayer perceptron and an asymmetric parallel Transformer network, we obtain a point cloud feature vector X∈R. N×C R represents the set of real numbers, N represents the number of points, and C represents the dimension of the channel; The single asymmetric parallel Transformer network in step (2) is processed in the following manner: S1. Asymmetrically split the point cloud feature vector into two feature vectors at the channel layer; S2. Process the two feature vectors into the local branch and the global branch respectively to obtain the local features and the global features; In S2, the point cloud features X∈R N×C It is asymmetrically divided into two parts along the channel dimension. and X l X represents the portion of the input that goes into the local branch. g This represents the part that inputs into the global branch, C l C represents the channel dimension of the portion of the input to the local branch. g This represents the channel dimension of the portion input to the global branch. As step (2) is repeated multiple times, the channel dimension C increases. g The channel dimension C is constantly increasing. l The continuous reduction in size allows for a greater focus on global features; S3. The obtained local and global features are concatenated together along the channel dimension and then input into the optimized point cloud feature vector obtained through the feedforward network as the output of the asymmetric parallel Transformer network.

2. The point cloud understanding and processing method based on asymmetric parallel Transformer network according to claim 1, characterized in that: Specifically, S3 involves: transferring the local feature Y... l and global features Y g After concatenating the features together in the channel dimension using tensor concatenation, the concatenated features are obtained. Then, the concatenated features are combined with the point cloud feature vectors of the original input asymmetric parallel Transformer network through residual connection to obtain fused features. The fused features are then input into the feedforward network to obtain feedback features. Finally, the feedback features and the fused features are combined through residual connection to obtain the optimized point cloud feature vector Y.

3. A point cloud understanding and processing method based on an asymmetric parallel Transformer network according to claim 1 or 2, characterized in that: The local branch mainly consists of a local grouping attention module. This module iterates through each element in the vector, using the K-nearest neighbor algorithm to find its 16 nearest neighbors. It then uses the feature values ​​of the iterated elements as the query vector (Query) and the feature values ​​of the neighboring elements as both the key and value vectors (Key and Value) as input vectors to cross-reference the attention module, thereby obtaining local features. The global branch mainly consists of a global principal component attention module. This module uses a farthest-point sampling algorithm to find principal elements evenly distributed globally. The feature values ​​of these principal elements are used as both the key vector (Key) and value vector (Value), while the feature values ​​of all elements are used as the query vector (Query). This input is then fed into a scalar cross-attention module to obtain global features.

4. The point cloud understanding and processing method based on asymmetric parallel Transformer network according to claim 1, characterized in that: During the first step (2) of the process, C g C is set to 1:8; During the second step (2), C g C is set to 1:8; During the third step (2) of the process, C g C is set to 1:4; During the fourth step (2) of the process, C g C is set to 1:

1.

5. The point cloud understanding and processing method based on asymmetric parallel Transformer network according to claim 1, characterized in that: In step (3), step (2) is repeated four times. The first time step (2) is repeated, it is processed by three consecutive asymmetric parallel Transformer networks. The second time step (2) is repeated, it is processed by four consecutive asymmetric parallel Transformer networks. The third time step (2) is repeated, it is processed by six consecutive asymmetric parallel Transformer networks. The fourth time step (2) is repeated, it is processed by three consecutive asymmetric parallel Transformer networks.

6. The point cloud understanding and processing method based on asymmetric parallel Transformer network according to claim 1, characterized in that: In step (3): The classification head is mainly composed of a global pooling operation (GAP) and a multilayer perceptron (MLP) connected sequentially. The segmentation head includes a decoder and a multilayer perceptron (MLP). The decoder includes multiple sampling extraction units and a transformation extraction unit. Each sampling extraction unit is composed of an upsampling operation and an asymmetric parallel Transformer network connected in sequence. Each transformation extraction unit is composed of a feature transformation operation and an asymmetric parallel Transformer network connected in sequence.

7. The point cloud understanding and processing method based on asymmetric parallel Transformer network according to claim 1, characterized in that: During training, repeat steps (1) to (2) multiple times until the preset number of training iterations is reached and training ends. The trained multilayer perceptron and asymmetric parallel Transformer network are then used in steps (1) to (3) to implement point cloud classification or segmentation.

Images