A method for object classification and segmentation based on 3D point cloud information

Abstract

The present invention relates to an object classification and object segmentation method based on 3D point cloud information, which belongs to the field of 3D point cloud processing, and particularly to the field of difficult processing of disordered point clouds and difficult extraction of point cloud features. In order to extract more effective information in extracting point cloud features, the present invention proposes and uses a new joint global feature and local feature structure. When extracting high-dimensional features, a new extraction mechanism is proposed to replace the traditional symmetric function. A large number of experiments have shown that the neural network we designed can solve a variety of point cloud tasks, including but not limited to object classification, object segmentation, scene segmentation, etc. The beneficial effects of the present invention: the designed point cloud recognition and segmentation algorithm mainly designs a new Mix‑Net dedicated neural network design, and innovates from multiple modules such as downsampling algorithm, feature value extraction and fusion mechanism, which significantly improves the accuracy of point cloud recognition and segmentation.

Description

Technical Field

[0001] The present invention relates to the field of 3D point cloud processing, and in particular to the field of difficulty in processing disordered point clouds and extracting point cloud features. A method for object classification and object segmentation based on 3D point cloud information is proposed. Background Art

[0002] 3D point cloud technology is currently used in the fields of intelligent driving, virtual reality, and robotics. The main purpose is to extract effective features from 3D point clouds to describe the category or segmentation of point clouds. When processing 3D point clouds, we should pay attention to the order disorder of point clouds and the permutation disorder of point clouds. Order disorder means that point clouds can have N representation forms in a container, but no matter which representation method is used, they still describe the same object. Permutation disorder is a simple operation of point clouds in the metric space after rotation and translation. In the standard coordinate system, the container value representing the point cloud changes accordingly, but still represents the same target object. In the above two cases, multiple representations of the container can still extract similar or identical features to represent the point cloud, which is the difficulty of point cloud processing. At present, the existing technology is only for the overall feature extraction of point clouds, and it is still difficult to involve efficient end-to-end neural networks. Compared with existing neural networks, the accuracy in object classification and object segmentation tasks has been significantly improved. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the present invention provides an object classification and object segmentation method based on 3D point cloud information, aiming to improve the accuracy of point cloud recognition and segmentation, and reduce information loss caused by dimensionality reduction or voxel change; and can be widely used in a variety of sensing devices.

[0004] The present invention will be further described below in conjunction with the accompanying drawings:

[0005] The object classification and object segmentation method based on 3D point cloud information comprises the following steps:

[0006] Step 1: Add position encoding to the input point cloud information. For the input point cloud information, add input relative position information. The relative position information is added in the first input layer and in the attention function. After a large number of experiments, it can be proved that the addition of relative position encoding can improve network performance and improve the final output accuracy.

[0007] Step 2: Data downsampling algorithm. Using a downsampling algorithm for input can reduce the complexity of the overall network, speed up network training, and suppress overfitting. The present invention proposes a new downsampling algorithm based on an attention mechanism to replace the traditional farthest point distance algorithm.

[0008] Step 3: Multi-head attention mechanism of relative position information fusion. As the main layer for extracting features in the network, we use the multi-head attention layer of relative position information fusion to replace the traditional convolution layer. Experiments show that the multi-head attention layer can increase network performance and improve the final output accuracy.

[0009] Step 4: Design of point cloud encoder structure. The point cloud encoder structure designed by the present invention is used to extract high-dimensional feature vectors from input point cloud data. The feature vectors include point cloud global feature information and point cloud local feature information. When fusing features to obtain the overall features of the point cloud, the present invention proposes a new feature fusion mechanism to replace the traditional symmetric function.

[0010] Step 5: Design of point cloud decoder structure. The present invention designs different decoder structures for different point cloud processing tasks, which are similar but different.

[0011] Step 6: Design the Mix-Net network training method. Mainly based on different point cloud processing tasks and different data sets, design loss functions, optimizers, and various training parameter indicators.

[0012] Furthermore, the implementation process of the position encoding in step 1 is as follows: Since there are multiple storage methods for the input point cloud container, it is difficult to directly obtain relative position information between points (between different elements of the same input) in the network. Therefore, relative position information is introduced to increase the position feature extraction between points. The position encoding formula of the present invention is as follows:

[0013]

[0014] Among them, P i is the input data or features, It is the average value of input data or features, and MLP is a commonly used multi-layer perceptron. The present invention has found in multiple experiments that the addition of position information in the first layer input and attention mechanism module has a significant improvement in the overall network performance.

[0015] Furthermore, the main implementation process of the data downsampling algorithm described in step 2 is as follows:

[0016] (1) Use an algorithm based on a multi-head attention mechanism to replace the traditional farthest point sampling algorithm. The features of the convolution output are used to obtain an attention score through a multi-head attention mechanism, and the effect similar to collecting k values ​​of the farthest point is achieved by selecting the first k maximum attention score values. Experiments have shown that the downsampling algorithm used in the present invention is more suitable for point clouds, and similar feature outputs can be obtained for a variety of different sequential inputs, which can solve the problem of point cloud disorder.

[0017] (2) According to the obtained k maximum values, the determined sequence points are matched in the original point cloud input, and these sequence points are represented as key points. The key points represent the most critical part of the point cloud. It can be easily seen from the key point visualization that the key points are the skeleton information of the point cloud.

[0018] (3) Based on the selected key points, the k-means algorithm is used to select feature points around each key point, all feature points that meet the k-means algorithm are obtained for feature extraction, and multi-layer features are fused using a multi-scale feature fusion method. The high-dimensional features after downsampling are output.

[0019] Furthermore, the implementation process of the multi-head attention mechanism for the relative position information fusion described in step 3 is mainly as follows:

[0020] (1) Using relative distance at the input can increase the overall performance of the network. After a large number of experiments, it can be seen that adding relative distance ρ in the attention function can still increase the overall network performance. The present invention adds relative distance to the attention function, and the formula is as follows:

[0021]

[0022] Among them, Q, K, V are query, key, value matrices obtained according to their respective change matrices, softmax is the activation function, ρ is the relative distance, and the present invention adds a relative distance value at the transpose of K and adds a relative distance value at V. The attention function designed by the present invention can extract more sufficient information between elements in the input, and is more suitable for point cloud processing.

[0023] (2) Introduce a multi-head attention mechanism. The input passes through multiple independent attention layers, and finally the output results of each layer are cascaded to obtain the output of the multi-head attention layer. This method can be run in parallel and has a significant speed advantage in processing input data. The formula of the multi-head attention mechanism is as follows:

[0024]

[0025] F i =Attention(Q i , K i , V i ), i=1, ..., h,

[0026] MultiHead(Q,K,V)=Concat(F1,F2,...,F h )W O

[0027] Where X represents the input, and There are three different sets of linear matrices. h represents the number of multi-head attention mechanisms. This invention uses h=4. Represents the output feature matrix. F i Represents the final output value of each multi-head attention layer. Concat represents cascade. The multi-head attention mechanism divides the input into h independent attention mechanism layers and operates each attention mechanism layer in parallel.

[0028] Furthermore, the implementation process of the point cloud encoder structure design described in step 4 is as follows:

[0029] (1) For input, the input of a point cloud sample is Where N represents the number of inputs. Each input point cloud has a d-dimensional vector. Common d is 3 or 6. When d is 3, the point cloud only contains position information. When d is 6, the point cloud contains position information and normal vector information. p is the output of the relative position encoding layer, F e Yes F p The output feature vector after the downsampling layer is F o F e After the output of 4 layers of multi-head attention mechanism, F o The output feature vector formula is as follows:

[0030] MSA(A)=Attention(A,A,A),

[0031] F1=MSA(F e ),

[0032] F i =MSA(F i-1 ), i = 2, 3, 4,

[0033] F o =Concat(F1,F2,...,F4)·W O ,

[0034] Among them, each MSA (F i ) represents the output feature of the i-th attention layer, and the output dimension of each layer is consistent with the input dimension. o are the weights of the linear layer, which change during network training.

[0035] (2) In order to extract the global feature F global With local features F localBased on the point cloud features, the present invention designs a new feature fusion layer to replace the traditional symmetric function. The newly proposed fusion layer can still solve the disorder problem of point clouds and greatly reduce the features lost by point clouds due to dimensionality reduction. The new fusion layer formula is as follows:

[0036] MCA(A,B)=Attention(A,B,B),

[0037] F lg =MCA(F local , F global ).

[0038] Beneficial effects of the invention: The point cloud recognition and segmentation algorithm designed by the patent of the present invention mainly designs a new Mix-Net dedicated neural network design, and innovates from multiple modules such as downsampling algorithm, feature value extraction and fusion mechanism, which significantly improves the accuracy of point cloud recognition and segmentation. In addition, the network is an end-to-end network and does not require the addition of other auxiliary modules to adjust the input, and directly processes the input point cloud to reduce information loss caused by dimensionality reduction or voxel changes. Sufficient experiments have been conducted on common data sets, common radar equipment, and common RGB-D cameras. The algorithm proposed by the present invention can be widely used in a variety of sensor perception devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Mix-Net network diagram

[0040] Figure 2 Mix-Net network overall structure flow chart

[0041] Figure 3 Mix-Net network downsampling module

[0042] Figure 4 Mix-Net network performance in classification tasks

[0043] Figure 5 Mix-Net network for aircraft model segmentation in the segmentation task. The left picture shows the human label segmentation, and the right picture shows the network output result.

[0044] Figure 6 Mix-Net network segmentation of the water cup model in the segmentation task. The left picture shows the human label segmentation, and the right picture shows the network output result.

[0045] Figure 7 Mix-Net network segmentation of chair model in segmentation task. The left picture is the human label segmentation, and the right picture is the network output result DETAILED DESCRIPTION

[0046] The following is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, so that the purpose of the invention, technical solutions, features, etc. are easier to understand. Obviously, the described embodiments are only part of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The embodiments are only used to explain the present invention, not to limit the present invention.

[0047] The present invention proposes an object classification and object segmentation method based on 3D point cloud information. The main structure flow chart is as follows: Figure 1 , Figure 2 , the complete method can be described by the following steps:

[0048] Step 1. Add position coding to the input. Position coding is a commonly used method in natural language processing. It can increase the physical connection between input features, help improve the overall performance of the network, and improve the accuracy of classification and segmentation tasks. Position information needs to be added during the input process, and position information can generally be divided into relative position and absolute position. In point cloud processing, the input data already contains clear three-dimensional position information. However, due to the existence of multiple storage methods for the input point cloud container, it is difficult to directly obtain relative position information between points (between different elements of the same input) in the network. Therefore, relative position information is introduced to increase the extraction of position features between points. The relative position coding formula of the present invention is as follows:

[0049]

[0050] Among them, P i is the input data or features, It is the average value of input data or features, and MLP is a commonly used multi-layer perceptron. The present invention has found in multiple experiments that the addition of position information in the first layer input and attention mechanism module has a significant improvement in the overall network performance.

[0051] Step 2: Data downsampling algorithm. The downsampling algorithm is to reduce the input data or feature data output, which can facilitate the subsequent network to target strong feature information, reduce the complexity of the overall network, reduce the overall network parameters, and speed up the network training process. The overall structure diagram is as follows Figure 3 The specific implementation is as follows:

[0052] 1. Use convolution processing for input point cloud. Convolutional network is easy to extract low-dimensional feature information compared with attention mechanism, through two convolution layers, each convolution layer contains one-dimensional convolution, batch normalization layer and ReLu activation function layer.

[0053] 2. Use an algorithm based on a multi-head attention mechanism to replace the traditional farthest point sampling algorithm. The features of the convolution output are used to obtain an attention score through a multi-head attention mechanism, and the effect similar to collecting k values ​​of the farthest point is achieved by selecting the first k maximum attention score values. Experiments have shown that the downsampling algorithm used in the present invention is more suitable for point clouds, and similar feature outputs can be obtained for a variety of different sequential inputs, which can solve the problem of point cloud disorder.

[0054] 3. According to the obtained k maximum values, the determined sequence points are matched in the original point cloud input, and these sequence points are represented as key points. The key points represent the most critical part of the point cloud. It can be easily seen from the key point visualization that the key points are the skeleton information of the point cloud.

[0055] 4. Based on the selected key points, use the k-means algorithm to select feature points around each key point, obtain all feature points that meet the k-means algorithm for feature extraction, and use the multi-scale feature fusion method to fuse multiple layers of features. Output the downsampled high-dimensional features.

[0056] Step 3: Multi-head attention mechanism for relative position information fusion. The specific implementation is as follows:

[0057] 1. The attention mechanism is used in the field of natural language processing to calculate the relevance between each word in the input sentence. Q, K, and V are the query, key, and value matrices obtained from the input sentence according to their respective change matrices. When dx=dy, it is a self-attention mechanism; when dx≠dy, it is a cross-attention mechanism. The calculation formula of the attention mechanism is as follows:

[0058] Q=F inx ·W Q , K=F iny ·W K , V=F iny ·W v

[0059] Among them, W Q , W K , W V are all linear matrices, and d q , d k , d v They are query, key, and value dimensions. It can be seen that query is obtained through F inx Determine, key, value is determined by F iny The present invention is used It can increase the computational efficiency of the attention mechanism.

[0060] An attention mechanism layer is mainly based on the product of Q and the relevant K, and then multiplied by the corresponding V through a specific activation function to get the output. The commonly used attention function formula is as follows:

[0061]

[0062] The attention weight is vector multiplied by Q and the corresponding K transpose and divided by d k , which can prevent the overall product from being too large. The softmax function is used to get the final attention weight. Finally, the attention weight is multiplied by the corresponding V to get the attention output.

[0063] Using relative distance in input can increase the overall performance of the network. After a lot of experiments, it can be seen that adding relative distance ρ in the attention function can still increase the overall network performance. The present invention adds relative distance to the attention function, and the formula is as follows:

[0064] The present invention adds a relative distance value at the transpose of K and adds a relative distance value at V. The attention function designed by the present invention can extract more sufficient information between elements in the input, and is more suitable for point cloud processing.

[0065] 2. Multi-head attention mechanism

[0066] Due to the limitation of input vector or feature being too large, the single-head attention mechanism may not extract sufficient features. In order to solve this problem, the present invention uses a multi-head attention mechanism, which can speed up the overall training speed of the network and reduce overfitting. The main idea is: the input passes through multiple independent attention layers, and finally the output results of each layer are cascaded to obtain the output of the multi-head attention layer. This method can be run in parallel and has obvious speed advantages in processing input data. The formula of the multi-head attention mechanism is as follows:

[0067]

[0068] F i =Attention(Q i , K i , V i ), i=1, ..., h,

[0069] MultiHead(Q,K,V)=Concat(F1,F2,...,F h )W O

[0070] Where h represents the number of multi-head attention mechanisms, and the present invention uses h=4. Represents the output feature matrix. F iRepresents the output value of each multi-head attention layer. and are three different sets of linear matrices. Finally, the multi-head attention mechanism divides the input into h independent attention mechanism layers and operates each attention mechanism layer in parallel.

[0071] Step 4: Point cloud encoder structure. The overall structure of Mix-Net is as follows: Figure 2 . The goal of Mix-Net is to obtain a new high-dimensional vector to represent the point cloud through a network encoder structure. The high-dimensional vector has a certain robustness to different inputs, and has a general effect on the rotation and translation of the input point cloud and the disordered input of the point cloud. The high-dimensional feature vector can be used as a key vector for most point cloud processing tasks. In the present invention, the feature vector is defined as a point cloud feature vector, which combines the local feature vector and the global feature vector.

[0072] The encoder structure designed by the present invention adds relative position information to the input point cloud, obtains more obvious point cloud feature data through the downsampling layer, reduces the complexity of the overall network, uses a multi-scale feature fusion algorithm, and uses a new downsampling algorithm with the attention mechanism as a score indicator. According to the output downsampling features, the global features of the point cloud and the local features of the point cloud are extracted respectively. The present invention sets the dimension of the global feature vector extraction to 512 and the dimension of the local feature vector to 128. After 4 layers of multi-head attention mechanisms and cascading the output of each multi-head attention mechanism, a residual structure, a data normalization layer and a ReLu activation layer are added to each multi-head attention layer, which can speed up the convergence of the network. A high-dimensional global feature vector F is obtained. global and the local eigenvector F local .

[0073] 1. For input, the input of a point cloud sample is Where N represents the number of inputs. Each input point cloud has a d-dimensional vector. Common d is 3 or 6. When d is 3, the point cloud only contains position information. When d is 6, the point cloud contains position information and normal vector information. p is the output of the relative position encoding layer, F e Yes F p The output feature vector after the downsampling layer is F o F e After the output of 4 layers of multi-head attention mechanism, F o The output feature vector formula is as follows:

[0074] MSA(A)=Attention(A,A,A),

[0075] F1=MSA(F e),

[0076] F i =MSA(F i-1 ), i = 2, 3, 4,

[0077]

[0078] Among them, each MSA (F i ) represents the output feature of the i-th attention layer, and the output dimension of each layer is consistent with the input dimension. o are the weights of the linear layer, which change during network training.

[0079] 2. In order to extract the global feature F global With local features F local Based on the point cloud features, the present invention designs a new feature fusion layer to replace the traditional symmetric function. The newly proposed fusion layer can still solve the disorder problem of point clouds, and can also greatly reduce the features lost by point clouds due to dimensionality reduction. The new fusion layer formula is as follows:

[0080] MCA(A,B)=Attention(A,B,B),

[0081] F lg =MCA(F local , F global ),

[0082] Step 5: Point cloud decoder structure. For different point cloud processing tasks, the structure of the decoder part is different due to the different output dimensions.

[0083] 1. Object classification task. The classification network is shown in the decoder part of the overall flowchart, see Figure 2 . The high-dimensional feature F lg Convert to N c There are N object catalog scores (such as airplanes, tables, and chairs) and they are labeled with numbers, for example, 0 represents the airplane category, 1 represents the table category, and 2 represents the chair category. c = 40 object categories. The present invention puts high-dimensional features into a preset classification decoder, which includes a convolutional network, two cascaded linear, normalized, ReLu activation function combination layers, and a dropout layer to prevent overfitting. Finally, a linear function is used to predict the final classification score The category corresponding to the sequence with the largest prediction score is the category predicted by the neural network this time.

[0084] 2. Object segmentation task. For object segmentation, the network needs to mark a label for each input point, for example, the table top part, the table leg part, we need to predict each point. Generally, each object category is divided into 2-4 parts, which are marked in sequence. There are a total of 20 object categories and 50 object segmentation areas. This invention uses high-dimensional features F lg In order to learn multiple parts of the common model, we convert the label into a one-dimensional target vector and a high-dimensional feature F lg Cascade. The subsequent object segmentation decoder is similar to the object classification decoder. It mainly uses convolution to extract features from high-dimensional information, uses two cascaded linear, normalized, and ReLu activation function combination layers for decoding, and uses a dropout layer to prevent overfitting. Finally, we can predict the segmentation score of each point Each point is identified as the label part corresponding to the highest score.

[0085] Step 6. Mix-Net network training method.

[0086] 1. Point cloud classification task and point cloud segmentation task dataset.

[0087] In the point cloud classification task, we use the ModelNet40 test set to verify our Mix-Net network. Each point cloud sample in this dataset contains 10,000 point cloud information and 40 object categories, including 9,843 training samples and 2,468 test samples. In the network training part, we performed data enhancement on the input point cloud, including range scaling with a scaling ratio of [0.8, 1.25], region translation with a translation range of [-0.1, 0.1], and random loss input with a loss ratio of [0, 12.5%]. During the experiment, we used N=1024, D=6(xyz+normal), local_dim=256, global_dim=512, and K=50. The results of the point cloud classification task are shown in the figure. Figure 4 As shown, the classification accuracy on the test set can reach 92.5%.

[0088] In the point cloud segmentation task, we use the very challenging ShapeNet dataset to verify our Mix-Net network. The dataset includes 13998 training sets and 2874 training sets. It contains 16 different categories, each category contains 2 to 4 part labels, and a total of 50 part labels. The segmentation goal is to classify the target for each input point. In the network training part, we performed data augmentation on the input point cloud, including range scaling, the scaling ratio is [0.8, 1.25], area translation, the translation range is [-0.1, 0.1], and also includes random loss input, the loss ratio is [0, 12.5%]. During the experiment, we used N = 1024, D = 6 (xyz + normal), local_dim = 256, global_dim = 512, K = 50. Some results of the point cloud segmentation task are shown as follows. Figure 5 As shown, the left picture is the marked point cloud, and the right picture is the neural network predicted point cloud.

[0089] 2. Other training details of Mix-Net: for point cloud classification tasks and point cloud segmentation tasks. The loss function uses the cross entropy loss function. In training, the stochastic gradient descent method is used as the most optimized method, and the momentum value is set to 0.9. The learning rate is set to 1e-4, batch_size=32, the GPU used is RTX-3070, and the neural network architecture used is pytorch.

[0090] In summary, the present invention proposes a classification and segmentation algorithm for point cloud processing, which can effectively and accurately realize the object classification and object segmentation tasks of point clouds. Compared with the commonly used algorithms, Mix-Net neural network is more suitable for point cloud processing. The permutation disorder and order disorder of point clouds can be better solved in the new feature extraction structure and the new fusion structure, reducing the overall parameter amount of the network and improving the accuracy of the overall output.

[0091] The above is a further detailed description of the present invention in combination with a specific implementation method, and it cannot be determined that the specific implementation of the present invention is limited to this. For technicians in the field of the patent to which the present invention belongs and related technical fields, based on the idea of ​​the technical solution of the patent of the present invention, the expansion and replacement of the operation method and structure should all fall within the scope of protection of the patent of the present invention.

Claims

1. A method for object classification and object segmentation based on 3D point cloud information, comprising the following steps: Step 1: Add position coding to the input point cloud information For the input point cloud information, the input relative position information is added. The relative position information is added in the first input layer and the attention function. After a large number of experiments, it can be proved that the addition of relative position encoding can improve network performance and improve the final output accuracy. Step 2: Data downsampling algorithm A downsampling algorithm is used for input to reduce the complexity of the overall network, speed up network training, and suppress overfitting. A downsampling algorithm based on an attention mechanism is used instead of the farthest point distance algorithm. The implementation process of the data downsampling algorithm is as follows: (1) Use an algorithm based on a multi-head attention mechanism to replace the traditional farthest point sampling algorithm The features of the convolution output are used through a multi-head attention mechanism to obtain an attention score. By selecting the first k maximum attention score values, an effect similar to collecting k values ​​at the farthest point is achieved. Experiments have shown that the downsampling algorithm used is more suitable for point clouds. Similar feature outputs can be obtained for a variety of different sequential inputs, which can solve the problem of point cloud disorder. (2) According to the obtained k maximum values, the determined sequence points are matched in the original point cloud input, and these sequence points are represented as key points. The key points represent the most critical parts of the point cloud. It can be easily seen from the key point visualization that the key points are the skeleton information of the point cloud; (3) Based on the selected key points, the k-means algorithm is used to select feature points around each key point, and all feature points that meet the k-means algorithm are obtained for feature extraction. Multi-layer features are fused using a multi-scale feature fusion method to output high-dimensional features after downsampling; Step 3: Multi-head attention mechanism for relative position information fusion As the main layer for extracting features in the network, a multi-head attention layer for fusing relative position information is used instead of a convolutional layer. Experiments show that the multi-head attention layer increases network performance and improves the final output accuracy. Step 4: Point cloud encoder structure design The designed point cloud encoder structure is used to extract high-dimensional feature vectors from input point cloud data. The feature vectors include global feature information of point cloud and local feature information of point cloud. When fusing features to obtain the overall features of point cloud, a new feature fusion mechanism is provided to replace the symmetric function. Step 5: Point cloud decoder structure design Different decoder structures are designed for different point cloud processing tasks, with similarities and differences; Step 6: Design Mix-Net network training method According to different point cloud processing tasks and for different data sets, loss functions, optimizers, and multiple training parameter indicators are designed.

2. The object classification and object segmentation method based on 3D point cloud information according to claim 1, characterized in that: The implementation process of the position encoding in step 1 is as follows: Since there are multiple storage methods for the input point cloud container, it is difficult to directly obtain the relative position information between points, that is, between different elements of the same input, in the network. Relative position information is introduced to increase the position feature extraction between points. The position encoding formula is as follows: Among them, P i is the input data or features, It is the average value of input data or features. MLP is a commonly used multi-layer perceptron. It is found in many experiments that the addition of position information in the first layer input and attention mechanism module has a significant improvement on the overall network performance.

3. The object classification and object segmentation method based on 3D point cloud information according to claim 1, characterized in that: The implementation process of the multi-head attention mechanism for relative position information fusion described in step 3 is as follows: (1) Using relative distance at the input can increase the overall performance of the network. After a large number of experiments, it can be seen that adding relative distance ρ to the attention function can still increase the overall network performance. Adding relative distance to the attention function, the formula is as follows: Among them, Q, K, V are query, key, value matrices obtained by inputting their respective change matrices, softmax is the activation function, ρ is the relative distance, a relative distance value is added at the transpose of K, and a relative distance value is added at V. The designed attention function extracts more sufficient information between elements in the input, which is more suitable for point cloud processing; (2) Introducing the multi-head attention mechanism: The input passes through multiple independent attention layers, and finally the output results of each layer are cascaded to obtain the output of the multi-head attention layer. By implementing parallel operation, there is a significant speed advantage in processing input data. The formula of the multi-head attention mechanism is as follows: F i =Attention(Q i ,K i ,V i ),i=1,...,h, MultiHead(Q,K,V)=Concat(F1,F2,...,F h )W o Where X represents the input, and There are three different sets of linear matrices. h represents the number of multi-head attention mechanisms. h = 4 is used. represents the output feature matrix, F i Represents the final output value of each multi-head attention layer. Concat represents cascade. The multi-head attention mechanism divides the input into h independent attention mechanism layers and operates each attention mechanism layer in parallel.

4. The object classification and object segmentation method based on 3D point cloud information according to claim 1, characterized in that: The implementation process of the point cloud encoder structure design described in step 4 is as follows: (1) For input, the input of a point cloud sample is Where N represents the number of inputs. Each input point has a d-dimensional vector, d is 3 or 6. When d is 3, the point cloud only contains position information. When d is 6, the point cloud contains position information and normal vector information. p is the output of the relative position encoding layer, F e Yes F p The output feature vector after the downsampling layer is F o F e After the output of 4 layers of multi-head attention mechanism, F o The output feature vector formula is as follows: MSA(A)=Attention(A,A,A), F1=MSA(F e ), F i =MSA(F i-1 ),i=2,3,4, F o =Concat(F1,F2,...,F4)·W O , Among them, each MSA (F i ) represents the output feature of the i-th attention layer. The output dimension of each layer is consistent with the input dimension. o are the weights of the linear layer, which change during network training: (2) In order to extract the global feature F global With local features F local Based on the point cloud features, a new feature fusion layer is designed to replace the traditional symmetric function. The newly proposed fusion layer can still solve the disorder problem of point clouds and greatly reduce the features lost by point clouds due to dimensionality reduction. The new fusion layer formula is as follows: MCA(A,B)=Attention(A,B,B), F lg =MCA(F local ,F global )。

Images