Three-dimensional model classification method based on point cloud position coding and multi-modal feature fusion

Abstract

The invention provides a three-dimensional model classification method based on point cloud position coding and multi-modal feature fusion. An existing three-dimensional model classification method has the problem of insufficient concern on the relation between points. In order to solve the problem, the invention provides a point cloud classification model which is combined with the relative position coding of the point cloud, the DRCNN (Dual-Ring Convolutional Neural Network) and the MVCNN (Multi-View Convolutional Neural Network). The invention further provides a method for classifying the point cloud according to the relative position coding of the point cloud, the DRCNN (Dual-Ring Convolutional Neural Network) and the MVCNN (Multi-View Convolutional Neural Network). According to the method, firstly, local neighborhood feature changes are sensed through the relative position codes of the point cloud, and then the DRCNN is input to obtain the global features of the point cloud. Then, multi-view features are extracted through the MVCNN to supplement point cloud representation; the method has the advantages that the correlation between the point cloud and multiple views is explored more effectively, multi-modal features with higher discrimination are generated, and the performance of three-dimensional model classification is improved through combination of the point cloud and the multiple views.

Description

Technical field:

[0001] The present invention relates to a three-dimensional model classification method based on point cloud position coding and multi-modal feature fusion, and the method has good application in the field of three-dimensional model classification. Background technology:

[0002] 3D model classification is an important task in computer vision and machine learning, and its goal is to classify 3D models to identify the category they belong to. With the rapid development of 3D sensing technology, the application of point clouds has become the main method in various industrial branches, and it is widely used in practical scenarios such as 3D modeling, forest dynamic observation, and autonomous driving. Therefore, the demand for understanding the shape of 3D point clouds is also increasing. Due to the improved availability and accuracy of LiDAR tools and photogrammetry software, it has become easier to obtain point clouds. Point clouds are composed of raw coordinates and are the simplest and basic representation of complex 3D scenes. Unlike the regular grid structure of 2D images, point clouds are a set of disordered spatial points and are not suitable for grid-based local convolution operations. Although deep learning networks such as CNN have achieved remarkable achievements in processing 2D images, it is not easy to directly transplant them to 3D point cloud analysis.

[0003] This article focuses on the classification task of point clouds. There are two problems in the process of point cloud classification. One is how to explore the inherent deep semantic relationship between each point in the point cloud model, and the other is how to selectively aggregate information in different neighborhoods of the point cloud. Therefore, the purpose of the network proposed in this article is to design a more sophisticated feature extractor and a more effective feature fusion module. The present invention proposes a point cloud classification model that combines the relative position encoding of point clouds, dual-ring dynamic graph convolution DRCNN (Dual-Ring Convolutional Neural Network) to extract three-dimensional features of point clouds, and MVCNN (Multi-view CNN) to extract multi-view features. First, the present invention proposes a feature encoding module based on the relative position relationship of point clouds, which is used to learn the semantic relationship between adjacent points of point clouds, combine spatial information to propose representative local features, and use relative position encoding to better understand the geometric structure and topological relationship in point clouds. In order to further aggregate the local features of the point cloud, the present invention proposes a dual-loop dynamic graph convolution DRCNN and a gated feature fusion module based on the attention mechanism, which can effectively extract local features with point cloud density information and aggregate them to obtain the global features of the point cloud. The gated propagation strategy can selectively filter out feature redundancy caused by the long distance between sampling points. In order to further improve the network's attention to the local information of the point cloud, the present invention introduces residual propagation of offset attention to capture the long-term dependency between points, which helps to alleviate the attenuation problem of information in the propagation process and avoid information loss or blurring. Then the original three-dimensional model is projected into a two-dimensional view, and the view features are extracted using MVCNN. The present invention proposes a multimodal fusion network of point clouds and views, which supplements the point cloud representation through multiple views. The network generates cross-modal correlation scores of point cloud and view features through an attention fusion module, and uses the correlation scores to filter out redundant information in the view. The filtered view features are aggregated, and the global features of the point cloud and view are fused. The advantage of the present invention is that, unlike the traditional method that treats each view equally, this method more effectively explores the correlation between point clouds and multiple views and generates more discriminative multimodal features. Point clouds provide structural information in space, while multi-views provide detailed information from different perspectives. The combination of the two can provide more comprehensive and richer information and improve the performance of 3D model classification. Summary of the invention:

[0004] In order to solve the problems existing in the field of three-dimensional model classification, the present invention discloses a three-dimensional model classification method based on point cloud position coding and multimodal feature fusion.

[0005] To this end, the present invention provides the following technical solutions:

[0006] 1. A three-dimensional model classification method based on point cloud position encoding and multimodal feature fusion, the method comprising the following steps:

[0007] Step 1: First, downsample the original point cloud to obtain the original point cloud dataset, input the point cloud dataset into the point cloud position encoding module for relative position encoding, and then project the point cloud to obtain a set of two-dimensional views. Each point cloud model is projected into 12 views to obtain a view dataset.

[0008] Step 2: In the training phase, the encoded point cloud features are input into the DRCNN network, and the view dataset is input into the MVCNN network. The outputs of the two network models are fused according to the correlation scores calculated by the attention fusion module, and the network parameters are optimized according to the category labels of the point clouds.

[0009] Step 3: Testing phase, i.e., the 3D model classification process, inputs the test data set into the optimized network model, calculates the probability distribution of the 3D model under different categories, and the category with the highest probability is the category of the 3D model.

[0010] 2. The three-dimensional model classification method based on point cloud position coding and multimodal feature fusion according to claim 1 is characterized in that, in the step 1, the original point cloud is first downsampled to obtain the original point cloud data set, the point cloud data set is input into the point cloud position coding module for relative position coding, and then the point cloud is projected to obtain a set of two-dimensional views, each point cloud model is projected into 12 views, a view data set is obtained, and the data set is divided into a training data set and a test data set, and the specific steps are:

[0011] Step 1-1 uses uniform sampling to downsample the original point cloud to obtain relatively uniform N points p1, p2, ..., p N , P=(p1,p2,…,p N )∈R N×3 As input feature F in , the high-dimensional feature F of the point cloud is obtained by embedding it into a higher-dimensional feature space through the MLP layer ch , as shown below:

[0012] F ch =MLP(P)

[0013] Step 1-2: original point cloud feature F in and high-dimensional point cloud features F ch Use the K nearest neighbor algorithm to group the neighborhood of the point cloud;

[0014] Step 1-2-1 The original point cloud feature F in Input K nearest neighbor algorithm to get feature F i,k ∈RN×K×3 ;

[0015] Step 1-2-2: High-dimensional feature F ch Input K nearest neighbor algorithm to get feature F c,k ∈R N×K×C 1, where C1 is the high-dimensional feature dimension;

[0016] Step 1-3 Feature F i,k Convert to relative position F' through trigonometric transformation i,k , as shown below:

[0017]

[0018]

[0019] Among them, α, β, δ are hyper parameters, δ = e -5 Ensure the stability of operation;

[0020] Steps 1-4 get the relative position coordinates F' i,k Take any point p in the neighborhood j =F' i,k,j =(x, y, z) to perform position encoding to obtain the relative position encoding F pos , as shown below:

[0021] F pos =PosEmb(p j )=(pos(x),pos(y),pos(z),x,y,z,dis(p))∈R C+4

[0022] Among them, (pos(x),pos(y),pos(z))∈R C Represents the point p j The angle encoding of the three-dimensional coordinate vector and the coordinate axis, C is the dimension of the angle encoding. Since the angles between the coordinates of different points and the coordinate axis may be the same, the coordinate features of the point and the distance dis(p) from the point to the origin of the coordinate are added on this basis. Taking pos(x) as an example, for different feature channels i∈[0,C / 6], it is as follows:

[0023]

[0024] Steps 1-5: Feature F c,k Input to the linear transformation module (LT) to get F' c,k , the transformation process is as follows:

[0025]

[0026] Where λ and μ are trainable parameters, δ = e-5 Ensure the stability of operation;

[0027] Step 1-6 F' c,k Input two layers of MLP to perform residual propagation to extract local features F h ;

[0028] F h =MLP(MLP(F′ c,k ))+F′ c,k

[0029] Among them, MLP is a multi-layer perceptron;

[0030] Step 1-7: Transform the position encoding matrix F pos With F h Multiply them together to calculate a high-dimensional feature matrix containing the location information of each point, and h After adding, the maximum pooling is performed to obtain the output feature F of the position encoding module out ;

[0031] F out =max((F pos ×F h )+F h )

[0032] Among them, max is the maximum pooling operation;

[0033] Steps 1-8 project the 3D model in Modelnet40 to obtain a 2D view set;

[0034] Step 1-8-1 Fix the 3D model at the center of the virtual sphere and select a fixed angle and amount for projection;

[0035] Step 1-8-2 places the virtual camera above the three-dimensional model, and the camera rotates a circle, and each step of 30 degrees projection obtains a set of two-dimensional views, which are recorded as V1, V2, V3, V4, V5, V6, V7, V8, V9, V10, V11 and V12;

[0036] 3. The three-dimensional model classification method based on point cloud position coding and multimodal feature fusion according to claim 1 is characterized in that in step 2, the encoded point cloud features are input into the DRCNN network, the view data set is input into the MVCNN network, the outputs of the two network models are fused according to the correlation scores calculated by the attention fusion module, and the network parameters are optimized according to the category labels of the point clouds. The specific process is as follows:

[0037] Step 2-1: The training model in ModelNet40 is processed by step 1 to obtain F out Input into the DRCNN model to obtain feature FD ;

[0038] Step 2-1-1 F out =(x1,x2,…,x n ) Choose two different K values ​​to construct two concentric neighborhoods to obtain F k1 、F k2 , where F k1 =((x 1,1 ,x 1,2 ,…,x 1,k1 ),…,(x n,1 ,x n,2 ,…,x n,k1 )), F k2 =((x 2,1 ,x 2,2 ,…,x 2,k2 ),…,(x n,1 ,x n,2 ,…,x n,k2 )), where x i is the feature vector of the i-th point in the point cloud, x i,j For a central point x i The feature vector of the jth neighboring point;

[0039] Step 2-1-2: F k1 and F k2 Input the double-ring feature extraction module to obtain the local feature F r1 、F r2 As shown below:

[0040] F r1 =f conv (F out ,F k1 )=Φ(Ψ([F out ,F k1 -F out ]))

[0041] F r2 =f conv (F out ,F k2 )=Φ(Ψ([F out ,F k2 -F out ]))

[0042] Among them, Φ represents the maximum pooling of aggregation function, and Ψ represents dynamic graph convolution;

[0043] Step 2-1-3: F r1 、F r2 Input to the attention gate feature fusion module to fuse the features obtained from different neighborhoods;

[0044] Step 2-1-3-1 Set F r1 、F r2 ∈R N×C Input two 1*1 convolutional layers to get F' r1 、F' r2 , F' r1 、F' r2 Put them together to get F r1,2 ∈R N×2C , as shown below:

[0045] F′ r1 =conv(F r1 ) F′ r2 =conv(F r2 )

[0046] F r1,2 =concat(F′ r1 ,F′ r2 )

[0047] Step 2-1-3-2: F r1,2 Input the ReLu activation function to get D, as shown below:

[0048] D = ReLU(F r1,2 )

[0049] Step 2-1-3-3 Input D into the convolutional layer to obtain the similarity matrix W, as shown below:

[0050] W = conv(D)

[0051] Among them, conv is a 1*1 convolutional layer, and W is the similarity matrix W∈R N×1 , where the i-th element w i ∈[0,1] represents the weight of the i-th point in the point cloud;

[0052] Step 2-1-3-4: F r1 Multiply it by the weight W to get F r1 The contribution F1 to the fused features is as follows:

[0053] F1=F r1 ×W

[0054] Step 2-1-3-5: F r2 Multiply it by the weight 1-W to get F r2 The contribution F2 to the fused features is as follows:

[0055] F2=F r2 ×(1-W)

[0056] Step 2-1-3-6 Add F1 and F2 to get the feature F after eliminating redundancy G , as shown below:

[0057] F G =F1+F2

[0058] Step 2-1-3-7: F G Perform linear transformations of Q, K, and V respectively as follows:

[0059] Q=W q ×F G

[0060] K=W k ×F G

[0061] V=W v ×F G

[0062] Among them, W q , W k , W v is the weight matrix;

[0063] Step 2-1-3-8 calculates the dot product of Q and K, and then performs a softmax operation to obtain the position attention weight A, as shown below:

[0064]

[0065] Among them, d is the dimension of the feature, which is used to scale the attention weight;

[0066] Step 2-1-3-9: Use the attention weight A to perform weighted summation on V to obtain F. sa , as shown below:

[0067] F sa =A·V

[0068] Step 2-1-3-10: F sa Subtract V and input it into the LBR layer to get F Δ , as shown below:

[0069] F Δ =LBR(F sa -V)

[0070] Among them, LBR is a linear layer, a normalization layer, and an activation layer;

[0071] Step 2-1-3-11 Change the feature F Δ 、F G and attention feature F sa Add together to get the global feature F o , as shown below:

[0072] F o =F Δ +F G +F sa

[0073] Step 2-1-3-12: F out =F o Repeat steps 2-1-1 to 2-1-3 for four cycles to obtain the local feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 , the local feature obtained in the i-th cycle is recorded as F i,r1 ;

[0074] Step 2-1-4 obtains the local neighborhood feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 The deep local features F of the point cloud are extracted by splicing together through the residual feature propagation module L ;

[0075] Step 2-1-4-1: Concatenate the local features output by each layer to obtain F c , as shown below:

[0076] F c =concat(F 1,r1 ,F 2,r1 ,F 3,r1 ,F 4,r1 )

[0077] Step 2-1-4-2: F c Input the convolutional layer to transform and get F' c , as shown below:

[0078] F′ c =conv(F c )

[0079] Among them, conv is a 1*1 convolutional layer;

[0080] Step 2-1-4-3 F' c Input group normalization layer to get F RG , as shown below:

[0081] F RG =RG(F′ c )

[0082] Among them, RG is ReLU plus GroupNormalization;

[0083] Step 2-1-4-4 F' c Perform linear transformations of Q, K, and V respectively as follows:

[0084] Q=W q ×F′ c

[0085] K=W k ×F′ c

[0086] V=W v ×F′ c

[0087] Among them, W q , W k , W v is the weight matrix;

[0088] Step 2-1-4-5 Calculate F' c The offset attention F OA , the calculation process is as follows:

[0089]

[0090] F OA = LBR(F′ c -F s )+F′ c

[0091] Among them, d is the feature dimension, F s is the self-attention feature matrix of the input features, and LBR is the linear layer, BN layer, and activation layer;

[0092] Step 2-1-4-6: F RG and F OA Add and normalize to get the local feature F of the point cloud L , as shown below:

[0093] F L =BN(F RG +F OA )

[0094] Among them, BN is BatchNormalization;

[0095] Step 2-1-4-7: F c =F L , repeat steps 2-1-4-1 to 2-1-4-6 to extract deep local features of the point cloud, and execute the cycle four times;

[0096] Step 2-1-5: The deep local features F of the point cloud L and the global feature F oSplice them together to get the complete point cloud feature representation F P , as shown below:

[0097] F P =concat(F o ,F L )

[0098] Step 2-1-6: F P Input convolutional layer to transform features to get F' P , as shown below:

[0099] F′ P =conv(F P )

[0100] Step 2-1-7 F' p Input pooling layer to get F D , as shown below:

[0101] F D =max(F′ P )+avg(F′ P )

[0102] Among them, max is the maximum pooling operation, and avg is the average pooling operation;

[0103] Step 2-2 Input the view dataset into the MVCNN network to obtain the view feature representation F of the point cloud V ;

[0104] Step 2-2-1 Combine the 12 views into a group to form a view feature representation F of the 3D model M , and adjusted to F' M , as shown below:

[0105] F′ M = reshape(F M )

[0106] Among them, reshape is the image resizing function to make it suitable for the input of the MVCNN network;

[0107] Step 2-2-2 F' M Input into the feature extraction module of MVCNN to obtain the deep semantic features F of the view of the 3D model. M , as shown below:

[0108] F″ M =AlexNet(F″ M )

[0109] Among them, AlexNet is a feature extractor;

[0110] Step 2-2-3: F” M The input pooling layer obtains the complete feature representation F of the 3D model projection into the view V , as shown below:

[0112] F V =max(F" M )

[0113] Among them, max is the maximum pooling operation;

[0114] Step 2-3: The point cloud representation F of the 3D model D and the view representation F V After concatenation, the correlation score F between the point cloud features and the view features is calculated through the convolution layer and the activation layer. C , as shown below:

[0115] F C =sig(conv(concat(F D ,F V ))

[0116] Among them, conv is 1*1 convolution, sig is the activation function;

[0117] Step 2-4 uses the correlation score to enhance the view representation of the point cloud to obtain the enhanced point cloud view feature F I , as shown below:

[0118] F I =(F V ×F C )+F V

[0119] Step 2-5: Fusion of point cloud features F of the 3D model D and view feature F I , and obtain the global feature representation F of the 3D model through the pooling layer J , as shown below:

[0120] F J =max(conv(concat(F I ,F D )))

[0121] Among them, conv is 1*1 convolution, and max is the maximum pooling operation;

[0122] Step 2-6: Find the 2, 3, and 4 views with the highest relevance scores. i With point cloud feature F D Fusion is performed in the above manner to obtain the hierarchical representation F of the 3D model S ;

[0123] F S2 =max(conv(concat(F2,F D )))

[0124] F S3 =max(conv(concat(F3,F D )))

[0125] F S4 =max(conv(concat(F4,F D )))

[0126] Among them, F i (i=2,3,4) indicates the i view features with the highest relevance scores;

[0127] Steps 2-7 will represent the layered representation F s2 、F s3 、F s4 Take the average and combine it with the global feature representation F of the 3D model J Concatenate them together to get the final feature representation F of the point cloud A , as shown below:

[0128] F A =concat(mean(F S2 ,F s3 ,F s4 ),F J )

[0129] Step 2-8 F A Input the fully connected layer to obtain the feature vector U of the three-dimensional model;

[0130] Step 2-9 Input the feature vector U into the softmax function to calculate the 3D model in the semantic category C i The predicted probability P under (i=1,2,…,n) i , as shown below:

[0131]

[0132] Among them, softmax(C i |U) is a three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U;

[0133] Step 2-10 uses the cross entropy loss function to calculate the loss as follows:

[0134]

[0135] Among them, Y iis the unique hot encoding of the true label of the 3D model, log is the natural logarithm, and N is the total number of 3D training models;

[0136] Step 2-11 is to back propagate the error loss and update the parameters layer by layer. The parameter update process is as follows:

[0137]

[0138] Among them, θ represents the parameter set, θ' represents the updated parameter set, and α is the learning rate;

[0139] Step 2-12 repeats step 2-1 to step 2-11 until the number of training times is reached to obtain an optimized three-dimensional model classification network.

[0140] 4. The three-dimensional model classification method based on point cloud position coding and multimodal feature fusion according to claim 1 is characterized in that in step 3, the test set data is classified using the optimized network model, and the specific process is:

[0141] Step 3-1 The test model in ModelNet40 is processed in step 1 to obtain F out Input into the DRCNN model to obtain feature F D ;

[0142] Step 3-1-1 F out =(x1,x2,…,x n ) Choose two different K values ​​to construct two concentric neighborhoods to obtain F k1 、F k2 , where F k1 =((x 1,1 ,x 1,2 ,…,x 1,k1 ),…,(x n,1 ,x n,2 ,…,x n,k1 )), F k2 =((x 2,1 ,x 2,2 ,…,x 2,k2 ),…,(x n,1 ,x n,2 ,…,x n,k2 )), where x i is the feature vector of the i-th point in the point cloud, x i,j For a central point x i The feature vector of the jth neighboring point;

[0143] Step 3-1-2 F k1 and F k2 Input the double-ring feature extraction module to obtain the local feature F r1、F r2 As shown below:

[0144] F r1 =f conv (F out ,F k1 )=Φ(Ψ([F out ,F k1 -F out ]))

[0145] F r2 =f conv (F out ,F k2 )=Φ(Ψ([F out ,F k2 -F out ]))

[0146] Among them, Φ represents the maximum pooling of aggregation function, and Ψ represents dynamic graph convolution;

[0147] Step 3-1-3: F r1 、F r2 Input to the attention gate feature fusion module to fuse the features obtained from different neighborhoods;

[0148] Step 3-1-3-1 Set F r1 、F r2 ∈R N×C Input two 1*1 convolutional layers to get F' r1 、F' r2 , F' r1 、F' r2 Put them together to get F r1,2 ∈R N×2C , as shown below:

[0149] F′ r1 =conv(F r1 ) F′ r2 =conv(F r2 )

[0150] F r1,2 =concat(F′ r1 ,F′ r2 )

[0151] Step 3-1-3-2: F r1,2 Input the ReLu activation function to get D, as shown below:

[0152] D = ReLU(F r1,2 )

[0153] Step 3-1-3-3 Input D into the convolutional layer to obtain the similarity matrix W, as shown below:

[0154] W = conv(D)

[0155] Among them, conv is a 1*1 convolutional layer, and W is the similarity matrix W∈R N×1 , where the i-th element w i ∈[0,1] represents the weight of the i-th point in the point cloud;

[0156] Step 3-1-3-4: F r1 Multiply it by the weight W to get F r1 The contribution F1 to the fused features is as follows:

[0157] F1=F r1 ×W

[0158] Step 3-1-3-5: F r2 Multiply it by the weight 1-W to get F r2 The contribution F2 to the fused features is as follows:

[0159] F2=F r2 ×(1-W)

[0160] Step 3-1-3-6 Add F1 and F2 to get the feature F after eliminating redundancy G , as shown below:

[0161] F G =F1+F2

[0162] Step 3-1-3-7: F G Perform linear transformations of Q, K, and V respectively as follows:

[0163] Q=W q ×F G

[0164] K=W k ×F G

[0165] V=W v ×F G

[0166] Among them, W q , W k , W v is the weight matrix;

[0167] Step 3-1-3-8 calculates the dot product of Q and K, and then performs a softmax operation to obtain the position attention weight A, as shown below:

[0168]

[0169] Among them, d is the dimension of the feature, which is used to scale the attention weight;

[0170] Step 3-1-3-9: Use the attention weight A to perform weighted summation on V to obtain F. sa , as shown below:

[0171] F sa =A·V

[0172] Step 3-1-3-10: F sa Subtract V and input it into the LBR layer to get F Δ , as shown below:

[0173] F Δ =LBR(F sa -V)

[0174] Among them, LBR is a linear layer, a normalization layer, and an activation layer;

[0175] Step 3-1-3-11 Change the feature F Δ 、F G and attention feature F sa Add together to get the global feature F o , as shown below:

[0176] F o =F Δ +F G +F sa

[0177] Step 3-1-3-12: F out =F o Repeat steps 3-1-1 to 3-1-3 for four cycles to obtain the local feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 , the local feature obtained in the i-th cycle is recorded as F i,r1 ;

[0178] Step 3-1-4 obtains the local neighborhood feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 The deep local features F of the point cloud are extracted by splicing together through the residual feature propagation module L ;

[0179] Step 3-1-4-1: Concatenate the local features output by each layer to obtain F c , as shown below:

[0180] F c =concat(F 1,r1 ,F2,r1 ,F 3,r1 ,F 4,r1 )

[0181] Step 3-1-4-2: F c Input the convolutional layer to transform and get F' c , as shown below:

[0182] F′ c =conv(F c )

[0183] Among them, conv is a 1*1 convolutional layer;

[0184] Step 3-1-4-3 F' c Input group normalization layer to get F RG , as shown below:

[0185] F RG =RG(F′ c )

[0186] Among them, RG is ReLU plus GroupNormalization;

[0187] Step 3-1-4-4 F' c Perform linear transformations of Q, K, and V respectively as follows:

[0188] Q=W q ×F′ c

[0189] K=W k ×F′ c

[0190] V=W v ×F′ c

[0191] Among them, W q , W k , W v is the weight matrix;

[0192] Step 3-1-4-5 Calculate F' c The offset attention F OA , the calculation process is as follows:

[0193]

[0194] F OA = LBR(F′ c -F s )+F′ c

[0195] Among them, d is the feature dimension, Fs is the self-attention feature matrix of the input features, and LBR is the linear layer, BN layer, and activation layer;

[0196] Step 3-1-4-6: F RG and F OA Add and normalize to get the local feature F of the point cloud L , as shown below:

[0197] F L =BN(F RG +F OA )

[0198] Among them, BN is BatchNormalization;

[0199] Step 3-1-4-7: F c =F L , repeat steps 3-1-4-1 to 3-1-4-6 to extract deep local features of the point cloud, and execute the cycle four times;

[0200] Step 3-1-5: The deep local features F of the point cloud L and the global feature F o Splice them together to get the complete point cloud feature representation F P , as shown below:

[0201] F P =concat(F o ,F L )

[0202] Step 3-1-6: F P Input convolutional layer to transform features to get F' P , as shown below:

[0203] F′ P =conv(F P )

[0204] Step 3-1-7 F' p Input pooling layer to get F D , as shown below:

[0205] F D =max(F′ P )+avg(F′ P )

[0206] Among them, max is the maximum pooling operation, and avg is the average pooling operation;

[0207] Step 3-2 Input the view dataset into the MVCNN network to obtain the view feature representation F of the point cloud V ;

[0208] Step 3-2-1 Combine the 12 views into a group to form a view feature representation F of the 3D model M , and adjusted to F' M , as shown below:

[0209] F′ M = reshape(F M )

[0210] Among them, reshape is the image resizing function to make it suitable for the input of the MVCNN network;

[0211] Step 3-2-2 F' M Input into the feature extraction module of MVCNN to obtain the deep semantic features F of the view of the 3D model. M , as shown below:

[0212] F″ M =AlexNet(F′ M )

[0213] Among them, AlexNet is a feature extractor;

[0214] Step 3-2-3 Place F” M The input pooling layer obtains the complete feature representation F of the 3D model projection into the view V , as shown below:

[0215] F V =max(F″ M )

[0216] Among them, max is the maximum pooling operation;

[0217] Step 3-3 The point cloud representation F of the 3D model D and the view representation F V After concatenation, the correlation score F between the point cloud features and the view features is calculated through the convolution layer and the activation layer. C , as shown below:

[0218] F C =sig(conv(concat(F D ,F V ))

[0219] Among them, conv is 1*1 convolution, sig is the activation function;

[0220] Step 3-4 uses the correlation score to enhance the view representation of the point cloud to obtain the enhanced point cloud view feature F I , as shown below:

[0221] FI =(F V ×F C )+F V

[0222] Step 3-5: Fusion of point cloud features F of the 3D model D and view feature F I , and obtain the global feature representation F of the 3D model through the pooling layer J , as shown below:

[0223] F J =max(conv(concat(F I ,F D )))

[0224] Among them, conv is 1*1 convolution, and max is the maximum pooling operation;

[0225] Step 3-6: Find the 2, 3, and 4 views with the highest relevance scores. i With point cloud feature F D Fusion is performed in the above manner to obtain the hierarchical representation F of the 3D model S ;

[0226] F S2 =max(conv(concat(F2,F D )))

[0227] F S3 =max(conv(concat(F3,F D )))

[0228] F S4 =max(conv(concat(F4,F D )))

[0229] Among them, F i (i=2,3,4) indicates the i view features with the highest relevance scores;

[0230] Steps 3-7 will represent the layered representation F s2 、F s3 、F s4 Take the average and combine it with the global feature representation F of the 3D model J Concatenate them together to get the final feature representation F of the point cloud A , as shown below:

[0231] F A =concat(mean(F S2 ,F s3 ,F s4 ),F J )

[0232] Step 3-8: F A Input the fully connected layer to obtain the feature vector U of the three-dimensional model;

[0233] Step 3-9 Input the feature vector U into the softmax function to calculate the 3D model in the semantic category C i The predicted probability P under (i=1,2,…,n) i , as shown below:

[0234]

[0235] Among them, softmax(C i |U) is a three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U;

[0236] Step 3-10 inputs the feature vector U into the softmax function to calculate the semantic category C of the three-dimensional model, as shown below:

[0237]

[0238] Among them, softmax(C i |U) is the three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U.

[0239] Beneficial effects:

[0240] The present invention is a three-dimensional model classification method based on point cloud position coding and multi-modal feature fusion.

[0241] 1. This paper proposes a new network model and a multimodal fusion network of point cloud and view, which supplements the point cloud representation through multi-view. Different from the traditional method that treats each view equally, this method more effectively explores the correlation between point cloud and multi-view and generates more discriminative multimodal features.

[0242] 2. The present invention proposes a point cloud relative position coding module. Relative position coding can provide additional information for each point to help enrich the representation of the point cloud. In addition to the coordinates and features of the point, relative position coding provides the relationship between the point and the surrounding points, thereby enhancing the representation ability of the point cloud.

[0243] 3. The present invention proposes a global-local feature extraction module, which includes dual-loop dynamic graph convolution and attention-gated feature fusion. This module enables the network to better adapt to the multi-scale feature distribution of point clouds, pays more attention to important features during the feature fusion process, and thus improves the model's attention to key information.

[0244] 4. The present invention introduces a feature residual propagation module of offset attention. Feature residual propagation helps to alleviate the problem of information attenuation in the propagation process by introducing residual connections in each layer. The introduction of offset attention can dynamically adjust the attention weight according to the characteristics of its neighboring nodes, thereby more effectively capturing the complex relationship between nodes.

[0245] 5. The present invention generates cross-modal correlation scores of point cloud and view features through an attention fusion module, and uses the correlation scores to filter out redundant information in the view. The filtered view features are aggregated, and the point cloud and view features are fused. Description of the drawings:

[0246] Figure 1 is a three-dimensional model to be classified in an embodiment of the present invention;

[0247] Figure 2 It is a flow chart of a three-dimensional model classification method based on point cloud position coding and multimodal feature fusion in an embodiment of the present invention;

[0248] Figure 3 The point cloud relative position encoding process of the three-dimensional model classification method based on point cloud position encoding and multimodal feature fusion in the embodiment of the present invention;

[0249] Figure 4 The attention gate feature fusion process of the three-dimensional model classification method based on point cloud position encoding and multimodal feature fusion in the embodiment of the present invention; Specific implementation method:

[0250] In order to clearly and completely describe the technical solutions in the embodiments of the present invention, the present invention is further described in detail below in conjunction with the drawings in the embodiments.

[0251] Take the 3D model airplane_0630 in the test set airplane class of the 3D model public dataset ModelNet40 as an example. Figure 1 The present invention is further described in detail.

[0252] The process of the three-dimensional model classification method based on point cloud position coding and multimodal feature fusion in the embodiment of the present invention is as follows: Figure 2 As shown, the following steps are included:

[0253] Step 1: First, downsample the original point cloud to obtain the original point cloud dataset, input the point cloud dataset into the point cloud position encoding module for relative position encoding, and then project the point cloud to obtain a set of two-dimensional views. Each point cloud model is projected into 12 views to obtain a view dataset.

[0254] Step 1-1 uses uniform sampling to downsample the original point cloud to obtain relatively uniform N points p1, p2, ..., p N , P=(p1,p2,…,p N )∈R N×3 As input feature F in , the high-dimensional feature F of the point cloud is obtained by embedding it into a higher-dimensional feature space through the MLP layer ch , as shown below:

[0255] F ch =MLP(P)

[0256] Figure 1 The high-dimensional features F of the three-dimensional model shown ch The calculation results are as follows:

[0257] P=F in =tensor([[[0.0710,-0.0285,-0.1019],[-0.5226,0.1267,-0.9449],[-0.4338,0.0814,0.7838],[0.2995,-0.1278,0.4642],[0.2928,-0 .0828,0.5213],[0.2773,-0.1069,-0.0097]],...,[-0.4083,-0.350 9,-0.0912],[0.3391,0.0816,0.5903],[0.1958,0.5988,-0.2142]]])

[0258] F ch=tensor([[[-2.0265e-02,2.2427e+00,4.7888e-01,...,-2.4620e-01,-2.4921e-01,-1.4994e-01],[-9.4325e-03,-1.6664e-01,1.364 0e-01,...,2.9525e-01,4.2975e-01,5.2598e-02],[3.5829e-01,1.2525e+00,-4.7227e-01,...,-8.7194e-02,-1.3221e-01,5.3969e-01 ],...,[-1.8939e-01,1.0313e+00,-2.2520e-03,...,2.9338e-01,3.8961e-01,-7.4574e-03],[-7.8142e-02,-3.4311e-02,1.8382e+00 ,...,-2.3231e-01,-2.9907e-03,1.5383e+00],[5.2619e-01,4.0073e-01,-4.2537e-01,...,1.0654e+00,3.0078e-01,-3.4815e-01]]])

[0259] Step 1-2: original point cloud feature F in and high-dimensional point cloud features F ch Use the K nearest neighbor algorithm to group the neighborhood of the point cloud;

[0260] Step 1-2-1 The original point cloud feature F in Input K nearest neighbor algorithm to get feature F i,k ∈R N×K×3 ;

[0261] Step 1-2-2: High-dimensional feature F ch Input K nearest neighbor algorithm to get feature F c,k ∈R N×K×C 1, where C1 is the high-dimensional feature dimension;

[0262] Step 1-3 Original point cloud F in The feature F obtained after grouping i,k Convert to relative position F' through trigonometric transformation i,k , as shown below:

[0263]

[0264] Among them, α, β, δ are hyper parameters, δ = e -5 Ensure the stability of operation;

[0265] Steps 1-4 get the relative position coordinates F' i,k Take any point p in the neighborhood i =F' i,k,j =(x,y,z) to get position encoding F pos , the relative position encoding is as follows:

[0266] F pos =PosEmb(p i )=(pos(x),pos(y),pos(z),x,y,z,dis(p))∈R C+4

[0267] where (pos(x),pos(y),pos(z))∈R C Represents the point p i The angle encoding of the three-dimensional coordinate vector and the coordinate axis, C is the dimension of the angle encoding. Since the angles between the coordinates of different points and the coordinate axis may be the same, the coordinate features of the point and the distance dis(p) from the point to the origin of the coordinate are added on this basis. Taking pos(x) as an example, for different feature channels i∈[0,C / 6], it is as follows:

[0268]

[0269] Steps 1-5: Feature F c,k Input to the linear transformation module (LT) to get F' c,k , the transformation process is as follows:

[0270]

[0271] Where λ and μ are trainable parameters, δ = e -5 Ensure the stability of operation;

[0272] Step 1-6 F' c,k Input two layers of MLP to perform residual propagation to extract local features F h ;

[0273] F h =MLP(MLP(F′ c,k ))+F′ c,k

[0274] Among them, MLP is a multi-layer perceptron;

[0275] Step 1-7: Transform the position encoding matrix F pos With F h Multiply them together to calculate a high-dimensional feature matrix containing the location information of each point, and h After adding, the maximum pooling is performed to obtain the output feature F of the position encoding module out ;

[0276] F out =max((F pos ×F h )+F h )

[0277] Among them, max is the maximum pooling operation,

[0278] F out =tensor([[[6.5426,3.2026,3.7978,...,3.4890,5.0449,4.0300],[5.2412,3.7259,2.9 002,...,4.1027,3.5450,4.9171],[5.9430,5.2097,5.1014,...,3.9943,3.2594,5.4150 ],...,[6.1941,4.4709,4.9885,...,4.4212,4.7609,4.7937],[5.2989,5.1901,5.0815,...,4.2212,2.8114,5.1066],[3.8876,5.6192,5.1170,...,3.9272,7.3280,4.7424]]])

[0279] Steps 1-8 project the 3D model in Modelnet40 to obtain a 2D view set;

[0280] Step 1-8-1 Fix the 3D model at the center of the virtual sphere and select a fixed angle and amount for projection;

[0281] Step 1-8-2 places the virtual camera above the three-dimensional model, and the camera rotates a circle, and each step of 30 degrees projection obtains a set of two-dimensional views, which are recorded as V1, V2, V3, V4, V5, V6, V7, V8, V9, V10, V11 and V12;

[0282] In step 2, during the training phase, the encoded point cloud features are input into the DRCNN network, and the view dataset is input into the MVCNN network. The outputs of the two network models are fused according to the correlation scores calculated by the attention fusion module, and the network parameters are optimized according to the category labels of the point clouds.

[0283] Step 2-1: The training data set in ModelNet40 is processed in step 1 to obtain F out Input into the DRCNN model to obtain feature F D ;

[0284] Step 2-1-1 F out =(x1,x2,…,xn ) Choose two different K values ​​to construct two concentric neighborhoods to obtain F k1 、F k2 , where F k1 =((x 1,1 ,x 1,2 ,…,x 1,k1 ),…,(x n,1 ,x n,2 ,…,x n,k1 )), F k2 =((x 2,1 ,x 2,2 ,…,x 2,k2 ),…,(x n,1 ,x n,2 ,…,x n,k2 )), where x i is the feature vector of the i-th point in the point cloud, x i,j For a central point x i The feature vector of the jth neighboring point;

[0285] Step 2-1-2: F k1 and F k2 Input the double-ring feature extraction module to obtain the local feature F r1 、F r2 As shown below:

[0286] F r1 =f conv (F out ,F k1 )=Φ(Ψ([F out ,F k1 -F out ]))

[0287] F r2 =f conv (F out ,F k2 )=Φ(Ψ([F out ,F k2 -F out ]))

[0288] Among them, Φ represents the maximum pooling of aggregation function, and Ψ represents dynamic graph convolution;

[0289] Step 2-1-3: F r1 、F r2 Input to the attention gate feature fusion module to fuse the features obtained from different neighborhoods;

[0290] Step 2-1-3-1 Set F r1 、F r2 ∈R N×CInput two 1*1 convolutional layers to get F' r1 、F' r2 , F' r1 、F' r2 Put them together to get F r1,2 ∈R N×2C , as shown below:

[0291] F′ r1 =conv(F r1 ) F′ r2 =conv(F r2 )

[0292] F r1,2 =concat(F′ r1 ,F′ r2 )

[0293] Step 2-1-3-2: F r1,2 Input the ReLu activation function to get D, as shown below:

[0294] D = ReLU(F r1,2 )

[0295] Step 2-1-3-3 Input D into the convolutional layer to obtain the similarity matrix W, as shown below:

[0296] W = conv(D)

[0297] Among them, conv is a 1*1 convolutional layer, and W is the similarity matrix W∈R N×1 , where the i-th element w i ∈[0,1] represents the weight of the i-th point in the point cloud;

[0298] Step 2-1-3-4: F r1 Multiply it by the weight W to get F r1 The contribution F1 to the fused features is as follows:

[0299] F1=F r1 ×W

[0300] Step 2-1-3-5: F r2 Multiply it by the weight 1-W to get F r2 The contribution F2 to the fused features is as follows:

[0301] F2=F r2 ×(1-W)

[0302] Step 2-1-3-6 Add F1 and F2 to get the feature F after eliminating redundancy G , as shown below:

[0303] F G=F1+F2

[0304] Step 2-1-3-7: F G Perform linear transformations of Q, K, and V respectively as follows:

[0305] Q=W q ×F G

[0306] K=W k ×F G

[0307] V=W v ×F G

[0308] Among them, W q , W k , W v is the weight matrix;

[0309] Step 2-1-3-8 calculates the dot product of Q and K, and then performs a softmax operation to obtain the position attention weight A, as shown below:

[0310]

[0311] Among them, d is the dimension of the feature, which is used to scale the attention weight;

[0312] Step 2-1-3-9: Use the attention weight A to perform weighted summation on V to obtain F. sa , as shown below:

[0313] F sa =A·V

[0314] Step 2-1-3-10: F sa Subtract V and input it into the LBR layer to get F Δ , as shown below:

[0315] F Δ =LBR(F sa -V)

[0316] Among them, LBR is a linear layer, a normalization layer, and an activation layer;

[0317] Step 2-1-3-11 Change the feature F Δ 、F G and attention feature F sa Add together to get the global feature F o , as shown below:

[0318] F o =F Δ +F G +F sa

[0319] Step 2-1-3-12: F out =F o Repeat steps 2-1-1 to 2-1-3 for four cycles to obtain the local feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 , the local feature obtained in the i-th cycle is recorded as F i,r1 ;

[0320] Step 2-1-4 obtains the local neighborhood feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 The deep local features F of the point cloud are extracted by splicing together through the residual feature propagation module L ;

[0321] Step 2-1-4-1: Concatenate the local features output by each layer to obtain F c , as shown below:

[0322] F c =concat(F 1,r1 ,F 2,r1 ,F 3,r1 ,F 4,r1 )

[0323] Step 2-1-4-2: F c Input the convolutional layer to transform and get F' c , as shown below:

[0324] F′ c =conv(F c )

[0325] Among them, conv is a 1*1 convolutional layer;

[0326] Step 2-1-4-3 F' c Input group normalization layer to get F RG , as shown below:

[0327] F RG =RG(F′ c )

[0328] Among them, RG is ReLU plus GroupNormalization;

[0329] Step 2-1-4-4 F' c Perform linear transformations of Q, K, and V respectively as follows:

[0330] Q=W q×F′ c

[0331] K=W k ×F′ c

[0332] V=W v ×F′ c

[0333] Among them, W q , W k , W v is the weight matrix;

[0334] Step 2-1-4-5 Calculate F' c The offset attention F OA , the calculation process is as follows:

[0335]

[0336] F OA = LBR(F′ c -F s )+F′ c

[0337] Among them, d is the feature dimension, F s is the self-attention feature matrix of the input features, and LBR is the linear layer, BN layer, and activation layer;

[0338] Step 2-1-4-6: F RG and F OA Add and normalize to get the local feature F of the point cloud L , as shown below:

[0339] F L =BN(F RG +F OA )

[0340] Among them, BN is BatchNormalization;

[0341] Step 2-1-4-7: F c =F L , repeat steps 2-1-4-1 to 2-1-4-6 to extract deep local features of the point cloud, and execute the cycle four times;

[0342] Step 2-1-5: The deep local features F of the point cloud L and the global feature F o Splice them together to get the complete point cloud feature representation F P , as shown below:

[0343] F P =concat(Fo ,F L )

[0344] Step 2-1-6: F P Input convolutional layer to transform features to get F' P , as shown below:

[0345] F′ P =conv(F P )

[0346] Step 2-1-7 F' p Input pooling layer to get F D , as shown below:

[0347] F D =max(F′ P )+avg(F′ P )

[0348] Among them, max is the maximum pooling operation, and avg is the average pooling operation;

[0349] Step 2-2 Input the view dataset into the MVCNN network to obtain the view feature representation F of the point cloud V ;

[0350] Step 2-2-1 Combine the 12 views into a group to form a view feature representation F of the 3D model M , and adjusted to F' M , as shown below:

[0351] F′ M = reshape(F M )

[0352] Among them, reshape is the image resizing function to make it suitable for the input of the MVCNN network;

[0353] Step 2-2-2 F' M Input into the feature extraction module of MVCNN to obtain the deep semantic features F of the view of the 3D model. M , as shown below:

[0354] F″ M =AlexNet(F′ M )

[0355] Among them, AlexNet is a feature extractor;

[0356] Step 2-2-3: F” M The input pooling layer obtains the complete feature representation F of the 3D model projection into the view V , as shown below:

[0357] F V =max(F″ M )

[0358] Among them, max is the maximum pooling operation;

[0359] Step 2-3: The point cloud representation F of the 3D model D and the view representation F V After concatenation, the correlation score F between the point cloud features and the view features is calculated through the convolution layer and the activation layer. C , as shown below:

[0360] F C =sig(conv(concat(F D ,F V ))

[0361] Among them, conv is 1*1 convolution, sig is the activation function;

[0362] Step 2-4 uses the correlation score to enhance the view representation of the point cloud to obtain the enhanced point cloud view feature F I , as shown below:

[0363] F I =(F V ×F C )+F V

[0364] Step 2-5: Fusion of point cloud features F of the 3D model D and view feature F I , and obtain the global feature representation F of the 3D model through the pooling layer J , as shown below:

[0365] F J =max(conv(concat(F I ,F D )))

[0366] Among them, conv is 1*1 convolution, and max is the maximum pooling operation;

[0367] Step 2-6: Find the 2, 3, and 4 views with the highest relevance scores. i With point cloud feature F D Fusion is performed in the above manner to obtain the hierarchical representation F of the 3D model S ;

[0368] F S2 =max(conv(concat(F2,F D )))

[0369] F S3=max(conv(concat(F3,F D )))

[0370] F S4 =max(conv(concat(F4,F D )))

[0371] Among them, F i (i=2,3,4) indicates the i view features with the highest relevance scores;

[0372] Steps 2-7 will represent the layered representation F s2 、F s3 、F s4 Take the average and combine it with the global feature representation F of the 3D model J Concatenate them together to get the final feature representation F of the point cloud A , as shown below:

[0373] F A =concat(mean(F S2 ,F s3 ,F s4 ),F J )

[0374] Step 2-8 F A Input the fully connected layer to obtain the feature vector U of the three-dimensional model;

[0375] Step 2-9 Input the feature vector U into the softmax function to calculate the 3D model in the semantic category C i The predicted probability P under (i=1,2,…,n) i , as shown below:

[0376]

[0377] Among them, softmax(C i |U) is a three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U;

[0378] Step 2-10 uses the cross entropy loss function to calculate the loss as follows:

[0379]

[0380] Among them, Y i is the unique hot encoding of the true label of the 3D model, log is the natural logarithm, and N is the total number of 3D training models;

[0381] Step 2-11 is to back propagate the error loss and update the parameters layer by layer. The parameter update process is as follows:

[0382]

[0383] Among them, θ represents the parameter set, θ' represents the updated parameter set, and α is the learning rate;

[0384] Step 2-12 repeats step 2-1 to step 2-11 until the number of training times is reached to obtain an optimized three-dimensional model classification network.

[0385] Step 3: Testing phase, i.e., the 3D model classification process, inputs the test data into the optimized network model, calculates the probability distribution of the 3D model under different categories, and the category with the highest probability is the category of the 3D model.

[0386] Step 3-1 The F out Input into the DRCNN model to obtain feature F D ;

[0387] Step 3-1-1F out =(x1,x2,…,x n ) Choose two different K values ​​to construct two concentric neighborhoods to obtain F k1 、F k2 , where F k1 =((x 1,1 ,x 1,2 ,…,x 1,k1 ),…,(x n,1 ,x n,2 ,…,x n,k1 )), F k2 =((x 2,1 ,x 2,2 ,…,x 2,k2 ),…,(x n,1 ,x n,2 ,…,x n,k2 )), where x i is the feature vector of the i-th point in the point cloud, x i,j For a central point x i The feature vector of the jth neighboring point;

[0388] Step 3-1-2 F k1 and F k2 Input the double-ring feature extraction module to obtain the local feature F r1 、F r2 As shown below:

[0389] F r1 =f conv (F out ,Fk1 )=Φ(Ψ([F out ,F k1 -F out ]))

[0390] F r2 =f conv (F out ,F k2 )=Φ(Ψ([F out ,F k2 -F out ]))

[0391] Among them, Φ represents the maximum pooling of aggregation function, and Ψ represents dynamic graph convolution;

[0392] F r1 =tensor([[[9.6227e-01,2.2891e+00,2.5058e+00,...,5.2719e-01,4.5935e-01,1.6420e-01],[1.0753e+00,1.1293e+00,1.1137e +00,...,2.1280e-01,-1.6072e-01,2.5352e-02],[1.8574e+00,1.9097e+00,1.4023e+00,...,8.8935e-01,1.0460e+00,-4.9903e-0 2],...,[1.2814e+00,2.1996e+00,1.0460e+00,...,4.4440e-01,1.3023e+00,2.1835e-01],[1.8137e+00,5.8721e-01,1.3135e+00 ,...,2.4003e-01,3.1433e-01,6.2611e-01],[2.0160e+00,9.5893e-01,1.4190e+00,...,8.6176e-01,7.2697e-01,6.0408e-01]]])

[0393] Step 3-1-3: F r1 、F r2 Input to the attention gate feature fusion module to fuse the features obtained from different neighborhoods;

[0394] Step 3-1-3-1 Set F r1 、F r2 ∈R N×C Input two 1*1 convolutional layers to get F' r1 、F' r2 , F' r1 、F' r2 Put them together to get Fr1,2 ∈R N×2C , as shown below:

[0395] F′ r1 =conv(F r1 ) F′ r2 =conv(F r2 )

[0396] F r1,2 =concat(F′ r1 ,F′ r2 )

[0397] Step 3-1-3-2: F r1,2 Input the ReLu activation function to get D, as shown below:

[0398] D = ReLU(F r1,2 )

[0399] Step 3-1-3-3 Input D into the convolutional layer to obtain the similarity matrix W, as shown below:

[0400] W = conv(D)

[0401] Among them, conv is a 1*1 convolutional layer, and W is the similarity matrix W∈R N×1 , where the i-th element w i ∈[0,1] represents the weight of the i-th point in the point cloud;

[0402] Step 3-1-3-4: F r1 Multiply it by the weight W to get F r1 The contribution F1 to the fused features is as follows:

[0403] F1=F r1 ×W

[0404] Step 3-1-3-5: F r2 Multiply it by the weight 1-W to get F r2 The contribution F2 to the fused features is as follows:

[0405] F2=F r2 ×(1-W)

[0406] Step 3-1-3-6 Add F1 and F2 to get the feature F after eliminating redundancy G , as shown below:

[0407] F G =F1+F2

[0408] F G=tensor([[[9.6238e-01,2.2908e+00,2.5063e+00,...,5.2840e-01,4.5951e-01,1.6695e-01],[1.0752e+00,1.1311e+00,1.1153e+ 00,...,2.1401e-01,-1.5902e-01,2.7075e-02],[1.8572e+00,1.9110e+00,1.4049e+00,...,8.8964e-01,1.0463e+00,-4.9521e-02 ],...,[1.7862e-01,5.5219e-01,5.1685e-01,...,5.4397e-01,8.0056e-01,4.0641e-01],[1.2998e+00,1.1037e+00,2.0545e+00,. ..,2.4297e-01,-3.3995e-02,9.3196e-01],[4.3208e-01,-7.9144e-02,7.0170e-01,...,1.1339e+00,6.8774e-01,1.8583e+00]]])

[0409] Step 3-1-3-7: F G Perform linear transformations of Q, K, and V respectively as follows:

[0410] Q=W q ×F G

[0411] K=W k ×F G

[0412] V=W v ×F G

[0413] Among them, W q , W k , W v is the weight matrix;

[0414] Step 3-1-3-8 calculates the dot product of Q and K, and then performs a softmax operation to obtain the position attention weight A, as shown below:

[0415]

[0416] Among them, d is the dimension of the feature, which is used to scale the attention weight;

[0417] Step 3-1-3-9: Use the attention weight A to perform weighted summation on V to obtain F. sa, as shown below:

[0418] F sa =A·V

[0419] F sa =tensor([[[0.2699,0.1164,0.2128,...,0.4504,0.3743,0.2482],[0.3675,0.4466,0.4 117,...,0.3201,0.4217,0.4140],[0.3050,0.3459,0.3158,...,0.2545,0.2616,0.2600 ],...,[0.3758,0.3455,0.3430,...,0.4157,0.4400,0.3828],[0.5140,0.5259,0.4199,...,0.3929,0.4545,0.4787],[0.2113,0.2436,0.2357,...,0.2331,0.1970,0.2359]]])

[0420] Step 3-1-3-10: F sa Subtract V and input it into the LBR layer to get F Δ , as shown below:

[0421] F Δ =LBR(F sa -V)

[0422] Among them, LBR is a linear layer, a normalization layer, and an activation layer;

[0423] F Δ=tensor([[[1.0293e+00,7.2997e-01,7.6899e-01,...,1.3558e+00,2.5385e-02,0.0000e+00],[0.0000e+00,0.0000e+00,0.0000e +00,...,0.0000e+00,0.0000e+00,2.5700e-01],[0.0000e+00,2.0476e+00,7.6412e-02,...,2.1709e-01,0.0000e+00,1.4078e+00 ],...,[5.9545e-01,0.0000e+00,0.0000e+00,...,0.0000e+00,0.0000e+00,0.0000e+00],[0.0000e+00,0.0000e+00,7.5174e-02, ...,5.1843e-01,0.0000e+00,0.0000e+00],[7.0393e-01,0.0000e+00,0.0000e+00,...,0.0000e+00,3.1810e-01,0.0000e+00]]])

[0424] Step 3-1-3-11 Change the feature F Δ 、F G and attention feature F sa Add together to get the global feature F o , as shown below:

[0425] F o =F Δ +F G +F sa

[0426] F o=tensor([[[1.8860e+00,2.8593e+00,3.0769e+00,...,1.6905e+00,2.2832e+00,2.5789e+00],[2.8247e-01,1.7836e+00,7.4470e -02,...,0.0000e+00,4.9427e-02,5.4470e-01],[1.7836e+00,0.0000e+00,0.0000e+00,...,1.8656e+00,0.0000e+00,0.0000e+00] ,...,[0.0000e+00,1.7886e+00,1.7856e+00,,...,0.0000e+00,3.0574e-01,1.3656e+00,],[1.3236e+00,0.0000e+00,0.0000e+00, ...,3.4698e-02,5.5326e-01,0.0000e+00],[0.0000e+00,1.7786e+00,,0.0000e+00,...,4.3954e-01,5.6064e-01,4.4845e-02]]])

[0427] Step 3-1-3-12: F out =F o Repeat steps 3-1-1 to 3-1-3 for four cycles to obtain the local feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 , the local feature obtained in the i-th cycle is recorded as F i,r1 ;

[0428] Step 3-1-4 obtains the local neighborhood feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 The deep local features F of the point cloud are extracted by splicing together through the residual feature propagation module L ;

[0429] Step 3-1-4-1: Concatenate the local features output by each layer to obtain F c , as shown below:

[0430] F c =concat(F 1,r1 ,F 2,r1 ,F 3,r1 ,F 4,r1 )

[0431] Step 3-1-4-2: Fc Input the convolutional layer to transform and get F' c , as shown below:

[0432] F′ c =conv(F c )

[0433] Among them, conv is a 1*1 convolutional layer;

[0434] Step 3-1-4-3 F' c Input group normalization layer to get F RG , as shown below:

[0435] F RG =RG(F′ c )

[0436] Among them, RG is ReLU plus GroupNormalization;

[0437] Step 3-1-4-4 F' c Perform linear transformations of Q, K, and V respectively as follows:

[0438] Q=W q ×F′ c

[0439] K=W k ×F′ c

[0440] V=W v ×F′ c

[0441] Among them, W q , W k , W v is the weight matrix;

[0442] Step 3-1-4-5 Calculate F' c The offset attention F OA , the calculation process is as follows:

[0443]

[0444] F OA = LBR(F′ c -F s )+F′ c

[0445] Among them, d is the feature dimension, F s is the self-attention feature matrix of the input features, and LBR is the linear layer, BN layer, and activation layer;

[0446] Step 3-1-4-6: FRG and F OA Add and normalize to get the local feature F of the point cloud L , as shown below:

[0447] F L =BN(F RG +F OA )

[0448] Among them, BN is BatchNormalization;

[0449] Step 3-1-4-7: F c =F L , repeat steps 3-1-4-1 to 3-1-4-6 to extract deep local features of the point cloud, and execute the cycle four times;

[0450] Step 3-1-5: The deep local features F of the point cloud L and the global feature F o Splice them together to get the complete point cloud feature representation F P , as shown below:

[0451] F P =concat(F o ,F L )

[0452] Step 3-1-6: F P Input convolutional layer to transform features to get F' P , as shown below:

[0453] F′ P =conv(F P )

[0454] Step 3-1-7 F' p Input pooling layer to get F D , as shown below:

[0455] F D =max(F′ P )+avg(F′ P )

[0456] F D=tensor([[2.9931,3.6464,3.4377,...,0.5517,0.3343,0.3412],[3.1057,3.0460,2.7 465,...,0.2652,0.1842,0.3813],[3.2991,3.7002,3.1012,...,0.2940,0.3461,0.2748 ],...,[3.3840,3.0341,3.2463,...,0.1948,0.1873,0.3846],[2.4733,19.8228,7.6581,...,0.3237,0.3374,0.1448],[3.1705,3.1975,3.6007,...,0.2905,0.3212,0.3223]])

[0457] Among them, max is the maximum pooling operation, and avg is the average pooling operation;

[0458] Step 3-2 Input the view dataset into the MVCNN network to obtain the view feature representation F of the point cloud V ;

[0459] Step 3-2-1 Combine the 12 views into a group to form a view feature representation F of the 3D model M , and adjusted to F' M , as shown below:

[0460] F′ M = reshape(F M )

[0461] Among them, reshape is the image resizing function to make it suitable for the input of the MVCNN network;

[0462] Step 3-2-2 F' M Input into the feature extraction module of MVCNN to obtain the deep semantic features F of the view of the 3D model. M , as shown below:

[0463] F″ M =AlexNet(F′ M )

[0464] Among them, AlexNet is the feature extractor; F″ M=tensor([[0.0000,0.0000,0.0000,...,0.0000,0.0000,1.1480],[0.5555,0.0000,0.0 000,...,0.0000,0.0000,0.2828],[0.0000,0.0000,0.0000,...,0.0000,0.0000,0.0000 ],...,[0.7483,0.0000,0.0000,...,0.0000,0.0000,0.0000],[6.6639,0.0000,0.0000,...,0.0000,0.0000,4.4495],[2.3945,0.0000,0.0000,...,0.0000,0.0000,0.2641]])

[0465] Step 3-2-3 Place F” M The input pooling layer obtains the complete feature representation F of the 3D model projection into the view V , as shown below:

[0466] F V =max(F″ M )

[0467] Among them, max is the maximum pooling operation;

[0468] Step 3-3 The point cloud representation F of the 3D model D and the view representation F V After concatenation, the correlation score F between the point cloud features and the view features is calculated through the convolution layer and the activation layer. C , as shown below:

[0469] F C =sig(conv(concat(F D ,F V ))

[0470] Among them, conv is 1*1 convolution, sig is the activation function;

[0471] Step 3-4 uses the correlation score to enhance the view representation of the point cloud to obtain the enhanced point cloud view feature F I , as shown below:

[0472] F I =(F V ×F C )+F V

[0473] F I=tensor([[[0.0087,1.4142,1.5946,...,0.0000,1.0638,0.1363],[0.6605,0.0000,1.5 400,...,0.0000,1.9027,0.3500],[0.0000,0.1923,1.0834,...,0.0000,1.7147,0.2447 ],...,[0.0000,1.1548,0.8931,...,0.0000,0.6254,0.1822],[0.0000,0.0000,0.0000,...,0.3083,1.9311,0.0000],[0.0579,0.0000,0.0747,...,0.0000,1.5251,0.0000]]])

[0474] Step 3-5: Fusion of point cloud features F of the 3D model D and view feature F I , and obtain the global feature representation F of the 3D model through the pooling layer J , as shown below:

[0475] F J =max(conv(concat(F I ,F D )))

[0476] Among them, conv is 1*1 convolution, and max is the maximum pooling operation;

[0477] F J =tensor([[0.0000,1.2271,0.5415,...,0.6519,0.0000,0.0000],[0.5039,1.6612,0.0 887,...,1.6458,1.0419,0.0000],[0.0000,1.7236,0.0000,...,1.2842,0.1134,0.0000 ],...,[0.0324,2.2678,0.2398,...,0.4141,0.4859,0.0000],[3.4902,0.0000,3.5696,...,0.0000,2.8185,3.7185],[0.1416,1.3231,0.0000,...,0.9984,0.1984,0.3281]])

[0478] Step 3-6: Find the 2, 3, and 4 views with the highest relevance scores. i With point cloud feature F DFusion is performed in the above manner to obtain the hierarchical representation F of the 3D model S ;

[0479] F S2 =max(conv(concat(F2,F D )))

[0480] F S3 =max(conv(concat(F3,F D )))

[0481] F S4 =max(conv(concat(F4,F D )))

[0482] Among them, F i (i=2,3,4) indicates the i view features with the highest relevance scores;

[0483] Steps 3-7 will represent the layered representation F s2 、F s3 、F s4 Take the average and combine it with the global feature representation F of the 3D model J Concatenate them together to get the final feature representation F of the point cloud A , as shown below:

[0484] F A =concat(mean(F S2 ,F s3 ,F s4 ),F J )

[0485] F A =tensor([[1.9710,0.0000,1.3905,...,0.0000,0.0000,0.0000],[0.0000,0.0000,4.1 429,...,0.0000,0.0000,0.8378],[0.0000,0.1041,0.0000,...,0.0000,1.4279,0.0000 ],...,[0.2597,0.0000,0.0000,...,3.1098,0.0000,0.5707],[0.1126,0.0000,0.0087,...,0.0000,1.7866,0.4852],[0.0000,0.0000,0.0000,...,0.0000,0.0000,0.0000]])

[0486] Step 3-8: F AInput the fully connected layer to obtain the feature vector U of the three-dimensional model;

[0487] Step 3-9 Input the feature vector U into the softmax function to calculate the 3D model in the semantic category C i The predicted probability P under (i=1,2,…,n) i , as shown below:

[0488]

[0489] Among them, softmax(C i |U) is a three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U;

[0490] P=tensor([0.1599,0.0276,0.0172,0.0264,0.0311,0.0197,0.0142,0.0354,0.0376 ,0.0009,0.0158,0.0168,0.0173,0.0154,0.0306,0.0256,0.0250,0.0163,0.0200,0. 0.0185,0.0239,0.0312,0.0169,0.0223,0.0273,0.0138,0.0267,0.0188,0.0218,0.0257,0.0249,0.0100,0.0197,0.0151,0.0209,0.0254,0.0164,0.0370,0.0074,0.0233])

[0491] The calculation process of the semantic category C of the three-dimensional model in step 3-10 is as follows:

[0492]

[0493] Among them, softmax(C i |U) is the three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U,

[0494] C = 0;

[0495] Using the optimized network model, the three-dimensional model of the test case "airplane_0630" is classified, and its predicted category "airplane" is the same as the true category.

[0496] The optimized model is used to classify the airplane category models in all test data with an accuracy rate of 93.8%.

[0497] The three-dimensional model classification method based on point cloud position coding and multimodal feature fusion in the embodiment of the present invention effectively integrates the original point cloud features and multi-view features of the three-dimensional model, and achieves higher classification performance.

[0498] The above is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, and the specific implementation methods herein are only used to help understand the method of the present invention. For those of ordinary skill in the art, according to the idea of ​​the present invention, changes and modifications can be made in the specific implementation methods and application scopes, so the present invention should not be understood as limiting the present invention.

Claims

1. A three-dimensional model classification method based on point cloud position encoding and multimodal feature fusion, the method comprising the following steps: Step 1: First, downsample the original point cloud to obtain the original point cloud dataset, input the point cloud dataset into the point cloud position encoding module for relative position encoding, and then project the point cloud to obtain a set of two-dimensional views. Each point cloud model is projected into 12 views to obtain a view dataset. Step 2: In the training phase, the encoded point cloud features are input into the DRCNN network, and the view dataset is input into the MVCNN network. The outputs of the two network models are fused according to the correlation scores calculated by the attention fusion module, and the network parameters are optimized according to the category labels of the point clouds. Step 3: Testing phase, i.e., the 3D model classification process, inputs the test data set into the optimized network model, calculates the probability distribution of the 3D model under different categories, and the category with the highest probability is the category of the 3D model.

2. The three-dimensional model classification method based on point cloud position coding and multimodal feature fusion according to claim 1 is characterized in that: In step 1, the original point cloud is first downsampled to obtain the original point cloud data set, the point cloud data set is input into the point cloud position encoding module for relative position encoding, and then the point cloud is projected to obtain a set of two-dimensional views. Each point cloud model is projected into 12 views to obtain a view data set, and the data set is divided into a training data set and a test data set. The specific steps are: Step 1-1 uses uniform sampling to downsample the original point cloud to obtain relatively uniform N points p1, p2, ..., p N , P=(p1,p2,…,p N )∈R N×3 As input feature F in , the high-dimensional feature F of the point cloud is obtained by embedding it into a higher-dimensional feature space through the MLP layer ch , as shown below: F ch =MLP(P) Step 1-2: original point cloud feature F in and high-dimensional point cloud features F ch Use the K nearest neighbor algorithm to group the neighborhood of the point cloud; Step 1-2-1 The original point cloud feature F in Input K nearest neighbor algorithm to get feature F i,k ∈R N×K×3 ; Step 1-2-2: High-dimensional feature F ch Input K nearest neighbor algorithm to get feature F c,k ∈R N×K×C 1, where C1 is the high-dimensional feature dimension; Step 1-3 Feature F i,k Convert to relative position F' through trigonometric transformation i,k , as shown below: Among them, α, β, δ are hyper parameters, δ = e -5 Ensure the stability of operation; Steps 1-4 get the relative position coordinates F' i,k Take any point p in the neighborhood j =F' i,k,j =(x, y, z) to perform position encoding to obtain the relative position encoding F pos , as shown below: F pos =PosEmb(p j )=(pos(x),pos(y),pos(z),x,y,z,dis(p))∈R C+4 Among them, (pos(x),pos(y),pos(z))∈R C Represents the point p j The angle encoding of the three-dimensional coordinate vector and the coordinate axis, C is the dimension of the angle encoding. Since the angles between the coordinates of different points and the coordinate axis may be the same, the coordinate features of the point and the distance dis(p) from the point to the origin of the coordinate are added on this basis. Taking pos(x) as an example, for different feature channels i∈[0,C / 6], it is as follows: Steps 1-5: Feature F c,k Input to the linear transformation module (LT) to get F' c,k , the transformation process is as follows: Where λ and μ are trainable parameters, δ = e -5 Ensure the stability of operation; Step 1-6 F' c,k Input two layers of MLP to perform residual propagation to extract local features F h ; F h =MLP(MLP(F′ c,k ))+F′ c,k Among them, MLP is a multi-layer perceptron; Step 1-7: Transform the position encoding matrix F pos With F h Multiply them together to calculate a high-dimensional feature matrix containing the location information of each point, and h After adding, the maximum pooling is performed to obtain the output feature F of the position encoding module out ; F out =max((F pos ×F h )+F h ) Among them, max is the maximum pooling operation; Steps 1-8 project the 3D model in Modelnet40 to obtain a 2D view set; Step 1-8-1 Fix the 3D model at the center of the virtual sphere and select a fixed angle and amount for projection; Step 1-8-2 places the virtual camera above the three-dimensional model, and the camera rotates a circle, and each step of 30 degrees projection obtains a set of two-dimensional views, which are recorded as V1, V2, V3, V4, V5, V6, V7, V8, V9, V10, V11 and V12; 3. The three-dimensional model classification method based on point cloud position coding and multimodal feature fusion according to claim 1 is characterized in that: In step 2, the encoded point cloud features are input into the DRCNN network, the view data set is input into the MVCNN network, the outputs of the two network models are fused according to the correlation scores calculated by the attention fusion module, and the network parameters are optimized according to the category labels of the point cloud. The specific process is as follows: Step 2-1: The training model in ModelNet40 is processed by step 1 to obtain F out Input into the DRCNN model to obtain feature F D ; Step 2-1-1 Set F out =(x1,x2,…,x n ) Choose two different K values ​​to construct two concentric neighborhoods to obtain F k1 、F k2 , where F k1 =((x 1,1 ,x 1,2 ,…,x 1,k1 ),…,(x n,1 ,x n,2 ,…,x n,k1 )), F k2 =((x 2,1 ,x 2,2 ,…,x 2,k2 ),…,(x n,1 ,x n,2 ,…,x n,k2 )), where x i is the feature vector of the i-th point in the point cloud, x i,j For a central point x i The feature vector of the jth neighboring point; Step 2-1-2: F k1 and F k2 Input the double-ring feature extraction module to obtain the local feature F r1 、F r2 As shown below: F r1 =f conv (F out ,F k1 )=Φ(Ψ([F out ,F k1 -F out ])) F r2 =f conv (F out ,F k2 )=Φ(Ψ([F out ,F k2 -F out ])) Among them, Φ represents the maximum pooling of aggregation function, and Ψ represents dynamic graph convolution; Step 2-1-3: F r1 、F r2 Input to the attention gate feature fusion module to fuse the features obtained from different neighborhoods; Step 2-1-3-1 Set F r1 、F r2 ∈R N×C Input two 1*1 convolutional layers to get F' r1 、F' r2 , F' r1 、F' r2 Put them together to get F r1,2 ∈R N×2C , as shown below: F′ r1 =conv(F r1 )F′ r2 =conv(F r2 ) F r1,2 =concat(F′ r1 ,F′ r2 ) Step 2-1-3-2: F r1,2 Input the ReLu activation function to get D, as shown below: D=ReLU(F r1,2 ) Step 2-1-3-3 Input D into the convolutional layer to obtain the similarity matrix W, as shown below: W = conv(D) Among them, conv is a 1*1 convolutional layer, and W is the similarity matrix W∈R N×1 , where the i-th element w i ∈[0,1] represents the weight of the i-th point in the point cloud; Step 2-1-3-4: F r1 Multiply it by the weight W to get F r1 The contribution F1 to the fused features is as follows: F1=F r1 ×W Step 2-1-3-5: F r2 Multiply it by the weight 1-W to get F r2 The contribution F2 to the fused features is as follows: F2=F r2 ×(1-W) Step 2-1-3-6 Add F1 and F2 to get the feature F after eliminating redundancy G , as shown below: F G =F1+F2 Step 2-1-3-7: F G Perform linear transformations of Q, K, and V respectively as follows: Q=W q ×F G K=W k ×F G V=W v ×F G Among them, W q , W k , W v is the weight matrix; Step 2-1-3-8 calculates the dot product of Q and K, and then performs a softmax operation to obtain the position attention weight A, as shown below: Among them, d is the dimension of the feature, which is used to scale the attention weight; Step 2-1-3-9: Use the attention weight A to perform weighted summation on V to obtain F. sa , as shown below: F sa =A·V Step 2-1-3-10: F sa Subtract V and input it into the LBR layer to get F Δ , as shown below: F Δ =LBR(F sa -V) Among them, LBR is a linear layer, a normalization layer, and an activation layer; Step 2-1-3-11 Change the feature F Δ 、F G and attention feature F sa Add together to get the global feature F o , as shown below: F o =F Δ +F G +F sa Step 2-1-3-12: F out =F o Repeat steps 2-1-1 to 2-1-3 for four cycles to obtain the local feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 , the local feature obtained in the i-th cycle is recorded as F i,r1 ; Step 2-1-4 obtains the local neighborhood feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 The deep local features F of the point cloud are extracted by splicing together through the residual feature propagation module L ; Step 2-1-4-1: Concatenate the local features output by each layer to obtain F c , as shown below: F c =concat(F 1,r1 ,F 2,r1 ,F 3,r1 ,F 4,r1 ) Step 2-1-4-2: F c Input convolutional layer to transform to get F' c , as shown below: F′ c =conv(F c ) Among them, conv is a 1*1 convolutional layer; Step 2-1-4-3 F' c Input group normalization layer to get F RG , as shown below: F RG =RG(F′ c ) Among them, RG is ReLU plus GroupNormalization; Step 2-1-4-4 F' c Perform linear transformations of Q, K, and V respectively as follows: Q=W q ×F′ c K=W k ×F′ c V=W v ×F′ c Among them, W q , W k , W v is the weight matrix; Step 2-1-4-5 Calculate F' c The offset attention F OA , the calculation process is as follows: F OA =LBR(F′ c -F s )+F′ c Among them, d is the feature dimension, F s is the self-attention feature matrix of the input features, and LBR is the linear layer, BN layer, and activation layer; Step 2-1-4-6: F RG and F OA Add and normalize to get the local feature F of the point cloud L , as shown below: F L =BN(F RG +F OA ) Among them, BN is BatchNormalization; Step 2-1-4-7: F c =F L , repeat steps 2-1-4-1 to 2-1-4-6 to extract deep local features of the point cloud, and execute the cycle four times; Step 2-1-5: The deep local features F of the point cloud L and the global feature F o Splice them together to get the complete point cloud feature representation F P , as shown below: F P =concat(F o ,F L ) Step 2-1-6: F P Input convolutional layer to transform the features to get F' P , as shown below: F′ P =conv(F P ) Step 2-1-7 F' p Input pooling layer to get F D , as shown below: F D =max(F′ P )+avg(F′ P ) Among them, max is the maximum pooling operation, and avg is the average pooling operation; Step 2-2 Input the view dataset into the MVCNN network to obtain the view feature representation F of the point cloud V ; Step 2-2-1 Combine the 12 views into a group to form a view feature representation F of the 3D model M , and adjusted to F' M , as shown below: F′ M =reshape(F M ) Among them, reshape is the image resizing function to make it suitable for the input of the MVCNN network; Step 2-2-2 F' M Input into the feature extraction module of MVCNN to obtain the deep semantic features F of the view of the 3D model. M , as shown below: F″ M =AlexNet(F′ M ) Among them, AlexNet is a feature extractor; Step 2-2-3: F” M The input pooling layer obtains the complete feature representation F of the 3D model projection into the view V , as shown below: F V =max(F″ M ) Among them, max is the maximum pooling operation; Step 2-3: The point cloud representation F of the 3D model D and the view representation F V After concatenation, the correlation score F between the point cloud features and the view features is calculated through the convolution layer and the activation layer. C , as shown below: F C =sig(conv(concat(F D ,F V )) Among them, conv is 1*1 convolution, sig is the activation function; Step 2-4 uses the correlation score to enhance the view representation of the point cloud to obtain the enhanced point cloud view feature F I , as shown below: F I =(F V ×F C )+F V Step 2-5: Fusion of point cloud features F of the 3D model D and view feature F I , and obtain the global feature representation F of the 3D model through the pooling layer J , as shown below: F J =max(conv(concat(F I ,F D ))) Among them, conv is 1*1 convolution, and max is the maximum pooling operation; Step 2-6: Find the 2, 3, and 4 views with the highest relevance scores. i With point cloud feature F D Fusion is performed in the above manner to obtain the hierarchical representation F of the 3D model S ; F S2 =max(conv(concat(F2,F D ))) F S3 =max(conv(concat(F3,F D ))) F S4 =max(conv(concat(F4,F D ))) Among them, F i (i=2,3,4) indicates the i view features with the highest relevance scores; Steps 2-7 will represent the layered representation F s2 、F s3 、F s4 Take the average and combine it with the global feature representation F of the 3D model J Concatenate them together to get the final feature representation F of the point cloud A , as shown below: F A =concat(mean(F S2 ,F s3 ,F s4 ),F J ) Step 2-8 F A Input the fully connected layer to obtain the feature vector U of the three-dimensional model; Step 2-9 Input the feature vector U into the softmax function to calculate the 3D model in the semantic category C i The predicted probability P under (i=1,2,…,n) i , as shown below: Among them, softmax(C i |U) is a three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U; Step 2-10 uses the cross entropy loss function to calculate the loss as follows: Among them, Y i is the unique hot encoding of the true label of the 3D model, log is the natural logarithm, and N is the total number of 3D training models; Step 2-11 is to back propagate the error loss and update the parameters layer by layer. The parameter update process is as follows: Among them, θ represents the parameter set, θ' represents the updated parameter set, and α is the learning rate; Step 2-12 repeats step 2-1 to step 2-11 until the number of training times is reached to obtain an optimized three-dimensional model classification network.

4. The three-dimensional model classification method based on point cloud position coding and multimodal feature fusion according to claim 1, characterized in that: In step 3, the optimized network model is used to classify the test set data. The specific process is as follows: Step 3-1 The test model in ModelNet40 is processed in step 1 to obtain F out Input into the DRCNN model to obtain feature F D ; Step 3-1-1 Set F out =(x1,x2,…,x n ) Choose two different K values ​​to construct two concentric neighborhoods to obtain F k1 、F k2 , where F k1 =((x 1,1 ,x 1,2 ,…,x 1,k1 ),…,(x n,1 ,x n,2 ,…,x n,k1 )), F k2 =((x 2,1 ,x 2,2 ,…,x 2,k2 ),…,(x n,1 ,x n,2 ,…,x n,k2 )), where x i is the feature vector of the i-th point in the point cloud, x i,j For a central point x i The feature vector of the jth neighboring point; Step 3-1-2 F k1 and F k2 Input the double-ring feature extraction module to obtain the local feature F r1 、F r2 As shown below: F r1 =f conv (F out ,F k1 )=Φ(Ψ([F out ,F k1 -F out ])) F r2 =f conv (F out ,F k2 )=Φ(Ψ([F out ,F k2 -F out ])) Among them, Φ represents the maximum pooling of aggregation function, and Ψ represents dynamic graph convolution; Step 3-1-3: F r1 、F r2 Input to the attention gate feature fusion module to fuse the features obtained from different neighborhoods; Step 3-1-3-1 Set F r1 、F r2 ∈R N×C Input two 1*1 convolutional layers to get F' r1 、F' r2 , F' r1 、F' r2 Put them together to get F r1,2 ∈R N×2C , as shown below: F’ r1 =conv(F r1 ) F’ r2 =conv(F r2 ) F r1,2 =concat(F’ r1 ,F’ r2 ) Step 3-1-3-2: F r1,2 Input the ReLu activation function to get D, as shown below: D=ReLU(F r1,2 ) Step 3-1-3-3 Input D into the convolutional layer to obtain the similarity matrix W, as shown below: W = conv(D) Among them, conv is a 1*1 convolutional layer, and W is the similarity matrix W∈R N×1 , where the i-th element w i ∈[0,1] represents the weight of the i-th point in the point cloud; Step 3-1-3-4: F r1 Multiply it by the weight W to get F r1 The contribution F1 to the fused features is as follows: F1=F r1 ×W Step 3-1-3-5: F r2 Multiply it by the weight 1-W to get F r2 The contribution F2 to the fused features is as follows: F2=F r2 ×(1-W) Step 3-1-3-6 Add F1 and F2 to get the feature F after eliminating redundancy G , as shown below: F G =F1+F2 Step 3-1-3-7: F G Perform linear transformations of Q, K, and V respectively as follows: Q=W q ×F G K=W k ×F G V=W v ×F G Among them, W q , W k , W v is the weight matrix; Step 3-1-3-8 calculates the dot product of Q and K, and then performs a softmax operation to obtain the position attention weight A, as shown below: Among them, d is the dimension of the feature, which is used to scale the attention weight; Step 3-1-3-9: Use the attention weight A to perform weighted summation on V to obtain F. sa , as shown below: F sa =A·V Step 3-1-3-10: F sa Subtract V and input it into the LBR layer to get F Δ , as shown below: F Δ =LBR(F sa -V) Among them, LBR is a linear layer, a normalization layer, and an activation layer; Step 3-1-3-11 Change the feature F Δ 、F G and attention feature F sa Add together to get the global feature F o , as shown below: F o =F Δ +F G +F sa Step 3-1-3-12: F out =F o Repeat steps 3-1-1 to 3-1-3 for four cycles to obtain the local feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 , the local feature obtained in the i-th cycle is recorded as F i,r1 ; Step 3-1-4 obtains the local neighborhood feature F 1,r1 、F 2,r1 、F 3,r1 、F 4,r1 The deep local features F of the point cloud are extracted by splicing together through the residual feature propagation module L ; Step 3-1-4-1: Concatenate the local features output by each layer to obtain F c , as shown below: F c =concat(F 1,r1 ,F 2,r1 ,F 3,r1 ,F 4,r1 ) Step 3-1-4-2: F c Input the convolutional layer to transform and get F' c , as shown below: F′ c =conv(F c ) Among them, conv is a 1*1 convolutional layer; Step 3-1-4-3 F' c Input group normalization layer to get F RG , as shown below: F RG =RG(F′ c ) Among them, RG is ReLU plus GroupNormalization; Step 3-1-4-4 F' c Perform linear transformations of Q, K, and V respectively as follows: Q=W q ×F′ c K=W k ×F′ c V=W v ×F′ c Among them, W q , W k , W v is the weight matrix; Step 3-1-4-5 Calculate F' c The offset attention F OA , the calculation process is as follows: F OA =LBR(F′ c -F s )+F′ c Among them, d is the feature dimension, F s is the self-attention feature matrix of the input features, and LBR is the linear layer, BN layer, and activation layer; Step 3-1-4-6: F RG and F OA Add and normalize to get the local feature F of the point cloud L , as shown below: F L =BN(F RG +F OA ) Among them, BN is BatchNormalization; Step 3-1-4-7: F c =F L , repeat steps 3-1-4-1 to 3-1-4-6 to extract deep local features of the point cloud, and execute the cycle four times; Step 3-1-5: The deep local features F of the point cloud L and the global feature F o Splice them together to get the complete point cloud feature representation F P , as shown below: F P =concat(F o ,F L ) Step 3-1-6: F P Input convolutional layer to transform the features to get F' P , as shown below: F P ’=conv(F P ) Step 3-1-7 F' p Input pooling layer to get F D , as shown below: F D =max(F’ P )+avg(F’ P ) Among them, max is the maximum pooling operation, and avg is the average pooling operation; Step 3-2 Input the view dataset into the MVCNN network to obtain the view feature representation F of the point cloud V ; Step 3-2-1 Combine the 12 views into a group to form a view feature representation F of the 3D model M , and adjusted to F' M , as shown below: F’ M =reshape(F M ) Among them, reshape is the image resizing function to make it suitable for the input of the MVCNN network; Step 3-2-2 F' M Input into the feature extraction module of MVCNN to obtain the deep semantic features F of the view of the 3D model. M , as shown below: F” M =AlexNet(F’ M ) Among them, AlexNet is a feature extractor; Step 3-2-3 Place F” M The input pooling layer obtains the complete feature representation F of the 3D model projection into the view V , as shown below: F V =max(F” M ) Among them, max is the maximum pooling operation; Step 3-3 The point cloud representation F of the 3D model D and the view representation F V After concatenation, the correlation score F between the point cloud features and the view features is calculated through the convolution layer and the activation layer. C , as shown below: F C =sig(conv(concat(F D ,F V )) Among them, conv is 1*1 convolution, sig is the activation function; Step 3-4 uses the correlation score to enhance the view representation of the point cloud to obtain the enhanced point cloud view feature F I , as shown below: F I =(F V ×F C )+F V Step 3-5: Fusion of point cloud features F of the 3D model D and view feature F I , and obtain the global feature representation F of the 3D model through the pooling layer J , as shown below: F J =max(conv(concat(F I ,F D ))) Among them, conv is 1*1 convolution, and max is the maximum pooling operation; Step 3-6: Find the 2, 3, and 4 views with the highest relevance scores. i With point cloud feature F D Fusion is performed in the above manner to obtain the hierarchical representation F of the 3D model S ; F S2 =max(conv(concat(F2,F D ))) F S3 =max(conv(concat(F3,F D ))) F S4 =max(conv(concat(F4,F D ))) Among them, F i (i=2,3,4) indicates the i view features with the highest relevance scores; Steps 3-7 will represent the layered representation F s2 、F s3 、F s4 Take the average and combine it with the global feature representation F of the 3D model J Concatenate them together to get the final feature representation F of the point cloud A , as shown below: F A =concat(mean(F S2 ,F s3 ,F s4 ),F J ) Step 3-8: F A Input the fully connected layer to obtain the feature vector U of the three-dimensional model; Step 3-9 Input the feature vector U into the softmax function to calculate the 3D model in the semantic category C i The predicted probability P under (i=1,2,…,n) i , as shown below: Among them, softmax(C i |U) is a three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U; The calculation process of the semantic category C of the three-dimensional model in step 3-10 is as follows: Among them, softmax(C i |U) is the three-dimensional model of category C i The probability of U i is the i-th component of the eigenvector U.

Images