Multi-view three-dimensional model classification method based on triple attention

Abstract

The invention relates to a multi-view three-dimensional model classification method based on triple attention. According to the method, firstly, a three-dimensional model is projected to obtain a two-dimensional view set, a binary contour data enhancement method is adopted for the two-dimensional view set to reduce the influence of a redundant part in a view, secondly, a triple attention module is introduced into network construction based on a convolutional neural network to extract view features, and finally, a three-dimensional image is obtained. According to the method, a GAT module is introduced, a cross-dimension dependency relationship of a view is captured when attention is calculated, a GAT module is introduced again, a connection relationship between nodes is optimized by extracting a graph data structure of the view, so that richer feature expression is obtained, and finally, a Softmax function is used for determining the category of a three-dimensional model. The method has a good three-dimensional model classification effect, and can more accurately solve the problem of three-dimensional model classification.

Description

Technical field:

[0001] The present invention relates to a multi-view method based on triple attention. Figure 3 This method has a good application in the field of three-dimensional model classification. Background technology:

[0002] With the development of 3D technology and the widespread application of small 3D sensors, the generation process of 3D models has become easier and the number has increased exponentially. Compared with 2D images, 3D models can provide richer spatial visual perception information for objects and have been widely used in digital entertainment, industrial design, virtual reality, biomedicine and other fields. 3D model classification is a basic research in its application field, and the quality of classification directly affects the actual application effect. Since 3D models are usually high-dimensional and unstructured, how to effectively extract features and efficiently classify a large number of 3D models has become an urgent problem to be solved. As an intuitive representation of 3D models, 2D views can describe the contour information of the surface of 3D models to a certain extent, and using views to realize 3D model classification is consistent with basic human cognition, that is, if two 3D models are roughly the same when viewed from multiple different perspectives, then the two 3D models can be considered similar. 2D views are a kind of structured data, and neural network algorithms can be directly used to extract deep features. Therefore, the view-based 3D model classification method has attracted widespread attention and attention from researchers.

[0003] Since the attention mechanism has the ability to establish interdependencies between channels or spatial positions, it has been widely studied and applied in various computer vision tasks in recent years. Figure 3 The present invention uses a two-dimensional model classification method to preprocess the view by a data enhancement method. Based on the convolutional neural network model, a triple attention module is introduced, which is a new method for calculating attention weights by capturing cross-dimensional interactions using a three-branch structure. The present invention renders the two-dimensional view by a data enhancement method for binary contours, and extracts the two-dimensional view features by the triple attention module and the GAT module, further improving the model classification accuracy. Summary of the invention:

[0004] In order to solve the problem of three-dimensional model classification in the field of computer vision, the present invention discloses a multi-view triplet attention-based Figure 3 Dimensional model classification method.

[0005] To this end, the present invention provides the following technical solutions:

[0006] 1. Multi-view based on triplet attention Figure 3 The method for classifying dimensional models is characterized in that the method mainly comprises the following steps:

[0007] Step 1: Project the 3D model in Modelnet40 to obtain a 2D view set.

[0008] Step 2: Render the 2D view set using data augmentation methods for binary contours.

[0009] Step 3: Use the triplet attention module to perform feature extraction on the 2D view set.

[0010] Step 4: Construct a neural network model and add the SK attention module.

[0011] Step 5: Use the GAT module to process the view features into a graph structure.

[0012] Step 6: Extract training data and test data from ModelNet40.

[0013] Step 7: Use the training data to optimize the model, and use the optimized model to process the test data. After convolution, pooling, and fusion, the classification result is obtained in the output layer.

[0014] 2. The triplet attention-based multi-view method according to claim 1 Figure 3 The three-dimensional model classification method is characterized in that, in step 1, the three-dimensional model is preprocessed, and the specific steps are:

[0015] Step 1-1: Fix the 3D model at the center of the virtual sphere and select a fixed angle and amount for projection;

[0016] Step 1-2 places the virtual camera above the 3D model. The camera rotates a circle and constructs a set of 2D view sets V(M) at 30-degree projections per step, as shown below:

[0017] V(M)={V x |V1,V2,…,V 12},V x ∈R H×W×C

[0018] Among them, M represents the three-dimensional model, V x Represents a 2D view of a 3D model.

[0019] 3. The triplet attention-based multi-view method according to claim 1 Figure 3 The method for classifying a two-dimensional model is characterized in that, in step 2, a data enhancement method of a binary contour is used to render the two-dimensional view set, and the specific steps are as follows:

[0020] Step 2-1 selects an image V(x,y) from the two-dimensional view set V(M) as the input of the original image;

[0021] Step 2-2 uses the global threshold method to convert V(x, y) into a binary image B(x, y), selects a global threshold T, sets the pixels with grayscale values ​​greater than the threshold to 1, and sets the pixels with grayscale values ​​less than the threshold to 0, as shown below:

[0022]

[0023] Among them, V(x,y) represents the original image, B(x,y) represents the binary image, and T represents the threshold.

[0024] 4. The triplet attention-based multi-view method according to claim 1 Figure 3 The dimensional model classification method is characterized in that, in step 3, a triple attention module is used to extract view features, and the specific steps are:

[0025] Step 3-1 defines a three-branch module and takes the binary image B(x,y) as the input F(x,y) of the module;

[0026] Step 3-2 In the first branch, rotate F(x,y) 90° counterclockwise along the H axis to obtain F H′ , the calculation formula is as follows:

[0027] F H′ =Permute(F(x,y))

[0028] Among them, Permute() represents the rotation operation;

[0029] Step 3-2-1 For the rotated F H′ , calculate its attention weight A H′ , the calculation formula is as follows:

[0030] A H′ =σ(Conv(F H′ ))

[0031] Among them, σ represents the sigmoid activation function, and Conv represents the convolution operation;

[0032] Step 3-2-2 generates the attention weight A H′ Applied to F H′ Perform weighted fusion, rotate the output 90° clockwise along the H axis to keep the same shape as the input, and obtain the final feature representation F′ H′ , the calculation formula is as follows:

[0033] F′ H′ =F H′ ⊙A H′

[0034] Among them, ⊙ represents weighted operation;

[0035] Step 3-3 In the second branch, rotate F(x,y) 90° counterclockwise along the W axis to obtain F W′ , the calculation formula is as follows:

[0036] F W′ =Permute(F W )

[0037] Among them, Permute() represents the rotation operation;

[0038] Step 3-3-1 For the rotated F W′ , calculate its attention weight A W′ , the calculation formula is as follows:

[0039] A W′ =σ(Conv(F W′ ))

[0040] Among them, σ represents the sigmoid activation function, and Conv represents the convolution operation;

[0041] Step 3-3-2 generates the attention weight A W′ Applied to F W′ Perform weighted fusion, rotate the output 90° clockwise along the W axis to keep the same shape as the input, and obtain the final feature representation F′ W′ , the calculation formula is as follows:

[0042] F′ W′ =F W′ ⊙A W′

[0043] Among them, ⊙ represents weighted operation;

[0044] Step 3-4 In the third branch, rotate F(x,y) 90° counterclockwise along the C axis to obtain F C′ , the calculation formula is as follows:

[0045] F C′ =Permute(F C )

[0046] Among them, Permute() represents the rotation operation;

[0047] Step 3-4-1 For the rotated F C′ , calculate its attention weight A C′ , the calculation formula is as follows:

[0048] A C′ =σ(Conv(F C′ ))

[0049] Among them, σ represents the sigmoid activation function, and Conv represents the convolution operation;

[0050] Step 3-4-2 generates the attention weight A C′ Applied to F C′ Perform weighted fusion, rotate the output 90° clockwise along the C axis to keep the same shape as the input, and obtain the final feature representation F′ C′ , the calculation formula is as follows:

[0051] F′ C′ =F C′ ⊙A C′

[0052] Among them, ⊙ represents weighted operation;

[0053] Step 3-5 represents the feature F′ H′ , F′ W′ and F′ C′ Perform weighted summation to obtain the final feature representation F′. The calculation formula is as follows:

[0054] F′=F′ H′ +F′ W′ +F′ C′

[0055] 5. The triplet attention-based multi-view method according to claim 1 Figure 3 The dimensional model classification method is characterized in that in step 4, a neural network model is designed and an SK attention module is added, and the specific steps are as follows:

[0056] Step 4-1: For the neural network model used to extract features, the SK attention module is serially added after the triple attention module. For the feature representation F′, multiple convolution operations of different scales are first performed on F′. The calculation formula is as follows:

[0057] U=Conv k (F′)

[0058] Among them, k represents the convolution kernel size, Conv k () represents the convolution operation, and U represents the feature representation after the convolution kernel operation;

[0059] Step 4-2 In order to capture global context information, the SK attention module performs global average pooling on the fused feature representation U to generate a channel vector Z. The calculation formula is as follows:

[0060] Z = GlobalAvgPool(U)

[0061] Among them, GlobalAvgPool() represents the global average pooling operation;

[0062] Step 4-3 inputs the channel vector Z into a shared multi-layer perceptron to generate the attention weight a of each channel. The calculation formula is as follows:

[0063] a=σ(W2·δ(W1·z))

[0064] Among them, W1 and W2 represent the weights of the multi-layer perceptron, δ represents the activation function, and σ represents the normalization function;

[0065] Step 4-4 performs weighted summation on each feature representation based on the generated weight a to obtain the final representation V. The calculation formula is as follows:

[0066] V=a·U

[0067] 6. The triplet attention-based multi-view method according to claim 1 Figure 3 The dimensional model classification method is characterized in that, in step 5, the graph structure information of the view is extracted using the GAT module, and the specific steps are:

[0068] Step 5-1 For the feature representation V, define a feature vector V i , a shared learnable linear transformation is applied to each node, calculated as follows:

[0069] g i =WV i

[0070] Where W represents the learnable weight matrix;

[0071] Step 5-2 uses the attention function to calculate the original attention coefficient. The calculation formula is as follows:

[0072]

[0073] Among them, e ij represents the attention score from node j to node i, g i and g j Represent the embedding of node i and node j respectively, a represents the weight vector, and LeakyReLU() represents the activation function;

[0074] Step 5-3 normalizes all adjacent nodes of a node to obtain a normalized attention coefficient. The calculation formula is as follows:

[0075]

[0076] Among them, k∈N(i) represents all neighbor nodes connected to node i, αij Represents the attention weight from node j to node i;

[0077] Step 5-4 uses the standardized attention coefficient to perform weighted summation of the features of adjacent views. The calculation formula is as follows:

[0078]

[0079] Where σ represents the normalization function S i Indicates the summation result.

[0080] 7. The triplet attention-based multi-view method according to claim 1 Figure 3 The method for classifying a dimensional model is characterized in that in step 6, training data and test data are extracted from ModelNet40, and the specific steps are as follows:

[0081] Step 6-1 projects the three-dimensional training model in ModelNet40 to obtain training data of a two-dimensional view;

[0082] Step 6-2 projects the three-dimensional test model in ModelNet40 to obtain the test data of the two-dimensional view.

[0083] 8. The triplet attention-based multi-view method according to claim 1 Figure 3 The method for classifying a dimensional model is characterized in that in step 7, the training data is used to optimize the network model, the optimized network model is used to process the test data, and the classification result is obtained in the output layer after convolution, pooling and fusion. The specific steps are as follows:

[0084] Step 7-1 Use the training data to optimize the network model, input the two-dimensional view in the test data into the optimized network model, and output the result S after convolution and pooling. i ;

[0085] Step 7-2: S i Input the Softmax classifier to get the classification result, as shown below:

[0086] V i =Softmax(W C S i )

[0087] Among them, Softmax() represents the maximum classification operation, W C Represents the weight matrix of the classification layer, V i Represents the classification probability vector of the 3D model.

[0088] Beneficial effects:

[0089] The present invention proposes a multi-view triplet attention-based Figure 3 Dimensional model classification method.

[0090] 1. The present invention performs a two-dimensional projection on the three-dimensional model and uses a set of two-dimensional view data to express the shape representation of the three-dimensional model. For the geometric shape of the three-dimensional model, the present invention uses a data enhancement method for a binary wheel image. The method focuses on capturing view information, reduces the influence of redundant parts, has good rotation robustness, can well express the geometric shape characteristics of the three-dimensional model, and improves the classification accuracy of the three-dimensional model.

[0091] 2. This invention uses the triplet attention module, which is a new method to calculate attention weights by capturing cross-dimensional interactions using a three-branch structure. For the input view, the triplet attention establishes inter-dimensional dependencies through rotation operations, and then performs residual transformation to encode inter-channel and spatial information, thereby improving the learning ability of the network model.

[0092] 3. The present invention refers to the GAT module, which is a new neural network architecture based on graph structured data. The GAT module implicitly specifies different weights for different nodes in the neighborhood by stacking nodes in the layers. Different weight matrices are used to express the relationship between input features and output features, so as to optimize the defects of convolutional neural networks.

[0093] 4. The present invention uses the public dataset ModelNet40 model library to verify the proposed 3D model classification method. The results show that the classification method proposed by the present invention is effective. Description of the drawings: Figure 1 is a three-dimensional model instance to be classified in an embodiment of the present invention; Figure 2 The triplet attention-based multi-view Figure 3 Dimensional model classification framework diagram; Figure 3 The twelve angle views of the three-dimensional model are obtained by a virtual camera in the embodiment of the present invention; Figure 4 A two-dimensional view rendered by a binary contour method in an embodiment of the present invention; Specific implementation method:

[0094] In order to make the technical solution in the embodiment of the present invention clearly and specifically described, a 3D model in the test set airplane class of the 3D model public data set ModelNet40 is taken as an example. Figure 1 As shown, a test is performed, and there are 1200 three-dimensional models in the test set airplane class. The present invention is further described in detail in conjunction with the drawings in the embodiments.

[0095] The present invention implements a multi-view based on triple attention Figure 3 Dimensional model classification framework, such as Figure 2 As shown, the following steps are included:

[0096] Step 1: preprocess the 3D model. The specific steps are as follows:

[0097] Step 1-1 will Figure 1 The three-dimensional model shown is fixed at the center of the virtual sphere, and a fixed angle of 30 degrees and a projection number of 12 are selected for projection, as shown in Figure 3 As shown;

[0098] Step 1-2 places the virtual camera above the 3D model. The camera rotates a circle and projects 30 degrees at each step to obtain a set of 2D view sets V(M), as shown below:

[0099] V(M)={V x |V1,V2,…,V 12},V x ∈R H×W×C

[0100] Among them, M represents the three-dimensional model, V x Represents a 2D view of a 3D model.

[0101] Step 2 uses the data enhancement method of binary contours to render the two-dimensional view set. The specific steps are:

[0102] Step 2-1 selects an image V(x,y) from the two-dimensional view set V(M) as the input of the original image;

[0103] Step 2-2 uses the global threshold method to convert V(x, y) into a binary image B(x, y), selects a global threshold T, sets the pixels with grayscale values ​​greater than the threshold to 1, and sets the pixels with grayscale values ​​less than the threshold to 0, as shown below:

[0104]

[0105] Among them, V(x,y) represents the original image, B(x,y) represents the binary image, and T represents the threshold.

[0106] Steps 2-3 are as follows Figure 4 As shown, taking airplane as an example, the rendered view is used as the input of the network model, and the view feature vector is as follows: B(x,y) Feature=[[0.00093348 -0.00207696 -0.00019165 -0.000191590.00229465 -0.00013335 0.00027862 0.00013897 0.00082097 0.00153777 0.0009845-2.19558875e-04 1.77114632e-03 6.31115836e-05 -0.0012298 0.000798640.00065512 -2.96645449e-05 -8.78741121e-05 1.99694638e-04 0.000110980.00016649 -0.00012466 0.001154 -0.00033617 0.00015574 -0.00031423 -0.00022482 -0.00140581 -0.00013382 0.00024798 0.00150477 -0.00010429 -0.00149231 0.00188234 0.00033907 0.00023123 8.52104640e-05 1.53082318e-031.95166387e-04 -0.00012976 -0.00018275 0.00111355 -2.16975954e-04 -2.25743534e-05 -1.24966132e-03]…]

[0107] Step 3: Build a neural network model, such as Figure 2 As shown, the triple attention module is used to extract view features. The specific steps are:

[0108] Step 3-1 defines a three-branch module and takes the binary image B(x,y) as the input F(x,y) of the module;

[0109] Step 3-2 In the first branch, rotate F(x,y) 90° counterclockwise along the H axis to obtain F H′ , the calculation formula is as follows:

[0110] F H′ =Permute(F(x,y))

[0111] Among them, Permute() represents the rotation operation;

[0112] Step 3-2-1 For the rotated F H′ , calculate its attention weight A H′ , the calculation formula is as follows:

[0113] A H′ =σ(Conv(F H′ ))

[0114] Among them, σ represents the sigmoid activation function, Conv represents the convolution operation, and the attention weight A H′ As shown below: H′ =[[0.4625304 0.47821882 0.5064253 0.5032532 0.5093386 0.49877496 0.504263460.50664175 0.5053692 0.5012332 0.5004162 0.49784237 0.505688 0.50111420.5008647 0.4942028 0.49796405 0.4991069 0.5064934 0.4963113 0.51026870.50125698 0.5024856 0.49675384 0.49365704 0.5027958 0.5076529 0.508145230.51006527 0.50361942 0.50422731 0.4635842]…]

[0115] Step 3-2-2 generates the attention weight A H′ Applied to F H′ Perform weighted fusion, rotate the output 90° clockwise along the H axis to keep the same shape as the input, and obtain the final feature representation F′ H′ , the calculation formula is as follows:

[0116] F′ H′ =F H′ ⊙A H′

[0117] Among them, ⊙ represents the weighted operation, and the feature represents F′ H′ As shown below: F′ H′=[[4.77198141 -6.7637410-0.544546 0.43041419 3.537576 -2.778062 0.9133042 -6.160861 -0.479400073.9928354 0.8046306 -0.17706413 -1.0724556 -5.6582732 2.32978193 3.46658083.6545703 -3.5565758 -2.44673312 5.7698212 0.15064502 1.7720605 -7.12969924.42422149 -2.1723745 4.765295 3.3866224 4.0162743 -0.50122583 4.03275693.6713355 0.25896169]…];

[0118] Step 3-3 In the second branch, rotate F(x,y) 90° counterclockwise along the W axis to obtain F W′ , the calculation formula is as follows:

[0119] F W′ =Permute(F W )

[0120] Among them, Permute() represents the rotation operation;

[0121] Step 3-3-1 For the rotated F W′ , calculate its attention weight A W′ , the calculation formula is as follows:

[0122] A W′ =σ(Conv(F W′ ))

[0123] Among them, σ represents the sigmoid activation function, Conv represents the convolution operation, and the attention weight A W′ As shown below: W′=[[0.4625304 0.47821882 0.5064253 0.5032532 0.5093386 0.49877496 0.504263460.50664175 0.5053692 0.5012332 0.5004162 0.49784237 0.505688 0.50111420.5008647 0.4942028 0.49796405 0.4991069 0.5064934 0.4963113 0.51026870.50125698 0.502524856 0.49675384 0.49365704 0.5027958 0.5076529 0.508145230.51006527 0.50361942 0.50422731 0.46358421]…]

[0124] Step 3-3-2 generates the attention weight A W′ Applied to F W′ Perform weighted fusion, rotate the output 90° clockwise along the W axis to keep the same shape as the input, and obtain the final feature representation F′ W′ , the calculation formula is as follows:

[0125] F′ W′ =F W′ ⊙A W′

[0126] Among them, ⊙ represents the weighted operation, and the feature represents F′ W′ As shown below: F′ W′ =[[-0.3459873

[0127] -0.3557174-0.3563259-0.3565601-0.3572626-0.360821-0.3672708-0.37664406-0.3943893-0.45269319-2.85039903 2.9823452 1.26611581.03112015.848585 3.8677542-1.64920701 3.8796155 4.77871008-3.3944633 3.4810942-0.5368205-7.878970 1.6911377 8.2825433 1.04997415.956989 1.6831661 1.247368681.89665401 1.95309573 2.61062604]…];

[0128] Step 3-4 In the third branch, rotate F(x,y) 90° counterclockwise along the C axis to obtain F C′ , the calculation formula is as follows:

[0129] F C′ =Permute(F C )

[0130] Among them, Permute() represents the rotation operation;

[0131] Step 3-4-1 For the rotated F C′ , calculate its attention weight A C′ , the calculation formula is as follows:

[0132] A C′ =σ(Conv(F C′ ))

[0133] Among them, σ represents the sigmoid activation function, Conv represents the convolution operation, and the attention weight A C′ As shown below: C′ =[[0.0323739 0.04996519 0.0190248 0.032744970.03221277 0.00206719 0.040750380.01613641 0.03390235 0.040246490.03553107 0.00632962 0.03442683 0.045882910.01900478 0.021441210.03710039 0.03861086 0.05596253 0.0439686 0.036119210.048507760.00716817 0.02596463 0.00525256 0.03657651 0.02809189 0.03490375 0.04528182 0.03938764 0.00690786 0.04449471]…]

[0134] Step 3-4-2 generates the attention weight A C′ Applied to F C′ Perform weighted fusion, rotate the output 90° clockwise along the C axis to keep the same shape as the input, and obtain the final feature representation F′ C′ , the calculation formula is as follows:

[0135] F′ C′ =F C′ ⊙A C′

[0136] Among them, ⊙ represents the weighted operation, and the feature represents F′ C′ As shown below: F′ C′ =[[-0.3485827-0.3523013-0.3565928-0.3561681-0.3552799-0.35704249-0.357736-0.3434403-0.35661432-0.3946474-0.250659806-0.290766809-0.3384064-0.3242772-0.2823318-0.1 5220749-0.31063866-0.34583302-0.3499622-0.3363541-0.21726081-0.26183593-0.2960491-0.2852742-0.2441485-0.12823562-0.272830445-0.303783197-0.307941853-0.29709096]…];

[0139] Step 3-5 represents the feature F′ H′ , F′ W′ and F′ C′ Perform weighted summation to obtain the final feature representation F′. The calculation formula is as follows:

[0140] F′=F′ H′ +F′ W′ +F′ C′

[0141] The feature representation F′ is as follows: F′ = [[5.81147151 1.51588341 1.757212122.100594341.62918585 1.54040498 1.44374291 8.71880878 5.267586574.14263371 7.176068446.22190594 1.11205845 7.69002490 2.181827302.29565939 9.32599080 1.108053271.40071468 2.58543039 5.679342201.06132064 9.10082146 4.02163211 2.978837782.45860956 4.020669282.84401506]…]

[0142] Step 4: Build a neural network model, such as Figure 2 As shown, add the SK attention module. The specific steps are:

[0143] Step 4-1: For the neural network model used to extract features, the SK attention module is serially added after the triple attention module. For the feature representation F′, multiple convolution operations of different scales are first performed on F′. The calculation formula is as follows:

[0144] U=Conv k (F′)

[0145] Among them, k represents the size of the convolution kernel, and a 5x5 convolution kernel is selected to implement the convolution operation. k () represents the convolution operation, and U represents the feature representation after the convolution kernel operation, as shown below: U = [[0.07231558 0.79005039 -0.13183269 -0.43141243 -0.68138295 -0.21127866 0.41436425 -0.43990028 -0.18998088 -0.3895573 0.22411817 -0.13861802 0.41436425 0.28807738 0.042108630.00457299 -0.2547403 -0.21260692 0.0931876 -0.88285303 0.09628438 -0.43763191 0.70846903 0.05549512 -0.15196016 -0.40317017 0.09173188 -0.01785285 0.37077516 0.00979392 0.28809568 -0.06224051-0.10304005 0.19141217-0.53525835 0.09173188 -0.25540993 -0.76822001 0.09173188]…]

[0146] Step 4-2 In order to capture global context information, the SK attention module performs global average pooling on the fused feature representation U to generate a channel vector Z. The calculation formula is as follows:

[0147] Z = GlobalAvgPool(U)

[0148] Among them, GlobalAvgPool() represents the global average pooling operation;

[0149] Step 4-3 inputs the channel vector Z into a shared multi-layer perceptron to generate the attention weight a of each channel. The calculation formula is as follows:

[0150] a=σ(W2·δ(W1·z))

[0151] Wherein, W1 and W2 represent the weights of the multilayer perceptron, δ represents the activation function, σ represents the normalization function, and the attention weight a is as follows: a=[[0.1714792 0.29500302 0.0090548 0.94925319 0.047275260.20607563 0.31138499 0.0088927 0.97436589 0.0264371 0.20684094 0.340250470.01586761 0.97802223 0.02616946 0.1753158 0.22335101 0.01814083 0.95669079 0.06807654 0.13569873 0.16864851 0.01110923 0.98184971 0.08071729 0.20234119 0.19833302 0.00605363 0.97499235 0.08789535]…]

[0152] Step 4-4 performs weighted summation on each feature representation based on the generated weight a to obtain the final representation V. The calculation formula is as follows:

[0153] V=a·U

[0154] The final representation V is as follows: V = [[0.15862386 0.28829003 0.01312718 0.95847013 0.06734496 0.13799926 0.20600414 0.00956759 0.99509772 0.05586494 0.15297039 0.28244306 0.02098287 0.9993007 0.03863685 0.11919519 0.16943096 0.01162596 0.91608276 0.06349515 0.14724827 0.22263547 0.01479444 0.98834798 0.073501210.14601764 0.20241677 0.0125573 0.94251468 0.04981716 0.16672565 0.266051180.01429782 0.98585538 0.02710078 0.17101579 0.2162787 0.01032009 0.999664950.02934226 0.16518528 0.23298483 0.01582752 0.93151943 0.06988643 0.15006842 0.25557972 0.01182973 0.9129331 0.04067946 0.14315967 0.20726701 0.00603898 0.97645555 0.03250151 0.13701123 0.18858687 0.01122402 0.99024566 0.02169669]…]

[0155] Step 5: Build a neural network model, such as Figure 2 As shown, the GAT module is used to extract the graph structure information of the view. The specific steps are:

[0156] Step 5-1 For the feature representation F′, define a feature vector V i , a shared learnable linear transformation is applied to each node, calculated as follows:

[0157] g i =WV i

[0158] Where W represents the learnable weight matrix, as shown below: W = [[7.2315581e-02 7.9005039e-01-1.3183269e-01 0.0000000e+00 9.3348004e-04 0.0000000e+00 1.7147920e-012.9500303e-01 9.0547958e-03 9.4925320e-01 4.7275260e-02 -4.3141243e-01 -6.8138295e-01 -2.1127866e-01 -2.0769592e-03 -1.9165310e-04 -1.9159351e-042.0607562e-01 3.1138498e-01 8.8926954e-03 9.7436589e-01 2.6437102e-024.1436425e-01 -4.3990028e-01 -1.8998088e-01 2.2946466e-03-1.3334538e-042.7861909e-04 2.0684095e-01 3.4025046e-01 1.5867610e-02 9.7802222e-012.6169458e-02 -3.8955730e-01 2.2411817e-01-1.3861802e-01 1.3897475e-048.2097109e-04 1.5377726e-03 1.7531580e-01 2.2335102e-01 1.8140826e-029.5669079e-01 6.8076544e-02 0.41436425 0.28807738 0.04210863 0.0009845 0-00.13569874 0.16864851 0.01110923 0.98184973 0.0807173 4.5729941e-03 -2.5474030e-01 -2.1260692e-01 -2.1955887e-04 1.7711463e-03 6.3111584e-052.0234118e-01 1.9833302e-01 6.0536270e-03 9.7499233e-01 8.7895356e-029.3187600e-02-8.8285303e-01 9.6284382e-02 -1.2297974e-03 7.9864333e-046.5512408e-04 1.5862386e-01 2.8829002e-01 1.3127178e-02 9.5847011e-016.7344956e-02 -4.3763191e-01 7.0846903e-01 5.5495117e-02 -2.9664545e-05 -8.7874112e-05 1.9969464e-04 1.3799927e-01 2.0600414e-01 9.5675886e-039.9509770e-01 5.5864941e-02 -1.51960164e-01 -4.03170168e-01 9.17318761e-021.10979316e-04 1.66487036e-04-1.24659913e-04 1.52970389e-01 2.82443076e-012.09828727e-02 9.99300718e-01 3.86368521e-02 -1.7852854e-02 3.7077516e-019.7939204e-03 1.1542997e-03 -3.3617453e-04 1.5574096e-04 1.1919519e-011.6943096e-01 1.1625960e-02 9.1608274e-01 6.3495152e-02 2.8809568e-01 -6.2240515e-02 9.7949460e-02 -0.0000000e+00 0.0000000e+00 -3.1423377e-041.4724827e-01 2.2263546e-01 1.4794442e-02 9.8834795e-01 7.3501207e-021.3822211e-01 5.6207303e-02-1.4967741e-01 -2.2481794e-04 -1.4058125e-03 -1.3382357e-04 1.4601764e-01 2.0241676e-01 1.2557301e-02 9.4251466e-014.9817160e-02 2.38039225e-01 6.07620418e-01 -6.70971796e-02 2.47978693e-04…].

[0159] Step 5-2 uses the attention function to calculate the original attention coefficient. The calculation formula is as follows:

[0160]

[0161] Among them, e ij represents the attention score from node j to node i, g i and g j Represents node i and node j Embedding, a represents the weight vector, and LeakyReLU() represents the activation function;

[0162] Step 5-3 normalizes all adjacent nodes of a node to obtain a normalized attention coefficient. The calculation formula is as follows:

[0163]

[0164] Among them, k∈N(i) represents all neighbor nodes connected to node i, α ij Represents the attention weight from node j to node i, as follows: α ij＝[[-0.8831216 -1.6897167 -2.3271856 0.5100326 13.42816452.3753273 -3.7631557 -0.97254246 -1.0438994 -2.0199893 -3.2511644 1.8822671 -3.9031086 2.933107 -1.7977371 0.5887007 -0.32918605 -0.96528184 -0.5948427-0.22485356 -1.691672 1.890091 -4.4370275 -1.3703587 -0.40264282 0.180496720.04062188 1.5536369 1.1114076 -1.3302454 -4.2111163 0.48550856 1.8495548 -0.25803518 -1.3284997 -3.0927162 3.3872619 0.16522811 4.164241 1.452566913.415513 0.6526762 -1.228251 -1.335132 -0.0290454 -2.2444367 1.13467450.14350769 0.5901849 -0.16744955 1.216979 -0.7608436 -1.4771589 0.96823364 -1.0448548 -0.6969638 -1.3235904 1.6367483 -0.3136755 1.4010174 -1.7071642 -0.79333377 -1.0764011 -0.99529624 0.9910239 -0.825001 -0.65857124 -2.36866350.40923834 0.14403646 -1.200581 1.0473725 1.8566219 0.6605684 1.60579280.03514231-1.9293506 -0.8340851 -0.37376013 0.4215227 -0.6050646 2.017055312.774362 -0.5791482 -1.5929656 -3.3783772 -0.95785636 3.926893 1.19568091.3766061 -1.5611833 -3.3548992 0.91985244 -1.0210743 1.7068307 -2.28401611.1651654 2.2388122 1.9091283 -1.9916228 0.98647463 -4.84623 -2.9944191.379924 -2.6909823 0.73469806 -1.2586002 1.0338769 0.5655411 -0.6225980.76023364 -2.098474 -2.0398748 1.2077656 2.6231623 0.65575266 0.54162765 -2.7446504 -2.6876256 -2.3410416]…].

[0165] Step 5-4 uses the standardized attention coefficient to perform weighted summation of the features of adjacent views. The calculation formula is as follows:

[0166]

[0167] Among them, σ represents the normalization function, S iThe summation result is as follows: [[6.618561652818589 -1.1647728093067984 -10.486749836547618 0.1519479288810843 0.164396019860760721.2080140986170955 -0.34331302722330026 -4.112129864902629 -2.8828319406699703 -0.5196450890211213 -7.98606146085749 0.48123702861344886-6.127844405613091 3.8357137736429037 -2.173754356963368 1.2660843919061966 -4.919631249887141 -5.128360748083804 -1.9071944240581218 -0.51964508902112131.2092708726918542 -0.9227561263021111 7.696135644482613 4.802492685405509 -7.180920637443266 -3.9065770434182223 -0.6209995267247632 5.9239294597013230.2358944807600696 -0.5196450890211213]…]

[0168] Step 6: Extract training data and test data from ModelNet40. The specific steps are:

[0169] Step 6-1 projects the three-dimensional training model in ModelNet40 to obtain training data of a two-dimensional view;

[0170] Step 6-2 projects the three-dimensional test model in ModelNet40 to obtain the test data of the two-dimensional view.

[0171] Step 7 uses the training data to optimize the network model, and uses the optimized network model to process the test data. After convolution, pooling, and fusion, the classification results are obtained in the output layer. The specific steps are as follows:

[0172] Step 7-1 Use the training data to optimize the network model, input the two-dimensional view in the test data into the optimized network model, and output the result S after convolution and pooling. i ;

[0173] Step 7-2: S i Input the Softmax classifier to get the classification result, as shown below:

[0174] V i =Softmax(W C S i )

[0175] Among them, Softmax() represents the maximum classification operation, W C Represents the weight matrix of the classification layer, V i Represents the classification probability vector of the 3D model. Figure 1 The classification result of the 3D model shown is airplane class, and the classification accuracy of the test data is as follows:

[0176] Classification accuracy = 1095 / 1200 ≈ 91.26%

[0177] Triple Attention-Based Multi-View in the Implementation of the Present Invention Figure 3 The three-dimensional model classification method projects the three-dimensional model into a two-dimensional view, uses the binary contour method to render the two-dimensional view, constructs a neural network model that combines the triplet attention module and the GAT module, and combines the Softmax function to determine the category of the three-dimensional model with high accuracy.

[0178] The above is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, and the specific implementation methods herein are only used to help understand the method of the present invention. For those of ordinary skill in the art, according to the idea of ​​the present invention, changes and modifications can be made in the specific implementation methods and application scopes, so the present invention should not be understood as limiting the present invention.

Claims

1. A multi-view 3D model classification method based on triplet attention, characterized in that: The method mainly includes the following steps: Step 1: Project the 3D model in Modelnet40 to obtain a 2D view set. Step 2: Render the 2D view set using data augmentation methods for binary contours. Step 3: Use the triplet attention module to perform feature extraction on the 2D view set. Step 4: Construct a neural network model and add the SK attention module. Step 5: Use the GAT module to process the view features into a graph structure. Step 6: Extract training data and test data from ModelNet40. Step 7: Use the training data to optimize the model, and use the optimized model to process the test data. After convolution, pooling, and fusion, the classification result is obtained in the output layer.

2. The multi-view 3D model classification method based on triplet attention according to claim 1, characterized in that: In step 1, the three-dimensional model is preprocessed, and the specific steps are: Step 1-1: Fix the 3D model at the center of the virtual sphere and select a fixed angle and amount for projection; Step 1-2 places the virtual camera above the 3D model. The camera rotates a circle and constructs a set of 2D view sets V(M) at 30-degree projections per step, as shown below: V(M)={V x |V1,V2,…,V 12 },V x ∈R H×W×C Among them, M represents the three-dimensional model, V x Represents a 2D view of a 3D model.

3. The multi-view 3D model classification method based on triplet attention according to claim 1, characterized in that: In step 2, the two-dimensional view set is rendered using a data enhancement method for binary contours, and the specific steps are as follows: Step 2-1 selects an image V(x,y) from the two-dimensional view set V(M) as the input of the original image; Step 2-2 uses the global threshold method to convert V(x, y) into a binary image B(x, y), selects a global threshold T, sets the pixels with grayscale values ​​greater than the threshold to 1, and sets the pixels with grayscale values ​​less than the threshold to 0, as shown below: Among them, V(x,y) represents the original image, B(x,y) represents the binary image, and T represents the threshold.

4. The method for multi-view 3D model classification based on triplet attention according to claim 1, characterized in that: In step 3, the view features are extracted using a triplet attention module, and the specific steps are as follows: Step 3-1 defines a three-branch module and takes the binary image B(x,y) as the input F(x,y) of the module; Step 3-2 In the first branch, rotate F(x,y) 90° counterclockwise along the H axis to obtain F H′ , the calculation formula is as follows: F H′ =Permute(F(x,y)) Among them, Permute() represents the rotation operation; Step 3-2-1 For the rotated F H′ , calculate its attention weight A H′ , the calculation formula is as follows: A H′ =σ(Conv(F H′ )) Among them, σ represents the sigmoid activation function, and Conv represents the convolution operation; Step 3-2-2 generates the attention weight A H′ Applied to F H′ Perform weighted fusion, rotate the output 90° clockwise along the H axis to keep the same shape as the input, and obtain the final feature representation F′ H′ , the calculation formula is as follows: F′ H′ =F H′ ⊙A H′ Among them, ⊙ represents weighted operation; Step 3-3 In the second branch, rotate F(x,y) 90° counterclockwise along the W axis to obtain F W′ , the calculation formula is as follows: F W′ =Exchanges(F W ) Among them, Permute() represents the rotation operation; Step 3-3-1 For the rotated F W′ , calculate its attention weight A W′ , the calculation formula is as follows: A W′ =σ(Conv(F W′ )) Among them, σ represents the sigmoid activation function, and Conv represents the convolution operation; Step 3-3-2 generates the attention weight A W′ Applied to F W′ Perform weighted fusion, rotate the output 90° clockwise along the W axis to keep the same shape as the input, and obtain the final feature representation F′ W′ , the calculation formula is as follows: F′ W′ =F W′ ⊙A W′ Among them, ⊙ represents weighted operation; Step 3-4 In the third branch, rotate F(x,y) 90° counterclockwise along the C axis to obtain F C′ , the calculation formula is as follows: F C′ =Exchanges(F C ) Among them, Permute() represents the rotation operation; Step 3-4-1 For the rotated F C′ , calculate its attention weight A C′ , the calculation formula is as follows: A C′ =σ(Conv(F C′ )) Among them, σ represents the sigmoid activation function, and Conv represents the convolution operation; Step 3-4-2 generates the attention weight A C′ Applied to F C′ Perform weighted fusion, rotate the output 90° clockwise along the C axis to keep the same shape as the input, and obtain the final feature representation F′ C′ , the calculation formula is as follows: F′ C′ =F C′ ⊙A C′ Among them, ⊙ represents weighted operation; Step 3-5 represents the feature F′ H′ , F′ W′ and F′ C′ Perform weighted summation to obtain the final feature representation F′. The calculation formula is as follows: F′=F′ H′ +F′ W′ +F′ C′ 5. The multi-view 3D model classification method based on triplet attention according to claim 1, characterized in that: In step 4, a neural network model is designed and an SK attention module is added. The specific steps are as follows: Step 4-1: For the neural network model used to extract features, the SK attention module is serially added after the triple attention module. For the feature representation F′, multiple convolution operations of different scales are first performed on F′. The calculation formula is as follows: U=Conv k (F′) Among them, k represents the convolution kernel size, Conv k () represents the convolution operation, and U represents the feature representation after the convolution kernel operation; Step 4-2 In order to capture global context information, the SK attention module performs global average pooling on the fused feature representation U to generate a channel vector Z. The calculation formula is as follows: Z = GlobalAvgPool(U) Among them, GlobalAvgPool() represents the global average pooling operation; Step 4-3 inputs the channel vector Z into a shared multi-layer perceptron to generate the attention weight a of each channel. The calculation formula is as follows: a=σ(W2·δ(W1·z)) Among them, W1 and W2 represent the weights of the multi-layer perceptron, δ represents the activation function, and σ represents the normalization function; Step 4-4 performs weighted summation on each feature representation based on the generated weight a to obtain the final representation V. The calculation formula is as follows: V=a·U 6. The method for multi-view 3D model classification based on triplet attention according to claim 1, characterized in that: In step 5, the GAT module is used to extract the graph structure information of the view, and the specific steps are as follows: Step 5-1 For the feature representation V, define a feature vector V i , a shared learnable linear transformation is applied to each node, calculated as follows: g i =WV i Where W represents the learnable weight matrix; Step 5-2 uses the attention function to calculate the original attention coefficient. The calculation formula is as follows: Among them, e ij represents the attention score from node j to node i, g i and g j Represent the embedding of node i and node j respectively, a represents the weight vector, and LeakyReLU() represents the activation function; Step 5-3 normalizes all adjacent nodes of a node to obtain a normalized attention coefficient. The calculation formula is as follows: Among them, k∈N(i) represents all neighbor nodes connected to node i, α ij Represents the attention weight from node j to node i; Step 5-4 uses the standardized attention coefficient to perform weighted summation of the features of adjacent views. The calculation formula is as follows: Where σ represents the normalization function S i Indicates the summation result.

7. The method for multi-view 3D model classification based on triplet attention according to claim 1, characterized in that In step 6, training data and test data are extracted from ModelNet40, and the specific steps are as follows: Step 6-1 projects the three-dimensional training model in ModelNet40 to obtain training data of a two-dimensional view; Step 6-2 projects the three-dimensional test model in ModelNet40 to obtain the test data of the two-dimensional view.

8. The method for multi-view 3D model classification based on triplet attention according to claim 1, characterized in that In step 7, the training data is used to optimize the network model, and the optimized network model is used to process the test data. After convolution, pooling, and fusion, the classification result is obtained in the output layer. The specific steps are: Step 7-1 Use the training data to optimize the network model, input the two-dimensional view in the test data into the optimized network model, and output the result S after convolution and pooling. i ; Step 7-2: S i Input the Softmax classifier to get the classification result, as shown below: V i =Softmax(W C S i ) Among them, Softmax() represents the maximum classification operation, W C Represents the weight matrix of the classification layer, V i Represents the classification probability vector of the 3D model.

Images