A multi-view three-dimensional model classification method fusing ResNet and CoTNet

Abstract

The present application relates to a kind of multi-view three-dimensional model classification methods of fusing ResNet and CoTNet.The present application is first projected to three-dimensional model, obtains two-dimensional view set.CBAM attention module and SENet attention module are added in 2DResNet model to enhance the learning ability of network.2DResNet-CoTNet network is constructed based on 2DResNet model, and 2DResNet is combined with 2DCoTNet in parallel, view feature is extracted from two-dimensional view.Shape descriptor D1, D2, D3, Zernike moment and Fourier descriptor are used to describe two-dimensional view, and 1DResNet is used to extract shape feature.View feature and shape feature are fused, and finally Softmax function is used to determine the category of three-dimensional model.The present application has good three-dimensional model classification effect, can more accurately solve three-dimensional model classification problem.

Description

Technical Field

[0001] This invention relates to a multi-view method that integrates ResNet and CoTNet. Figure 3 This method is a 3D model classification method, which has been widely used in the field of 3D model classification. Background Technology

[0002] With the improvement of knowledge systems and the continuous development of technology, the number of various application software is also growing rapidly, including 3D modeling software. In the past, people mainly obtained information through images, videos, and audio. Now, 3D models are gradually entering people's daily lives. Researchers use software and technology to model target objects in the virtual or real world and persistently store them, thus forming 3D models of various shapes. 3D models can not only display the shape and contour information of objects, but also contain rich spatial structural information, thus becoming a popular multimedia data type in recent years. As scientists have successively proposed various excellent machine learning algorithms, 3D model recognition and retrieval technology has also been widely used. In recent years, the number of 3D models in model libraries has become enormous, their structures complex, and their types diverse. How to accurately find the target model from the numerous categories in the model library has become crucial. Research on 3D model classification methods is of great significance and has become a hot topic.

[0003] In recent years, the Transformer has ushered in a new era in the field of natural language processing. With the emergence of new Transformer-based architectures, many computer vision tasks have achieved excellent results. This invention introduces a novel Transformer module, the Contextual Transformer (CoT), which fully utilizes the contextual information between input keys to guide the learning of dynamic attention matrices, thereby enhancing feature representation capabilities. Based on the ResNet convolutional neural network, a multi-view approach integrating ResNet and CoTNet is proposed. Figure 3 The 2D model classification method consists of ResNet and CoTNet in parallel. It extracts 2D view features and then fuses them with 1D shape features, which further improves the model classification accuracy. Summary of the Invention

[0004] To address the 3D model classification problem in the field of computer vision, this invention discloses a multi-view algorithm that integrates ResNet and CoTNet. Figure 3 3D model classification method.

[0005] Therefore, the present invention provides the following technical solution:

[0006] 1. Multi-view integration of ResNet and CoTNet Figure 3A 3D model classification method, characterized by the following main steps:

[0007] Step 1: Project the 3D model in Modelnet10 to obtain a 2D view set.

[0008] Step 2: Use the D1 descriptor to describe the distance between the centroid of the 2D view and the random sampling point on the boundary, use the D2 descriptor to describe the distance between two random sampling points on the boundary of the 2D view, and use the D3 descriptor to describe the square root of the area of ​​the triangle formed by the three random sampling points on the boundary of the 2D view.

[0009] Step 3: Extract the Zernike moments of the 2D view and normalize them to the (0,1) range using the standard moment method.

[0010] Step 4: Obtain the outline view of the two-dimensional view, extract its one-dimensional Fourier operator, divide by the DC component and normalize to the (0,1) range.

[0011] Step 5: Use the S1 descriptor to describe the compactness of the 2D view boundary, and use the S2 descriptor to describe the perimeter of the quadrilateral enclosed by the four random sampling points on the 2D view boundary.

[0012] Step 6: Extract the 2D view of the 3D model in ModelNet10, and extract the D1, D2, D3, Zernike moments, Fourier operators, S1, and S2 shape descriptors of the 2D view to form training and test data.

[0013] Step 7: Design the 2DResNet network, combine it in parallel with 2DCoTNet to obtain the 2DResNet-CoTNet network, and design the 1DResNet network.

[0014] Step 8: Optimize 2DResNet-CoTNet and 1DResNet using the training data. Process the test data using the optimized 2DResNet-CoTNet and 1DResNet. After convolution, pooling, and fusion, obtain the classification result in the output layer.

[0015] 2. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3 The 3D model classification method is characterized by preprocessing the 3D model in step 1, specifically the following steps:

[0016] Step 1-1: Fix the 3D model at the center of the virtual sphere and select a fixed angle and number of projections;

[0017] Step 1-2: Place the virtual camera above the 3D model. Rotate the camera around the entire circle, projecting 60 degrees at each step to obtain a set of 2D views, denoted as V1, V2, V3, V4, V5, and V6.

[0018] 3. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3 The dimensional model classification method is characterized by extracting the D1, D2, and D3 features of the view in step 2, specifically through the following steps:

[0019] Step 2-1 uses the D1 descriptor to describe the distance between the centroid of the 2D view and random sampling points on the boundary. The specific steps are as follows:

[0020] Step 2-1-1 involves randomly sampling points at equal intervals on the boundary of the two-dimensional view. The resulting point set is Point={(x1,y1),…,(x i ,y i ),…,(x m ,y m )}, where (x i ,y i () are the coordinates of a point;

[0021] Step 2-1-2: Randomly select N points from Point to form a set PD1={ai1,ai2,…,ai...} N}, where Bins represents the number of intervals and BinsSize represents the length of the interval, as shown below:

[0022]

[0023] Where dist() represents the Euclidean distance between two points, O represents the centroid of the 2D view, P1 represents a point on the view boundary, and max() represents taking the maximum value.

[0024] Step 2-1-3 uses the D1 descriptor to describe the distance between the centroid of the 2D view and random sampling points on the boundary, D1_v i The calculation formula is as follows:

[0025]

[0026] Step 2-2 uses the D2 descriptor to describe the distance between two random sampling points on the boundary of the two-dimensional view. The specific steps are as follows:

[0027] Step 2-2-1: Randomly select N pairs of points from Point to form a set PD2={(ai1,bi1),(ai2,bi2),…,(ai1,bi1),…,(ai2,bi2 ... N ,bi NBins represents the number of intervals, and BinsSize represents the length of the interval, as shown below:

[0028]

[0029] Where dist() represents the Euclidean distance between two points, P1 and P2 represent points on the view boundary, and max() represents taking the maximum value.

[0030] Step 2-2-2 uses the D2 descriptor to describe the distance between two random sampling points on the boundary of the two-dimensional view, D2_v i The calculation formula is as follows:

[0031] Steps 2-3 use the D3 descriptor to describe the square root of the area of ​​the triangle formed by three random sampling points on the boundary of the two-dimensional view. The specific steps are as follows:

[0032] Step 2-3-1: Randomly select N points from Point to form a set PD3 = {(ai1, bi1, ci1), (ai2, bi2, ci2), ..., (ai...} N ,bi N ,ci N Bins represents the number of intervals, and BinsSize represents the length of the interval, as shown below:

[0033]

[0034] Here, herson() represents the Helen formula, max() represents taking the maximum value, and P1, P2, and P3 represent points on the view boundary;

[0035] Step 2-3-2 uses the D3 descriptor to represent the square root of the area of ​​the triangle formed by three random sampling points on the boundary of the two-dimensional view, D3_v i The calculation formula is as follows:

[0036]

[0037] Where a = dist(P1,P2), b = dist(P1,P3), c = dist(P2,P3), dist() represents the Euclidean distance between the two points, and s represents the area of ​​the triangle. The calculation formula is as follows:

[0038]

[0039] 4. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3The dimensional model classification method is characterized by the following steps in step 3: extracting the Zernike moments of the view.

[0040] Step 3-1: Define the Zernike moment as the Zernike polynomial inside the unit circle, and define a set of complex-valued functions W on the unit circle. nm Let (x, y) represent any square-integrable function within the unit circle, calculated as follows:

[0041]

[0042] Where ρ represents the vector length from the origin to the point (x,y), θ is the angle between vector ρ and the counterclockwise direction of the x-axis, ϳ is the complex unit, n and m are the dimensions of the Zernike moments, and T is the Zernike polynomial. nm (ρ) is shown below:

[0043]

[0044] The Zernike moments of the two-dimensional view v(x,y) in step 3-2 are shown below:

[0045]

[0046] 5. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3 The dimensional model classification method is characterized in that, in step 4, the Fourier operator of the view is extracted and normalized to the (0,1) range using the standard moments method. The specific steps are as follows:

[0047] Step 4-1 performs a Fourier transform on the 2D view. Before performing the Fourier transform, it needs to be converted into a one-dimensional shape representation. The centroid distance function is adopted, which is achieved by calculating the distance from the outer contour boundary points to the centroid. According to the definition of the centroid, the Fourier coefficients Q(n) are obtained by the discrete Fourier transform, as shown in the following formula:

[0048]

[0049] Where j is the complex unit, {(x i ,y i Let |i=1, 2, …, N} be the boundary points of the contour;

[0050] Step 4-2: Dividing the Fourier coefficients Q(n) by the first coefficient Q(0) yields the Fourier operator, as shown below:

[0051]

[0052] 6. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3 The 2D model classification method is characterized by extracting S1 and S2 features from the 2D view in step 5, specifically as follows:

[0053] Step 5-1 uses the S1 descriptor to describe the compactness of the 2D view contour, extracting the perimeter and area of ​​the 2D view contour. The calculation process of the S1 feature is as follows:

[0054]

[0055] Where l represents the perimeter of the view outline and s represents the area of ​​the view;

[0056] Step 5-2 uses the S2 descriptor to describe the perimeter of the quadrilateral formed by four random sampling points on the boundary of the two-dimensional view. The specific steps are as follows:

[0057] Step 5-2-1: Randomly select N points from Point to form a quadruple set PS2={(ai1,bi1,ci1,di1),(ai2,bi2,ci2,di2), …, (ai N ,bi N ,ci N ,di N Bins represents the number of intervals, and BinsSize represents the length of the interval, as shown below:

[0058]

[0059] Where max() means taking the maximum value, C represents the perimeter of the quadrilateral, and P1, P2, P3, and P4 represent points on the view boundary;

[0060] Step 5-2-2 uses the S2 descriptor to represent the perimeter of the quadrilateral formed by four random sampling points on the boundary of the two-dimensional view, S2_v i The calculation process is as follows:

[0061]

[0062] Where u=dist(P1,P2), v=dist(P1,P3), x=dist(P3,P4), y=dist(P2,P4), and dist() represents the Euclidean distance between the two points.

[0063] 7. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3The dimensional model classification method is characterized by extracting training and test data from ModelNet10 in step 6, specifically through the following steps:

[0064] Step 6-1 Project the 3D training model in ModelNet10 to obtain a 2D view, extract the D1, D2, D3, Zernike moments, Fourier operators, S1, and S2 descriptors of the 2D view, and combine them with the 2D view to form training data;

[0065] Step 6-2 Project the 3D test model in ModelNet10 to obtain a 2D view, extract the D1, D2, D3, Zernike moments, Fourier operators, S1, and S2 descriptors from the 2D view, and combine them with the 2D view to form test data.

[0066] 8. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3 The dimensional model classification method is characterized by the following steps in step 7: designing a 2DResNet network, combining it in parallel with a 2DCoTNet to obtain a 2DResNet-CoTNet network, and designing a 1DResNet network.

[0067] Step 7-1: 2DResNet contains 5 convolutional layers: conv_1, conv_2, conv_3, conv_4, and conv_5. A CBAM attention module is added between conv_5 and the average pooling layer, and a SENet attention module is added sequentially afterward.

[0068] Step 7-2 constructs the 2DCoTNet model, which consists of three 1×1 convolutions, key values ​​(K), query values ​​(Q), variable values ​​(V), weight values ​​(W), and two dot product operations. The dot product of key value K and query value Q yields weight value W, which reflects the local context representation of the input. Then, the dot product of weight value W and variable value V yields the global context representation. Finally, the output of the model is obtained.

[0069] Step 7-3 combines the improved 2DResNet model with the 2DCoTNet model in parallel to form the 2DResNet-CoTNet convolutional neural network model.

[0070] Step 7-4: 1DResNet consists of 5 convolutional layers, activation functions, max pooling layers, and average convolutional layers. The extracted D1, D2, D3, Zernike moments, Fourier operators, and S1 and S2 shape descriptors are input into the 1DResNet convolutional neural network and fused before output.

[0071] 9. The multi-view method that integrates ResNet and CoTNet according to claim 1 Figure 3 The dimensional model classification method is characterized by the following steps in step 8: 2DResNet-CoTNet and 1DResNet are optimized using training data; the optimized 2DResNet-CoTNet and 1DResNet are then used to process the test data; after convolution, pooling, and fusion, the classification result is obtained in the output layer. The specific steps are as follows:

[0072] Step 8-1: Optimize 2DResNet-CoTNet using training data. Input the two-dimensional view from the test data into the optimized 2DResNet-CoTNet, and output the feature F after convolutional pooling.

[0073] Step 8-2: Optimize 1DResNet using training data. Input the D1, D2, D3, Zernike moments, Fourier operators, S1, and S2 descriptors from the test data into the optimized 1DResNet. After convolutional pooling, output feature G.

[0074] Step 8-3 inputs features F and G into the fusion layer for fusion, as shown below:

[0075]

[0076] Step 8-4: Input X into the Softmax classifier to obtain the classification result, as shown below:

[0077]

[0078] Where Y = (y1, y2, …, y M W and B represent the parameters of the Softmax layer, and P(S) represents the parameters of the Softmax layer. i |m) represents the 3D model m in category S i The probability of occurrence of the following is calculated as follows:

[0079]

[0080] The class with the highest probability in the probability distribution is selected as the predicted class S of the 3D model m, as shown below:

[0081]

[0082] Beneficial effects:

[0083] This invention presents a multi-view method that integrates ResNet and CoTNet. Figure 3 3D model classification method.

[0084] 1. This invention improves the three-dimensional shape distribution features to make them applicable to two-dimensional views, using a shape distribution function to describe the shape information of a three-dimensional model view. For the geometry of a three-dimensional model, this invention integrates view features and shape features to comprehensively describe the three-dimensional model. The shape features of the three-dimensional model are obtained by fusing D1, D2, D3, Zernike, Fourier, S1, and S2 features, exhibiting good rotational robustness and effectively representing the geometric characteristics of the three-dimensional model.

[0085] 2. This invention introduces the CBAM attention module and the SENet attention module. These two attention modules can be seamlessly integrated into any CNN architecture for end-to-end training. This invention adds these two attention modules to 2DResNet to focus on capturing view information, reduce the impact of redundant parts, and improve the classification accuracy of the 3D model.

[0086] 3. This invention incorporates a Transformer-style module, namely the Contextual Transformer (CoT) block, for visual recognition. This design fully utilizes the contextual information between input keys to guide the learning process of the dynamic attention matrix, thereby enhancing visual representation capabilities. By combining the CoT module with the 2DResNet module in parallel, the learning ability of the network model is further improved.

[0087] 4. This invention integrates view features and shape features, proposing a parallel combination of ResNet and CoTNet for 3D model classification. It fuses view features extracted by 2DResNet-CoTNet and shape features extracted by 1DResNet to classify 3D models. Compared with traditional view-based convolutional neural networks, this method enriches and preserves the descriptive power of 3D models, improving the accuracy of 3D model classification.

[0088] 5. This invention uses the publicly available ModelNet10 dataset to validate the proposed 3D model classification method. The results show that the proposed classification method is effective. Attached Figure Description

[0089] Figure 1 This is an example of a three-dimensional model to be classified in an embodiment of the present invention;

[0090] Figure 2 This is a diagram of the 3D model classification framework that integrates ResNet and CoTNet in an embodiment of the present invention.

[0091] Figure 3 These are six angle views of a three-dimensional model obtained through a virtual camera in an embodiment of the present invention;

[0092] Figure 4 These refer to the four shape features D1, D2, D3, and S2 in the embodiments of the present invention. Detailed Implementation

[0093] To clearly and specifically describe the technical solutions in the embodiments of the present invention, a 3D model from the toilet class of the ModelNet10 test set is used as an example, such as... Figure 1 As shown, tests were conducted, and the test set contained 600 3D models in the toilet class. The invention will be further described in detail with reference to the accompanying drawings in the embodiments.

[0094] This invention implements a 3D model classification framework based on the fusion of ResNet and CoTNet, such as... Figure 2 As shown, it includes the following steps:

[0095] Step 1 involves preprocessing the 3D model. The specific steps are as follows:

[0096] Step 1-1 will Figure 1 The 3D model shown is fixed at the center of the virtual sphere, and a fixed angle of 60° and a projection number of 6 are selected for projection. Figure 3 As shown;

[0097] Steps 1-2 involve placing a virtual camera above the 3D model, rotating the camera one full circle, and projecting a set of 2D views V1, V2, V3, V4, V5, and V6 at 60-degree intervals, as shown below. Figure 3 As shown;

[0098] Step 2: Extract the D1, D2, and D3 features from the 3D model view. The specific steps are as follows:

[0099] Step 2-1 uses the D1 descriptor to describe the distance between the centroid of the 2D view and random sampling points on the boundary, such as... Figure 4 As shown, the specific steps are as follows:

[0100] Step 2-1-1 involves randomly sampling points at equal intervals on the boundary of the two-dimensional view. The resulting point set is Point={(x1,y1),…,(x i ,y i ),…,(x m ,y m )}, where (x i ,y i () are the coordinates of a point;

[0101] Step 2-1-2: Randomly select N points from Point to form a set PD1={ai1,ai2,…,ai...} N}, where Bins represents the number of intervals and BinsSize represents the length of the interval, as shown below:

[0102]

[0103] Where dist() represents the Euclidean distance between two points, O represents the centroid of the 2D view, P1 represents a point on the view boundary, and max() represents taking the maximum value.

[0104] Step 2-1-3 uses the D1 descriptor to describe the distance between the centroid of the 2D view and random sampling points on the boundary, D1_v i The calculation formula is as follows:

[0105]

[0106] from Figure 1 Extract the D1 feature from the 3D model view shown, D1=[[-0.25065980677170285,-0.2907668093480549,-0.3384064348130071,-0.3242772307360641,-0.2823318914308117,-0.15220749439027823,-0.3106386685966566,-0.34583302205963834,-0.3499622401134148,-0.3363541195004869]…];

[0107] Step 2-2: Use the D2 descriptor to describe the distance between two random sampling points on the boundary of the two-dimensional view, such as... Figure 4 As shown, the specific steps are as follows:

[0108] Step 2-2-1: Randomly select N pairs of points from Point to form a set PD2={(ai1,bi1),(ai2,bi2),…,(ai1,bi1),…,(ai2,bi2 ... N ,bi N Bins represents the number of intervals, and BinsSize represents the length of the interval, as shown below:

[0109]

[0110] Where dist() represents the Euclidean distance between two points, P1 and P2 represent points on the view boundary, and max() represents taking the maximum value.

[0111] Step 2-2-2 uses the D2 descriptor to describe the distance between two random sampling points on the boundary of the two-dimensional view, D2_v i The calculation formula is as follows:

[0112]

[0113] from Figure 1 Extract the D2 feature from the 3D model view shown, D2=[[-0.348582754939446,-0.3523013800446598,-0.3565928788310495,-0.35616814599616237,-0.3552799569323815,-0.35704249957918305,-0.3577363140574128,-0.3434403027525801,-0.35661432854682595,-0.39464747524442684]…];

[0114] Steps 2-3 use the D3 descriptor to describe the square root of the area of ​​the triangle formed by three random sampling points on the boundary of the two-dimensional view, such as... Figure 4 As shown, the specific steps are as follows:

[0115] Step 2-3-1: Randomly select N points from Point to form a set PD3 = {(ai1, bi1, ci1), (ai2, bi2, ci2), ..., (ai...} N ,bi N ,ci N Bins represents the number of intervals, and BinsSize represents the length of the interval, as shown below:

[0116]

[0117] Here, herson() represents the Helen formula, max() represents taking the maximum value, and P1, P2, and P3 represent points on the view boundary.

[0118] Step 2-3-2 uses the D3 descriptor to represent the square root of the area of ​​the triangle formed by three random sampling points on the boundary of the two-dimensional view, D3_v i The calculation formula is as follows:

[0119]

[0120] Where a = dist(P1,P2), b = dist(P1,P3), c = dist(P2,P3), dist() represents the Euclidean distance between the two points, and s represents the area of ​​the triangle, as shown in the formula below:

[0121]

[0122] from Figure 1Extract the D3 feature from the 3D model shown, D3=[[-0.34598737645375033,-0.35571749252008333,-0.3563259059806642,-0.3565601800698186,-0.3572626055376695,-0.360821304531894,-0.36727088142811404,-0.37664406433175407,-0.394389311200879,-0.4526931928937365]…];

[0123] Step 3: Extract the Zernike moments of the 2D view and normalize them to the (0,1) range using the standard moment method. The specific steps are as follows:

[0124] Step 3-1: Define the Zernike moment as the Zernike polynomial inside the unit circle, and define a set of complex-valued functions W on the unit circle. nm Let (x, y) represent any square-integrable function within the unit circle, calculated as follows:

[0125]

[0126] Where ρ represents the vector length from the origin to the point (x,y), θ is the angle between vector ρ and the counterclockwise direction of the x-axis, ϳ is the complex unit, n and m are the dimensions of the Zernike moments, and T is the Zernike polynomial. nm (ρ) is shown below:

[0127]

[0128] The Zernike moments of the two-dimensional view v(x,y) in step 3-2 are shown below:

[0129]

[0130] from Figure 1The Zernike moment features extracted from the 3D model view shown are: [[4.771981413511788,-6.7637410311195145,-0.5445469157595518,0.43041419701618233,3.5375762054384925,-2.778062128379721,0.9133042414782983,-6.160861616919954,-0.47940] 007849910937,3.9928354723193715,0.8046306208221686,-0.1770641369702642,-1.0724556062173323,-5.658273261083465,2.3297819390647936,3.4665808786931733,3.654570332221941,-3.556575838157568]…];

[0131] Step 4: Extract the Fourier operator of the view and normalize it to the (0,1) range using the standard moments method. The specific steps are as follows:

[0132] Step 4-1 performs a Fourier transform on the 2D view. Before performing the Fourier transform, it needs to be converted into a one-dimensional shape representation. The centroid distance function is adopted, which is achieved by calculating the distance from the outer contour boundary points to the centroid. According to the definition of the centroid, the Fourier coefficients Q(n) are obtained by the discrete Fourier transform, as shown in the following formula:

[0133]

[0134] Where j is the complex unit, {(x i ,y i Let |i=1, 2, …, N} be the boundary points of the contour;

[0135] Step 4-2: Dividing the Fourier coefficients Q(n) by the first coefficient Q(0) yields the Fourier operator, as shown below:

[0136]

[0137] from Figure 1The Fourier features extracted from the 3D model view shown are: [[-2.4467331226353455,5.769821202676518,0.15064502448105016,1.7720605432305985,-7.129699292405384,4.4242214906] 87125, -2.172374504139878, 4.765295855236491, 3.3866224809158405, 4.0162743125664715, -0.5012258374062019, 4.032756946678989, 3.671335516 606025, 0.25896169930334356, -2.850399030072648, 2.9823452754906397, 1.2661158108826323, 1.0311201934136491, 5.848585436377673, 3.8677542 501704525,-1.6492070154837855,3.8796155865145057,4.7787100859154865,-3.3944633257708032,3.4810942278125663,-0.5368205028596302]…];

[0138] Step 5 extracts the S1 and S2 features from the 2D view. The specific steps are as follows:

[0139] Step 5-1 uses the S1 descriptor to describe the compactness of the 2D view contour, extracting the perimeter and area of ​​the 2D view contour. The calculation process of the S1 feature is as follows:

[0140]

[0141] Where l represents the perimeter of the view outline and s represents the area of ​​the view;

[0142] from Figure 1 The S1 feature extracted from the 3D model view shown is 17.95177430889018;

[0143] Step 5-2 uses the S2 descriptor to describe the perimeter of the quadrilateral formed by four random sampling points on the boundary of the two-dimensional view, such as... Figure 4 As shown, the specific steps are as follows:

[0144] Step 5-2-1: Randomly select N points from Point to form a quadruple set PS2={(ai1,bi1,ci1,di1),(ai2,bi2,ci2,di2), …, (ai N ,bi N ,ci N ,di N Bins represents the number of intervals, and BinsSize represents the length of the interval, as shown below:

[0145]

[0146] Where max() means taking the maximum value, C represents the perimeter of the quadrilateral, and P1, P2, P3, and P4 represent points on the view boundary;

[0147] Step 5-2-2 uses the S2 descriptor to represent the perimeter of the quadrilateral formed by four random sampling points on the boundary of the two-dimensional view, S2_v i The calculation process is as follows:

[0148]

[0149] Where u=dist(P1,P2), v=dist(P1,P3), x=dist(P3,P4), y=dist(P2,P4), and dist() represents the Euclidean distance between the two points.

[0150] from Figure 1 The S2 feature extracted from the 3D model view shown is [[77.87897032689895,113.69113771244177,98.28254336742447,113.04997415169906,58.95698987036835,133.68316611822965,81.24736845153328,152.8966540027138,115.95309573204813,92.61062604628412]…];

[0151] Step 6 extracts training and test data from ModelNet10. The specific steps are as follows:

[0152] Step 6-1 Project the 3D training model in ModelNet10 to obtain a 2D view, extract the D1, D2, D3, Zernike moments, Fourier operators, S1, and S2 descriptors of the 2D view, and combine them with the 2D view to form training data;

[0153] Step 6-2 Project the 3D test model in ModelNet10 to obtain a 2D view, extract the D1, D2, D3, Zernike moments, Fourier operators, S1, and S2 descriptors of the 2D view, and combine them with the 2D view to form test data;

[0154] Step 7: Design the 2DResNet network and combine it in parallel with 2DCoTNet to obtain the 2DResNet-CoTNet network. Design the 1DResNet network, as follows: Figure 2 As shown, the specific steps are as follows:

[0155] Step 7-1: 2DResNet contains 5 convolutional layers: conv_1, conv_2, conv_3, conv_4, and conv_5. A CBAM attention module is added between conv_5 and the average pooling layer, and a SENet attention module is added sequentially afterward.

[0156] Step 7-2 constructs the 2DCoTNet model, which consists of three 1×1 convolutions, key values ​​(K), query values ​​(Q), variable values ​​(V), weight values ​​(W), and two dot product operations. The dot product of key value K and query value Q yields weight value W, which reflects the local context representation of the input. Then, the dot product of weight value W and variable value V yields the global context representation. Finally, the output of the model is obtained.

[0157] Step 7-3 combines the improved 2DResNet model with the 2DCoTNet model in parallel to form the 2DResNet-CoTNet convolutional neural network model.

[0158] Step 7-4: 1DResNet consists of 5 convolutional layers, activation functions, max pooling layers, and average convolutional layers. The extracted D1, D2, D3, Zernike moments, Fourier operators, S1, and S2 shape descriptors are input into the 1DResNet convolutional neural network and fused before output.

[0159] Step 8: Optimize 2DResNet-CoTNet and 1DResNet using the training data. Process the test data using the optimized 2DResNet-CoTNet and 1DResNet, performing convolution, pooling, and fusion to obtain the classification result in the output layer. The specific steps are as follows:

[0160] Step 8-1: Optimize 2DResNet-CoTNet using the training data. Input the 2D view from the test data into the optimized 2DResNet-CoTNet. After convolutional pooling, the output feature F is shown below. The extracted local features are as follows:

[0161] F=[[-0.21726081659465926,-0.26183593137350897,-0.29604916366376716,-0.2852742947486958,-0.24414850859207496,-0.1282356244373974,-0.2728304457330694,-0.30378319748832827,-0.30794185380824873,-0.29709096561504356],

[0162] [-0.21726081664447058,-0.2618359313735137,-0.29604916366377076,-0.2852742947486953,-0.24414850858971893,-0.12823562456102489,-0.27283044573308396,-0.30378319748832827,-0.30794185380824884,-0.2970909656150294],

[0163] [-0.21726081664447058,-0.26183593137352257,-0.2960491636637707,-0.28527429474868793,-0.2441485085937578,-0.12823562458162946,-0.27283044573311527,-0.30378319748832827,-0.3079418538082488,-0.2970909656150255],

[0164] [-0.21726081661381746,-0.2618359313735066,-0.29604916366376677,-0.28527429474869104,-0.24414850859162618,-0.12823562458162946,-0.2728304457330847,-0.30378319748832827,-0.30794185380824884,-0.2970909656150397],

[0165] [-0.21726081663297567,-0.26183593137351663,-0.29604916366376544,-0.2852742947486966,-0.2441485085925798,-0.1282356240871195,-0.27283044573311527,-0.30378319748832827,-0.30794185380824873,-0.2970909656150307],

[0166] [-0.21726081664447058,-0.26183593137352257,-0.29604916366376965,-0.2852742947486913,-0.2441485085927481,-0.12823562458162946,-0.2728304457330668,-0.30378319748832827,-0.30794185380824857,-0.2970909656150397]];

[0167] Step 8-2 optimizes 1DResNet using the training data. The D1, D2, D3, Zernike moments, Fourier operators, and S1 and S2 descriptors from the test data are input into the optimized 1DResNet. After convolutional pooling, the output feature G is obtained. The extracted local features are shown below:

[0168] G=[[-0.3098837669228543,-0.3110944956060788,-0.3118035609505005,-0.31243418855261684,-0.3108672772307213, -0.31301686177476257,-0.31409184539678087,-0.30388142341890206,-0.31690922478679967,-0.35611187347357165],

[0169] [-0.30988376692299663,-0.311094495606091,-0.31180356095049855,-0.3124341885526172,-0.3108672772307214,-0.313016861774763,-0.3140918453965931,-0.3038814234188965,-0.31690922479296696,-0.35611194925944706],

[0170] [-0.3098837669229729,-0.31109449560609237,-0.3118035609505002,-0.3124341885526171,-0.3108672772307215,-0.3130168617747619,-0.314091845396532,-0.30388142341890084,-0.31690922476829775,-0.35611193410227204],

[0171] [-0.3098837669231627,-0.311094495606091,-0.31180356095049805,-0.31243418855261673,-0.3108672772307213,-0.31301686177476273,-0.3140918453967198,-0.30388142341890473,-0.31690922481455247,-0.3561119189450969],

[0172] [-0.3098837669232102,-0.3110944956061046,-0.31180356095049866,-0.31243418855261673,-0.3108672772307213,-0.313016861774763,-0.3140918453966025,-0.30388142341890456,-0.31690922479913425,-0.3561118431592215],

[0173] [-0.30988376692309155,-0.31109449560610863,-0.31180356095049977,-0.3124341885526169,-0.3108672772307214,-0.31301686177476273,-0.3140918453965931,-0.3038814234188989,-0.31690922478834144,-0.3561119644166222]];

[0174] Step 8-3 inputs features F and G into the fusion layer for fusion, as shown below:

[0175]

[0176] After fusion through the fusion layer, from Figure 1 The fusion feature X extracted from the 3D model shown is as follows:

[0177] X=[[1.2092708726918542,-0.9227561263021111,7.696135644482613,4.802492685405509,-7.180920637443266, -3.9065770434182223,-0.6209995267247632,5.923929459701323,0.2358944807600696,-0.5196450890211213],

[0178] [-5.359509614146529,-8.900756853292249,-8.775007491209433,-5.562669994246831,5.120143894187047,3.2566768337449976,2.38159153826989,-2.8909490762071957,3.891646608618246,-0.5196450890211213],

[0179] [6.618561652818589,-1.1647728093067984,-10.486749836547618,0.1519479288810843,0.16439601986076072,1.2080140986170955,-0.34331302722330026,-4.112129864902629,-2.8828319406699703,-0.5196450890211213],

[0180] [-7.3740493800704705,-11.642282623230896,-14.780942778496499,-0.45485400825023603,-8.48318322013078,-2.192559391548071,1.536087511572395,-9.719270605067354,1.5505271499405844,-0.5196450890211213],

[0181] [-7.98606146085749,0.48123702861344886,-6.127844405613091,3.8357137736429037,-2.173754356963368,1.2660843919061966,-4.919631249887141,-5.128360748083804,-1.9071944240581218,-0.5196450890211213],

[0182] [-0.6237351453203983,-7.456750577446568,-0.5163357332916354,-9.372856586831839,-0.6630227013222232,4.312915395956567,-1.5562553217679302,4.207388357811751,-4.983510223664951,-0.5196450890211213]];

[0183] Step 8-4: Input X into the Softmax classifier to obtain the classification result, as shown below:

[0184]

[0185] Where Y = (y1, y2, …, y 10 W and B represent the parameters of the Softmax layer, and P(S) represents the parameters of the Softmax layer. i |m) represents the 3D model m in category S i The probability of occurrence of the following, i=1, 2, …, 10, is calculated as follows:

[0186]

[0187] Figure 1 The three-dimensional model shown is in S1, S2, ..., S 10 The probability distribution is: [0.84882, 0.78611, 0.88286, 0.807610, 0.841522, 0.885219, 0.863211, 0.8200554, 0.865333, 0.898861]. The class with the highest probability in this distribution is selected as the predicted class S for the 3D model m, as shown below:

[0188]

[0189] Figure 1 The 3D model classification result shown is for the "toilet" class. The classification accuracy of the test data is as follows:

[0190] Classification accuracy = 539 / 600 ≈ 89.9%

[0191] Multi-view fusion of ResNet and CoTNet in embodiments of the present invention Figure 3 The 3D model classification method projects a 3D model into a 2D view, uses various shape descriptors to describe the 2D view, and combines the Softmax function to determine the category of the 3D model, achieving high accuracy.

[0192] The above description, in conjunction with the accompanying drawings, provides a detailed account of the embodiments of the present invention. The specific implementation methods described herein are merely illustrative of the invention. Those skilled in the art will recognize that variations and modifications can be made to the specific implementation methods and applications based on the principles of this invention; therefore, this document should not be construed as limiting the scope of the invention.

Claims

1. A multi-view three-dimensional model classification method fusing ResNet and CoTNet, characterized in that, The method mainly comprises the following steps: Step 1: project the three-dimensional model in Modelnet10 to obtain a two-dimensional view set; Step 2: use D1 descriptor to describe the distance between the two-dimensional view centroid and the random sampling points on the boundary, use D2 descriptor to describe the distance between two random sampling points on the two-dimensional view boundary, and use D3 descriptor to describe the square root of the area of the triangle formed by three random sampling points on the two-dimensional view boundary; Step 3: extract the Zernike moment of the two-dimensional view, and normalize it to the range of (0, 1) by using the standard moment method; Step 4: obtain the contour view of the two-dimensional view, extract the one-dimensional Fourier operator of the contour view, and divide by the direct current component to normalize to the range of (0, 1); Step 5: use S1 descriptor to describe the compactness of the two-dimensional view boundary, and use S2 descriptor to describe the perimeter of the quadrilateral surrounded by four random sampling points on the two-dimensional view boundary; Step 6: extract the two-dimensional view V1-V6 of the three-dimensional model in ModelNet10, input V1-V6 as input data into the two-dimensional residual-context transformation network 2DResNet-CoTNet, extract D1, D2, D3, Zernike moment, Fourier operator, S1, S2 shape descriptor of V1-V6, and input them as input data into the one-dimensional residual network 1DResNet; Step 7: design 2DResNet network, combine it with 2DCoTNet in parallel to obtain 2DResNet-CoTNet network, and design 1DResNet network, the specific steps are as follows: Step 7-1: 2DResNet contains five convolutional layers, namely conv_1, conv_2, conv_3, conv_4 and conv_5, a CBAM attention module is added between conv_5 and the average pooling layer, and a SENet attention module is added in series at the back; Step 7-2: construct 2DCoTNet model, which consists of three 1x1 convolution, key value keys(K), query value query(Q), variable value value(V), weight value weights(W) and two point multiplication operations, multiply the key value K and the query value Q to get the weight value W, which reflects the local context representation of the input, then multiply the weight value W and the variable value V to get the global context representation, and finally get the output of the model; Step 7-3: combine the improved 2DResNet model with the 2DCoTNet model in parallel to form a 2DResNet-CoTNet convolutional neural network model; Step 7-4: 1DResNet consists of five convolutional layers, activation function, maximum pooling layer and average convolution layer, and the extracted D1, D2, D3, Zernike moment, Fourier operator, S1, S2 shape descriptor is input into the 1DResNet convolutional neural network for fusion and output; Step 8: Optimize 2DResNet-CoTNet and 1DResNet using training data, process test data using optimized 2DResNet-CoTNet and 1DResNet, and obtain classification results in the output layer after convolution, pooling, and fusion.

2. The multi-view 3D model classification method of claim 1, wherein, In step 1, the three-dimensional model is preprocessed, and the specific steps are as follows: Step 1-1: Fix the three-dimensional model at the center of a virtual sphere and select a fixed angle and number for projection; Step 1-2: Place a virtual camera above the three-dimensional model, rotate the camera by 60 degrees at each step, and project a set of two-dimensional views, denoted as V1, V2, V3, V4, V5, and V6.

3. The method of claim 1, wherein the method further comprises: In step 2, extract D1, D2, and D3 features of the views, and the specific steps are as follows: Step 2-1: Use D1 descriptor to describe the distance between the two-dimensional view centroid and the randomly sampled points on the boundary, and the specific steps are as follows: Step 2-1-1: Randomly and equally sample points on the boundary of the two-dimensional view, and the collected point set is Point={(x1,y1),…,(xi,yi),…,(xm,ym)}, where (xi,yi) is the point coordinate; Step 2-1-2: Randomly select N points from Point to form set PD1={ai1,ai2,…,aiN}, Bins represents the number of intervals, and BinsSize represents the interval length, as shown below: where dist() represents the Euclidean distance between two points, O represents the centroid of the two-dimensional view, P1 represents the point on the boundary of the view, and max() represents the maximum value; Step 2-1-3: Use D1 descriptor to describe the distance between the two-dimensional view centroid and the randomly sampled points on the boundary, and the calculation formula of D1_vi is as follows: Step 2-2: Use D2 descriptor to describe the distance between two randomly sampled points on the boundary of the two-dimensional view, and the specific steps are as follows: Step 2-2-1: Randomly select N pairs of points from Point to form set PD2={(ai1,bi1),(ai2,bi2),…,(aiN,biN)}, Bins represents the number of intervals, and BinsSize represents the interval length, as shown below: where dist() represents the Euclidean distance between two points, P1 and P2 represent the points on the boundary of the view, and max() represents the maximum value; Step 2-2-2: Use D2 descriptor to describe the distance between two randomly sampled points on the boundary of the two-dimensional view, and the calculation formula of D2_vi is as follows: Step 2-3: Use D3 descriptor to describe the square root of the area of the triangle formed by three randomly sampled points on the boundary of the two-dimensional view, and the specific steps are as follows: Step 2-3-1: Randomly select N triplets of points from Point to form set PD3={(ai1,bi1,ci1),(ai2,bi2,ci2),…,(aiN,biN,ciN)}, Bins represents the number of intervals, and BinsSize represents the interval length, as shown below: Wherein, herson() represents Helen formula, max() represents maximum value, P1, P2, P3 represent points on view boundary; Step 2-3-2 utilizes D3 descriptor to represent square root of triangle area formed by three random sampling points on two-dimensional view boundary, calculation formula of D3_vi is as follows: Wherein, a=dist(P1,P2), b=dist(P1,P3), c=dist(P2,P3), dist() represents Euclidean distance between two points, s represents area of triangle.

4. The multi-view 3D model classification method of claim 1, wherein, In step 3, Zernike moment of view is extracted, and specific steps are as follows: Step 3-1 Zernike moment is defined as Zernike polynomial inside unit circle, a set of complex value function set Wnm(x,y) on unit circle is defined, which represents any square integrable function inside unit circle, calculation formula is as follows: Wherein, ρ represents vector length from origin to point (x,y), θ is included angle between vector ρ and x axis in counterclockwise direction, ϳ is complex unit, n and m are dimension of Zernike moment, Zernike polynomial Tnm(ρ) is as follows: Step 3-2 Zernike moment of two-dimensional view v(x,y) is as follows: 。 5. The method of claim 1, wherein, In step 4, Fourier operator of view is extracted, and normalized to (0,1) range by using standard moment method, and specific steps are as follows: Step 4-1 Fourier transform is carried out on two-dimensional view, before Fourier transform, it needs to be converted into one-dimensional shape representation, barycentric distance function is adopted, barycentric distance is realized by calculating distance between contour boundary point and barycenter, according to barycenter definition, Fourier coefficient Q(n) is obtained by discrete Fourier transform, formula is as follows: Wherein, j is complex unit, {(xi,yi)|i=1, 2, …, N} is boundary point of contour; Step 4-2 Fourier coefficient Q(n) is divided by first coefficient Q(0) to obtain Fourier operator, as follows: 。 6. The method of claim 1, wherein In step 5, S1 and S2 features of two-dimensional view are extracted, and specific steps are as follows: Step 5-1 S1 descriptor is utilized to describe compactness of two-dimensional view contour, perimeter and area of two-dimensional view contour are extracted, calculation process of S1 feature is as follows: Wherein, l represents perimeter of view contour, s represents area of view; Step 5-2 S2 descriptor is utilized to describe perimeter of quadrangle formed by four random sampling points on two-dimensional view boundary, and specific steps are as follows: Step 5-2-1 four tuples of N points are randomly selected from Point to form set PS2={(ai1,bi1,ci1,di1),(ai2,bi2,ci2,di2), …, (aiN,biN,ciN,diN)}, Bins represents interval number, BinsSize represents interval length, as follows: Wherein, max() represents maximum value, C represents perimeter of quadrangle, P1, P2, P3, P4 represent points on view boundary; Step 5-2-2 uses S2 descriptor to represent the perimeter of the quadrilateral formed by four random sampling points on the two-dimensional view boundary, and the calculation process of S2_vi is as follows: Wherein, u=dist(P1, P2), v=dist(P1, P3), x=dist(P3, P4), y=dist(P2, P4), dist() represents the Euclidean distance between two points.

7. The method of claim 1, wherein In step 6, the training data and test data are extracted from ModelNet10, and the specific steps are as follows: Step 6-1 Project the three-dimensional training model in ModelNet10 to obtain a two-dimensional view, extract D1, D2, D3, Zernike moments, Fourier operator, S1, S2 descriptors of the two-dimensional view, and construct training data with the two-dimensional view. Step 6-2 Project the three-dimensional test model in ModelNet10 to obtain a two-dimensional view, extract D1, D2, D3, Zernike moments, Fourier operator, S1, S2 descriptors of the two-dimensional view, and construct test data with the two-dimensional view.

8. The multi-view 3D model classification method fusing ResNet and CoTNet according to claim 1, characterized in that In step 8, 2DResNet-CoTNet and 1DResNet are optimized using training data, and the optimized 2DResNet-CoTNet and 1DResNet are used to process test data, and after convolution, pooling and fusion, the classification result is obtained in the output layer. The specific steps are as follows: Step 8-1 uses training data to optimize 2DResNet-CoTNet, and inputs the two-dimensional view in the test data into the optimized 2DResNet-CoTNet, and outputs the feature F after convolution and pooling; Step 8-2 uses training data to optimize 1DResNet, and inputs D1, D2, D3, Zernike moments, Fourier operator, S1, S2 descriptors in the test data into the optimized 1DResNet, and outputs the feature G after convolution and pooling; Step 8-3 inputs the features F and G into the fusion layer for fusion, as shown below: Step 8-4 inputs X into the Softmax classifier to obtain the classification result, as shown below: Wherein, Y=(y1, y2, …, yM), W and B represent the parameters of the Softmax layer, P(Si|m) represents the probability of three-dimensional model m in class Si, and the calculation process is as follows: In the probability distribution, the class with the maximum probability is selected as the predicted class S of the three-dimensional model m, as shown below: 。

Images