A lightweight three-dimensional model classification system based on multi-view grouping

Abstract

The application discloses a lightweight three-dimensional model classification system based on multi-view grouping. A view sampling module projects a three-dimensional model into N views and sequentially renders N two-dimensional depth maps. A view grouping module divides the two-dimensional depth maps into multiple groups according to the geometric semantics of the three-dimensional model. A feature extraction module extracts features from each group of two-dimensional depth maps and outputs multiple feature maps. A feature fusion module inputs the multiple feature maps into a MobileViT Block module for further analysis to obtain a final prediction result. The two-dimensional depth maps are divided into multiple groups through good geometric semantics, and then multiple groups of views use an improved MobileNet V2 network to extract effective features. Finally, multi-view feature fusion obtains the feature recognition of the three-dimensional model. The network reduces the network complexity while ensuring the effectiveness of three-dimensional model classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a lightweight three-dimensional model classification system based on multi-view grouping. BACKGROUND

[0002] With the development and maturity of three-dimensional acquisition technology, more and more three-dimensional models are generated to represent the geometric shapes of real world or virtual three-dimensional objects. Accordingly, classification or retrieval of three-dimensional models from digitized three-dimensional models has become one of the important research topics. In recent years, scholars have proposed a large number of three-dimensional model classification and retrieval methods. These methods can be divided into two categories: hand-crafted descriptor-based methods and deep learning descriptor-based methods.

[0003] For the hand-crafted descriptor-based methods, Osada et al. proposed model distribution descriptors that can measure global geometric properties of a model based on the distribution of attributes. Other methods, such as the multi-Fourier spectrum descriptor-based and spectrum clustering enhanced three-dimensional model retrieval algorithm, use signal processing techniques and spectrum clustering to capture the inherent features of the model. Some shape descriptors are based on the native three-dimensional representation of the model, such as the three-dimensional voxel grid descriptor proposed by Knopp et al., which extends the two-dimensional feature descriptor SURF to three dimensions. View-based descriptors use a set of views to describe a three-dimensional model, such as the panoramic view used in the panoramic view-based three-dimensional model descriptor unsupervised three-dimensional model retrieval algorithm, which describes the position and direction of a shape surface. Shu et al. proposed a thickness map for three-dimensional model classification.

[0004] For the deep learning descriptor-based methods, Su et al. proposed a multi-view convolutional neural network MVCNN for three-dimensional model recognition, which extracts features from each view separately and fuses them through a pooling operation. Three-dimensional model recognition based on unsupervised multi-view pose estimation estimates the pose and class of an object together by taking the view point label as a latent variable. Feng et al. proposed a group view convolutional neural network GVCNN, which includes view, group, and shape level descriptors. Wei et al. proposed a view-based three-dimensional model classification graph convolutional neural network View-GCN framework, which uses view-graph modeling to model the spatial relationship between views. Liu et al. proposed a hierarchical multi-view context modeling HMVCM method for three-dimensional object classification and retrieval, which models the multi-view context for classification and retrieval. Chen et al. proposed a multi-view visual transformer MVT framework for three-dimensional model recognition to communicate between patches from different views. Han et al. proposed a SeqViews2SeqLabels method that learns three-dimensional global features by RNN attention aggregation of sequential views.Figure 3 D2SeqViews uses a convolutional neural network to aggregate sequential views.

[0005] Existing three-dimensional model classification algorithms can be divided into two types: model-based and view-based. The former is used for descriptors of three-dimensional models themselves, while the latter is used for descriptors of two-dimensional images rendered under multiple views. Generally speaking, due to the rapid development of deep learning, especially the convolutional neural network (hereinafter referred to as CNN) technology, many deep learning-based algorithms have achieved high accuracy in three-dimensional model classification, but they usually have complex neural network (NN) structures, have high requirements for the size and computing power of the graphics processing unit (hereinafter referred to as GPU) memory, and increase the difficulty of training. Therefore, these algorithms may not be practically deployed on platforms without high configurations. SUMMARY

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a lightweight three-dimensional model classification system based on multi-view grouping, comprising a view sampling module, a view grouping module, a feature extraction module and a feature fusion module, wherein the view sampling module projects a three-dimensional model into N views to obtain N two-dimensional depth maps; the view grouping module divides the two-dimensional depth maps into multiple groups through geometric semantic division; the feature extraction module extracts features from each group of two-dimensional depth maps using an improved lightweight network MobileNetV2 and outputs multiple feature maps; and the feature fusion module inputs the multiple feature maps into a MobileViTblock for analysis to obtain a prediction result, which is used to determine the class of a single three-dimensional model.

[0008] In a possible implementation, the view sampling module projects a three-dimensional model into N views to sequentially render N two-dimensional depth maps, comprising:

[0009] The three-dimensional model is normalized to be within a unit boundary sphere;

[0010] An inscribed triangular mesh with N vertices is used for view sampling, a virtual camera is placed on the vertex, the vertex faces the centroid of the three-dimensional model, and a two-dimensional depth map is captured by orthogonal projection, and the single three-dimensional model is converted into multiple two-dimensional depth map representations.

[0011] In a possible implementation, the view grouping module divides the two-dimensional depth map into multiple groups through geometric semantic division, including: a single three-dimensional model divides multiple two-dimensional depth maps into three groups according to three different direction dimensions of the sampling viewpoints, the three groups of views are orthogonal to each other, and together form the entire three-dimensional model view set, assuming that the size of each depth map is WxH, each view group has M views, and each view group can be mapped to form a tensor X i = R W*H*C , i = 1, 2, 3, C = M.

[0012] In a possible implementation, the improved MobileNetV2 includes MobileNetV2, and two C1 modules and one C2 module arranged at the input end of the MobileNetV2, the C1 module includes a 3x3 convolution module, a regularization processing module, and an activation function, and the C2 module includes a 3x3 convolution module, a regularization processing module, an activation function, and an effective channel attention ECA module.

[0013] In a possible implementation, the feature extraction module extracts features of each group of two-dimensional views through the improved MobileNetV2 and outputs multiple feature maps, including: inputting the corresponding two-dimensional depth map in the equatorial direction of the sphere into the C2 module, and inputting the two-dimensional depth maps in the remaining two directions into two C1 modules respectively, the two-dimensional depth maps are subjected to the C1 module and the C2 module to extract shallow-level feature information and adjust the size of the feature map; and then input to the lightweight network MobileNetV2 for further feature extraction.

[0014] In a possible implementation, the feature fusion module includes a C3 module and a MobileViTblock, and the C3 module includes a 1x1 convolution module, a regularization processing module, an activation function, a maximum pooling module, and an ECA module.

[0015] In a possible implementation, the MobileViTblock includes a local modeling module at the input end, the output end of the local modeling module is connected to a global modeling module, the output end of the global modeling module is connected to a fusion module, the local modeling module includes a first n x n convolution block and a first 1x1 convolution block, the global modeling module includes an unfolding unit and a folding unit, and a Transformer module between the unfolding unit and the folding unit, and the global modeling module includes a second 1x1 convolution block and a second n x n convolution block.

[0016] In a possible implementation, the feature fusion module inputs the plurality of feature maps after fusion into a MobileViT block for analysis to obtain a prediction result, including: after the plurality of feature maps are input into the MobileViT block, local feature extraction is first performed through n x n convolution, and then dimension enhancement is performed using 1 x 1 convolution; then, global feature extraction is performed using an unfolding unit, a Transformer module folding unit, and the number of channels is adjusted back to the original size using 1 x 1 convolution, the output is connected with the original input feature map using a skip connection, and finally, n x n convolution is used to fuse the features to obtain feature Y.

[0017] In a possible implementation, the feature Y provides a high-level multi-view representation of the three-dimensional model, for similarity calculation in a three-dimensional model retrieval process.

[0018] In the embodiment of the present application, the obtained two-dimensional depth map is divided into a plurality of groups through good geometric semantics, the determination of views with different feature information amounts can be implemented, then the effective features of the view groups are extracted using the improved lightweight network respectively, and finally, the final output is obtained after fusion and further analysis, the recognition of the three-dimensional model can be implemented, the overall network is designed from the perspective of lightweight, and the classification effectiveness is ensured while the calculation complexity and the network parameter amount are low. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A structure diagram of a lightweight three-dimensional model classification system based on multi-view grouping provided by the embodiment of the present application;

[0020] Figure 2 A multi-view grouping diagram from three different direction dimensions of a three-dimensional model provided by the embodiment of the present application;

[0021] Figure 3 A structure diagram of an ECA module provided by the embodiment of the present application;

[0022] Figure 4 A structure diagram of a MobileViT block provided by the embodiment of the present application. DETAILED DESCRIPTION

[0023] The present scheme will be described below in combination with the accompanying drawings and specific embodiments.

[0024] Figure 1 A lightweight three-dimensional model classification system based on multi-view grouping provided by the embodiment of the present application, referring to Figure 1 The lightweight three-dimensional model classification system based on multi-view grouping in the embodiment includes a view sampling module, a view grouping module, a feature extraction module, and a feature fusion module.

[0025] For a given 3D model, it is first projected into N views to obtain N 2D depth maps. Then, the depth maps are divided into multiple groups based on the geometric semantic partition of views. Next, the improved MobileNetV2 is used to extract features for each group of views. Finally, the obtained multiple feature maps are input into the MobileViT block for further parsing, and then the prediction results are obtained through the Softmax function, which are used for class determination of the 3D model.

[0026] How to obtain multiple depth maps around a single 3D model is an important problem, and the embodiments of the present application use an inscribed N-vertex triangular mesh for view sampling, including: first, normalize the model to be in a unit boundary sphere, and the boundary sphere surface is composed of an N-vertex triangular mesh. Then, we place virtual cameras on the vertices of the boundary sphere surface, which face the centroid of the 3D model, and capture 2D depth maps through orthogonal projection. A single 3D model is represented by multiple 2D depth maps.

[0027] In the embodiments, taking N = 18 as an example, an inscribed 18-vertex triangular mesh is used to visualize the view sampling in Figure 2 .

[0028] Next, the views are divided into three groups by three different directional dimensions of the 3D model, and the specific directions of the three different directional dimensions are shown in Figure 2 (a), Figure 2 (b), Figure 2 (c). Each group contains 8 two-dimensional depth maps of viewpoints, and it should be noted that there is a slight redundancy between groups because some viewpoints are repeatedly included in different groups. Assuming that the size of each depth map is W x H. The depth maps in each view group form a tensor, X i = R W*H*C , i = 1, 2, 3, C = 8. The argument of the proposed view grouping scheme is as follows. First, the division of viewpoints follows good geometric semantics: first, the viewpoints of each group are located on a complete circle, and the widest view of the 3D model is obtained by 18 virtual cameras; the three large circles are orthogonal to each other and together form the most complete and widest view of the entire 3D model. Second, compared with extracting features from all views, view grouping feature extraction can greatly reduce the complexity of the network.

[0029] The model will extract features for each view group respectively. Specifically, the three groups of tensors are input into three feature extraction branches. Compared with extracting features from all views through the overall network module, grouped feature extraction greatly reduces the parameter count and computational complexity, as shown in the following analysis.

[0030] Assume for a CNN layer that maps features of shape W1 x H1 x C1 to W2 x H2 x C2. Assume that a k x k convolution is used, and let N1 = W1 H1 C1 and N2 = W2 H2 C2. The parameter count for this network layer is P = k2 C1 C2, and the approximate number of floating point operations is Q = 2k 2 C1 N2. If we average the number of channels into g input and output groups, the total parameter count for this group of convolution layers is P' = k2 C1g C2g g = P / g, and the total number of floating point operations is approximately Q' = 2k2 C1g N2g g = Q / g. Even more savings are achieved for networks with more levels.

[0031] For each group of feature extraction branches, the view group is extracted by using the improved lightweight network MobileNetV2. The improved MobileNetV2 includes MobileNetV2 and two C1 modules and one C2 module set at the input end of the MobileNetV2, the C1 module includes a 3 x 3 convolution module, a regularization processing module and an activation function, and the C2 module includes a 3 x 3 convolution module, a regularization processing module, an activation function and an effective channel attention ECA module. As can be seen, several layers of convolution are added in front of the standard MobileNetV2 in each branch in order to effectively extract features and adjust the feature map size, as shown in the C1 module and the C2 module in Figure 1 The used MobileNetV2 has been pre-trained on Imagenet1K.

[0032] The two-dimensional depth map in the equatorial direction of the ball is input into the C2 module, and the two-dimensional depth maps in the remaining two circles are input into the two C1 modules respectively, and the two-dimensional depth maps are extracted by the C1 module and the C2 module. Shallow features; then input to the lightweight network MobileNetV2 for further feature extraction.

[0033] In particular, the effective channel attention ECA module is used to introduce the channel attention mechanism into the C2 module. This helps to more effectively extract features. Considering the reduction of network parameter quantity and computational quantity, only ECA is added to the C2 module corresponding to the view in the equatorial direction dimension, because the most important projection in the three-dimensional model view set comes from the view in the equatorial direction dimension. The structure of the ECA module is shown in Figure 3 It improves the baseline squeeze-and-excitation (SE) module, which uses 1 x 1 convolution instead of fully connected to capture channel-level information. This approach significantly reduces the number of parameters required while still achieving the required performance. After group feature extraction, we get the feature maps F1, F2 and F3 of the three view groups respectively.

[0034] The feature fusion module in this embodiment includes a C3 module and a MobileViT block. The C3 includes a 1x1 convolution module, a regularization processing module, an activation function, a max pooling module, and an ECA module. The MobileViT block includes a local modeling module at the input end, the output end of the local modeling module is connected to a global modeling module, the output end of the global modeling module is connected to a fusion module, the local modeling module includes a first n x n convolution block and a first 1x1 convolution block, the global modeling module includes an unfolding unit and a folding unit, and a Transformer module between the unfolding unit and the folding unit, and the global modeling module includes a second 1x1 convolution block and a second n x n convolution block.

[0035] The feature fusion module is also designed to be lightweight and effective. First, the view group features are concatenated by F = aggregate(F1, F2, F3) and sent to the C3 module. The C3 block shrinks F by a 1x1 convolution to reduce the parameter count and computational cost of the following layers and uses an ECA module to improve the effectiveness of feature extraction. Second, the output of the C3 block is input into the MobileViT block for further analysis. MobileViT is a lightweight version of the visual Transformer (ViT) model, with the MobileViT block as the core module. The structure of the MobileViT block is as shown in Figure 4 Within the MobileViT block, local feature extraction is first performed by an n x n convolution, and then dimension enhancement is performed using a 1x1 convolution. Then, global feature extraction is performed using an unfolding Transformer folding structure, and a 1x1 convolution is used to adjust the number of channels back to the original size. The output is connected to the original input feature map using a skip connection, and finally the features are fused by an n x n convolution to obtain the feature Y.

[0036] The feature Y provides a high-level multi-view representation of the three-dimensional model. Y can be used for similarity calculation in the three-dimensional model retrieval process, and as input to the Softmax layer for three-dimensional model classification prediction.

[0037] To validate the multi-view grouping based lightweight 3D model classification system in this application, the performance of the proposed LiteMVGNet for 3D model classification is evaluated on two benchmark 3D model datasets, ModelNet10 and ModelNet40. ModelNet40 consists of 12,311 models from 40 classes. We follow the same data split as most related works, where 9843 models and 2456 models are used for training and testing, respectively. ModelNet10 is a subset of ModelNet40, which consists of 4,899 models from 10 classes. We use 3,991 models as the training set and 908 models as the test set, respectively.

[0038] The input size of the depth map is set to 224x224x1. LiteMVGNet uses the PyTorch deep learning framework, and we use stochastic gradient descent (SGD) as the optimizer during the training process. The initial learning rate is set to 0.01, and the weight decay is set to 0.01. The network model is trained on a GPU (NVIDIA 3090 24GB), and the network is trained for 200 rounds. The learning rate is reduced to 1 / 10 of the previous value twice at the 60th round and the 160th round.

[0039] We compare LiteMVGNet with benchmark methods, including MVCNN, MVCNN-MultiRes, GVCNN, RotationNet, MVT-small, and the method proposed by Lin et al. We compare the classification accuracy and network complexity, and the results are shown in Table 1. In Table 1, we can see that LiteMVGNet achieves the lowest network complexity with the least floating-point operations (FLOPs) and network parameters (Params). Specifically, the parameter amount of LiteMVGNet is about 2 / 3 of MVT-small and even about 1 / 6 of the method proposed by Lin et al.; while the computational complexity of LiteMVGNet is one or even two orders of magnitude smaller than other models. We also notice that LiteMVGNet has higher classification accuracy than most other methods. When using 20 views, RotationNet and MVTsmall obtain higher classification accuracy because they use two more views than us.

[0040] Table 1 Classification Comparison Results

[0041]

[0042] The embodiment of the application proposes a lightweight three-dimensional model classification system LiteMVGNet based on multi-view grouping. It is composed of multiple stages, and each stage is designed to be simple and effective. First, the three-dimensional model is projected into 18 views to obtain two-dimensional depth maps. Then, the two-dimensional depth maps are divided into multiple groups according to good geometric semantics, and the features of each group are extracted by an improved lightweight network MobilenetV2 branch network. Then, a lightweight MobileViT-based module is used to fuse the view group features and further analyze the features. The channel attention mechanism is introduced in the network, which can promote effective feature extraction. Experiments show that LiteMVGNet has significant advantages in model complexity.

[0043] In the embodiments of the application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the cases of A alone, A and B together, and B alone. Wherein A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" and the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b and c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0044] The above is only a specific embodiment of the application, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the application, which should be covered within the protection scope of the application. The protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A lightweight 3D model classification system based on multi-view grouping, characterized by: include: A view sampling module, a view grouping module, a feature extraction module, and a feature fusion module, wherein the view sampling module projects the 3D model into N views and sequentially renders them to generate N 2D depth maps; the view grouping module divides the 2D depth maps into multiple groups based on geometric semantics; the feature extraction module extracts features from each group of 2D views using an improved lightweight network MobileNetV2 and outputs multiple feature maps; the feature fusion module fuses the multiple feature maps and inputs them into the network MobileViTblock module for further analysis to obtain a prediction result of the 3D model, which is used to determine the class of a single 3D model; The view sampling module projects the 3D model into N views and sequentially renders and generates N 2D depth maps, including: Normalizing the three-dimensional model to be within a unit bounding sphere; Use an inscribed triangle mesh of N vertices for view sampling, place a virtual camera on the vertex, the vertex faces the centroid of the 3D model, and capture a 2D depth map through orthogonal projection, the single 3D model is converted into multiple 2D depth map representations; The improved MobileNetV2 includes MobileNetV2 and two C1 modules and one C2 module set at the input end of the MobileNetV2. The C1 module includes a 3×3 convolution module, a regularization processing and an activation function. The C2 module includes a 3×3 convolution module, a regularization processing module, an activation function and an effective channel attention ECA module. The feature extraction module extracts features from each set of two-dimensional views using the improved MobileNetV2 and outputs multiple feature maps, including: The two-dimensional depth map corresponding to the equatorial direction of the bounding sphere is input into the C2 module, and the two-dimensional depth maps in the other two directions are respectively input into two C1 modules. The two-dimensional depth map is passed through the C1 module and the C2 module to extract shallow feature information and adjust the size of the feature map; then it is input into the lightweight network MobileNetV2 for further feature extraction.

2. The lightweight 3D model classification system based on multi-view grouping according to claim 1 is characterized in that: The view grouping module divides the two-dimensional depth map into multiple groups by geometric semantic partitioning, including: a single three-dimensional model divides multiple two-dimensional depth maps into three groups according to the sampling viewpoints in three different direction dimensions. The three groups of views are orthogonal to each other and together form the entire three-dimensional model view set. Assuming that the size of each depth map is W×H and each view group has M views, each view group can be mapped to form a tensor: X i =R W*H*C ,i=1,2,3,C=M.

3. The lightweight 3D model classification system based on multi-view grouping according to claim 1 is characterized in that: The feature fusion module includes a C3 module and a MobileViTblock, and the C3 includes a 1×1 convolution module, a regularization processing module, an activation function, a maximum pooling module and an ECA module.

4. The lightweight 3D model classification system based on multi-view grouping according to claim 3 is characterized in that: The MobileViTblock includes a local modeling module at the input end, the output end of the local modeling module is connected to the global modeling module, the output end of the global modeling module is connected to the fusion module, the local modeling module includes a first n×n convolution block and a first 1×1 convolution block, the global modeling module includes an expansion unit and a folding unit, and a Transformer module between the expansion unit and the folding unit, and the global modeling module includes a second 1×1 convolution block and a second n×n convolution block.

5. The lightweight 3D model classification system based on multi-view grouping according to claim 4 is characterized in that: The feature fusion module fuses the multiple feature maps and inputs them into MobileViTblock for parsing to obtain prediction results, including: the multiple feature maps are fused and input into MobileViTblock, first local features are extracted through n×n convolution, and then dimensionality enhancement is performed using 1×1 convolution; then global features are extracted using the expansion unit and the Transformer module folding unit, and the number of channels is adjusted back to the original size using 1×1 convolution, the output is connected to the original input feature map using jump connection, and finally feature Y is obtained by fusing features through n×n convolution.

6. The lightweight 3D model classification system based on multi-view grouping according to claim 5 is characterized in that: Feature Y provides a high-level multi-view representation of the 3D model and is used for similarity calculation in the 3D model retrieval process.

Images