A multi-view 3D object recognition method and system

Abstract

The application discloses a kind of multi-view 3D object identification method and system, comprising: step 1, for each 3D object, multiple virtual cameras are set from different angles to obtain the multi-view representation of 3D object, and the multi-scale feature of each view is extracted;Step 2, the multi-scale feature of each view is fused;Step 3, the fusion feature of different views is interacted by using global Transformer module, and is aggregated into global descriptor by average pooling layer after information interaction;Step 4, global descriptor is input into classifier for classification.The method of the application realizes the most advanced recognition performance on ModelNet40 and ModelNet10 set in 12 views.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning technology, and specifically relates to a multi-view-based... Figure 3 Methods and systems for classifying D objects. Background Technology

[0002] 3D object recognition is one of the important problems in computer vision. With the rapid development of fields such as medical image analysis, autonomous driving, and virtual reality, the demand for 3D object recognition has become urgent.

[0003] Deep learning-based 3D object recognition methods can be roughly divided into three groups: view-based methods, voxel-based methods, and point-based methods. Voxel-based and point-based methods can also be collectively referred to as model-based methods.

[0004] Model-based methods directly use 3D networks to process the raw representations of 3D objects, such as polygonal meshes, voxel meshes, and point clouds. In voxel-based methods, 3D objects are represented as 3D meshes, and 3D convolutional filters are used to convolve on the voxels. In point-based methods, 3D objects are represented as a set of unordered points, which are directly used for prediction after modeling. Some classic networks, such as VoxNet, fit point clouds to voxel meshes and use volumetric occupancy rasteres to classify the point clouds of 3D objects.

[0005] View-based methods use 2D convolutional filters to convolve 2D images rendered from 3D objects. Since view-based methods do not rely on complex 3D features, and with the rapid development of deep learning and the success of convolutional neural networks in image recognition, view-based approaches generally outperform model-based methods. However, view-based 3D object recognition still faces many challenges. One of the biggest challenges is how to effectively correlate features between different views.

[0006] In view-based 3D object recognition methods, MVCNN is a pioneering work. It extracts features for each view using a CNN and aggregates features across different views using max pooling layers. To more effectively aggregate features across different views, GVCNN groups views into multiple sets and assigns higher weights to sets containing more critical information, then aggregates features from all sets into a global descriptor based on these weights. Inspired by the great success of the Vision Transformer, a multi-view Vision Transformer (MVT) using a local-global Transformer was proposed. MVT overcomes the limitation of multi-view CNN models in effectively capturing the correlation between features across different views; however, the Transformer degrades the details of local features when extracting them. Summary of the Invention

[0007] Inspired by the competitive performance of the Multi-View Transformer (MVT) and previous multi-view CNN models, we propose a hybrid network that leverages the strengths of both CNNs and Transformers to process 2D rendered images of 3D objects. The Transformer addresses the insufficient feature interaction between different viewpoints in CNN models, while the CNN addresses the degradation of the Transformer in local feature extraction. Compared to previous methods, this approach can extract high-quality features while fully associating feature information from different viewpoints.

[0008] To achieve the above objectives, the technical solution provided by this invention is: a multi-view... Figure 3 The D-object recognition method includes the following steps:

[0009] Step 1: For each 3D object, set up multiple virtual cameras from different angles to obtain a multi-view representation of the 3D object and extract multi-scale features for each view;

[0010] Step 2: Fuse the multi-scale features of each view;

[0011] Step 3: Use the global Transformer structure to exchange information on the fusion features of different views. After the information exchange, the features are aggregated into a global descriptor through an average pooling layer.

[0012] Step 4: Input the global descriptor into the classifier for classification.

[0013] Furthermore, in step 1, GoogLeNet is used as the backbone network, and the first six, twelve, and fifteen convolutional layers of GoogLeNet are used to extract features at different scales.

[0014] Furthermore, in step 2, the multi-scale features of each view are fused using the multi-scale feature clustering module MSFF based on the SE module. The specific implementation method is as follows;

[0015] For each multi-scale feature set , This indicates the number of views, with subscripts 1, 2, and 3 representing different scales. Convolutional kernels of different sizes are used to process features at different scales. Projecting onto the same width, height, and channel dimensions yields...

[0016]

[0017] in, Represents the features after projection. These represent the width, height, and number of channels of the feature map, respectively; then... The channels are directly added together to obtain a new feature map. This new feature map is then compressed to a smaller size using an average pooling layer. Three sets of weight values ​​were obtained using a layer perceptron (MLP) layer and a sigmoid activation function. , and Features at three scales were processed using weighted mapping. Weighting is applied along the channel dimension, and the feature maps at each scale are expanded to the desired size using the torch.expand_as() function. The three dilated feature maps are then used to extract view-level features through a max-pooling layer. ;

[0018]

[0019]

[0020] in It is the sigmoid function. and These represent the expansion and compression phases of the SE module, respectively.

[0021] Furthermore, the global Transformer structure includes multiple Transformer blocks, each of which includes a multi-head self-attention layer (MSA) and a feedforward network layer (FFN). The FFN contains two MLPs, and each MSA and FFN is preceded by a normalization layer (LN). The inputs and outputs of the MSA and FFN are connected using residual connections.

[0022] Furthermore, the specific processing procedure for the global Transformer structure is as follows;

[0023] Each input view-level feature ∈R w×h×c First, it was flattened into wh×c Size, and then pass through an average pooling layer to obtain view-level markers. ,Then, It is linked to the global token matrix:

[0024]

[0025] The input x0 of the global Transformer structure and the data processing of x0 when it passes through the t-th Transformer block are represented by the following equation:

[0026]

[0027]

[0028]

[0029] in, It is an absolute position encoding matrix. As an intermediate variable, These represent the multi-head self-attention layer and the feedforward network layer, respectively. LN represents the normalization layer. After passing through T Transformer blocks, the output will be... Input to average pooling layer:

[0030]

[0031]

[0032] in, Represents a view View-level tags The view-level marker after weighted mapping of T Transformer blocks, where y represents the global descriptor. This indicates average pooling.

[0033] Furthermore, the classifier is a fully connected layer.

[0034] The present invention also provides a multi-view Figure 3 The D-object recognition system includes the following modules:

[0035] The multi-scale feature extraction module sets up multiple virtual cameras from different angles for each 3D object to obtain a multi-view representation of the 3D object and extract multi-scale features for each view.

[0036] The multi-scale feature fusion module is used to fuse multi-scale features of each view;

[0037] The information interaction module is used to exchange information on the fusion features of different views using the global Transformer structure. After the information interaction, the features are aggregated into a global descriptor through the average pooling layer.

[0038] The classification module is used to input global descriptors into the classifier for classification.

[0039] Furthermore, the multi-scale feature fusion module uses the multi-scale feature clustering module MSFF based on the SE module to fuse the multi-scale features of each view. The specific implementation method is as follows:

[0040] For each multi-scale feature set , This indicates the number of views, with subscripts 1, 2, and 3 representing different scales. Convolutional kernels of different sizes are used to process features at different scales. Projecting onto the same width, height, and channel dimensions yields...

[0041]

[0042] in, Represents the features after projection. These represent the width, height, and number of channels of the feature map, respectively; then... The channels are directly added together to obtain a new feature map. This new feature map is then compressed to a smaller size using an average pooling layer. Three sets of weight values ​​were obtained using a layer perceptron (MLP) layer and a sigmoid activation function. , and Features at three scales were processed using weighted mapping. Weighting is applied along the channel dimension, and the feature maps at each scale are expanded to the desired size using the torch.expand_as() function. The three dilated feature maps are then used to extract view-level features through a max-pooling layer. ;

[0043]

[0044]

[0045] in It is the sigmoid function. and These represent the expansion and compression phases of the SE module, respectively.

[0046] Furthermore, the specific processing procedure for the global Transformer structure is as follows;

[0047] Each input view-level feature ∈R w×h×c First, it was flattened into wh×c Size, and then pass through an average pooling layer to obtain view-level markers. ,Then, It is linked to the global token matrix:

[0048]

[0049] The input x0 of the global Transformer structure and the data processing of x0 when it passes through the t-th Transformer block are represented by the following equation:

[0050]

[0051]

[0052]

[0053] in, It is an absolute position encoding matrix. As an intermediate variable, These represent the multi-head self-attention layer and the feedforward network layer, respectively. LN represents the normalization layer. After passing through T Transformer blocks, the output will be... Input to average pooling layer:

[0054]

[0055]

[0056] in, Represents a view View-level tags The view-level marker after weighted mapping of T Transformer blocks, where y represents the global descriptor. This indicates average pooling.

[0057] Furthermore, the global Transformer structure includes multiple Transformer blocks, each of which includes a multi-head self-attention layer (MSA) and a feedforward network layer (FFN). The FFN contains two MLPs, and each MSA and FFN is preceded by a normalization layer (LN). The inputs and outputs of the MSA and FFN are connected using residual connections.

[0058] Compared with the prior art, the advantages and beneficial effects of the present invention are: ① It utilizes the advantages of CNN and Transformer respectively to process the view obtained by projecting a three-dimensional object;

[0059] ② Using the Multi-Scale Feature Aggregation Module (MSFF) can effectively fuse features of different scales and enable the Transformer to better capture the correlation between features of different views.

[0060] ③ Although the proposed network is a simple structure, it achieves state-of-the-art recognition performance on ModelNet40 and ModelNet10 with 12-view settings. Attached Figure Description

[0061] Figure 1 A general framework for 3D object recognition.

[0062] Figure 2 A diagram of the Multiscale Feature Aggregation Module (MSFF).

[0063] Figure 3 It is a Transformer structural framework. Detailed Implementation

[0064] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0065] The proposed network consists of three parts: a multi-scale feature extraction module, a multi-scale feature aggregation module (MSFF), and a global Transformer module (GF). The first part uses a backbone network to extract features at different scales; the second part fuses features at different scales; and the third part uses a Transformer to capture the correlation between information in different views.

[0066] The method of this invention mainly includes the following steps:

[0067] Step 1: Extract multi-scale features

[0068] For each 3D object, we set up multiple virtual cameras from different angles to obtain a multi-view representation of the 3D object. We then project the 3D object onto M views. In, each view The size is We use GoogLeNet as the backbone network and extract features at different scales using the first six, twelve, and fifteen convolutional layers of GoogLeNet, respectively. Each view... Feature sets with three different scales were obtained by using different convolutional layers of GoogleNet. , All feature sets extracted for a set of views.

[0069] Step 2: Multi-scale Feature Aggregation Module (MSFF)

[0070] The Transformer architecture is used to capture the correlations between features from different views. The Multi-Head Self-Attention (MSA) module within the Transformer architecture enables sufficient information exchange between features from different views. However, the MSA module itself is computationally expensive, and using MSA layers for feature association across multiple scales leads to an exponential increase in computational cost. An effective method to save computational costs is to fuse features from different scales into a single feature.

[0071] Given the excellent performance of Squeeze and Excitation (SE) in fusing spatial and channel information, we designed an MSFF module based on the SE module to fuse multi-scale features for each view. Therefore, in this invention, we input the multi-scale feature set into the multi-scale feature clustering module (MSFF module) to fuse the multi-scale features for each view. Then, the outputs of the MSFF modules for all views are concatenated into a global feature set, such as... Figure 2 As shown.

[0072] The specific operation flow of the MSFF module is as follows:

[0073] For each feature set We use convolution kernels of different sizes to... Project onto the same width, height, and channel dimensions, and obtain:

[0074]

[0075] We The channels are directly added together to obtain a new feature map with a size of [size missing]. The new feature map is compressed to a smaller size using an average pooling layer. Three sets of weight values ​​were obtained using a layer perceptron (MLP) layer and a sigmoid activation function. , and Features at three scales were processed using weighted mapping. Weighting is applied along the channel dimension, and the feature maps at each scale are expanded to the desired size using the torch.expand_as() function. The three dilated feature maps are then used to extract view-level features through a max-pooling layer. The detailed procedure for the MSFF module is as follows.

[0076]

[0077]

[0078] in It is the sigmoid function. Furthermore, and These represent the expansion and compression phases of the SE module, respectively.

[0079] Step 3: Global Transformer Module (GF)

[0080] In the global Transformer module, view-level features from different perspectives interact through a Transformer structure to capture the correlations between different features. The input to this module is a set of view-level features processed by the MSFF module. The view-level feature set is aggregated into a global descriptor through an average pooling layer after information interaction. .

[0081] 3.1 Structural Composition

[0082] The global Transformer module jointly processes view-level features for each view. A standard Transformer module consists of an MSA layer and a feedforward network (FFN) layer. The FFN layer contains two MLPs, with a normalization layer (LN) preceding each MSA and FFN. The inputs and outputs of both the MSA and FFN layers use residual connections.

[0083] 3.2, Reasons for selection

[0084] The main reason we chose the Transformer architecture for feature interaction is that the multi-head self-attention (MSA) layer in the Transformer architecture allows all features from different views to be simultaneously correlated. In the feature interaction structures used in previous view-based methods (such as recurrent neural networks and graph convolutional neural networks), the view can only have local feature interactions and cannot simultaneously perceive features from all views. The basic Transformer module is as follows: Figure 3 As shown. We briefly describe the Transformer components used in our approach, namely position embedding, multi-head self-attention, and masking mechanisms.

[0085] 3.3, Absolute Position Encoding

[0086] We use absolute position encoding and add it to the token of each view to store location information. In our network, absolute position encoding... It was used to simulate the positional relationship between tokens in different views.

[0087] 3.4, Multi-head Self-Attention Module

[0088] The multi-head self-attention (MSA) layer is a core component of the Transformer. It uses three learnable linear projections to transform the input into a D-dimensional Query, Key, and Value, i.e., Q, K, and V. The attention module is calculated using the softmax formula (QK). T / D)V. In the MSA layer, a multi-head attention block uses n distinct heads to linearly project Q, K, and V. After projection, all n heads perform attention computations in parallel, and then the outputs of the n heads are concatenated and projected back to the dimension D. The MSA block can model the correlations between tokens in any view; however, an increase in the number of tokens and the dimension D leads to an exponential increase in computational cost. In other words, it is necessary to reduce the number of tokens and the dimension D to save computation, which is the main motivation for designing our MSFF aggregation module.

[0089] In the global Transformer, each input view-level feature ∈R w×h×c All will be flattened into wh×c Size, and then pass through an average pooling layer to obtain view-level markers. .Then, It is linked to the global token matrix:

[0090]

[0091] The input x0 of the global Transformer module and the data processing of x0 when it passes through the t-th Transformer block are represented by this equation:

[0092]

[0093]

[0094]

[0095] in It is an absolute positional encoding matrix. After passing through T Transformer blocks, we will output... Input to average pooling layer:

[0096]

[0097]

[0098] here, Represents a view View-level tags After the view-level label is weighted and mapped by T Transformer blocks, we use y as the global descriptor and input it into the fully connected (FC) layer for classification.

[0099] Step 4, perform the classification task.

[0100] Finally, the output of the global Transformer encoder is fused into the global descriptor through average pooling. We use this descriptor to perform 3D object classification tasks.

[0101] The present invention also provides a multi-view Figure 3 The D-object recognition system includes the following modules:

[0102] The multi-scale feature extraction module sets up multiple virtual cameras from different angles for each 3D object to obtain a multi-view representation of the 3D object and extract multi-scale features for each view.

[0103] The multi-scale feature fusion module is used to fuse multi-scale features of each view;

[0104] The information interaction module is used to exchange information on the fusion features of different views using the global Transformer structure. After the information interaction, the features are aggregated into a global descriptor through the average pooling layer.

[0105] The classification module is used to input global descriptors into the classifier for classification.

[0106] The specific implementation methods and steps of each module are the same, and will not be described in this invention.

[0107] To evaluate the performance of our proposed network method, we conducted experiments on the ModelNet40 and ModelNet10 datasets.

[0108] ModelNet40 contains 12,311 CAD shapes, divided into 40 categories. 9,843 3D shapes were used for training, and 2,468 shapes were used for testing. ModelNet10, a part of ModelNet40, contains 4,899 shapes across 10 categories.

[0109] Our model is implemented using the PyTorch platform. It was trained on a server with a GeForce RTX 3090 GPU. In our experiments, each 3D object was projected onto 12 views. A total of 200 epochs were trained. The AdamW optimizer was used to train the network with an initial learning rate of 0.0001, β1=0.9, and β2=0.98. Our hybrid 3D object recognition configuration is shown in Table I.

[0110] Table I. Configuration for Small Hybrid 3D Object Recognition

[0111]

[0112] We conducted comparative tests on the ModelNet10 dataset, comparing the MSFF module with the connection layer. The experimental results are shown in Table II.

[0113] Ablation studies using Table IIMSFF (Multiscale Feature Convergence) module

[0114]

[0115] Experimental results show that on the ModelNet10 dataset, the MSFF module can significantly improve the overall accuracy of the model and significantly reduce the model size. The model size using the connection layer is 245.6MB, while the model size using the MSFF module is 75.6MB.

[0116] To validate the effectiveness of our network, we compared it with a large number of typical network models, including model-based and view-based methods.

[0117] As shown in Table III, among model-based methods, LP-3DCNN and PointNet++ achieved recognition accuracies of 92.1% and 92.5%, respectively. Compared to view-based methods (including MVCNN, 3D2SeqViews, SeqViews2SeqLabels, GVCNN, MVT, and CARNet), our method outperforms in 12-view settings (i.e., each 3D object is rendered onto 12 images). On both the ModelNet10 and ModelNet40 datasets, our method achieves the highest recognition accuracy.

[0118] Table III compares the methods with state-of-the-art approaches on the MODELNET40 dataset.

[0119]

[0120] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A multi-view 3D object recognition method, characterized in that, The steps include the following: Step 1: For each 3D object, set up multiple virtual cameras from different angles to obtain a multi-view representation of the 3D object and extract multi-scale features for each view; Step 2: Fuse the multi-scale features of each view; In step 2, the multi-scale features of each view are fused using the multi-scale feature clustering module MSFF based on the SE module. The specific implementation method is as follows; For each multi-scale feature set , This indicates the number of views, with subscripts 1, 2, and 3 representing different scales. Convolutional kernels of different sizes are used to process features at different scales. Projecting onto the same width, height, and channel dimensions yields... in, Represents the features after projection. These represent the width, height, and number of channels of the feature map, respectively; then... The channels are directly added together to obtain a new feature map. This new feature map is then compressed to a smaller size using an average pooling layer. Three sets of weight values ​​were obtained using a layer perceptron (MLP) layer and a sigmoid activation function. , and Features at three scales were processed using weighted mapping. Weighting is applied along the channel dimension, and the feature maps at each scale are expanded to the desired size using the torch.expand_as() function. The three dilated feature maps are then used to extract view-level features through a max-pooling layer. ; in It is the sigmoid function. and These represent the expansion and compression phases of the SE module, respectively. Step 3: Use the global Transformer structure to exchange information on the fusion features of different views. After the information exchange, the features are aggregated into a global descriptor through an average pooling layer. Step 4: Input the global descriptor into the classifier for classification.

2. The multi-view 3D object recognition method as described in claim 1, characterized in that: In step 1, GoogLeNet is used as the backbone network, and the first six, twelve, and fifteen convolutional layers of GoogLeNet are used to extract features at different scales.

3. The multi-view 3D object recognition method as described in claim 1, characterized in that: The global Transformer structure consists of multiple Transformer blocks. Each Transformer block includes a multi-head self-attention layer (MSA) and a feedforward network layer (FFN). The FFN contains two MLPs. Each MSA and FFN is preceded by a normalization layer (LN). The inputs and outputs of the MSA and FFN are connected using residual connections.

4. The multi-view 3D object recognition method as described in claim 3, characterized in that: The specific processing procedure for the global Transformer structure is as follows; Each input view-level feature ∈R w×h×c First, it was flattened into wh×c Size, and then pass through an average pooling layer to obtain view-level markers. ,Then, It is linked to the global token matrix: The input x0 of the global Transformer structure and the data processing of x0 when it passes through the t-th Transformer block are represented by the following equation: in, It is an absolute position encoding matrix. As an intermediate variable, These represent the multi-head self-attention layer and the feedforward network layer, respectively. LN represents the normalization layer. After passing through T Transformer blocks, the output will be... Input to average pooling layer: in, Represents a view View-level tags The view-level marker after weighted mapping of T Transformer blocks, where y represents the global descriptor. This indicates average pooling.

5. The multi-view 3D object recognition method as described in claim 1, characterized in that: The classifier is a fully connected layer.

6. A multi-view 3D object recognition system, characterized in that, Includes the following modules: The multi-scale feature extraction module sets up multiple virtual cameras from different angles for each 3D object to obtain a multi-view representation of the 3D object and extract multi-scale features for each view. The multi-scale feature fusion module is used to fuse multi-scale features of each view; The multi-scale feature fusion module uses the multi-scale feature clustering module MSFF based on the SE module to fuse the multi-scale features of each view. The specific implementation method is as follows; For each multi-scale feature set , This indicates the number of views, with subscripts 1, 2, and 3 representing different scales. Convolutional kernels of different sizes are used to process features at different scales. Projecting onto the same width, height, and channel dimensions yields... in, Represents the features after projection. These represent the width, height, and number of channels of the feature map, respectively; then... The channels are directly added together to obtain a new feature map. This new feature map is then compressed to a smaller size using an average pooling layer. Three sets of weight values ​​were obtained using a layer perceptron (MLP) layer and a sigmoid activation function. , and Features at three scales were processed using weighted mapping. Weighting is applied along the channel dimension, and the feature maps at each scale are expanded to the desired size using the torch.expand_as() function. The three dilated feature maps are then used to extract view-level features through a max-pooling layer. ; in It is the sigmoid function. and These represent the expansion and compression phases of the SE module, respectively. The information interaction module is used to exchange information on the fusion features of different views using the global Transformer structure. After the information interaction, the features are aggregated into a global descriptor through the average pooling layer. The classification module is used to input global descriptors into the classifier for classification.

7. The multi-view 3D object recognition system as described in claim 6, characterized in that: The specific processing procedure for the global Transformer structure is as follows; Each input view-level feature ∈R w×h×c First, it was flattened into wh×c Size, and then pass through an average pooling layer to obtain view-level markers. ,Then, It is linked to the global token matrix: The input x0 of the global Transformer structure and the data processing of x0 when it passes through the t-th Transformer block are represented by the following equation: in, It is an absolute position encoding matrix. As an intermediate variable, These represent the multi-head self-attention layer and the feedforward network layer, respectively. LN represents the normalization layer. After passing through T Transformer blocks, the output will be... Input to average pooling layer: in, Represents a view View-level tags The view-level marker after weighted mapping of T Transformer blocks, where y represents the global descriptor. This indicates average pooling.

8. The multi-view 3D object recognition system as described in claim 7, characterized in that: The global Transformer structure consists of multiple Transformer blocks. Each Transformer block includes a multi-head self-attention layer (MSA) and a feedforward network layer (FFN). The FFN contains two MLPs. Each MSA and FFN is preceded by a normalization layer (LN). The inputs and outputs of the MSA and FFN are connected using residual connections.

Images