Method and system for carrying out 3D shape recognition based on multi-modal neural network
By combining graph convolution and Transformer multimodal neural networks, the problems of complexity and high computational cost of multimodal fusion methods are solved, achieving efficient 3D shape recognition and improving recognition accuracy and model adaptability.
Patent Information
- Application Number
- CN202410909175.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-08
- Publication Date
- 2024-11-08
Smart Images

Figure CN118918576A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a method and system for 3D shape recognition based on a multimodal neural network of graph convolution and transformer. Background Art
[0002] At present, with the rapid development of 3D sensing technology (such as LiDAR) and the emergence of low-cost, miniaturized sensing devices (such as Microsoft Kinect and Google Tango), the collection of 3D data is more convenient and feasible than ever before. How to analyze and recognize these 3D data has become an important research topic in the field of computer vision and pattern recognition. According to the various forms of 3D shape data, the current 3D shape recognition methods can be divided into four categories: multi-view based methods, point cloud based methods, voxel based methods and multimodal fusion based methods. There are certain limitations in the representation of 3D shapes by single modal data, so the methods based on multimodal fusion show better performance. The methods based on multimodal fusion aim to integrate multimodal data to make up for the lack of data and obtain better 3D shape descriptors. For example, FusionNet uses multi-view and voxel data. PVNet uses point cloud and multi-view data. Both have achieved good results in 3D shape recognition.
[0003] Multi-view based methods: MVCNN is a pioneering work based on multi-view methods, which uses the initial CNN to extract features of each view and uses the maximum pooling method to compile the features of multiple views into a unified global descriptor. However, excessive use of max-pooling to aggregate multi-view features will only retain the maximum elements of each view, resulting in a large amount of information loss. Subsequent methods, such as GVCNN and MVCNN-multiRES, have improved the multi-view fusion strategy to obtain better 3D shape descriptors. Wei et al. proposed the View-GCN method, which abandoned the traditional CNN and introduced the concept of graph into multi-view learning for the first time. Experimental results show that introducing graph structure into multi-view learning is very effective. Since the camera angle for multi-view sampling of a 3D shape is often fixed, it is difficult to avoid the loss of structural information of the 3D shape when collecting multiple views. To solve the above problems, Hamdi et al. proposed the MVTN model. MVTN can find the best view of a 3D shape, perform end-to-end training, and migrate to any multi-view network to find the best multi-view. Sun et al. proposed a view set attention network (ViewFormer) model. They introduced an attention mechanism in view selection, proposed a new perspective, and provided flexibility for view selection. In reality, it is not possible to obtain a complete view of a 3D shape. Kanezaki et al. proposed a RotationNet model that feeds the view pose label into the network as a learned latent variable and automatically aligns the object pose of the 3D model in the image when predicting the 3D model. This method greatly improves the recognition accuracy and only requires a part of the multiple views to complete the recognition.
[0004] Point cloud-based methods: Qi et al. introduced the PointNet method to directly process the original point cloud. This method takes the point cloud as input, realizes permutation invariance through symmetric functions, and solves the irregular characteristics of the point cloud. PointNet learns features point by point through multiple MLP layers and extracts global shape features through the maximum pooling layer. However, since each point in PointNet learns features independently, the local relationship information between points is lost. Subsequently, Qi et al. introduced the PointNet++ method to address the shortcomings of the original PointNet. PointNet++ learns features from local geometry and abstracts them layer by layer. Due to the powerful functions of PointNet and PointNet++, many networks are developed based on them, such as PointWeb and PointASNL. Traditional point cloud sampling methods are mostly based on mathematical statistics, except for random sampling and farthest point sampling. Wu et al. proposed a point cloud edge sampling (APES) method based on non-generative attention for 3D point cloud recognition. Inspired by the Canny image edge detection algorithm and attention mechanism, APES calculates the correlation between points and samples edge points with high correlation. Zhao et al. proposed MANet and introduced a soft attention mechanism. Different from the above methods, PointView-GCN proposes a new data model to migrate multiple views to point cloud data. PointView-GCN directly samples the point cloud data of the 3D shape from different angles. The resulting view is the point cloud data projected from the 3D shape onto the 2D plane.
[0005] Methods based on multimodal fusion: You et al. proposed a PVNet model that combines point clouds and multi-views. PVNet is divided into two parts: point clouds and multi-views. First, the initial CNN is used to extract the features of the two parts, and then an embedded attention fusion strategy is proposed to fuse the features of the two parts. Hegde et al. proposed Fusionnet using both multi-view and voxel data. Voxel data is concretized as three-dimensional objects scattered into binary voxels in space. Fusionnet is a fusion of two convolutional neural networks, one trained on multiple views and the other trained on voxelized 3D shapes. Each network of FusionNet considers multiple views or directions of each object before classification, and Fusionnet uses multi-views of 20 views and 60 voxels.
[0006] Graph Convolutional Networks: Kipf et al. first proposed a graph convolutional network (GCN). They used spectral graph theory to implement convolution operations on topology. Li et al. proposed an adaptive graph convolutional neural network (AGCN) that uses residual graphs to enhance graphs, which are formed by determining the distance between pairs of nodes. AGCN identifies hidden structural relationships by using the adjacency matrix of the graph and constructs the adjacency matrix of the residual graph through a learnable distance function with the features of two nodes as input. DGCN introduces a dual graph convolution structure with two parallel graph convolutional layers. Chen et al. introduced the concept of fast learning in graph convolutional networks through importance sampling (FastGCN). FastGCN adopts a fixed node sampling strategy for each convolutional layer. FastGCN understands graph convolution as an integral transformation of the node embedding function on a probability measure. Cluster-GCN uses a graph clustering algorithm to sample subgraphs and then performs graph convolution on the nodes within the selected subgraphs.
[0007] Transformer method: Transformer was first used in natural language processing (NLP). Due to the excellent performance of Transformer model in NLP, transferring Transformer model to computer vision has gradually become a new research direction. Carion et al. proposed the End-to-End Object Detection Using Transformer (DERT) model, which first introduced Transformer for object detection. DERT proposed a set prediction method to solve the target detection task, using the model architecture of CNN+Transformer encoder+Transformer decoder. Chen et al. proposed a pixel-based generative pre-training model, using GPT to perform unsupervised pre-training on large-scale image samples. Vision Transformer (VIT) is a pioneering model of Transformer in the field of CV. VIT reuses the model structure of Transformer in NLP to solve image problems and processes images into a form similar to token sequences in NLP. Experiments have shown that VIT can achieve good results by pre-training on a large amount of sufficient data. Han et al. proposed the TNT model based on the VIT model, subdividing VIT into sub-blocks and treating each block as a sentence. The elements obtained by further subdividing the blocks are regarded as words. TNT believes that it is necessary to divide the image into blocks more coarsely. It is assumed that the block is decomposed into sub-blocks with finer granularity.
[0008] Through the above analysis, the problems and defects of the prior art are as follows:
[0009] (1) Complexity of multimodal fusion methods: Multimodal fusion methods are highly complex, and the training and inference processes require a lot of computing resources. In addition, the feature alignment and fusion strategies between different modal data in multimodal fusion methods still need to be further optimized.
[0010] (2) Node sampling problem of graph convolutional network (GCN) methods: For example, FastGCN and Cluster-GCN have certain limitations in node sampling strategies and cannot fully and efficiently capture the complex relationships in the graph structure. In addition, graph convolutional networks still face computing and storage challenges when processing large-scale graph data.
[0011] (3) Computational overhead of the Transformer method: The Transformer model has a high computational overhead, especially when processing high-resolution images and large-scale data. The model training and inference process requires high hardware resources, making it difficult to apply efficiently in resource-limited environments. Summary of the invention
[0012] In view of the problems existing in the prior art, the present invention provides a method and system for 3D shape recognition based on a multimodal neural network of graph convolution and transformer.
[0013] The present invention is implemented as follows: a method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer, comprising:
[0014] Capture multiple views and complete the rendering of 3D and 2D multi-views of 3D objects; use the convolutional neural network CNN to extract the features of the rendered 3D and 2D multi-views, and fuse the extracted features to complete the construction of the graph; send the graph to the graph convolution-transformer network to obtain the global descriptor, and complete the recognition of the 3D object based on the descriptor.
[0015] Furthermore, the graph convolution-transformer network is a hierarchical network based on optimal view embedding, local graph-transformer, non-local graph labeling and selective view sampling, including:
[0016] A confidence-based best view embedding method is used to select the best 3D shape view and embed the features of the best view into other views;
[0017] Perform non-local graph tag embedding and feed all nodes into the local graph-transformer encoder to update node features;
[0018] The embedded tokens are extracted from the embedding, the non-local graph tokens are recorded as the multi-view aggregation features of the current layer of the graph G, and the non-local graph tokens are concatenated with the multi-view aggregation features of other layers to obtain the global features of the 3D shape.
[0019] Further, it is characterized in that the method steps are:
[0020] Step S1: Prepare the 2D multi-views to be trained, and render 20 2D multi-views for each 3D shape; select the traditional CNN network as the feature extractor of the 2D multi-view, and the extracted 2D multi-view features are recorded as I i ;
[0021] Step S2: Prepare the 3D multi-views to be trained. 3D multi-views refer to points projected onto a 2D plane from different perspectives. Use the farthest point sampling FPS to sample 2048 3D shape points from each angle, and sample 20 angles for each 3D shape. After generating the 3D multi-view dataset, select Pointnet++ as the feature extractor for point cloud multi-views, perform T-Net affine transformation on the input unordered point cloud data, and record the extracted features as P i ;
[0022] Step S3: Build the graph and mark the i-th node of graph G as V i , V i The feature carried is denoted as F i , F i represents the fusion feature of the i-th view, and the virtual camera coordinate is defined as C i ; V i and F i The definition of V is subject to the following equation: i =(C i , F i ), It is a tensor concatenation operation that concatenates two-dimensional and three-dimensional features to enable the network to learn more graphic node information about 3D shapes;
[0023] Step S4: Use V i Construct the graph G. Choose to construct the graph G by constructing the adjacency matrix S. Define S as follows: V ij =ψ(G ij θ s ), G ij =[V i , V j , V i -V j ,||V i -V j ||2], G ijrepresents the spatial relationship between two views, [] represents the process of connecting elements into vector form; ψ represents the process of connecting elements with parameters θ s For nonlinear embedding of features, ψ is configured as a four-layer multilayer perceptron with LeakyReLU activation;
[0024] Step S5: Define the neural network for training graph G, namely graph transformer network GTN, which is based on optimal view embedding, local Figure 1 transformer, non-local graph labeling, and selective view sampling; use the confidence-based best view embedding method to select the best 3D shape view and embed the features of the best view into other views, and denote the best view as B v ;
[0025] Step S6: After obtaining B v (where 1≤B v ≤N), B v Embed into other views. The embedding operation is as follows:
[0026]
[0027] Step S7: Perform non-local graph tag embedding. First, perform a tag embedding operation on the graph of the current layer, and the tag is recorded as a token. Then define a learnable tensor as a token. The token has the same shape as the graph node and its initial value is 0. The token is embedded in each layer of the graph G. The operation is as follows: G l The number of nodes increases from N to N+;
[0028] Step S8: Feed all nodes into the local graph-transformer encoder to update node features; after the nodes are fed in, the nodes are first pos_embeddning operated, pos_embeddning embeds the node coordinates into the node features, and the nodes with coordinate information are sent to the network for learning. The node feature update operation in the local graph transformer encoder is shown as follows: i =φ(S i F i C i W;θ e ), where φ represents a nonlinear transformation, W is a learnable weight matrix, and φ is defined as a sequence of Batch Normalization and LeakyReLU activations, with parameters θ e Control; The updated node is the same as VIT and needs to go through multiple layers of Norm, Multi-Head Attention and MLP operations;
[0029] Step S9: Local Figure 1 After the transformer updates the node features, it extracts the embedded tokens from the embedding, records the non-local graph tokens as the multi-view aggregation features of the current layer of the graph G, and concatenates the non-local graph tokens with the multi-view aggregation features of other layers to obtain the global features of the 3D shape; the global features are defined as F g , the L-layer non-local graph token is recorded as The specific implementation of G is as follows:
[0030] The non-local graph token is a learnable tensor with an initial value of 0;
[0031] Step S10: adopt a selective sampling strategy to coarsen the graph;
[0032] Step S11: In the test phase, for the input 2D and 3D multi-views, first use the traditional CNN to extract the 2D and 3D multi-view features respectively, then fuse the features and construct the graph; finally, update the graph node features through GTN to obtain distinguishable global feature descriptors, then perform probability mapping on the global feature descriptors, and select the appropriate classification according to the obtained probability.
[0033] Furthermore, the complete definition of the best view obtained in step S5 is as follows:
[0034] Ω is a CNN, and Ω.fc represents the fully connected layer of CNN.
[0035] Further, the step S10 includes:
[0036] First, perform FPS sampling on the graph nodes of the current level, and select the node with the largest distance between views each time; use KNN and view selector. After performing FPS sampling on the graph of the current level, iteratively select sampling points for KNN operation, and send the points selected by KNN and the sampling points to the view selector; the implementation of the view selector is shown in the following formula: where Vs(·) represents a learnable view selector; Vs(·) is implemented using a 2-layer MLP, where V fk Represents the node set selected by FPS and KNN. The Max operation returns the maximum value in the Nc class, and argmax returns the number of maximum values of all nodes in KNN, that is, the number of nodes selected by the view selector. The nodes selected by the view selector are used as the graph nodes of the next layer, and the node features updated by the local graph transformer are used as the input features of the graph nodes of the next layer.
[0037] Another object of the present invention is to provide a 3D shape recognition system based on a multimodal neural network of graph convolution and transformer, comprising:
[0038] Data acquisition module: used to capture 3D and 2D multi-views of 3D objects and render them.
[0039] Feature extraction module: Use convolutional neural network (CNN) to extract features of rendered 3D and 2D multi-views.
[0040] Graph construction module: fuses the extracted features and builds the graph structure.
[0041] Graph convolution-Transformer network module: The graph structure is sent into the graph convolution-Transformer network to obtain the global descriptor.
[0042] Recognition module: recognizes three-dimensional objects based on global descriptors.
[0043] Furthermore, the graph convolution-Transformer network module includes:
[0044] Best View Embedding Unit: Selects the best 3D shape view based on confidence and embeds its features into other views.
[0045] Local Graph-Transformer Encoder: Non-local graph token embedding is performed and all nodes are fed into the local graph-Transformer encoder to update node features.
[0046] Non-local graph tag embedding unit: extracts the embedded token from the embedding, uses it as the multi-view aggregate feature of the current layer of the graph, and concatenates it with the multi-view aggregate features of other layers to obtain the global features of the 3D shape.
[0047] Furthermore, the identification module includes:
[0048] Feature fusion unit: extracts 2D and 3D multi-view features respectively and fuses them.
[0049] Graph construction unit: constructs the graph, splices the features, and forms graph node information.
[0050] Node update unit: Update graph node features through GTN to obtain distinguishable global feature descriptors.
[0051] Classification unit: Probability mapping of global feature descriptors and selection of appropriate classification based on the obtained probability.
[0052] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer.
[0053] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of a method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer.
[0054] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:
[0055] First, MVPAI-GTN integrates the features of multi-view and point cloud modalities to achieve an organic combination of 2D view information and 3D shape structure information. This fusion effectively makes up for the shortcomings of single modality data in representing 3D shapes and significantly improves the accuracy of 3D shape recognition.
[0056] MVPAI-GTN innovatively combines graph structure and transformer methods, which can effectively aggregate information from different views and generate representative global descriptors. Through the synergy of local graph-transformer and non-local graph token, efficient aggregation of multi-level graph features is achieved, enhancing the recognition ability of the model.
[0057] The proposed best view embedding method effectively reduces the negative impact of low-quality views on model accuracy by selecting the best 3D shape view and embedding it into other views. This strategy improves recognition accuracy by acquiring more discriminative deep features.
[0058] MVPAI-GTN adopts a modular design, which allows multi-view features to be processed and fused at different stages. For example, point cloud multi-view features are extracted through Pointnet++, image multi-view features are extracted using VGG16, and node features are updated through the graph transformer network. This design not only improves the flexibility of the model, but also enhances its scalability and adaptability.
[0059] MVPAI-GTN introduces a confidence-based best view embedding and best view loss to force the network to select more discriminative views, thereby optimizing the accuracy of the model. Through the combination of global view loss, best view loss and global feature loss, the model achieves efficient optimization of view features at different levels during training.
[0060] Second, the present invention solves the following major technical problems in the prior art:
[0061] Limitations of unimodal data: Traditional unimodal methods risk information loss when describing three-dimensional shapes and have difficulty in fully capturing all the characteristics of the shape.
[0062] Information loss in multi-view methods: For example, MVCNN uses the max-pooling method to fuse multi-view features, which results in the loss of a large amount of useful information.
[0063] Local relationship loss of point cloud methods: For example, PointNet methods cannot effectively capture the local relationship information between points in the point cloud.
[0064] High computational cost of complex networks: Existing complex networks (such as multi-view, point cloud, and voxel-based methods) have high computational costs during training and inference, making them difficult to use in practice.
[0065] Diversity and complexity of 3D shape data: Due to the diversity and complexity of 3D shape data, a single method is difficult to adapt to various application scenarios.
[0066] The present invention solves the above problems through the following innovations and technical advances:
[0067] Multimodal fusion: Combining 2D multi-view and 3D point cloud data, using CNN to extract features and build graph structures, we achieve efficient fusion of multimodal data and make up for the shortcomings of single-modal data.
[0068] Best view embedding: The confidence-based best view embedding method is used to select the best 3D shape view and embed its features into other views, effectively improving the recognition accuracy.
[0069] Graph Convolution-Transformer Network: It introduces a hierarchical network structure of graph convolution and Transformer, and achieves efficient global feature extraction through non-local graph label embedding and selective view sampling.
[0070] Adaptive feature fusion: During the graph construction process, two-dimensional and three-dimensional features are adaptively fused through the adjacency matrix and nonlinear embedding methods, which improves the network's understanding and recognition of three-dimensional shapes.
[0071] Efficient training and reasoning: By introducing a selective sampling strategy and an optimized graph convolution-Transformer network structure, the computational cost is greatly reduced and the efficiency of training and reasoning is improved.
[0072] Flexible network structure: A hierarchical network structure based on local graph-Transformer and non-local graph labeling is designed, which can flexibly adapt to different types of 3D shape data and improve the generalization ability of the model.
[0073] The significant technical advances of the present invention are:
[0074] Improved recognition accuracy: The accuracy of 3D shape recognition is significantly improved through optimal view embedding and multimodal data fusion.
[0075] Optimization of information retention: In the process of feature extraction and fusion, the loss of information is effectively avoided, and the overall performance of the model is improved.
[0076] Improved computational efficiency: Through selective sampling and optimized network structure, the computational cost is significantly reduced, and the training and reasoning efficiency of the model is improved.
[0077] Enhanced adaptability and generalization capabilities: The flexible network structure enables the model to adapt to various types of 3D shape data, improving the generalization capabilities in practical applications.
[0078] Third, traditional 3D shape recognition methods usually use single modality feature extraction, which cannot fully capture multi-view information, resulting in insufficient recognition accuracy and robustness.
[0079] The present invention uses traditional convolutional neural network (CNN) and PointNet++ to extract multi-view features by setting the sampling parameters of 2D multi-view (20 views) and 3D multi-view in detail. Specific parameters include sampling 2048 points from different viewing angles for each 3D shape and sampling 20 angles for each shape.
[0080] The optimization of multi-view feature extraction parameters improves the accuracy and diversity of feature extraction, fully utilizes the complementary information of two-dimensional and three-dimensional views, and greatly improves the accuracy and robustness of 3D shape recognition.
[0081] Traditional methods lack effective multimodal feature fusion and global feature extraction, resulting in poor recognition performance of complex shapes.
[0082] The present invention proposes a multimodal neural network based on graph convolution and Transformer, which adopts a hierarchical network structure of confidence-based optimal view embedding, local graph-Transformer encoding, non-local graph labeling and selective view sampling to fuse two-dimensional and three-dimensional view features.
[0083] The combination of graph convolution and Transformer network realizes the efficient fusion of multimodal features and the extraction of global features, significantly improving the recognition ability of complex 3D shapes.
[0084] In terms of graph construction and node feature updating, existing methods fail to fully express multi-view information, resulting in insufficient node features.
[0085] The present invention constructs a graph through farthest point sampling (FPS) and K nearest neighbor (KNN), defines node features and fuses two-dimensional and three-dimensional view information. The node features are updated using local graph-Transformer, and non-local graph tags are obtained by embedding Tokens to enrich node features.
[0086] The optimized graph construction and node feature updating strategies enhance the expressiveness of node features and improve the comprehensive utilization efficiency of multi-view information, thereby improving the accuracy and stability of 3D shape recognition.
[0087] The lack of an effective view selection mechanism results in the failure to fully utilize some view information, affecting the recognition effect.
[0088] The present invention optimizes view selection through the confidence-based optimal view embedding method, combines FPS and KNN with a view selector, and implements a selective sampling strategy to ensure that the selected views have high representativeness and information content.
[0089] By optimizing view selection and sampling strategies, the accuracy and effectiveness of view selection are improved, ensuring the full utilization of view information, thereby significantly improving the overall performance and efficiency of 3D shape recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0090] Figure 1 It is a flow chart of a method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer provided by an embodiment of the present invention;
[0091] Figure 2 This is a structural overview of MVPAI-GTN provided by an embodiment of the present invention;
[0092] Figure 3 It is an overview of the graph-transformer network (GTN) in MVPAI-GTN provided by an embodiment of the present invention;
[0093] Figure 4 It is an overview of the best view embedding (Best View-embedding) in MVPAI-GTN provided by an embodiment of the present invention;
[0094] Figure 5 It is an overview of the local graph-transformer in MVPAI-GTN provided by an embodiment of the present invention;
[0095] Figure 6This is an overview of selective view sampling in MVPAI-GTN provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0096] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0097] Example 1: 3D shape recognition of industrial parts
[0098] In industrial manufacturing, accurate recognition and classification of the 3D shapes of various parts is crucial for the efficient operation of automated production lines. Traditional methods are easily affected by complex shapes and diverse viewing angles, and the recognition accuracy and speed are not ideal.
[0099] Implementation steps:
[0100] 1) Multi-view capture:
[0101] Place industrial parts on a rotating platform and render them in 3D and 2D multi-views from 20 different angles.
[0102] Use high-resolution cameras and laser scanners to obtain 2D images and 3D point cloud data at each angle, respectively.
[0103] 2) Feature extraction:
[0104] Use convolutional neural network (CNN) to extract 2D multi-view features. Set the network structure of CNN and use the pre-trained ResNet50 model to extract the features of each 2D view.
[0105] Use PointNet++ to extract 3D multi-view features. Sample 2048 points for each view and extract point cloud features through T-Net affine transformation.
[0106] 3) Graph construction and node labeling:
[0107] Each view is constructed as a node of the graph G, and the node features are marked as the concatenation results of the extracted 2D and 3D features.
[0108] Define the virtual camera coordinates, convert the view coordinates into node feature coordinates, and construct the adjacency matrix S of the graph.
[0109] 4) Graph Convolution-Transformer Network Processing:
[0110] Using the confidence-based best view embedding method, the best 3D shape view is selected and the features of the best view are embedded into other views.
[0111] Perform non-local graph label embedding and feed all nodes into the local graph-Transformer encoder to update node features.
[0112] Extract the embedded tokens, perform multi-view aggregation feature splicing, and obtain the global features of the 3D shape.
[0113] 5) Classification and Identification:
[0114] Use the global feature descriptor to perform probability mapping through the Softmax classifier and select the appropriate classification label.
[0115] During the testing phase, the same process is used to extract features and update node features to achieve rapid identification and classification of industrial parts.
[0116] Effect:
[0117] In this embodiment, multi-view feature extraction and graph convolution-Transformer network processing are used to improve the accuracy and speed of 3D shape recognition of industrial parts.
[0118] It achieves efficient processing of complex shapes and diverse viewing angles, and is suitable for real-time monitoring and quality inspection of automated production lines.
[0119] Example 2: 3D shape recognition and restoration of cultural relics
[0120] In the protection and restoration of cultural relics, accurately identifying and classifying the 3D shapes of cultural relics is crucial for the refinement and precision of restoration work. Traditional methods often fail to achieve ideal results when dealing with complex shapes and detailed features of cultural relics.
[0121] Implementation steps:
[0122] 1) Multi-view capture:
[0123] The cultural relics are 3D scanned and photographed from multiple perspectives, and three-dimensional and two-dimensional multiple views of the cultural relics are obtained from 20 different angles.
[0124] Use high-resolution cameras and 3D scanners to obtain two-dimensional images and three-dimensional point cloud data at each angle, respectively.
[0125] 2) Feature extraction:
[0126] Use convolutional neural network (CNN) to extract 2D multi-view features. Set the network structure of CNN and use the pre-trained VGG16 model to extract features for each 2D view.
[0127] Use PointNet++ to extract 3D multi-view features. Sample 2048 points for each view and extract point cloud features through T-Net affine transformation.
[0128] 3) Graph construction and node labeling:
[0129] Each view is constructed as a node of the graph G, and the node features are marked as the concatenation results of the extracted 2D and 3D features.
[0130] Define the virtual camera coordinates, convert the view coordinates into node feature coordinates, and construct the adjacency matrix S of the graph.
[0131] 4) Graph Convolution-Transformer Network Processing:
[0132] Using the confidence-based best view embedding method, the best 3D shape view is selected and the features of the best view are embedded into other views.
[0133] Perform non-local graph label embedding and feed all nodes into the local graph-Transformer encoder to update node features.
[0134] Extract the embedded tokens, perform multi-view aggregation feature splicing, and obtain the global features of the 3D shape.
[0135] 5) Classification, identification and repair:
[0136] Use global feature descriptors, perform probability mapping through SVM classifiers, and select appropriate classification labels to assist restorers in classifying and restoring cultural relics.
[0137] During the testing phase, the same process is used to extract features and update node features to achieve rapid identification and restoration guidance of cultural relics.
[0138] Effect:
[0139] In this embodiment, the accuracy and speed of 3D shape recognition of cultural relics are improved through multi-view feature extraction and graph convolution-Transformer network processing.
[0140] The invention realizes efficient processing of complex shapes and detailed features of cultural relics, assists restoration personnel in fine restoration, and protects the integrity and historical value of cultural relics. The invention discloses a 3D shape recognition method based on a multimodal neural network of graph convolution and transformer. The method is implemented by the following steps:
[0141] Step S1: Figure 2As shown in the figure, prepare the 2D multi-views to be trained. We refer to the method in MVCNN and render 20 2D multi-views for each 3D shape. Each view is a 2D projection of the 3D shape under the camera angle. We choose the traditional CNN network (VGG16 or ResNet50) as the feature extractor of the 2D multi-view. In order to merge with the features of the 3D multi-view, the shape of the 2D multi-view features is the same as that of the 3D multi-view. The extracted 2D multi-view features are recorded as I i .
[0142] Step S2: Figure 2 As shown, prepare the 3D multi-views that need to be trained. 3D multi-views refer to points projected onto a 2D plane from different perspectives (eliminating duplicate points). Compared with 2D multi-views, 3D multi-views retain the structural information of 3D shapes under different views. We refer to the method in PointView-Gcn. We use the farthest point sampling (FPS) to sample 2048 3D shape points from each angle, and sample 20 angles for each 3D shape. After generating the 3D multi-view dataset, we further extract the initial features of each 3D view. We choose Pointnet++ as the feature extractor for point cloud multi-views, and perform T-Net affine transformation on the input unordered point cloud data. In this way, the influence caused by the perturbation of the point cloud is greatly reduced, and the extracted features are recorded as P i .
[0143] Step S3: After obtaining the 2D and 3D multi-view features, we need to construct the graph. We mark the i-th node of the graph G as V i , V i The feature carried is denoted as F i , F i represents the fusion feature of the i-th view. The virtual camera coordinate is defined as C i . V i and F i The definition of V is subject to the following equation: i =(C i , F i ), It is a tensor concatenation operation. Concatenating two-dimensional and three-dimensional features allows the network to learn more graphic node information about 3D shapes.
[0144] Step S4: Get graph node V i Then, we use V i Construct the graph G. We choose to construct the graph G by constructing the adjacency matrix S, and we define S as follows: V ij =ψ(G ij θ s ), G ij =[Vi , V j , V i -V j ,||V i -V j ||2], G ij represents the spatial relationship between two views, [] represents the process of connecting elements into vector form. ψ represents the parameter θ s is the nonlinear embedding of the features. We configure ψ as a four-layer multilayer perceptron (MLP) with LeakyReLU activation.
[0145] Step S5: After completing the construction of graph G, we further define the neural network for training graph G, which is the graph transformer network (GTN) we proposed. Figure 3 As shown, GTN is based on the best view embedding, local Figure 1 Transformer, non-local graph labeling and selective view sampling. Different views have different contributions to 3D shape recognition. Some views contribute little to 3D shape recognition or even reduce the recognition effect. To this end, we propose Figure 4 The confidence-based best view embedding method shown in Figure 1 selects the best 3D shape view and embeds the features of the best view into other views to improve recognition performance. We denote the best view as B v , the complete definition of the optimal view is as follows:
[0146] Ω is a CNN, and Ω.fc represents the fully connected layer of CNN.
[0147] Step S6: After obtaining B v (where 1≤B v ≤N), we need to convert B v Embed into other views. The embedding operation is as follows:
[0148]
[0149] Step S7: After completing the best view embedding, we perform non-local graph label embedding. Figure 5 As shown in the figure, we first perform a token embedding operation on the graph of the current layer, and the token is recorded as a token. Then we define a learnable tensor as a token. The token has the same shape as the graph node and its initial value is 0. We embed tokens in each layer of the graph G, and the operation is as follows: G l The number of nodes increases from N to N+1.
[0150] Step S8: After the non-local graph label embedding is completed, we feed all nodes into Figure 5 The part shown Figure 1 Update node features in the transformer encoder. After the node is fed in, we first perform a pos_embeddning operation on the node. The pos_embeddning embedding embeds the node coordinates into the node features and sends the nodes with coordinate information to the network for learning. In this way, the network can learn the structured information of the three-dimensional shape to a certain extent. In summary, the operation of updating node features in the local graph transformer encoder is as follows: F i =φ(S i F i C i W;θ e ), where φ represents a nonlinear transformation and W is a learnable weight matrix. We define φ as a sequence of BatchNormalization and LeakyReLU activations, with parameters θ e The updated nodes are the same as VIT and need to go through multiple layers of Norm, Multi-HeadAttention, and MLP operations.
[0151] Step S9: After the local graph-transformer updates the node features, Figure 2 As shown in Figure 2, we extract the embedded token from the embedding, record the non-local graph token as the multi-view aggregation feature of the current layer of graph G, and concatenate the non-local graph token with the multi-view aggregation features of other layers to obtain the global feature of the 3D shape. We define the global feature as F g . The L-layer non-local graph token is recorded as The specific implementation of G is as follows:
[0152] As a learnable tensor with an initial value of 0, the non-local graph token can effectively learn the features of other nodes. Compared with the brute force maximum pooling operation on all nodes, the loss of multi-view information is greatly reduced.
[0153] Step S10: GTN is a hierarchical network. Each time through GTN, the graph nodes are reduced to expand the receptive field. We adopt a selective sampling strategy to coarsen the graph. First, FPS sampling is performed on the graph nodes of the current level, and each time the node with the largest distance between views is selected. Although FPS sampling can maintain the diversity of views, it cannot guarantee the discriminability of the learning task after subsampling. In order to solve this shortcoming of FPS, we adopt the following method: Figure 6The KNN and view selector shown in the figure. After FPS sampling of the current level of graphics, iteratively selects sampling points for KNN operation, and sends the points selected by KNN and the sampling points to the view selector. The implementation of the view selector is shown in the following formula: where Vs(·) represents a learnable view selector. This view selector outputs the probability that a view belongs to Nc classes. We implement Vs(·) using a 2-layer MLP, where V fk Represents the set of nodes selected by FPS and KNN. The Max operation returns the maximum value in the Nc class, and argmax returns the number of maximum values of all nodes in KNN, that is, the number of nodes selected by the view selector. The nodes selected by the view selector are used as the graph nodes of the next layer, and the node features updated by the local graph transformer are used as the input features of the graph nodes of the next layer.
[0154] Step S11: In the test phase, for the input 2D and 3D multi-views, we first use traditional CNN to extract 2D and 3D multi-view features respectively, and then fuse the features and construct the graph. Finally, we update the graph node features through the GTN we proposed to obtain distinguishable global feature descriptors, and then perform probability mapping on the global feature descriptors, and select the appropriate classification according to the obtained probability.
[0155] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.
[0156] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.
Claims
1. A method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer, characterized in that: include: Capture multiple views and complete the rendering of 3D and 2D multi-views of 3D objects; use the convolutional neural network CNN to extract the features of the rendered 3D and 2D multi-views, and fuse the extracted features to complete the construction of the graph; send the graph to the graph convolution-transformer network to obtain the global descriptor, and complete the recognition of the 3D object based on the descriptor.
2. The method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer as claimed in claim 1, characterized in that: The graph convolution-transformer network is a hierarchical network based on optimal view embedding, local graph-transformer, non-local graph labeling and selective view sampling, including: A confidence-based best view embedding method is used to select the best 3D shape view and embed the features of the best view into other views; Perform non-local graph tag embedding and feed all nodes into the local graph-transformer encoder to update node features; The embedded tokens are extracted from the embedding, the non-local graph tokens are recorded as the multi-view aggregation features of the current layer of the graph G, and the non-local graph tokens are concatenated with the multi-view aggregation features of other layers to obtain the global features of the 3D shape.
3. The method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer as claimed in claim 1, characterized in that: The method steps are: Step S1: Prepare the 2D multi-views to be trained, and render 20 2D multi-views for each 3D shape; select the traditional CNN network as the feature extractor of the 2D multi-view, and the extracted 2D multi-view features are recorded as I i ; Step S2: Prepare the 3D multi-views to be trained. The 3D multi-views refer to the points projected onto the 2D plane from different perspectives. Use the farthest point sampling FPS to sample 2048 3D shape points from each angle, and sample 20 angles for each 3D shape. After generating the 3D multi-view dataset, select Pointnet++ as the feature extractor for the point cloud multi-view. Perform T-Net affine transformation on the input unordered point cloud data, and the extracted features are recorded as P i ; Step S3: Build the graph and mark the i-th node of graph G as V i , V i The feature carried is denoted as F i , F i represents the fusion feature of the i-th view, and the virtual camera coordinate is defined as C i ; V i and F i The definition of V is subject to the following equation: i =(C i ,F i ), It is a tensor concatenation operation that concatenates two-dimensional and three-dimensional features to enable the network to learn more graphic node information about 3D shapes; Step S4: Use V i Construct the graph G. Choose to construct the graph G by constructing the adjacency matrix S. Define S as follows: V ij =ψ(G ij θ s ),G ij =[V i ,V j ,V i -V j ,||V i -V j ||2], G ij represents the spatial relationship between two views, [] represents the process of connecting elements into vector form; ψ represents the process of connecting elements with parameters θ s For nonlinear embedding of features, ψ is configured as a four-layer multilayer perceptron with LeakyReLU activation; Step S5: define a neural network for training graph G, namely graph transformer network GTN, which is a hierarchical network based on best view embedding, local graph-transformer, non-local graph labeling and selective view sampling; use the confidence-based best view embedding method to select the best 3D shape view, and embed the features of the best view into other views, and record the best view as B v ; Step S6: After obtaining B v (where 1≤B v ≤N), B v Embed into other views. The embedding operation is as follows: B v ≠i。 Step S7: Perform non-local graph tag embedding. First, perform a tag embedding operation on the graph of the current layer, and the tag is recorded as a token. Then define a learnable tensor as a token. The token has the same shape as the graph node and its initial value is 0. Embed the token in each layer of the graph G. The operation is as follows: G l The number of nodes increases from N to N+; Step S8: Feed all nodes into the local graph-transformer encoder to update node features; After the node is sent in, the node is first pos_embeddning operated. The pos_embeddning embeds the node coordinates into the node features, and the nodes with coordinate information are sent to the network for learning. The node feature update operation in the local graph transformer encoder is shown as follows: i =φ(S i F i C i W;θ e ), where φ represents a nonlinear transformation, W is a learnable weight matrix, and φ is defined as a sequence of Batch Normalization and LeakyReLU activations, with parameters θ e Control; The updated nodes are the same as VIT and need to go through multiple layers of Norm, Multi-Head Attention and MLP operations; Step S9: After the local graph-transformer updates the node features, it extracts the embedded token from the embedding, records the non-local graph token as the multi-view aggregation feature of the current layer of the graph G, and concatenates the non-local graph token with the multi-view aggregation features of other layers to obtain the global features of the three-dimensional shape; Define the global feature as F g , the L-layer non-local graph token is recorded as The specific implementation of G is as follows: The non-local graph token is a learnable tensor with an initial value of 0; Step S10: adopt a selective sampling strategy to coarsen the graph; Step S11: In the test phase, for the input two-dimensional and three-dimensional multi-views, first use the traditional CNN to extract the two-dimensional and three-dimensional multi-view features respectively, and then fuse the features and construct the graph; finally, update the graph node features through GTN to obtain distinguishable global feature descriptors, and then perform probability mapping on the global feature descriptors, and select the appropriate classification according to the obtained probability.
4. The method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer as claimed in claim 3, characterized in that: The complete definition of the best view obtained in step S5 is as follows: Ω is a CNN, and Ω.fc represents the fully connected layer of CNN.
5. The method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer as claimed in claim 3, characterized in that: The step S10 comprises: First, perform FPS sampling on the graph nodes of the current level, and select the node with the largest distance between views each time; use KNN and view selector. After performing FPS sampling on the graph of the current level, iteratively select sampling points for KNN operation, and send the points selected by KNN and the sampling points to the view selector; the implementation of the view selector is shown in the following formula where Vs(·) represents a learnable view selector; Vs(·) is implemented using a 2-layer MLP, where V fk Represents the node set selected by FPS and KNN. The Max operation returns the maximum value in the Nc class, and argmax returns the number of maximum values of all nodes in KNN, that is, the number of nodes selected by the view selector. The nodes selected by the view selector are used as the graph nodes of the next layer, and the node features updated by the local graph transformer are used as the input features of the graph nodes of the next layer.
6. A 3D shape recognition system based on graph convolution and transformer multimodal neural network according to the method of claim 1, characterized in that: include: Data acquisition module: used to capture 3D and 2D multi-views of 3D objects and render them. Feature extraction module: Use convolutional neural network (CNN) to extract features of rendered 3D and 2D multi-views. Graph construction module: fuses the extracted features and builds the graph structure. Graph convolution-Transformer network module: The graph structure is sent into the graph convolution-Transformer network to obtain the global descriptor. Recognition module: recognizes three-dimensional objects based on global descriptors.
7. The 3D shape recognition system based on graph convolution and transformer multimodal neural network as claimed in claim 6, characterized in that: The graph convolution-Transformer network module includes: Best View Embedding Unit: Selects the best 3D shape view based on confidence and embeds its features into other views. Local Graph-Transformer Encoder: Non-local graph token embedding is performed and all nodes are fed into the local graph-Transformer encoder to update node features. Non-local graph tag embedding unit: extracts the embedded token from the embedding, uses it as the multi-view aggregate feature of the current layer of the graph, and concatenates it with the multi-view aggregate features of other layers to obtain the global features of the 3D shape.
8. The 3D shape recognition system based on graph convolution and transformer multimodal neural network as claimed in claim 6, characterized in that: The identification module comprises: Feature fusion unit: extracts 2D and 3D multi-view features respectively and fuses them. Graph construction unit: constructs the graph, splices the features, and forms graph node information. Node update unit: Update graph node features through GTN to obtain distinguishable global feature descriptors. Classification unit: Probability mapping of global feature descriptors and selection of appropriate classification based on the obtained probability.
9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer as claimed in claim 1.
10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the method for 3D shape recognition based on a multimodal neural network of graph convolution and transformer as claimed in claim 1.
Citation Information
Cited By
Novel CAD model feature recognition method based on graph and point cloud neural network
CN119623271A
A new CAD model feature recognition method based on graph and point cloud neural network
CN119623271B
