A three-dimensional model retrieval method based on multi-view aggregation

Abstract

The application provides a three-dimensional model retrieval method based on multi-view aggregation, comprising the following steps: step 1: designing a view-based three-dimensional model retrieval network, which can extract the features of a three-dimensional model from the view representation of the three-dimensional model for three-dimensional model retrieval; the three-dimensional model retrieval network MVAT comprises a single-view feature extraction network and a multi-view feature aggregation network, the single-view feature extraction network is responsible for independently extracting the local features of the three-dimensional model from each view, and the multi-view feature aggregation network is responsible for generating a global feature representation through information fusion of multiple views; step 2: introducing a weight freezing and contrast learning technology, and adopting a three-stage training method to train the three-dimensional model retrieval network; step 3: performing performance evaluation on the trained three-dimensional model retrieval network on a test set. The three-dimensional model retrieval method has the advantages of strong feature expression capability, efficient training method, high precision, and good generalization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional model retrieval, and in particular to a three-dimensional model retrieval method based on multi-view aggregation. BACKGROUND

[0002] Existing three-dimensional model retrieval methods mainly fall into two categories: traditional three-dimensional model retrieval methods and deep learning-based three-dimensional model retrieval methods. Traditional three-dimensional model retrieval methods mainly rely on hand-designed feature descriptors, which have limited expressive power and often fail to adequately reflect the high-level semantic information of three-dimensional models. Deep learning-based three-dimensional model retrieval methods can automatically learn high-level features with richer semantic information, thereby improving the accuracy and robustness of retrieval. Deep learning-based methods can be further divided into four subcategories according to the data representation of input three-dimensional models: view-based three-dimensional model retrieval methods, point cloud-based three-dimensional model retrieval methods, voxel-based three-dimensional model retrieval methods, and multi-modal three-dimensional model retrieval methods.

[0003] Traditional three-dimensional model retrieval methods mainly rely on hand-designed feature descriptors, which capture the geometric or visual features of the model to support effective retrieval. Patent CN119066231A discloses an industrial-level three-dimensional model fast retrieval method based on binary decomposition, which generates three-dimensional model feature descriptors using recursive segmentation and surface area functions. Patent CN118350072A discloses a vehicle component three-dimensional model retrieval method, device, equipment, medium, and product, which generates a frequency spectrum graph through a two-dimensional projection sequence to describe the features of a three-dimensional model. CN117852152A discloses a building model retrieval method, system, device, and storage medium based on three-dimensional spatial feature matching, which uses voxel color average values as three-dimensional model feature descriptors to achieve three-dimensional model retrieval.

[0004] The core idea of view-based three-dimensional model retrieval methods is to arrange multiple virtual cameras around the three-dimensional model, render views from multiple different angles, extract features from each view, and aggregate these view features into a global feature representation for the three-dimensional model retrieval task. Patent CN119089000A discloses a multi-view three-dimensional model retrieval method and system based on a dual-parameter fusion network, which obtains high-expression multi-view features through feature fusion of light multi-view, depth multi-view, and thickness multi-view. Patent CN117743616A discloses a three-dimensional model retrieval method, system, device, and storage medium based on images, which uses image domain networks and model domain networks to fuse the features of multiple views for three-dimensional model retrieval. Figure Three

[0005] ​Point cloud-based three-dimensional model retrieval method is a mature technology, its general paradigm mainly includes two steps: first, according to the three-dimensional space coordinate information of each point in the point cloud, the feature is extracted, that is, each component point of the point cloud is coded to capture its spatial geometric structure; Then, these point-level features are effectively aggregated to form a global feature descriptor, thereby providing the basis for three-dimensional model retrieval. Patent CN117033686A discloses a mechanical part three-dimensional model retrieval method based on feature extraction, which uses PointNet to extract features from the point cloud representation of the three-dimensional model.

[0006] The voxel-based three-dimensional model retrieval method extracts features from the voxel representation of the three-dimensional model. Patent CN113052298A discloses a three-dimensional model retrieval method based on capsule network, which converts the three-dimensional model into discrete voxels, uses convolutional neural network and capsule network to extract the features of voxel grid, and uses dynamic routing algorithm to optimize weight iteration, realizing efficient three-dimensional model retrieval.

[0007] Multi-modal three-dimensional model retrieval method shows excellent performance by fusing three-dimensional model features from different data representations, promotes the in-depth understanding of high-level semantic features of three-dimensional models, and thus improves the effect of three-dimensional model retrieval. By utilizing the complementary characteristics of multi-modal, better results than single-modal methods can be achieved in collaborative processing and information fusion. CN112270762A discloses a three-dimensional model retrieval method based on multi-modal fusion, which uses PointNet to extract point cloud features, uses MVCNN to extract multi-view and panoramic view features, and fuses them into a multi-modal feature for retrieval. Patent CN110163091A discloses a three-dimensional model retrieval method based on LSTM network multi-modal information fusion, which uses two layers of LSTM to fuse skeleton feature domain view features, and constructs multi-modal features of three-dimensional models.

[0008] Problems and defects of the prior art:

[0009] Traditional three-dimensional model retrieval methods mainly rely on manually designed feature descriptors, which have limited expressive ability and cannot fully reflect the high-level semantic information of three-dimensional models. In addition, traditional methods have poor robustness when facing data noise, rotation, scale changes and other problems, and are easily disturbed by these factors, leading to a decrease in retrieval accuracy. For new three-dimensional models in different scenarios or different retrieval needs, traditional methods lack flexibility and often need to redesign features or methods, which undoubtedly increases the workload and difficulty of retrieval. For example, patents CN119066231A, CN118350072A, CN117852152A, CN107807970A, CN107066547A, CN106599053A, etc. all use manually designed feature descriptors, which are difficult to adapt to complex geometric structures and diverse application scenarios.

[0010] In contrast, three-dimensional model retrieval methods based on deep learning can automatically learn and extract high-level features with more semantic information, thereby improving the accuracy and robustness of retrieval. Through training on large-scale datasets, deep learning models can better cope with noise, transformation and other challenges, and exhibit good generalization ability between different datasets. However, existing deep learning-based retrieval methods still have deficiencies in feature expression ability, making it difficult to extract feature representations with high discriminability. For example, the feature extraction networks used in patents CN119089000A, CN117743616A, CN117033686A, CN113052298A, CN112270762A, CN110163091A have significantly fallen behind the current technical level, and their feature extraction capabilities are not superior in today's application scenarios.

[0011] The retrieval performance of deep learning-based methods is closely related to their data representation, among which view-based three-dimensional model retrieval methods have become a relatively mainstream algorithm due to their high frequency of use, excellent comprehensive performance, and wide range of application scenarios. However, existing view-based three-dimensional model retrieval methods still have certain defects. Patent CN119089000A discloses a multi-view three-dimensional model retrieval method and system based on a dual parameter fusion network, which extracts features from light multi-view, depth multi-view, and thickness multi-view and calculates similarity. Figure Three This method has the problem of view information redundancy, as the depth and thickness information of three-dimensional models are already contained in the light multi-view, using all three as inputs not only doubles the computational burden of the network, but also cannot effectively improve the expressive ability of the features. Furthermore, when obtaining multi-views, this patent uses an axisymmetric method to set virtual cameras, which significantly reduces the retrieval performance when the pose of the three-dimensional model changes, lacking sufficient rotational robustness. Patent CN111382300A discloses a multi-view three-dimensional model retrieval method based on group pair deep feature learning, which uses a group pair network to learn deep features from multi-views. However, this method still has the problem of feature redundancy, as the features extracted from different views are not sufficiently distinct, which reduces the discriminability of the features and the retrieval performance of the method.Figure Three The view-based three-dimensional model retrieval method and system adopt a maximum pooling method to aggregate the features of multiple views. This pooling method completely discards non-maximal feature components, resulting in severe loss of view information and failing to fully utilize the associated information between multiple views. In addition, the two patents respectively select obsolete FCN and CNN to extract the features of single views, and their feature extraction capabilities have obvious technical generation gap compared with the current advanced ViT. SUMMARY

[0012] In order to overcome the deficiencies of the existing view-based three-dimensional model retrieval method in multi-view acquisition, single-view feature extraction and multi-view feature aggregation, the present application proposes a three-dimensional model retrieval method based on multi-view aggregation, aiming to realize highly symmetric multi-view acquisition, efficient single-view feature extraction, and fully utilize the associated information between views for multi-view feature aggregation.

[0013] The present application provides a three-dimensional model retrieval method based on multi-view aggregation, comprising the following steps:

[0014] Step one: design a view-based three-dimensional model retrieval network, which can extract the features of three-dimensional models from the view representation of three-dimensional models for three-dimensional model retrieval; the three-dimensional model retrieval network MVAT includes a single-view feature extraction network and a multi-view feature aggregation network, the single-view feature extraction network is responsible for independently extracting the local features of three-dimensional models from each view and inputting the multi-view feature aggregation network, and the multi-view feature aggregation network is responsible for receiving the view information sent by the single-view feature extraction network and generating a global feature representation through the information fusion of multiple views;

[0015] Step two: introduce weight freezing and contrast learning technology, and use a three-stage training method to train the three-dimensional model retrieval network of step one;

[0016] Step three: performance evaluation of the three-dimensional model retrieval network trained in step two on the test set.

[0017] As a further improvement of the present application, in step one, the multi-view feature aggregation network includes a multi-view attention module, a channel attention aggregation module and a residual channel attention module connected in sequence, the multi-view attention module is used to focus on the related information between different views through the self-attention mechanism, enhance the perception of important features by the network, the channel attention aggregation module is used to further improve the effect of feature fusion by weighting the features of different channels, and the residual channel attention module is used to introduce residual connection to solve the problem of channel information loss and improve the expression ability of features.

[0018] As a further improvement of the present application, in the step one, the visual Transformer is selected as the single-view feature extraction network, and a weight sharing mechanism is introduced to cope with the view rotation change, each view is extracted by a group of visual Transformer with shared weights, through the self-attention mechanism of the visual Transformer, the single-view feature extraction network can not only flexibly capture the local detail information in the image, but also pay attention to the overall structure and semantic association of the image, so as to extract high-level features with global expression ability.

[0019] As a further improvement of the present application, the input and output of the multi-view attention module are both feature tensors with the shape of N x V x C, which are spliced by a group of view feature sets with the batch size N, the view number V and the feature channel number C;

[0020] The multi-view attention module comprises a global attention module, a surface attention module and a vertex attention module connected in sequence, the global attention module, the surface attention module and the vertex attention module capture the correlation information between features from the global range, the local surface and the vertex level respectively, the global attention module, the surface attention module and the vertex attention module all adopt the architecture based on the Transformer encoder, and through the hierarchical focusing manner, the multi-view attention module obtains comprehensive and detailed feature expression.

[0021] As a further improvement of the present application, the global attention module introduces a standard Transformer encoder to process the input view feature set, in this process, the self-attention mechanism is used to calculate the correlation between the view features, and no mask is set to ensure that the features of all views are treated equally, so as to capture the complete global dependency relationship, and in this way, the features of a single view and the feature information of other views are fused through adaptive weights, so as to generate a new feature tensor set;

[0022] When the surface attention module extracts the correlation information between the view features, only the self-attention of the co-surface view features in the same group is calculated to capture the deep correlation between the view features in the group, and the self-attention calculation between different groups is omitted, which reduces the calculation complexity and avoids introducing irrelevant features;

[0023] When the vertex attention module calculates the attention, only the self-attention calculation of the co-point view in the group is performed, and the cross-group self-attention calculation is omitted.

[0024] As a further improvement of the present application, the channel attention aggregation module aggregates the features in the view dimension V to generate global features with the shape of N x C, and the specific steps are as follows:

[0025] Step S1: Using max pooling and average pooling, the input features are aggregated along the view dimension V to obtain two pooled features of shape N×C. Max pooling is used to capture significant values ​​in the features, while average pooling is used to provide global statistical information.

[0026] Step S2: The max pooling and average pooling features obtained in step S1 are respectively input into the MLP layer with shared weights for processing. After processing by the MLP layer, the features obtained by different pooling methods are added together to perform information fusion. The MLP optimizes the pooling features through compression and decompression processes.

[0027] Step S3: Combine the activation function to perform a nonlinear transformation on the fused features of step S2, further enhancing the effective components in the features and suppressing irrelevant components. At the same time, normalize the value range of the feature tensor and adjust the output to a range that is easy to measure similarity. Finally, a feature tensor with shape N×C is obtained for similarity calculation in subsequent tasks.

[0028] As a further improvement of the present invention, the channel attention aggregation module introduces the tanh function as the activation function. The tanh function makes the features distributed in all subspaces with the same sign in the C-dimensional space, increasing the diversity of feature distribution and extending the range of the angles between features to [-180°, 180°], thereby improving the discriminative ability of the angle-based similarity measurement method.

[0029] As a further improvement of the present invention, the output feature z after introducing residual connection in the residual channel attention module is represented as:

[0030] z=x+y=x*(1+tanh[MLP(x)]) (3)

[0031] Where x is the input of the module and y is the output of the channel attention;

[0032] Through this design, the variation of the feature scale is effectively limited to an interval with a mean of 1 and a value range of (0, 2), thereby ensuring the statistical stability of the feature components in multi-level cascades, as shown in equation (4):

[0033] E(|z i |)≈E(|x i |) (4)

[0034] Among them, E(|z i |) and E(|x i |) represent the expectations of the scales of the z and x feature components, respectively.

[0035] As a further improvement of the present application, in the step two, the three-stage training method comprises the following steps:

[0036] Step y1, stage one: remove the multi-view feature aggregation network, connect a classification head behind the single-view feature extraction network, and train the single-view feature extraction network alone with a single-view classification task;

[0037] Step y2, stage two: remove the classification head added in the stage one, re-connect the multi-view feature aggregation network behind the single-view feature extraction network, at the same time, freeze the weights of the single-view feature extraction network, and train the multi-view feature aggregation network alone with a contrast learning method;

[0038] Step y3, stage three: unfreeze the weights of the single-view feature extraction network frozen in the stage two, and train the complete network using the same contrast learning method as in the stage two.

[0039] The beneficial effects of this invention are: 1. Strong feature representation capability—Based on the self-attention mechanism, a multi-view attention module is proposed, including three sub-modules: global attention, surface attention, and vertex attention. These modules capture the correlation information between views from different spatial levels, and adopt a progressive refinement strategy from global to local and from coarse-grained to fine-grained, which improves the descriptive ability of multi-view correlation information. Based on the channel attention mechanism, a channel attention aggregation module and a residual channel attention module are proposed. The channel attention aggregation module combines the advantages of max pooling and average pooling, effectively reducing the loss of view information in feature aggregation. At the same time, it further filters noise by compressing and expanding the feature dimension. The residual channel attention module utilizes the characteristics of residual connections to deepen the network depth through multi-level concatenation and mine the deep semantic information of the aggregated features; 2. Efficient training method—To address the limitations of the traditional two-stage training method, a three-stage training method is designed by combining weight freezing and contrastive learning techniques. In Phase 1, a single-view feature extraction network is trained independently using a single-view classification task. In Phase 2, the weights of the single-view feature extraction network are frozen, and a multi-view feature aggregation network is trained independently using contrastive learning. In Phase 3, the weights of the single-view feature extraction network are unfrozen, and the entire network is jointly trained using contrastive learning. This three-stage training method ensures that each component of MVAT is fully trained. To address the characteristics of the 3D model retrieval task, the symmetric cross-entropy loss is adaptively improved, and an absolute cosine similarity is designed. The improved symmetric cross-entropy loss ensures consistency in the loss scale across the three training stages; the absolute cosine similarity enhances the ability to distinguish between different categories of features and adjusts the feature distribution to a more stable state, improving the applicability of contrastive learning. 3. High accuracy and good generalization—MVAT's retrieval performance was evaluated on three large-scale datasets: ModelNet10, ModelNet40, and MCB-B, and visualization results are provided. Experimental results show that MVAT outperforms other mainstream 3D model retrieval methods in all metrics, demonstrating that the proposed method has high accuracy and good generalization. Attached Figure Description

[0040] Figure 1 This is the network structure of the MVAT of this invention;

[0041] Figure 2 This refers to the position and orientation of the virtual camera in this invention;

[0042] Figure 3 This is the network structure of the MVA of this invention;

[0043] Figure 4 This is the network structure of the MVA submodule of the present invention;

[0044] Figure 5is the network structure of the FSA of the present application;

[0045] Figure 6 is the network structure of the VSA of the present application;

[0046] Figure 7 is the network structure of the CAA of the present application;

[0047] Figure 8 is the network structure of the RCA of the present application;

[0048] Figure 9 is the first stage of the three-stage training method of the present application;

[0049] Figure 10 is the second and third stages of the three-stage training method of the present application;

[0050] Figure 11 is the test method of the MVAT of the present application;

[0051] Figure 12 is the top 10 retrieval results of the MVAT on the ModelNet40 dataset of the present application;

[0052] Figure 13 is the top 10 retrieval results of the MVAT on the MCB-B dataset of the present application. DETAILED DESCRIPTION

[0053] Traditional three-dimensional model retrieval methods mainly rely on hand-designed feature descriptors. However, these descriptors are limited by the designer's prior knowledge and are difficult to effectively handle complex geometric structures and diverse data representations. Unlike traditional methods, deep learning-based three-dimensional model retrieval methods break through the bottleneck of traditional methods by end-to-end feature learning and optimization, eliminating the dependence on hand-designed features, and achieving feature abstraction and expression through multi-layer network structure. Among them, view-based three-dimensional model retrieval methods have become the mainstream retrieval method due to their high application frequency, excellent comprehensive performance and wide range of application scenarios. However, existing view-based three-dimensional model retrieval methods fail to fully exploit the correlation information between view features, and have certain limitations in improving feature discriminability. To solve these problems, the present application proposes a three-dimensional model retrieval method based on multi-view aggregation, aiming to achieve more discriminative feature expression and more efficient training optimization, thereby significantly improving the retrieval performance of three-dimensional models.

[0054] The present application discloses a three-dimensional model retrieval method based on multi-view aggregation, comprising the following steps:

[0055] Step one: design a three-dimensional model retrieval network-MVAT (Multi-View Aggregation Transformer) for three-dimensional model retrieval;

[0056] Step two: train the three-dimensional model retrieval network using a three-stage training method;

[0057] Step three: evaluate the performance of the trained three-dimensional model retrieval network on the test set.

[0058] 1. Network structure of MVAT

[0059] The present application designs a three-dimensional model retrieval network based on view-MVAT, which can extract the features of three-dimensional models from the view representation of three-dimensional models for three-dimensional model retrieval, and the network structure is as shown in Figure 1 The whole MVAT can be divided into two parts: single view feature extraction network and multi-view feature aggregation network. The single view feature extraction network is responsible for independently extracting the local features of the three-dimensional model from each view, and the multi-view feature aggregation network generates global feature representation through the information fusion of multiple views to improve the retrieval accuracy. The multi-view feature extraction network can be further divided into three key modules: multi-view attention module (Multi-View Attention, MVA), channel attention aggregation module (Channel Attention Aggregation, CAA), and residual channel attention module (Residual Channel Attention, RCA). MVA focuses on the relevant information between different views through self-attention mechanism to enhance the network's perception of important features; CAA further improves the feature fusion effect by weighting different channel features; and RCA introduces residual connection to solve the problem of channel information loss, effectively improving the feature expression ability. Next, according to the above network structure division, the design details of each part and its role in the whole network will be described in detail.

[0060] 1.1 Acquisition of multi-view

[0061] The present application selects a regular dodecahedron as a virtual camera configuration framework to obtain more rich symmetric view information, and the position and pose of the virtual camera are consistent with Figure 2The regular dodecahedron has twenty vertices, each of which is adjacent to three edges. To render the view images, the three-dimensional model is enclosed in a regular dodecahedron, and three virtual cameras are arranged on each vertex in the direction of the adjacent edges. The front vector of each virtual camera points to the center of the regular dodecahedron, and the up vector is perpendicular to the front vector and coplanar with the front vector and one of the adjacent edges of the vertex. Through this configuration, sixty virtual cameras collectively render sixty view images at different viewing angles, which will serve as the input of the MVAT. In addition, in order to achieve view lightweighting while ensuring that this method utilizes the symmetry of the regular dodecahedron, not only is the balanced distribution of viewing angles guaranteed, but also various view features of the three-dimensional model can be effectively captured, enhancing the rotational robustness during retrieval. In addition, a supersampling anti-aliasing algorithm is used during the rendering process: first, rendering is performed at 16 (4x4) times the resolution, then high-frequency signals are removed using a Gaussian smoothing algorithm, and finally, the view is scaled to the target size through a downsampling algorithm. In this way, the rendered view not only has clear details, but also does not place excessive burden on the calculation of the MVAT.

[0062] 1.2 Single-view feature extraction network

[0063] Existing three-dimensional model retrieval methods often use convolutional neural networks (CNN) to extract single-view features. However, CNN has some limitations that cannot be ignored. It mainly focuses on local receptive fields and is difficult to effectively capture global information. In addition, when processing different regions of an image, CNN usually applies the same convolution operation to all positions, which cannot distinguish the importance of different image regions and limits the understanding of the global semantics of the image. To solve the above problems, the invention chooses visual Transformer as the single-view feature extraction network, which has strong representation ability and advantage in capturing global features in image processing tasks. At the same time, to cope with the ability of view rotation change, the invention introduces a weight sharing mechanism. Each view is extracted by a group of visual Transformers with shared weights. Through the self-attention mechanism of visual Transformer, the network can not only flexibly capture local detail information in the image, but also pay attention to the overall structure and semantic association of the image, thereby extracting high-level features with global expression ability.

[0064] 1.3 Multi-view attention module

[0065] After extracting the feature of each single view by the visual Transformer, a set of sixty view feature tensors is obtained. These feature tensors correspond to different viewing angles of the 3D model, and comprehensively reflect the geometric structure of the 3D model in space. However, these single view features are independent of each other and cannot directly reflect the correlation between multiple views and global information. In order to overcome the above problems, the present application proposes a multi-view attention module (Multi-View Attention, MVA), as shown in Figure 3 The input and output of the MVA are both feature tensors with the shape of N x V x C, which are concatenated by a set of view features with the batch size N, the number of views V and the number of feature channels C.

[0066] The MVA consists of three sub-modules: global attention, surface attention and vertex attention. According to the hierarchical analysis of the view distribution, these sub-modules capture the correlation information between features from the global range, the local surface and the vertex level respectively. The three sub-modules all adopt the architecture of the Transformer encoder, and their common structure is shown in Figure 4 Although the basic architectures of the three are similar, they use different masks in the self-attention calculation, and focus on capturing the correlation information at different levels of the 3D model, thereby differentiating the three different self-attention mechanisms. Through this hierarchical focusing way, the MVA can obtain more comprehensive and detailed feature expression.

[0067] 1.3.1 Global Attention

[0068] The spatial distribution of views presents a high degree of symmetry, and this symmetry makes there be more or less a certain correlation between every two views. In order to deeply mine the potential correlation information between each view and all other views, a standard Transformer encoder is introduced to process the input view feature set. In this process, the self-attention mechanism is used to calculate the correlation between the view features, and no mask is set to ensure that the features of all views are treated equally, so as to capture the complete global dependency. In this way, not only the features of a single view can be identified, but also the feature information of other views can be effectively fused through adaptive weights, so as to generate a new and more expressive feature tensor set. The MVA sub-module is named as global attention (Global Attention, GA) by the present application.

[0069] 1.3.2 Surface Attention

[0070] After extracting the correlation information between view features at the global level, the correlation of features in the local spatial structure is also crucial. With this local correlation, the information implied in the view features can be more comprehensively mined, thereby improving the quality of the overall feature expression and the retrieval performance. As described in the foregoing, the spatial distribution of virtual cameras adopts a regular dodecahedron structure, which has high symmetry and regularity in three-dimensional space. Each face of the regular dodecahedron is a regular pentagon, and each vertex of the pentagon has three virtual cameras, and there is always a special virtual camera whose forward vector and the up vector determine a plane that passes through the central normal of the regular pentagon, thereby forming a view distribution mode with geometric characteristics. In order to fully utilize this feature, the sixty views on the regular dodecahedron are divided into twelve groups, each containing five views, corresponding to a regular pentagon face of the regular dodecahedron. The forward vector of the virtual camera of each view in the group intersects with the plane determined by the up vector and the central normal of the regular pentagon face, forming a highly correlated spatial arrangement. When extracting the correlation information between view features, only the self-attention of the view features in the same group is calculated to capture the deep correlation between the views in the group, while the self-attention calculation between different groups is omitted, thereby reducing the computational complexity while avoiding the introduction of irrelevant features, as shown in Figure 5 In the figure, the regular pentagon represents a face of the regular dodecahedron, each black dot represents a vertex of the regular dodecahedron, and each arrow represents the up vector of the virtual camera. The multiplication sign represents matrix multiplication. The self-attention mechanism focusing on the regular pentagon face is referred to as face self-attention (FSA) in the present disclosure, and the MVA sub-module designed based on FSA is referred to as face attention (FA).

[0071] 1.3.3 Vertex attention

[0072] On the basis of establishing the correlation information of coplanar view features, the correlation information between view features at other levels is also worthy of in-depth research and mining. In the view generation method described in the foregoing, three virtual cameras are placed on each vertex of the regular dodecahedron. The forward vectors of these virtual cameras are consistent, but the up vectors are different. Through this setting, three views are rendered on each vertex, and the three views have certain correlation and are different due to the difference in viewing angle. Drawing on the design idea of FSA, the sixty view features on the regular dodecahedron are divided into twenty groups, each containing three views, corresponding to a vertex of the regular dodecahedron. The three views share the same vertex and naturally have high correlation. When calculating the attention, only the self-attention calculation within the group is performed, and the self-attention calculation across groups is omitted. This way not only reduces the computational complexity, but also effectively avoids the introduction of irrelevant information, as shown in Figure 6The present application refers to this self-attention mechanism based on the co-point vertex view feature as Vertex Self-Attention (VSA), and the MVA sub-module designed based on VSA is called Vertex Attention (VA).

[0073] 1.4 Channel Attention Aggregation Module

[0074] The present application migrates the channel attention mechanism to the field of three-dimensional model retrieval and makes adaptive adjustments for the specific needs of this field to be compatible with the heterogeneous features of shape N x V x C extracted by visual Transformer and MVA. With the weight calculation idea in channel attention, the features are aggregated on the view dimension V to generate a global feature of shape N x C. The present application names this module as Channel Attention Aggregation (CAA), and its structure is as shown in Figure 7 , and the specific steps are as follows.

[0075] Step S1: First, two classical feature aggregation methods, max-pooling and average-pooling, are used to aggregate the input features along the view dimension V, respectively, to obtain two pooled features of shape N x C. Max-pooling can capture significant values in the features, while average-pooling can provide global statistical information, and the combination of the two can retain information in the input features more comprehensively.

[0076] Step S2: Subsequently, the two pooled features are input into a shared weight MLP layer for processing. Unlike ordinary MLPs, the MLP in channel attention has a unique design: the channel dimensions of its input and output remain the same, while the number of channels in the hidden layer is significantly reduced. The purpose is to first compress the channel dimension, extract key features, and filter out redundant information and noise, and then restore important information by expanding the channel dimension, so as to ensure that the final output feature is both compact and has key expression ability. That is, the role of this MLP is similar to that of a filter, which efficiently optimizes the pooled features through the compression and restoration process. After the MLP layer processing, CAA adds the features obtained by different pooling methods for information fusion. In this way, CAA can integrate the complementary information provided by different pooling methods.

[0077] Step S3: Finally, the fused features are nonlinearly transformed by combining the activation function, further enhancing the effective components in the features and suppressing irrelevant components. At the same time, the value range of the feature tensor is normalized to adjust the output to a range that is easy to measure similarity. Finally, a feature tensor of shape N x C is obtained, which is used for similarity calculation in subsequent tasks.

[0078] In the above structural design, the choice of activation function plays a crucial role in the expressiveness of features and the final performance of the model. In channel attention, the Sigmoid function is usually chosen as the activation function, which maps the input value to the interval (0, 1), facilitating the weighting of feature components. However, the Sigmoid function has obvious limitations in the three-dimensional model retrieval scenario, especially for angle-based similarity measurement methods (such as cosine similarity). When using different activation functions, the distribution range of features will be significantly different. When using the Sigmoid function, since its output is only positive, the features can only be distributed in a same-sign subspace of the C-dimensional feature space, thereby limiting the range of the included angle between features, making the included angle only change between (-90°, 90°). Then, features of different categories often have difficulty in widening the distance in the angle space, thereby reducing the discrimination ability of the angle-based similarity measurement method. To solve this problem, CAA introduces the tanh function as the activation function. Similar to the Sigmoid function, tanh is also a smooth nonlinear transformation function, but its value range is expanded to (-1, 1). This feature enables tanh to effectively widen the distribution range of features. When using the tanh function, features can be distributed in all same-sign subspaces of the C-dimensional space. This not only increases the diversity of feature distribution, but also expands the range of the included angle between features to [-180°, 180°], thereby significantly improving the discrimination ability of the angle-based similarity measurement method. h

[0079] 1.5 Residual Channel Attention Module

[0080] In CAA, through the adaptive optimization of the channel attention mechanism, efficient feature aggregation is achieved, and the important role of different channels in feature expression is fully played. However, the potential of channel attention goes far beyond that. As a dynamic weight distribution mechanism, channel attention can generate channel weights using internal feature information, thereby enhancing the expressiveness of important features while suppressing redundant information. This feature makes channel attention very suitable for feature optimization tasks. By introducing the channel attention mechanism, the aggregated comprehensive features can be further mapped to an optimized feature space, making these features exhibit higher discriminability and robustness in similarity measurement tasks, thereby significantly improving the overall performance of three-dimensional model retrieval.

[0081] First, apply channel attention to the output features of CAA. Let the output features of CAA (i.e., the input features of this module) be x, and the output features after applying channel attention be y, then y is defined as follows:

[0082] y = x * tanh[MLP(x)] (1)

[0083] ​where * denotes element-wise multiplication. It is worth noting that neither max-pooling nor average-pooling operation appears when the channel attention (CA) is unfolded, because the feature aggregation has already been done in CAA, thus there is no need to do the pooling operation again.

[0084] The scale of feature components is a problem that needs to be focused on in 3D model retrieval, which directly affects the discriminability of features in similarity measurement. By examining the changes of the scale expectations E(|x i |) and E(|y i |) of each component of features x and y, since the output range of the tanh function is the open interval (-1, 1), the scale expectation of feature components will decrease, as shown in equation (2).

[0085] E(|y i |)<E(|x i |) (2)

[0086] The scale of features decreases layer by layer as the network deepens, resulting in the gradual gathering of features near the origin in high-dimensional space. When the distribution of features tends to concentrate, the effect of similarity measurement will be greatly weakened, which will reduce the network's ability to distinguish between different categories of 3D models, and will have a significant negative impact on the performance of the entire retrieval system. In order to effectively deal with this problem while ensuring that the depth of the network is sufficient to extract complex semantic features, the proposed residual connection mechanism in ResNet is worth learning. Residual connection introduces a shortcut that directly connects the input and output, allowing the input features to be directly preserved and superimposed on the output features. This design not only alleviates the problem of gradient vanishing, but also effectively avoids the excessive contraction of feature scale, so that the discriminability of features can still be maintained in a deeper network. The output feature z after introducing the residual connection can be represented as:

[0087] z = x + y = x * (1 + tanh[MLP(x)]) (3)

[0088] where x is the input of the module, and y is the output of the channel attention;

[0089] Through this design, the change of feature scale is effectively limited to the interval with a mean of 1 and a value range of (0, 2), thereby ensuring the statistical stability of feature components in multi-level series, as shown in equation (4).

[0090] E(|z i |)≈E(|x i |) (4)

[0091] where E(|z i |) and E(|x i |) represent the expectations of the scales of z and x feature components, respectively.

[0092] Benefiting from this scale stability design, the introduction of multi-stage cascaded structure can further increase the depth of the network, so as to capture more complex and abstract features. This deep feature representation is particularly crucial for similarity measurement, which can more accurately depict the subtle differences between input data, thereby significantly improving the performance of the network in the retrieval task. The network module formed by cascading the above structure through multiple stages is referred to as a residual channel attention module (RCA) in the present application, and the single constituent unit thereof is referred to as an RCA block. The structure of RCA and its constituent unit is shown in Figure 8 .

[0093] 2 Training and testing method of MVAT

[0094] 2.1 Three-stage training method

[0095] The traditional two-stage training method is as follows:

[0096] (1) Stage one: remove the multi-view feature aggregation network, connect a classification head behind the single-view feature extraction network, and train the single-view feature extraction network alone with a single-view classification task.

[0097] (2) Stage two: remove the classification head added in stage one, re-connect the multi-view feature aggregation network behind the single-view feature extraction network, and train the complete network with a multi-view classification task.

[0098] The above two-stage training method has two significant defects. Firstly, in stage two, the parameters of the multi-view feature aggregation network are randomly initialized. This disordered initialization method may lead to the generation of low-quality or even incorrect feature representations when the network aggregates multi-view features. These undesirable features can mislead the overall update direction of the network, further negatively affecting the weights of the single-view feature extraction network which has been finely trained in the first stage, thereby reducing the overall efficiency and effectiveness of feature extraction. Secondly, in stage two, the overall network is trained through a classification task. Although the classification task can extract features with strong expression ability, these features are often more suitable for classification requirements and may not meet the specific requirements of the retrieval task. The retrieval task pays more attention to the intra-class consistency and inter-class discriminativeness of features, that is, the similarity of features of the same class is significantly higher than that between different class samples. This property is often difficult to fully guarantee in the classification task.

[0099] In order to overcome the inherent defects of the traditional two-stage training method, the present application introduces the weight freezing and contrast learning technology, and proposes a three-stage training method, the specific steps of which are as follows:

[0100] Step y1, phase one: remove the multi-view feature aggregation network, connect a classification head behind the single-view feature extraction network, and train the single-view feature extraction network alone with a single-view classification task, as shown in Figure 9 .

[0101] Step y2, phase two: remove the classification head added in phase one, re-connect the multi-view feature aggregation network behind the single-view feature extraction network, and at the same time, freeze the weights of the single-view feature extraction network, train the multi-view feature aggregation network alone with a contrast learning method, as shown in Figure 10 .

[0102] Step y3, phase three: unfreeze the weights of the single-view feature extraction network frozen in phase two, and train the complete network using the same contrast learning method as in phase two, as shown in Figure 10 .

[0103] 2.1.1 Contrast learning

[0104] The contrast learning method proposed in the present application is shown in Figure 10 , and the specific steps are as follows:

[0105] Step 1: In each training, first extract a batch of data of a fixed size from the training set, and ensure that the number of samples in the batch is sufficient to maintain the diversity and statistical significance of sample pairs.

[0106] Step 2: Then, use MVAT to extract the feature vector of each sample. After extracting the features, traverse all possible pairs of sample features (F i , F j ) that can be formed in the batch, calculate their similarity s ij , and form a similarity matrix S. At the same time, generate a real value matrix GT of the same shape as S to label whether each pair of samples belongs to the same class.

[0107] Step 3: Finally, calculate the loss function by comparing the difference between S and GT, and update the network weights using the optimizer, thus completing one training iteration. Wherein, GT is defined as: the matrix elements of the same class sample pairs are 1, and the matrix elements of different class sample pairs are 0, as shown in equation (5), where C i and C j represent the classes of the i-th and j-th samples in the batch, respectively.

[0108]

[0109] 2.1.2 Symmetric cross-entropy loss

[0110] Loss function is one of the key factors to determine the optimization of neural network weights, directly affecting the learning efficiency and final performance of the model. In the first stage of the three-stage training method, the single-view feature extraction network is trained separately using the classification task, so the cross-entropy loss L ce is used as the loss function. Cross-entropy loss guides the network to learn more accurate feature representation by measuring the difference between the predicted distribution and the true distribution. However, in the second and third stages, the training method changes from classification task to contrastive learning. At this time, simply relying on the traditional cross-entropy loss is no longer applicable. In order to better handle the contrastive learning task, the present invention uses the symmetric cross-entropy loss as the loss function. By calculating the cross-entropy loss for the rows and columns of the similarity matrix respectively, and taking the average of the two as the final loss value, the bidirectional matching between image features and text features is considered in the optimization process. The symmetric cross-entropy loss L sce is defined as follows:

[0111]

[0112] where L ce (GT, S) represents the cross-entropy loss calculated along the row direction of the matrix, and L ce (GT T , S T ) represents the cross-entropy loss calculated along the column direction of the matrix. Since the three-dimensional model retrieval task belongs to the same modality retrieval task, its true value matrix GT and similarity matrix S are both symmetric matrices. Therefore, the cross-entropy loss calculated along the row and column must be equal, and can be combined into a unified loss term.

[0113] 2.1.3 Absolute cosine similarity

[0114] Cosine similarity has certain instability in the training process, mainly reflected in the limitation of feature geometric distribution in the optimization process. Specifically, for cosine similarity, when the similarity between two features is high, its value is close to 1, indicating that the angle between the features is close to 0°, i.e. the features tend to be co-linear in the same direction; on the contrary, when the similarity is low, its value is close to -1, indicating that the angle between the features is close to ±180°, i.e. the features tend to be co-linear in the opposite direction. Therefore, after the weight optimization is completed, features of the same class will tend to be co-linear in the same direction, while features of different classes will tend to be co-linear in the opposite direction. However, when the number of feature categories is greater than 2, this optimization result cannot be achieved geometrically. In other words, for multi-class feature distribution, the features cannot simultaneously satisfy the requirement of multiple pairs of opposite co-linearity. Due to the geometric conflict that cannot be completely solved, the features can only reach a dynamic balance state in the end. This dynamic balance, although it can complete the task to some extent, may reduce the convergence efficiency and performance of the model, and at the same time limit the applicability of cosine similarity in multi-class scenarios.

[0115] To solve the inherent defects of traditional cosine similarity in specific scenarios, the application proposes an improved similarity measurement function, absolute cosine similarity, which is defined as shown in equation (7). Absolute cosine similarity effectively solves the geometric constraint problem that occurs in the optimization process by taking the absolute value of the cosine similarity. Specifically, under the measurement of absolute cosine similarity, when the similarity between two features is high, the similarity value is close to 1, and the corresponding feature vector angle is close to 0° or ±180°, that is, the two feature vectors tend to be collinear; when the similarity between two features is low, the similarity value is close to 0, and the angle between the feature vectors is close to ±90°, that is, the feature vectors are perpendicular to each other. This geometric relationship has a clear geometric meaning in high-dimensional feature space, and as long as the feature dimension is greater than or equal to the number of categories, it can be achieved through optimization. Since the same feature vectors tend to be collinear and different feature vectors tend to be perpendicular to each other, a stable static balance is finally formed between the features, thereby improving the performance in the retrieval task.

[0116]

[0117] 2.2 Test method

[0118] After completing the training of the three-dimensional model retrieval network, the next step is to evaluate the performance of the trained network on the test set. The three-dimensional model retrieval task is divided into two stages: preprocessing stage and retrieval stage. When testing the three-dimensional model retrieval network, the same operation needs to be performed according to the two stages, and the process is as shown in Figure 11 . The specific steps are as follows:

[0119] Step 1, preprocessing stage: Before testing begins, the MVAT that has been trained is first used to extract features from the multiple views of each three-dimensional model in the test set, and all extracted features are stored in a three-dimensional model database, thereby improving retrieval efficiency and avoiding repeated calculation of features during each retrieval.

[0120] Step 2, retrieval stage: In each test, a three-dimensional model is selected from the test set as a query. Next, using the three-dimensional model database generated in the preprocessing stage, the absolute cosine similarity between the features of the query and all three-dimensional models in the database is calculated. According to the size of the similarity, all three-dimensional models in the test set are sorted in descending order, and the sorted three-dimensional model sequence is the retrieval result of the query.

[0121] 3 Experimental results and analysis

[0122] 3.1 Dataset and evaluation index

[0123] 3.1.1 ModelNet data set and its evaluation index

[0124] ModelNet data set contains 127915 high-quality three-dimensional models distributed in 662 categories, widely used in computer vision, computer graphics, robotics, cognitive science and other fields, especially in the related research of three-dimensional model analysis and processing. Its comprehensiveness and extensive coverage of categories make it one of the important benchmark data sets in three-dimensional model retrieval field. However, due to the large size of the ModelNet data set, directly using the full set for experiment has significant challenges in computing resources and time cost. Therefore, in order to more conveniently evaluate the performance of three-dimensional model retrieval network, researchers have constructed two commonly used subsets from ModelNet data set: ModelNet10 and ModelNet40. ModelNet10 data set contains 4899 three-dimensional models distributed in 10 categories, of which 3991 are used for training and 908 are used for testing. These categories are all common object categories, with strong representativeness. ModelNet40 data set is further expanded to 40 categories, containing a total of 12311 three-dimensional models, of which 9843 are used for training and 2468 are used for testing. This data set covers more diverse object categories, suitable for more complex and extensive three-dimensional model research tasks. In the evaluation system of ModelNet, the core index for evaluating retrieval performance is mAP.

[0125] 3.1.2 MCB-B data set and its evaluation index

[0126] MCB dataset is an important dataset in three-dimensional vision tasks in recent years. The dataset is divided into two subsets: MCB-A and MCB-B. Among them, MCB-B dataset has become the preferred test benchmark for many three-dimensional model retrieval networks due to its moderate data volume, diverse rotation poses, and rich evaluation index system, which is used to comprehensively evaluate the performance of three-dimensional model retrieval. MCB-B dataset is based on mechanical parts, covering 25 types of mechanical parts, a total of 18038 three-dimensional models, of which 14451 are used for training and 3587 are used for testing, providing a good verification environment for the generalization ability of the network under different conditions. In its evaluation system, the core retrieval performance indicators include: F1-score, mAP, NDCG. According to whether the average value is calculated based on the data size of each category, MCB-B dataset further divides each evaluation indicator into micro and macro versions. Micro average treats each query and its retrieval results equally, and calculates the average value directly on the entire dataset; while macro average first calculates the average value within each category, and then takes the inter-class average value of all category intra-class average values.

[0127] 3.2 Experimental details and hyperparameter settings

[0128] An advanced visual Transformer, MaxViT, is selected as the single-view feature extraction network of MVAT in the experiment. In order to ensure that MaxViT can exert strong feature extraction ability in the experiment, and at the same time avoid the problem of difficult convergence when visual Transformer is trained from scratch, ImageNet-1K dataset is used to pre-train MaxViT in the experiment. In the experiment, the pixel size of each view is set to 224x224, and the background color is uniformed as black to reduce the interference of background noise on feature extraction. At the same time, in order to reduce information redundancy on the basis of ensuring feature extraction ability, the feature dimension size output by the single-view feature extraction network is set to 512. This dimension not only serves as the input feature dimension of the multi-view feature aggregation network, but also remains unchanged after RCA, CAA and RCA, and finally becomes the feature dimension of the network output. In RCA, the number of series of RCA blocks is an important parameter that affects the performance of the network, which is set to 16 in this experiment.

[0129] In the training process, Adam, which is widely used in deep learning and has superior performance, is chosen as the optimizer. Its stability and efficiency in handling complex models have been widely verified. In the three-stage training method proposed in the present application, the training rounds of each stage are set to 50 to balance the training efficiency and training quality. After many experimental attempts, the ideal configuration of the learning rate is determined: in stage one, the learning rate is set to 10 -4 ; while in stage two and stage three, the learning rate is reduced to 10 -5 . This learning rate adjustment strategy can effectively promote the convergence of the model to obtain near-optimal experimental results.

[0130] 3.3 Performance evaluation experiment

[0131] 3.3.1 Performance evaluation experiment on ModelNet dataset

[0132] According to the experimental setup in the foregoing, MVAT is trained and tested on ModelNet10 dataset and ModelNet40 dataset respectively to comprehensively evaluate its performance in the three-dimensional model retrieval task. In order to accurately evaluate the performance of MVAT, it is compared with various three-dimensional model retrieval methods. The performance evaluation results of MVAT and these methods on ModelNet10 dataset and ModelNet40 dataset are shown in Table 1 and Table 2 respectively.

[0133] Table 1 Performance evaluation experiment on ModelNet110 data

[0134]

[0135] Table 2 Performance evaluation experiment on ModelNet140 data

[0136]

[0137] From the experimental results, it can be seen that on the ModelNet10 dataset, the mAP of MVAT reaches 95.1%, which is significantly better than other methods in performance. Compared with the previous method SPNet, the mAP is improved by 0.9%; compared with VAM-IAM which also introduces attention mechanism, the performance is improved by 1.5% mAP. On the ModelNet40 dataset, MVAT also achieves remarkable results, with mAP reaching 93.2%. Compared with the method MVTN, the mAP is improved by 0.3%; compared with VAM-IAM which also introduces attention mechanism, the mAP is improved by 0.4%. These results show that MVAT has strong generalization ability and excellent performance in the three-dimensional model retrieval task.

[0138] To further verify the actual performance of MVAT, several samples were randomly selected from the test set of the ModelNet40 dataset, and the top 10 retrieval results were visualized as shown in Figure 12 Fig. 3. In the figure, the three-dimensional model on the left is displayed as a query and its category label, and the top 10 retrieval results are displayed on the right. Among them, the three-dimensional model marked with a box represents a retrieval error sample, and the label below the box represents its true category.

[0139] From the figure, it can be observed that MVAT achieves excellent retrieval accuracy, even though there are a few incorrect retrievals, these incorrect results have certain relevance to the correct results in semantic categories. For some complex categories of three-dimensional models, MVAT can also identify high-level features in their geometric structures, thus showing high robustness and semantic perception ability. In summary, whether comparing overall performance or visualizing actual retrieval results, MVAT has shown significant advantages in the three-dimensional model retrieval task and has performed excellently on complex and large-scale datasets.

[0140] 3.3.2 Performance evaluation experiment on MCB-B dataset

[0141] According to the experimental setup in the foregoing, MVAT was comprehensively trained and tested on the MCB-B dataset to evaluate its performance in the three-dimensional model retrieval task in depth. In order to accurately evaluate the performance of MVAT, a variety of three-dimensional model retrieval methods were selected for comparison, and the experimental results are shown in Table 3.

[0142] Table 3 Performance evaluation experiment on ModelNet140 data

[0143]

[0144] From the experimental results, it can be seen that MVAT performs significantly better than other methods on the MCB-B dataset. In terms of micro metrics, the F1-score, mAP, and NDCG of MVAT reached 89.6%, 96.1%, and 96.7%, respectively, which is an increase of 22.0%, 4.8%, and 4.2% compared to previous methods. In terms of macro metrics, the F1-score, mAP, and NDCG of MVAT reached 87.6%, 94.7%, and 95.3%, respectively, which is an increase of 2.2%, 3.8%, and 8.2% compared to previous methods. Unlike the ModelNet dataset, which focuses on class distribution diversification, the MCB-B dataset is a database of three-dimensional models focused on mechanical parts, which usually have highly structured and standardized features. The outstanding performance of MVAT on the MCB-B dataset indicates its strong ability in feature extraction and retrieval of structured models. Especially the significant improvement in micro mAP, which further reflects the excellent general retrieval performance of MVAT on high-frequency and easily divided classes. This not only proves the superiority of MVAT in complex three-dimensional model retrieval tasks, but also indicates its potential application in the engineering field.

[0145] To further evaluate the actual performance of MVAT on the MCB-B dataset, a number of samples were randomly selected from the test set of the MCB-B dataset according to the same visualization method as before, and the top 10 retrieval results were displayed as shown in FIG. 6. Figure 13

[0146] The results show that the retrieval accuracy of MVAT on the MCB-B dataset is also excellent. Although there are a small number of false positives, these false positive samples still have high similarity in geometric structure with the correct results. This further verifies that MVAT can exhibit excellent three-dimensional model retrieval capability whether it faces the class-rich ModelNet dataset or the class-sparse MCB-B dataset, fully demonstrating its excellent retrieval performance.

[0147] 3.4 Ablation experiment

[0148] Detailed ablation experiments were conducted on the structure of MVAT and each part involved in the training process, aiming to verify the effectiveness of the structure and method proposed in the present application. The experimental dataset is ModelNet40, and a simple network composed of MaxViT and max pooling directly connected in series, MaxViT→MaxPool, is constructed as the baseline network for ablation experiments.

[0149] 3.4.1 Ablation experiment of channel attention aggregation module

[0150] ​Firstly, the role of the channel attention aggregation module (CAA) in the three-dimensional model retrieval network is studied. CAA is a typical feature aggregation module, and its function is similar to the max pooling operation in the benchmark network, but it is more flexible and efficient in design. In order to comprehensively evaluate the influence of different aggregation modules on the performance of three-dimensional model retrieval, the max pooling operation in the benchmark network is replaced by other aggregation modules, and the comparison of retrieval performance is observed through experiments. On the ModelNet40 dataset, the retrieval performance of CAA and two traditional aggregation methods is shown in Table 4.

[0151] Table 4 Ablation experiment of CAA on ModelNet140 data

[0152]

[0153] The experimental results show that the retrieval network using CAA as the aggregation method achieves an mAP index of 89.7%, which is 3.7% and 0.9% higher than that of the retrieval network using max pooling and average pooling, respectively. This fully reflects the significant advantage of CAA in feature aggregation. This advantage is due to the unique structure of CAA, which combines the advantages of max pooling and average pooling, not only can play the strengths of the two, but also further optimize the feature distribution by introducing MLP and tanh activation function, so as to realize more efficient feature expression and aggregation.

[0154] 3.4.2 Ablation experiment of residual channel attention

[0155] The residual channel attention module (RCA) further improves the distribution of features after CAA, and is one of the key components of MVAT. RCA is formed by a plurality of RCA blocks with the same structure connected in series, and each RCA block enhances the feature expression capability of the network through residual connection and channel attention mechanism. After replacing the max pooling layer in the benchmark network with CAA, a plurality of RCA blocks are further connected in series after it to evaluate the influence of the introduction of RCA and the number of RCA blocks connected in series on the performance of three-dimensional model retrieval. The detailed settings and experimental results of the related ablation experiment on the ModelNet40 dataset are shown in Table 5.

[0156] Table 5 Ablation experiment of RCA on ModelNet140 data

[0157]

[0158] The experimental results show that the introduction of RCA has a significant effect on the retrieval performance. The mAP is increased by 0.3% by introducing only one RCA block. When the number of RCA blocks in series reaches 16, the retrieval performance of the network is most significantly improved. Compared with the benchmark network with only CAA, the mAP is increased by 2.2%. This result shows that the multi-stage RCA block in series can continuously optimize the distribution of features and map the features to a more discriminative feature space, thereby effectively improving the discrimination ability of different categories of features, confirming the key role of the RCA module and its multi-stage series structure in the three-dimensional model retrieval task.

[0159] 3.4.3 Ablation experiment of multi-view attention module

[0160] The multi-view attention module (MVA) is a subsequent module of the single-view feature extraction network, which is used to capture the correlation information between multi-view features, so as to provide more rich and accurate input features for the subsequent feature aggregation module CAA. MVA is composed of three sub-modules, namely global attention (GA), face attention (FA) and vertex attention (VA). The design philosophy of these sub-modules is different, and they model different levels of feature relationships from global to local. In order to comprehensively evaluate the individual role and synergistic effect of each sub-module of MVA, a set of ablation experiments are designed. On the basis of the benchmark network with CAA and RCA added, different sub-module combinations and arrangement methods are introduced, and their retrieval performance on the ModelNet40 dataset is tested. The experimental results are shown in Table 6.

[0161] Table 6 Ablation experiment of MVA on ModelNet140 data

[0162]

[0163] The experimental results show that even if only one sub-module is introduced alone, the retrieval performance can be significantly improved, and the mAP is increased by more than 0.8%. Among them, the VA sub-module has the most significant improvement, and the mAP reaches 93.0% when used alone. In the two-by-two combination of sub-modules, the VA→FA arrangement performs best, with an mAP of 91.5%, which is slightly lower than the benchmark network by 0.4%. When the three sub-modules are integrated together and combined in different arrangement orders, the experiment further reveals the significant influence of order on performance. Among them, the GA→FA→VA arrangement order performs best, with an mAP of 93.2%, which is improved by 1.3% compared with the benchmark network. This result shows that the strategy of gradually obtaining feature correlation information from global to local has important value. The introduction of MVA converts the original 60 single-view features into a new feature set with more complex information expression ability, thereby significantly enhancing the representation ability and retrieval performance of the overall features.

[0164] 3.4.4 Ablation experiments of contrastive learning and symmetric cross-entropy loss

[0165] The application uses a contrastive learning method in the training process, and selects symmetric cross-entropy as the loss function. To verify the effectiveness of contrastive learning and symmetric cross-entropy loss, a plurality of ablation experiments are set on the ModelNet40 dataset, and different training techniques are used to train the MVAT with complete structure, so as to analyze the influence of the use of contrastive learning and the selection of loss function on the retrieval performance, and the experimental results are shown in Table 7.

[0166] Table 7 Ablation experiments of contrastive learning and symmetric cross-entropy loss on ModelNet140 data

[0167]

[0168] The experimental results show that, compared with the non-contrastive learning method, the use of the contrastive learning method for training can significantly improve the retrieval performance, and at least can increase the mAP by 1.1%. In addition, in the contrastive learning method, the influence of different loss functions on the performance is also different. Among them, the symmetric cross-entropy loss performs best in all experiments, and compared with the non-contrastive learning method, the contrastive learning method using the symmetric cross-entropy loss increases the mAP by 2.0%, further verifying the superiority of the symmetric cross-entropy loss in the contrastive learning.

[0169] 3.4.5 Ablation experiments of absolute cosine similarity

[0170] The selection of the similarity measurement function plays a key role in the training and testing of the three-dimensional model retrieval network. In order to verify the effectiveness of the absolute cosine similarity in the training and testing of the three-dimensional model retrieval network, on the basis of the experimental setting in the foregoing, only the similarity measurement function is replaced, and a new ablation experiment is carried out on the ModelNet40 dataset, and the experimental results are shown in Table 8.

[0171] Table 8 Ablation experiments of absolute cosine similarity on ModelNet140 data

[0172]

[0173] The experimental results show that, compared with the traditional cosine similarity, the use of absolute cosine similarity makes the mAP of MVAT in the three-dimensional model retrieval task increase by 1.2%. This result fully proves the advantages of absolute cosine similarity in the training and testing process of the three-dimensional model retrieval task. In the training stage, the absolute cosine similarity can more effectively guide the feature to optimize in a more discriminative direction, and finally form a static and stable feature distribution, thereby enhancing the model's ability to distinguish different three-dimensional model features. While in the testing stage, the absolute cosine similarity can more accurately measure the similarity between different three-dimensional models, further improving the accuracy and reliability of the retrieval results.

[0174] 3.4.6 Ablation experiment of three-stage training method

[0175] On the basis of the two-stage training method, the invention introduces weight freezing and contrast learning technology, effectively ensuring that each part of the single-view feature extraction network and the multi-view feature aggregation network can be fully trained. In order to further explore the reasons behind the excellent performance of the three-stage training method and verify the necessity of each training stage, the invention designs a new ablation experiment on the ModelNet40 dataset, and the experimental settings and results are shown in Table 9.

[0176] Table 9 Ablation experiment of three-stage training method on ModelNet140 data

[0177]

[0178] The experimental results show that the single-stage training method performs the worst, with an mAP of only 87.1%. The two-stage training method without weight freezing improves this problem to some extent, with a significant increase of 4.3% in mAP. In the two-stage training method with weight freezing, the weight freezing technology is further applied, which further improves the mAP by 1.1%. In the three-stage training method, the introduction of the weight unfreezing stage effectively improves the parameter adjustment of the network, which further improves the final mAP by 0.7%. The three-stage training method fully combines the advantages of the traditional two-stage method and solves the shortcomings of the two-stage method by introducing the weight freezing technology. This method not only speeds up the training process, but also improves the convergence performance of the network. Finally, through this method, the network achieves better overall performance, demonstrating the effectiveness and superiority of the three-stage training method in the three-dimensional model retrieval task.

[0179] The application discloses a three-dimensional model retrieval method based on multi-view aggregation, which has the advantages of realizing more discriminative feature expression and more efficient training optimization, simultaneously proposing a multi-view attention module, a channel attention aggregation module and a residual channel attention module, and a three-stage training method, and realizing high-precision retrieval performance on three data sets, and specific advantages are as follows.

[0180] 1. Strong feature expression capability - based on the self-attention mechanism, a multi-view attention module is proposed, including three sub-modules of global attention, surface attention and vertex attention. These modules capture the correlation information between views from different spatial levels, adopt a step-by-step refinement strategy from global to local and from coarse granularity to fine granularity, and improve the description capability of multi-view correlation information. Based on the channel attention mechanism, a channel attention aggregation module and a residual channel attention module are proposed. The channel attention aggregation module combines the advantages of maximum pooling and average pooling, effectively reduces the loss of view information in feature aggregation, and further filters noise through feature dimension compression and expansion. The residual channel attention module uses the characteristics of residual connection to deepen the network depth through multi-level series, and excavates the deep semantic information of the aggregated features.

[0181] 2. Efficient training method - in view of the limitations of the traditional two-stage training method, a three-stage training method is designed by combining weight freezing and contrast learning technology. In stage one, the single-view feature extraction network is independently trained using the single-view classification task; in stage two, the weights of the single-view feature extraction network are frozen, and the multi-view feature aggregation network is independently trained using contrast learning; in stage three, the weights of the single-view feature extraction network are unfrozen, and the entire network is jointly trained using contrast learning. This three-stage training method ensures the full training of the MVAT components. In view of the characteristics of the three-dimensional model retrieval task, the symmetric cross-entropy loss is adaptively improved, and the absolute cosine similarity is designed. The improved symmetric cross-entropy loss ensures the consistency of the loss scale in the three-stage training; the absolute cosine similarity enhances the discrimination ability of different class features, adjusts the feature distribution to a more stable state, and improves the applicability of contrast learning.

[0182] 3. High precision and good generalization - the retrieval performance of MVAT is evaluated on ModelNet10, ModelNet40 and MCB-B three large-scale data sets, and the visual results are given. The experimental results show that MVAT is superior to other mainstream three-dimensional model retrieval methods in various indicators, which can prove that the method proposed in the application has high precision and good generalization.

[0183] The above is further detailed description of the present application in combination with specific preferred embodiments, and cannot be deemed as limitation of the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, and all should be deemed as falling within the protection scope of the present application.

Claims

1. A method for 3D model retrieval based on multi-view aggregation, characterized in that, The method comprises the following steps: Step one: design a view-based three-dimensional model retrieval network, which can extract the features of the three-dimensional model from the view representation of the three-dimensional model for three-dimensional model retrieval; the three-dimensional model retrieval network MVAT comprises a single-view feature extraction network and a multi-view feature aggregation network, the single-view feature extraction network is responsible for independently extracting local features of the three-dimensional model from each view and inputting the multi-view feature aggregation network, and the multi-view feature aggregation network is responsible for receiving view information sent by the single-view feature extraction network and generating global feature representation through information fusion of multiple views; Step two: introduce weight freezing and contrast learning technology, and use a three-stage training method to train the three-dimensional model retrieval network of step one; Step three: evaluate the performance of the three-dimensional model retrieval network trained in step two on the test set; In step one, the multi-view feature aggregation network comprises a multi-view attention module, a channel attention aggregation module and a residual channel attention module connected in sequence, the multi-view attention module is used to focus on the relevant information between different views through a self-attention mechanism, enhance the perception of important features by the network, the channel attention aggregation module is used to further improve the effect of feature fusion by weighting the features of different channels, and the residual channel attention module is used to introduce residual connection to solve the problem of channel information loss and improve the expression ability of the features; In step one, the visual Transformer is selected as the single-view feature extraction network, and a weight sharing mechanism is introduced to cope with the view rotation change, each view is extracted by a group of visual Transformers with shared weights, through the self-attention mechanism of the visual Transformer, the single-view feature extraction network can not only flexibly capture the local detail information in the image, but also pay attention to the overall structure and semantic association of the image, so as to extract high-level features with global expression ability; In step two, the three-stage training method comprises the following steps: Step y1, phase one: remove the multi-view feature aggregation network, connect a classification head behind the single-view feature extraction network, and train the single-view feature extraction network alone with a single-view classification task; Step y2, phase two: remove the classification head added in step one, re-connect the multi-view feature aggregation network behind the single-view feature extraction network, freeze the weights of the single-view feature extraction network, and train the multi-view feature aggregation network alone with a contrast learning method; Step y3, phase three: unfreeze the weights of the single-view feature extraction network frozen in step two, and train the complete network using the same contrast learning method as in step two; In step three, when testing the three-dimensional model retrieval network, the following preprocessing stage and retrieval stage are performed, and the specific steps are as follows: Step 1, preprocessing stage: before the test starts, first use the trained three-dimensional model retrieval network to extract features from the multiple views of each three-dimensional model in the test set, and persistently store all the extracted features in a three-dimensional model database, thereby improving retrieval efficiency and avoiding repeated calculation of features at each retrieval; Step 2, retrieval stage: in each test, a three-dimensional model is selected from the test set as a query, then the absolute cosine similarity between the features of the query and all three-dimensional models in the database is calculated using the three-dimensional model database generated in the preprocessing stage of step 1, and all three-dimensional models in the test set are sorted in descending order according to the similarity, and the sorted three-dimensional model sequence is the retrieval result of the query.

2. The three-dimensional model search method according to Claim 1, wherein The input and output of the multi-view attention module are both feature tensors with a shape of N x V x C which are spliced from a set of view features with a batch size of N , a number of views of V , and a number of feature channels of C ; The multi-view attention module comprises a global attention module, a surface attention module and a vertex attention module connected in sequence, the global attention module, the surface attention module and the vertex attention module capture correlation information between features from global range, local surface and vertex level respectively, the global attention module, the surface attention module and the vertex attention module all adopt a Transformer encoder-based architecture, and through hierarchical focusing, the multi-view attention module obtains comprehensive and detailed feature expression; The global attention module introduces a standard Transformer encoder to process the input view feature set, in this process, self-attention mechanism is used to calculate the correlation between view features, and no mask is set to ensure that the features of all views are treated equally, thereby capturing complete global dependency, and in this way, the features of a single view and the feature information of other views are identified and fused through adaptive weights, thereby generating a new feature tensor set; When the surface attention module extracts correlation information between view features, it only calculates self-attention for co-surface view features in the same group to capture deep correlations between views within the group, while self-attention calculation is not performed between different groups, thereby reducing computational complexity and avoiding the introduction of irrelevant features; When the vertex attention module calculates attention, only co-point view self-attention calculation is performed within the group, while cross-group self-attention calculation is omitted.

3. The three-dimensional model search method according to Claim 1, wherein The channel attention aggregation module is in a view dimension V The features are aggregated above to generate global features in a shape of N x C The specific steps are as follows: Step S1: using the feature aggregation method of max-pooling and average-pooling, respectively, the input features are aggregated along the view dimension V aggregated, two pooled features with shapes of N x C are obtained, max-pooling is used to capture significant values in the features, and average-pooling is used to provide global statistical information; Step S2: input the maximum pooling and average pooling features obtained in step S1 into the MLP layer with shared weights for processing, and then add the features obtained through different pooling methods after the MLP layer processing to perform information fusion; the MLP optimizes the pooling features through compression and restoration processes; Step S3: a nonlinear transformation is performed on the fused features of step S2 by combining an activation function, further enhancing the effective components in the features, suppressing irrelevant components, normalizing the value range of the feature tensor, adjusting the output to a range that is easy to measure similarity, and finally obtaining a feature tensor with a shape of N x C for subsequent similarity calculation tasks; the channel attention aggregation module introduces a tanh function as the activation function, which makes the feature distribution in all same sign subspaces of the C dimensional space, increases the diversity of the feature distribution, and expands the range of the angle between the features to , thereby improving the discrimination ability of the angle-based similarity measurement method.

4. The three-dimensional model search method according to Claim 1, characterized by, In the residual channel attention module, the output feature z after introducing residual connection is represented as: (3), wherein, is an input to the module, is an output of the channel attention; Through this design, the change of feature scale is effectively limited in the interval with mean value 1 and value range (0, 2), thereby ensuring the statistical stability of feature components in multi-level series, as shown in equation (4): (4), wherein and respectively represent and the desired eigen-component scale.

5. The three-dimensional model search method according to Claim 1, wherein The contrast learning method comprises the following steps: Step 1: First, a fixed size batch data is extracted from the training set, ensuring that the number of samples in the batch is sufficient to maintain the diversity of sample pairs and statistical significance; Step 2: Extract the feature vector of each sample using the three-dimensional model retrieval network, after extracting the features, traverse all possible pairs of sample features in this batch of data , calculate the similarity between them , form a similarity matrix , at the same time, generate a true value matrix with the same shape as according to the sample category , to mark whether each pair of samples belongs to the same category; Step 3: Compute the loss function by comparing the difference between and Update the network weights with an optimizer to complete one training iteration. wherein, is defined as: 1 for the matrix element of the sample pair of the same category, 0 for the matrix element of the sample pair of different categories, as shown in equation (5), wherein and respectively represent the category of the th and j th sample in the batch. (5)。 6. The three-dimensional model search method according to Claim 1, wherein In the three-stage training method, a symmetric cross-entropy loss is used as a loss function, the cross-entropy loss is calculated for the rows and columns of the similarity matrix respectively, and the average of the two is taken as the final loss value, so that the bidirectional matching between the image features and the text features is considered in the optimization process, and the definition of the symmetric cross-entropy loss is as follows: (6), wherein, represents the cross-entropy loss calculated along the row direction of the matrix, represents the cross-entropy loss calculated along the column direction of the matrix; In the three-stage training method, the absolute cosine similarity measurement function is used to solve the geometric constraint problem of the cosine similarity in the optimization process. Specifically, under the measurement of the absolute cosine similarity, when the similarity between two features is high, the similarity value is close to 1, and the corresponding feature vector angle is close to or That is, the two feature vectors tend to be collinear, and when the similarity between two features is low, the similarity value is close to 0, and the angle between the feature vectors is close to That is, the feature vectors are perpendicular to each other.

7. The three-dimensional model search method according to Claim 1, wherein In the step one, the acquisition step of multi-view is further included, and specifically as follows: A regular dodecahedron is selected as a virtual camera configuration framework, a three-dimensional model is enclosed in a regular dodecahedron, and three virtual cameras are arranged on each vertex along the adjacent edge direction, the forward vector of each virtual camera points to the center of the regular dodecahedron, and the upward vector is perpendicular to the forward vector and coplanar with the forward vector and one of the adjacent edges of the vertex, and the multi-view images obtained through this configuration are used as the input of the three-dimensional model retrieval network; In the acquisition step of multi-view, the supersampling anti-aliasing algorithm is also used in the rendering process of view images: first, rendering is performed with 4*4 times resolution, then high-frequency signals are removed by using a Gaussian smoothing algorithm, and finally the view is scaled to the target size through a downsampling algorithm.

Images