Three-dimensional model retrieval method and system based on multi-view fusion and prototype guidance

Abstract

The application provides a three-dimensional model retrieval method and system based on multi-view fusion and prototype guidance, and relates to the field of computer vision. First, a double-path adaptive perception network DPAP-Net is used in combination with a static prior and a dynamic anchoring mechanism to identify and strengthen a "sketch-friendly view", thereby solving the feature mismatch problem caused by perspective preference. Second, a prototype guidance driven dynamic sampling strategy PGDS is used to realize adaptive adsorption of sampling points to effective stroke areas, extract key local structures in sparse sketches, and overcome noise interference of large-area blank backgrounds. Finally, an auxiliary classification loss is introduced, and a cross-modal transfer loss is applied to the sketch branch to eliminate cross-domain distribution differences, thereby improving the accuracy and robustness of retrieval.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a method and system for retrieving 3D models based on multi-view fusion and prototype guidance. Background Technology

[0002] With the rapid development of digital modeling technology, 3D model data has exploded, making 3D model retrieval a huge application potential in fields such as industrial design, digital twins and intelligent manufacturing. The core task of 3D model retrieval is to quickly and accurately locate the target geometric entity from a massive database based on the input information provided by the user. According to the difference in query methods, existing 3D model retrieval methods are mainly divided into three categories: model-based retrieval, text-based retrieval and view-based retrieval.

[0003] Model-based retrieval uses the data of the 3D model itself, such as point clouds and meshes, as query input. Although such methods can deeply explore the spatial geometric features of 3D models, the threshold for ordinary users to obtain complete 3D models is extremely high, and the storage and computational costs of model data are huge. Text-based retrieval queries through descriptive attributes. Although the interaction is simple, it is limited by the semantic ambiguity of natural language and is difficult to accurately describe complex shape details. View-based retrieval uses two-dimensional visual features for matching, achieving a better balance between interactive convenience and representational ability.

[0004] In view-based retrieval, sketch-based retrieval holds irreplaceable value in rapid prototyping and creative search scenarios due to its advantages such as convenient drawing, low interaction threshold, and direct reflection of users' core design concepts. However, despite these advantages, existing sketch retrieval methods still face severe technical challenges in practical applications. On the one hand, traditional view feature extraction methods, such as multi-view convolutional neural networks, are mainly designed for information-rich images and perform poorly in the sketch domain. Due to strong perspective preferences when users draw, some views in the 3D model contribute very little to sketch matching or even interfere. Existing view filtering mechanisms lack optimization for sketch characteristics, limiting the accuracy of feature alignment. On the other hand, hand-drawn sketches are highly sparsity, with large areas of blank space. How to accurately extract useful global features and capture key local details from extremely sparse input remains the core bottleneck for improving retrieval performance. Therefore, how to construct a multi-view adaptive fusion mechanism that can identify "sketch-friendly" views and achieve accurate extraction of sparse sketch line information is of great research significance for improving the performance of 3D model retrieval. Summary of the Invention

[0005] This invention proposes a 3D model retrieval method and system based on multi-view fusion and prototype guidance, using the dual-path adaptive perception network DPAP-Net and the prototype-guided dynamic sampling strategy PGDS method. This solves the problems of "sketch-friendly" view recognition and fusion caused by sketch perspective preference and local feature extraction in sparse sketches in view-based retrieval, thereby improving the accuracy and stability of the retrieval results.

[0006] This invention proposes a dual-path adaptive perception network DPAP-Net and a prototype-guided dynamic sampling strategy PGDS method, providing a 3D model retrieval method and system based on multi-view fusion and prototype guidance, including the following steps:

[0007] S1. Preprocess the 3D model and sketch to obtain the basic feature set of the 3D model in multiple views. and basic features of the sketch ;

[0008] S2. Construct a dual-path adaptive perception network DPAP-Net and process the multi-view basic feature set. and basic features of the sketch Perform feature filtering and fusion;

[0009] S3. Process the sketch to obtain the sketch feature map. and global features of the sketch Execute prototype-guided PG operations to generate a spatial guide map. ;

[0010] S4. Perform dynamic sampling (DS) using the spatial guidance map. With embedded feature maps The interaction enables adaptive adjustment of the feature extraction location;

[0011] S5. Introduce auxiliary classification constraints for the two modal branches respectively to achieve cross-domain feature alignment between the sketch and the 3D model;

[0012] S6. Input the sketch to be queried, calculate the cosine similarity between the features of the sketch and the features of the 3D model in the database, sort them in descending order of score, and output the search results.

[0013] Preferably, in step S1, the set of basic features of the three-dimensional model in multiple views is obtained. and basic features of the sketch Its features are:

[0014] Let the cross-modal dataset be... ,in Represents the sketch dataset, Representing a 3D model dataset, to distinguish the relevant variables of the two modalities, the superscript 's' represents the sketch and the superscript 'v' represents the 3D model. First, the 3D model is preprocessed; specifically, a bounding sphere observation space is established around the geometric center of the 3D model as the origin; and uniformly spaced variables are set on the same horizontal plane on the sphere. Using multiple virtual camera viewpoints, the 3D model is projected and rendered onto each viewpoint, resulting in a set of multi-view images that reflect the multi-dimensional spatial structure of the model. Subsequently, a sketch was randomly sampled from the same category in the sketch dataset. and Construct positive sample pairs and and Input to pre-trained feature extractor Basic feature extraction is performed to obtain the view's basic feature set. Basic features of the sketch , For feature dimensions.

[0015] Preferably, by establishing a bounding sphere observation space and uniformly setting virtual camera viewpoints on the sphere, the spatial structure information of the 3D model can be captured comprehensively from multiple perspectives, overcoming the information loss problem caused by a single perspective or fixed projection method. At the same time, using a pre-trained CNN for basic feature extraction not only ensures the effectiveness of the features but also reduces the learning difficulty of subsequent networks. In addition, constructing positive sample pairs between the sketch and the multi-view images of the 3D model lays the data foundation for subsequent cross-modal feature alignment, making the association learning between the sketch and the 3D model more direct and effective.

[0016] Preferably, in step S2, a dual-path adaptive sensing network DPAP-Net is constructed, characterized in that:

[0017] In obtaining the basic feature set of multi-view Basic features of the sketch Then, DPAP-Net is constructed to filter the importance of 3D view information. This network works in conjunction with the static prior perception path SPA and the dynamic sketch anchoring path DSA.

[0018] S21. In SPA, the aim is to comprehensively evaluate the underlying information content of each viewpoint by quantifying three key dimensions of the view, thereby constructing an evaluation distribution that reflects the objective quality of the view. Specifically, let... For view Binarized object mask, For the projected area, Representing pixel coordinates in image space, they are evaluated using the following three metrics: (I) High-frequency energy distribution (HFE) is used to measure the structural richness of a view: , in, It is the absolute value. This indicates that the Laplace operator is used to extract the second derivative components; For the three-dimensional model Each view image coordinate; For the three-dimensional model HFE score for each view; (II) Local Binary Mode Entropy (LBP) is used to measure the texture richness of a view. Let... The probability distribution of a uniform local binary pattern within the object region: , For the three-dimensional model LBP score for each view; (III) Saliency score (SMS) is used to measure the visual appeal of a view: , in, This indicates the generation of the first [item] using the spectral residual method. A saliency plot, For the three-dimensional model The SMS score for each view is calculated by obtaining the scores for the three dimensions mentioned above, and then using a linear scaling method to calculate the normalized weight distribution of each viewpoint under different dimensions. , in, For the first The original score of each view on the k-th evaluation dimension For the first Normalized weights of each view in the k-th dimension The total score for all views on the k-th dimension. To prevent small constants with denominators of zero, a subsequent constant is introduced. Normalized learnable dimension importance parameters We obtain the static prior weights for each perspective by weighted summation of the normalized weights of the three dimensions: , in, The importance weight of the k-th evaluation dimension; For the first Static prior weights of each view; S22. In DSA, utilize the basic features of the sketch. As a semantic anchor, a cross-modal attention mechanism is used to capture a set of basic features from multiple views. Basic features of the sketch To determine the correlation strength between them, firstly, a multilayer perceptron (MLP) is used to capture the basic features of the sketch. With multi-view basic feature set Nonlinear interaction: , in, For addition operation, For the first Basic features of each view For the first First, match the score for each view; second, introduce a temperature coefficient. The scores are sharpened to generate dynamic semantic response weights for each view: , The dynamic semantic response weight for the i-th view; S23. Finally, the system uses learnable fusion factors. The static prior weights of each view are added together with the dynamic semantic response weights to obtain the final view weights: , Let the final view weight be the i-th view; apply this weight to the multi-view basic feature set. Generate the final 3D model features. : , In addition, to eliminate redundant interference from outlier views, a low-rank tensor loss LRTLoss is introduced for global constraints: , in, It represents the sum of the singular values ​​of a matrix, enhancing consistency between views.

[0019] Preferably, the core innovation of this step lies in the dual-path collaborative mechanism; the static prior perception path SPA comprehensively evaluates the view quality from three objective dimensions: high-frequency energy, texture richness, and visual saliency, avoiding the overfitting risk caused by simply relying on data-driven approaches; the dynamic sketch anchoring path DSA uses sketch features as semantic anchors and dynamically evaluates the correlation strength between each view and the sketch through cross-modal attention; the two paths reflect the "inherent quality" and "semantic relevance to the query" of the view, respectively, and after being fused by learnable factors, they can adaptively select the optimal view for different retrieval tasks; in addition, the low-rank tensor loss enhances the consistency between view features through nuclear norm constraints, effectively suppressing the interference of outlier views.

[0020] Preferably, in step S3, a sketch feature map is obtained. and global features of the sketch Spatial guidance map Its features are: First, input the sketch. Using sketch feature extractor Obtain the sketch feature map ,in, represents the channel dimension of the feature map. and These represent the height and width of the feature map in the spatial dimension, respectively; subsequently, global average pooling (GAP) is used to refine the sketch feature map. The spatial dimension is compressed, and a multilayer perceptron (MLP) is used to perform linear mapping and dimensionality reduction on the pooled vectors. Normalization ensures that features are aligned on the hypersphere, thus obtaining global features of the sketch. To achieve prototype guidance for accurately locating discriminative local strokes in the feature map, the following operations are further performed: (I) Channel Alignment: To ensure that the feature map is aligned with the embedding space dimension, a 1×1 convolutional layer Conv is used to align the sketch feature map. Projected to dimension The embedding space is used to generate embedding feature maps. ; (II) Prototype Extraction: From the pre-defined sketch classification layer weight matrix of the sketch branch Extract the corresponding category column vectors, The total number of categories, after normalization, serves as the prototype for that class. ; (III) Spatial guidance map generation: by calculating the embedded feature map Each spatial location and prototype Cosine similarity is used to generate spatial guidance maps. And using the Sigmoid function and temperature coefficient Map to the (0,1) interval: , It is a spatial guidance map in location The weight of the position, Calculation of module length; (IV) Guiding Consistency Constraint: To prevent prototype guidance from degrading during training and causing response failure, a guiding constraint loss is introduced. This loss encourages the spatial guidance map to generate response intensity at spatial locations, ensuring that dynamic sampling can capture key local information: .

[0021] Preferably, this step generates a spatial guidance map through category prototype guidance, which can accurately locate discriminative local stroke regions in the sketch, rather than simply relying on global features. Specifically, the prototype vector of the corresponding category is extracted from the sketch classification layer weights, and the positional cosine similarity is calculated with the embedded feature map, so that high-response regions automatically focus on the typical stroke structure of that category. At the same time, the guidance constraint loss encourages the guidance map to generate response intensity, effectively preventing the prototype guidance from degenerating and failing during training, and ensuring that dynamic sampling can always capture key local information.

[0022] Preferably, in step S4, the dynamic sampling DS driven by the guide is characterized by: In obtaining the spatial guidance map With embedded feature maps Then, using spatial guidance maps With embedded feature maps The interaction between the two enables adaptive adjustment of the feature extraction location. First, the embedded feature map is... Spatial guidance map Channel stitching is performed, and the spatial offset of sampling points is predicted using lightweight convolutional layers. : , in, The offset is limited to the (-1, 1) interval to ensure training stability. By controlling the movement range of sampling points, the sampling points are encouraged to snap towards stroke regions with semantic information, thus overcoming the limitations of regular grid sampling and enabling spatial offset. Guided by this, deformable convolution is used to embed the feature map. Perform irregular sampling to generate dynamic feature maps with structural robustness. : , Subsequently, spatial guidance maps were used. As a spatial weight matrix, for dynamic feature maps By performing a weighted aggregation operation, key stroke regions are assigned high weights while large blank areas are suppressed, and local structural features of the sketch are extracted. : , in, It is a dynamic feature map at position The feature vector at that location, It is a spatial guidance map in location The weight of the position, To prevent small constants with a denominator of zero, For element-wise multiplication; finally, the global features of the sketch are... Local structural features of the sketch Perform weighted fusion and utilize learnable fusion coefficients. Adjusting the feature ratio of the two, through After normalization, the final feature vector of the sketch is obtained, which can simultaneously represent the global semantics and key local structures. : .

[0023] Preferably, the core advantage of this step lies in breaking the limitations of traditional convolutional neural network regular grid sampling; through the interaction of spatial guide map and embedded feature map, lightweight convolutional layer can predict the spatial offset of each sampling point, enabling sampling points to adaptively gather towards stroke regions with semantic information; subsequently, deformable convolution performs irregular sampling at these offset positions, generating dynamic feature maps that are robust to geometric changes such as deformation, rotation, and scaling; finally, the spatial guide map is used as a weight matrix to perform weighted aggregation of dynamic feature maps, which not only highlights the contribution of key stroke regions, but also effectively suppresses the interference of large areas of blank background, realizing the coordinated expression of global semantics and local structure.

[0024] Preferably, in step S5, cross-domain feature alignment is performed, characterized in that: To achieve cross-modal feature space alignment and improve semantic discriminative power, firstly, auxiliary classification constraints are introduced for both modal branches. For the 3D model branch, additive angular margin softmax loss is used. Features of 3D Models Imposing constraints enhances intra-class compactness and inter-class separability: , in, Representing the features of a 3D model Weight matrix of classification layer in 3D model No. Cosine similarity of columns For the first Three-dimensional model features Scaling factor This is an additive margin, where N is the number of samples in the batch. This is the index of the current sample. yes when Special cases at that time For the first The true class labels of each sample; for the sketch branch, cross-entropy loss is used. For the final eigenvector of the sketch Apply category prediction constraints to ensure that the final feature vector of the sketch has semantic discriminative ability: , in, Real Category The corresponding weight column vector, Real Category The corresponding weight column vector, For the first The final feature vector of the sketch is then used to further reduce the distribution difference between the sketch modality and the view modality, a cross-modal transfer loss is introduced. : , in, True Class in 3D Model Classifier The weight vector, It is the square of the Euclidean distance, and the loss is achieved by... The final feature vector of the sketch Weight matrix of classification layer in 3D model Corresponding category The weight vectors are narrowed to directly map the final feature vectors of the sketch to the feature space of the 3D model. Finally, the auxiliary classification loss and cross-modal transfer loss are combined to achieve accurate cross-domain alignment between the sketch and the 3D model. The specific formula for the total loss is as follows: .

[0025] Preferably, this step achieves cross-domain feature alignment between the sketch and the 3D model through triple loss co-optimization. First, the 3D model branch uses additive angular margin Softmax loss, which forcibly increases the inter-class distance and reduces the intra-class distance in the cosine space, significantly improving the discriminative ability of the features. Second, the sketch branch uses cross-entropy loss to ensure that the sketch features are semantically separable. Most importantly, the cross-modal transfer loss directly pulls the sketch features closer to the weight vector of the corresponding class in the 3D model classifier, so that the sketch features are explicitly mapped to the feature space of the 3D model, fundamentally reducing the distribution difference between the two modalities. After the triple loss co-optimization, the sketch and the 3D model are comparable in the same metric space, and the retrieval accuracy is significantly improved.

[0026] Preferably, a 3D model retrieval system based on multi-view fusion and prototype guidance is characterized by comprising: Training Phase: Data from the 3D model database and sketch database are preprocessed to obtain a multi-view basic feature set and sketch basic features. A dual-path adaptive perception network is used to filter and fuse the features from the multi-view basic feature set and sketch basic features to obtain 3D model features. Sketches are processed to obtain sketch feature maps and sketch global features. Prototype guidance is performed to generate a spatial guidance map, followed by dynamic sampling. The interaction between the spatial guidance map and the embedded feature map is used to adaptively adjust the feature extraction position to obtain the final sketch feature vector. Then, the 3D model features and the final sketch feature vector are used to perform cross-domain alignment between the two modalities, finally obtaining the 3D model feature space and the trained sketch features. Retrieval Phase: The user inputs a sketch to be queried, and the features of the sketch are extracted; based on the 3D model feature space obtained during the training phase and the trained sketch features, the cosine similarity between the sketch features and all 3D model features in the database is calculated; finally, the most matching 3D model detection results are returned, sorted from high to low similarity.

[0027] Preferably, the main innovations of this system include: First, a dual-path adaptive perception network that integrates the selection of features from multiple views and sketches, thereby enhancing feature expressiveness; Second, a prototype-guided dynamic sampling mechanism that uses a spatial guidance map to adaptively adjust the feature extraction position, thereby enhancing the ability to capture key areas of the sketch; Third, a cross-domain alignment strategy that constructs a unified feature space and achieves modality matching; During retrieval, results are returned by quickly sorting based on cosine similarity, balancing accuracy and efficiency.

[0028] Compared with the prior art, the present invention has the following technical effects:

[0029] This method significantly improves the accuracy of sketch-based 3D model retrieval. First, by employing the dual-path adaptive perceptual network DPAP-Net, combined with static priors and dynamic anchoring mechanisms, it effectively identifies and enhances "sketch-friendly" views, resolving feature mismatch issues caused by viewpoint bias. Second, utilizing the prototype-guided dynamic sampling strategy PGDS, it achieves adaptive adsorption of sampling points towards effective stroke regions, accurately extracting key local structures in sparse sketches and overcoming noise interference from large areas of blank background. This method is rationally designed and structurally simple. While improving cross-modal feature alignment accuracy, it is easy to implement and possesses good scalability and industrial application prospects. Attached Figure Description

[0030] Figure 1 This invention presents a flowchart of a 3D model retrieval method and system based on multi-view fusion and prototype guidance.

[0031] Figure 2 This is a diagram of the dual-path adaptive sensing network structure proposed in this invention.

[0032] Figure 3 This is a prototype guidance operation structure diagram proposed in this invention.

[0033] Figure 4 This is the dynamic sampling structure proposed in this invention.

[0034] Figure 5 This is the overall flowchart of the training and retrieval process proposed in this invention. Detailed Implementation

[0035] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0036] This invention utilizes the dual-path adaptive perceptual network DPAP-Net to achieve efficient fusion of multi-view features. By objectively quantifying the underlying information of the view through static prior paths and combining it with real-time capture of semantic relevance through dynamic sketch anchoring paths, the system can adaptively identify and enhance views consistent with user drawing preferences. Combined with low-rank tensor loss to suppress outlier interference, the accuracy and stability of cross-domain feature space alignment are significantly enhanced. Simultaneously, through the prototype-guided dynamic sampling strategy PGDS, category priors guide sampling points to actively "adsorb" towards stroke regions with semantic information. This dynamic offset mechanism breaks the limitations of traditional convolutional regular grid sampling, enabling the model to bypass large areas of meaningless blank space and accurately capture fine-grained topological structures in discontinuous lines, fundamentally enhancing the discriminative power of local features under sparse input. The method proposed in this application significantly improves the accuracy and stability of retrieval results while ensuring computational efficiency and ease of implementation.

[0037] Please see Figures 1 to 5 The 3D model retrieval method and system based on multi-view fusion and prototype guidance in this application aims to improve the accuracy and stability of retrieval results while ensuring computational efficiency and ease of implementation through a dual-path adaptive perception network and a prototype-guided dynamic sampling strategy.

[0038] S1. Preprocess the 3D model and sketch to obtain the basic feature set of the 3D model in multiple views. and basic features of the sketch ;

[0039] Furthermore, in step S1, the set of basic features for the multi-view model of the 3D model is obtained. and basic features of the sketch The specific steps are as follows:

[0040] Let the cross-modal dataset be... ,in Represents the sketch dataset, Representing a 3D model dataset, to distinguish the relevant variables of the two modalities, the superscript 's' represents the sketch and the superscript 'v' represents the 3D model. First, the 3D model is preprocessed; specifically, a bounding sphere observation space is established around the geometric center of the 3D model as the origin; and uniformly spaced variables are set on the same horizontal plane on the sphere. Using multiple virtual camera viewpoints, the 3D model is projected and rendered onto each viewpoint, resulting in a set of multi-view images that reflect the multi-dimensional spatial structure of the model. Subsequently, a sketch was randomly sampled from the same category in the sketch dataset. and Construct positive sample pairs and and Input to pre-trained feature extractor Basic feature extraction is performed to obtain the view's basic feature set. Basic features of the sketch , For feature dimensions.

[0041] S2. Construct a dual-path adaptive perception network DPAP-Net and process the multi-view basic feature set. and basic features of the sketch Perform feature filtering and fusion;

[0042] Furthermore, in step S2, the dual-path adaptive sensing network DPAP-Net is constructed, and the specific steps are as follows:

[0043] In obtaining the basic feature set of multi-view Basic features of the sketch Then, DPAP-Net is constructed to filter the importance of 3D view information. This network works in conjunction with the static prior perception path SPA and the dynamic sketch anchoring path DSA. S21. In SPA, the aim is to comprehensively evaluate the underlying information content of each viewpoint by quantifying three key dimensions of the view, thereby constructing an evaluation distribution that reflects the objective quality of the view. Specifically, let... For view Binarized object mask, For the projected area, Representing pixel coordinates in image space, they are evaluated using the following three metrics: (I) High-frequency energy distribution (HFE) is used to measure the structural richness of a view: , in, It is the absolute value. This indicates that the Laplace operator is used to extract the second derivative components; For the three-dimensional model Each view image coordinate; For the three-dimensional model HFE score for each view; (II) Local Binary Mode Entropy (LBP) is used to measure the texture richness of a view. Let... The probability distribution of a uniform local binary pattern within the object region: , For the three-dimensional model LBP score for each view; (III) Saliency score (SMS) is used to measure the visual appeal of a view: , in, This indicates the generation of the first [item] using the spectral residual method. A saliency plot, For the three-dimensional model The SMS score for each view is calculated by obtaining the scores for the three dimensions mentioned above, and then using a linear scaling method to calculate the normalized weight distribution of each viewpoint under different dimensions. , in, For the first The original score of each view on the k-th evaluation dimension For the first Normalized weights of each view in the k-th dimension The total score for all views on the k-th dimension. To prevent small constants with a denominator of zero, Then introduced Normalized learnable dimension importance parameters ,and We obtain the static prior weights for each perspective by weighted summation of the normalized weights of the three dimensions: , in, The importance weight of the k-th evaluation dimension; For the first Static prior weights of each view; S22. In DSA, utilize the basic features of the sketch. As a semantic anchor, a cross-modal attention mechanism is used to capture a set of basic features from multiple views. Basic features of the sketch To determine the correlation strength between them, firstly, a multilayer perceptron (MLP) is used to capture the basic features of the sketch. With multi-view basic feature set Nonlinear interaction: , in, For addition operation, For the first Basic features of each view For the first First, match the score for each view; second, introduce a temperature coefficient. The scores are sharpened to generate dynamic semantic response weights for each view: , The dynamic semantic response weight for the i-th view; ; S23. Finally, the system uses learnable fusion factors. The static prior weights of each view are added together with the dynamic semantic response weights to obtain the final view weights: , Let be the final view weight of the i-th view; ,in, The initial value is 0; Apply this weight to the multi-view basic feature set Generate the final 3D model features. : , In addition, to eliminate redundant interference from outlier views, a low-rank tensor loss LRTLoss is introduced for global constraints: , in, It represents the sum of the singular values ​​of a matrix, enhancing consistency between views.

[0044] S3. Process the sketch to obtain the sketch feature map. and global features of the sketch Execute prototype-guided PG operations to generate a spatial guide map. ;

[0045] Furthermore, in step S3, the sketch feature map is obtained. and global features of the sketch Spatial guidance map The specific steps are as follows:

[0046] First, input the sketch. Using sketch feature extractor Obtain the sketch feature map ,in, represents the channel dimension of the feature map. and These represent the height and width of the feature map in the spatial dimension, respectively; subsequently, global average pooling (GAP) is used to refine the sketch feature map. The spatial dimension is compressed, and a multilayer perceptron (MLP) is used to perform linear mapping and dimensionality reduction on the pooled vectors. Normalization ensures that features are aligned on the hypersphere, thus obtaining global features of the sketch. To achieve prototype guidance for accurately locating discriminative local strokes in the feature map, the following operations are further performed: (I) Channel Alignment: To ensure that the feature map is aligned with the embedding space dimension, a 1×1 convolutional layer Conv is used to align the sketch feature map. Projected to dimension The embedding space is used to generate embedding feature maps. ; (II) Prototype Extraction: From the pre-defined sketch classification layer weight matrix of the sketch branch Extract the corresponding category column vectors, The total number of categories, after normalization, serves as the prototype for that class. ; (III) Spatial guidance map generation: by calculating the embedded feature map Each spatial location and prototype Cosine similarity is used to generate spatial guidance maps. And using the Sigmoid function and temperature coefficient Map to the (0,1) interval: , It is a spatial guidance map in location The weight of the position, Calculation of module length; ; (IV) Guiding Consistency Constraint: To prevent prototype guidance from degrading during training and causing response failure, a guiding constraint loss is introduced. This loss encourages the spatial guidance map to generate response intensity at spatial locations, ensuring that dynamic sampling can capture key local information: .

[0047] S4. Perform dynamic sampling (DS) using the spatial guidance map. With embedded feature maps The interaction enables adaptive adjustment of the feature extraction location;

[0048] Furthermore, in step S4, the dynamic sampling DS of the bootstrap driver is guided by the following steps:

[0049] In obtaining the spatial guidance map With embedded feature maps Then, using spatial guidance maps With embedded feature maps The interaction between the two enables adaptive adjustment of the feature extraction location. First, the embedded feature map is... Spatial guidance map Channel stitching is performed, and the spatial offset of sampling points is predicted using lightweight convolutional layers. : , in, The offset is limited to the (-1, 1) interval to ensure training stability. Controlling the movement range of the sampling points This encourages sampling points to snap towards stroke regions with semantic information, overcoming the limitations of regular grid sampling, and allowing for spatial offset. Guided by this, deformable convolution is used to embed the feature map. Perform irregular sampling to generate dynamic feature maps with structural robustness. : , Subsequently, spatial guidance maps were used. As a spatial weight matrix, for dynamic feature maps By performing a weighted aggregation operation, key stroke regions are assigned high weights while large blank areas are suppressed, and local structural features of the sketch are extracted. : , in, It is a dynamic feature map at position The feature vector at that location, It is a spatial guidance map in location The weight of the position, To prevent small constants with a denominator of zero, , For element-wise multiplication; finally, the global features of the sketch are... Local structural features of the sketch Perform weighted fusion and utilize learnable fusion coefficients. Adjusting the feature ratio of the two, through After normalization, the final feature vector of the sketch is obtained, which can simultaneously represent the global semantics and key local structures. : , in, and The initial value is 0.

[0050] S5. Introduce auxiliary classification constraints for the two modal branches respectively to achieve cross-domain feature alignment between the sketch and the 3D model;

[0051] Furthermore, in step S5, cross-domain feature alignment is performed, specifically through the following steps:

[0052] To achieve cross-modal feature space alignment and improve semantic discriminative power, firstly, auxiliary classification constraints are introduced for both modal branches. For the 3D model branch, additive angular margin softmax loss is used. Features of 3D Models Imposing constraints enhances intra-class compactness and inter-class separability: , in, Representing the features of a 3D model Weight matrix of classification layer in 3D model No. Cosine similarity of columns For the first Three-dimensional model features The scaling factor is set to 16. The additive margin is set to 0.5, and N is the number of samples in the batch. This is the index of the current sample. yes when Special cases at that time For the first The true class labels of each sample; for the sketch branch, cross-entropy loss is used. For the final eigenvector of the sketch Apply category prediction constraints to ensure that the final feature vector of the sketch has semantic discriminative ability: , in, Real Category The corresponding weight column vector, Real Category The corresponding weight column vector, For the first The final feature vector of the sketch is then used to further reduce the distribution difference between the sketch modality and the view modality, a cross-modal transfer loss is introduced. : , in, True Class in 3D Model Classifier The weight vector, It is the square of the Euclidean distance, and the loss is achieved by... The final feature vector of the sketch Weight matrix of classification layer in 3D model Corresponding category The weight vectors are narrowed to directly map the final feature vectors of the sketch to the feature space of the 3D model. Finally, the auxiliary classification loss and cross-modal transfer loss are combined to achieve accurate cross-domain alignment between the sketch and the 3D model. The specific formula for the total loss is as follows: , in, , Weighting coefficient A dynamic adjustment strategy is adopted, setting a small value in the early stage of training to stabilize the learning of the 3D model classifier, and gradually increasing it to the target value as the training rounds increase; specifically, let the current training round be . The total number of training rounds is ,but: ,in, , , .

[0053] S6. Input the sketch to be queried, calculate the cosine similarity between the features of the sketch and the features of the 3D model in the database, sort them in descending order of score, and output the search results.

[0054] Furthermore, the search results are output in S6. The specific steps are as follows: After the system training is completed, the user inputs a sketch to be queried; the system quickly calculates the features of the sketch according to the process of S1, S3, and S4; then, it calculates the cosine similarity with the features of all 3D models in the database; finally, it sorts the similarity from high to low and returns the detection result of the most matching 3D model to the user.

[0055] Furthermore, the 3D model retrieval system based on multi-view fusion and prototype guidance is characterized by comprising: Training Phase: Data from the 3D model database and sketch database are preprocessed to obtain a multi-view basic feature set and sketch basic features. A dual-path adaptive perception network is used to filter and fuse the features from the multi-view basic feature set and sketch basic features to obtain 3D model features. Sketches are processed to obtain sketch feature maps and sketch global features. Prototype guidance is performed to generate a spatial guidance map, followed by dynamic sampling. The interaction between the spatial guidance map and the embedded feature map is used to adaptively adjust the feature extraction position to obtain the final sketch feature vector. Then, the 3D model features and the final sketch feature vector are used to perform cross-domain alignment between the two modalities, finally obtaining the 3D model feature space and the trained sketch features. Retrieval Phase: The user inputs a sketch to be queried, and the features of the sketch are extracted; based on the 3D model feature space obtained during the training phase and the trained sketch features, the cosine similarity between the sketch features and all 3D model features in the database is calculated; finally, the most matching 3D model detection results are returned, sorted from high to low similarity.

[0056] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A 3D model retrieval method based on multi-view fusion and prototype guidance, characterized in that, Includes the following steps: S1. Preprocess the 3D model and sketch to obtain the basic feature set of the 3D model in multiple views. and basic features of the sketch ; S2. Construct a dual-path adaptive perception network DPAP-Net and process the multi-view basic feature set. and basic features of the sketch Perform feature filtering and fusion; In obtaining the basic feature set of multi-view Basic features of the sketch Then, DPAP-Net is constructed to filter the importance of 3D view information. This network works in conjunction with the static prior perception path SPA and the dynamic sketch anchoring path DSA. S21. In SPA, the aim is to comprehensively evaluate the underlying information content of each perspective by quantifying the three key dimensions of the view, thereby constructing an evaluation distribution that reflects the objective quality of the view. S22. In DSA, utilize the basic features of the sketch. As a semantic anchor, a cross-modal attention mechanism is used to capture a set of basic features from multiple views. Basic features of the sketch The strength of the correlation between them; S23. Finally, the system uses learnable fusion factors. The static prior weights of each view are added together with the dynamic semantic response weights to obtain the final view weights. S3. Process the sketch to obtain the sketch feature map. and global features of the sketch Execute prototype-guided PG operations to generate a spatial guide map. ; S4. Perform dynamic sampling DS, utilizing the spatial guidance map. With embedded feature maps The interaction between the features enables adaptive adjustment of the feature extraction location; the embedded feature map It uses a 1×1 convolutional layer (Conv) to extract the sketch feature map. Projected to dimension The embedded space is generated; S5. Introduce auxiliary classification constraints for the two modal branches respectively to achieve cross-domain feature alignment between the sketch and the 3D model; S6. Input the sketch to be queried, calculate the cosine similarity between the features of the sketch and the features of the 3D model in the database, sort them in descending order of score, and output the search results.

2. The 3D model retrieval method based on multi-view fusion and prototype guidance according to claim 1, in step S1, obtains the multi-view basic feature set of the 3D model. and basic features of the sketch Its features are: set up Cross-modal dataset is ,in Represents the sketch dataset, Representing a 3D model dataset, to distinguish the relevant variables of the two modalities, the superscript 's' represents the sketch and the superscript 'v' represents the 3D model. First, the 3D model is preprocessed; specifically, a bounding sphere observation space is established around the geometric center of the 3D model as the origin; and uniformly spaced variables are set on the same horizontal plane on the sphere. Using multiple virtual camera viewpoints, the 3D model is projected and rendered onto each viewpoint, resulting in a set of multi-view images that reflect the multi-dimensional spatial structure of the model. Subsequently, a sketch was randomly sampled from the same category in the sketch dataset. and Construct positive sample pairs and and Input to pre-trained feature extractor Basic feature extraction is performed to obtain the view's basic feature set. Basic features of the sketch , For feature dimensions.

3. The 3D model retrieval method based on multi-view fusion and prototype guidance according to claim 2, wherein in step S2, a dual-path adaptive perception network DPAP-Net is constructed, characterized in that: In obtaining the basic feature set of multi-view Basic features of the sketch Then, DPAP-Net is constructed to filter the importance of 3D view information. This network works in conjunction with the static prior perception path SPA and the dynamic sketch anchoring path DSA. S21. In SPA, the aim is to comprehensively evaluate the underlying information content of each viewpoint by quantifying three key dimensions of the view, thereby constructing an evaluation distribution that reflects the objective quality of the view. Specifically, let... For view Binarized object mask, For the projected area, Representing pixel coordinates in image space, they are evaluated using the following three metrics: (I) High-frequency energy distribution (HFE) is used to measure the structural richness of a view: ， in, It is the absolute value. This indicates that the Laplace operator is used to extract the second derivative components; For the three-dimensional model Each view image coordinate; For the three-dimensional model HFE score for each view; (II) Local Binary Mode Entropy (LBP) is used to measure the texture richness of a view. Let... The probability distribution of a uniform local binary pattern within the object region: ， For the three-dimensional model LBP score for each view; (III) Saliency score (SMS) is used to measure the visual appeal of a view: ， in, This indicates the generation of the first [item] using the spectral residual method. A saliency plot, For the three-dimensional model The SMS score for each view is calculated by obtaining the scores for the three dimensions mentioned above, and then using a linear scaling method to calculate the normalized weight distribution of each viewpoint under different dimensions. ， in, For the first The original score of each view on the k-th evaluation dimension For the first Normalized weights of each view in the k-th dimension The total score for all views on the k-th dimension. To prevent small constants with denominators of zero, a subsequent constant is introduced. Normalized learnable dimension importance parameters We obtain the static prior weights for each perspective by weighted summation of the normalized weights of the three dimensions: ， in, The importance weight of the k-th evaluation dimension; For the first Static prior weights of each view; S22. In DSA, utilize the basic features of the sketch. As a semantic anchor, a cross-modal attention mechanism is used to capture a set of basic features from multiple views. Basic features of the sketch To determine the correlation strength between them, firstly, a multilayer perceptron (MLP) is used to capture the basic features of the sketch. With multi-view basic feature set Nonlinear interaction: ， in, For addition operation, For the first Basic features of each view For the first First, match the score for each view; second, introduce a temperature coefficient. The scores are sharpened to generate dynamic semantic response weights for each view: ， The dynamic semantic response weight for the i-th view; S23. Finally, the system uses learnable fusion factors. The static prior weights of each view are added together with the dynamic semantic response weights to obtain the final view weights: ， Let be the final view weight of the i-th view; Apply this weight to the multi-view basic feature set Generate the final 3D model features. : ， In addition, to eliminate redundant interference from outlier views, a low-rank tensor loss LRTLoss is introduced for global constraints: ， in, It represents the sum of the singular values ​​of a matrix, enhancing consistency between views.

4. In the 3D model retrieval method based on multi-view fusion and prototype guidance according to claim 3, in step S3, the sketch feature map is obtained. and global features of the sketch Spatial guidance map Its features are: First, input the sketch. Using sketch feature extractor Obtain the sketch feature map ,in, represents the channel dimension of the feature map. and These represent the height and width of the feature map in the spatial dimension, respectively; subsequently, global average pooling (GAP) is used to refine the sketch feature map. The spatial dimension is compressed, and a multilayer perceptron (MLP) is used to perform linear mapping and dimensionality reduction on the pooled vectors. Normalization ensures that features are aligned on the hypersphere, thus obtaining global features of the sketch. To achieve prototype guidance for accurately locating discriminative local strokes in the feature map, the following operations are further performed: (I) Channel Alignment: To ensure that the feature map is aligned with the embedding space dimension, a 1×1 convolutional layer Conv is used to align the sketch feature map. Projected to dimension The embedding space is used to generate embedding feature maps. ; (II) Prototype Extraction: From the pre-defined sketch classification layer weight matrix of the sketch branch Extract the corresponding category column vectors, The total number of categories, after normalization, serves as the prototype for that class. ; (III) Spatial guidance map generation: by calculating the embedded feature map Each spatial location and prototype Cosine similarity is used to generate spatial guidance maps. And using the Sigmoid function and temperature coefficient Map to the (0,1) interval: ， It is a spatial guidance map in location The weight of the position, Calculation of module length; (IV) Guiding Consistency Constraint: To prevent prototype guidance from degrading during training and causing response failure, a guiding constraint loss is introduced. This loss encourages the spatial guidance map to generate response intensity at spatial locations, ensuring that dynamic sampling can capture key local information: 。 5. The 3D model retrieval method based on multi-view fusion and prototype guidance according to claim 4, wherein in step S4, the guided dynamic sampling DS is characterized in that: In obtaining the spatial guidance map With embedded feature maps Then, using spatial guidance maps With embedded feature maps The interaction between the two enables adaptive adjustment of the feature extraction location. First, the embedded feature map is... Spatial guidance map Channel stitching is performed, and the spatial offset of sampling points is predicted using lightweight convolutional layers. : ， in, The offset is limited to the (-1, 1) interval to ensure training stability. By controlling the movement range of sampling points, the sampling points are encouraged to snap towards stroke regions with semantic information, thus overcoming the limitations of regular grid sampling and enabling spatial offset. Guided by this, deformable convolution is used to embed the feature map. Perform irregular sampling to generate dynamic feature maps with structural robustness. : ， Subsequently, spatial guidance maps were used. As a spatial weight matrix, for dynamic feature maps By performing a weighted aggregation operation, key stroke regions are assigned high weights while large blank areas are suppressed, and local structural features of the sketch are extracted. : , in, It is a dynamic feature map at position The feature vector at that location, It is a spatial guidance map in location The weight of the position, To prevent small constants with a denominator of zero, For element-wise multiplication; finally, the global features of the sketch are... Local structural features of the sketch Perform weighted fusion and utilize learnable fusion coefficients. Adjusting the feature ratio of the two, through After normalization, the final feature vector of the sketch is obtained, which can simultaneously represent the global semantics and key local structures. : .

6. The 3D model retrieval method based on multi-view fusion and prototype guidance according to claim 5, wherein cross-domain feature alignment is performed in step S5, characterized in that: To achieve cross-modal feature space alignment and improve semantic discriminative power, firstly, auxiliary classification constraints are introduced for both modal branches. For the 3D model branch, additive angular margin softmax loss is used. Features of 3D Models Imposing constraints enhances intra-class compactness and inter-class separability: ， in, Representing the features of a 3D model Weight matrix of classification layer in 3D model No. Cosine similarity of columns For the first Three-dimensional model features Scaling factor This is an additive margin, where N is the number of samples in the batch. This is the index of the current sample. yes when Special cases at that time For the first The true class labels of each sample; for the sketch branch, cross-entropy loss is used. For the final eigenvector of the sketch Apply category prediction constraints to ensure that the final feature vector of the sketch has semantic discriminative ability: ， in, Real Category The corresponding weight column vector, Real Category The corresponding weight column vector, For the first The final feature vector of the sketch is then used to further reduce the distribution difference between the sketch modality and the view modality, a cross-modal transfer loss is introduced. : ， in, True Class in 3D Model Classifier The weight vector, It is the square of the Euclidean distance, and the loss is achieved by... The final feature vector of the sketch Weight matrix of classification layer in 3D model Corresponding category The weight vectors are narrowed to directly map the final feature vectors of the sketch to the feature space of the 3D model. Finally, the auxiliary classification loss and cross-modal transfer loss are combined to achieve accurate cross-domain alignment between the sketch and the 3D model. The specific formula for the total loss is as follows: 。 7. A 3D model retrieval system based on multi-view fusion and prototype guidance, implementing the 3D model retrieval method based on multi-view fusion and prototype guidance as described in any one of claims 1-6, characterized in that, include: Training phase: The data in the 3D model database and sketch database are preprocessed to obtain the multi-view basic feature set and sketch basic features. The dual-path adaptive perception network is used to filter and fuse the features of the multi-view basic feature set and sketch basic features. Obtain the features of the three-dimensional model; The sketch is processed to obtain a sketch feature map and a global sketch feature map. A prototype guidance operation is performed to generate a spatial guidance map. Then, dynamic sampling is performed, and the interaction between the spatial guidance map and the embedded feature map is used to achieve adaptive adjustment of the feature extraction position. The final feature vector of the sketch is obtained; Then, the two modalities are aligned across domains using the 3D model features and the final feature vector of the sketch, finally obtaining the 3D model feature space and the trained sketch features. Retrieval phase: The user inputs a sketch to be queried, and the features of the sketch to be queried are extracted; Based on the 3D model feature space obtained during the training phase and the trained sketch features, the cosine similarity between the sketch features to be queried and all 3D model features in the database is calculated. Finally, the most matching 3D model detection results are returned by sorting the similarity from high to low.

Images