Two-stage 3D model sketch retrieval method based on feature transfer

Abstract

The present invention provides a two-stage 3D model sketch retrieval method based on feature migration, which has the following characteristics: Step S1, annotating category labels on existing hand-drawn sketch data as a first training data set, and annotating category labels on existing 3D model data and its two-dimensional view data as a second training data set; Step S2, training a hand-drawn sketch feature extraction network based on the first training data to obtain category center vectors of each category label; Step S3, training a 3D model feature extraction network based on the second training data set; Step S4, extracting 3D model features of the 3D model based on the 3D model feature extraction network, and extracting sketch features of the sketch to be queried based on the hand-drawn sketch feature extraction network; Step S5, calculating the similarity between the sketch features and each 3D model feature, and outputting the 3D model based on the similarity. In summary, this method can improve the accuracy of retrieval of 3D models based on hand-drawn sketches.

Description

Technical Field

[0001] The present invention relates to the field of sketch-based three-dimensional model retrieval, and in particular to a two-stage three-dimensional model sketch retrieval method based on feature migration. Background Art

[0002] Three-dimensional models are widely used in many fields, including industrial design, VR and AR, film and television animation, and smart homes. In recent years, with the advent of the "Internet + 3D" big data era, the scale of three-dimensional model data on the Internet has grown exponentially, and the market demand for effective and rapid searches of three-dimensional models has also increased rapidly. The development of three-dimensional model retrieval technology is becoming increasingly important. In the field of three-dimensional model retrieval, due to the complexity of three-dimensional models in terms of spatial structure and semantic details, traditional keyword and example-based retrieval methods cannot fully express query intent and are difficult to retrieve three-dimensional models that meet user intent. Hand-drawn sketches can more flexibly and intuitively express details such as the geometric shape, topological structure, and spatial position relationships of objects, more effectively expressing user query intent. Therefore, sketch-based three-dimensional model retrieval is an effective, easy-to-use, and very promising retrieval method.

[0003] Currently, sketch-based 3D model retrieval methods can be divided into two main categories: traditional feature engineering-based methods and deep learning-based methods. Traditional feature engineering-based methods mostly use handcrafted features, using low-order geometric features and image descriptors to represent sketches and 3D models. However, since these shallow, manually designed features often lack high-level semantic information, these methods suffer from significant deficiencies in retrieval accuracy and robustness.

[0004] Deep learning, based on big data and an end-to-end learning approach, avoids the drawbacks of traditional feature engineering methods. Currently, most deep learning-based methods are single-stage approaches based on twin neural networks. For example, Wang et al. proposed learning similarity features between sketches and 3D model projections using twin convolutional neural networks; He et al. proposed a triple center loss function to generate a unified feature space; and Lei et al. proposed a method that directly uses features from multiple views of a 3D model to measure similarity with sketch features without feature fusion.

[0005] However, these single-stage methods still have the following problems: on the one hand, the current single-stage feature space learning method learns the feature space of the sketch and the 3D model at the same time, which causes the sketch and the 3D model feature space to interfere with each other during the learning process, resulting in a decrease in learning effect; on the other hand, the drawing of hand-drawn sketches is highly subjective and abstract. How to solve the impact of overly abstract and low-quality sketch images mixed in the algorithm training set on the final training results is also an urgent problem to be solved. Summary of the Invention

[0006] The present invention is made to solve the above problems, and its purpose is to provide a two-stage three-dimensional model sketch retrieval method based on feature migration.

[0007] The present invention provides a two-stage three-dimensional model sketch retrieval method based on feature migration, which has the following characteristics and includes the following steps: step S1, annotating category labels on existing hand-drawn sketch data as a first training data set, and annotating category labels on existing three-dimensional model data and multi-angle two-dimensional view data of three-dimensional model data as a second training data set; step S2, constructing a hand-drawn sketch feature extraction network based on a classification neural network, training the hand-drawn sketch feature extraction network according to the first training data, inputting the first training data set into the trained hand-drawn sketch feature extraction network, and obtaining the category center vector of each category label; step S3, constructing a three-dimensional model feature extraction network, and training the three-dimensional model feature extraction network according to the second training data set; step S4, extracting three-dimensional model features of each three-dimensional model in a three-dimensional model database according to the three-dimensional model feature extraction network, and extracting sketch features of a sketch to be queried according to the hand-drawn sketch feature extraction network; step S5, calculating the similarity between the sketch features and each three-dimensional model feature, and outputting the corresponding three-dimensional models in descending order of similarity.

[0008] The two-stage three-dimensional model sketch retrieval method based on feature migration provided by the present invention may also have the following features: wherein, step S1 includes the following sub-steps: step S1-1, annotating category labels for hand-drawn sketch data, adjusting the image size of the hand-drawn sketch data to a fixed size, and normalizing the pixel range to between 0 and 1; step S1-2, based on multiple virtual cameras, obtaining multi-angle two-dimensional view data of the three-dimensional model data, annotating category labels for the three-dimensional model data and the two-dimensional view data, adjusting the image size of the two-dimensional view data to a fixed size, and normalizing the pixel range to between 0 and 1.

[0009] In the two-stage three-dimensional model sketch retrieval method based on feature migration provided by the present invention, it can also have the following characteristics: wherein, step S2 includes the following sub-steps: step S2-1, a hand-drawn sketch feature extraction network based on a classification neural network is formed by a convolutional neural network Sketch-CNN, a first branch fully connected network FC1, a second branch fully connected network FC2 and a fully connected classification layer FC3; step S2-2, according to the first training data set, combined with a joint loss function L, a hand-drawn sketch feature extraction network based on a classification neural network is trained, and the joint loss function L is composed of the classification loss function L classification and uncertainty-based regularization loss function L regIn step S2-3, the first training data set is input into the trained hand-drawn sketch feature extraction network to obtain the sketch feature vector. The vector space jointly constituted by the sketch feature vector is used as the sketch feature space. The average feature vector is calculated based on the sketch feature vectors of the same category label as the category center vector of the category label.

[0010] The two-stage three-dimensional model sketch retrieval method based on feature migration provided by the present invention may also have the following characteristics: wherein, in step S2-2, the classification loss function L classification The expression is: Regularization loss function L reg The expression is: The expression of the joint loss function L is: L = L classification +αL reg , where f i Input sketch features of the fully connected classification layer FC3 in the training phase, yi is the category label of sketch feature i, is the weight vector corresponding to the category label yi in the fully connected classification layer FC3, m is the hyperparameter representing the class interval size, s is the hyperparameter representing the scaling factor, N sketch is the size of the sketch training batch, μ is the output feature vector of the first branch fully connected network FC1, and σ is the N of the second branch fully connected network FC2. sketch Output feature vector, α is a manually set parameter.

[0011] The two-stage 3D model sketch retrieval method based on feature migration provided by the present invention may also have the following features: wherein step S3 includes the following sub-steps: step S3-1, a 3D model feature extraction network is formed by a basic convolutional neural network Shape-CNN and a multi-view feature fusion module Fusion-Layer; step S3-2, based on the second training data set, a feature migration loss function L is used to extract the 3D model feature. trans Train the 3D model feature extraction network.

[0012] The two-stage three-dimensional model sketch retrieval method based on feature migration provided by the present invention may also have the following characteristics: wherein, in step S3-2, the feature migration loss function L trans The expression is: Where, t i is the feature of the 3D model data i extracted by the 3D model feature extraction network, c yi is the category center vector corresponding to the category label yi of the 3D model data i in the hand-drawn sketch feature extraction network, N shape The size of the training batch for the 3D model.

[0013] The two-stage three-dimensional model sketch retrieval method based on feature migration provided by the present invention may also have the following features: wherein, step S4 includes the following sub-steps: step S4-1, obtaining multi-angle two-dimensional view data of each three-dimensional model, adjusting the image size of the two-dimensional view data to a fixed size, and normalizing the pixel range to between 0 and 1, inputting the two-dimensional view data of each three-dimensional model into the three-dimensional model feature extraction network, and obtaining the three-dimensional model features of each three-dimensional model; step S4-2, adjusting the image size of the sketch to be queried to a fixed size, and normalizing the pixel range to between 0 and 1, and inputting the sketch to be queried into the hand-drawn sketch feature extraction network to obtain sketch features.

[0014] The two-stage three-dimensional model sketch retrieval method based on feature migration provided by the present invention may also have the following feature: wherein the fixed size is 224×224 pixels.

[0015] The two-stage three-dimensional model sketch retrieval method based on feature migration provided by the present invention may also have the following feature: wherein, in step S5, the similarity is cosine similarity.

[0016] Functions and effects of the invention

[0017] According to the two-stage three-dimensional model sketch retrieval method based on feature transfer involved in the present invention, because a hand-drawn sketch feature extraction network is constructed based on an uncertainty learning method, and the hand-drawn sketch feature extraction network and the three-dimensional model feature extraction network are trained separately according to different training data, the two-stage three-dimensional model sketch retrieval method based on feature transfer of the present invention can achieve good results in training the hand-drawn sketch feature extraction network and the three-dimensional model feature extraction network, thereby improving the accuracy of retrieving three-dimensional models based on hand-drawn sketches. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 1 is a flow chart of a two-stage 3D model sketch retrieval method based on feature migration in an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of the principle of feature matching between a hand-drawn sketch and a three-dimensional model through a two-stage three-dimensional model sketch retrieval method based on feature migration in an embodiment of the present invention;

[0020] Figure 3 3. It is a schematic diagram comparing feature spaces generated by a traditional single-stage method and a two-stage 3D model sketch retrieval method based on feature migration under noise interference in an embodiment of the present invention;

[0021] Figure 4It is a schematic diagram of the results of applying a two-stage 3D model sketch retrieval method based on feature migration based on the SHREC2014 dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments and drawings specifically illustrate the two-stage three-dimensional model sketch retrieval method based on feature migration of the present invention.

[0023] In this embodiment, the public data sets SHREC 2013 and SHREC 2014 are used as training and verification data sets. The SHREC 2013 data set provides 1,258 three-dimensional models and 7,200 sketches, which are divided into 90 categories in total. On average, each category includes 14 three-dimensional models and 80 sketches. SHREC 2014 provides 8,987 three-dimensional models and 13,680 sketches, which are divided into 171 categories in total. On average, each category includes 52 three-dimensional models and 80 sketches. In this embodiment, 50 sketches from each category in the public data sets SHREC 2013 and SHREC 2014 are used as existing hand-drawn sketch data, and all three-dimensional models from each category in the public data sets SHREC 2013 and SHREC 2014 are used as existing three-dimensional model data to train the hand-drawn sketch feature extraction network and the three-dimensional model feature extraction network. In addition to the existing hand-drawn sketch data, 30 sketches from each category in the public datasets SHREC 2013 and SHREC 2014 are selected as query sketches to test the effectiveness of the two-stage 3D model sketch retrieval method based on feature transfer for 3D model retrieval.

[0024] Figure 1 3D model sketch retrieval method based on two-stage feature migration in an embodiment of the present invention.

[0025] like Figure 1 As shown, the two-stage 3D model sketch retrieval method based on feature migration in an embodiment of the present invention includes the following steps:

[0026] Step S1-1: annotate the hand-drawn sketch data with category labels, adjust the image size of the hand-drawn sketch data to a fixed size, and normalize the pixel range to between 0 and 1. Save the hand-drawn sketch data in different folders according to different category labels.

[0027] In step S1-2, based on multiple virtual cameras, multi-angle two-dimensional view data of the three-dimensional model data is obtained, category labels are annotated on the three-dimensional model data and the two-dimensional view data, the image size of the two-dimensional view data is adjusted to a fixed size, and the pixel range is normalized to between 0 and 1. The three-dimensional model data is stored in different folders according to different category labels.

[0028] Among them, in Blender 3D graphics software, virtual cameras are set orthogonally around the 3D model data, totaling 16 virtual cameras. Each virtual camera renders a 3D projection image from its perspective, and a total of 16 views are obtained as 2D view data of the 3D model data.

[0029] Figure 2 It is a schematic diagram of the principle of feature matching between a hand-drawn sketch and a three-dimensional model through a two-stage three-dimensional model sketch retrieval method based on feature migration in an embodiment of the present invention.

[0030] like Figure 2 As shown in Figure 1, in stage 1, the hand-drawn sketch, or hand-drawn sketch data, is passed through the Sketch CNN network, or the convolutional neural network Sketch-CNN. It then passes through the fully connected layer FC1, or the first branch of the fully connected network FC1, and the fully connected layer FC2, or the second branch of the fully connected network FC2, to obtain feature vectors μ and σ, respectively. Based on the feature vectors μ and σ, the Gaussian distribution N = (μ, σ) of the sketch features is obtained. This is then input into the fully connected layer FC3, or the fully connected classification layer FC3. The output is trained using a joint loss function L to obtain a hand-drawn sketch feature extraction network. The hand-drawn sketch is then input into the trained hand-drawn sketch feature extraction network to calculate the sketch category center, or the category center vector. In stage 2, the 3D model, or 3D model data, is passed through the View CNN network, or the convolutional neural network Shape-CNN, to obtain 2D view data. Feature fusion is then performed to obtain a 3D model feature descriptor. This 3D model feature descriptor is then trained using a transfer loss to cluster the model features of the 3D model based on the sketch category center.

[0031] Step S2-1, a hand-drawn sketch feature extraction network based on a classification neural network is formed by a convolutional neural network Sketch-CNN, a first branch fully connected network FC1, a second branch fully connected network FC2 and a fully connected classification layer FC3.

[0032] Among them, the hand-drawn sketch feature extraction network is based on uncertainty learning technology. It extracts the basic features of the sketch through the convolutional neural network Sketch-CNN and inputs the basic features into the two-branch fully connected layer, namely the first branch fully connected network FC1 and the second branch fully connected network FC2, which output feature vectors μ and σ respectively. In the training stage, the sketch feature is modeled as a Gaussian distribution with a mean of μ and a variance of σ, that is, f i =μ+εσ. Where ε is a random number randomly sampled from a normal distribution, and the variance σ represents the uncertainty of the sketch sample.

[0033] Step S2-2: Based on the first training data set, combined with the joint loss function L, a hand-drawn sketch feature extraction network based on the classification neural network is trained. The joint loss function L is composed of the classification loss function L classification and uncertainty-based regularization loss function L reg In the classification loss function L classification Under the supervision of , the hand-drawn sketch feature extraction network learns to generate a feature space that separates features of different category labels and aggregates features of the same category labels; under the regularization loss function L reg Under the supervision of

[15] , the hand-drawn sketch feature extraction network learns to reduce the weight of noise samples with higher uncertainty on loss optimization and focuses more on fitting samples with lower uncertainty, thereby reducing the negative impact of noise samples on network training.

[0034] Among them, the classification loss function L classification The expression is:

[0035]

[0036] Regularization loss function L reg The expression is:

[0037]

[0038] The expression of the joint loss function L is:

[0039] L=L classification +αL reg ,

[0040] Where f i Input sketch features of the fully connected classification layer FC3 in the training phase, yi is the category label of sketch feature i, is the weight vector corresponding to the category label yi in the fully connected classification layer FC3, m is the hyperparameter representing the class interval size, s is the hyperparameter representing the scaling factor, N sketch is the size of the sketch training batch, μ is the output feature vector of the first branch fully connected network FC1, and σ is the N of the second branch fully connected network FC2.sketch Output feature vector, α is a manually set parameter.

[0041] In step S2-3, the first training data set is input into the trained hand-drawn sketch feature extraction network to obtain the sketch feature vector. The vector space jointly constituted by the sketch feature vectors is used as the sketch feature space. The average feature vector is calculated based on the sketch feature vectors of the same category label as the category center vector of the category label.

[0042] In step S3-1, a 3D model feature extraction network is constructed by the basic convolutional neural network Shape-CNN and the multi-view feature fusion module Fusion-Layer.

[0043] Among them, the feature fusion module uses channel average pooling to perform feature fusion.

[0044] Step S3-2: Based on the second training data set, the feature transfer loss function L trans Train the 3D model feature extraction network.

[0045] The convolutional neural network Shape-CNN is used to extract the two-dimensional view data of the three-dimensional model in the second training data set, and the data is input into the feature fusion module Fusion-Layer for feature fusion to generate a unique three-dimensional model feature descriptor for the three-dimensional model. The three-dimensional model feature descriptor is converted using the transfer loss function L trans Training is performed. Transfer loss function L trans The purpose is to transfer the 3D model features into the sketch feature space, so that the sketch and 3D model features of the same category label are clustered together in the feature space.

[0046] Feature transfer loss function L trans The expression is:

[0047]

[0048] Where t i is the feature of the 3D model data i extracted by the 3D model feature extraction network, c yi is the category center vector corresponding to the category label yi of the 3D model data i in the hand-drawn sketch feature extraction network, N shape The size of the training batch for the 3D model.

[0049] Figure 3 3D is a schematic diagram comparing feature spaces generated by a traditional single-stage method and a two-stage three-dimensional model sketch retrieval method based on feature migration under noise interference in an embodiment of the present invention.

[0050] like Figure 3As shown, the normal sketch data is the sketch feature vector, the noise sketch data is the feature vector of the noise sketch, the 3D model data is the 3D model feature, the category center is the category center vector, Figure 3 The left part (a) is the feature space generated by the traditional single-stage method under noise interference. Figure 3 Part (b) on the right is the feature space generated by the two-stage three-dimensional model sketch retrieval method based on feature migration under noise interference. It can be seen that compared with the traditional single-stage method, the two-stage three-dimensional model sketch retrieval method based on feature migration of the present invention can make the sketch features and three-dimensional model features of the same category labels more clustered together in the feature space.

[0051] Step S4-1: For the 3D model database to be matched, obtain multi-angle 2D view data of each 3D model in the 3D model database, adjust the image size of the 2D view data to a fixed size, and normalize the pixel range to between 0 and 1, input the 2D view data of each 3D model into the 3D model feature extraction network to obtain the 3D model features of each 3D model.

[0052] Among them, in Blender 3D graphics software, virtual cameras are set orthogonally around the 3D model data, totaling 16 virtual cameras. Each virtual camera renders a 3D projection image from its perspective, and a total of 16 views are obtained as 2D view data of the 3D model data.

[0053] In step S4-2, the image size of the query sketch is adjusted to a fixed size, and the pixel range is normalized to between 0 and 1. The query sketch is input into the hand-drawn sketch feature extraction network to obtain sketch features.

[0054] In this embodiment, the fixed size of the image is 224×224 pixels.

[0055] Step S5: Calculate the similarity between the sketch features and the features of each 3D model, and output the corresponding 3D models in descending order of similarity.

[0056] Among them, the feature vector μ is directly used as the sketch feature to participate in the similarity calculation of the model feature, and the similarity calculation is performed based on the cosine similarity.

[0057] Figure 4 It is a schematic diagram of the results of applying a two-stage 3D model sketch retrieval method based on feature migration based on the SHREC2014 dataset in an embodiment of the present invention.

[0058] like Figure 4As shown in the figure, on the left side of the vertical line are some sketches selected from the SHREC2014 dataset as query sketches, including sketches of a hand, a tree, a helicopter, a telephone, and a chair from top to bottom. On the right side of the vertical line is a two-stage 3D model sketch retrieval method based on feature transfer, and the 3D models in the SHREC2014 dataset are matched. It can be seen that the query sketch and the matched 3D model have a high degree of similarity.

[0059] In this embodiment, the two-stage feature transfer-based 3D model sketch retrieval method of the present invention, i.e., the method of the present invention, is compared with other sketch-based 3D model retrieval methods. Other sketch-based 3D model retrieval methods include: Deep Correlation Metric Learning (DCML), Wasserstein Center of Gravity Representation Learning (LWBR), Triple Center Loss (TCL), Deep Cross-Modal Adaptation (DCA), Deep Sketch Model Hashing (DSSH), Deep Point Pair Subspace Metric Learning (DPSML), Attention and Hierarchical Representation Learning (HEAR), and Sketch Uncertainty Learning (SUL). Each retrieval method uses the AlexNet convolutional neural network structure as the Sketch-CNN network and the ResNet50 convolutional neural network structure as the Shape-CNN network, respectively, for a more comprehensive comparison. The SHREC 2013 and SHREC 2014 datasets are used as training and test datasets. Six common indicators are used as the performance measurement standards of each method, namely: Nearest Neighbor (NN), First Tier (FT), Second Tier (ST), E-Measure (E), Discounted Cumulative Gain (DCG) and mean Average Precision (mAP).

[0060] Each method is verified using the SHREC 2013 dataset, and the obtained general indicator data are as follows:

[0061]

[0062] The first column shows the various retrieval methods, the second column shows the basic models used by each retrieval method, and the third to eighth columns show the data of various indicators based on the SHREC 2013 dataset after each retrieval method adopts different basic models. For example, the third column of the twelfth row shows the two-stage 3D model sketch retrieval method based on feature transfer described in the present invention, which uses the AlexNet convolutional neural network structure as the Sketch-CNN network. Based on the SHREC 2013 dataset, the nearest neighbor accuracy obtained is 78.9. As shown in the table, using the SHREC 2013 dataset as the validation dataset, the two-stage 3D model sketch retrieval method based on feature transfer described in the present invention performs better in all six indicators than other sketch-based 3D model retrieval methods, with the lead margin being mostly within one percentage point.

[0063] Each method is verified using the SHREC 2014 dataset, and the obtained general indicator data are as follows:

[0064]

[0065] The first column shows the various retrieval methods, the second column shows the basic models used by the various retrieval methods, and the third to eighth columns show the data of various indicators of the various retrieval methods based on the dataset SHREC 2014 after adopting different basic models. For example, the third column of the twelfth row shows the two-stage 3D model sketch retrieval method based on feature transfer of the present invention, which uses the AlexNet convolutional neural network structure as the Sketch-CNN network. Based on the dataset SHREC 2014, the nearest neighbor accuracy obtained is 75.8. As shown in the table, using the more challenging dataset SHREC 2014 as the verification dataset, the two-stage 3D model sketch retrieval method based on feature transfer of the present invention still performs better in all six indicators compared to other sketch-based 3D model retrieval methods, indicating that the two-stage 3D model sketch retrieval method based on feature transfer of the present invention can maintain excellent retrieval performance even in more complex environments.

[0066] Functions and Effects of the Embodiments

[0067] The two-stage feature transfer-based 3D model sketch retrieval method of this embodiment, on the one hand, by separately training the hand-drawn sketch feature extraction network and the 3D model feature extraction network, avoids the degradation of learning results caused by mutual interference during the learning process. On the other hand, based on an uncertainty learning method, by constructing a hand-drawn sketch feature extraction network with a two-branch fully connected network, weights are assigned according to sketch quality during training, reducing the impact of low-quality sketch training data on the algorithm's retrieval performance. In summary, this method achieves excellent results for training the hand-drawn sketch feature extraction network and the 3D model feature extraction network, thereby improving the accuracy of 3D model retrieval from hand-drawn sketches.

[0068] The above embodiments are preferred examples of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A two-stage 3D model sketch retrieval method based on feature migration, characterized in that: The following steps are involved: Step S1: annotating category labels on existing hand-drawn sketch data as a first training data set, and annotating the category labels on existing 3D model data and multi-angle 2D view data of the 3D model data as a second training data set; Step S2: constructing a hand-drawn sketch feature extraction network based on a classification neural network, training the hand-drawn sketch feature extraction network according to the first training data, inputting the first training data set into the trained hand-drawn sketch feature extraction network, and obtaining a category center vector of each category label; Step S3, constructing a three-dimensional model feature extraction network, and training the three-dimensional model feature extraction network according to the second training data set; Step S4, extracting 3D model features of each 3D model in the 3D model database according to the 3D model feature extraction network, and extracting sketch features of the sketch to be queried according to the hand-drawn sketch feature extraction network; Step S5, calculating the similarity between the sketch feature and each of the three-dimensional model features, and outputting the corresponding three-dimensional models in descending order of similarity; Wherein, the step S1 includes the following sub-steps: Step S1-1, marking the category label on the hand-drawn sketch data, adjusting the image size of the hand-drawn sketch data to a fixed size, and normalizing the pixel range to between 0 and 1; Step S1-2: obtaining the two-dimensional view data of the three-dimensional model data from multiple angles based on multiple virtual cameras, labeling the three-dimensional model data and the two-dimensional view data with the category labels, adjusting the image size of the two-dimensional view data to the fixed size, and normalizing the pixel range to between 0 and 1. The step S3 includes the following sub-steps: Step S3-1, forming the 3D model feature extraction network by using the basic convolutional neural network Shape-CNN and the multi-view feature fusion module Fusion-Layer; Step S3-2: Based on the second training data set, the feature transfer loss function L trans training the three-dimensional model feature extraction network, In step S3-2, the feature migration loss function L trans The expression is: Where, t i is the feature of the three-dimensional model data i extracted by the three-dimensional model feature extraction network, c yi is the category center vector corresponding to the category label yi of the 3D model data i in the hand-drawn sketch feature extraction network, N shape The size of the training batch for the 3D model.

2. The two-stage 3D model sketch retrieval method based on feature migration according to claim 1, characterized in that: in, The step S2 includes the following sub-steps: Step S2-1, the hand-drawn sketch feature extraction network based on the classification neural network is composed of the convolutional neural network Sketch-CNN, the first branch fully connected network FC1, the second branch fully connected network FC2 and the fully connected classification layer FC3; Step S2-2: training the hand-drawn sketch feature extraction network based on the classification neural network based on the first training data set and the joint loss function L, wherein the joint loss function L is a function of the classification loss function L. classification and uncertainty-based regularization loss function L reg composed of; Step S2-3: input the first training data set into the trained hand-drawn sketch feature extraction network to obtain sketch feature vectors. The vector space jointly constituted by the sketch feature vectors is used as the sketch feature space. The average feature vector is calculated based on the sketch feature vectors of the same category label as the category center vector of the category label.

3. The two-stage 3D model sketch retrieval method based on feature transfer according to claim 2, characterized in that: in, In step S2-2, the classification loss function L classification The expression is: The regularized loss function L reg The expression is: The expression of the joint loss function L is: L=L classification +αL reg , Where, f i The sketch features of the fully connected classification layer FC3 are input in the training phase, yi is the category label of the sketch feature i, is the weight vector corresponding to the category label yi in the fully connected classification layer FC3, m is a hyperparameter representing the class interval size, s is a hyperparameter representing the scaling factor, N sketch is the size of the sketch training batch, μ is the output feature vector of the first branch fully connected network FC1, and σ is the N of the second branch fully connected network FC2. sketch Output feature vector, α is a manually set parameter.

4. The two-stage 3D model sketch retrieval method based on feature migration according to claim 1, Its characteristics are: Wherein, the step S4 includes the following sub-steps: Step S4-1, obtaining multi-angle 2D view data of each 3D model, adjusting the image size of the 2D view data to a fixed size, and normalizing the pixel range to between 0 and 1, inputting the 2D view data of each 3D model into the 3D model feature extraction network to obtain the 3D model features of each 3D model; Step S4-2: Adjust the image size of the query sketch to the fixed size, and normalize the pixel range to between 0 and 1, input the query sketch into the hand-drawn sketch feature extraction network to obtain the sketch features.

5. The two-stage 3D model sketch retrieval method based on feature migration according to claim 1 or 4, characterized in that: in, The fixed size is 224×224 pixels.

6. The two-stage 3D model sketch retrieval method based on feature transfer according to claim 1, characterized in that: in, In step S5, the similarity is cosine similarity.

Images