A sketch 3D model retrieval method based on prototype contrast learning

Abstract

The application discloses a sketch three-dimensional model retrieval method based on prototype contrast learning, and relates to the technical field of multimedia information retrieval; the sketch three-dimensional model retrieval method comprises the following steps: S1, selecting a data set sample; S2, data set sample preprocessing; S3, training a sketch domain feature extractor and a three-dimensional domain model feature extractor; respectively calculating a clustering prototype for each sample, calculating a prototype contrast loss, and iteratively updating the network parameters of the domain feature extractor; S4, training a cross-domain alignment mapping network; respectively calculating the domain features of a sketch and a three-dimensional model, matching the sketch and the three-dimensional model into pairs according to categories, calculating a common clustering prototype through the mapping network, calculating a prototype contrast loss, and iteratively updating the cross-domain alignment mapping network; the application can effectively reduce the semantic gap between the sketch and the three-dimensional model while increasing the difference between categories, and can quickly and accurately retrieve a three-dimensional model of a corresponding category by inputting a sketch.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multimedia information retrieval, and particularly relates to a sketch-based three-dimensional model retrieval method based on prototype contrast learning. Figure Three The present application belongs to the technical field of multimedia information retrieval, and particularly relates to a sketch-based three-dimensional model retrieval method based on prototype contrast learning. BACKGROUND

[0002] In recent years, with the rapid development of multimedia technology and Internet technology, three-dimensional models are widely used in the entertainment industry and the industrial field and play an increasingly important role. In order to help users easily find the three-dimensional models they need and reuse and share them, three-dimensional model retrieval has become a popular research direction in the field of multimedia information retrieval technology. The widespread popularity of touch screen devices enables people to draw sketches anytime and anywhere, and many people have begun to try to express and communicate by hand-drawing sketches. Since sketches can contain more abundant meanings than texts and are easier to obtain than three-dimensional models, compared with traditional methods based on keywords and three-dimensional model samples, hand-drawing sketches are a more suitable interactive way for three-dimensional model retrieval.

[0003] In existing research, in 2013, Li et al. proposed "SHREC'13 track: Large scale sketch-based 3D shape retrieval" and "SHREC'14 track: Extended large scale sketch-based 3D shape retrieval", which can be seen as early sketch-based three-dimensional model retrieval methods. Figure Three The existing sketch-based three-dimensional model retrieval methods are built on the features handcrafted by researchers, which is time-consuming and laborious. In 2015, Wang et al. proposed using a twin network to solve the cross-domain problem in "Sketch-based 3d shape retrieval using convolutional neural networks", which first introduced deep learning into the task of sketch-based three-dimensional model retrieval; but this method ignored the three-dimensionality of three-dimensional models and only selected one view of the three-dimensional models to pair with the sketches for learning, so the performance was low. In 2018, Dai et al. proposed a sketch-based three-dimensional model retrieval method based on holistic metric learning in "Deep correlated holistic metric learning for sketch-based 3D shape retrieval", which used a deep neural network to learn the global features of three-dimensional models and sketches, and then used the global features to calculate the similarity between the three-dimensional models and the sketches, but this method also ignored the three-dimensionality of three-dimensional models and only selected one view of the three-dimensional models to pair with the sketches for learning, so the performance was low. Figure Three Figure Three ​The cross-domain retrieval method uses discriminative loss to improve the distinguishability of different classes of goods within the domain, and uses correlation loss to reduce the distance of the same class of goods between domains. The performance of this method has greatly improved compared with the past. Unfortunately, alignment operations need to be performed at multiple nodes of the network during training, which is somewhat cumbersome. In 2020, Dai et al. proposed a cross-domain guidance training based on the sketch of the sketch-based 3D shape retrieval in the knowledge distillation model. Figure Three The cross-domain retrieval method has good retrieval performance through a large number of experiments.

[0004] However, although the sketch-based three-dimensional model retrieval is fast and convenient, and has made some progress, it still faces many challenges. First of all, although sketches and three-dimensional models belong to the same visual media, they are different in dimension and generation method, and there is a great semantic gap between the two domains. Secondly, the sketch media itself has high abstraction, and the sketch style of the same class of goods drawn by different people may be very different.

[0005] The present application proposes a sketch-based three-dimensional model retrieval method based on prototype contrast learning. Figure Three The cross-domain retrieval method enables samples from two different domains of sketches and three-dimensional models to be gathered around the common prototypes of their respective categories in a common semantic space, and as far away from the common prototypes of other categories as possible, thereby effectively reducing the semantic gap between domains while increasing the difference between categories. Through the input sketch, the corresponding three-dimensional model of the category is quickly and accurately retrieved. SUMMARY

[0006] The purpose of the present application is to provide a sketch-based three-dimensional model retrieval method based on prototype contrast learning. Figure Three The cross-domain retrieval method solves the problems of the sketch-based three-dimensional model retrieval in the background technology, such as the high abstraction of sketches, the great difference in sketch style of the same class of goods drawn by different people, and the great semantic gap between the two domains.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] A sketch-based three-dimensional model retrieval method based on prototype contrast learning, comprising the following steps: Figure Three The cross-domain retrieval method based on prototype contrast learning comprises the following steps:

[0009] S1, select dataset samples;

[0010] The dataset samples are selected from the sketch Figure Three The cross-domain retrieval dataset is selected from the sketch Figure ThreeThe 3D model retrieval dataset includes a subset of sketch data and a subset of 3D model data;

[0011] S2, Dataset Sample Preprocessing;

[0012] The sketch data in the sketch data subset of S1 is unified into a binary image, and each three-dimensional model data in the three-dimensional model data subset is rendered to obtain 12 two-dimensional grayscale views.

[0013] S3, training the sketch domain feature extractor and the 3D domain model feature extractor;

[0014] Design a sketch domain feature extractor network and a 3D model domain feature extractor network. Use the sketch domain feature extractor network and the 3D model domain feature extractor network to extract the features of the preprocessed samples in S2. Calculate the clustering prototypes using the sample features so that each sample clusters near the corresponding category prototype. Calculate the prototype contrast loss and iteratively update the parameters of the domain feature extractor network to obtain the sketch domain feature extractor and the 3D domain model feature extractor.

[0015] S4. Train the cross-domain alignment mapping network;

[0016] Using the sketch domain feature extractor and the 3D model domain feature extractor obtained in S3, the domain features of the sketch and the 3D model are calculated respectively. The sketch and the 3D model are matched into pairs according to category. The common cluster prototype is calculated through the mapping network. The prototype contrast loss is calculated. The cross-domain alignment mapping network is iteratively updated.

[0017] Preferably, the grass Figure Three The 3D model retrieval dataset includes data of different categories. For each category of sketch data, the training and testing samples are divided in a 5:3 ratio, and for each category of 3D models, the training and testing samples are divided in an 8:2 ratio.

[0018] Preferably, in step S2, each 3D model data in the subset of 3D model data is rendered into 12 2D grayscale views, as follows:

[0019] The 3D model is placed on a plane, and 12 virtual camera positions are evenly placed around it at 30 degrees. Lighting is added during rendering, and the brightness and darkness of the 2D view are used to represent the surface depth of the 3D model, resulting in 12 2D grayscale views.

[0020] Preferably, in step S3, clustering is used to calculate category prototypes, and contrastive learning is used to bring similar prototypes closer to each other and move away from dissimilar prototypes, as follows:

[0021] S31. Design a sketch domain feature extractor network; Utilize AlexNet to design a sketch domain feature extractor network tailored to the characteristics of sketch data. The first layer of the sketch domain feature extractor network has a 15*15 convolutional kernel, and the dropout rate after the last fully connected layer is 0.5, which is used to extract sketch features.

[0022] S32. Design a 3D model domain feature extractor network; using the MVCNN method for 3D model multi-view as a framework, and using ResNet-50 pre-trained on the large image dataset ImageNet as the backbone convolutional neural network to process the grayscale rendering image of each view, a 3D model domain feature extractor network is obtained to extract the features of the 3D model.

[0023] S33. Use the sketch domain feature extractor network and the 3D model domain feature extractor network to extract sketch data training samples and 3D model training samples to train their respective features. Calculate the cluster prototypes of the training samples so that each sample is clustered near the corresponding category prototype and as far away as possible from the prototypes of other categories. Calculate the prototype contrast loss and iteratively update the parameters of the domain feature extractor network to obtain the sketch domain feature extractor and the 3D domain model feature extractor.

[0024] S34. Validate the intra-domain feature classification accuracy of the sketch domain feature extractor and the 3D domain model feature extractor using sketch data test samples and 3D model test samples.

[0025] Preferably, the loss function used to iteratively update the parameters of the in-domain feature extractor network during training in S33 is the prototype contrastive loss.

[0026] The corresponding calculation formula is as follows:

[0027]

[0028] Where v is the feature vector extracted from the sample; sim is a similarity measurement function; τ is a control coefficient; Pc is the cluster prototype of c-class features in the memory data; c+ represents the positive class; and P is the set of all cluster prototypes.

[0029] Preferably, in S4, the prototype contrast loss is calculated according to formula (1) in S3, and the cross-domain aligned mapping network is iteratively updated.

[0030] Preferably, the mapping network is a three-layer fully connected network with a size of 2048-1024-256, and there is one feature extractor in the sketch domain and one feature extractor in the 3D domain.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] (1) This invention enables samples from two different domains, sketch and 3D model, to gather in a common semantic space near the common prototype of their respective categories and as far away as possible from the common prototype of other categories. This effectively reduces the semantic gap between domains while increasing the differences between categories, and allows for quick and accurate retrieval of the corresponding category's 3D model by inputting a sketch.

[0033] (2) The method provided by this invention is simple and easy to implement. The main content is to train two intra-domain feature extractors and a cross-domain alignment mapping network using a prototype contrastive learning training method. Extensive experiments on the two general datasets SHREC'13 and SHREC'14 show that this method has excellent retrieval performance, which is no less than the best current method, and has good application prospects. Attached Figure Description

[0034] Figure One This is a schematic diagram of the overall framework of the present invention;

[0035] Figure Two This is a schematic diagram illustrating the core principle of the present invention;

[0036] Figure Three This is an example image of the search results obtained by the present invention on the two datasets SHREC'13 and SHREC'14. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1:

[0039] A grass based on prototype contrastive learning Figure Three The dimensional model retrieval method includes the following steps:

[0040] Step 1: Selecting the dataset.

[0041] The dataset selects the common SHREC'13 and SHREC'14 drafts. Figure Three The 3D model retrieval dataset contains subsets of sketch data and subsets of 3D model data.

[0042] The SHREC'13 dataset contains 7,200 sketches and 1,258 3D models, totaling 90 different categories. Each sketch category contains 50 training samples and 30 test samples. The sample size for each 3D model category is different. During the experiment, the samples were randomly divided into training and test samples at a ratio of 80% to 20%.

[0043] The SHREC'14 dataset contains 13,680 sketches and 8,987 3D models, totaling 171 different categories. Each sketch category contains 50 training samples and 30 test samples. The sample size for each 3D model category is different. During the experiment, the models were randomly divided into training and test samples at a ratio of 80% to 20%.

[0044] Step 2: Dataset preprocessing.

[0045] Data preprocessing is performed on the samples in the dataset selected in step (1).

[0046] For the sketch data, the original samples in the dataset are 8-channel grayscale images with a size of 1111*1111. To facilitate network learning and processing, they are uniformly transformed into binary images with a size of 224*224.

[0047] For 3D model data, the original samples of the dataset are 3D models in OFF format. This embodiment provides a method to render them as multiple 2D views: assuming the model is placed on a plane with a consistent orientation (such as perpendicular to the horizontal line), 12 virtual camera positions are evenly placed around it at 30 degrees. At the same time, to more clearly highlight the depth information of the 3D model, lighting is added during rendering, and the brightness and darkness of the 2D views represent the depth of the 3D model surface. This results in 12 2D views. Further processing is performed to unify the view size to 256*256 and transform it into an 8-channel grayscale image.

[0048] Step 3: Train the two intra-domain feature extractors corresponding to the sketch and the 3D model respectively.

[0049] The method provided by this invention believes that the foundation for solving cross-domain retrieval problems lies in a thorough understanding and learning of the data features within a single domain. Therefore, as... Figure One As shown, the overall framework consists of two parts: intra-domain feature extraction and cross-domain feature alignment. This step is described in detail below. Figure One The left side shows the intra-domain feature extraction.

[0050] For sketch feature extractors, since there are large blank areas in sketches and huge style differences between different sketch samples, although they are all two-dimensional images, this invention does not choose to directly use convolutional neural networks designed for real photos like existing technologies. Instead, it redesigns a new network based on AlexNet to extract sketch features: the convolutional kernel of the first layer is increased from 11*11 to 15*15 to filter blank information, and the dropout rate after the last fully connected layer is increased to 0.5 to reduce the solidification of sketch style learning.

[0051] For the 3D model feature extractor, this invention uses the high-performance 3D model multi-view method MVCNN as a framework, and uses ResNet-50 pre-trained on the large image dataset ImageNet as the backbone convolutional neural network to process the grayscale rendering image of each view.

[0052] For both the sketch feature extractor and the 3D model feature extractor, this invention utilizes the samples preprocessed in step two to calculate their clustering prototypes, ensuring that each sample clusters near its corresponding category prototype and as far away as possible from prototypes of other categories, such as... Figure Two As shown. More specifically, the loss function used during training to iteratively update the parameters of the in-domain feature extractor network is the prototype contrast loss. The corresponding calculation formula is as follows:

[0053]

[0054] Where v is the feature vector extracted from the sample; sim is a similarity measurement function, using the L2 norm in the experiment; τ is a control coefficient, set to 0.07 according to the contrastive learning tradition in the experiment; Pc is the cluster prototype of c-class features in the memory data; c+ represents the positive class; P is the set of all cluster prototypes.

[0055] In step three, the feature extractor is trained by calculating a series of cluster prototypes using the feature extractor, and then using the cluster prototypes to calculate the loss function of the prototype comparison loss, which updates the parameters of the feature extractor, making the classification more accurate during the feature extraction and classification process.

[0056] Step 4: Train the cross-domain aligned mapping network.

[0057] So far, two intra-domain feature extractors for the corresponding sketches and 3D models have been trained using a prototype-contrastive learning approach. Next, according to the requirements of cross-domain retrieval tasks, features of sketches and 3D models that semantically belong to the same category should be mapped to adjacent positions in a common semantic space. This step is called... Figure One Cross-domain feature alignment is shown on the right.

[0058] Due to such Figure Two The prototype contrastive learning method shown is not limited to specific multimedia formats. Here, it is still used to train the cross-domain alignment mapping network: using the two feature extractors obtained in step three, the intra-domain features of the sketch and the 3D model are calculated respectively. They are matched into pairs according to their categories, and the common cluster prototype is calculated through the corresponding mapping network. The common prototypes are kept as far away from other categories as possible. Then, the prototype contrastive loss is calculated according to formula (1), and the cross-domain alignment mapping network is iteratively updated. The mapping network is a three-layer fully connected network with a size of 2048-1024-256, and there is one sketch intra-domain feature extractor and one 3D intra-domain model feature extractor.

[0059] The method provided by this invention uses two common herbal formulas, SHREC'13 and SHREC'14. Figure Three Experiments were conducted on the dimensional model dataset, and the retrieval metrics FT, DCG, E, and mAP were recorded on the two datasets as shown in Tables 1 and 2. For the metrics, E should be as close to 0% as possible, while FT, DCG, and mAP should be as close to 100% as possible.

[0060] Table 1. Comparison of retrieval performance (%) of different schemes on the SHREC'13 dataset

[0061] Scheme FT DCG E mAP Comparative Scheme 1 14.2 20.8 76.1 20.8 Comparative Scheme 2 43.3 51.0 54.7 52.5 Comparative Scheme 3 52.0 60.8 53.4 58.8 Inventive Scheme 54.7 71.9 45.4 65.9

[0062] Table 2 Comparison of retrieval performance (%) of different schemes on the SHREC'14 dataset

[0063] Scheme FT DCG E mAP Comparative Scheme 1 12.9 20.0 76.1 20.7 Comparative Scheme 2 32.5 41.4 61.2 41.9 Comparative Scheme 3 35.0 41.7 61.6 44.2 Inventive Scheme 40.3 45.5 58.1 49.1

[0064] In the table above, comparison scheme 1 is the Siamese network method proposed by Wang et al. in "Sketch-based 3D shape retrieval using convolutional neural networks"; comparison scheme 2 is the holistic metric learning method proposed by Dai et al. in "Deepcorrelated holistic metric learning for sketch-based 3D shape retrieval"; and comparison scheme 3 is the cross-domain guided training method proposed by Dai et al. in "Cross-modal guidance network for sketch-based 3D shape retrieval". Observing Tables 1 and 2, it can be seen that the proposed scheme has certain advantages over the comparison schemes in the four retrieval metrics (FT, DCG, E, and mAP) on the two general datasets SHREC'13 and SHREC'14. It can retrieve the corresponding category of 3D models by inputting a sketch, showing good potential for application.

[0065] Examples of retrieving results using the method provided in this invention on two datasets are as follows: Figure Three The left side shows the input sketch, and the right side shows the top 5 retrieved 3D models. Failed cases are highlighted with boxes and their corresponding category names. It can be seen that even the failed cases have a certain visual similarity to the successful cases, which to some extent corroborates the effectiveness of the method provided by this invention.

[0066] The above description is only for the purpose of helping to understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, any equivalent substitutions or modifications made to the technical solution and inventive concept disclosed in the present invention within the scope of the technology disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for retrieving 3D sketch models based on prototype contrastive learning, characterized in that, Includes the following steps: S1. Select dataset samples; The dataset sample selection is a sketch 3D model retrieval dataset, which includes a subset of sketch data and a subset of 3D model data; S2, Dataset Sample Preprocessing; The sketch data in the sketch data subset of S1 is unified into a binary image, and each three-dimensional model data in the three-dimensional model data subset is rendered to obtain 12 two-dimensional grayscale views. S3, training the sketch domain feature extractor and the 3D domain model feature extractor; Design a sketch domain feature extractor network and a 3D model domain feature extractor network. Use the sketch domain feature extractor network and the 3D model domain feature extractor network to extract the features of the preprocessed samples in S2. Calculate the cluster prototype using the sample features. Calculate the category prototype using clustering. Use contrastive learning to bring similar prototypes closer to each other and away from dissimilar prototypes, so that each sample clusters near the corresponding category prototype. Calculate the prototype contrastive loss and iteratively update the parameters of the domain feature extractor network to obtain the sketch domain feature extractor and the 3D domain model feature extractor. S4. Train the cross-domain alignment mapping network; Using the sketch domain feature extractor and the 3D model domain feature extractor obtained in S3, the domain features of the sketch and the 3D model are calculated respectively. The sketch and the 3D model are matched into pairs according to category. The common cluster prototype is calculated through the mapping network. The prototype contrast loss is calculated. The cross-domain alignment mapping network is iteratively updated.

2. The sketch 3D model retrieval method according to claim 1, characterized in that: The sketch 3D model retrieval dataset includes different categories of data. Each category of sketch data is divided into training samples and test samples in a 5:3 ratio, and each category of 3D models is divided into training samples and test samples in an 8:2 ratio.

3. The sketch 3D model retrieval method according to claim 2, characterized in that: In step S2, each 3D model data in the subset of 3D model data is rendered into 12 2D grayscale views, as follows: The 3D model is placed on a plane, and 12 virtual camera positions are evenly placed around it at 30 degrees. Lighting is added during rendering, and the brightness and darkness of the 2D view are used to represent the surface depth of the 3D model, resulting in 12 2D grayscale views.

4. The sketch 3D model retrieval method according to claim 3, characterized in that: S3 is specifically as follows: S31. Design a sketch domain feature extractor network; utilize AlexNet to design a sketch domain feature extractor network tailored to the characteristics of sketch data. The first layer of the sketch domain feature extractor network uses a convolutional kernel of... Finally, the dropout rate after the fully connected layer is 0.5, which is used to extract features from the sketch. S32. Design a feature extractor network for the 3D model domain; Using the MVCNN method for multi-view 3D models as a framework, and taking ResNet-50 pre-trained on the large image dataset ImageNet as the backbone convolutional neural network to process the grayscale rendering image of each view, a feature extractor network in the 3D model domain is obtained to extract the features of the 3D model. S33. Use the sketch domain feature extractor network and the 3D model domain feature extractor network to extract sketch data training samples and 3D model training samples to train their respective features. Calculate the cluster prototypes of the training samples so that each sample is clustered near the corresponding category prototype and as far away as possible from the prototypes of other categories. Calculate the prototype contrast loss and iteratively update the parameters of the domain feature extractor network to obtain the sketch domain feature extractor and the 3D domain model feature extractor. S34. Validate the intra-domain feature classification accuracy of the sketch domain feature extractor and the 3D domain model feature extractor using sketch data test samples and 3D model test samples.

5. The sketch 3D model retrieval method according to claim 4, characterized in that: In S33, the loss function used to iteratively update the parameters of the in-domain feature extractor network during training is the prototype contrastive loss. , The corresponding calculation formula is as follows: （1） Where v is the feature vector extracted from the sample; sim is a similarity measurement function; τ is a control coefficient; Pc is the cluster prototype of c-class features in the memory data; c+ represents the positive class; and P is the set of all cluster prototypes.

6. The sketch 3D model retrieval method according to claim 5, characterized in that: In S4, the prototype contrast loss is calculated according to formula (1) in S3, and the cross-domain aligned mapping network is iteratively updated.

7. The sketch 3D model retrieval method according to claim 6, characterized in that: The mapping network is a three-layer fully connected network with a size of 2048-1024-256, and there is one feature extractor in the sketch domain and one feature extractor in the 3D domain.

Images