Cross-modal retrieval method for sketch retrieval three-dimensional model based on spatiotemporal feature information

Abstract

The invention discloses a cross-modal retrieval method for a sketch retrieval three-dimensional model based on spatiotemporal feature information, which comprises the following steps: firstly, selecting data, then constructing a sketch three-dimensional model image sequence and a spatiotemporal feature information extraction network, extracting spatiotemporal feature information of a sketch and athree-dimensional model by using the spatiotemporal feature information extraction network, combining the spatial and temporal feature information of the sketch with the spatial and temporal feature information of the three-dimensional model by using deep metric learning; and finally, calculating the similarity according to the Euclidean distance between the spatial and temporal feature information of the sketch and the spatial and temporal feature information of the three-dimensional model in the spatial and temporal feature information combination. The method is outstanding in retrieval performance, capable of effectively completing cross-modal retrieval of the sketch retrieval three-dimensional model, better in accuracy, easy to operate and high in practicability.

Description

technical field [0001] The invention relates to the technical fields of computer graphics, computer vision and intelligent recognition, in particular to a cross-modal retrieval method for retrieving a 3D model from a sketch based on spatio-temporal feature information. Background technique [0002] With the rapid development of computer-aided design and computer vision, three-dimensional objects, as an important data type, have become one of the main carriers of information after sound, image and video, and have a wide range of applications in the fields of industrial manufacturing, virtual reality and augmented reality. Applications. How to effectively identify and retrieve 3D models is the research basis of many applications, and has become a topic of concern to researchers. Hand-drawn sketches are easy to construct, very intuitive, and are not affected by external factors such as geography, profession, age, etc. As a very effective means of communication. In recent years...

AI Technical Summary

Problems solved by technology

[0006] In the feature cross-domain embedding part, part of the 3D model retrieval method based on sketches performs similarity evaluation directly after the initial feature extraction of 3D models and sketches. For example, Liu et al. used CNN to extract the features of sketches and 3D models. Minimum distance method for similarity evaluation [Liu Yujie, Song Yang, Li Zongmin, et al. Sketch-based 3D shape retrieval with representative view and convolutional neural network [J]. Journal of Graphics, 2018, 39(4): 735-741( in Chinese) (Liu Jie, Song Yang, Zong, et al. Hand-painted 3D model retrieval based on fusion of information entropy and CNN [J]. Journal of Graphics, 2018, 39(4): 735-741)]. Taking full account of the differences between cross-domain data, the retrieval effect of this method is not ideal.

[0007] More feature embedding methods use metric learning to embed the initial features of sketches and 3D models into a common space, so that the distance of similar data (same domain and cross-domain) in the feature space is closer, and the distance of heterogeneous data is farther. For example, Wang et al. used the binary metric learning network Siamese to force the distance between similar cross-domain data to be close enough to complete the representation and embedding of cross-domain data [Wang F, Kang L, Li Y. Sketch-based 3d shape retrieval using convolutional neural networks[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:1875-1883]. Qi et al. [Qi A, Song Y Z, Xiang T. Semantic Embedding for Sketch-Based 3D Shape Retrieval[C] / / BMVC.2018,2(7):8], Bai et al [J.Bai, M.Wang, and D.Kong, Deep Common Semantic Space Embedding for Sketch-Based3D Model Retrieval, Entropy, vol.21, no. 4, pp.369, 2019] considered the relationship between similar data and heterogeneous data at the same time, and proposed a cross-domain data embedding based on triplet loss (Triplet Loss), which achieved good results. However, based on triplet loss The method of learning (Triplet Loss) needs to consider each pair of positive and negative sample combinations, and network training is time-consuming and difficult.

Images