Sketch-based three-dimensional model retrieval method, device, equipment and medium

Abstract

The application discloses a sketch-based three-dimensional model retrieval method, device and equipment and a medium, wherein the sketch-based three-dimensional model retrieval method comprises the following steps: projecting a training sketch based on a line-of-sight direction of the training sketch by using a preset three-dimensional model viewpoint group to obtain a two-dimensional sketch view group; obtaining an enhanced sketch set of the two-dimensional sketch view group by using an edge detection algorithm and a sample enhancement algorithm; extracting sample features corresponding to the enhanced sketch set to generate a sample feature set, obtaining a public feature embedding space based on the sample feature set; and retrieving a to-be-retrieved sketch based on the public feature embedding space to obtain a model retrieval result corresponding to a three-dimensional model. The method can effectively, accurately and quickly identify the types of three-dimensional objects with various line-of-sight directions and has generalization and can be applied to the type identification of different types of three-dimensional objects.

Description

Technical Field

[0001] This invention relates to the field of 3D model retrieval technology, and in particular to a sketch-based 3D model retrieval method, apparatus, device, and medium. Background Technology

[0002] In recent years, with the rapid development of 3D printing and computer technologies, the amount of 3D model data has exploded. 3D models have been applied to industries such as autonomous driving, medical biology, industrial production, and gaming. The vast number of 3D models provides designers with more choices, but it also brings another challenge: how to find the target model of interest from the enormous 3D model library—that is, 3D model retrieval.

[0003] 3D model retrieval using sketches as input is a flexible and convenient method. Sketch-based 3D model retrieval is essentially a similarity comparison problem between sketches and 3D models. Sketches are highly abstract; they are only a rough description of a 3D model from one viewpoint.

[0004] Traditional methods utilize manually designed coding algorithms to extract view feature descriptors from sketches and the projected 3D models, then perform similarity comparisons to achieve retrieval. These traditional methods are further divided into global feature-based and local feature-based methods. Global feature-based methods calculate features based on the overall shape of the sketch. Local feature-based methods use the calculation of local features to represent the overall global features. Traditional sketch-based 3D model retrieval methods have limited application scope and suffer from low extraction efficiency, low versatility, and low retrieval accuracy. Summary of the Invention

[0005] This invention provides a sketch-based 3D model retrieval method, apparatus, device, and medium to address the problems of limited application scope, low extraction efficiency, low versatility, and low retrieval accuracy of traditional sketch-based 3D model retrieval methods.

[0006] A sketch-based 3D model retrieval method, comprising:

[0007] Obtain a training sketch of a 3D object. Based on the viewing direction of the training sketch, project the training sketch using a preset 3D model viewpoint group to obtain a 2D sketch view group.

[0008] An enhanced sketch set for a two-dimensional sketch view group is obtained using edge detection and sample augmentation algorithms.

[0009] Extract sample features corresponding to the enhanced sketch set to generate a sample feature set. Based on the sample feature set, obtain the common feature embedding space of the same type of image corresponding to the 3D object, which is distributed in a cluster.

[0010] Obtain the sketch of the 3D model to be searched, and search the sketch based on the common feature embedding space to obtain the model search results corresponding to the 3D model.

[0011] A sketch-based 3D model retrieval device, comprising:

[0012] The module for acquiring training sketches is used to acquire training sketches of 3D objects. Based on the viewing direction of the training sketches, a preset 3D model viewpoint group is used to project the training sketches to acquire a 2D sketch view group.

[0013] The module for acquiring enhanced sketch sets is used to acquire enhanced sketch sets of two-dimensional sketch view groups using edge detection algorithms and sample augmentation algorithms.

[0014] The module for obtaining the common feature embedding space is used to extract sample features corresponding to the enhanced sketch set to generate a sample feature set. Based on the sample feature set, the common feature embedding space of the same type of image corresponding to the 3D object is obtained in a clustered distribution.

[0015] The module for obtaining model retrieval results is used to obtain the sketch of the 3D model to be retrieved, and to search the sketch based on the common feature embedding space to obtain the model retrieval results corresponding to the 3D model.

[0016] An apparatus includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the sketch-based 3D model retrieval method described above.

[0017] A computer-readable medium storing a computer program that, when executed by a processor, implements the above-described sketch-based 3D model retrieval method.

[0018] The aforementioned sketch-based 3D model retrieval method, apparatus, device, and medium project training sketches into a group of 2D sketch views along their viewing direction. This is used to train common features of images of the same type corresponding to 3D objects to be clustered and embedded in space. This is used to identify different types of 3D objects across spatial domains. It can effectively, accurately, and quickly identify the types of 3D objects in various viewing directions and has generalization properties that can be applied to the identification of different types of 3D objects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the application environment of a sketch-based 3D model retrieval method according to an embodiment of the present invention.

[0021] Figure 2 This is a flowchart of a sketch-based 3D model retrieval method according to an embodiment of the present invention;

[0022] Figure 3 This is a first flowchart of a sketch-based 3D model retrieval method according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of a 3D model viewpoint group in a sketch-based 3D model retrieval method according to an embodiment of the present invention;

[0024] Figure 5 This is a second flowchart of a sketch-based 3D model retrieval method in one embodiment of the present invention;

[0025] Figure 6 This is the third flowchart of a sketch-based 3D model retrieval method in one embodiment of the present invention;

[0026] Figure 7 This is the fourth flowchart of a sketch-based 3D model retrieval method in one embodiment of the present invention;

[0027] Figure 8 This is a comparative schematic diagram of different levels of training sketches, two-dimensional views, and multi-level pseudo-sketches in a sketch-based 3D model retrieval method according to an embodiment of the present invention.

[0028] Figure 9 This is the fifth flowchart of a sketch-based 3D model retrieval method in one embodiment of the present invention;

[0029] Figure 10 A sixth flowchart of a sketch-based 3D model retrieval method in one embodiment of the present invention;

[0030] Figure 11 This is a schematic diagram of the ternary network model of a sketch-based 3D model retrieval method in one embodiment of the present invention;

[0031] Figure 12 This is a schematic diagram of the overall process of a sketch-based 3D model retrieval method in one embodiment of the present invention;

[0032] Figure 13 This is a schematic diagram of a sketch-based 3D model retrieval device according to an embodiment of the present invention;

[0033] Figure 14 This is a schematic diagram of the device in one embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] The sketch-based 3D model retrieval method provided in this invention can be applied to, for example... Figure 1 In this application environment, the sketch-based 3D model retrieval method is applied to a sketch-based 3D model retrieval system. This system includes a client and a server, where the client communicates with the server via a network. The client, also known as the user terminal, is a program that provides local services to the client, corresponding to the server. This client can be installed on various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0036] In one embodiment, such as Figure 2 As shown, a sketch-based 3D model retrieval method is provided, which can be applied to... Figure 1 Taking the server in the example, the specific steps are as follows:

[0037] S10. Obtain a training sketch of the 3D object. Based on the viewing direction of the training sketch, project the training sketch using a preset 3D model viewpoint group to obtain a 2D sketch view group.

[0038] In this context, the training sketch for a 3D object is a drawing that represents the initial design or shape of a known object type. The sketch in this embodiment can be either 2D or 3D. When creating a 2D sketch, the plane on which the sketch is attached must first be determined, i.e., the coordinate plane defined by the sketch coordinate system. Once the coordinate plane of the sketch is determined, its viewing direction relative to the unit cube can be determined. It can be understood that a unit cube has six faces, which means it has six viewing directions: up, down, left, right, front, and back.

[0039] A 3D model viewpoint group is a set of at least three viewpoints (one main viewpoint and two auxiliary viewpoints symmetrical to the main viewpoint) that project the training sketch into a 2D sketch view for each viewing direction. In other words, the 2D sketch view is the 2D sketch formed by projecting the training sketch through the 3D model viewpoint group. The 2D sketch corresponding to each viewpoint in a single direction constitutes the 2D sketch view group corresponding to the training sketch in that direction.

[0040] Specifically, a unit cube can be circumscribed on a sphere, and multiple points on each face of the unit cube can be projected onto the sphere to form at least three viewpoints of that face (the direction of the line of sight) relative to the sphere.

[0041] The training sketches drawn can obtain their viewing direction relative to the unit cube through the coordinate plane determined by the sketch coordinate system, and be projected based on the three-dimensional model viewpoint group corresponding to the viewing direction to form at least three two-dimensional sketch views corresponding to a single direction.

[0042] In step S10, the training sketch can obtain a two-dimensional sketch view in the accurate direction through the three-dimensional model viewpoint group, which is beneficial for subsequent steps to efficiently process the two-dimensional sketch view and reduce computational complexity.

[0043] S20. Use edge detection algorithm and sample augmentation algorithm to obtain the augmented sketch set of two-dimensional sketch view group.

[0044] Among them, the edge detection algorithm refers to the method of segmenting the target region based on the discontinuous property of gray values ​​and the abrupt change of gray values, which is used to extract image features. In this embodiment, the Canny operator, Marr-Hildreth operator, etc. can be used, but no specific limitation is made here.

[0045] Sample augmentation algorithms, also known as sample expansion algorithms, are used to make limited data produce value equivalent to more data without substantially increasing the amount of data. For example, for single-sample data augmentation, there are geometric transformations, color transformations, etc., which are not specifically limited here.

[0046] S30. Extract the sample features corresponding to the enhanced sketch set to generate a sample feature set. Based on the sample feature set, obtain the common feature embedding space of the features of the same type of image corresponding to the 3D object in a clustered distribution.

[0047] In this embodiment, the sample features are the image drawings, which are mainly divided into four categories: intuitive features, gray-scale statistical features, transformation coefficient features, and algebraic features.

[0048] The common feature embedding space is an abstract concept that stores feature vectors corresponding to object types in all images. The common feature embedding space can be constrained by a loss function, bringing feature vectors closer together from images of the same type and distancing them from images of different types, ultimately achieving a clustered distribution of images of the same type. The loss function evaluates the degree to which the model's predictions differ from the true values; a better loss function generally indicates better model performance.

[0049] Specifically, in order to fully reflect the type characteristics of three-dimensional objects, this embodiment can extract as many sample features as possible corresponding to the enhanced sketch set, including standard type features, features of objects of the same type, and features of objects far from the standard type, and save the aforementioned extracted sample features into the public feature embedding space through the type index mode.

[0050] S40. Obtain the sketch of the 3D model to be retrieved, and perform a search on the sketch based on the common feature embedding space to obtain the model retrieval results corresponding to the 3D model.

[0051] Specifically, by extracting features from the 3D model sketch to be retrieved and matching them with those stored in the common feature embedding space, the model retrieval results corresponding to the 3D model can be retrieved quickly and effectively. During the retrieval, the 3D model sketch to be retrieved is used as input, and features are extracted using CNN (Convolutional Neural Networks). Similarity evaluation is performed in the common feature embedding space, and finally, the 3D model most similar to the sketch to be retrieved is retrieved.

[0052] The sketch-based 3D model retrieval method provided in this embodiment projects training sketches into a group of 2D sketch views along their viewing direction. This is used to train common features embedded in a space where the features of images of the same type corresponding to 3D objects are clustered together. This is used to identify different types of 3D objects across different spatial domains. It can effectively, accurately, and quickly identify the types of 3D objects in various viewing directions and has generalization properties that can be applied to the identification of different types of 3D objects.

[0053] In one specific embodiment, such as Figure 3 As shown, before step S10, that is, before obtaining the training sketch of the 3D model, the following steps are also included:

[0054] S101. Obtain a unit cube inscribed in a sphere, where the six view faces of the unit cube correspond to a line of sight.

[0055] Among them, the unit cube is a special type of cube, which is a cube with each edge having a length of 1 unit and a total surface area of ​​6 square units. The view is the six faces of the unit cube.

[0056] The direction of sight is the direction of sight corresponding to the viewing plane, including six directions: up, down, left, right, front, and back.

[0057] S102. Project the center of each viewing plane onto the surface of the sphere in the corresponding line of sight to obtain the main viewpoint of the center of the viewing plane on the surface of the sphere.

[0058] The main viewpoint is the projection point of the center of the viewing plane onto the sphere in the direction of the line of sight.

[0059] Step S102 confirms the position of the main viewpoint on the circumsphere of the unit cube, facilitating subsequent confirmation of the corresponding main viewpoint of the 3D model based on the rotation direction of the 3D object on the sphere, thereby confirming the viewing direction of the 3D model. Specifically, different types of sketches will have biases when drawing; for example, airplanes tend to be drawn from the side rather than the front, and ants tend to be drawn from the top rather than the side. The viewing direction of the 3D model can be selected and confirmed based on the characteristics of different types of sketches.

[0060] S103. On a sphere opposite to the viewing plane, at least one pair of mutually symmetrical auxiliary viewpoints are set around the main viewpoint, and the main viewpoint and auxiliary viewpoints constitute a unidirectional viewpoint group in a single line of sight.

[0061] Specifically, to increase data diversity and accurately locate the viewpoint orientation of the sketch, thereby enhancing the robustness of the Triplet model, this embodiment sets up four pairs of mutually symmetrical auxiliary viewpoints around the main viewpoint. Eight auxiliary viewpoints are set above, below, left, right, upper left and lower right, and lower left and upper right of each main viewpoint, resulting in a total of nine viewpoints per viewpoint group (main viewpoint plus eight auxiliary viewpoints), for a total of 54 viewpoints. The viewpoint group setup for the 3D model is as follows: Figure 4 As shown.

[0062] S104. Combine all unidirectional viewpoint groups to form a 3D model viewpoint group.

[0063] Specifically, this embodiment can generate a group of three-dimensional model viewpoints including six viewing directions.

[0064] In steps S101 to S104, this embodiment can form a three-dimensional model viewpoint group to increase the diversity of data and accurately locate the viewpoint direction of the sketch.

[0065] In one specific embodiment, the three-dimensional model viewpoint group includes at least one unidirectional viewpoint group in the viewing direction, and the unidirectional viewpoint group includes at least three viewpoint positions. For example... Figure 5 As shown, in step S10, the training sketch is projected onto the training sketch using a preset three-dimensional model viewpoint group based on the viewing direction of the training sketch to obtain a two-dimensional sketch view group. Specifically, the steps are as follows: S11. Based on the viewing direction corresponding to the training sketch, the corresponding unidirectional viewpoint group is matched from the three-dimensional model viewpoint group.

[0066] Specifically, in this embodiment, the main viewpoint on the corresponding sphere can be determined by the center of the three-dimensional object based on the rotation direction of the circumscribed sphere, thereby obtaining the unidirectional viewpoint group in the corresponding three-dimensional model viewpoint group through the main viewpoint.

[0067] S12. Based on a unidirectional viewpoint group, project at least three viewpoint positions of the training sketch in a single line of sight to obtain a 2D sketch view group corresponding to the training sketch.

[0068] Specifically, this embodiment continues to illustrate the concept based on nine viewpoints in each viewing direction. Based on the sketch viewpoint directions, two-dimensional sketches are generated in projection form of the original sketch, with slight differences in direction and position across the nine minor directions. These nine two-dimensional sketches form a group of two-dimensional sketch views corresponding to the training sketch.

[0069] In steps S11 to S12, this embodiment can generate a group of two-dimensional sketch views corresponding to the training sketch, so as to project the three-dimensional object into multiple two-dimensional sketches generated from a training sketch in the correct viewing direction, thereby enhancing the recognition robustness.

[0070] In one specific embodiment, such as Figure 6 As shown, before step S20, that is, before obtaining the enhanced sketch set of the two-dimensional sketch view group using the edge detection algorithm and the sample augmentation algorithm, the following steps are also specifically included:

[0071] S201. Using the two-dimensional sketch view group as the standard sample set, obtain a positive sample set that is in the same category as the three-dimensional object and a negative sample set that is not in the same category as the three-dimensional object.

[0072] S202. Combine the standard sample set, the positive sample set, and the negative sample set to generate the first augmented sample set corresponding to the training sketch.

[0073] Specifically, the number of samples in the first augmented sample set (C t The number of positive samples (C) p ), the number of standard sample sets (C) a ) and the number of negative samples (C) n The maximum number of unique first augmented sample sets (consisting of triples) that can be generated is determined to be:

[0074] C t =C a ×C p ×C

[0075] The standard sample set is a set of sketches of the same category as the training sketches, the positive sample set is a set of pseudo-sketches of the same category as the standard samples, and the negative sample set is a set of pseudo-sketches of a different category than the standard samples.

[0076] In steps S201 to S202, this embodiment can combine the positive and negative pseudo-sketch sets with the first enhanced sample set of the standard sample set, which is beneficial for effectively highlighting the concentrated features close to the sample set in the subsequent process, while also possessing the features of being far away from the standard sample set.

[0077] In one specific embodiment, the first augmented sample set includes at least three augmented sample images. For example... Figure 7 As shown, in step S20, the enhanced sketch set of the two-dimensional sketch view group is obtained by using edge detection algorithm and sample augmentation algorithm, which specifically includes the following steps:

[0078] S211. An edge detection algorithm is used to generate a multi-level pseudo-image group corresponding to each enhanced sample image in the first enhanced sample set. The multi-level pseudo-image group includes at least two two-dimensional pseudo-images.

[0079] In this context, pseudo-sketches are obtained by adjusting the threshold in the edge detection algorithm to obtain two-dimensional pseudo-images with different thresholds. Multi-level pseudo-sketches are sketches obtained by combining two-dimensional pseudo-images with several thresholds selected from multiple two-dimensional pseudo-images and participating in subsequent training.

[0080] Specifically, this embodiment takes the Canny operator as an example, and the process of obtaining the view of the enhanced sketch set of the two-dimensional sketch view group is as follows:

[0081] The goal of the Canny operator is to find an optimal edge, defined as follows: 1. Good detection – the algorithm can identify as many actual edges in the image as possible; 2. Good localization – the identified edge should be as close as possible to the actual edge in the image; and 3. Minimal response – the edge in the image should only be identified once, and any image noise should not be identified as an edge.

[0082] The specific implementation steps of the Canny algorithm are as follows:

[0083] (1). Gaussian filtering: The gray values ​​of the pixels and their neighbors in the enhanced sample image to be filtered are weighted and averaged according to the parameter rules generated by the Gaussian formula, which can effectively filter out the high-frequency noise superimposed in the image.

[0084] (2) Calculating the gradient image and angle image: The gradient is calculated using a Gaussian filter, resulting in a Sobel-like effect, where pixels closer to the center point have higher weights. Calculating the angle image is simpler; its function is to guide the direction of non-maximum suppression.

[0085] (3). Non-maximum suppression of gradient image: The gradient image obtained in the previous step has problems such as thick and wide edges and weak edge interference. Non-maximum suppression is used to find the local maximum value of the pixel and set the gray value corresponding to the non-maximum value to 0, which can eliminate most of the non-edge pixels.

[0086] (4) Edge Connectivity Using Dual Thresholds: After the above three steps, the resulting image has high edge quality, but many pseudo-edges still exist. Therefore, the Canny algorithm uses a dual threshold method, specifically selecting two thresholds: a low threshold and a high threshold, to filter edge pixels. Taking training sketches of a brain and an airplane as examples, the high threshold is set to 50. When the low threshold is set to different values, the comparison images of different levels of training sketches, 2D views, and multi-level pseudo-sketches are as follows: Figure 8 As shown.

[0087] In step S212, the multi-level pseudo-image group generated by the edge detection algorithm not only solves the problem of excessive difference between pseudo-sketches caused by the difference in detail information between different sketches, but also increases data diversity, which can effectively improve the retrieval accuracy.

[0088] S212. Combine all multi-level pseudo-graph groups to generate a second enhanced sample set.

[0089] S213. The sample augmentation algorithm is used to expand the second augmented sample set to obtain the augmented sketch set corresponding to the two-dimensional sketch view group.

[0090] Among them, sample augmentation algorithms are used when photos taken due to environmental factors such as lighting have low clarity and contrast, failing to highlight key elements in the image. These algorithms enhance image details or increase the number of images. The sample augmentation process improves image contrast, making people or objects more prominent, which is beneficial for subsequent image recognition and other processing.

[0091] In the data equalization process involved in steps S211 to S213, since the training sketch and the augmented sample image still belong to different domain spaces, this embodiment can extract multi-level pseudo-sketches for each augmented sample image through an edge detection algorithm. This not only solves the problem of losing view detail information in a single pseudo-sketch but also effectively reduces domain differences. Here, "domain space" is an abstract term. The augmented sketch is a real image obtained from a 3D model; its shape, position, and size are directly obtained from the 3D model corresponding to the view domain. The training sketch, on the other hand, is drawn manually based on the sketch domain. Sketches drawn by different people, or even by the same person for the same object each time, are different, containing different amounts of information, and varying in size and shape.

[0092] In one specific embodiment, such as Figure 9 As shown, in step S20, the edge detection algorithm and the sample augmentation algorithm obtain the augmented sketch set of the two-dimensional sketch view group, which specifically includes the following steps:

[0093] S221. Use an edge detection algorithm to obtain the second enhanced sample set corresponding to the two-dimensional sketch view group.

[0094] S222. The sample augmentation algorithm is used to expand the second augmented sample set to obtain the augmented sketch set corresponding to the two-dimensional sketch view group. The sample augmentation algorithm includes rotating and / or cropping the second augmented sample set, wherein the rotation angle is 0°-40°.

[0095] Specifically, rotation does not change the image size, while cropping does. Typically, random cropping is used during training, and during testing, the middle portion is cropped or no cropping is performed. It's important to note that cropping multiple versions of the input and then merging the results significantly improves prediction accuracy. Neither rotation nor cropping causes image distortion.

[0096] In steps S212 to S222, this embodiment can obtain an enhanced sketch set containing more images corresponding to the two-dimensional sketch view group by rotation and cropping, which can be used to improve the robustness of the network during training.

[0097] In one specific embodiment, the sample feature set includes a standard sample feature set, a positive sample feature set, and a negative sample feature set. For example... Figure 10 As shown, in step S30, based on the sample feature set, the common feature embedding space of the images of the same type corresponding to the 3D object is obtained in a clustered distribution, which specifically includes the following steps:

[0098] S31. Train a shared weighted triple network model based on the standard sample feature set, positive sample feature set, and negative sample feature set respectively, and obtain the triple network retrieval results corresponding to the training sketch.

[0099] In this embodiment, a Triplet network can be used as a triplet network model. Triplet loss is a loss function in deep learning within a Triplet network, primarily used for training samples with low variance. Furthermore, triplet loss is frequently used in tasks where the training objective is to obtain sample embeddings, such as text and image embeddings.

[0100] Specifically, firstly, this embodiment uses the Sketch-A-Net (hand-drawn image recognition) network to extract image features from the standard sample feature set, positive sample feature set, and negative sample feature set, respectively, for subsequent learning of a feature map. Then, a Triplet network is used, with three parameters input to the triplet loss formula: anchor point a, positive example p, and negative example n, corresponding to the aforementioned standard sample set, positive sample set, and negative sample set in this embodiment, respectively. The goal of Triplet loss is to make samples with the same label as close as possible in the embedding space, and to make samples with different labels as far apart as possible in the embedding space. Following these two points, samples of the same category in the embedding space will eventually cluster into a very small group, meaning the distance between samples within the same category cluster is very small.

[0101] Furthermore, the feature embedding network of the Triplet network, such as Figure 11 As shown in the figure, the three input branches correspond to three CNN networks with shared weights for feature extraction.

[0102] S32. A sketch index is established using the ternary network retrieval results, which maps the sample feature set to the common feature embedding space to constrain the distribution of the standard sample feature set and the positive sample feature set to be clustered.

[0103] The sketch index refers to creating an index based on the sketch category and view location information. The trained network then establishes a correspondence between the retrieval similarity of each sketch category and the view. The input to the ternary network model is the training sketch, which is mapped using Triplet loss to output the retrieval results of the training sketch, mapped into a common feature embedding space.

[0104] In steps S31 to S32, this embodiment can reduce the difference between training sketches and pseudo-sketches through the Triplet network, thereby improving the accuracy of retrieval.

[0105] The sketch-based 3D model retrieval method provided in this embodiment projects training sketches into a group of 2D sketch views along their viewing direction. This is used to train common features embedded in a space where the features of images of the same type corresponding to 3D objects are clustered together. This is used to identify different types of 3D objects across different spatial domains. It can effectively, accurately, and quickly identify the types of 3D objects in various viewing directions and has generalization properties that can be applied to the identification of different types of 3D objects.

[0106] In one specific embodiment, the overall process of the sketch-based 3D model retrieval method is as follows: Figure 12 As shown, the specific steps include the following:

[0107] S1, Data Alignment:

[0108] A model sketch----》

[0109] Nine 2D viewpoint projection views----》

[0110] S2, Data Balance:

[0111] Each 2D viewpoint projection sketch generates N multi-level pseudo-sketches (up to this point, 9*N pseudo-sketches have been generated) ----> Data augmentation is performed on these 9*N pseudo-sketches to obtain at least 225×225 augmented images as triplet augmented data (augmented data includes three types: positive sample set, standard sample set, and negative sample set) ---->

[0112] S3, Triple Pair Training:

[0113] Extracting features from triplet augmented data ---->

[0114] The output value is obtained by training a Triplet network with features from triplet augmented data.

[0115] The output values ​​obtained from the Triplet network are used as indices and then mapped to the common feature embedding space.

[0116] S4, Sketch Retrieval.

[0117] Furthermore, the sketch-based 3D model retrieval method provided in this embodiment can also construct a 3D model viewpoint group to increase data diversity and accurately locate the viewpoint direction of the sketch; generate a 2D sketch view group corresponding to the training sketch to project the 3D object as multiple 2D sketches generated from a single training sketch in the correct viewing direction, enhancing recognition robustness; by combining the positive and negative pseudo-sketch sets with the first enhanced sample set of the standard sample set, it is beneficial to effectively highlight the concentrated features of the sample set that are close to it, while also possessing features that are far away from the standard samples; by extracting multi-level pseudo-sketches for each enhanced sample image through an edge detection algorithm, it can not only solve the problem of losing view detail information in a single pseudo-sketch, but also effectively reduce domain differences; by rotating and cropping, an enhanced sketch set containing more images corresponding to the 2D sketch view group can be obtained, which can be used to improve the robustness of the network during training; by using a Triplet network, the difference between the training sketch and the pseudo-sketch is reduced, improving the retrieval accuracy.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0119] In one embodiment, a sketch-based 3D model retrieval device is provided, which corresponds one-to-one with the sketch-based 3D model retrieval method described in the above embodiments. For example... Figure 13 As shown, the sketch-based 3D model retrieval device includes a training sketch acquisition module 10, an enhanced sketch set acquisition module 20, a common feature embedding space acquisition module 30, and a model retrieval result acquisition module 40. Detailed descriptions of each functional module are as follows:

[0120] The training sketch acquisition module 10 is used to acquire training sketches of three-dimensional objects. Based on the viewing direction of the training sketches, a preset three-dimensional model viewpoint group is used to project the training sketches to acquire a two-dimensional sketch view group.

[0121] The module 20 for acquiring an enhanced sketch set is used to acquire an enhanced sketch set of a two-dimensional sketch view group using an edge detection algorithm and a sample augmentation algorithm.

[0122] The common feature embedding space module 30 is used to extract sample features corresponding to the enhanced sketch set to generate a sample feature set. Based on the sample feature set, the common feature embedding space of the same type of image corresponding to the 3D object is obtained in a clustered distribution.

[0123] The module 40 for obtaining model retrieval results is used to obtain the sketch of the three-dimensional model to be retrieved, and to retrieve the sketch based on the common feature embedding space to obtain the model retrieval results corresponding to the three-dimensional model.

[0124] Preferably, the sketch-based 3D model retrieval device further includes a module for acquiring a unit cube, a module for acquiring a main viewpoint, a module for setting auxiliary viewpoints, and a module for constructing a 3D viewpoint group. Detailed descriptions of each functional module are as follows:

[0125] The module for obtaining a unit cube is used to obtain a unit cube inscribed in a sphere. The six view faces of the unit cube correspond to a line of sight.

[0126] The main viewpoint acquisition module is used to project the center of each view plane onto the surface of a sphere in the corresponding line of sight, and to obtain the main viewpoint where the center of the view plane is on the surface of the sphere.

[0127] The auxiliary viewpoint module is used to set at least one pair of mutually symmetrical auxiliary viewpoints around the main viewpoint on a sphere opposite to the viewing plane. The main viewpoint and the auxiliary viewpoints constitute a unidirectional viewpoint group in a single line of sight.

[0128] The module for constructing a 3D viewpoint group is used to combine all unidirectional viewpoint groups to form a 3D model viewpoint group.

[0129] Preferably, the training sketch acquisition module 10 includes a unidirectional viewpoint matching submodule and a 2D sketch view acquisition submodule. Detailed descriptions of each functional module are as follows:

[0130] The matching one-way viewpoint group submodule is used to match the corresponding one-way viewpoint group from the 3D model viewpoint group based on the line of sight corresponding to the training sketch.

[0131] The submodule for obtaining a 2D sketch view group is used to project at least three viewpoint positions of the training sketch in a single line of sight based on a unidirectional viewpoint group, so as to obtain the 2D sketch view group corresponding to the training sketch.

[0132] Preferably, the sketch-based 3D model retrieval device further includes a sample set acquisition module and a first sample set generation module. Detailed descriptions of each functional module are as follows:

[0133] The sample set acquisition module is used to take the 2D sketch view group as the standard sample set, and acquire a positive sample set that is in the same category as the 3D object and a negative sample set that is not in the same category as the 3D object.

[0134] The module for generating the first sample set is used to combine the standard sample set, the positive sample set, and the negative sample set to generate the first augmented sample set corresponding to the training sketch.

[0135] Preferably, the enhanced sketch set acquisition module 20 includes a multi-level pseudo-map group generation submodule, a second sample set generation submodule, and an enhanced sketch set acquisition submodule. Detailed descriptions of each functional module are as follows:

[0136] The multi-level pseudo-image group generation submodule is used to generate a multi-level pseudo-image group corresponding to each enhanced sample image in the first enhanced sample set using an edge detection algorithm. The multi-level pseudo-image group includes at least two two-dimensional pseudo-images.

[0137] A second sample set generation submodule is used to combine all multi-level pseudo-map groups to generate a second enhanced sample set;

[0138] The module for obtaining the enhanced sketch set is used to expand the second enhanced sample set using a sample enhancement algorithm to obtain the enhanced sketch set corresponding to the two-dimensional sketch view group.

[0139] Preferably, the module 20 for acquiring the enhanced sketch set includes a sub-module for acquiring the enhanced sample set and a sub-module for expanding the enhanced samples. Detailed descriptions of each functional module are as follows:

[0140] The module for obtaining the enhanced sample set is used to obtain the second enhanced sample set corresponding to the two-dimensional sketch view group using an edge detection algorithm. The module for expanding the enhanced sample set is used to expand the second enhanced sample set using a sample enhancement algorithm to obtain the enhanced sketch set corresponding to the two-dimensional sketch view group. The sample enhancement algorithm includes rotating and / or cropping the second enhanced sample set, wherein the rotation angle is 0°-40°.

[0141] Preferably, the common feature embedding space acquisition module 30 includes a retrieval result acquisition submodule and a mapping ternary network retrieval result submodule. Detailed descriptions of each functional module are as follows:

[0142] The retrieval results submodule is used to train a shared-weights ternary network model based on the standard sample feature set, the positive sample feature set, and the negative sample feature set, and to obtain the retrieval results of the ternary network corresponding to the training sketch.

[0143] The submodule for mapping ternary network retrieval results is used to build a sketch index through ternary network retrieval results, mapping the sample feature set to the common feature embedding space, and constraining the distribution of the standard sample feature set and the positive sample feature set to be clustered.

[0144] Specific limitations regarding the sketch-based 3D model retrieval device can be found in the limitations of the sketch-based 3D model retrieval method described above, and will not be repeated here. Each module in the sketch-based 3D model retrieval device described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the device in hardware form or independent of it, or stored in the memory of the device in software form, so that the processor can call and execute the operations corresponding to each module.

[0145] In one embodiment, a device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, the device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile media and internal memory. The non-volatile media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs on the non-volatile media. The database contains data related to a sketch-based 3D model retrieval method. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a sketch-based 3D model retrieval method.

[0146] In one embodiment, a device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the sketch-based 3D model retrieval method of the above embodiments, for example... Figure 2 Steps S10 to S40 are shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit of the sketch-based 3D model retrieval device in the above embodiments, for example... Figure 13 The functions of modules 10 to 40 are shown. To avoid repetition, they will not be described again here.

[0147] In one embodiment, a computer-readable medium is provided having a computer program stored thereon. When executed by a processor, the computer program implements the sketch-based 3D model retrieval method of the above embodiments, for example... Figure 2 Steps S10 to S40 are shown. Alternatively, when the computer program is executed by a processor, it implements the functions of each module / unit in the sketch-based 3D model retrieval device in the above-described device embodiment, for example... Figure 13 The functions of modules 10 to 40 are shown. To avoid repetition, they will not be described again here.

[0148] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments of this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A sketch-based 3D model retrieval method, characterized in that, include: Obtain a unit cube inscribed in a sphere, wherein the six view faces of the unit cube each correspond to a line of sight. Project the center of each viewing surface onto the surface of the sphere in the corresponding line of sight to obtain the main viewing point of the center of the viewing surface on the surface of the sphere; On a sphere opposite to the viewing plane, four pairs of mutually symmetrical auxiliary viewpoints are arranged around the main viewpoint, and the main viewpoint and the auxiliary viewpoints constitute a unidirectional viewpoint group in a single line of sight. The three-dimensional model viewpoint group is formed by combining all the aforementioned unidirectional viewpoint groups; A training sketch of a three-dimensional object is obtained. Based on the viewing direction of the training sketch, a preset three-dimensional model viewpoint group is used to project the training sketch to obtain a two-dimensional sketch view group. The three-dimensional model viewpoint group includes six unidirectional viewpoint groups in the viewing direction, and the unidirectional viewpoint group includes nine viewpoint positions. An enhanced sketch set for the two-dimensional sketch view group is obtained using edge detection and sample augmentation algorithms. Extract the sample features corresponding to the enhanced sketch set to generate a sample feature set. Based on the sample feature set, obtain the common feature embedding space in which the features of the same type of image corresponding to the 3D object are clustered and distributed. Obtain the sketch of the three-dimensional model to be retrieved, and perform a retrieval on the sketch based on the common feature embedding space to obtain the model retrieval results corresponding to the three-dimensional model; The sample feature set includes a standard sample feature set, a positive sample feature set, and a negative sample feature set; The step of obtaining a common feature embedding space based on the sample feature set, where the features of similar images corresponding to the 3D object are clustered, includes: Based on the standard sample feature set, positive sample feature set, and negative sample feature set, train a shared weighted triple network model to obtain the triple network retrieval result corresponding to the training sketch; A sketch index is established using the ternary network retrieval results, and the sample feature set is mapped to the common feature embedding space to constrain the distribution of the standard sample feature set and the positive sample feature set to be clustered. The ternary network model mentioned above uses a Triplet network. The training sketch is a drawing that represents the initial design or shape of a known type of object.

2. The sketch-based 3D model retrieval method according to claim 1, characterized in that, The training sketch is projected onto a pre-defined set of 3D model viewpoints based on the viewing direction to obtain a 2D sketch view set, including: Based on the viewing direction corresponding to the training sketch, a corresponding unidirectional viewpoint group is matched from the viewpoint group of the 3D model; Based on the unidirectional viewpoint group, the eight viewpoint positions of the training sketch in a single line of sight are projected to obtain the two-dimensional sketch view group corresponding to the training sketch.

3. The sketch-based 3D model retrieval method according to claim 1, characterized in that, Before obtaining the enhanced sketch set of the two-dimensional sketch view group using the edge detection algorithm and the sample augmentation algorithm, the method further includes: Using the two-dimensional sketch view group as a standard sample set, obtain a positive sample set that is of the same category as the three-dimensional object and a negative sample set that is of a different category from the three-dimensional object; The first augmented sample set corresponding to the training sketch is generated by combining the standard sample set, the positive sample set, and the negative sample set.

4. The sketch-based 3D model retrieval method according to claim 3, characterized in that, The first augmented sample set includes at least three augmented sample images; The process of obtaining the enhanced sketch set of the two-dimensional sketch view group using edge detection and sample augmentation algorithms includes: An edge detection algorithm is used to generate a multi-level pseudo-image group corresponding to each of the enhanced sample images in the first enhanced sample set, and the multi-level pseudo-image group includes at least two two-dimensional pseudo-images. A second enhanced sample set is generated by combining all the aforementioned multi-level pseudo-map sets; The second enhanced sample set is expanded using a sample augmentation algorithm to obtain the enhanced sketch set corresponding to the two-dimensional sketch view group.

5. The sketch-based 3D model retrieval method according to claim 1, characterized in that, The process of obtaining the enhanced sketch set of the two-dimensional sketch view group using edge detection and sample augmentation algorithms includes: An edge detection algorithm is used to obtain the second enhanced sample set corresponding to the two-dimensional sketch view group; The sample augmentation algorithm is used to expand the second augmented sample set to obtain the augmented sketch set corresponding to the two-dimensional sketch view group. The sample augmentation algorithm includes rotating and / or cropping the second augmented sample set, wherein the rotation angle is 0°-40°.

6. A sketch-based 3D model retrieval device, characterized in that, include: The training sketch acquisition module is used to acquire training sketches of three-dimensional objects. Based on the viewing direction of the training sketches, the training sketches are projected using a preset three-dimensional model viewpoint group to acquire a two-dimensional sketch view group. The three-dimensional model viewpoint group includes six unidirectional viewpoint groups in the viewing direction, and the unidirectional viewpoint group includes nine viewpoint positions. An enhanced sketch set acquisition module is used to acquire an enhanced sketch set of the two-dimensional sketch view group using an edge detection algorithm and a sample augmentation algorithm. A common feature embedding space module is used to extract sample features corresponding to the enhanced sketch set to generate a sample feature set. Based on the sample feature set, a common feature embedding space is obtained where the features of the same type of image corresponding to the 3D object are clustered. The module for obtaining model retrieval results is used to obtain the sketch of the three-dimensional model to be retrieved, and to retrieve the sketch based on the common feature embedding space to obtain the model retrieval results corresponding to the three-dimensional model. Before obtaining the training sketch of the 3D object, the method further includes: Obtain a unit cube inscribed in a sphere, wherein the six view faces of the unit cube each correspond to a line of sight. Project the center of each viewing surface onto the surface of the sphere in the corresponding line of sight to obtain the main viewing point of the center of the viewing surface on the surface of the sphere; On a sphere opposite to the viewing plane, four pairs of mutually symmetrical auxiliary viewpoints are arranged around the main viewpoint, and the main viewpoint and the auxiliary viewpoints constitute a unidirectional viewpoint group in a single line of sight. The three-dimensional model viewpoint group is formed by combining all the aforementioned unidirectional viewpoint groups; The sample feature set includes a standard sample feature set, a positive sample feature set, and a negative sample feature set; The step of obtaining a common feature embedding space based on the sample feature set, where the features of similar images corresponding to the 3D object are clustered, includes: Based on the standard sample feature set, positive sample feature set, and negative sample feature set, train a shared weighted triple network model to obtain the triple network retrieval result corresponding to the training sketch; A sketch index is established using the ternary network retrieval results, and the sample feature set is mapped to the common feature embedding space to constrain the distribution of the standard sample feature set and the positive sample feature set to be clustered. The ternary network model mentioned above uses a Triplet network. The training sketch is a drawing that represents the initial design or shape of a known type of object.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the sketch-based 3D model retrieval method as described in any one of claims 1 to 5.

8. A computer-readable medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the sketch-based 3D model retrieval method as described in any one of claims 1 to 5.

Images