Cross-modal retrieval method and device for retrieving three-dimensional models based on sketches and medium

Abstract

The application provides a cross-modal retrieval method and device for retrieving three-dimensional models based on sketches, and a medium. The method comprises the following steps: constructing a shared data space based on sample sketches and sample three-dimensional CAD models, rendering the sample three-dimensional CAD models into a plurality of sample two-dimensional views of different perspectives, and representing the three-dimensional CAD models by the two-dimensional views of the plurality of perspectives; constructing a deep learning model based on two ResNet networks sharing weights, and then training based on the shared data space; embedding the features of the sketches and the three-dimensional CAD models into a unified semantic space, and reducing the difference between the modes; obtaining a cross-modal joint feature corresponding to each sample three-dimensional CAD model based on Triplet metric learning; when a target sketch is obtained, inputting the target sketch into the deep learning model to extract a target embedding feature, using the feature distance of the cross-modal joint feature to represent the semantic similarity, and determining the most similar target three-dimensional CAD model, thereby improving the accuracy of sketch retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-modal retrieval, and in particular to a cross-modal retrieval method for retrieving three-dimensional models based on sketches, a device and a medium. BACKGROUND

[0002] In recent years, with the rapid development of three-dimensional scanning, modeling and printing technologies, three-dimensional model data has shown explosive growth, especially in the field of industrial design and manufacturing. As a key digital asset, CAD models have become an urgent need for reuse, sharing and efficient retrieval. Three-dimensional model retrieval technology has gradually become a hot issue in the field of computer graphics and intelligent manufacturing due to its advantages in visual stimulation and multi-scene application.

[0003] With the popularity of touch screens and handwriting devices, and people's preference for visual expression, hand-drawn sketches have become a natural and efficient way of information transmission. Users can quickly express their design intentions through sketches, fully utilizing the randomness and creativity of input. Compared with traditional retrieval methods based on keywords or examples, sketches are more suitable for intuitive description of three-dimensional shapes and spatial structures, especially for the rapid query and preliminary screening of three-dimensional models in industrial environments. In this context, three-dimensional model retrieval based on sketches is an important branch of cross-modal retrieval.

[0004] However, three-dimensional model retrieval based on sketches still faces many challenges. First, sketches and three-dimensional models belong to two different modalities, two-dimensional hand-drawn trajectories and three-dimensional parameterized representations, respectively. The feature distribution of these two modalities has essential differences, resulting in a significant semantic gap between modalities. Second, hand-drawn sketches based on industrial parts are usually simple, and different users have different drawing styles for the same part, and lack of color and texture information, further increasing the difficulty of feature extraction and matching. In addition, the number of available sketch samples for training in industrial scenarios is often limited, which also restricts the generalization ability of the retrieval model. Therefore, the current accuracy of retrieving three-dimensional models based on sketches is low, affecting the user experience. SUMMARY

[0005] The present application aims to at least solve one of the problems in the prior art. To this end, the present application provides a cross-modal retrieval method for retrieving three-dimensional models based on sketches, a device and a medium, which can embed sketches and three-dimensional models of different modalities into a unified semantic space, implement cross-modal retrieval through Triplet metric learning, and improve the retrieval accuracy of three-dimensional models based on sketches.

[0006] In a first aspect, the present application provides a cross-modal retrieval method for retrieving three-dimensional models based on sketches, comprising:

[0007] constructing a shared data space based on a plurality of sample sketches and sample three-dimensional CAD models, rendering each of the sample three-dimensional CAD models into a two-dimensional view set, wherein the two-dimensional view set comprises a plurality of sample two-dimensional views of the sample three-dimensional CAD models at different viewing angles;

[0008] constructing a deep learning model based on a first ResNet network and a second ResNet network sharing weight values, and training the deep learning model based on the shared data space, wherein the first ResNet network is configured to extract first embedding features of the sample sketches, the second ResNet network is configured to extract second embedding features of the sample two-dimensional views, and a plurality of the first embedding features and a plurality of the second embedding features are configured to obtain a cross-modal joint feature corresponding to each of the sample three-dimensional CAD models based on Triplet metric learning;

[0009] when a target sketch is obtained, inputting the target sketch into the deep learning model to extract a target embedding feature, and determining a target three-dimensional CAD model based on a retrieval feature distance corresponding to each of the cross-modal joint features and the target embedding feature, wherein the target three-dimensional CAD model corresponds to a smallest retrieval feature distance.

[0010] According to some embodiments of the present application, before constructing a shared data space based on a plurality of sample sketches and sample three-dimensional CAD models, the method further comprises:

[0011] obtaining a sketch dataset constructed based on industrial parts, wherein the sketch dataset comprises a plurality of sample three-dimensional CAD models corresponding to mechanical parts of different categories;

[0012] based on any of the sample three-dimensional CAD models, obtaining a plurality of original sketches input by a user, wherein each of the original sketches corresponds to a different viewing angle of the sample three-dimensional CAD model;

[0013] based on any of the original sketches, generating an enhanced sketch through stroke deletion and stroke deformation, and associating the enhanced sketch with the corresponding sample three-dimensional CAD model;

[0014] determining the enhanced sketch and the original sketch as the sample sketch.

[0015] According to some embodiments of the present application, generating an enhanced sketch through stroke deletion and stroke deformation comprises:

[0016] determining a plurality of sketch stroke paths based on SVG path data of the original sketch, and determining a drawing order and a stroke length corresponding to each of the sketch stroke paths;

[0017] Determine a deletion probability of each of the sketch stroke paths based on the drawing order and the stroke length, and delete a plurality of the sketch stroke paths based on the deletion probability, wherein the drawing order is positively correlated with the deletion probability, and the stroke length is negatively correlated with the deletion probability;

[0018] Based on any remaining sketch stroke path, determine the start and end points of a plurality of stroke segments and the control points of a Bezier curve as path control points, determine a target deformation strength based on the stroke length and a preset deformation strength, determine an offset amount of each of the path control points based on the target deformation strength and a preset trigonometric function, and adjust the position of each of the path control points based on the respective offset amount.

[0019] Generate the enhanced sketch based on the adjusted remaining sketch stroke paths.

[0020] According to some embodiments of the present application, the first ResNet network and the second ResNet network have the same structure, and both of them sequentially include an input layer, seven hidden layers and an output layer, wherein the first layer of the hidden layers sequentially includes a 7x7 convolutional layer, a batch normalization layer, a ReLU activation function and a 3x3 maximum pooling layer, the second layer of the hidden layers includes three Bottleneck residual blocks, each of which has three convolutional layers and a skip connection structure, the third layer of the hidden layers includes four Bottleneck residual blocks, the fourth layer of the hidden layers includes six Bottleneck residual blocks, the fifth layer of the hidden layers includes three Bottleneck residual blocks, the sixth layer of the hidden layers includes a global average pooling layer, the seventh layer of the hidden layers sequentially includes a fully connected layer, a ReLU activation function and a Dropout layer, and the output layer is a fully connected layer.

[0021] According to some embodiments of the present application, training the deep learning model based on the shared data space includes:

[0022] Input each of the sample sketches into the input layer of the first ResNet network, extract the first visual feature corresponding to each of the sample sketches through the first four layers of the hidden layers of the first ResNet network, perform a View-MaxPooling operation on the first visual features of a plurality of sample sketches associated with a unified sample three-dimensional CAD model to obtain a first fusion feature, perform deep abstraction on the first fusion feature through the fifth layer of the hidden layers to obtain a deep feature, and sequentially pass the deep feature through the last two layers of the hidden layers and the output layer to obtain the first embedding feature.

[0023] Each of the sample 2D views is input into the input layer of the second ResNet network. The second visual features corresponding to each sample 2D view are extracted through the first four hidden layers of the second ResNet network. View-Max Pooling is performed on the second visual features of multiple sample 2D views belonging to the same 2D view group to obtain second fusion features. High-level semantic information is extracted from the second fusion features through the fifth hidden layer. The high-level semantic information is then passed through the last two hidden layers and the output layer to obtain the second embedding features. Each sample 3D CAD model corresponds to one second embedding feature.

[0024] According to some embodiments of the present invention, before training the deep learning model based on the shared data space, the method further includes:

[0025] Multiple positive samples and multiple negative samples are determined based on multiple sample sketches and multiple sample two-dimensional views;

[0026] Based on any of the first or second embedded features, a first sample distance is determined with each of the positive samples, a second sample distance is determined with each of the negative samples, and a loss function for Triplet metric learning is determined based on the difference between the second sample distance and the first sample distance.

[0027] According to some embodiments of the present invention, training the deep learning model based on the shared data space includes:

[0028] Based on any of the first or second embedding features, the current gradient of the current training step is determined by stochastic gradient descent, and the original parameters of the current training step, the sum of Newtonian momentum accumulated in previous training steps, the preset momentum weights, and the learning rate are determined.

[0029] The original parameters are updated to target parameters based on the current gradient, the original parameters, the sum of Newtonian momentum, the momentum weights, and the learning rate, and the current training step is executed based on the target parameters.

[0030] According to some embodiments of the present invention, determining a target 3D CAD model based on the retrieval feature distances corresponding to the target embedded features and each of the cross-modal joint features includes:

[0031] Based on any of the cross-modal joint features, determine the feature Euclidean distance between the target embedded feature and the first embedded feature, determine the feature bulldozer distance between the target embedded feature and the second embedded feature, and determine the sum of the feature Euclidean distance and the feature bulldozer distance as the retrieval feature distance;

[0032] The target 3D CAD model is determined based on the cross-modal joint feature with the smallest retrieval feature distance.

[0033] Secondly, embodiments of the present invention provide a cross-modal retrieval device for 3D models based on sketch retrieval, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the cross-modal retrieval method for 3D models based on sketch retrieval as described in the first aspect above.

[0034] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions for performing the cross-modal retrieval method for 3D models based on sketch retrieval as described in the first aspect above.

[0035] The cross-modal retrieval method for 3D models based on sketch retrieval according to embodiments of the present invention has at least the following beneficial effects: A shared data space is constructed based on multiple sample sketches and sample 3D CAD models; each sample 3D CAD model is rendered into a group of 2D views, wherein the 2D view group includes multiple sample 2D views of the sample 3D CAD model from different perspectives; a deep learning model is constructed based on a first ResNet network and a second ResNet network with shared weights; the deep learning model is trained based on the shared data space; wherein the first ResNet network is used to extract first embedding features of the sample sketches, and the second ResNet network is used to extract second embedding features of the sample 2D views; multiple first embedding features and multiple second embedding features are used to obtain cross-modal joint features corresponding to each sample 3D CAD model based on Triplet metric learning; when a target sketch is obtained, the target sketch is input into the deep learning model to extract target embedding features; the target 3D CAD model is determined based on the retrieval feature distances corresponding to the target embedding features and each of the cross-modal joint features, wherein the target 3D CAD model corresponds to the retrieval feature distance with the smallest numerical value. According to the technical solution of the present invention, a three-dimensional CAD model can be represented by two-dimensional views from multiple perspectives. By embedding the features of sketches and three-dimensional CAD models into a unified semantic space through a deep learning network, the differences between modalities are reduced. The Triplet metric is used to learn and constrain the distance between different classes, forming cross-modal joint features of the three-dimensional CAD model. When performing cross-modal retrieval, the feature distance with the cross-modal joint features is used to represent semantic similarity, thereby determining the most similar target three-dimensional CAD model and improving the accuracy of sketch retrieval. Attached Figure Description

[0036] Figure 1 This is a flowchart of a cross-modal retrieval method for 3D models based on sketch retrieval provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of a cross-modal retrieval method for 3D models based on sketch retrieval provided in another embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of a two-dimensional view group corresponding to a three-dimensional CAD model provided in another embodiment of the present invention;

[0039] Figure 4 This is an enhanced schematic diagram provided by another embodiment of the present invention;

[0040] Figure 5 This is a structural diagram of a cross-modal retrieval device for three-dimensional models based on sketch retrieval provided in another embodiment of the present invention. Detailed Implementation

[0041] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0042] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0043] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0044] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0045] This invention provides a cross-modal retrieval method, apparatus, and medium for retrieving 3D models based on sketches. The cross-modal retrieval method for retrieving 3D models based on sketches includes: constructing a shared data space based on multiple sample sketches and sample 3D CAD models; rendering each sample 3D CAD model into a group of two-dimensional views, wherein the two-dimensional view group includes multiple sample two-dimensional views of the sample 3D CAD model from different perspectives; constructing a deep learning model based on a first ResNet network and a second ResNet network with shared weights; training the deep learning model based on the shared data space; wherein the first ResNet network is used to extract first embedding features from the sample sketches, and the second ResNet network is used to extract second embedding features from the sample two-dimensional views; multiple first embedding features and multiple second embedding features are used to obtain cross-modal joint features corresponding to each sample 3D CAD model based on Triplet metric learning; when a target sketch is obtained, the target sketch is input into the deep learning model to extract target embedding features; the target 3D CAD model is determined based on the retrieval feature distances corresponding to the target embedding features and each of the cross-modal joint features, wherein the target 3D CAD model corresponds to the retrieval feature distance with the smallest numerical value. According to the technical solution of the present invention, a three-dimensional CAD model can be represented by two-dimensional views from multiple perspectives. By embedding the features of sketches and three-dimensional CAD models into a unified semantic space through a deep learning network, the differences between modalities are reduced. The Triplet metric is used to learn and constrain the distance between different classes, forming cross-modal joint features of the three-dimensional CAD model. When performing cross-modal retrieval, the feature distance with the cross-modal joint features is used to represent semantic similarity, thereby determining the most similar target three-dimensional CAD model and improving the accuracy of sketch retrieval.

[0046] The technical solutions of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0047] Reference Figure 1 , Figure 1 A flowchart of a cross-modal retrieval method for 3D models based on sketch retrieval provided in this embodiment of the invention. This cross-modal retrieval method for 3D models based on sketch retrieval includes, but is not limited to, the following steps:

[0048] S10: Construct a shared data space based on multiple sample sketches and sample 3D CAD models, and render each sample 3D CAD model into a 2D view group, wherein the 2D view group includes multiple sample 2D views of the sample 3D CAD model from different perspectives.

[0049] It should be noted that the sample's 3D CAD model can correspond to various mechanical devices, such as Figure 3The example shown includes push-button switches, stepper motors, and ball screws. In this embodiment, after obtaining the sample 3D CAD model, each sample 3D CAD model is rendered into a group of multiple sample 2D views, allowing a sample 3D CAD model to be represented through multiple 2D views, each 2D view corresponding to a perspective of the sample 3D CAD model.

[0050] It is worth noting that, during the rendering process, this embodiment employs the view rendering method proposed by the Multi-View Convolutional Neural Network (MVCNN). The pose normalization process is performed on all sample 3D CAD models in the dataset. Furthermore, a lighting model is introduced during virtual camera imaging to enhance the expression of surface depth information through image brightness variations, thereby more clearly presenting the geometric structure and three-dimensional features. The brightness information of the view effectively reflects the curvature, edges, and hierarchical relationships of the model surface.

[0051] For example, such as Figure 3 As shown, the sample 3D CAD model is placed in a standard orientation (e.g., perpendicular to the horizontal plane), allowing 12 virtual cameras to be evenly distributed around it to capture 12 views from different angles. Alternatively, the sample 3D CAD model can be in any orientation, with its centroid aligned with the geometric center of a regular dodecahedron, so that the model is surrounded by this polyhedron. Virtual cameras facing the center of the model are placed at all 20 vertices of the dodecahedron. At each vertex, the virtual camera is rotated sequentially around its optical axis by 0°, 90°, 180°, and 270°, and the view is rendered accordingly. Finally, the 20 vertex positions, combined with the four rotation angles at each position, generate a total of 80 2D views for each model.

[0052] It should be noted that in this embodiment, each sample 3D CAD model is converted into a set of 2D views through the above rendering process to represent its 3D shape. Simultaneously, multiple sample sketches are further obtained. These sketches can be pre-drawn manually and entered into the CAD system. The original sketch input pixel size is 1111×1111. During the preprocessing stage, the CAD system uniformly adjusts the size of the sample sketches to 256×256 pixels to adapt to the input requirements of the subsequent deep learning network.

[0053] S20, a deep learning model is constructed based on a first ResNet network and a second ResNet network with shared weights, and the deep learning model is trained based on a shared data space. The first ResNet network is used to extract the first embedding features of the sample sketch, and the second ResNet network is used to extract the second embedding features of the sample two-dimensional view. Multiple first embedding features and multiple second embedding features are used to obtain the cross-modal joint features corresponding to each sample three-dimensional CAD model based on Triplet metric learning.

[0054] It should be noted that this embodiment uses two ResNet networks with the same structure to construct the deep resting model by sharing weights, such as... Figure 2 As shown, the deep learning model in this embodiment is an end-to-end deep learning framework. During training, multiple sample sketches are input into the first ResNet network, which extracts features from the sample sketches. The extracted first embedding features are placed into the shared data space. Similarly, the second ResNet network extracts features from multiple sample 2D views of the 2D view group to obtain the second embedding features. The same 2D view group corresponds to one second embedding feature. The second embedding features are used to characterize the features of different perspectives of the sample 3D CAD model. The second embedding features are also saved into the shared data space, so that the shared data space stores the first and second embedding features to achieve feature embedding.

[0055] It should be noted that the sample sketch and sample 2D view are representations of the same type of data in different domains. Therefore, this embodiment adopts a triplet-based metric learning method when training the model, thereby completing the joint feature mapping between different domain data and data in the same domain during the learning process. After training, the feature mapping between the sample sketch and the sample 3D CAD model is constructed using the similarity between the first and second embedded features, thus forming a stable state in which cross-domain data are clustered according to category, which plays a good supporting role in retrieval based on the distance between features.

[0056] It should be noted that the cross-modal joint features in this implementation include the first embedded features of the sample sketch corresponding to a sample 3D CAD model and the second embedded features corresponding to the sample 3D CAD model. The cross-modal joint features are in the form of a feature set, in which all embedded features are used to represent the same sample 3D CAD model. Of course, after training, similarity evaluation and other operations can be performed using the test set, which will not be elaborated on here.

[0057] S30, when the target sketch is obtained, the target sketch is input into the deep learning model to extract the target embedding features. The target 3D CAD model is determined based on the retrieval feature distances corresponding to the target embedding features and each cross-modal joint feature. The target 3D CAD model corresponds to the retrieval feature distance with the smallest value.

[0058] It should be noted that after obtaining the target sketch, because the deep learning model has already been trained to obtain multiple cross-modal joint features, the 3D CAD models that can be identified have a retrieval basis. For models belonging to the same category, their feature distributions in the embedding space are usually quite similar. Therefore, when using the target sketch as a query sample, such as Figure 1As shown, the target sketch is input into a deep learning model to extract target embedding features. Based on the target embedding features, the mapped neighborhood region is determined among multiple cross-modal joint features, thus completing the retrieval. This embodiment uses the average Euclidean distance as a similarity measure. Taking cross-modal joint features as the unit, the feature distance between the target embedding feature and each first and second embedding feature in the cross-modal joint features is determined and averaged. This average value is the retrieval feature distance corresponding to the cross-modal joint feature. The smaller the retrieval feature distance, the higher the similarity between the target embedding feature and the cross-modal joint feature. Since each cross-modal joint feature corresponds to a sample 3D CAD model, the sample 3D CAD model corresponding to the cross-modal joint feature with the smallest retrieval feature distance can be determined as the target 3D CAD model, thereby achieving effective cross-modal retrieval from sketch to 3D model.

[0059] In another embodiment, before performing step S10, the following steps are included, but are not limited to:

[0060] S11, Obtain a sketch dataset based on industrial parts, wherein the sketch dataset includes sample 3D CAD models corresponding to mechanical parts of multiple categories;

[0061] S12, Based on any sample 3D CAD model, obtain multiple original sketches input by the user, where each original sketch corresponds to a different viewpoint of the sample 3D CAD model;

[0062] S13, Based on any original sketch, generate an enhanced sketch by stroke deletion and stroke deformation, and associate the enhanced sketch with the corresponding sample 3D CAD model;

[0063] S14, the enhanced sketch and the original sketch are identified as the sample sketch.

[0064] It should be noted that this embodiment uses the InduSketch dataset, a sketch dataset built based on industrial parts. This dataset covers 151 categories, including typical mechanical parts such as push-button switches, handles, and stepper motors. Each category contains 10 to 30 3D CAD models, for a total of 1700 models. For each 3D CAD model, sketches can be drawn by three artists from seven angles: top, bottom, left, right, front, rear, and a random free viewpoint. The sketches are then input into the CAD system as the original sketches.

[0065] It should be noted that, since different users have different sketching habits, in order to improve the accuracy of retrieval with a small number of sketch samples, this embodiment further performs stroke reduction and stroke deformation on the original sketch to obtain an enhanced sketch. One enhanced sketch is generated for each original sketch, and both the enhanced sketch and the original sketch are determined as sample sketches, thereby expanding the number of sketch samples.

[0066] In another embodiment, in step S13, an enhanced sketch is generated by stroke deletion and stroke deformation, specifically including but not limited to the following steps:

[0067] S131, Based on the SVG path data of the original sketch, determine multiple sketch stroke paths, and determine the drawing order and stroke length of each sketch stroke path;

[0068] S132, determine the deletion probability of each sketch stroke path based on the drawing order and stroke length, and delete multiple sketch stroke paths based on the deletion probability. The drawing order is positively correlated with the deletion probability, and the stroke length is negatively correlated with the deletion probability.

[0069] S133, based on any remaining sketch stroke path, determine the start and end points of multiple stroke line segments and the control points of the Bézier curve as path control points, determine the target deformation intensity based on the stroke length and preset deformation intensity, determine the offset of each path control point based on the target deformation intensity and preset trigonometric function, and adjust the position of each path control point based on their respective offsets.

[0070] S134, Generate an enhanced sketch based on the adjusted remaining sketch stroke paths.

[0071] It should be noted that in the CAD field, SVG path data can draw arbitrarily complex shapes by defining a series of coordinate points and curve instructions. Therefore, this embodiment can determine multiple sketch stroke paths based on the SVG path data of the original sketch, split an original sketch into multiple paths, and determine the drawing order and stroke length of each sketch stroke path.

[0072] It should be noted that, in order to enhance the sketch by deleting strokes and deforming them, this embodiment introduces stroke length and drawing order when determining the deletion probability of each sketch stroke path. Stroke length is negatively correlated with the deletion probability; longer strokes better represent the shape of the object, therefore longer strokes are given a higher retention weight. Similarly, since the sketch is drawn manually, strokes drawn later in the order may represent details rather than the shape of the model, so their retention weight is appropriately reduced, meaning the drawing order is positively correlated with the deletion probability. This embodiment deletes multiple sketch stroke paths based on the deletion probability, see reference [reference needed]. Figure 4 As shown, you can delete strokes until the remaining number is between 5 and 15.

[0073] It should be noted that after reducing the sketch stroke paths, this embodiment further deforms the strokes. For each retained sketch stroke path, all path control points (the start and end points of the stroke segments and the control points of the Bézier curves) are extracted. The deformation intensity is adaptively adjusted according to the length of the stroke (longer strokes allow for larger offsets), and an offset generated by sine and cosine functions is applied to each point, thereby introducing natural hand-drawn fluctuations while maintaining the continuity of the overall shape. Finally, the processed paths are recombined and output. This method expands the dataset at a 1:1 ratio, generating one enhanced sketch for each original sketch, consistent with the size of the original sketch dataset.

[0074] In another embodiment, the first ResNet network and the second ResNet network have the same structure. Both the first ResNet network and the second ResNet network sequentially include an input layer, seven hidden layers, and an output layer. The first hidden layer sequentially includes a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, and a 3×3 max pooling layer. The second hidden layer includes three Bottleneck residual blocks, each of which has three convolutional layers and a skip connection structure. The third hidden layer includes four Bottleneck residual blocks. The fourth hidden layer includes six Bottleneck residual blocks. The fifth hidden layer includes three Bottleneck residual blocks. The sixth hidden layer includes a global average pooling layer. The seventh hidden layer sequentially includes a fully connected layer, a ReLU activation function, and a Dropout layer. The output layer is a fully connected layer.

[0075] It should be noted that this embodiment uses the ResNet-50 deep residual network as its basic architecture to extract common semantic features from the input sketch and multiple views of the 3D model. This network fully utilizes its deep residual learning mechanism to effectively extract high-level semantic representations of cross-modal data.

[0076] It should be noted that the first ResNet network and the second ResNet network in this embodiment have the same network structure, consisting of nine layers, as detailed below:

[0077] The first layer is the input layer, which is used to receive preprocessed RGB image input, namely the target sketch, sample sketch or sample two-dimensional view in this embodiment;

[0078] The second layer (the hidden layer of the first layer) contains a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, and a 3×3 max pooling layer for initial feature extraction and downsampling;

[0079] The third layer (the hidden layer of the second layer) corresponds to Stage 1, which consists of 3 Bottleneck residual blocks. Each block contains 3 convolutional layers and skip connection structures, and the output feature map size is 56×56.

[0080] The fourth layer (the hidden layer of the third layer) corresponds to Stage 2, which consists of 4 Bottleneck residual blocks. Downsampling is achieved through the convolution stride, and the output feature map size is 28×28.

[0081] The fifth layer (the hidden layer of the fourth layer) corresponds to Stage 3, which consists of 6 Bottleneck residual blocks. It further extracts abstract features and outputs a feature map with a size of 14×14.

[0082] The sixth layer (the hidden layer of the fifth layer) corresponds to Stage 4, which consists of 3 Bottleneck residual blocks. The output feature map size is 7×7 and the number of channels is 2048.

[0083] The seventh layer (a hidden layer of the sixth layer) contains a global average pooling layer, which compresses the 7×7×2048 spatial features into a 2048-dimensional feature vector;

[0084] The eighth layer (a hidden layer of the seventh layer) is a fully connected layer (which can be added as needed for the task), followed by a ReLU activation function and a Dropout layer for feature transformation and regularization;

[0085] The ninth layer (output layer) is a fully connected layer that outputs the final common semantic feature vector, the dimension of which can be set according to the requirements of the cross-modal retrieval task.

[0086] This network effectively alleviates the gradient vanishing problem through residual connections, enabling the model to be stably trained up to 50 layers, thereby achieving layer-by-layer abstraction and efficient extraction of multi-level semantic features of the input image.

[0087] In another embodiment, in step S20, training a deep learning model based on a shared data space specifically includes, but is not limited to, the following steps:

[0088] S211, each sample sketch is input into the input layer of the first ResNet network. The first visual features corresponding to each sample sketch are extracted through the first four hidden layers of the first ResNet network. The View-Max Pooling operation is performed on the first visual features of multiple sample sketches associated with the unified sample 3D CAD model to obtain the first fusion features. The first fusion features are deeply abstracted through the fifth hidden layer to obtain deep features. The deep features are then passed through the last two hidden layers and the output layer to obtain the first embedding features.

[0089] S212, input each sample 2D view into the input layer of the second ResNet network, extract the second visual features corresponding to each sample 2D view through the first four hidden layers of the second ResNet network, perform View-Max Pooling operation on the second visual features of multiple sample 2D views belonging to the same 2D view group to obtain the second fusion features, extract high-level semantic information from the second fusion features through the fifth hidden layer, and pass the high-level semantic information through the last two hidden layers and the output layer to obtain the second embedding features, wherein each sample 3D CAD model corresponds to one second embedding feature.

[0090] It should be noted that, as Figure 2 As shown, this embodiment employs end-to-end model training. The first ResNet network serves as a sketch feature extractor. The original sketch and the augmented sketch are input into the input layer of the first ResNet network. After processing through the hidden layer of the first layer, the first visual features of each sample sketch are extracted through the first three stages of the first ResNet network (Stages 1 to 3, corresponding to the third to fifth layers of the network, i.e., the hidden layers of the second to fourth layers). Subsequently, the features from different viewpoints are initially fused using View-Max Pooling to obtain the first fused feature, thus aggregating multi-view information. Further, Stage 4 (the hidden layer of the fifth layer) of the first ResNet network is used to perform deep abstraction on the fused features. Finally, a global average pooling operation is used to extract the discriminative overall sketch representation features as the first embedded feature.

[0091] It should be noted that after inputting all sample 2D views corresponding to the sample 3D CAD model into the input layer of the second ResNet network, the steps for the first 5 layers are the same as those for the sample sketch, and will not be elaborated here. It is worth noting that in this embodiment, during feature fusion, the second visual features corresponding to each sample 2D view are fused together to obtain the second fused feature, which is the multi-view information of the sample 3D CAD model. Then, the Stage 4 (the hidden layer of the fifth layer) of the second ResNet network is used to extract high-level semantic information from the fused feature. Finally, the second embedded feature corresponding to the sample 3D CAD model is output through a global average pooling layer.

[0092] In another embodiment, before training the deep learning model based on the shared data space in step S20, the following steps are included, but are not limited to:

[0093] S221, determine multiple positive samples and multiple negative samples based on multiple sample sketches and multiple sample two-dimensional views;

[0094] S222, based on any first or second embedded feature, determine the first sample distance with each positive sample, determine the second sample distance with each negative sample, and determine the loss function of Triplet metric learning based on the difference between the second sample distance and the first sample distance.

[0095] It should be noted that when constructing the joint feature distribution, a shared weight method is used for feature extraction in the feature extraction network. This ensures that the feature maps of data from different distributions are located in the same high-dimensional space. To ensure that similar data are similar and dissimilar data are dissimilar, this embodiment chooses a method combining shared weights and Triplet metric learning to establish the joint feature distribution of cross-domain data. In this process, this embodiment divides multiple sample sketches and multiple sample two-dimensional views into training and test sets in the shared data space. In the training set, multiple positive samples and multiple negative samples are determined, and the distances between positive and negative sample pairs are denoted as follows: and ,in, The distance to the first sample. The second sample distance is used to make positive sample pairs relatively close and negative sample pairs relatively far apart. Therefore, it can be determined The loss function is then defined as follows: In the formula, s represents the query database, which in this embodiment is the shared data space. For similar view samples within the library, For heterogeneous view samples, N is the total number of samples in a batch. This represents the relative distance between positive and negative samples. This is the mapping function learned by the network.

[0096] In another embodiment, step S20, training a deep learning model based on a shared data space, includes:

[0097] S223, based on any first or second embedding feature, determine the current gradient of the current training step based on stochastic gradient descent, and determine the original parameters of the current training step, the sum of Newtonian momentum accumulated in previous training steps, the preset momentum weights and learning rate.

[0098] S224 updates the original parameters to the target parameters based on the current gradient, original parameters, Newton's momentum sum, momentum weights, and learning rate, and executes the current training step based on the target parameters.

[0099] It should be noted that this embodiment uses a fully weight-sharing approach during training, combining stochastic gradient descent (SGD) and Newton's momentum method to optimize the training process. This is because for large datasets, SGD has fast training and convergence speeds, but it relies on robust initial values ​​and good learning rate adjustment, which can lead to instability. The addition of Newton's momentum can suppress oscillations to some extent and improve the stability of network learning.

[0100] It should be noted that during the training process of each embedded feature, a parameter is obtained at each training step. In this embodiment, the original parameters are determined based on the current training step, and the target parameters are obtained by updating the original parameters using Newton's momentum method. Each step generates Newton's momentum, a process well-known to those skilled in the art. The expression for the target parameter in this embodiment is as follows: In the formula, For the target parameters, These are the original parameters. For the current gradient, Let Newton's momentum be the sum of his and his. The preset momentum weight, for example, 0.9, This represents the current learning rate, initialized to 0.0001.

[0101] It is worth noting that, in order to further improve the yield rate and ensure the stability of training, this embodiment... Use the following formula for adaptive adjustment: In the formula, and These represent the current learning rate and the original learning rate, respectively. The number of iterations; and In this embodiment, the parameters set in the learning rate update formula are 0.75 and 0.0001, respectively.

[0102] In another embodiment, in step S30, the target 3D CAD model is determined based on the retrieval feature distances corresponding to the target embedded features and the respective cross-modal joint features. This includes, but is not limited to, the following steps:

[0103] S31, based on any cross-modal joint feature, determine the feature Euclidean distance between the target embedded feature and the first embedded feature, determine the feature bulldozer distance between the target embedded feature and the second embedded feature, and determine the sum of the feature Euclidean distance and the feature bulldozer distance as the retrieval feature distance;

[0104] S32, determine the target 3D CAD model based on the cross-modal joint features with the minimum retrieval feature distance.

[0105] It should be noted that for similar models, since their distribution in high-dimensional space is similar, it is only necessary to measure the mapping position of the input query (target sketch) in the retrieval database. That is, using the target embedding feature as the retrieval input, the feature Euclidean distance between the target embedding feature and each of the first and second embedding features and the target embedding feature is determined. The cross-modal joint feature corresponds to a sample 3D CAD model and includes multiple first and second embedding features. Therefore, the multiple feature Euclidean distances corresponding to the cross-modal joint feature can be determined as the retrieval feature distance. The smaller the value of the retrieval feature distance, the more similar the target view is to a certain view of the corresponding sample 3D CAD model. Based on this, the sample 3D CAD model to which the cross-modal joint feature with the smallest retrieval feature distance belongs is determined as the target 3D CAD model.

[0106] For example, when multiple cross-modal joint features are stored in the shared data space, the distribution space of joint features of cross-domain data can be determined. For sample sketches For its first embedding feature For the sample 3D CAD model ,use Two-dimensional view of a sample To represent the sample 3D CAD model View features of a sample two-dimensional view ,in, Let M be the weights to be learned, and M be the model data space. Define the sample sketch. To sample 3D CAD model The retrieval feature distance is ,satisfy As shown in the formula, Let the feature Euclidean distance be the distance between a sample sketch and a sample 3D CAD model, and the j-th sample 2D view. Define the sample sketch. To sample 3D CAD model The minimum distance is ,satisfy .

[0107] Once the target sketch is obtained, it is input into a deep learning model to extract target embedding features. It can be characterized as 3D feature vector The distance between the target sketch and the sample 3D CAD model is calculated using Euclidean distance and Wasserstein distance, respectively, with the following formulas: and In the formula yes arrive Spatial distance, yes arrive The distance traveled.

[0108] like Figure 5 As shown, Figure 5 This is a structural diagram of a cross-modal retrieval device for 3D models based on sketch retrieval, provided in one embodiment of the present invention. The present invention also provides a cross-modal retrieval device for 3D models based on sketch retrieval, comprising:

[0109] The processor 401 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0110] The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401 to execute the cross-modal retrieval method for 3D models based on sketch retrieval according to the embodiments of this application.

[0111] Input / output interface 403 is used to implement information input and output;

[0112] The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0113] Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0114] The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0115] This application also provides an electronic device, including the cross-modal retrieval device for three-dimensional models based on sketch retrieval as described above.

[0116] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described cross-modal retrieval method for retrieving 3D models based on sketches.

[0117] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0119] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A cross-modal retrieval method for 3D models based on sketch retrieval, characterized in that, include: A shared data space is constructed based on multiple sample sketches and sample 3D CAD models. Each of the sample 3D CAD models is rendered into a group of 2D views, wherein the group of 2D views includes multiple sample 2D views of the sample 3D CAD model from different perspectives. A deep learning model is constructed based on a first ResNet network and a second ResNet network with shared weights, and trained on the shared data space. The first ResNet network extracts first embedding features from the sample sketch, and the second ResNet network extracts second embedding features from the sample 2D view. Multiple first and second embedding features are used to obtain cross-modal joint features corresponding to each sample 3D CAD model based on Triplet metric learning. The first and second ResNet networks have identical structures, each consisting of an input layer, seven hidden layers, and an output layer. In the above, the first hidden layer consists of a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, and a 3×3 max pooling layer. The second hidden layer consists of three Bottleneck residual blocks, each of which includes three convolutional layers and a skip connection structure. The third hidden layer consists of four Bottleneck residual blocks. The fourth hidden layer consists of six Bottleneck residual blocks. The fifth hidden layer consists of three Bottleneck residual blocks. The sixth hidden layer consists of a global average pooling layer. The seventh hidden layer consists of a fully connected layer, a ReLU activation function, and a Dropout layer. The output layer is a fully connected layer. Once the target sketch is obtained, the target sketch is input into the deep learning model to extract the target embedding features. The target 3D CAD model is determined based on the retrieval feature distances corresponding to the target embedding features and each of the cross-modal joint features, wherein the target 3D CAD model corresponds to the retrieval feature distance with the smallest value.

2. The cross-modal retrieval method for 3D models based on sketch retrieval according to claim 1, characterized in that, Before constructing a shared data space based on multiple sample sketches and sample 3D CAD models, the method further includes: Obtain a sketch dataset based on industrial parts, wherein the sketch dataset includes sample 3D CAD models corresponding to mechanical parts of multiple categories; Based on any of the sample 3D CAD models, multiple original sketches input by the user are obtained, wherein each original sketch corresponds to a different viewpoint of the sample 3D CAD model; Based on any of the original sketches, an enhanced sketch is generated by stroke reduction and stroke deformation, and the enhanced sketch is associated with the corresponding sample 3D CAD model; The enhanced sketch and the original sketch are identified as the sample sketch.

3. The cross-modal retrieval method for 3D models based on sketch retrieval according to claim 2, characterized in that, Enhanced sketches are generated through stroke reduction and stroke deformation, including: Based on the SVG path data of the original sketch, multiple sketch stroke paths are determined, and the drawing order and stroke length of each sketch stroke path are determined. The deletion probability of each sketch stroke path is determined based on the drawing order and the stroke length. Multiple sketch stroke paths are deleted based on the deletion probability. The drawing order is positively correlated with the deletion probability, and the stroke length is negatively correlated with the deletion probability. Based on any of the remaining sketch stroke paths, the start and end points of multiple stroke segments and the control points of the Bézier curve are determined as path control points. The target deformation intensity is determined based on the stroke length and the preset deformation intensity. The offset of each path control point is determined based on the target deformation intensity and the preset trigonometric function. The position of each path control point is adjusted based on its corresponding offset. The enhanced sketch is generated based on the adjusted remaining sketch stroke paths.

4. The cross-modal retrieval method for 3D models based on sketch retrieval according to claim 1, characterized in that, Training the deep learning model based on the shared data space includes: Each of the sample sketches is input into the input layer of the first ResNet network. The first visual features corresponding to each sample sketch are extracted through the first four hidden layers of the first ResNet network. The first visual features of multiple sample sketches associated with the unified sample 3D CAD model are subjected to View-Max Pooling to obtain the first fusion features. The first fusion features are deeply abstracted through the fifth hidden layer to obtain deep features. The deep features are then passed through the last two hidden layers and the output layer to obtain the first embedding features. Each of the sample 2D views is input into the input layer of the second ResNet network. The second visual features corresponding to each sample 2D view are extracted through the first four hidden layers of the second ResNet network. The second visual features of multiple sample 2D views belonging to the same 2D view group are subjected to View-Max Pooling to obtain the second fusion features. High-level semantic information is extracted from the second fusion features through the fifth hidden layer. The high-level semantic information is then passed through the last two hidden layers and the output layer to obtain the second embedding features. Each sample 3D CAD model corresponds to one second embedding feature.

5. The cross-modal retrieval method for 3D models based on sketch retrieval according to claim 4, characterized in that, Before training the deep learning model based on the shared data space, the method further includes: Multiple positive samples and multiple negative samples are determined based on multiple sample sketches and multiple sample two-dimensional views; Based on any of the first or second embedded features, a first sample distance is determined with each of the positive samples, a second sample distance is determined with each of the negative samples, and a loss function for Triplet metric learning is determined based on the difference between the second sample distance and the first sample distance.

6. The cross-modal retrieval method for 3D models based on sketch retrieval according to claim 5, characterized in that, Training the deep learning model based on the shared data space includes: Based on any of the first or second embedding features, the current gradient of the current training step is determined by stochastic gradient descent, and the original parameters of the current training step, the sum of Newtonian momentum accumulated in previous training steps, the preset momentum weights, and the learning rate are determined. The original parameters are updated to target parameters based on the current gradient, the original parameters, the sum of Newtonian momentum, the momentum weights, and the learning rate, and the current training step is executed based on the target parameters.

7. The cross-modal retrieval method for 3D models based on sketch retrieval according to claim 1, characterized in that, Determining the target 3D CAD model based on the target embedding features and the retrieval feature distances corresponding to each of the cross-modal joint features includes: Based on any of the cross-modal joint features, determine the feature Euclidean distance between the target embedded feature and the first embedded feature, determine the feature bulldozer distance between the target embedded feature and the second embedded feature, and determine the sum of the feature Euclidean distance and the feature bulldozer distance as the retrieval feature distance; The target 3D CAD model is determined based on the cross-modal joint feature with the smallest retrieval feature distance.

8. A cross-modal retrieval device for 3D models based on sketch retrieval, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the cross-modal retrieval method for retrieving 3D models based on sketches as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the cross-modal retrieval method for a 3D model based on sketch retrieval as described in any one of claims 1 to 6.

Images