A method for retrieving sketch-based mechanical parts library by incorporating dual-scale features

Abstract

This invention provides a method for retrieving a sketch-based mechanical parts library by fusing dual-scale features. The method includes: acquiring 3D models of each mechanical part; projecting these models to obtain multiple views of each part; selecting the optimal view set from the multiple views using joint image entropy; constructing a mechanical sketch model; extracting edge contours from the optimal view set of each mechanical part to obtain its outline sketch; constructing a bag-of-words model and an improved MVCNN model to extract local and global features from the outline sketches of each mechanical part, respectively; constructing a dual-scale feature extraction model; coupling the local and global features of each mechanical part to obtain aggregated feature descriptors; using the dual-scale feature extraction model to extract sketch feature descriptors from the input sketch; calculating similarity between the sketch descriptors and the aggregated feature descriptors; and outputting matching results based on the calculation results. Implementing this invention can eliminate modal differences between sketches and 3D models, improving retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer-aided design technology, and in particular to a method for retrieving a sketch-based mechanical parts library that integrates dual-scale features. Background Technology

[0002] With the increasing prevalence of Computer-Aided Design (CAD) systems in modern industry, a large number of 3D digital models are generated and stored on the internet or in enterprise databases, forming parts libraries. Product designers can use these libraries to view and access the CAD models of the parts they need, accelerating product development, reducing design costs, and improving product quality. For parts suppliers, parts libraries serve as a platform for promoting and selling their products and for communicating and collaborating with customers. However, finding the required 3D CAD model within a vast and diverse parts library is a time-consuming and labor-intensive process.

[0003] Currently, traditional text-based and feature-description-based 3D model retrieval methods cannot meet the needs of rapid retrieval, and also suffer from problems such as inaccurate feature descriptions and ambiguous 3D model names. Therefore, how to quickly and effectively retrieve the model needed by the user from a massive number of part models has become an urgent need in the field of product design.

[0004] In recent years, sketch-based 3D model retrieval, with its advantage of flexibly expressing the designer's intent, has gradually attracted the attention of researchers. It can be broadly divided into two directions: one is to process the 3D model itself using features such as contour shape, point cloud, voxels, and geometric similarity to generate features. This type of method mainly extracts local features of the 3D model (local features refer to the features possessed by local areas of the 3D model, such as the balls of a bearing); the other is a view-based method. This method first uses projection to project the 3D model into two-dimensional views, and then extracts depth features from these two-dimensional views to obtain a compact representation (compact representation here refers to converting a complex 3D model into a lower-dimensional feature representation) of the 3D model's features, resulting in better performance. For example, Chao Zhang et al. proposed a view-based 3D CAD model retrieval framework for the entire product lifecycle, representing each 3D model using a series of orthogonal two-dimensional views, and using deep learning methods for view-based CAD model retrieval.

[0005] Regarding the view representation of 3D models, Liu Yujie et al. proposed a 3D model retrieval method based on hand-drawn image fusion information entropy and CNN, which obtains the representative view of the model by calculating the one-dimensional information entropy of the model projection map.

[0006] In terms of view edge extraction, traditional edge detection algorithms such as the Canny operator and the Prewitt operator are commonly used. For example, Manda B et al. proposed a weighted combination of Canny edge detection and Gaussian blur to generate computer sketches.

[0007] However, both of the above methods have shortcomings, namely:

[0008] 1) In methods based on the 3D model itself, the computational load is large; in view-based methods, the views of the 3D model mainly use a single information entropy or complexity measurement to eliminate redundant views, without fully considering the combination of multi-dimensional information entropy to improve the effect of view expression.

[0009] 2) In traditional edge extraction algorithms, view edge extraction inevitably retains a large amount of unnecessary noise;

[0010] 3) Feature extraction of 3D models ignores the local features of the view (local features refer to local information, shape, etc. of a small area in the view), while local features are very important for feature extraction of 3D models.

[0011] Therefore, in order to address the significant modal differences between sketches and 3D models, minimize the gap between them, and improve retrieval accuracy, it is necessary to propose a retrieval technology framework for sketch-based mechanical parts libraries that integrates dual-scale features. Summary of the Invention

[0012] The technical problem to be solved by the embodiments of the present invention is to provide a method for retrieving sketched mechanical parts library by integrating dual-scale features, which can eliminate modal differences between sketches and 3D models and improve retrieval accuracy.

[0013] To address the aforementioned technical problems, embodiments of the present invention provide a method for retrieving a sketch-based mechanical parts library by fusing dual-scale features, the method comprising the following steps:

[0014] The three-dimensional models of each mechanical part are obtained, and the three-dimensional models of each mechanical part are projected to obtain multiple views of each mechanical part. Furthermore, the optimal view set of each mechanical part is selected by using the joint image entropy method.

[0015] Construct a mechanical sketch model for extracting the edge contours of part views, and import all selected optimal view sets into the mechanical sketch model to extract the edge contours of the views, thereby obtaining the contour sketches of each mechanical part.

[0016] A visual bag-of-words model and an improved MVCNN model are constructed, and local and global features are extracted from the outline sketches of each mechanical part in the visual bag-of-words model and the improved MVCNN model, respectively.

[0017] A dual-scale feature extraction model based on feature fusion is constructed, which couples the local features extracted from each mechanical part with their corresponding global features to obtain the aggregated feature descriptor of each mechanical part;

[0018] The dual-scale feature extraction model is used to extract the sketch feature descriptors of the input sketch, and the similarity is calculated with the aggregated feature descriptors of each mechanical part. Based on the calculation results, the matching results are output.

[0019] The method of projecting the 3D model of each mechanical part using a fixed projection method to obtain multiple views of each mechanical part is achieved by performing the following steps, specifically including:

[0020] The current 3D model to be projected is vertically aligned with a constant z-axis.

[0021] The virtual camera is positioned in a pre-defined manner, specifically at a 30-degree angle to the horizontal plane of the 3D model to be projected, with its direction pointing directly to the center of the 3D model to be projected, and it is positioned to circle the 3D model to be projected every 30 degrees.

[0022] The virtual camera captures images of the 3D model to be projected, resulting in 12 views of each mechanical part.

[0023] The specific steps for selecting the optimal view of each mechanical part using the joint image entropy method include:

[0024] One-dimensional entropy calculation is performed on all views of each mechanical part to obtain the first entropy value of all views of each mechanical part, and two-dimensional entropy calculation is performed on all views of each mechanical part to obtain the second entropy value of all views of each mechanical part.

[0025] The first entropy value and the second entropy value of the same view in the same mechanical part are assigned weight ratios and then weighted and added together to obtain the comprehensive entropy value of all views of each mechanical part.

[0026] Sort the comprehensive entropy values ​​of all views in the same mechanical part in descending order, and select the views with the larger comprehensive entropy values ​​in each mechanical part as the optimal view set.

[0027] The mechanical sketch model includes a Sobel operator, a U-Net structure, and a PatchGAN structure. The Sobel operator is incorporated into the U-Net structure as the first convolutional layer. The U-Net structure serves as a generator to generate a contour sketch from the input part view. The PatchGAN structure serves as a discriminator to judge the quality of the contour sketch generated by the generator.

[0028] The visual bag-of-words model includes a SIFT keypoint detection module, a K-Means clustering module for obtaining the visual dictionary, a module for extracting and fusing contour sketch features, and an MLP module for extracting high-level features; wherein,

[0029] The SIFT key point detection module is used to extract SIFT key points from the contour sketch of each mechanical part using the SIFT algorithm, and to calculate the feature descriptors of the key points; wherein, each feature descriptor is represented as a vector, which is used to characterize the gradient direction and intensity of the region surrounding the key point.

[0030] The K-Means clustering visual dictionary acquisition module is used to assign each feature descriptor to the nearest cluster center based on K-Means clustering, and to form a visual dictionary from the final cluster centers;

[0031] The module for extracting and fusing contour sketch features is used to extract the fusing features of the contour sketches of each mechanical part based on a visual dictionary, so as to form a histogram matrix of each mechanical part.

[0032] The MLP high-level feature extraction module is used to take the histogram matrix of each mechanical part as input, and output the local feature descriptor of each mechanical part through multi-layer nonlinear transformation of the MLP network.

[0033] The improved MVCNN model includes a multi-view encoding module, a multi-view fusion module, and a global feature extraction module; wherein,

[0034] The multi-view encoding module is used to encode the contour sketches of each mechanical part through a convolutional layer CONV. 1-5 Perform convolution operations separately;

[0035] The multi-view fusion module is used to aggregate the output results of the convolution operation on the contour sketches of each mechanical part through the pooling aggregation layer View-pool.

[0036] The global feature extraction module is used to extract features from the aggregation results through three fully connected layers and output global feature descriptors for each mechanical part.

[0037] The steps of extracting sketch feature descriptors from the input sketch using the dual-scale feature extraction model, calculating similarity between these descriptors and the aggregated feature descriptors of each mechanical part, and further outputting matching results based on the calculation results specifically include:

[0038] The similarity between sketch feature descriptors and aggregated feature descriptors of each mechanical part is measured based on Euclidean distance. The features are then sorted and output in ascending order according to the Euclidean distance to achieve matching retrieval of sketches and 3D models in the mechanical parts library.

[0039] Implementing the embodiments of the present invention has the following beneficial effects:

[0040] This invention proposes a 3D model projection view representation method based on image entropy to reduce redundancy among multiple views. It also proposes a mechanical sketch model for extracting the edge contours of part views to deeply extract the contour sketches of parts to approximate the similarity between views and sketches. Furthermore, it constructs a visual bag-of-words model and an improved MVCNN (Multi-view Convolutional Neural Networks) model to extract local and global features of the contour sketches, respectively. By fusing dual-scale features and using Euclidean distance, it performs sketch-based 3D model matching, thereby eliminating modal differences between sketches and 3D models and improving retrieval accuracy. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0042] Figure 1 A flowchart illustrating a sketch-based mechanical parts library retrieval method incorporating dual-scale features, provided in an embodiment of the present invention;

[0043] Figure 2 This invention provides a method for retrieving a sketch-based mechanical parts library by fusing dual-scale features, which uses joint image entropy to select the optimal view.

[0044] Figure 3 This is a basic structural diagram of the mechanical sketch model in a sketch-based mechanical parts library retrieval method that integrates dual-scale features, provided in an embodiment of the present invention.

[0045] Figure 4This diagram illustrates the process of local feature extraction using a visual bag-of-words model in a sketch-based mechanical parts retrieval method that integrates dual-scale features, as provided in an embodiment of the present invention.

[0046] Figure 5 for Figure 4 The structure diagram of the histogram matrix output by the module that extracts and fuses the contour sketch features;

[0047] Figure 6 This diagram illustrates the process of global feature extraction using an improved MVCNN model in a sketch-based mechanical parts library retrieval method that integrates dual-scale features, as provided in an embodiment of the present invention.

[0048] Figure 7 This is a diagram illustrating the process of coupling local and global features into an aggregated feature descriptor in a sketch-based mechanical parts library retrieval method that integrates dual-scale features, as provided in an embodiment of the present invention.

[0049] Figure 8 This is a schematic diagram illustrating the principle framework of a sketch-based mechanical parts library retrieval method that integrates dual-scale features, as provided in an embodiment of the present invention.

[0050] Figure 9 A comparison of recall and precision curves in a contour sketch extraction experiment provided by an embodiment of the present invention for a sketch-based mechanical parts library retrieval method incorporating dual-scale features;

[0051] Figure 10 This is a comparison chart of recall and precision curves in an ablation experiment for a sketch-based mechanical parts library retrieval method that integrates dual-scale features, provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown in the figure, a sketch-based mechanical parts library retrieval method integrating dual-scale features is proposed in an embodiment of the present invention. The method includes the following steps:

[0054] Step S1: Obtain the 3D model of each mechanical part, and project the 3D model of each mechanical part to obtain multiple views of each mechanical part. Then, use the joint image entropy method to select the optimal view set of each mechanical part.

[0055] The specific process is as follows: First, the 3D model is read and displayed using Blender software.

[0056] Secondly, a fixed projection method is used to project the 3D model of each mechanical part. Specifically, the 3D model to be projected is vertically aligned with a constant z-axis. A virtual camera is positioned in a pre-defined location, specifically at a 30-degree angle to the horizontal plane of the 3D model to be projected, with its direction pointing directly to the center of the 3D model. The camera then circles the 3D model to be projected at 30-degree intervals. The virtual camera is used to photograph the 3D model to be projected, resulting in 12 views of each mechanical part.

[0057] Because the fixed projection method generates multiple views with information redundancy, some views cannot effectively express the features of the corresponding 3D model. A joint image entropy method is used to determine the amount of information contained in the projected views of the 3D model, selecting the optimal view, removing redundant views, and retaining the views with more information. Specifically, one-dimensional entropy is calculated for all views of each mechanical part to obtain the first entropy value, and two-dimensional entropy is calculated for all views of each mechanical part to obtain the second entropy value. The first and second entropy values ​​of the same view within the same mechanical part are weighted and then weighted and added together to obtain the comprehensive entropy value of all views of the same mechanical part. The comprehensive entropy values ​​of all views within the same mechanical part are sorted in descending order, and the view with the largest comprehensive entropy value among all mechanical parts is selected as the optimal view set.

[0058] Image entropy is a method of statistically analyzing image features. One-dimensional entropy represents the amount of information contained in the clustering features of gray-level distribution in an image, and its calculation formula is shown in formula (1):

[0059]

[0060] In the formula, h_1 represents the one-dimensional entropy; P i It represents the proportion of pixels with a gray value of i (0≤i≤255) in an image, and its entropy is non-negative.

[0061] Two-dimensional entropy is calculated by performing probability distribution calculation on the gray values ​​of each pixel and its surrounding pixels in an image, and then calculating the entropy value of the image

[18] . The calculation formula is shown in formulas (2) to (3):

[0062]

[0063]

[0064] In the formula, i represents the gray value of a pixel (0≤i≤255); j represents the average gray value of the neighborhood (0≤j≤255); (i,j) is the feature pair; f(i,j) is the frequency of occurrence of the feature pair (i,j); N is the image scale; h_2 represents the two-dimensional entropy; Pij This represents the probability of f(i,j) occurring in the defined image.

[0065] One-dimensional entropy is relatively simple to calculate and is used to describe the distribution of pixel values. It has a certain degree of rotation and scaling invariance, but it cannot reflect the spatial characteristics of the image's grayscale distribution. Two-dimensional entropy can better reflect the overall spatial characteristics of an image, but it is more sensitive to image resolution, rotation, and scaling, and it is invariant.

[0066] To balance the overall spatial features and invariance of the image, a method combining one-dimensional and two-dimensional image entropy is proposed, which respectively assigns the view... Figure 1 The two-dimensional entropy is weighted by different weights ω1 and ω2. These weights are then fused and added together to obtain a comprehensive entropy value T. The comprehensive entropy values ​​T are then sorted, and the view with the largest comprehensive entropy value T is selected as the representation of the 3D model. Figure 2 As shown.

[0067] Step S2: Construct a mechanical sketch model for extracting the edge contours of the part views, and import all the selected optimal view sets into the mechanical sketch model to extract the edge contours of the views, thereby obtaining the contour sketches of each mechanical part.

[0068] The specific process involves drawing inspiration from the anime2sketch algorithm in the animation field and incorporating traditional Sobel edge detection technology to propose a mechanical sketch model for extracting the edge contours of part views. The basic structure is as follows: Figure 3 As shown. The mechanical sketch model includes the Sobel operator, the U-Net structure, and the PatchGAN structure; the Sobel operator is incorporated into the U-Net structure as the first convolutional layer, the U-Net structure serves as a generator to generate a contour sketch from the input part view, and the PatchGAN structure serves as a discriminator to judge the quality of the contour sketch generated by the generator.

[0069] At this point, the Sobel operator takes the optimal view of each mechanical part as input and extracts features through the U-Net structure encoder (i.e., the left half of the U-Net). This part mainly consists of convolution and downsampling (pooling layers). In the convolution part, the Sobel operator is introduced as the first convolutional layer. This operator contains matrices in both horizontal and vertical directions, respectively... and Convolving the aforementioned matrix with the optimal view yields approximate values ​​of the horizontal and vertical brightness differences of the view's pixels. Pixels with larger approximate brightness differences are identified as edges of the view. Using the Sobel operator as the first convolutional layer allows the model to initially learn edge information of the view, simplifying the learning of subsequent convolutional layers and improving the model's training efficiency and performance. Simultaneously, to adapt to feature extraction of the optimal view's edge contours and improve the network's feature extraction capability, the kernel size in the ordinary generative adversarial network is changed from 3×3 to 4×4. During downsampling, the number of channels doubles, continuously shrinking the image and extracting shallow information. The decoder (right half of U-Net) employs convolution and upsampling (deconvolution). During upsampling, the number of channels is halved, continuously enlarging the image and extracting deeper information. A copy-and-crop shortcut connection is used between the encoder and decoder to fuse features from both sides of U-Net, combining shallow and deep features.

[0070] During convolution calculations in the U-Net generator, padding refers to adding extra pixel values ​​around the input image. This controls the size of the output feature map and helps preserve image edge information. There are two padding modes: "SAME" and "VALID". When padding is set to "SAME", zeros are added around the image to preserve edge information; when padding is set to "VALID", no zeros are added. The advantage of VALID is that it reduces the size of the feature map, thus reducing network parameters and computational cost. However, to preserve image edge information as much as possible and ensure that the input and output images are the same size, the convolution mode is set to SAME.

[0071] The PatchGAN discriminator receives a contour sketch generated by the generator as input, divides the sketch into multiple small blocks, and evaluates the realism of each block within a local region. Based on feedback from the discriminator, the generator adjusts the generated contour sketch to gradually approximate the realistic sketch of the part. Through the discriminative mechanism of the PatchGAN structure, the generator can benefit from the evaluation and optimization of local details, thereby generating more realistic and accurate contour sketches.

[0072] Therefore, the mechanical sketch model can achieve rapid contour extraction, learn higher-level features of the optimal view, avoid the influence of redundant lines on the overall contour, and make the resulting contour sketch have stronger generalization ability. Thus, by importing all selected optimal views into the mechanical sketch model for view edge contour extraction, contour sketches of various mechanical parts can be obtained quickly.

[0073] Step S3: Construct a visual bag-of-words model and an improved MVCNN model, and extract local and global features from the outline sketches of each mechanical part in the visual bag-of-words model and the improved MVCNN model, respectively.

[0074] The specific process involves extracting deep features from the generated contour sketch to achieve accurate matching between the sketch and the outline sketch. To address the weak representational power of features extracted by a single model, a visual bag-of-words model is used to extract local features from the contour sketch, while an improved MVCNN model is used to extract global features. The local and global features are then coupled to improve the feature extraction performance of the contour sketch.

[0075] First, a visual bag-of-words model is constructed, including a SIFT keypoint detection module, a K-Means clustering module for obtaining the visual dictionary, a module for extracting and fusing contour sketch features, and an MLP module for extracting high-level features, such as... Figure 4 As shown. It is understandable that the above four modules are all executable software functional modules.

[0076] The SIFT key point detection module is used to extract SIFT key points from the contour sketches of each mechanical part using the SIFT algorithm, and to calculate the feature descriptors of the key points. Each feature descriptor is represented as a vector, which is used to characterize the gradient direction and intensity of the region surrounding the key point.

[0077] The K-Means clustering module for obtaining a visual dictionary is used to assign each feature descriptor to the nearest cluster center based on K-Means clustering, and then form the visual dictionary from the final cluster centers. The module executes the following steps: First, it randomly selects k cluster centers from all sample points; then, it calculates the Euclidean distance from each sample point to a cluster center and assigns it to the nearest cluster; next, it uses the mean of the sample points in each cluster as the new center point for the next iteration; this assignment and update process is repeated until the cluster centers no longer change, completing the clustering. The final k cluster centers are the k visual words, and all visual words form the visual dictionary.

[0078] The feature extraction and fusion module for contour sketches is used to extract fusion features from the contour sketches of each mechanical part based on a visual dictionary, thereby constructing a histogram matrix for each mechanical part. The module executes the following steps: First, it calculates the distance from each sample point of the contour sketch to all cluster centers; then, it finds the cluster center with the smallest distance, assigns the sample point to this cluster center, and increments the corresponding visual word histogram number by 1. Each sample point can be approximated by a visual word. This process is repeated for all sample points to obtain a feature histogram of a contour view; therefore, each contour view can be represented by a feature histogram. Finally, the feature histograms of multiple contour sketches of a 3D model are merged into a single fusion feature, meaning each model can be represented by multiple feature histograms. The feature histograms of all models constitute a histogram matrix, as shown in the image. Figure 5 As shown, This represents the frequency of the Mth view of the 3D model N under visual word k.

[0079] The MLP (Multi-Layer Perceptron) module extracts advanced features by taking the histogram matrix of each mechanical part as input and outputting local feature descriptors for each part through multi-layer nonlinear transformations of the MLP network. Since feature histograms contain frequency information of local features but lack the ability to represent the features themselves, the MLP network is introduced to take the feature histogram matrix extracted by the bag-of-words model as input. Through multi-layer nonlinear transformations of the MLP network, richer and more abstract feature representations (such as shape information) are learned from the original local features, thereby improving the representational power of the feature histogram and ultimately outputting local feature descriptors of the contour sketch. In this case, the MLP network is a 3-layer multilayer perceptron with 2 hidden layers. All connections between layers are fully connected, and each node is connected to every node in the previous layer. The network parameters are trained using the backpropagation algorithm.

[0080] Secondly, the adaptability and feature extraction capabilities of the network are enhanced by constructing an improved MVCNN network architecture, such as... Figure 6 As shown. This improved MVCNN model includes a multi-view encoding module, a multi-view fusion module, and a global feature extraction module; among them, the multi-view encoding module is used to extract the contour sketches of each mechanical part through convolutional layers CONV. 1-5 Convolution operations are performed separately; a multi-view fusion module is used to aggregate the output results of the convolution operation on the contour sketches of each mechanical part through a pooling aggregation layer (View-pool); a global feature extraction module is used to extract features from the aggregation results through three fully connected layers and output global feature descriptors for each mechanical part.

[0081] The MVCNN model is a network architecture for 3D model classification that can handle input images from multiple viewpoints. Compared to other networks, the MVCNN model extracts the overall features of each image from multiple viewpoints, without relying on specific local regions or keypoints; therefore, the features extracted by the MVCNN model are called global features.

[0082] At this point, each convolutional layer of the MVCNN model includes Local Response Normalization (Lrn), a common technique used to control model complexity and reduce overfitting. However, when Lrn layers are applied to part view feature extraction, they suffer from high computational complexity and difficulty in parameter tuning. Therefore, this invention replaces Lrn layers with Dropout. Dropout discards 50% of neurons, reducing the number of parameters, thus preventing overfitting and improving the model's generalization ability. Furthermore, the SVM loss function used in the MVCNN network model is typically used for binary classification, performing poorly on multi-class problems. The 3D parts library retrieval proposed in this invention is a multi-class problem. To better adapt to multi-class classification tasks, this invention improves the SVM loss function to a simplified cross-entropy function, directly operating on the labels and the feature maps extracted by the convolutional layers. In practice, classification is first performed using the softmax function, and then the cross-entropy loss function is used to measure the difference between the model's predictions and the true labels, i.e., calculating the loss value. This optimization improves the training efficiency and accuracy of the feature extraction model.

[0083] Finally, local features primarily capture detailed information such as the local shape in multi-view images of parts, exhibiting high discriminative ability. By focusing on local features, the model's discriminative ability and robustness to changes in viewpoint can be improved. Global features, on the other hand, capture the overall structure and semantic information of multi-view images of parts, reflecting the overall shape and layout of the images. By considering global features, the model can better understand the overall semantics of multi-view images and possess stronger generalization ability.

[0084] Step S4: Construct a dual-scale feature extraction model based on feature fusion, and couple the local features extracted from each mechanical part with their corresponding global features to obtain the aggregated feature descriptor of each mechanical part.

[0085] The specific process is as follows: First, a dual-scale feature extraction model based on feature fusion is constructed. It should be noted that the dual-scale feature extraction model is constructed from a commonly used multi-scale feature extraction module, and its process is relatively simple, so it will not be described in detail here.

[0086] Secondly, in the dual-scale feature extraction model, features are extracted from the optimal view of each mechanical part to obtain dual-scale feature descriptors for each mechanical part. To comprehensively leverage the model's discriminative and generalization abilities, this invention integrates local and global features to obtain aggregated feature descriptors for each mechanical part. The specific process is as follows: Figure 7 As shown.

[0087] There are generally two feature fusion methods: add and concat. The add method increases the information content of each dimension of the view feature while keeping the dimensionality unchanged. The concat method merges the number of channels, meaning the view feature dimension increases while the information content of each dimension remains unchanged. This invention uses the concat fusion method to fuse the local feature vectors extracted by the bag-of-words model and the global features extracted by MVCNN to retain the original information to the greatest extent. To appropriately reduce the feature dimensionality and improve robustness, after dual-scale feature fusion, an average pooling layer is constructed to reduce the dimensionality, resulting in the final dual-scale feature descriptor, achieving effective complementarity and fusion of local and global features.

[0088] Step S5: Use the proposed dual-scale feature extraction model to extract the sketch feature descriptors of the input sketch, and calculate the similarity with the aggregated feature descriptors of each mechanical part. Then, output the matching results based on the calculation results.

[0089] The specific process involves measuring the similarity between the sketch feature descriptor of the input sketch and the aggregated feature descriptor of the 3D model in the model library based on Euclidean distance. The calculated Euclidean distances are then sorted in ascending order and output to achieve matching and retrieval of sketches and 3D models in the mechanical parts library.

[0090] For example, the distance dist(X,Y) between the sketch feature descriptor and the aggregate feature descriptor is shown in Equation (4):

[0091]

[0092] In the formula, X = {x1, x2, ..., x} n} and Y = {y1, y2, ..., y n} represent feature descriptors for arbitrary sketches and 3D models, respectively.

[0093] At this point, the smaller the Euclidean distance between the two, the more similar the two feature vectors are. The matching results are then sorted in ascending order of distance and output.

[0094] like Figure 8As shown, the principle framework of the sketch-based mechanical parts library retrieval method fused with dual-scale features provided by the embodiments of the present invention includes three-dimensional model projection representation based on image entropy, contour extraction of projected view based on mechanical sketch, contour sketch feature extraction coupled with local and global features, sketch feature extraction, and similarity calculation of sketch and contour sketch based on Euclidean distance.

[0095] 1) 3D model projection view representation based on image entropy. A fixed projection method is used to project the 3D model to form multiple views. In order to eliminate the influence of redundant views, the optimal view set is selected from multiple views by using joint image entropy.

[0096] 2) Projection view contour extraction based on mechanical sketch. To extract edge contours with high accuracy and reduce the gap between edge contours and sketches, a mechanical sketch model for edge contour extraction of part views is constructed. By introducing the Sobel operator, a generative adversarial network is built to progressively refine the extraction of edge contours and transform them into contour sketches.

[0097] 3) Contour sketch feature extraction with coupling of local and global features. A visual bag-of-words model is constructed to extract local features of the contour sketch, and the MVCNN is improved to extract global features of the contour sketch. The local and global features are coupled to obtain a more representative aggregated feature descriptor.

[0098] 4) Sketch feature extraction. For the input sketch, a dual-scale feature extraction model is constructed to extract features and obtain sketch feature descriptors.

[0099] 5) Calculate the similarity between sketches and outline sketches based on Euclidean distance. The sketch feature descriptors and aggregated feature descriptors are similarized using Euclidean distance. The smaller the Euclidean distance, the more similar the two feature vectors are. The matching results are then sorted in ascending order of distance and output.

[0100] In this embodiment of the invention, the ESB (engineeringshape benchmark) dataset was used to evaluate the performance of the proposed retrieval method. The ESB dataset contains 867 models across 42 categories of parts. 66 models were grouped into three miscellaneous categories. Since these miscellaneous models cannot be further subdivided, they would affect model training. Additionally, several categories had too few models, which were ignored in the experiment. Ultimately, the experimental dataset of this invention contained 712 models.

[0101] The experiment was run on a server configured with an Intel i7 CPU, 24GB of memory and an NVIDIA GeForce GTX 1650 GPU, with Python 3.6.13 and TensorFlow 1.14.0 as the software environment.

[0102] The experiment divided the dataset into training, validation, and test sets in a 6:2:2 ratio. In the joint image entropy view representation experiment, the one-dimensional entropy weight ω1 = 0.4, and the two-dimensional entropy weight ω2 = 0.6. In the mechanical sketch-based projected view contour extraction experiment, the input image size was set to 256×256×3, the batch size was set to 1, the learning rate was set to 2e–4, and the Adam algorithm was used for parameter optimization, with 200 epochs. When training the improved MVCNN model, each model selected 6 views as the representation of the 3D model, the image data resolution was adjusted to 227×227×3, the batch size was set to 4, the learning rate was set to 1e-6, the cross-entropy loss used the sparse_softmax_cross_entropy_with_logits function, and the Adam algorithm was used for parameter optimization. The model file was saved to a checkpoint file every 1000 iterations, with 500 epochs. When evaluating the test set, all sketches in the test set are first input into the retrieval model proposed in this invention. For each query sketch, the retrieved models are sorted according to similarity. Then, the average value of the evaluation index of all sketch retrieval results in the test set is calculated as the final evaluation result.

[0103] To fully evaluate various aspects of model retrieval performance, this invention employs six evaluation metrics commonly used in the field of 3D model retrieval to assess the experimental results:

[0104] 1. Nearest Neighbor (NN): Represents the search precision of the most accurate result in the current search results.

[0105] 2. First Tier (FT): Represents the recall rate of the top K most relevant 3D models in the search results, where K is the cardinality of the query category.

[0106] 3. Second Tier (ST): Represents the recall rate of the top 2K most relevant 3D models in the search results, where K is the cardinality of the query category.

[0107] 4. E-measure (E): The precision and recall of the first n search results are calculated, and then the harmonic mean of these two metrics is solved to obtain the value of the E-measure.

[0108] 5. Discounted Cumulative Gain (DCG) is an indicator that comprehensively considers the relevance and ranking quality of search results. It assigns weights to search results with relevance as a positive proportional coefficient and discounts the weights to obtain the cumulative gain value.

[0109] 6. Mean Average Precision (MAP) is a retrieval system evaluation metric used for multiple queries. It calculates the average precision score for each query and averages the average precision scores of all queries to obtain the result. The MAP metric is one of the important indicators for measuring the performance of a retrieval system, as it reflects the system's accuracy and stability for different queries.

[0110] Using the ESB dataset, this invention compares the proposed method with commonly used sketch-based 3D model retrieval methods, including the Cross-Modal Manifold Evaluation (CDMR) algorithm, the View-Based Backpropagation (SBR-VC) algorithm, the Siamese convolutional neural network algorithm, and the 3D shape centroid representation algorithm (LWBR). Detailed results are shown in Table 1. Because the CDMR algorithm extracts descriptors using traditional methods, it is affected by the differences between sketches and different domains of the 3D model, as well as the complexity of the 3D model, making manual feature extraction extremely difficult. While the 3D shape centroid representation algorithm (LWBR) solves the feature extraction problem using depth metric learning, the excessive views generated by the 3D model projection cause feature redundancy, negatively impacting the accuracy of matching.

[0111] The method proposed in this invention improves upon the four commonly used methods mentioned above across various evaluation metrics. For example, compared to the Siamese method, it achieves a 39.6% improvement in the ST metric, and compared to the LWBR method, it achieves a 6.9% improvement in the mAP metric. Similarly, the algorithm of this invention also demonstrates significant advantages in NN, FT, E, and DCG metrics. Current commonly used methods either utilize dual-network models to extract view and sketch features separately, or seek the optimal viewpoint to improve cross-domain learning capabilities. In contrast, the method proposed in this invention not only solves the information redundancy problem of multiple views and reduces the gap between edge extraction contours and sketches, but also focuses on the fusion of global and local features, thus exhibiting better performance.

[0112] Table 1

[0113]

[0114] To verify the effectiveness of the proposed computer-generated sketch (contour sketch), this invention designed two sets of experiments and used the PR curve evaluation index for verification. The experimental results are as follows: Figure 9As shown, the precision-recall curve (PR curve) generates different PR points by varying the number of models returned by the query, thus distinguishing between the relevance and irrelevance of model retrieval and comprehensively demonstrating retrieval performance. In Experiment 1, since the ESB dataset did not provide corresponding hand-drawn sketches, this invention collected and hand-drawn sketches of 46 models from different perspectives, totaling 552 sketches, which served as the test set. Experiment 2 used the corresponding computer-generated sketches as the test set. To evaluate the consistency between the two, PR curves for both experiments were generated, starting from... Figure 9 As can be seen, the overall trends of the PR curves obtained from the two sets of experiments are similar, so it can be considered that computer-generated sketches and hand-drawn sketches have a high degree of similarity and consistency.

[0115] This invention designed two sets of ablation experiments for comparison. Experiment 1 used only the optimized MVCNN model to extract global features, while Experiment 2 used only the visual bag-of-words model to extract local features. These experiments compared recall and precision on the ESB dataset, and the results are as follows: Figure 10 . Figure 10 The PR curve of combining the Bag-of-Visual-Words (BOVW) and MVCNN models is higher than that of using only a single MVCNN model or Bag-of-Visual-Words model.

[0116] Implementing the embodiments of the present invention has the following beneficial effects:

[0117] This invention proposes a 3D model projection view representation method based on image entropy to reduce redundancy among multiple views. It also proposes a mechanical sketch model for extracting the edge contours of part views to deeply extract the contour sketches of parts to approximate the similarity between views and sketches. Furthermore, it constructs a visual bag-of-words model and an improved MVCNN (Multi-view Convolutional Neural Networks) model to extract local and global features of the contour sketches, respectively. By fusing dual-scale features and using Euclidean distance, it performs sketch-based 3D model matching, thereby eliminating modal differences between sketches and 3D models and improving retrieval accuracy.

[0118] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disk, etc.

[0119] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A sketch-based mechanical part library retrieval method fusing bi-scale features, characterized in that, The method comprises the following steps: acquiring three-dimensional models of each mechanical part, and projecting the three-dimensional models of each mechanical part to obtain multiple views of each mechanical part, and further selecting an optimal view set of each mechanical part by using a joint image entropy method; constructing a mechanical sketch model for part view edge contour extraction, and importing all selected optimal view sets into the mechanical sketch model for view edge contour extraction to obtain contour sketches of each mechanical part; constructing a visual bag-of-words model and an improved MVCNN model, and performing local feature and global feature extraction on the contour sketches of each mechanical part in the visual bag-of-words model and the improved MVCNN model, respectively; constructing a double-scale feature extraction model based on feature fusion, coupling the extracted local features of each mechanical part with the corresponding global features to obtain an aggregated feature descriptor of each mechanical part; extracting a sketch feature descriptor of an input sketch using the double-scale feature extraction model, and performing similarity calculation on the sketch feature descriptor and the aggregated feature descriptor of each mechanical part, and further outputting a matching result according to the calculation result; the specific steps of selecting an optimal view set of each mechanical part by using a joint image entropy method comprise: performing one-dimensional entropy calculation on all views of each mechanical part to obtain first entropy values of all views of each mechanical part, and performing two-dimensional entropy calculation on all views of each mechanical part to obtain second entropy values of all views of each mechanical part; performing weight ratio assignment on the first entropy value and the second entropy value of the same view in the same mechanical part, and performing weight fusion addition calculation to obtain a comprehensive entropy value of all views of each mechanical part; sorting the comprehensive entropy values of all views in the same mechanical part from large to small, and selecting the view with the largest comprehensive entropy value in each mechanical part as the optimal view set; the visual bag-of-words model comprises a SIFT key point detection module, a K-Means clustering visual dictionary acquisition module, a fused contour sketch feature extraction module, and an MLP high-level feature extraction module; wherein the SIFT key point detection module is configured to extract SIFT key points of the contour sketches of each mechanical part by using a SIFT algorithm, and calculate feature descriptors of the key points; wherein each feature descriptor is represented as a vector, which is used to represent the gradient direction and intensity of the region around the key point; the K-Means clustering visual dictionary acquisition module is configured to assign each feature descriptor to the nearest cluster center based on K-Means clustering, and form a visual dictionary with the final cluster centers; the fused contour sketch feature extraction module is configured to extract fused features of the contour sketches of each mechanical part based on the visual dictionary to form a histogram matrix of each mechanical part; the MLP high-level feature extraction module is configured to take the histogram matrix of each mechanical part as input, and output a local feature descriptor of each mechanical part through multi-layer nonlinear transformation of an MLP network.

2. The sketch-based mechanical part library retrieval method integrating dual-scale features as claimed in claim 1, wherein, The method of fixed projection is used to project the three-dimensional model of each mechanical part to obtain multiple views of each mechanical part, which is obtained by performing the following steps, specifically including: According to the constant z-axis vertical current three-dimensional model to be projected; The virtual camera is surrounded according to the previously placed position, specifically with a 30-degree angle with the horizontal plane of the current three-dimensional model to be projected, and its direction is directly pointing to the center of the current three-dimensional model to be projected, and it surrounds the current three-dimensional model to be projected in every 30 degrees. Through the virtual camera, the current three-dimensional model to be projected is photographed to obtain 12 views of each mechanical part.

3. The sketch-based mechanical part library retrieval method of fusing dual-scale features according to claim 1, characterized in that, The mechanical sketch model includes a sobel operator, a U-Net structure and a PatchGAN structure; wherein, The sobel operator is integrated into the U-Net structure as the first layer convolution, the U-Net structure is used as a generator for generating a contour sketch of the input part view, and the PatchGAN structure is used as a discriminator for judging the quality of the contour sketch generated by the generator.

4. The sketch-based mechanical part library retrieval method integrating dual-scale features as claimed in claim 1, wherein, The improved MVCNN model includes a multi-view coding module, a multi-view fusion module and a global feature extraction module; wherein, The multi-view coding module is used to perform convolution operation on the contour sketch of each mechanical part through convolution layer CONV 1-5 respectively; The multi-view fusion module is used to aggregate the output results of the contour sketch of each mechanical part after convolution operation through the pooling aggregation layer View-pool; The global feature extraction module is used to extract features of the aggregated results through three fully connected layers, and output global feature descriptors of each mechanical part.

5. The sketch-based retrieval of mechanical parts library with fusion of dual-scale features method of claim 1, wherein, The step of using the double-scale feature extraction model to extract the sketch feature descriptor of the input sketch, and calculating the similarity with the aggregated feature descriptor of each mechanical part, and further outputting the matching result according to the calculation result, specifically includes: The similarity between the sketch feature descriptor and the aggregated feature descriptor of each mechanical part is measured based on the Euclidean distance, and the Euclidean distance is sorted in order from small to large and outputted, so as to realize the matching retrieval of the sketch and the three-dimensional model in the mechanical part library.

Images