Three-dimensional cad model retrieval method based on image and attribute graph fusion network

Abstract

The application provides a three-dimensional CAD model retrieval method based on an image and attribute graph fusion network, which not only fully utilizes the high-level geometric information of the image in the three-dimensional CAD model and the topological structure information contained in the attribute graph, but also introduces a multi-head attention mechanism, enhances the topological structure information in the attribute graph through the geometric information in the view, and finally fuses the information of the two modes, fully utilizes the respective mode information, and guides and enhances other mode information, so that the characteristic information of the model can be distinguished more clearly. The application is widely applicable to various CAD models and is not limited to specific data sets, can realize efficient retrieval of three-dimensional CAD models, has higher retrieval accuracy than other three-dimensional model retrieval methods with better performance, and further promotes the application of the attention mechanism and cross-modal fusion in the three-dimensional model retrieval problem.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and particularly relates to a three-dimensional model retrieval method based on a neural network and an attention mechanism, aiming to solve the problems of low three-dimensional CAD model retrieval performance and low model reuse rate in industrial production by using the technology, so that the highest similarity model can be retrieved from the model library according to the existing model, the model reuse rate is improved, the model design cost is reduced, and the production efficiency is improved. BACKGROUND

[0002] With the rapid increase in the number and complexity of three-dimensional CAD models, three-dimensional CAD model reuse and retrieval problems have attracted widespread attention. Three-dimensional CAD models play an important role in industrial manufacturing because they are the basis of the entire product life cycle. According to statistics, the reuse rate of three-dimensional CAD models has reached nearly 75%. In practical applications, model designers select similar models to the demand from the existing model library and slightly modify them to obtain new models. Therefore, a high-performance retrieval method can effectively reuse models and plays a key role in the product life cycle, which is an important factor to improve the core competitiveness of enterprises. In view of this problem, scholars have introduced graph theory, computer vision and machine learning technologies to automatically identify and retrieve large-scale three-dimensional model datasets. For example, geometric structure analysis method, function projection view method, topological correlation analysis method, semantic feature recognition method, etc. However, due to the increasing size of the dataset and the limitations of traditional computer vision and machine learning technologies, the retrieval performance cannot meet the needs of industrial production.

[0003] In recent years, with the improvement of computer computing power, deep learning has developed greatly and has been successfully applied in the field of three-dimensional model retrieval. Deep learning uses convolutional neural networks to extract local features of general three-dimensional models, and gradually constructs model abstract features through convolution and pooling methods, so as to perform model retrieval. However, the general model retrieval method faces the following problems. The voxel-based method has a 3rd power increase in the amount of calculation and memory consumption with the resolution, so the network structure performs poorly in a three-dimensional model dataset with high resolution. The point cloud-based method is difficult to process three-dimensional point cloud data with traditional 2D deep network due to the irregularity and disorder of point cloud data. The unordered point cloud data is usually converted into regular voxels in a three-dimensional grid for processing, but this method reduces the resolution of the three-dimensional model and ignores a lot of geometric structure information. The multi-view method benefits from the highly mature related technology of deep learning in the field of two-dimensional image processing, and the large labeled dataset of two-dimensional images can effectively pre-train the model. Compared with the voxel-based and point cloud-based methods, the three-dimensional model retrieval algorithm based on views has better results. However, since the two-dimensional images are taken by rotating the model once around, this will cause the loss of part of the local information due to the existence of the shooting gap.

[0004] Therefore, in view of the low performance of the existing retrieval method based on traditional machine learning through topological relationship and semantic feature retrieval and the problem that the deep learning method is affected by resolution and shooting angle, which cannot effectively meet the retrieval needs of the industry, it is necessary to propose a new retrieval method to meet the needs of industrial production. SUMMARY

[0005] The present application proposes a three-dimensional CAD model retrieval method based on image and attribute graph fusion network (VGNet) to overcome the shortcomings of the prior art.

[0006] The three-dimensional CAD model retrieval method based on image and attribute graph fusion network proposed by the present application not only fully utilizes the high-level geometric information of the image in the three-dimensional CAD model and the topological structure information contained in the attribute graph, but also introduces a multi-head attention mechanism to enhance the topological structure information in the attribute graph through the geometric information in the view. Finally, the information of the two modalities is fused to fully utilize the respective modal information and guide the enhancement of other modal information, so that the characteristic information of the model can be distinguished more clearly. The present application is widely applicable to various CAD models and is not limited to specific datasets, can realize efficient retrieval of three-dimensional CAD models, and has higher retrieval accuracy than other three-dimensional model retrieval methods with better performance at present. At the same time, the present application further promotes the application of attention mechanism and cross-modal fusion in the problem of three-dimensional model retrieval.

[0007] The method of the present application specifically comprises the following steps:

[0008] The method specifically comprises the following three steps: data set preprocessing, VGNet network model construction and training, and VGNet network model prediction result evaluation.

[0009] Step 1, data preprocessing of the VGNet network model, and the specific operation is as follows:

[0010] In order to obtain the data format of the network specified input, the original data set needs to be preprocessed before training. In the present application, the three-dimensional CAD model adopts the.sldprt format, and before inputting into the network framework, it needs to be converted into image data and graph structure data. In the present application, 12 images are used to describe a model.

[0011] Step 2, VGNet network model construction and training, and the specific operation is as follows:

[0012] The VGNet network model construction and training are divided into two steps: construction of the neural network model and training of the network model.

[0013] Step 2.1: construction of the neural network model

[0014] The neural network model is composed of two parts, which are a feature extractor and an attention guidance fusion module.

[0015] Step 2.1.1: feature extractor

[0016] In the two modalities, the neural network of each modality is used to extract features from the original data. In the view branch, multi-layer convolution layers are used to gradually extract features, and the convolution layers with convolution channels of 64, 128, 256 and 512 are used to extract the feature information of the view, and finally the feature information of each view is obtained. In the attribute graph branch, due to the small size of the attribute graph converted by the model, a single-layer network cannot well represent the information contained, therefore, for the attribute graph branch, a multi-layer graph neural network plus a JK-Net network is used to extract feature information.

[0017] Step 2.1.2: attention guidance fusion module

[0018] The attention guidance fusion module is composed of three sub-modules: Multi-head Attention (MHA), residual enhancement and cross-modal fusion.

[0019] Step 2.1.2.1: MHA

[0020] In order to fully tap the potential relationship between image features and attribute graph features, the application finds the connection between the two through MHA. The feature vectors of the two modalities are input to interact with each other, so as to find more valuable regional information to improve the discriminability of the features.

[0021] Step 2.1.2.2: residual enhancement

[0022] After the interaction learning between the two modalities in the above step, a score is finally obtained, and the score is between 0 and 1. The score also represents the closeness of the connection between the two modalities, and also reflects that the higher the score, the more important the region. On the contrary, the lower the score, the lower the importance. Therefore, the score reflects the importance of the region, and the features can be enhanced according to the attention score, so that important features are more obvious and edge features are more edge.

[0023] Step 2.1.2.3: cross-modal fusion

[0024] After the above steps, the modal feature information has fully utilized the high-level geometric structure information of the image features and the topological semantic information of the attribute graph features. At this time, the feature information of the two modalities is fused to form the final model feature representation. The application uses a multi-layer perception machine to realize the fusion of the two modalities.

[0025] Step 2.2: network model training

[0026] Since the image branch and the attribute graph branch are difficult to train synchronously, the feature extraction network of the two branches is pre-trained first. After pre-training is completed, the entire network framework is trained, and the parameters of the feature extraction network are updated only in the first 10 stages, and the attention fusion module is fine-tuned. After that, all parameters will participate in the update together to achieve better performance.

[0027] The model training adopts the Adam optimization algorithm to adjust the parameters, the learning rate is 1e-4, the weight decay is 1e-3, the first-order exponential decay rate is 0.9, the second-order exponential decay rate is 0.999, and a total of 200 rounds of training are performed.

[0028] Step 3, the VGNet network model prediction result is evaluated, and the specific operation is as follows:

[0029] In order to evaluate the performance and generalization ability of the method, appropriate evaluation indicators need to be selected for comparative analysis. In the application, the evaluation indicators include precision, recall, F1 score and mAP. Before introducing the evaluation indicators, the related concepts are first described:

[0030] · TP (True Positive): positive samples predicted as positive by the model

[0031] • TN (True Negative): negative samples predicted as negative class by the model

[0032] • FP (False Positive): negative samples predicted as positive class by the model

[0033] • FN (False Negative): positive samples predicted as negative class by the model

[0034] Accuracy: the ratio of correctly predicted results to the total samples, the formula is as follows

[0035]

[0036] Precision: its meaning is the probability of actual positive samples in all predicted positive samples. The formula is as follows:

[0037]

[0038]

[0039] mAP (Mean Average Precision): its meaning is the average value of precision. The formula is as follows:

[0040]

[0041]

[0042] Recall: its meaning is the probability of predicted positive samples in actual positive samples. The formula is as follows:

[0043]

[0044]

[0045] F1 Score: in order to balance precision and recall, F1 score is introduced. The formula is as follows:

[0046]

[0047] Since it is a multi-classification problem, the calculation method of precision and recall is changed, first, the precision and recall of each class are calculated, then the average precision and recall are calculated by adding according to the weight.

[0048] After the model obtains four indexes, compared with the current mainstream model, it is proved that the performance is more excellent.

[0049] The beneficial results of the present application are:

[0050] 1. The technology proposes a method of fusing view modalities and attribute graph modalities. Unlike traditional deep models, the VGNet network uses high-level geometric features of multiple views to guide the learning of attribute graph features, and fuses multiple view features containing rich global information with attribute graph features containing rich topological information and local information to form complementary advantages.

[0051] 2. The technology proposes an attention-guided fusion mechanism. From the embedded view features, an attention mask can be adaptively obtained to generate attention-aware features of the attribute graph model, which makes retrieval of three-dimensional CAD models more effective. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart for data set preprocessing;

[0053] Figure 2 A framework diagram of the VGNet network;

[0054] Figure 3 A feature extraction detail diagram;

[0055] Figure 4 A multi-head attention mechanism diagram. DETAILED DESCRIPTION

[0056] The present application comprises three steps: data set preprocessing, VGNet network model construction and training, and VGNet network model prediction result evaluation.

[0057] Step 1: Data set preprocessing

[0058] In order to obtain the data format specified by the network input, the original data set needs to be preprocessed before training. The main work is to convert the original data set into image data and graph structure data, and the conversion process is as follows Figure 1The data conversion of both formats needs to start from a STEP file, which is a standard file for exchanging product model data. The STEP file can be formed by batch conversion of the CAD model file format of SLDPRT through SOLIDWORKS tools. After obtaining the STEP file, different conversions are carried out for the two branches respectively. For the image branch, first of all, the STEP file needs to be converted into an STL file format, which is composed of the definition of a plurality of triangular facets, and the definition of each triangular facet includes the three-dimensional coordinates of the three points of the triangular facet and the normal vector of the triangular facet. Secondly, the model composed of the facets is recorded according to the sequence around the corresponding image to form image data. On the other hand, for the attribute graph branch, the information of the three-dimensional model can be directly extracted from the STEP file, such as the number of faces, the number of edges, the type of faces, the type of edges and the like. After extraction, the information can be organized into the graph structure required by the attribute graph branch.

[0059] Step 2: VGNet network model construction and training

[0060] As shown in Figure 2 , the VGNet network model construction and training is divided into two steps: construction of a neural network model and training of the network model.

[0061] Step 2.1: Construction of a neural network model

[0062] The neural network model is composed of two parts, namely a feature extractor and an attention-guided fusion module.

[0063] Step 2.1.1: Feature extractor

[0064] After obtaining the original input data of the two modalities in step 1, they are respectively input into the neural network of the respective modality to extract features.

[0065] View branch: Each three-dimensional model is represented by a set of rendered views captured by a pre-defined camera array. The present application adopts 12 views to represent a three-dimensional model. Each CAD model has a set of views, as shown in Figure 3 .a), which is input into a convolutional layer with a convolutional channel of 64, 128, 256 and 512 in turn to obtain the features of each view. The calculation formula is as follows:

[0066] z = Conv(BN(ReLU(x)))

[0067] where x is the input image, where C represents the number of output channels, S is the step of the input image, ReLU is the activation function, BN (batch Normalization) is batch normalization, and Conv is a convolution operation with a convolution kernel of 3*3.

[0068] Graph Branch: The classic network GCN in the graph neural network is used to extract features. The nodes in the graph are constantly updated and aggregated through the message passing mechanism. Since the attributed graph is in the small graph category in the graph neural network, the feature information of the single-layer network layer cannot well represent the contained information, so Figure 3 .b) method, the feature information is fully extracted through multi-layer graph convolution layers, and the front, middle and late features are spliced to form the final graph features by using a jump network. The calculation formula is as shown below:

[0069]

[0070] wherein is the feature of the node at the lth layer, σ is a nonlinear transformation, c ij is a normalization factor, N i is all neighbors of the node i including the node itself, w (l) is the weight of the lth layer, b (l) is the intercept of the lth layer.

[0071] Step 2.1.2: Attention-guided fusion module

[0072] As shown in Figure 4 , the attention-guided fusion module is composed of three sub-modules: Multi-head Attention (MHA), residual enhancement and cross-modal fusion.

[0073] Step 2.1.2.1: MHA

[0074] In order to fully exploit the potential relationship between image features and attributed graph features, the present application finds the relationship between the two through MHA. The feature vectors of the two modalities are input to interact with each other, so as to find out the more attention-worthy area information to improve the discriminability of the features. The calculation method of MHA is as shown below:

[0075] F(Q,K,V)=ξ(Concat(head1,…,head t )W 0 ),

[0076] head i reflects the relationship between the ith view and the attributed graph, head i is realized by a softmax function. Wherein, Q i =QWi i Q , K i =KW i K , V i =VWi V W is the weight parameter matrix, Q represents the attribute graph feature, K and V both represent the view feature, d model represents the model feature dimension, and ξ is a normalization function, which is realized by a sigmoid function in the experiment, and the final output guide score P1 is between 0 and 1.

[0077] Step 2.1.2.2: residual enhancement

[0078] The guide score obtained in the above step reflects the strength of the correlation between different views and attribute graphs. The higher the score value, the stronger the correlation. Conversely, the weaker. For feature enhancement, local information with stronger correlation with cross-modal features should be given greater importance. Therefore, we use the guide score F(Q, K, V) to enhance the features through residual connection:

[0079] f i ′ =f i *(1+F(Q,K,V))

[0080] Where f i *F(Q,K,V) is to refine the features through the guide score, and then added to the original features f i to generate enhanced features f i ′ . Enhanced graph features f i ′ contain view information and have higher model discrimination.

[0081] Step 2.1.2.3: cross-modal fusion

[0082] After the above steps, the modal feature information has fully utilized the high-level geometric structure information of the image features and the topological semantic information of the attribute graph features. At this time, the feature information of the two modalities is fused to form the final model feature representation. The present application uses a multi-layer perceptron to realize the fusion of the two modalities, which is calculated as follows:

[0083] f final =MLP(Concat(f g ,f v ))

[0084] Where MLP is a multi-layer perceptron consisting of two fully connected layers. f g is the attribute graph feature, f v is the view feature, and f final is the final feature of the three-dimensional CAD model.

[0085] Step 2.2: network model training

[0086] Since the training of the image branch and the attribute graph branch is difficult to synchronize, the feature extraction networks of the two branches are pre-trained first. After the pre-training is completed, the training of the entire network framework is performed, and the parameter update of the feature extraction network is frozen in the first 10 stages, and only the attention fusion module is fine-tuned. After that, all parameters will participate in the update together to achieve better performance.

[0087] The model training adopts the Adam optimization algorithm to adjust the parameters, the learning rate is 1e-4, the weight decay is 1e-3, the first-order exponential decay rate is 0.9, the second-order exponential decay rate is 0.999, and a total of 200 rounds of training are performed.

[0088] Step 3: VGNet network model prediction result evaluation

[0089] As shown in Table 1, the VGNet method proposed in the application achieves mAP of 34.4 and P@N of 95.8 on the retrieval task. Compared with CurveNet, which has the best performance in point cloud, the accuracy of VGNet is improved by 3.9, and the mAP is improved by 0.8. Compared with the multi-view based method, the VGNet framework still needs to perform better. Compared with the MVCNN implemented by GoogleNet, the accuracy is improved by 1.3, and the mAP is improved by 0.5. It shows that the method of the application has better retrieval effect than other classical three-dimensional model retrieval methods.

[0090] Table 1. Comparison of retrieval performance of advanced models

[0091]

Claims

1. A three-dimensional CAD model retrieval method based on image and attribute graph fusion network, characterized in that The method specifically comprises the following steps: Step 1, data preprocessing of the VGNet network model; Step 2, construction and training of the VGNet network model, and the specific operation is as follows: The construction and training of the VGNet network model is divided into two steps: construction of the neural network model and training of the network model; Step 2.1: construction of the neural network model; the neural network model is composed of two parts, namely a feature extractor and an attention guidance fusion module; The feature extractor uses the neural network of each modality to extract features from the original data in the two modalities; the view branch: each three-dimensional model is represented by a set of rendered views captured by a pre-defined camera array; it is input into the convolutional layer with the convolutional channel being 64, 128, 256 and 512 in turn to obtain the features of each view; the calculation formula is as follows: z = Conv(BN(ReLU(x))) where x is an input image, where C represents the number of output channels, S is the step of the input image, ReLU is its activation function, BN is batch normalization, and Conv is a convolution operation with a 3*3 convolution kernel. The attribute graph branch: the classic network GCN in the graph neural network is used to extract features; the nodes in the graph continuously update and aggregate features through the message passing mechanism; since the attribute graph is in the small graph category in the graph neural network, the feature information of a single layer network layer cannot well represent the contained information, so a multi-layer graph convolutional layer is used to fully extract feature information and a jump network is used to splice the features at the early, middle and late stages to form the final graph features; the calculation formula is as follows: wherein is the feature of the node at the l-th layer, σ is a non-linear transformation, c ij is a normalization factor, N i is all neighbors of node i including the node itself, w (l) is the weight of the l-th layer, b (l) is the intercept of the l-th layer; The attention guidance fusion module is composed of three sub-modules: Multi-head Attention (MHA), residual enhancement and cross-modal fusion; Step 2.1.1: MHA MHA is used to find the connection between image features and attribute graph features; the feature vectors of the two modalities are input to interact with each other, and an attention score is obtained; Step 2.1.2: residual enhancement According to the attention score, the features are enhanced, so that the important features are more obvious and the edge features are more edge; Step 2.1.3: cross-modal fusion The feature information of the two modalities is fused to form the final model feature representation; Step 2.2: network model training The feature extraction networks of the two branches are pre-trained; after pre-training, the entire network framework is trained, the parameters of the feature extraction network are frozen in the first N stages, and only the attention fusion module is fine-tuned; then all the parameters are updated together; Step 3, evaluation of the prediction result of the VGNet network model.

2. The three-dimensional CAD model retrieval method based on image and attribute graph fusion network according to claim 1, characterized in that: The data preprocessing of the VGNet network model; the specific steps are as follows: The three-dimensional CAD model adopts the.sldprt format, and before inputting into the network framework, it is converted into image data and graph structure data; 12 images are used to describe a model.

3. The three-dimensional CAD model retrieval method based on image and attribute graph fusion network according to claim 1, characterized in that: The calculation method of MHA is as follows: F(Q, K, V) = ξ(Concat(head1,..., headt) W 0 ), head i reflects the connection between the ith view and the attribute graph, head i implemented by the softmax function; wherein, Q i = QW i Q , K i = KW i K , V i = VW i V ; W 0 is the weight parameter matrix, Q represents the attribute graph feature, and K and V both represent the view feature; d model represents the dimension of the model feature; ξ is a normalization function, which is implemented by the sigmoid function in the experiment, and the final output guidance score P1 is between 0-1.

4. The three-dimensional CAD model retrieval method based on image and attribute graph fusion network according to claim 3, characterized in that: According to the attention score, the features are enhanced, specifically: the guidance score F(Q, K, V) is used to enhance the features through residual connection: f i ′=f i *(1+F(Q,K,V)) where f i F(Q, K, V) is to refine the features by guidance scores and then added to the original features f i to generate the enhanced features f i ′. The enhanced graph features f i ′ contain view information.

5. The image and attributed graph fusion network based 3D CAD model retrieval method according to claim 1, characterized in that: The feature information of the two modalities is fused to form the final model feature representation; specifically, a multi-layer perceptron is used to realize the fusion of the two modalities; the calculation method is as follows: f final = MLP(Concat(f g ,f v )) where MLP is a multi-layer perceptron consisting of two fully connected layers; f g is an attribute graph feature, f v is a view feature, f final is a final feature of the three-dimensional CAD model. 6.The 3D CAD model retrieval method based on image and attributed graph fusion network according to claim 1, characterized in that: The network model training adopts an Adam optimization algorithm to adjust parameters, a learning rate is 1e-4, weight decay is 1e-3, a first-order exponential decay rate is 0.9, a second-order exponential decay rate is 0.999, and a total of 200 rounds of training are performed.

Images