A 3D model retrieval method based on multi-ring views and latent class mining

Abstract

The present invention relates to the field of three-dimensional model retrieval technology, and provides a three-dimensional model retrieval method based on multi-ring views and latent class mining. The present invention selects a data set, configures a virtual camera with the three-dimensional model in the data set as the center, and the virtual camera rotates around the Y axis at equal angles and then moves equidistantly along the Z axis and shoots, thereby jointly generating a multi-ring view; the multi-ring view is processed by the multi-ring view feature learning network to obtain a three-dimensional model representation; the sketch in the data set is processed by the fine-grained sketch clustering network to align the features of the sketch with the features of the three-dimensional model to obtain a three-dimensional model retrieval result. According to the solution of the present invention, the enhancement technology of the multi-ring view feature learning network can capture the features of the three-dimensional model from multiple angles to enhance the model's understanding of the three-dimensional shape and more accurately identify and match the three-dimensional model in the sketch; and the latent class mining technology of the fine-grained sketch clustering network further improves the performance of sketch retrieval of three-dimensional models.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional model retrieval, and in particular relates to a three-dimensional model retrieval method based on multi-ring views and latent class mining. Background Art

[0002] In the digital age, 3D models have become a key information carrier due to their rich information expression capabilities. Rapid advances in 3D modeling technology have made creating 3D models much easier. Faced with the surging number of 3D models, efficiently and accurately retrieving the specific models users need has become a pressing challenge. Sketch-based 3D model retrieval technology has emerged as a solution to this problem. It allows users to quickly locate and retrieve target models anytime, anywhere using a simple hand-drawn sketch.

[0003] Sketch-based 3D model retrieval technology has undergone a transition from manual feature extraction to deep learning. Early research relied on inefficient manual features. Deep learning was then applied to address cross-domain issues through Siamese networks, but this was limited by ignoring the stereoscopic nature of 3D models. Subsequently, methods based on holistic metric learning were proposed, leveraging discriminative and relevance losses to improve performance, but the training process was complex. In recent years, knowledge distillation models and cross-domain guided training have significantly improved the performance of sketch-based 3D model retrieval. However, the field still faces challenges, such as the semantic differences between sketches and 3D models, the abstractness of sketches, and the varying importance of view features.

[0004] Existing methods for generating 2D views of 3D models and sketch-based retrieval face several challenges. First, the modal differences between sketches and 3D objects create a significant domain gap, which hinders feature extraction and matching. Second, the limited viewing angles of 2D views projected from 3D models limit their ability to express 3D models. Summary of the Invention

[0005] The present invention provides a three-dimensional model retrieval method based on multi-ring views and latent class mining, which can improve the technical problems in related technologies such as the significant domain gap caused by the modal difference between sketches and three-dimensional objects, the extraction and matching of obstacles, and the limited viewing angle of the three-dimensional model projected into a two-dimensional view, which leads to certain limitations in the expression of the three-dimensional model by the two-dimensional view.

[0006] The present invention provides a three-dimensional model retrieval method based on multi-ring views and latent class mining, comprising the following steps:

[0007] Step S1: Select a dataset, which includes several 3D models and sketches. Using 3D imaging software, configure a virtual camera with each 3D model in the dataset as the center. The virtual camera rotates around the Y axis at equal angles and takes pictures. At each angle, the virtual camera is then displaced along the Z axis at equal distances and takes pictures, thereby generating a multi-ring view.

[0008] Step S2: The multi-ring view is processed by the multi-ring view feature learning network to obtain a three-dimensional model representation;

[0009] Step S3: Under the guidance of the 3D model representation, the sketches in the dataset are cross-modally learned through a fine-grained sketch clustering network to share the common features of sketches and 3D models, align the features of the sketches with those of the 3D models, and obtain the 3D model retrieval results.

[0010] Furthermore, step S1 includes the following steps:

[0011] Step S101: The dataset selected is the SHREC'13 sketch 3D model retrieval dataset, which includes a sketch data subset and a 3D model data subset;

[0012] Step S102: Use the 3D graphics software Blender to construct a rectangular empty background model and configure a virtual camera. Select the center point of the rectangular empty background model as the center, set the light source at the edge of the cube of the rectangular empty background model, define a virtual sphere with the 3D model as the center, and rotate the virtual camera around the Y axis at equal angles and shoot. At each angle, the virtual camera is then displaced along the Z axis at equal distances and shoots, thereby generating a multi-ring view. Set the coordinates of the virtual camera to:

[0013] ; ; ;

[0014] Where X is the coordinate of the virtual camera on the x-axis; r is the radius of the virtual sphere; is the angle of the virtual camera relative to the x-axis; Y is the coordinate of the virtual camera on the y-axis; Z is the coordinate of the virtual camera on the z-axis; Z0 is the position of the virtual camera on the z-axis.

[0015] Furthermore, in step S102, the virtual camera is rotated around the Y axis at equal angles, and a photo is taken at each position rotated 30 degrees, for a total of 12 times. For each position rotated 30 degrees around the Y axis, the virtual camera is displaced along 6 equidistant points on the Z axis and a photo is taken at each equidistant point, finally generating 72 multi-ring views.

[0016] Furthermore, step S2 includes the following steps:

[0017] Step S201: A multi-loop view feature learning network is composed of a feature extraction module with the first ResNet-50 as the backbone network, a multi-loop view feature encoder block based on the Transformer architecture, and a two-layer multi-layer perceptron. The multi-loop view is input into the multi-loop view feature learning network, and the multi-loop view of each ring is independently passed through the feature extraction module with the first ResNet-50 as the backbone network to capture high-level features of a single view of the multi-loop view of each ring. The high-level features of a single view of the multi-loop view of each ring are then processed through a normalization layer.

[0018] Step S202: The high-level features of each ring in the multi-ring view after the normalization layer processing are average pooled to fuse information from different perspectives to form a fused feature. The fused feature is then input into the multi-ring view feature encoder block based on the Transformer architecture. The fused feature is further refined and fused using the multi-head self-attention and spatial cross-attention mechanisms. The image features of the fused feature are converted into a serialized form to obtain the feature vector of the 3D model.

[0019] Step S203: After being processed by several Transformer-based multi-loop view feature encoder blocks, the feature vectors of the 3D model are max-pooled to obtain a unified feature vector of the 3D model and further enhance the feature expression capability. Finally, the unified feature vector of the 3D model is input into a two-layer multi-layer perceptron for category prediction, ultimately obtaining the output category of the 3D model as the fine-grained category center.

[0020] Among them, the two-layer multilayer perceptron consists of a first input layer, a hidden layer and a second output layer. The layers of the first input layer, the hidden layer and the second output layer are connected by weights, and the nonlinear characteristics of the unified feature vector of the three-dimensional model are learned through a nonlinear activation function.

[0021] Furthermore, the high-level features of a single view in step S201 are processed by a normalization layer, specifically in the following steps:

[0022] Step S2011: Euclidean norm normalizes the high-level features of a single view of the multi-ring view of each ring so that the high-level features of a single view of the multi-ring view of each ring are on the same scale. The formula is:

[0023] ;

[0024] Where: is the feature vector of high-level features; is the feature vector of the high-level features of the i-th multi-ring view; is the feature vector of the high-level features of the i-th multi-ring view Eigenvector after Euclidean norm normalization; is the feature vector of high-level features Dimensions;

[0025] Step S2012, calculating the discriminant loss function by comparing each Euclidean norm normalized feature vector with the class representation feature vector of each Euclidean norm normalized feature vector , the formula is:

[0026] ;

[0027] Where: is the discriminant loss function; is the total number of multi-ring views; To find the maximum function; is the margin, which controls the minimum distance between the feature vectors that are similar to the high-level features of the multi-ring view and the feature vectors that are similar but not similar to the high-level features of the multi-ring view; D is the cosine distance of the distance metric function; is a feature vector similar to the high-level features of the i-th multi-ring view; min is the minimum function; is the output category set of the predicted three-dimensional model; j is the category index, which is used to identify each category in the output category set c of the predicted three-dimensional model; y mi is the predicted category of the m-th 3D model for the i-th multi-ring view; is the feature vector that is dissimilar to the high-level features of the i-th multi-ring view;

[0028] Step S2013, calculate the cross entropy loss function by comparing the predicted probability distribution of the 3D model with the probability distribution of the true label , the formula is:

[0029] ;

[0030] Where, is the cross entropy loss function; e is the base of the natural logarithm; is the total number of output category sets of the predicted three-dimensional model; For the The true category of the multi-ring view; For the The true category of the multi-ring view The weight vector of is the bias term for the output category of the j-th predicted 3D model; For the The true category of the multi-ring view The bias term; is the weight vector of the output category of the j-th predicted 3D model;

[0031] Step S2014, final loss function is the discriminant loss function and cross entropy loss function The combination of, the formula is:

[0032] ;

[0033] Where, is the final loss function; is the cross entropy loss function of the three-dimensional model; is a hyperparameter; is the discriminant loss function.

[0034] Furthermore, in step S202, the fused features are input into a multi-ring view feature encoder block based on the Transformer architecture, and the specific steps are as follows:

[0035] Step S2021, linearly transforming the fused features through the first weight matrix, then transforming the fused features through the second weight matrix, and then linearly transforming the fused features through the third weight matrix to obtain an embedding vector;

[0036] Step S2022: Input the embedding vector into the multi-head attention mechanism. For each head in the multi-head attention mechanism, calculate the dot product between the query vector Q and the key vector K, then normalize it through the cross entropy loss function, and finally multiply it by the sequence length to obtain the output of each head in the multi-head attention mechanism.

[0037] Step S2023, concatenate the output of each head in the multi-head attention mechanism in step S2022 into one, and output the weight matrix Integrate to obtain the splicing output of the multi-head attention mechanism, the formula is:

[0038] ;

[0039] Where, is a multi-head attention mechanism; Q, K, and V are the feature vectors of high-level features respectively The query vector, key vector, and value vector obtained by linear mapping; Concat is a concatenation operation; head1 is the output of the first attention head in the multi-head attention mechanism, head2 is the output of the second attention head in the multi-head attention mechanism, and head h is the output of the hth attention head in the multi-head attention mechanism;

[0040] In step S2024, the concatenated output of the multi-head attention mechanism is processed by a normalization layer and then input into the spatial cross attention mechanism to capture the long-range interaction of the fused features in the spatial dimension. The formula is:

[0041] ;

[0042] Where, is the spatial cross attention mechanism; softmax is the normalized exponential function; d k is the feature vector of high-level features The dimension of the key vector K obtained by linear mapping; is the feature vector of high-level features The transpose of the key vector K matrix obtained by linear mapping;

[0043] Step S2025, finally input the output processed in the spatial cross attention mechanism Feature vectors with high-level features Perform residual connection and apply normalization layer, the formula is:

[0044] ;

[0045] Where, is the normalization layer; is the final output, which is the sum of the cross-attention output after the normalization layer and the input of the residual connection.

[0046] Furthermore, step S3 includes the following steps:

[0047] Step S301: A fine-grained sketch clustering network is formed by a feature extraction module with a second ResNet-50 as the backbone network, a fully connected classifier, and a two-layer multilayer perceptron. The 3D model representation and the sketch data subset are input into the fine-grained sketch clustering network. Each sketch in the sketch data subset is passed through the feature extraction module with the second ResNet-50 as the backbone network to obtain a feature vector of the sketch.

[0048] Step S302: The feature vector of the sketch enters the fully connected classifier and the two-layer multilayer perceptron in turn, and then undergoes Euclidean norm normalization. Then, by minimizing the correlation loss function, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category to obtain the three-dimensional model retrieval result.

[0049] Furthermore, in step S302, the feature vector of the sketch is aligned with the unified feature vector of the 3D model at the center of the fine-grained category. The specific steps are as follows:

[0050] Step S3021: Calculate the distance between the feature vector of each sketch and the corresponding fine-grained category center, and pull the feature vector of the sketch closer to the nearest fine-grained category center, and minimize the correlation loss function. The formula is:

[0051] ;

[0052] Where, is the minimum correlation loss function; is the total number of sketches; is the correct category of the p-th sketch; is the correct category of the p-th sketch the number of associated fine-grained categories; is the feature vector of the p-th sketch; is the correct category of the p-th sketch Fine-grained category center of is the index in the correct sketch set;

[0053] Fine-grained sketch clustering network combined with cross entropy loss function and minimum correlation loss function Perform joint supervised training and calculate the joint loss function , where the cross entropy loss function The formula is:

[0054] ;

[0055] Where, is the cross entropy loss function; is the correct category of the p-th sketch The weight vector of is the feature vector of the p-th sketch; Pass model parameters for the p-th sketch The transformed feature vector; is the correct category label of the p-th sketch The bias term; is the total number of sketch categories; is the weight vector of the oth sketch category; b o is the bias term of the o-th sketch category;

[0056] Joint loss function The formula is:

[0057] ;

[0058] Where, is the joint loss function; L m is the cross entropy loss function of the sketch; is the hyperparameter of the sketch; L c is the minimum correlation loss function.

[0059] The present invention provides a 3D model retrieval method based on multi-ring views and latent class mining, which has at least the following technical effects:

[0060] (1) Through multi-ring view enhancement technology, the features of the 3D model can be captured from multiple angles to enhance the model's understanding of the 3D shape. By constructing multiple ring views, each ring represents a different perspective, a comprehensive analysis of the 3D model is achieved. The 3D shape can be observed and understood from different directions, capturing richer and more detailed feature information. This multi-angle feature extraction enables the retrieval dataset to more accurately identify and match the 3D model in the sketch when processing the sketch retrieval task, thereby improving the accuracy and efficiency of the retrieval.

[0061] (2) Through latent class mining technology, fine-grained subcategories that may exist in 3D models can be identified and mined. By applying clustering algorithms, similar features in 3D models are clustered together to form different categories. These fine-grained category representations help to better align the feature distributions of sketches and 3D models, allowing the retrieval dataset to more accurately understand and match the 3D model in the sketch. This not only improves the retrieval dataset's understanding of the internal structure of the 3D model, but also enhances the retrieval dataset's robustness when faced with complex and diverse sketch inputs, further improving the performance of sketch-based retrieval of 3D models. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0063] Figure 1 A flowchart of a three-dimensional model retrieval method based on multi-ring views and latent class mining provided by an embodiment of the present invention;

[0064] Figure 2 This is an example of the result of a three-dimensional model retrieval in a specific embodiment provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs; the terms used herein are only for the purpose of describing specific embodiments rather than limiting the present invention.

[0067] like Figure 1 As shown, an embodiment of the present invention provides a three-dimensional model retrieval method based on multi-ring views and latent class mining, comprising the following steps:

[0068] Step S1: Select a dataset, which includes several 3D models and sketches. Using 3D imaging software, configure a virtual camera with each 3D model in the dataset as the center. The virtual camera rotates around the Y axis at equal angles and takes pictures. At each angle, the virtual camera is then displaced along the Z axis at equal distances and takes pictures, thereby generating a multi-ring view.

[0069] Step S2: The multi-ring view is processed by the multi-ring view feature learning network to obtain a three-dimensional model representation;

[0070] Step S3: Under the guidance of the 3D model representation, the sketches in the dataset are cross-modally learned through a fine-grained sketch clustering network to share the common features of sketches and 3D models, align the features of the sketches with those of the 3D models, and obtain the 3D model retrieval results.

[0071] Specifically, step S1 includes the following steps:

[0072] Step S101: The dataset selected is the SHREC'13 sketch 3D model retrieval dataset, which includes a sketch data subset and a 3D model data subset;

[0073] Step S102: Use the 3D graphics software Blender to construct a rectangular empty background model and configure a virtual camera. Select the center point of the rectangular empty background model as the center, set the light source at the edge of the cube of the rectangular empty background model, define a virtual sphere with the 3D model as the center, and rotate the virtual camera around the Y axis at equal angles and shoot. At each angle, the virtual camera is then displaced along the Z axis at equal distances and shoots, thereby generating a multi-ring view. Set the coordinates of the virtual camera to:

[0074] ; ; ;

[0075] Where X is the coordinate of the virtual camera on the x-axis; r is the radius of the virtual sphere; is the angle of the virtual camera relative to the x-axis; Y is the coordinate of the virtual camera on the y-axis; Z is the coordinate of the virtual camera on the z-axis; Z0 is the position of the virtual camera on the z-axis.

[0076] Specifically, in step S102, the virtual camera is rotated around the Y axis at equal angles, and a photo is taken at each 30-degree rotation position for a total of 12 times. For each 30-degree rotation position around the Y axis, the virtual camera is then displaced along the Z axis to six equidistant points and a photo is taken at each equidistant point. Finally, 72 multi-ring views are generated.

[0077] Specifically, step S2 includes the following steps:

[0078] Step S201: A multi-loop view feature learning network is composed of a feature extraction module with the first ResNet-50 as the backbone network, a multi-loop view feature encoder block based on the Transformer architecture, and a two-layer multi-layer perceptron. The multi-loop view is input into the multi-loop view feature learning network, and the multi-loop view of each ring is independently passed through the feature extraction module with the first ResNet-50 as the backbone network to capture high-level features of a single view of the multi-loop view of each ring. The high-level features of a single view of the multi-loop view of each ring are then processed through a normalization layer.

[0079] Step S202: The high-level features of each ring in the multi-ring view after the normalization layer processing are average pooled to fuse information from different perspectives to form a fused feature. The fused feature is then input into the multi-ring view feature encoder block based on the Transformer architecture. The fused feature is further refined and fused using the multi-head self-attention and spatial cross-attention mechanisms. The image features of the fused feature are converted into a serialized form to obtain the feature vector of the 3D model.

[0080] Step S203: After being processed by several Transformer-based multi-loop view feature encoder blocks, the feature vectors of the 3D model are max-pooled to obtain a unified feature vector of the 3D model and further enhance the feature expression capability. Finally, the unified feature vector of the 3D model is input into a two-layer multi-layer perceptron for category prediction, ultimately obtaining the output category of the 3D model as the fine-grained category center.

[0081] Among them, the two-layer multilayer perceptron consists of a first input layer, a hidden layer and a second output layer. The layers of the first input layer, the hidden layer and the second output layer are connected by weights, and the nonlinear characteristics of the unified feature vector of the three-dimensional model are learned through a nonlinear activation function.

[0082] It is understandable that since the feature vector dimensions of the 3D model and the sketch may be inconsistent, they are input into the same two-layer multilayer perceptron network, and the first and second output layers of the two-layer multilayer perceptron network are designed with the same number of neurons to ensure that the feature vectors of the 3D model and the sketch can be mapped to the same vector space.

[0083] Specifically, the high-level features of a single view in step S201 are processed through a normalization layer, specifically in the following steps:

[0084] Step S2011: Euclidean norm normalizes the high-level features of a single view of the multi-ring view of each ring so that the high-level features of a single view of the multi-ring view of each ring are on the same scale. The formula is:

[0085] ;

[0086] Where: is the feature vector of high-level features; is the feature vector of the high-level features of the i-th multi-ring view; is the feature vector of the high-level features of the i-th multi-ring view Eigenvector after Euclidean norm normalization; is the feature vector of high-level features Dimensions;

[0087] Step S2012, calculating the discriminant loss function by comparing each Euclidean norm normalized feature vector with the class representation feature vector of each Euclidean norm normalized feature vector , the formula is:

[0088] ;

[0089] Where: is the discriminant loss function; is the total number of multi-ring views; To find the maximum function; is the margin, which controls the minimum distance between the feature vectors that are similar to the high-level features of the multi-ring view and the feature vectors that are similar but not similar to the high-level features of the multi-ring view; D is the cosine distance of the distance metric function; is a feature vector similar to the high-level features of the i-th multi-ring view; min is the minimum function; is the output category set of the predicted three-dimensional model; j is the category index, which is used to identify each category in the output category set c of the predicted three-dimensional model; y mi is the predicted category of the m-th 3D model for the i-th multi-ring view; is the feature vector that is dissimilar to the high-level features of the i-th multi-ring view;

[0090] Step S2013, calculate the cross entropy loss function by comparing the predicted probability distribution of the 3D model with the probability distribution of the true label , the formula is:

[0091] ;

[0092] Where, is the cross entropy loss function; e is the base of the natural logarithm; is the total number of output category sets of the predicted three-dimensional model; For the The true category of the multi-ring view; For the The true category of the multi-ring view The weight vector of is the bias term for the output category of the j-th predicted 3D model; For the The true category of the multi-ring view The bias term; is the weight vector of the output category of the j-th predicted 3D model;

[0093] Step S2014, final loss function is the discriminant loss function and cross entropy loss function The combination of, the formula is:

[0094] ;

[0095] Where, is the final loss function; is the cross entropy loss function of the three-dimensional model; is a hyperparameter; is the discriminant loss function.

[0096] Specifically, in step S202, the fused features are input into the multi-ring view feature encoder block based on the Transformer architecture. The specific steps are as follows:

[0097] Step S2021, linearly transforming the fused features through the first weight matrix, then transforming the fused features through the second weight matrix, and then linearly transforming the fused features through the third weight matrix to obtain an embedding vector;

[0098] Step S2022: Input the embedding vector into the multi-head attention mechanism. For each head in the multi-head attention mechanism, calculate the dot product between the query vector Q and the key vector K, then normalize it through the cross entropy loss function, and finally multiply it by the sequence length to obtain the output of each head in the multi-head attention mechanism.

[0099] Step S2023, concatenate the output of each head in the multi-head attention mechanism in step S2022 into one, and output the weight matrix Integrate to obtain the splicing output of the multi-head attention mechanism, the formula is:

[0100] ;

[0101] Where, is a multi-head attention mechanism; Q, K, and V are the feature vectors of high-level features respectively The query vector, key vector, and value vector obtained by linear mapping; Concat is a concatenation operation; head1 is the output of the first attention head in the multi-head attention mechanism, head2 is the output of the second attention head in the multi-head attention mechanism, and headh is the output of the hth attention head in the multi-head attention mechanism;

[0102] In step S2024, the concatenated output of the multi-head attention mechanism is processed by a normalization layer and then input into the spatial cross attention mechanism to capture the long-range interaction of the fused features in the spatial dimension. The formula is:

[0103] ;

[0104] Where, is the spatial cross attention mechanism; softmax is the normalized exponential function; d k is the feature vector of high-level features The dimension of the key vector K obtained by linear mapping; is the feature vector of high-level features The transpose of the key vector K matrix obtained by linear mapping;

[0105] Step S2025, finally input the output processed in the spatial cross attention mechanism Feature vectors with high-level features Perform residual connection and apply normalization layer, the formula is:

[0106] ;

[0107] Where, is the normalization layer; is the final output, which is the sum of the cross-attention output after the normalization layer and the input of the residual connection.

[0108] Specifically, step S3 includes the following steps:

[0109] Step S301: A fine-grained sketch clustering network is formed by a feature extraction module with a second ResNet-50 as the backbone network, a fully connected classifier, and a two-layer multilayer perceptron. The 3D model representation and the sketch data subset are input into the fine-grained sketch clustering network. Each sketch in the sketch data subset is passed through the feature extraction module with the second ResNet-50 as the backbone network to obtain a feature vector of the sketch.

[0110] Step S302: The feature vector of the sketch enters the fully connected classifier and the two-layer multilayer perceptron in turn, and then undergoes Euclidean norm normalization. Then, by minimizing the correlation loss function, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category to obtain the three-dimensional model retrieval result.

[0111] Specifically, in step S302, the feature vector of the sketch is aligned with the unified feature vector of the 3D model at the center of the fine-grained category. The specific steps are:

[0112] Step S3021: Calculate the distance between the feature vector of each sketch and the corresponding fine-grained category center, and pull the feature vector of the sketch closer to the nearest fine-grained category center, and minimize the correlation loss function. The formula is:

[0113] ;

[0114] Where, is the minimum correlation loss function; is the total number of sketches; is the correct category of the p-th sketch; is the correct category of the p-th sketch the number of associated fine-grained categories; is the feature vector of the p-th sketch; is the correct category of the p-th sketch Fine-grained category center of is the index in the correct sketch set;

[0115] Fine-grained sketch clustering network combined with cross entropy loss function and minimum correlation loss function Perform joint supervised training and calculate the joint loss function , where the cross entropy loss function The formula is:

[0116] ;

[0117] Where, is the cross entropy loss function; is the correct category of the p-th sketch The weight vector of is the feature vector of the p-th sketch; Pass model parameters for the p-th sketch The transformed feature vector; is the correct category label of the p-th sketch The bias term; is the total number of sketch categories; is the weight vector of the oth sketch category; b o is the bias term of the o-th sketch category;

[0118] Joint loss function The formula is:

[0119] ;

[0120] Where, is the joint loss function; Lm is the cross entropy loss function of the sketch; is the hyperparameter of the sketch; L c is the minimum correlation loss function.

[0121] Example 1:

[0122] The public dataset SHREC'13 dataset is selected:

[0123] The SHREC'13 dataset contains 7,200 sketch images and 1,258 3D models across 90 different categories. For the sketch data, each category is allocated 50 samples for training and 30 samples for testing. For the 3D models, the images are randomly divided into 80% for training and 20% for testing.

[0124] This example extracts 50 sketches from each category in the SHREC'13 dataset as sketch training data. All 3D models from each category are used as training 3D model data to train a multi-loop view feature learning network and a fine-grained sketch clustering network. An additional 30 sketches from each category are selected as query sketches to evaluate the effectiveness of the retrieval method proposed in this paper.

[0125] Enter the "motorbike" (motorcycle) sketch query, and the search results are the first five Figure 2 Although the fifth retrieval result is incorrectly identified as a “race car”, the first four retrieval results are all correct “motorbike” 3D models because the “motorbike” category in the SHREC’13 dataset only contains four 3D models.

[0126] It can be seen that there is a high similarity between the query sketch and the retrieved 3D model.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D model retrieval method based on multi-ring views and latent class mining, characterized in that: The following steps are involved: Step S1: Select a data set, which includes several 3D models and sketches; Using 3D imaging software, a virtual camera is configured with each 3D model in the dataset as the center. The virtual camera rotates around the Y axis at equal angles and takes pictures. At each angle, the virtual camera is then displaced along the Z axis at equal distances and takes pictures, generating a multi-ring view. Specifically: Step S101: The dataset selected is the SHREC'13 sketch 3D model retrieval dataset, which includes a sketch data subset and a 3D model data subset; Step S102: Use the 3D graphics software Blender to construct a rectangular empty background model and configure a virtual camera. Select the center point of the rectangular empty background model as the center, set the light source at the edge of the cube of the rectangular empty background model, define a virtual sphere with the 3D model as the center, and rotate the virtual camera around the Y axis at equal angles and shoot. At each angle, the virtual camera is then displaced along the Z axis at equal distances and shoots, thereby generating a multi-ring view. Set the coordinates of the virtual camera to: ； ； ； Where X is the coordinate of the virtual camera on the x-axis; r is the radius of the virtual sphere; is the angle of the virtual camera relative to the x-axis; Y is the coordinate of the virtual camera on the y-axis; Z is the coordinate of the virtual camera on the z-axis; Z0 is the position of the virtual camera on the z-axis; Step S2: After the multi-ring view is processed by the multi-ring view feature learning network, a three-dimensional model representation is obtained; specifically: Step S201: A multi-loop view feature learning network is composed of a feature extraction module with the first ResNet-50 as the backbone network, a multi-loop view feature encoder block based on the Transformer architecture, and a two-layer multi-layer perceptron. The multi-loop view is input into the multi-loop view feature learning network, and the multi-loop view of each ring is independently passed through the feature extraction module with the first ResNet-50 as the backbone network to capture high-level features of a single view of the multi-loop view of each ring. The high-level features of a single view of the multi-loop view of each ring are then processed through a normalization layer. Step S202: The high-level features of each ring in the multi-ring view after the normalization layer processing are average pooled to fuse information from different perspectives to form a fused feature. The fused feature is then input into the multi-ring view feature encoder block based on the Transformer architecture. The fused feature is further refined and fused using the multi-head self-attention and spatial cross-attention mechanisms. The image features of the fused feature are converted into a serialized form to obtain the feature vector of the 3D model. Step S203: After being processed by several Transformer-based multi-loop view feature encoder blocks, the feature vectors of the 3D model are max-pooled to obtain a unified feature vector of the 3D model and further enhance the feature expression capability. Finally, the unified feature vector of the 3D model is input into a two-layer multi-layer perceptron for category prediction, ultimately obtaining the output category of the 3D model as the fine-grained category center. The two-layer multilayer perceptron consists of a first input layer, a hidden layer, and a second output layer. The first input layer, the hidden layer, and the second output layer are connected by weights, and the nonlinear features of the unified feature vector of the three-dimensional model are learned through a nonlinear activation function. Step S3: Under the guidance of the 3D model representation, the sketches in the dataset are cross-modally learned using a fine-grained sketch clustering network to share common features between sketches and 3D models. The features of the sketches are aligned with those of the 3D models to obtain 3D model retrieval results. Specifically: Step S301: A fine-grained sketch clustering network is formed by a feature extraction module with a second ResNet-50 as the backbone network, a fully connected classifier, and a two-layer multilayer perceptron. The 3D model representation and the sketch data subset are input into the fine-grained sketch clustering network. Each sketch in the sketch data subset is passed through the feature extraction module with the second ResNet-50 as the backbone network to obtain a feature vector of the sketch. Step S302: The feature vector of the sketch enters the fully connected classifier and the two-layer multilayer perceptron in turn, and then undergoes Euclidean norm normalization. Then, by minimizing the correlation loss function, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category to obtain the three-dimensional model retrieval result.

2. The 3D model retrieval method based on multi-ring views and latent class mining according to claim 1, characterized in that: In step S102, the virtual camera is rotated around the Y axis at equal angles, and a photo is taken at each 30-degree rotation position for a total of 12 times. For each 30-degree rotation position around the Y axis, the virtual camera is then displaced along the Z axis to six equidistant points and a photo is taken at each equidistant point. Finally, 72 multi-ring views are generated.

3. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 2, characterized in that: In step S201, the high-level features of a single view are processed through a normalization layer. The specific steps are as follows: Step S2011: Euclidean norm normalizes the high-level features of a single view of the multi-ring view of each ring so that the high-level features of a single view of the multi-ring view of each ring are on the same scale. The formula is: ； Where: is the feature vector of high-level features; is the feature vector of the high-level features of the i-th multi-ring view; is the feature vector of the high-level features of the i-th multi-ring view Eigenvector after Euclidean norm normalization; is the feature vector of high-level features Dimensions; Step S2012, calculating the discriminant loss function by comparing each Euclidean norm normalized feature vector with the class representation feature vector of each Euclidean norm normalized feature vector , the formula is: ； Where: is the discriminant loss function; is the total number of multi-ring views; To find the maximum function; is the margin, which controls the minimum distance between the feature vectors that are similar to the high-level features of the multi-ring view and the feature vectors that are similar but not similar to the high-level features of the multi-ring view; D is the cosine distance of the distance metric function; is a feature vector similar to the high-level features of the i-th multi-ring view; min is the minimum function; is the output category set of the predicted three-dimensional model; j is the category index, which is used to identify each category in the output category set c of the predicted three-dimensional model; y mi is the predicted category of the m-th 3D model for the i-th multi-ring view; is the feature vector that is dissimilar to the high-level features of the i-th multi-ring view; Step S2013, calculate the cross entropy loss function by comparing the predicted probability distribution of the 3D model with the probability distribution of the true label , the formula is: ； Where, is the cross entropy loss function; e is the base of the natural logarithm; is the total number of output category sets of the predicted three-dimensional model; For the The true category of the multi-ring view; For the The true category of the multi-ring view The weight vector of is the bias term for the output category of the j-th predicted 3D model; For the The true category of the multi-ring view The bias term; is the weight vector of the output category of the j-th predicted 3D model; Step S2014, final loss function is the discriminant loss function and cross entropy loss function The combination of, the formula is: ； Where, is the final loss function; is the cross entropy loss function of the three-dimensional model; is a hyperparameter; is the discriminant loss function.

4. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 3, characterized in that: In step S202, the fused features are input into the multi-ring view feature encoder block based on the Transformer architecture. The specific steps are as follows: Step S2021, linearly transforming the fused features through the first weight matrix, then transforming the fused features through the second weight matrix, and then linearly transforming the fused features through the third weight matrix to obtain an embedding vector; Step S2022: Input the embedding vector into the multi-head attention mechanism. For each head in the multi-head attention mechanism, calculate the dot product between the query vector Q and the key vector K, then normalize it through the cross entropy loss function, and finally multiply it by the sequence length to obtain the output of each head in the multi-head attention mechanism. Step S2023, concatenate the output of each head in the multi-head attention mechanism in step S2022 into one, and output the weight matrix Integrate to obtain the splicing output of the multi-head attention mechanism, the formula is: ； Where, is a multi-head attention mechanism; Q, K, and V are the feature vectors of high-level features respectively The query vector, key vector, and value vector obtained by linear mapping; Concat is a concatenation operation; head1 is the output of the first attention head in the multi-head attention mechanism, head2 is the output of the second attention head in the multi-head attention mechanism, and head h is the output of the hth attention head in the multi-head attention mechanism; In step S2024, the concatenated output of the multi-head attention mechanism is processed by a normalization layer and then input into the spatial cross attention mechanism to capture the long-range interaction of the fused features in the spatial dimension. The formula is: ； Where, is the spatial cross attention mechanism; softmax is the normalized exponential function; d k is the feature vector of high-level features The dimension of the key vector K obtained by linear mapping; is the feature vector of high-level features The transpose of the key vector K matrix obtained by linear mapping; Step S2025, finally input the output processed in the spatial cross attention mechanism Feature vectors with high-level features Perform residual connection and apply normalization layer, the formula is: ； Where, is the normalization layer; is the final output, which is the sum of the cross-attention output after the normalization layer and the input of the residual connection.

5. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 4, characterized in that: In step S302, the feature vector of the sketch is aligned with the unified feature vector of the 3D model at the center of the fine-grained category. The specific steps are as follows: Step S3021: Calculate the distance between the feature vector of each sketch and the corresponding fine-grained category center, and pull the feature vector of the sketch closer to the nearest fine-grained category center, and minimize the correlation loss function. The formula is: ； Where, is the minimum correlation loss function; is the total number of sketches; is the correct category of the p-th sketch; is the correct category of the p-th sketch the number of associated fine-grained categories; is the feature vector of the p-th sketch; is the correct category of the p-th sketch Fine-grained category center of is the index in the correct sketch set; Fine-grained sketch clustering network combined with cross entropy loss function and minimum correlation loss function Perform joint supervised training and calculate the joint loss function , where the cross entropy loss function The formula is: ； Where, is the cross entropy loss function; is the correct category of the p-th sketch The weight vector of is the feature vector of the p-th sketch; Pass model parameters for the p-th sketch The transformed feature vector; is the correct category label of the p-th sketch The bias term; is the total number of sketch categories; is the weight vector of the oth sketch category; b o is the bias term of the o-th sketch category; Joint loss function The formula is: ； Where, is the joint loss function; L m is the cross entropy loss function of the sketch; is the hyperparameter of the sketch; L c is the minimum correlation loss function.

Images