Three-dimensional model retrieval method based on multi-ring view and hidden class mining

Abstract

The invention relates to the technical field of three-dimensional model retrieval, and provides a three-dimensional model retrieval method based on multi-view and hidden class mining. According to the method, a data set is selected, a virtual camera is configured with a three-dimensional model in the data set as the center, the virtual camera rotates around the Y axis at equal angles and then displaces along the Z axis at equal intervals and shoots, and a multi-ring view is generated jointly; after the multi-ring view is subjected to multi-ring view feature learning network learning processing, three-dimensional model representation is obtained; the sketch in the data set is processed through a fine-grained sketch clustering network, features of the sketch are aligned with features of the three-dimensional model, and a three-dimensional model retrieval result is obtained. According to the scheme of the invention, the enhancement technology of the multi-view feature learning network can capture the features of the three-dimensional model from multiple angles, so as to enhance the understanding of the model on the three-dimensional shape and more accurately identify and match the three-dimensional model in the sketch; and the hidden class mining technology of the fine-grained sketch clustering network further improves the performance of retrieving the three-dimensional model by the sketch.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional model retrieval, and in particular relates to a three-dimensional model retrieval method based on multi-ring views and latent class mining. Background Art

[0002] In the digital age, 3D models have become key information carriers due to their rich information expression capabilities. With the rapid advancement of 3D modeling technology, creating 3D models has become more convenient. Faced with the surge in the number of 3D models, how to efficiently and accurately retrieve the specific models required by users has become an urgent problem that needs to be solved. Sketch-based 3D model retrieval technology has emerged, which allows users to quickly locate and retrieve target models anytime and anywhere through simple hand-drawn sketches.

[0003] Sketch-based 3D model retrieval technology has undergone a transformation from manual feature extraction to deep learning. Early research relied on manual features and was inefficient. Then deep learning was applied to deal with cross-domain issues through twin networks, but it was limited by ignoring the stereoscopic nature of 3D models. Subsequently, a method based on holistic metric learning was proposed to improve performance by using discriminative and correlation losses, but the training process was relatively complicated. In recent years, the performance of sketch-based 3D model retrieval has been significantly improved through knowledge distillation models and cross-domain guided training. However, this field still faces challenges, such as the semantic differences between sketches and 3D models, the abstractness of sketches, and the differences in the importance of view features.

[0004] In the field of 2D view generation and sketch-based retrieval of 3D models, existing methods face some challenges. First, the modality difference between sketches and 3D objects creates a significant domain gap, which is an obstacle to feature extraction and matching. Second, the 3D model is projected into a 2D view with limited viewing angles, which leads to certain limitations in expressing the 3D model. Summary of the invention

[0005] The present invention provides a three-dimensional model retrieval method based on multi-ring views and latent class mining, which can improve the technical problems in the related technology that the modal difference between the sketch and the three-dimensional object causes a significant domain gap, hinders the extraction and matching of features, and the viewing angle of the three-dimensional model projected into a two-dimensional view is limited, resulting in certain limitations of the two-dimensional view in expressing the three-dimensional model.

[0006] The present invention provides a three-dimensional model retrieval method based on multi-ring views and latent class mining, comprising the following steps: Step S1: Select a data set, which includes several three-dimensional models and sketches; use three-dimensional image software to configure a virtual camera with each three-dimensional model in the data set as the center, and the virtual camera rotates around the Y axis at equal angles and shoots, and the virtual camera at each angle is displaced along the Z axis at equal distances and shoots, and a multi-ring view is generated together; Step S2: After the multi-ring view is processed by the multi-ring view feature learning network, a three-dimensional model representation is obtained; Step S3: Under the guidance of the 3D model representation, the sketches in the dataset are cross-modally learned through a fine-grained sketch clustering network to share the common features of sketches and 3D models, align the features of the sketches with those of the 3D models, and obtain the 3D model retrieval results.

[0007] Furthermore, step S1 includes the following steps: Step S101: the dataset selects the SHREC'13 sketch 3D model retrieval dataset, which includes a sketch data subset and a 3D model data subset; Step S102: Use Blender, a 3D graphics software, to construct a rectangular empty background model and configure a virtual camera. Select the center point of the rectangular empty background model as the center, set the light source at the edge of the cube of the rectangular empty background model, define a virtual sphere with the 3D model as the center for the 3D model, rotate the virtual camera around the Y axis at equal angles and shoot, and move the virtual camera at each angle along the Z axis at equal distances and shoot, and jointly generate a multi-ring view; set the coordinates of the virtual camera to: ; ; ; Where X is the coordinate of the virtual camera on the x-axis; r is the radius of the virtual sphere; is the angle of the virtual camera relative to the x-axis; Y is the coordinate of the virtual camera on the y-axis; Z is the coordinate of the virtual camera on the z-axis; Z0 is the position of the virtual camera on the z-axis.

[0008] Furthermore, in step S102, the virtual camera is rotated around the Y axis at equal angles, and photographed once at each position rotated 30 degrees for a total of 12 times. For each position rotated 30 degrees around the Y axis, the virtual camera is displaced along 6 equidistant points on the Z axis and photographed once at each equidistant point, and finally 72 multi-ring views are generated together.

[0009] Further, step S2 includes the following steps: Step S201: A multi-loop view feature learning network is composed of a feature extraction module with the first ResNet-50 as the backbone network, a multi-loop view feature encoder block based on the Transformer architecture, and a double-layer multi-layer perceptron; the multi-loop view is input into the multi-loop view feature learning network, and the multi-loop view of each ring is independently passed through the feature extraction module with the first ResNet-50 as the backbone network to capture the high-level features of a single perspective of the multi-loop view of each ring, and then the high-level features of a single perspective of the multi-loop view of each ring are processed by a normalization layer; Step S202: The high-level features of a single perspective of the multi-ring view of each ring after the normalization layer processing are average pooled to fuse the information of different perspectives to form a fused feature; the fused feature is then input into the multi-ring view feature encoder block based on the Transformer architecture, and the fused feature is further refined and fused using the multi-head self-attention and spatial cross-attention mechanisms, and the image features of the fused feature are converted into a serialized form to obtain a feature vector of the three-dimensional model; Step S203: After being processed by several multi-loop view feature encoder blocks based on the Transformer architecture, the feature vectors of the 3D model are integrated by maximum pooling to obtain a unified feature vector of the 3D model and further enhance the feature expression capability; finally, the unified feature vector of the 3D model is input into a double-layer multi-layer perceptron for category prediction, and finally the output category of the 3D model is obtained as the fine-grained category center; Among them, the double-layer multilayer perceptron consists of a first input layer, a hidden layer and a second output layer. The layers of the first input layer, the hidden layer and the second output layer are connected by weights, and the nonlinear characteristics of the unified feature vector of the three-dimensional model are learned through a nonlinear activation function.

[0010] Furthermore, the high-level features of a single view in step S201 are processed by a normalization layer, and the specific steps are as follows: Step S2011, normalize the high-level features of a single view of the multi-ring view of each ring by Euclidean norm, so that the high-level features of a single view of the multi-ring view of each ring are on the same scale, and the formula is: ; Where: is the feature vector of high-level features; is the feature vector of the high-level features of the i-th multi-ring view; is the feature vector of the high-level features of the i-th multi-ring view Eigenvector after Euclidean norm normalization; is the feature vector of high-level features Dimensions; Step S2012, calculating the discriminant loss function by comparing each Euclidean norm normalized feature vector with the class representation feature vector of each Euclidean norm normalized feature vector , the formula is: ; Where: is the discriminant loss function; is the total number of multi-ring views; To find the maximum function; is the margin, which controls the minimum distance between the feature vectors that are similar to the high-level features of the multi-ring view and the feature vectors that are similar but not similar to the high-level features of the multi-ring view; D is the distance metric function cosine distance; is a feature vector similar to the high-level features of the i-th multi-ring view; min is the minimum function; is the output category set of the predicted three-dimensional model; j is the category index, which is used to identify each category in the output category set c of the predicted three-dimensional model; y mi is the predicted category of the m-th 3D model for the i-th multi-ring view; is the feature vector that is dissimilar to the high-level features of the i-th multi-ring view; Step S2013, calculating the cross entropy loss function by comparing the predicted probability distribution of the 3D model with the probability distribution of the true label , the formula is: ; In the formula, is the cross entropy loss function; e is the base of the natural logarithm; is the total number of output category sets of the predicted three-dimensional model; For the The true category of the multi-ring view; For the The true category of the multi-ring view The weight vector of is the bias term for the output category of the j-th predicted 3D model; For the The true category of the multi-ring view The bias term of is the weight vector of the output category of the j-th predicted 3D model; Step S2014, final loss function is the discriminant loss function and the cross entropy loss function The combination of, the formula is: ; In the formula, is the final loss function; is the cross entropy loss function of the three-dimensional model; is a hyperparameter; is the discriminant loss function.

[0011] Furthermore, in step S202, the fused features are input into a multi-ring view feature encoder block based on the Transformer architecture, and the specific steps are as follows: Step S2021, linearly transforming the fused features through the first weight matrix, then transforming the fused features through the second weight matrix, and then linearly transforming the fused features through the third weight matrix; obtaining an embedding vector; Step S2022, input the embedding vector into the multi-head attention mechanism, for each head in the multi-head attention mechanism, calculate the dot product between the query vector Q and the key vector K, then normalize it through the cross entropy loss function, and finally multiply it by the sequence length to obtain the output of each head in the multi-head attention mechanism; Step S2023, concatenate the output of each head in the multi-head attention mechanism in step S2022 into one, and output the weight matrix Integrate to obtain the concatenated output of the multi-head attention mechanism, the formula is: ; In the formula, is a multi-head attention mechanism; Q, K, and V are the feature vectors of high-level features respectively The query vector, key vector, and value vector obtained by linear mapping; Concat is a concatenation operation; head1 is the output of the first attention head in the multi-head attention mechanism, head2 is the output of the second attention head in the multi-head attention mechanism, and head h is the output of the hth attention head in the multi-head attention mechanism; In step S2024, the concatenated output of the multi-head attention mechanism is processed by a normalization layer and then input into the spatial cross attention mechanism to capture the long-range interaction of the fused features in the spatial dimension. The formula is: ; In the formula, is the spatial cross attention mechanism; softmax is the normalized exponential function; d k is the feature vector of high-level features The dimension of the key vector K obtained by linear mapping; is the feature vector of high-level features The transpose of the key vector K matrix obtained by linear mapping; Step S2025, finally input the output processed in the spatial cross attention mechanism Feature vectors with high-level features Make a residual connection and apply a normalization layer, the formula is: ; In the formula, is the normalization layer; is the final output, which is the sum of the cross-attention output after the normalization layer and the input of the residual connection.

[0012] Further, step S3 includes the following steps: Step S301: a feature extraction module with the second ResNet-50 as the backbone network, a fully connected classifier, and a two-layer multilayer perceptron form a fine-grained sketch clustering network; the three-dimensional model representation and the sketch data subset are input into the fine-grained sketch clustering network, and each sketch in the sketch data subset is passed through the feature extraction module with the second ResNet-50 as the backbone network to obtain a feature vector of the sketch; Step S302: The feature vector of the sketch enters the fully connected classifier and the two-layer multilayer perceptron in turn, and then undergoes Euclidean norm normalization. Then, by minimizing the correlation loss function, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category to obtain the three-dimensional model retrieval result.

[0013] Furthermore, in step S302, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category, and the specific steps are as follows: Step S3021: Calculate the distance between the feature vector of each sketch and the corresponding fine-grained category center, and pull the feature vector of the sketch closer to the nearest fine-grained category center, the minimum correlation loss function The formula is: ; In the formula, is the minimum correlation loss function; is the total number of sketches; is the correct category of the p-th sketch; is the correct category for the pth sketch the number of associated fine-grained categories; is the feature vector of the p-th sketch; is the correct category of the pth sketch The fine-grained category center of is the index in the correct sketch set; Fine-grained sketch clustering network combined with cross entropy loss function And the minimum correlation loss function Perform joint supervised training and calculate the joint loss function , where the cross entropy loss function The formula is: ; In the formula, is the cross entropy loss function; is the correct category of the pth sketch The weight vector of is the feature vector of the p-th sketch; Pass model parameters for the pth sketch The transformed feature vector; is the correct category label of the p-th sketch The bias term of is the total number of sketch categories; is the weight vector of the oth sketch category; b o is the bias term of the o-th sketch category; Joint loss function The formula is: ; In the formula, is the joint loss function; L m is the cross entropy loss function of the sketch; is the hyperparameter of the sketch; L c is the minimum correlation loss function.

[0014] The present invention provides a 3D model retrieval method based on multi-ring views and latent class mining, which has at least the following technical effects: (1) Through multi-ring view enhancement technology, the features of the 3D model can be captured from multiple angles to enhance the model's understanding of the 3D shape. By constructing multiple ring views, each ring represents a different perspective, a comprehensive analysis of the 3D model can be achieved. The 3D shape can be observed and understood from different directions, capturing richer and more detailed feature information. This multi-angle feature extraction enables the retrieval dataset to more accurately identify and match the 3D model in the sketch when processing the sketch retrieval task, thereby improving the accuracy and efficiency of the retrieval.

[0015] (2) Through latent class mining technology, fine-grained subcategories that may exist in 3D models can be identified and mined. By applying clustering algorithms, similar features in 3D models are clustered together to form different categories. These fine-grained category representations help to better align the feature distributions of sketches and 3D models, allowing the retrieval dataset to more accurately understand and match the 3D model in the sketch. This not only improves the retrieval dataset's understanding of the internal structure of the 3D model, but also enhances the retrieval dataset's robustness when faced with complex and diverse sketch inputs, further improving the performance of sketch retrieval of 3D models. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings: Figure 1 A flowchart of a three-dimensional model retrieval method based on multi-ring views and latent class mining provided by an embodiment of the present invention; Figure 2 The present invention provides a specific example of a three-dimensional model retrieval result in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs; the terms used herein are only for the purpose of describing specific embodiments rather than limiting the present invention.

[0019] like Figure 1 As shown, an embodiment of the present invention provides a 3D model retrieval method based on multi-ring views and latent class mining, comprising the following steps: Step S1: Select a data set, which includes several three-dimensional models and sketches; use three-dimensional image software to configure a virtual camera with each three-dimensional model in the data set as the center, and the virtual camera rotates around the Y axis at equal angles and shoots, and the virtual camera at each angle is displaced along the Z axis at equal distances and shoots, and a multi-ring view is generated together; Step S2: After the multi-ring view is processed by the multi-ring view feature learning network, a three-dimensional model representation is obtained; Step S3: Under the guidance of the 3D model representation, the sketches in the dataset are cross-modally learned through a fine-grained sketch clustering network to share the common features of sketches and 3D models, align the features of the sketches with those of the 3D models, and obtain the 3D model retrieval results.

[0020] Specifically, step S1 includes the following steps: Step S101: the dataset selects the SHREC'13 sketch 3D model retrieval dataset, which includes a sketch data subset and a 3D model data subset; Step S102: Use Blender, a 3D graphics software, to construct a rectangular empty background model and configure a virtual camera. Select the center point of the rectangular empty background model as the center, set the light source at the edge of the cube of the rectangular empty background model, define a virtual sphere with the 3D model as the center for the 3D model, rotate the virtual camera around the Y axis at equal angles and shoot, and move the virtual camera at each angle along the Z axis at equal distances and shoot, and jointly generate a multi-ring view; set the coordinates of the virtual camera to: ; ; ; Where X is the coordinate of the virtual camera on the x-axis; r is the radius of the virtual sphere; is the angle of the virtual camera relative to the x-axis; Y is the coordinate of the virtual camera on the y-axis; Z is the coordinate of the virtual camera on the z-axis; Z0 is the position of the virtual camera on the z-axis.

[0021] Specifically, in step S102, the virtual camera is rotated around the Y axis at equal angles, and photographed once for every 30-degree rotation for a total of 12 times. For every 30-degree rotation around the Y axis, the virtual camera is displaced to 6 equidistant points along the Z axis and photographed once at each equidistant point, and finally 72 multi-ring views are generated together.

[0022] Specifically, step S2 includes the following steps: Step S201: A multi-loop view feature learning network is composed of a feature extraction module with the first ResNet-50 as the backbone network, a multi-loop view feature encoder block based on the Transformer architecture, and a double-layer multi-layer perceptron; the multi-loop view is input into the multi-loop view feature learning network, and the multi-loop view of each ring is independently passed through the feature extraction module with the first ResNet-50 as the backbone network to capture the high-level features of a single perspective of the multi-loop view of each ring, and then the high-level features of a single perspective of the multi-loop view of each ring are processed by a normalization layer; Step S202: The high-level features of a single perspective of the multi-ring view of each ring after the normalization layer processing are average pooled to fuse the information of different perspectives to form a fused feature; the fused feature is then input into the multi-ring view feature encoder block based on the Transformer architecture, and the fused feature is further refined and fused using the multi-head self-attention and spatial cross-attention mechanisms, and the image features of the fused feature are converted into a serialized form to obtain a feature vector of the three-dimensional model; Step S203: After being processed by several multi-loop view feature encoder blocks based on the Transformer architecture, the feature vectors of the 3D model are integrated by maximum pooling to obtain a unified feature vector of the 3D model and further enhance the feature expression capability; finally, the unified feature vector of the 3D model is input into a double-layer multi-layer perceptron for category prediction, and finally the output category of the 3D model is obtained as the fine-grained category center; Among them, the double-layer multilayer perceptron consists of a first input layer, a hidden layer and a second output layer. The layers of the first input layer, the hidden layer and the second output layer are connected by weights, and the nonlinear characteristics of the unified feature vector of the three-dimensional model are learned through a nonlinear activation function.

[0023] It can be understood that since the dimensions of the feature vectors of the 3D model and the sketch may be inconsistent, they are input into the same two-layer multilayer perceptron network, and the first output layer and the second output layer of the two-layer multilayer perceptron network are designed with the same number of neurons to ensure that the feature vectors of the 3D model and the sketch can be mapped to the same vector space.

[0024] Specifically, the high-level features of a single view in step S201 are processed by a normalization layer, and the specific steps are: Step S2011, normalize the high-level features of a single view of the multi-ring view of each ring by Euclidean norm, so that the high-level features of a single view of the multi-ring view of each ring are on the same scale, and the formula is: ; Where: is the feature vector of high-level features; is the feature vector of the high-level features of the i-th multi-ring view; is the feature vector of the high-level features of the i-th multi-ring view Eigenvector after Euclidean norm normalization; is the feature vector of high-level features Dimensions; Step S2012, calculating the discriminant loss function by comparing each Euclidean norm normalized feature vector with the class representation feature vector of each Euclidean norm normalized feature vector , the formula is: ; Where: is the discriminant loss function; is the total number of multi-ring views; To find the maximum function; is the margin, which controls the minimum distance between the feature vectors that are similar to the high-level features of the multi-ring view and the feature vectors that are similar but not similar to the high-level features of the multi-ring view; D is the distance metric function cosine distance; is a feature vector similar to the high-level features of the i-th multi-ring view; min is the minimum function; is the output category set of the predicted three-dimensional model; j is the category index, which is used to identify each category in the output category set c of the predicted three-dimensional model; y mi is the predicted category of the m-th 3D model for the i-th multi-ring view; is the feature vector that is dissimilar to the high-level features of the i-th multi-ring view; Step S2013, calculating the cross entropy loss function by comparing the predicted probability distribution of the 3D model with the probability distribution of the true label , the formula is: ; In the formula, is the cross entropy loss function; e is the base of the natural logarithm; is the total number of output category sets of the predicted three-dimensional model; For the The true category of the multi-ring view; For the The true category of the multi-ring view The weight vector of is the bias term for the output category of the j-th predicted 3D model; For the The true category of the multi-ring view The bias term of is the weight vector of the output category of the j-th predicted 3D model; Step S2014, final loss function is the discriminant loss function and the cross entropy loss function The combination of, the formula is: ; In the formula, is the final loss function; is the cross entropy loss function of the three-dimensional model; is a hyperparameter; is the discriminant loss function.

[0025] Specifically, in step S202, the fused features are input into a multi-ring view feature encoder block based on the Transformer architecture, and the specific steps are as follows: Step S2021, linearly transforming the fused features through the first weight matrix, then transforming the fused features through the second weight matrix, and then linearly transforming the fused features through the third weight matrix; obtaining an embedding vector; Step S2022, input the embedding vector into the multi-head attention mechanism, for each head in the multi-head attention mechanism, calculate the dot product between the query vector Q and the key vector K, then normalize it through the cross entropy loss function, and finally multiply it by the sequence length to obtain the output of each head in the multi-head attention mechanism; Step S2023, concatenate the output of each head in the multi-head attention mechanism in step S2022 into one, and output the weight matrix Integrate to obtain the concatenated output of the multi-head attention mechanism, the formula is: ; In the formula, is a multi-head attention mechanism; Q, K, and V are the feature vectors of high-level features respectively The query vector, key vector, and value vector obtained by linear mapping; Concat is a concatenation operation; head1 is the output of the first attention head in the multi-head attention mechanism, head2 is the output of the second attention head in the multi-head attention mechanism, and head h is the output of the hth attention head in the multi-head attention mechanism; In step S2024, the concatenated output of the multi-head attention mechanism is processed by a normalization layer and then input into the spatial cross attention mechanism to capture the long-range interaction of the fused features in the spatial dimension. The formula is: ; In the formula, is the spatial cross attention mechanism; softmax is the normalized exponential function; d k is the feature vector of high-level features The dimension of the key vector K obtained by linear mapping; is the feature vector of high-level features The transpose of the key vector K matrix obtained by linear mapping; Step S2025, finally input the output processed in the spatial cross attention mechanism Feature vectors with high-level features Make a residual connection and apply a normalization layer, the formula is: ; In the formula, is the normalization layer; is the final output, which is the sum of the cross-attention output after the normalization layer and the input of the residual connection.

[0026] Specifically, step S3 includes the following steps: Step S301: a feature extraction module with the second ResNet-50 as the backbone network, a fully connected classifier, and a two-layer multilayer perceptron form a fine-grained sketch clustering network; the three-dimensional model representation and the sketch data subset are input into the fine-grained sketch clustering network, and each sketch in the sketch data subset is passed through the feature extraction module with the second ResNet-50 as the backbone network to obtain a feature vector of the sketch; Step S302: The feature vector of the sketch enters the fully connected classifier and the two-layer multilayer perceptron in turn, and then undergoes Euclidean norm normalization. Then, by minimizing the correlation loss function, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category to obtain the three-dimensional model retrieval result.

[0027] Specifically, in step S302, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category. The specific steps are: Step S3021: Calculate the distance between the feature vector of each sketch and the corresponding fine-grained category center, and pull the feature vector of the sketch closer to the nearest fine-grained category center, the minimum correlation loss function The formula is: ; In the formula, is the minimum correlation loss function; is the total number of sketches; is the correct category of the p-th sketch; is the correct category for the pth sketch the number of associated fine-grained categories; is the feature vector of the p-th sketch; is the correct category of the pth sketch The fine-grained category center of is the index in the correct sketch set; Fine-grained sketch clustering network combined with cross entropy loss function And the minimum correlation loss function Perform joint supervised training and calculate the joint loss function , where the cross entropy loss function The formula is: ; In the formula, is the cross entropy loss function; is the correct category of the pth sketch The weight vector of is the feature vector of the p-th sketch; Pass model parameters for the pth sketch The transformed feature vector; is the correct category label of the p-th sketch The bias term of is the total number of sketch categories; is the weight vector of the oth sketch category; b o is the bias term of the o-th sketch category; Joint loss function The formula is: ; In the formula, is the joint loss function; L m is the cross entropy loss function of the sketch; is the hyperparameter of the sketch; L c is the minimum correlation loss function.

[0028] Embodiment 1: The public dataset SHREC'13 dataset is selected: The SHREC'13 dataset contains 7,200 sketch images and 1,258 3D models, covering 90 different categories. In the sketch data, each category is allocated 50 samples for training and 30 samples for testing; in the 3D models, 80% are used for training and 20% are used for testing.

[0029] This embodiment extracts 50 sketches of each category from the SHREC'13 dataset as sketch training data, and uses all 3D models of each category as training 3D model data for training the multi-loop view feature learning network and the fine-grained sketch clustering network. An additional 30 sketches are selected from each category as query sketches to evaluate the effectiveness of the retrieval method provided by the present invention.

[0030] Enter the "motorbike" (motorcycle) sketch query, and the search results are the first five Figure 2 Although the fifth retrieval result is incorrectly identified as a “race car”, since the “motorbike” category in the SHREC’13 dataset only contains 4 3D models, the first 4 retrieval results are all correct “motorbike” 3D models.

[0031] It can be seen that there is a high similarity between the query sketch and the retrieved 3D model.

[0032] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D model retrieval method based on multi-loop view and latent class mining, characterized in that: The following steps are involved: Step S1: Select a data set, which includes several three-dimensional models and sketches; Using 3D imaging software, a virtual camera is configured with each 3D model in the data set as the center. The virtual camera rotates around the Y axis at equal angles and takes pictures. The virtual camera at each angle is then displaced along the Z axis at equal distances and takes pictures, thus generating a multi-ring view. Step S2: After the multi-ring view is processed by the multi-ring view feature learning network, a three-dimensional model representation is obtained; Step S3: Under the guidance of the 3D model representation, the sketches in the dataset are cross-modally learned through a fine-grained sketch clustering network to share the common features of sketches and 3D models, align the features of the sketches with those of the 3D models, and obtain the 3D model retrieval results.

2. The three-dimensional model retrieval method based on multi-ring view and latent class mining according to claim 1, characterized in that: Step S1 includes the following steps: Step S101: the dataset selects the SHREC'13 sketch 3D model retrieval dataset, which includes a sketch data subset and a 3D model data subset; Step S102: Use Blender, a 3D graphics software, to construct a rectangular empty background model and configure a virtual camera. Select the center point of the rectangular empty background model as the center, set the light source at the edge of the cube of the rectangular empty background model, define a virtual sphere with the 3D model as the center for the 3D model, rotate the virtual camera around the Y axis at equal angles and shoot, and move the virtual camera at each angle along the Z axis at equal distances and shoot, and jointly generate a multi-ring view; set the coordinates of the virtual camera to: ； ； ； Where X is the coordinate of the virtual camera on the x-axis; r is the radius of the virtual sphere; is the angle of the virtual camera relative to the x-axis; Y is the coordinate of the virtual camera on the y-axis; Z is the coordinate of the virtual camera on the z-axis; Z0 is the position of the virtual camera on the z-axis.

3. The 3D model retrieval method based on multi-ring views and latent class mining according to claim 2, characterized in that: In step S102, the virtual camera is rotated around the Y axis at equal angles, and photographed once for every 30-degree rotation, for a total of 12 times. For every 30-degree rotation around the Y axis, the virtual camera is displaced along 6 equidistant points on the Z axis and photographed once at each equidistant point, and finally 72 multi-ring views are generated together.

4. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 3, characterized in that: Step S2 includes the following steps: Step S201: A multi-loop view feature learning network is composed of a feature extraction module with the first ResNet-50 as the backbone network, a multi-loop view feature encoder block based on the Transformer architecture, and a double-layer multi-layer perceptron; the multi-loop view is input into the multi-loop view feature learning network, and the multi-loop view of each ring is independently passed through the feature extraction module with the first ResNet-50 as the backbone network to capture the high-level features of a single perspective of the multi-loop view of each ring, and then the high-level features of a single perspective of the multi-loop view of each ring are processed by a normalization layer; Step S202: The high-level features of a single perspective of the multi-ring view of each ring after the normalization layer processing are average pooled to fuse the information of different perspectives to form a fused feature; the fused feature is then input into the multi-ring view feature encoder block based on the Transformer architecture, and the fused feature is further refined and fused using the multi-head self-attention and spatial cross-attention mechanisms, and the image features of the fused feature are converted into a serialized form to obtain a feature vector of the three-dimensional model; Step S203: After being processed by several multi-loop view feature encoder blocks based on the Transformer architecture, the feature vectors of the 3D model are integrated by maximum pooling to obtain a unified feature vector of the 3D model and further enhance the feature expression capability; finally, the unified feature vector of the 3D model is input into a double-layer multi-layer perceptron for category prediction, and finally the output category of the 3D model is obtained as the fine-grained category center; Among them, the double-layer multilayer perceptron consists of a first input layer, a hidden layer and a second output layer. The layers of the first input layer, the hidden layer and the second output layer are connected by weights, and the nonlinear characteristics of the unified feature vector of the three-dimensional model are learned through a nonlinear activation function.

5. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 4, characterized in that: In step S201, the high-level features of a single view are processed by a normalization layer, and the specific steps are as follows: Step S2011, normalize the high-level features of a single view of the multi-ring view of each ring by Euclidean norm, so that the high-level features of a single view of the multi-ring view of each ring are on the same scale, and the formula is: ； Where: is the feature vector of high-level features; is the feature vector of the high-level features of the i-th multi-ring view; is the feature vector of the high-level features of the i-th multi-ring view Eigenvector after Euclidean norm normalization; is the feature vector of high-level features Dimensions; Step S2012, calculating the discriminant loss function by comparing each Euclidean norm normalized feature vector with the class representation feature vector of each Euclidean norm normalized feature vector , the formula is: ； Where: is the discriminant loss function; is the total number of multi-ring views; To find the maximum function; is the margin, which controls the minimum distance between the feature vectors that are similar to the high-level features of the multi-ring view and the feature vectors that are similar but not similar to the high-level features of the multi-ring view; D is the distance metric function cosine distance; is a feature vector similar to the high-level features of the i-th multi-ring view; min is the minimum function; is the output category set of the predicted three-dimensional model; j is the category index, which is used to identify each category in the output category set c of the predicted three-dimensional model; y mi is the predicted category of the m-th 3D model for the i-th multi-ring view; is the feature vector that is dissimilar to the high-level features of the i-th multi-ring view; Step S2013, calculating the cross entropy loss function by comparing the predicted probability distribution of the 3D model with the probability distribution of the true label , the formula is: ； In the formula, is the cross entropy loss function; e is the base of the natural logarithm; is the total number of output category sets of the predicted three-dimensional model; For the The true category of the multi-ring view; For the The true category of the multi-ring view The weight vector of is the bias term for the output category of the j-th predicted 3D model; For the The true category of the multi-ring view The bias term of is the weight vector of the output category of the j-th predicted 3D model; Step S2014, final loss function is the discriminant loss function and the cross entropy loss function The combination of, the formula is: ； In the formula, is the final loss function; is the cross entropy loss function of the three-dimensional model; is a hyperparameter; is the discriminant loss function.

6. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 5, characterized in that: In step S202, the fused features are input into a multi-ring view feature encoder block based on the Transformer architecture, and the specific steps are as follows: Step S2021, linearly transforming the fused features through the first weight matrix, then transforming the fused features through the second weight matrix, and then linearly transforming the fused features through the third weight matrix; obtaining an embedding vector; Step S2022, input the embedding vector into the multi-head attention mechanism, for each head in the multi-head attention mechanism, calculate the dot product between the query vector Q and the key vector K, then normalize it through the cross entropy loss function, and finally multiply it by the sequence length to obtain the output of each head in the multi-head attention mechanism; Step S2023, concatenate the output of each head in the multi-head attention mechanism in step S2022 into one, and output the weight matrix Integrate to obtain the concatenated output of the multi-head attention mechanism, the formula is: ； In the formula, is a multi-head attention mechanism; Q, K, and V are the feature vectors of high-level features respectively The query vector, key vector, and value vector obtained by linear mapping; Concat is a concatenation operation; head1 is the output of the first attention head in the multi-head attention mechanism, head2 is the output of the second attention head in the multi-head attention mechanism, and head h is the output of the hth attention head in the multi-head attention mechanism; In step S2024, the concatenated output of the multi-head attention mechanism is processed by a normalization layer and then input into the spatial cross attention mechanism to capture the long-range interaction of the fused features in the spatial dimension. The formula is: ； In the formula, is the spatial cross attention mechanism; softmax is the normalized exponential function; d k is the feature vector of high-level features The dimension of the key vector K obtained by linear mapping; is the feature vector of high-level features The transpose of the key vector K matrix obtained by linear mapping; Step S2025, finally input the output processed in the spatial cross attention mechanism Feature vectors with high-level features Make a residual connection and apply a normalization layer, the formula is: ； In the formula, is the normalization layer; is the final output, which is the sum of the cross-attention output after the normalization layer and the input of the residual connection.

7. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 6, characterized in that: Step S3 includes the following steps: Step S301: a feature extraction module with the second ResNet-50 as the backbone network, a fully connected classifier, and a two-layer multilayer perceptron form a fine-grained sketch clustering network; the three-dimensional model representation and the sketch data subset are input into the fine-grained sketch clustering network, and each sketch in the sketch data subset is passed through the feature extraction module with the second ResNet-50 as the backbone network to obtain a feature vector of the sketch; Step S302: The feature vector of the sketch enters the fully connected classifier and the two-layer multilayer perceptron in turn, and then undergoes Euclidean norm normalization. Then, by minimizing the correlation loss function, the feature vector of the sketch is aligned with the unified feature vector of the three-dimensional model at the center of the fine-grained category to obtain the three-dimensional model retrieval result.

8. The three-dimensional model retrieval method based on multi-ring views and latent class mining according to claim 7, characterized in that: In step S302, the feature vector of the sketch is aligned with the unified feature vector of the 3D model at the center of the fine-grained category. The specific steps are as follows: Step S3021: Calculate the distance between the feature vector of each sketch and the corresponding fine-grained category center, and pull the feature vector of the sketch closer to the nearest fine-grained category center, the minimum correlation loss function The formula is: ； In the formula, is the minimum correlation loss function; is the total number of sketches; is the correct category of the p-th sketch; is the correct category for the pth sketch the number of associated fine-grained categories; is the feature vector of the p-th sketch; is the correct category of the pth sketch The fine-grained category center of is the index in the correct sketch set; Fine-grained sketch clustering network combined with cross entropy loss function And the minimum correlation loss function Perform joint supervised training and calculate the joint loss function , where the cross entropy loss function The formula is: ； In the formula, is the cross entropy loss function; is the correct category of the pth sketch The weight vector of is the feature vector of the p-th sketch; Pass model parameters for the pth sketch The transformed feature vector; is the correct category label of the p-th sketch The bias term of is the total number of sketch categories; is the weight vector of the oth sketch category; b o is the bias term of the o-th sketch category; Joint loss function The formula is: ； In the formula, is the joint loss function; L m is the cross entropy loss function of the sketch; is the hyperparameter of the sketch; L c is the minimum correlation loss function.

Images