A shape-awareness enhanced 3D model retrieval method

Abstract

The application discloses a kind of based on shape perception enhancement three-dimensional model retrieval method, including the following steps, real picture is acquired and is handled, obtains the multiscale feature of real picture, the data obtained is input dynamic sparse feature correction module and is handled, obtains fusion feature, is randomly initialized in spherical space feature aggregation module Generation spherical space feature;Fusion feature is input spherical space feature aggregation module and is handled, obtain spherical vector, establish the matching relationship of spherical space feature and spherical vector, adopt cosine measure to calculate similarity and sort, the first 10 spherical space features of option similarity highest, the class label of the three-dimensional retrieval model corresponding to the first 10 spherical space features of similarity highest is regarded as the final three-dimensional model retrieval result output.The application is more accurate by the feature extraction of three-dimensional retrieval model to target shape in real picture, significantly improve the robustness of three-dimensional retrieval model when facing diversified real scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional model retrieval, and particularly relates to a three-dimensional model retrieval method based on shape perception enhancement. BACKGROUND

[0002] In real-world scenarios, real pictures are widely used in three-dimensional model retrieval due to their convenience in acquisition and rich information. However, current technologies still face significant challenges in handling the matching task between real images and three-dimensional models. Real pictures often contain complex backgrounds, lighting variations, and diverse colors and textures, which are non-geometric attributes that often interfere with the model's judgment of shape structure during feature extraction. When the target in the image has a significant difference in appearance from the three-dimensional model, even if its geometric shape remains consistent, it is often difficult to be accurately matched. This reflects the lack of ability of existing methods in distinguishing object structure and appearance information.

[0003] Existing three-dimensional model retrieval methods rely heavily on prior knowledge modules during model learning, and do not have efficient automatic adaptation capabilities, especially in large-scale and diverse data environments, showing low training efficiency and generalization performance. In recent years, a novel contrastive learning framework has been proposed, which effectively reduces the modal gap between two-dimensional images and three-dimensional models by jointly optimizing instance-level and class-level contrastive learning tasks, and improves the accuracy of cross-modal retrieval. A novel attention mechanism is introduced to improve three-dimensional shape segmentation and retrieval, and more accurately capture shape features. A hierarchical attention and metric learning deep three-dimensional shape retrieval method is used to focus on the key parts of the shape to achieve accurate retrieval.

[0004] The ability to recognize fine-grained structural differences is also a weak link in current technology. At the level of large category discrimination, existing methods can complete basic classification tasks, but they perform unstable when dealing with models with subtle differences within a category. For example, small changes in the backrest curve or armrest structure of a chair can cause the system to misjudge as different categories of objects. This lack of hierarchical expression mechanism in feature modeling limits its application effect in high-precision retrieval tasks.

[0005] In the face of the above problems, an image encoding and matching method that can focus on the essential feature extraction of the target shape and has strong robustness and fine-grained discrimination ability is needed to improve the accuracy and reliability of the three-dimensional model retrieval system driven by real pictures in practical applications. SUMMARY

[0006] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a three-dimensional model retrieval method based on shape perception enhancement, which solves the problems mentioned in the background art.

[0007] To achieve the above object, the present application provides the following technical scheme: a three-dimensional model retrieval method based on shape perception enhancement, comprising the following steps:

[0008] Step S1: obtaining a real picture;

[0009] Step S2: constructing a three-dimensional retrieval model; the three-dimensional retrieval model comprises a query encoder, a dynamic sparse feature correction module and a spherical space feature aggregation module;

[0010] Step S3: extracting the real picture through the query encoder to obtain multi-scale features of the real picture;

[0011] Step S4: inputting the multi-scale features of the real picture into the dynamic sparse feature correction module for processing to obtain fusion features;

[0012] Step S5: randomly initializing to generate spherical space features in the spherical space feature aggregation module; inputting the fusion features into the spherical space feature aggregation module for processing to obtain a spherical vector, and establishing a matching relationship between the spherical space features and the spherical vector;

[0013] Step S6: calculating the similarity of the matching relationship between the spherical space features and the spherical vector by using a cosine measurement method, sorting the similarity scores of the spherical space features and the spherical vector in descending order to obtain a sorting result, and selecting the top 10 spherical space features with the highest similarity from the sorting result, and taking the class labels of the three-dimensional retrieval model corresponding to the top 10 spherical space features with the highest similarity as the final three-dimensional model retrieval result.

[0014] Further, the specific process of step S3 is as follows:

[0015] The real picture is extracted through the query encoder to obtain multi-scale features of the real picture;

[0016] The query encoder is composed of a basic network, a shape overview network, a shape focusing network and an output network;

[0017] The basic network is composed of a first embedding feature, a first basic convolution block, a second embedding feature, a second basic convolution block, a third embedding feature and a third basic convolution block;

[0018] The shape overview network is composed of a fourth embedding feature, a fourth basic convolution block and a first priori knowledge module;

[0019] The shape focusing network is composed of a first dynamic convolution layer, a fifth embedding feature and a second dynamic convolution layer;

[0020] The output network comprises a second priori knowledge module; the real picture is sequentially input into a first embedding feature, a first basic convolution block, a second embedding feature, a second basic convolution block and a third embedding feature in the basic network for processing, to obtain an output of the third embedding feature, and the output of the third embedding feature is input into a third basic convolution block for processing, to obtain an output of the third basic convolution block, i.e. low-level features and middle-level features of the real picture;

[0021] The low-level features and the middle-level features of the real picture are sequentially input into a fourth embedding feature and a fourth basic convolution block in the shape overview network for processing, to obtain an output of the fourth basic convolution block, and the output of the fourth basic convolution block is input into the first priori knowledge module for processing, to obtain an output of the first priori knowledge module, i.e. context features of the real picture;

[0022] The context features of the real picture are sequentially input into a first dynamic convolution layer and a fifth embedding feature in the shape focus network for processing, to obtain an output of the fifth embedding feature, and the context features of the real picture and the output of the fifth embedding feature are input into a second dynamic convolution layer for processing, to obtain an output of the second dynamic convolution layer;

[0023] The output of the second dynamic convolution layer is input into a second priori knowledge module in the output network for processing, to obtain an output of the second priori knowledge module, i.e. multi-scale features of the real picture.

[0024] Further, the specific process of step S4 is as follows:

[0025] The multi-scale features of the real picture are processed by a dynamic sparse feature correction module to obtain fused features;

[0026] The dynamic sparse feature correction module comprises a first layer normalization, a residual feature perception module, a second layer normalization, an adaptive weight module and a sparse feature perception module;

[0027] Firstly, the multi-scale features of the real picture are input into the first layer normalization for processing, to obtain an output of the first layer normalization, and the output of the first layer normalization is input into the residual feature perception module for processing, to obtain an output of the residual feature perception module, i.e. final class-specific residual attention features;

[0028] The final class-specific residual attention features are added to the multi-scale features of the real picture to obtain a first added output, and the first added output is input into the second layer normalization for processing, to obtain an output of the second layer normalization, and the output of the second layer normalization is input into the sparse feature perception module for processing, to obtain an output of the sparse feature perception module;

[0029] The final class-specific residual attention features are input into the adaptive weight module for processing to obtain the output of the adaptive weight module, which is the feature fused by the adaptive weight module.

[0030] The output of the sparse feature perception module is added to the fused feature of the adaptive weight module to obtain the second added output. The second added output, the output of the sparse feature perception module, and the first added output are then added together to obtain the fused feature.

[0031] Furthermore, the processing procedure of the residual feature perception module is as follows:

[0032] The third layer of normalization of the input-output residual feature perception module of the first layer is normalized to obtain the feature tensor x;

[0033] The feature tensor is processed by a fully connected layer as a classifier. The process is performed to generate the raw classification score for each category, represented as follows:

[0034] ;

[0035] In the formula, Represents the original classification score; This means that the input feature tensor x is input into the classifier for computation;

[0036] Calculate the class-specific attention score based on the original classification score, and represent it as follows:

[0037] ;

[0038] In the formula, Indicates the first Class and First Class-specific attention scores; Represents an exponential function; Temperature parameters for controlling fractional sharpness; Indicates the first Transpose of class-specific feature vectors at spatial locations; Indicates the first Classifier weight vectors for each category; Indicates the first Transpose of class-specific feature vectors at spatial locations; For transpose; This is a dot product operation;

[0039] Based on the Class-specific attention scores generate class-specific feature vectors, representing:

[0040] ;

[0041] In the formula, represents the class-specific feature vector of the i-th class; represents the class-specific attention score of the i-th class at the k-th spatial position;

[0042] calculates the global average pooling feature based on the class-specific feature vectors of the first

[0043] ;

[0044] In the formula, represents the global average pooling feature;

[0045] combines the global average pooling feature and the class-specific feature vector to obtain the final class-specific residual attention feature, denoted as:

[0046] ;

[0047] In the formula, represents the final class-specific residual attention feature; represents a learnable scalar parameter.

[0048] Further, the processing process of the sparse feature perception module is as follows:

[0049] The output of the second layer normalization is input into the sparse feature perception module, which is first processed by 1x1 convolution and 3x3 convolution to generate Q, L, and V; wherein Q, L, and V are query matrix, key matrix, and value matrix, respectively.

[0050] The attention matrix is calculated based on Q and L, denoted as:

[0051] ;

[0052] In the formula, represents the attention matrix; is the dimension of the feature vector;

[0053] The largest attention score in the attention matrix is selected by the selection mechanism, denoted as:

[0054] ;

[0055] In the formula, represents the element in the i-th row and the j-th column of the sparse attention matrix; represents the j-th row of the attention matrix ; is the first k largest attention scores selected after sorting the attention scores of the j-th row; is the element in the i-th row and the j-th column of the attention matrix;

[0056] The​ and V are processed by an activation function to obtain the output of the sparse feature perception module, denoted as:

[0057] ;

[0058] wherein, denotes the output of the sparse feature perception module; denotes the activation function.

[0059] Further, the processing procedure of the adaptive weight module is as follows:

[0060] The final class-specific residual attention feature is input into the adaptive weight module for processing to obtain the adaptive weight module fusion feature.

[0061] The adaptive weight module comprises a deep feature pyramid module and a shallow feature pyramid module.

[0062] The final class-specific residual attention feature is input into the deep feature pyramid module and the shallow feature pyramid module respectively for processing to obtain the output of the deep feature pyramid module and the output of the shallow feature pyramid module respectively.

[0063] The output of the deep feature pyramid module and the output of the shallow feature pyramid module are added to obtain the adaptive weight module fusion feature, denoted as:

[0064] ;

[0065] wherein, denotes the adaptive weight module fusion feature of the i-th layer; denotes the output of the deep feature pyramid module of the i-th layer; denotes the output of the shallow feature pyramid module of the i-th layer; denotes the adaptive learning weight.

[0066] Further, the specific procedure of step S5 is as follows:

[0067] Randomly initialize spherical space features in the spherical space feature aggregation module, with a feature dimension D; ; the randomly initialized and the feature dimension D form a spherical matrix , wherein ; is the i-th spherical space feature, is the j-th spherical space feature; is the i-th spherical space feature, which is a real number vector with a dimension of ;

[0068] ​The fused features are mapped to a spherical space feature aggregation module using a predictor network for processing, representing:

[0069] ;

[0070] In the formula, It is the mapped feature vector. A function representing the predictor network; These represent the parameters of the predictor network; Features of fusion;

[0071] Then, the mapped feature vectors Normalization is performed to obtain the i-th spherical vector. ; ; The total number of spherical vectors;

[0072] At the same time, the first one to spherical vectors Stacked row by row, each spherical vector forms a row of the matrix, thus creating the batch sample feature matrix. ;

[0073] Based on the i-th spherical vector Calculate the average value of the current batch features, representing:

[0074] ;

[0075] In the formula, Let be the average value of the features in the t-th batch; Represents the i-th spherical vector in the t-th batch;

[0076] Use a predictor network to calculate the average of the features in the t-th batch. The mapping is performed in the spherical space feature aggregation module to calculate the spherical space features, which are represented as follows:

[0077] ;

[0078] In the formula, This indicates that in the (t+1)th iteration... A spherical spatial feature; Indicates the t-th iteration. A spherical spatial feature; Indicates the momentum parameter;

[0079] based on and batch sample feature matrix Calculate the similarity matrix, which is represented as:

[0080] ;

[0081] wherein, denotes a sample-prototype similarity matrix;

[0082] based on contrastive learning optimization, to obtain a contrastive learning optimization result, denoted as:

[0083] ;

[0084] wherein, denotes a natural logarithm function; denotes a temperature parameter; is the jth spherical space feature; is the total number of spherical space features;

[0085] and based on the contrastive learning optimization result, a matching relationship between and is established.

[0086] Further, the three-dimensional retrieval model comprises a rendering encoder;

[0087] The three-dimensional model picture is input into the rendering encoder for extraction, to obtain multi-scale features of the rendered image.

[0088] Further, the three-dimensional retrieval model is optimized through a comprehensive loss function, and the comprehensive loss function comprises an instance-level contrastive loss, a category-level contrastive loss, and a spherical space aggregation loss;

[0089] The instance-level contrastive loss is denoted as:

[0090] ;

[0091] wherein, denotes the instance-level contrastive loss; denotes the multi-scale features of the rendered image corresponding to the ith spherical vector; denotes the multi-scale features of the yth rendered image;

[0092] The category-level contrastive loss is denoted as:

[0093] ;

[0094] wherein, denotes the category-level contrastive loss; is a category label; denotes the category label to which the ith spherical vector belongs; denotes a feature vector of the category label e; ​​

[0095] spherical space aggregation loss, denoted as:

[0096]

[0097] wherein, denotes the spherical space aggregation loss; is the jth spherical space feature;

[0098] comprehensive loss function denoted as:

[0099]

[0100] wherein, wherein, is the weight coefficient of the category level contrast loss in the comprehensive loss function.

[0101] Further, the specific process of step S1 is as follows:

[0102] The public Pix3D dataset and the Stanford Cars dataset are selected.

[0103] The Pix3D dataset contains real pictures and three-dimensional model pictures of chairs and tables, and the Stanford Cars dataset contains real pictures and three-dimensional model pictures of car models.

[0104] Real pictures and three-dimensional model pictures are extracted from the Pix3D dataset or the Stanford Cars dataset as query data for inputting the three-dimensional retrieval model for processing.

[0105] Compared with the prior art, the present application has the following beneficial effects:

[0106] (1) The present application can efficiently capture multi-scale features of real pictures and three-dimensional model views by using a query encoder and a rendering encoder combined with dynamic convolution, and ensure accurate extraction of key area features. The shape focusing network in the query encoder focuses on the details of the target shape using a dynamic convolution layer, effectively reducing the interference of complex background, illumination changes and other non-geometric properties on feature extraction. Compared with traditional methods, the present application is more accurate in feature extraction of target shapes in real pictures, significantly improves the robustness of the three-dimensional retrieval model when facing diversified real scenes, and ensures the accuracy of subsequent retrieval.

[0107] ​​​(2) This invention establishes a cross-modal matching relationship between fused features and spherical space features by processing the fused features into a spherical space feature aggregation module, and uses cosine metric to sort the matching relationship to obtain the retrieval results. This cross-modal matching method fully considers the differences and connections between real images and 3D model views in the feature space. Through the optimization processing of the spherical space feature aggregation module, the modal gap between 2D images and 3D models is effectively reduced. Compared with traditional cross-modal matching methods, it can more accurately measure the similarity between the two, thereby improving the accuracy of cross-modal retrieval and providing a more reliable solution for 3D model retrieval based on real images. In practical applications, it can more accurately find 3D models that match real images for users. Attached Figure Description

[0108] Figure 1 This is a flowchart of the method of the present invention.

[0109] Figure 2 This is a framework diagram of the query encoder of the present invention.

[0110] Figure 3 This is a framework diagram of the dynamic sparse feature correction module of the present invention. Detailed Implementation

[0111] like Figure 1 As shown, the present invention provides a technical solution: a 3D model retrieval method based on shape-aware enhancement, comprising the following steps:

[0112] Step S1: Obtain the real image;

[0113] Step S2: Construct a 3D retrieval model; the 3D retrieval model includes a query encoder, a dynamic sparse feature correction module, and a spherical space feature aggregation module;

[0114] Step S3: Extract multi-scale features of the real image by using a query encoder;

[0115] Step S4: Input the multi-scale features of the real image into the dynamic sparse feature correction module for processing to obtain fused features;

[0116] Step S5: Randomly initialize and generate spherical spatial features in the spherical spatial feature aggregation module; input the fused features into the spherical spatial feature aggregation module for processing to obtain spherical vectors, and establish a matching relationship between spherical spatial features and spherical vectors;

[0117] Step S6: The similarity of the matching relationship between the spherical space feature and the spherical vector is calculated by using the cosine metric method, the similarity scores of the spherical space feature and the spherical vector are sorted in descending order, and the sorted result is obtained. The top 10 spherical space features with the highest similarity are selected from the sorted result, and the class label of the three-dimensional retrieval model corresponding to the top 10 spherical space features with the highest similarity is taken as the final three-dimensional model retrieval result output.

[0118] The specific process of step S1 is as follows:

[0119] The Pix3D dataset and the Stanford Cars dataset, which are widely recognized in the field of three-dimensional models, are selected.

[0120] The Pix3D dataset contains multiple categories of objects (such as chairs, tables, etc.), and provides real pictures and three-dimensional model pictures. The Stanford Cars dataset contains 196 categories of vehicles, and provides real pictures and three-dimensional model pictures.

[0121] Real pictures and three-dimensional model pictures are extracted from the Pix3D dataset or the Stanford Cars dataset as query data for input into the three-dimensional retrieval model for processing.

[0122] It is noted that the Pix3D dataset and the Stanford Cars dataset have the same processing process.

[0123] The specific process of step S3 is as follows:

[0124] As shown in FIG. 3, the real picture is extracted by the query encoder to obtain the multi-scale feature of the real picture. The query encoder is constructed based on ResNet50. Figure 2 The query encoder is composed of a basic network, a shape overview network, a shape focusing network, and an output network.

[0125] The basic network is composed of a first embedding feature, a first basic convolution block, a second embedding feature, a second basic convolution block, a third embedding feature, and a third basic convolution block.

[0126] The shape overview network is composed of a fourth embedding feature, a fourth basic convolution block, and a first priori knowledge module.

[0127] The shape focusing network is composed of a first dynamic convolution layer, a fifth embedding feature, and a second dynamic convolution layer.

[0128] The shape focusing network is composed of a first dynamic convolution layer, a fifth embedding feature, and a second dynamic convolution layer.

[0129] The output network comprises a second priori knowledge module; the real picture is sequentially input into a first embedding feature, a first basic convolution block, a second embedding feature, a second basic convolution block and a third embedding feature in the base network for processing, to obtain an output of the third embedding feature, and the output of the third embedding feature is input into a third basic convolution block for processing, to obtain an output of the third basic convolution block, that is, low-level features and middle-level features of the real picture;

[0130] The low-level features and the middle-level features of the real picture are sequentially input into a fourth embedding feature and a fourth basic convolution block in the shape overview network for processing, to obtain an output of the fourth basic convolution block, and the output of the fourth basic convolution block is input into the first priori knowledge module for processing, to obtain an output of the first priori knowledge module, that is, context features of the real picture;

[0131] The context features of the real picture are sequentially input into a first dynamic convolution layer and a fifth embedding feature in the shape focusing network for processing, to obtain an output of the fifth embedding feature, and the context features of the real picture and the output of the fifth embedding feature are input into a second dynamic convolution layer for processing, to obtain an output of the second dynamic convolution layer;

[0132] The output of the second dynamic convolution layer is input into a second priori knowledge module in the output network for processing, to obtain an output of the second priori knowledge module; that is, multi-scale features of the real picture.

[0133] The three-dimensional retrieval model comprises a rendering encoder;

[0134] The three-dimensional model picture is input into the rendering encoder for extraction, to obtain multi-scale features of a rendering image;

[0135] The rendering encoder is constructed based on ResNet18, the rendering encoder and the query encoder have the same processing mode, and the rendering encoder also comprises a base network, a shape overview network, a shape focusing network and an output network; however, the rendering encoder has lower computational complexity and is suitable for processing multi-view data, and the base network, the shape overview network, the shape focusing network and the output network are used to extract low-level features and middle-level features of the three-dimensional model picture, and the dynamic convolution layer is used to finely perceive the low-level features and the middle-level features; in addition, the query encoder and the rendering encoder both adopt the dynamic convolution layer, which improves the feature expression capability, so that the rendering encoder and the query encoder can better capture important information in the real picture and the three-dimensional model picture, thereby laying a foundation for subsequent feature fusion and matching tasks.

[0136] The specific process of step S4 is as follows:

[0137] As Figure 3As shown, the multi-scale features of the real picture are processed by a dynamic sparse feature correction module to obtain fusion features; the dynamic sparse feature correction module is built based on a Transformer architecture;

[0138] The dynamic sparse feature correction module is composed of a first layer normalization, a residual feature perception module, a second layer normalization, an adaptive weight module and a sparse feature perception module;

[0139] First, the multi-scale features of the real picture are input into the first layer normalization for processing to obtain the output of the first layer normalization, and the output of the first layer normalization is input into the residual feature perception module for processing to obtain the output of the residual feature perception module, i.e. the final class-specific residual attention feature;

[0140] The final class-specific residual attention feature is added to the multi-scale features of the real picture to obtain the first added output, the first added output is input into the second layer normalization for processing to obtain the output of the second layer normalization, and the output of the second layer normalization is input into the sparse feature perception module for processing to obtain the output of the sparse feature perception module;

[0141] The final class-specific residual attention feature is input into the adaptive weight module for processing to obtain the output of the adaptive weight module, i.e. the adaptive weight module fusion feature;

[0142] The output of the sparse feature perception module is added to the adaptive weight module fusion feature to obtain the second added output, and the second added output, the output of the sparse feature perception module and the first added output are added together to obtain the fusion feature.

[0143] The processing process of the residual feature perception module is as follows:

[0144] The output of the first layer normalization is input into the third layer normalization of the residual feature perception module for normalization processing to obtain a feature tensor x;

[0145] The feature tensor x is processed by a full connection layer (1x1 convolution) as a classifier FC to generate the original classification score of each class, which is represented as:

[0146] ;

[0147] In the formula, represents the original classification score; represents inputting the input feature tensor x into the classifier FC for calculation;

[0148] The class-specific attention score is calculated based on the original classification score, which is represented as:

[0149] ​;

[0150] wherein, denotes the i-th class-specific attention score; denotes the i-th class-specific attention score; denotes the i-th class-specific attention score; denotes the exponential function; is a temperature parameter for controlling the sharpness of the score; denotes the transpose of the i-th class-specific feature vector at the k-th spatial position; denotes the i-th class-specific attention score; denotes the transpose of the i-th class-specific feature vector at the k-th spatial position; denotes the i-th class-specific attention score; denotes the transpose of the i-th class-specific feature vector at the k-th spatial position; denotes the transpose; denotes the dot product operation; denotes the i-th class-specific attention score;

[0151] The class-specific feature vector is generated based on the i-th class-specific attention score, denoted as:

[0152] ;

[0153] wherein, denotes the i-th class-specific feature vector; denotes the i-th class-specific attention score at the k-th spatial position;

[0154] The global average pooling feature is calculated based on the i-th class-specific feature vector at the k-th spatial position, denoted as:

[0155] ;

[0156] wherein, denotes the global average pooling feature;

[0157] The global average pooling feature and the class-specific feature vector are combined to obtain the final class-specific residual attention feature, denoted as:

[0158] ;

[0159] wherein, denotes the final class-specific residual attention feature; denotes the learnable scalar parameter.

[0160] The processing procedure of the sparse feature perception module is as follows:

[0161] The output of the second layer normalization is input into the sparse feature perception module, which is first processed through 1x1 convolution and 3x3 convolution to generate Q, L, and V; wherein Q, L, and V are query matrix, key matrix, and value matrix, respectively.​​

[0162] The attention matrix is calculated based on Q and L, and is represented as:

[0163]

[0164] In the formula, represents the attention matrix; is the dimension of the feature vector;

[0165] The largest attention score in the attention matrix is selected by a selection mechanism, and is represented as:

[0166]

[0167] In the formula, represents the element in the i-th row and j-th column of the sparse attention matrix; represents the j-th row of the attention matrix ; is the top k largest attention scores selected from the attention scores of the j-th row; is the element in the i-th row and j-th column of the attention matrix;

[0168] and V are processed by an activation function to obtain the output of the sparse feature perception module, and is represented as:

[0169]

[0170] In the formula, represents the output of the sparse feature perception module; represents the activation function.

[0171] The processing process of the adaptive weight module is as follows:

[0172] The final class-specific residual attention feature is input into the adaptive weight module for processing to obtain the adaptive weight module fusion feature;

[0173] The adaptive weight module includes a deep feature pyramid module and a shallow feature pyramid module;

[0174] The final class-specific residual attention feature is input into the deep feature pyramid module and the shallow feature pyramid module respectively for processing to obtain the output of the deep feature pyramid module and the output of the shallow feature pyramid module respectively;

[0175] The output of the deep feature pyramid module and the output of the shallow feature pyramid module are added to obtain the adaptive weight module fusion feature, and is represented as:

[0176] ​​​​​

[0177] wherein, represents the adaptive weight module fusion feature of the i-th layer; represents the output of the i-th layer deep feature pyramid module; represents the output of the i-th layer shallow feature pyramid module; represents the adaptive learning weight.

[0178] The specific process of step S5 is as follows:

[0179] Randomly initializing spherical space features, with a feature dimension ; the randomly initialized and feature dimension D form a spherical matrix , wherein ; is the i-th spherical space feature, is the i-th spherical space feature, is a real number vector with a dimension of ; Using a predictor network to map the fusion feature to the spherical space feature aggregation module for processing, denoted as:

[0180]

[0181] ;

[0182] wherein, is the mapped feature vector, denotes the function of the predictor network; denotes the parameters of the predictor network; is the fusion feature; then, the mapped feature vector is normalized to obtain the i-th spherical vector ; ; is the total number of spherical vectors;

[0183] Meanwhile, the 1st to i-th spherical vectors are stacked by rows, with each spherical vector as a row of the matrix, to form a batch sample feature matrix ;

[0184] Based on the i-th spherical vector , the average value of the current batch of features is calculated, denoted as:

[0185] ;

[0186] wherein, is the average value of the t-th batch of features; ​​Represents the i-th spherical vector in the t-th batch;

[0187] The average value of the features in the t-th batch is obtained using a predictor network (composed of a multilayer perceptron, MLP). The mapping is performed in the spherical space feature aggregation module to calculate the spherical space features, which are represented as follows:

[0188] ;

[0189] In the formula, This indicates that in the (t+1)th iteration... A spherical spatial feature; Indicates the t-th iteration. A spherical spatial feature; This represents the momentum parameter, with a value of 0.99; based on and batch sample feature matrix Calculate the similarity matrix, which is represented as:

[0190] ;

[0191] In the formula, Represents the sample-prototype similarity matrix;

[0192] based on right and Perform comparative learning optimization to obtain the comparative learning optimization results, which are represented as follows:

[0193] ;

[0194] In the formula, Represent the natural logarithm function; Indicates temperature parameter; Let j be the j-th spherical spatial feature; This represents the total number of features in a spherical space.

[0195] And based on the results of comparative learning optimization, an optimization system was established. and The matching relationship.

[0196] The training optimizes the 3D retrieval model through a comprehensive loss function, which includes instance-level contrast loss, category-level contrast loss, and spherical space aggregation loss.

[0197] Instance-level contrastive loss, representing:

[0198] ;

[0199] In the formula, Indicates instance-level contrast loss; a multi-scale feature representing the i-th spherical vector corresponding to the rendered image; a multi-scale feature representing the y-th rendered image (including positive and negative samples);

[0200] a class-level contrastive loss, denoted as Lcls, is defined as:

[0201] ;

[0202] wherein, denotes the class-level contrastive loss; is the class label; denotes the class label of the i-th spherical vector to which the i-th spherical vector belongs; ; denotes the feature vector of the class label e;

[0203] a spherical space aggregation loss, denoted as Lssag, is defined as:

[0204] ;

[0205] wherein, denotes the spherical space aggregation loss; is the j-th spherical space feature;

[0206] a comprehensive loss function Lcomprehensive is defined as:

[0207] ;

[0208] wherein, is a weight coefficient of the class-level contrastive loss Lcls in the comprehensive loss function Lcomprehensive.

[0209] While the embodiments of the application have been illustrated and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the spirit and scope of the application, which is defined by the appended claims and their equivalents.​​

Claims

1. A shape-awareness based enhanced 3D model retrieval method, characterized in that, The method comprises the following steps: Step S1: obtaining a real picture; Step S2: constructing a three-dimensional retrieval model; the three-dimensional retrieval model comprises a query encoder, a dynamic sparse feature correction module and a spherical space feature aggregation module; Step S3: extracting the real picture through the query encoder to obtain multi-scale features of the real picture; Step S4: inputting the multi-scale features of the real picture into the dynamic sparse feature correction module for processing to obtain fused features; Step S5: randomly initializing to generate spherical space features in the spherical space feature aggregation module; inputting the fused features into the spherical space feature aggregation module for processing to obtain a spherical vector, and establishing a matching relationship between the spherical space features and the spherical vector; The specific process of step S5 is as follows: randomly initializing a spherical space feature in a spherical space feature aggregation module a feature dimension ; composing a spherical matrix with the randomly initialized and the feature dimension D , wherein ; is the th spherical space feature, is the th spherical space feature is a real number vector with a dimension of ​ The fused features are mapped into the spherical space feature aggregation module for processing by using a predictor network, which is represented as: ； wherein is the mapped feature vector, denotes a function of the predictor network; denotes a parameter of the predictor network; is the fused feature; After the mapped feature vectors are normalized to obtain the i-th spherical vector ; ; is the total number of spherical vectors; Meanwhile, the first to the spherical vectors are stacked by rows, each spherical vector being a row of a matrix, to form a batch sample feature matrix ; based on the ith spherical vector The average of the current batch of features is computed, denoted by: ； wherein is the average value for the tth batch of features; denotes the ith spherical vector in the tth batch. using the predictor network to process the average of the t-th batch of features are mapped into the spherical space feature aggregation module for processing to compute the spherical space features, denoted as ； wherein represents the t+1th iteration of the i th spherical spatial feature; represents the tth iteration of the i th spherical spatial feature; represents a momentum parameter; based on and the batch sample feature matrix computing a similarity matrix, denoted: ； wherein denotes the sample-prototype similarity matrix; based on to and contrastive learning optimization, obtain a contrastive learning optimization result, which is represented by: ； wherein denotes the natural logarithm function; denotes a temperature parameter; is the jth spherical spatial feature; is the total number of spherical spatial features; denotes the exponential function; And based on the contrast learning optimization results to establish And Matching relationship; Step S6: calculating the similarity of the matching relationship between the spherical space features and the spherical vector by using a cosine measurement method, sorting the similarity scores of the spherical space features and the spherical vector in descending order to obtain a sorting result, and selecting the top 10 spherical space features with the highest similarity from the sorting result, and taking the class labels of the three-dimensional retrieval model corresponding to the top 10 spherical space features as the final three-dimensional model retrieval result.

2. The shape-awareness based shape retrieval method according to claim 1, wherein: The specific process of step S3 is as follows: The real picture is extracted through the query encoder to obtain multi-scale features of the real picture; The query encoder is composed of a basic network, a shape overview network, a shape focusing network and an output network; The basic network is composed of a first embedding feature, a first basic convolution block, a second embedding feature, a second basic convolution block, a third embedding feature and a third basic convolution block; The shape overview network is composed of a fourth embedding feature, a fourth basic convolution block and a first priori knowledge module; The shape focusing network is composed of a first dynamic convolution layer, a fifth embedding feature and a second dynamic convolution layer; The output network comprises a second priori knowledge module; The real picture is sequentially input into the first embedding feature, the first basic convolution block, the second embedding feature, the second basic convolution block and the third embedding feature in the basic network for processing to obtain the output of the third embedding feature, and the output of the third embedding feature is input into the third basic convolution block for processing to obtain the output of the third basic convolution block, i.e. the low-level features and the middle-level features of the real picture; The low-level features and the middle-level features of the real picture are sequentially input into the fourth embedding feature and the fourth basic convolution block in the shape overview network for processing to obtain the output of the fourth basic convolution block, and the output of the fourth basic convolution block is input into the first priori knowledge module for processing to obtain the output of the first priori knowledge module, i.e. the context features of the real picture; The context features of the real picture are sequentially input into the first dynamic convolution layer and the fifth embedding feature in the shape focusing network for processing to obtain the output of the fifth embedding feature, and the context features of the real picture and the output of the fifth embedding feature are input into the second dynamic convolution layer for processing to obtain the output of the second dynamic convolution layer; The output of the second dynamic convolutional layer is input into the output network to be processed in the second priori knowledge module to obtain an output of the second priori knowledge module, that is, the multi-scale features of the real picture. 3.The shape-awareness enhanced 3D model retrieval method of claim 2, wherein: The specific process of step S4 is: The multi-scale features of the real picture are processed by the dynamic sparse feature correction module to obtain the fusion features. The dynamic sparse feature correction module is composed of a first layer normalization, a residual feature perception module, a second layer normalization, an adaptive weight module, and a sparse feature perception module. First, the multi-scale features of the real picture are input into the first layer normalization to be processed to obtain an output of the first layer normalization, and the output of the first layer normalization is input into the residual feature perception module to be processed to obtain an output of the residual feature perception module, that is, the final class-specific residual attention feature. The final class-specific residual attention feature is added to the multi-scale features of the real picture to obtain a first added output, and the first added output is input into the second layer normalization to be processed to obtain an output of the second layer normalization, and the output of the second layer normalization is input into the sparse feature perception module to be processed to obtain an output of the sparse feature perception module. The final class-specific residual attention feature is input into the adaptive weight module to be processed to obtain an output of the adaptive weight module, that is, the adaptive weight module fusion feature. The output of the sparse feature perception module is added to the adaptive weight module fusion feature to obtain a second added output, and the second added output, the output of the sparse feature perception module, and the first added output are added together to obtain the fusion features.

4. The shape-awareness based shape-enhanced 3D model retrieval method according to claim 3, characterized in that: The processing process of the residual feature perception module is: The output of the first layer normalization is input into the third layer normalization of the residual feature perception module to be normalized to obtain a feature tensor x. processing the feature tensor through a fully connected layer as a classifier to generate raw classification scores for each class, denoted as: ； In the formula, denotes the original classification score; denotes inputting the input feature tensor x into the classifier for calculation; The class-specific attention score is calculated based on the original classification score, which is represented as: ； wherein represents a class-specific attention score for the th class; represents a class-specific attention score for the is a temperature parameter for controlling the sharpness of the score; represents the transpose of the class-specific feature vector for the th spatial location; represents the transpose of the class-specific feature vector for the is a classifier weight vector for the represents the transpose of the class-specific feature vector for the th spatial location; is the transpose; is a dot product operation; Based on the first The class-specific attention score generates a class-specific feature vector, representing: ； In the formula, represents the class-specific feature vector of the i-th class; represents the class-specific attention score of the i-th class at the spatial position k. Based on the class-specific feature vectors of the first global average pooling features are calculated, denoted as: ； In the formula, denotes the global average pooling feature; The global average pooling feature and the class-specific feature vector are combined to obtain the final class-specific residual attention feature, which is represented as: ； In the formula, represents the final class-specific residual attention feature; represents a learnable scalar parameter.

5. The shape-awareness based shape-enhanced 3D model retrieval method according to claim 4, characterized in that: The processing process of the sparse feature perception module is: The output of the second layer normalization is input into the sparse feature perception module to be processed by a 1x1 convolution and a 3x3 convolution to generate Q, L, and V. Q, L, and V are query matrix, key matrix, and value matrix, respectively. The attention matrix is calculated based on Q and L, which is represented as: ； wherein represents an attention matrix; is the dimension of the feature vector; The maximum attention score in the attention matrix is selected by the selection mechanism, which is represented as: ； wherein, denotes the element in the i-th row and j-th column of the sparse attention matrix; denotes the j-th row of the attention matrix is the top k attention scores after sorting the attention scores of the j-th row; denotes the element in the i-th row and j-th column of the attention matrix;​ will be described in detail below. and V are processed by an activation function to obtain the output of the sparse feature perception module, denoted as: ； wherein denotes the output of the sparse feature perception module; denotes an activation function.

6. The three-dimensional model retrieval method based on shape perception enhancement according to claim 5, characterized in that: The processing process of the adaptive weight module is: The final class-specific residual attention feature is input into the adaptive weight module to be processed to obtain the adaptive weight module fusion feature. The adaptive weight module includes a deep feature pyramid module and a shallow feature pyramid module. The final class-specific residual attention feature is input into the deep feature pyramid module and the shallow feature pyramid module to be processed to obtain an output of the deep feature pyramid module and an output of the shallow feature pyramid module, respectively. The output of the deep feature pyramid module and the output of the shallow feature pyramid module are added together to obtain the adaptive weight module fusion feature, which is represented as: ； In the formula, denotes the adaptive weight module fusion feature of the i-th layer; denotes the output of the i-th layer deep feature pyramid module; denotes the output of the i-th layer shallow feature pyramid module; denotes the adaptive learning weight.

7. The shape-awareness enhanced 3D model retrieval method according to claim 6, wherein: The three-dimensional retrieval model comprises a rendering encoder; The three-dimensional model picture is input into the rendering encoder for extraction, and multi-scale features of a rendering image are obtained.

8. The shape-awareness enhanced 3D model retrieval method of claim 7, wherein: The three-dimensional retrieval model is optimized through a comprehensive loss function, and the comprehensive loss function comprises an instance-level contrast loss, a category-level contrast loss and a spherical space aggregation loss. The instance-level contrast loss is represented as: ； In the formula, represents an example level contrast loss; represents the multi-scale feature of the i-th spherical vector corresponding to the rendered image; represents the multi-scale feature of the y-th rendered image; The category-level contrast loss is represented as: ； In the formula, represents the category level contrast loss; is a category label; represents the i-th spherical vector the category label to which the spherical vector belongs ; represents the feature vector of the category label e; The spherical space aggregation loss is represented as: ； wherein represents the spherical space aggregation loss; is the jth spherical space feature; Synthetic loss function , which is represented by: ； In the formula, is the weight coefficient of the category level contrast loss in the comprehensive loss function. is the weight coefficient of the category level contrast loss in the comprehensive loss function.

9. The shape-awareness enhanced 3D model retrieval method of claim 8, wherein: The specific process of step S1 is as follows: The public Pix3D dataset and the Stanford Cars dataset are selected; The Pix3D dataset comprises real pictures and three-dimensional model pictures of chairs and tables, and the Stanford Cars dataset comprises real pictures and three-dimensional model pictures of vehicle models; Real pictures and three-dimensional model pictures are extracted from the Pix3D dataset or the Stanford Cars dataset as query data for input into the three-dimensional retrieval model for processing.

Images