Cross-modal three-dimensional model retrieval method for progressive optimization of compact center and accurate alignment

Abstract

The invention discloses a compact center and accurate alignment progressive optimization cross-modal three-dimensional model retrieval method, and belongs to the technical field of three-dimensional model retrieval. In order to solve the problems of insufficient utilization of multi-modal information, insufficient feature expression, low alignment precision among modals and the like in the existing cross-modal three-dimensional model retrieval, the method effectively bridges the modal gap between an image and a three-dimensional model through a compact center and accurate alignment progressive optimization retrieval framework, and improves the accuracy of the three-dimensional model retrieval. And the retrieval accuracy and robustness of the model in a complex scene are remarkably improved. Specifically, a point cloud mode is introduced to enrich three-dimensional structure expression, a three-dimensional feature guidance network fusing point cloud and views is constructed, high-quality discrimination features are learned through compact center loss, and the high-quality discrimination features are used as teacher features to guide two-dimensional image learning. A precise alignment mechanism is further provided, instance-level and category-level alignment is achieved in the shared space, and retrieval precision and robustness are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional model retrieval, and in particular relates to a cross-modal three-dimensional model retrieval method with progressive optimization of compact center and precise alignment. Background Art

[0002] With the rapid development of digital technology and computer vision, the application areas of 3D shapes continue to expand, covering a wide range of industries such as medical diagnosis, 3D printing, bioinformatics, and digital entertainment. In recent years, there has been a growing demand for the rapid generation and convenient acquisition of large numbers of 3D shapes, especially in applications such as virtual reality and augmented reality. This widespread demand has driven the development of 3D shape retrieval algorithms based on 2D images. Compared to directly acquiring 3D models, 2D images are cheaper and more convenient to obtain, such as by capturing natural images with a camera. This has important research value and practical significance in real-world applications.

[0003] Although image-based 3D model retrieval is flexible and convenient, significant differences exist in data representation between 2D images and 3D models. Effectively addressing the complexity of cross-modal 3D shape retrieval remains a daunting task. This difference stems from the inherent differences in the characteristics of each modality. 3D models typically include different data representations, such as point clouds and views, which are inherently different from the pixel values ​​of an image. This results in significant differences in the distribution of features between the two in the semantic space, increasing the difficulty of cross-modal matching. While diverse, these multimodal representations may share consistent relevant information, which is crucial for improving the performance of image-based 3D model retrieval. Existing methods focus on directly learning transferable feature representations and utilizing a single modality for matching, ignoring the distribution differences of samples and feature discriminability, and lacking the comprehensive utilization of multimodal information. Therefore, one of the key challenges in improving cross-modal 3D shape retrieval is how to fully utilize multimodal information and bridge the significant cross-modal gap, enabling effective matching and comparison of image and 3D model features in a unified semantic space. Summary of the Invention

[0004] In response to the problems existing in existing cross-modal 3D model retrieval, such as insufficient utilization of multimodal information, inadequate feature expression, and low inter-modal alignment accuracy, the present invention provides a cross-modal 3D model retrieval method with progressive optimization of compact center and precise alignment.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A cross-modal 3D model retrieval method with progressive optimization of compact center and precise alignment includes the following steps:

[0007] Step 1: Design a retrieval framework with compact center and precise alignment progressive optimization. The retrieval framework includes a 3D feature guidance network (3DFGN) and a 2D image learning network (2DGLN). The 3D feature guidance network inputs point cloud data samples into a DGCNN network to extract global point cloud features, and inputs multi-view image samples into an MVCNN network to extract local view features. The 2D image learning network (2DGLN) uses AlexNet and ResNet50 as the backbone networks of the convolutional neural network (CNN) to extract real image features.

[0008] The specific operations of step 1 are:

[0009] For point cloud data, the 3D point cloud data samples are first input into the DGCNN network, and the global point cloud features F are extracted through the 3D space transformation network, EdgeConv network layer and pooling operation. P , where the point cloud data is represented as , N1 is the number of sampling points;

[0010] Multi-view image samples The data is obtained by N2 virtual cameras evenly set around the 3D shape with an interval of 30°. The multi-view image samples are input into the MVCNN network, which is based on AlexNet and extracts local view features. The specific extraction process is shown in formula (1):

[0011]

[0012] Among them, F P Represents the global point cloud features, F V represents the local view feature, F V′ represents the global view feature after pooling aggregation, f DGCNN (·),f MVCNN (·) represent the point cloud feature extractor and the multi-view feature extractor, respectively, and MaxPooling represents the maximum pooling operation.

[0013] AlexNet and ResNet50 are used as the backbone network of convolutional neural network CNN to extract feature F I , where the real image data is represented as N2 is the number of images, and the specific extraction process is shown in formula (2):

[0014] F I =f ImageNet (I) (2)

[0015] Among them, f ImageNet Represents a real image feature extractor.

[0016] Step 2: Input the global point cloud features and local view features into the Attention-Aware Fusion Module (AAFM) to obtain local point cloud-view complementary features;

[0017] The specific steps of step 2 are:

[0018] Step 2.1, calculate the cosine similarity of the two modal features, obtain the relationship score between the point cloud and the view feature, and generate the attention weight coefficient of different view images This weight coefficient can effectively filter the multi-view features and then select the most critical view as the input of the next layer. The specific calculation process is shown in formula (3):

[0019]

[0020] Among them, F P Represents the global point cloud features, represents the i-th view feature, S θ represents the cosine cosine function, ξ sig Represents the sigmoid function;

[0021] Step 2.2: Multiply the view feature by the attention weight coefficient to obtain the multi-view feature based on the attention weight. On this basis, the residual connection method is used to further add the original multi-view feature to obtain the enhanced view feature. The connection process is shown in formula (4):

[0022]

[0023] In step 2.3, the relationship scores are used to aggregate the enhanced view features with strong correlation to obtain multi-view complementary features of different scales. The information of different perspectives is filtered and fused, providing the model with rich local visual details and geometric clues. On this basis, the point cloud and the filtered multi-scale view features are fused hierarchically and iteratively. The calculation method is shown in formula (5):

[0024]

[0025] in, represents the complementary features of the kth layer, MLP(·) represents the multi-layer perceptron;

[0026] Step 2.4: Aggregate the complementary features of each layer through maximum pooling to obtain the final local point cloud-view complementary feature F PV , which provides strong support for comprehensive modeling and comprehensive representation of three-dimensional shapes. The calculation method is shown in formula (6):

[0027]

[0028] Step 3: Aggregate the global point cloud features, local point cloud-view complementary features, and global view features in a cascade manner, and send the aggregated joint vector to the fully connected layer to obtain the final 3D fusion features;

[0029] The specific operations of step 3 are:

[0030] The global point cloud feature F is cascaded P , local point cloud-view complementary feature F PV , global view feature F V′ Aggregated into a joint vector, which is then fed into the fully connected layer to obtain the final 3D fusion feature. To enhance the synergy between modes, the calculation method is shown in formula (7):

[0031]

[0032] in, represents a cascade operation, and MLP(·) represents a multi-layer perceptron.

[0033] Step 4: Design a compact center loss to impose dual constraints on the cluster center (the feature center of each class of samples) and the feature spacing (the distance between features) of the 3D fusion features, thereby improving the intra-class compactness and inter-class separability of the samples.

[0034] The specific steps of step 4 are:

[0035] In step 4.1, the cluster center is constrained by controlling the distance between the 3D fusion feature and the center of the sample of the same type to be always smaller than the distance between the feature and the center of other classes, so as to preliminarily achieve the purpose of distinguishing between classes and aggregating shape representations within a class. The calculation formula is shown in formula (8):

[0036]

[0037] in, Represents the loss function after constraining the cluster center, T represents the number of small batch training samples selected from the samples at each iteration, represents the i-th sample The features calculated by the convolutional neural network CNN are Indicates the yth i The center vector of the class sample, c j Represents the class label y i The sample center vectors of different other class labels j, m represents the boundary value that can be manually set, and D(·) represents the square of the Euclidean distance function;

[0038] In step 4.2, we introduce a feature spacing constraint to ensure that the mean distance between the current sample feature and features of the same class is always smaller than the mean distance between the current sample feature and features of samples of different classes. This reduces the distance between features of the same class and improves the compactness within the class, while also shortening the distance between the boundary feature and the corresponding sample center, making it farther away from features of other classes. The feature spacing constraint loss function calculation formula is shown in Equation (9):

[0039]

[0040] in, represents the loss function after constraining the feature spacing, Represents the class label y i In the class except f i Other sample features besides f j Represents the class label y i The features of samples of other classes that belong to different classes, δ(·) represents the operation of taking the extreme value of the sample feature distance; m represents the boundary value that can be manually set. The compact center loss function CCL under the dual constraints is defined as shown in formula (10):

[0041]

[0042] Among them, L CC represents the compact center loss, represents the cluster center constraint function, represents the feature spacing constraint function, and η is a hyperparameter that controls the combination ratio of the above two functions.

[0043] Step 5: Design a 3D joint loss function, introduce cross entropy loss and compact center loss to jointly constrain the 3D feature guidance network;

[0044] The step 5 optimizes the discriminative ability and distribution structure of the features through the joint constraints of cross entropy loss and compact center loss. During the training process, the category center is continuously updated iteratively and gradually converges to a stable cluster center, providing three-dimensional cluster center guidance for subsequent two-dimensional image feature learning. The calculation process is shown in formula (11):

[0045]

[0046] Among them, L 3D is the three-dimensional joint loss, α is a hyperparameter that controls the ratio of cross entropy loss and compact center loss, Represents the cross entropy loss function used to constrain the three-dimensional fusion features, f c Represents a classifier.

[0047] Step 6: Design a precise alignment strategy in the 2D image learning network to continuously narrow the distance between image features and corresponding 3D fusion features and cluster centers;

[0048] The specific steps of step 6 are:

[0049] Step 6.1, introduce instance alignment loss L IL , align the 3D fusion features and image features at the instance level, and minimize the feature distribution differences of the samples by continuously bringing the image and 3D fusion features of the same instance closer, ensuring the consistency of features of different modalities in the shared space. The instance alignment loss is calculated as shown in formula (12):

[0050]

[0051] Where T represents the number of small batch training samples selected from the samples at each iteration, Represents the image features of the i-th sample obtained by the convolutional neural network, represents the 3D fusion feature extracted by the 3D feature guidance network for the i-th sample, and D(·) represents the square of the Euclidean distance function;

[0052] Step 6.2: Introduce the feature alignment loss L of the constrained category center CL , pushing image features toward the trained three-dimensional clustering center to achieve precise alignment between cross-modal embedding features of the same category. This enables the model to gradually optimize the feature distribution in the shared embedding space, reduce the feature distribution differences of samples within the same category, enhance intra-class compactness, and thus make the features of different modalities more consistent in the shared space, improving retrieval performance. The feature alignment loss function is defined as shown in formula (13):

[0053]

[0054] in, Indicates the label is y i The cluster center of the class samples.

[0055] Step 7: Design a cross-modal joint loss function that combines cross entropy loss, 3D joint loss, instance alignment loss, and feature alignment loss to jointly constrain the retrieval framework for progressive optimization of compact center and precise alignment.

[0056] The specific steps of step 7 are:

[0057] In the cross-modal weight space CMS, the cross entropy loss is introduced to improve the classification performance of cross-modal data and enhance the model's ability to recognize categories. 3D , instance alignment loss L IL , feature alignment loss L CL Jointly constrain the model to ensure that the different modal features learned by the model are discriminative and semantically consistent; the cross-modal joint loss function L totalThe calculation process is shown in formula (14):

[0058]

[0059] in, To constrain the cross entropy loss of image features, β and μ are the control instance alignment loss L IL , feature alignment loss L CL A hyperparameter for the proportion in the total function.

[0060] In step 8, based on the real image features, similarity matching with the three-dimensional fusion features is completed in the metric space to achieve cross-modal retrieval.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] This invention achieves collaborative optimization by introducing multimodal 3D representation, designing a multimodal fusion network, and constructing a precise alignment strategy. It effectively bridges the modality gap between images and 3D models, significantly improving the accuracy and robustness of the model in complex scenarios. It has good application prospects and practical promotion value. The specific technical advantages are as follows:

[0063] (1) In the traditional image and 3D model alignment task, the point cloud modality is introduced to enrich the structural expression of the 3D model. By constructing discriminative 3D fusion features as teachers to guide the feature learning of 2D real images, semantic consistency modeling between images and 3D models is achieved, thereby significantly improving the discriminative ability of image features and the accuracy of cross-modal alignment.

[0064] (2) In the 3D branch, a 3D feature guidance network is constructed that fuses point clouds and views. This effectively integrates global and local complementary information and learns detailed 3D shape descriptors. Furthermore, a compact center loss is designed to improve the intra-class compactness and inter-class separability of samples. This is used to constrain the model training and provide high-quality cluster center guidance for subsequent 2D image feature learning.

[0065] (3) We constructed a precise alignment mechanism based on instance-level and category-level feature alignment, further improving the matching accuracy between images and 3D models. This mechanism significantly optimizes the distribution structure of different modal features in a unified semantic space, effectively improving the transferability and distinguishability of cross-modal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A retrieval framework that is progressively optimized for compact centering and precise alignment;

[0067] Figure 2 The attention-aware fusion process in the network is guided by three-dimensional features. DETAILED DESCRIPTION

[0068] To gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description thereof. However, the present invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a comprehensive understanding of the present disclosure.

[0069] A cross-modal 3D model retrieval method with progressive optimization of compact center and precise alignment includes the following steps:

[0070] Step 1: Design a retrieval framework with compact center and precise alignment progressive optimization. The retrieval framework includes a 3D feature guidance network (3DFGN) and a 2D image learning network (2DGLN). The 3D feature guidance network inputs point cloud data samples into a DGCNN network to extract global point cloud features, and inputs multi-view image samples into an MVCNN network to extract local view features. The 2D image learning network (2DGLN) uses AlexNet and ResNet50 as the backbone networks of the convolutional neural network (CNN) to extract real image features.

[0071] The specific operations of step 1 are:

[0072] For point cloud data, the 3D point cloud data samples are first input into the DGCNN network, and the global point cloud features F are extracted through the 3D space transformation network, EdgeConv network layer and pooling operation. P , where the point cloud data is represented as , N1 is the number of sampling points;

[0073] Multi-view image samples The data is obtained by N2 virtual cameras evenly set around the 3D shape with an interval of 30°. The multi-view image samples are input into the MVCNN network, which is based on AlexNet and extracts local view features. The specific extraction process is shown in formula (1):

[0074]

[0075] Among them, F P Represents the global point cloud features, F V represents the local view feature, F V′ represents the global view feature after pooling aggregation, f DGCNN (·),f MVCNN (·) represent the point cloud feature extractor and the multi-view feature extractor, respectively, and MaxPooling represents the maximum pooling operation.

[0076] AlexNet and ResNet50 are used as the backbone network of convolutional neural network CNN to extract feature F I , where the real image data is represented as N2 is the number of images, and the specific extraction process is shown in formula (2):

[0077] F I =f ImageNet (I) (2)

[0078] Among them, f ImageNet Represents a real image feature extractor.

[0079] Step 2: Input the global point cloud features and local view features into the Attention-Aware Fusion Module (AAFM) to obtain local point cloud-view complementary features;

[0080] The specific steps of step 2 are:

[0081] Step 2.1, calculate the cosine similarity of the two modal features, obtain the relationship score between the point cloud and the view feature, and generate the attention weight coefficient of different view images This weight coefficient can effectively filter the multi-view features and then select the most critical view as the input of the next layer. The specific calculation process is shown in formula (3):

[0082]

[0083] Among them, F P Represents the global point cloud features, represents the i-th view feature, S θ represents the cosine cosine function, ξ sig Represents the sigmoid function;

[0084] Step 2.2: Multiply the view feature by the attention weight coefficient to obtain the multi-view feature based on the attention weight. On this basis, the residual connection method is used to further add the original multi-view feature to obtain the enhanced view feature. The connection process is shown in formula (4):

[0085]

[0086] In step 2.3, the relationship scores are used to aggregate the enhanced view features with strong correlation to obtain multi-view complementary features of different scales. The information of different perspectives is filtered and fused, providing the model with rich local visual details and geometric clues. On this basis, the point cloud and the filtered multi-scale view features are fused hierarchically and iteratively. The calculation method is shown in formula (5):

[0087]

[0088] in, represents the complementary features of the kth layer, MLP(·) represents the multi-layer perceptron;

[0089] Step 2.4: Aggregate the complementary features of each layer through maximum pooling to obtain the final local point cloud-view complementary feature F PV , which provides strong support for comprehensive modeling and comprehensive representation of three-dimensional shapes. The calculation method is shown in formula (6):

[0090]

[0091] Step 3: Aggregate the global point cloud features, local point cloud-view complementary features, and global view features in a cascade manner, and send the aggregated joint vector to the fully connected layer to obtain the final 3D fusion features;

[0092] The specific operations of step 3 are:

[0093] The global point cloud feature F is cascaded P , local point cloud-view complementary feature F PV , global view feature F V′ Aggregated into a joint vector, which is then fed into the fully connected layer to obtain the final 3D fusion feature. To enhance the synergy between modes, the calculation method is shown in formula (7):

[0094]

[0095] in, represents a cascade operation, and MLP(·) represents a multi-layer perceptron.

[0096] Step 4: Design a compact center loss to impose dual constraints on the cluster center (the feature center of each class of samples) and the feature spacing (the distance between features) of the 3D fusion features, thereby improving the intra-class compactness and inter-class separability of the samples.

[0097] The specific steps of step 4 are:

[0098] In step 4.1, the cluster center is constrained by controlling the distance between the 3D fusion feature and the center of the sample of the same type to be always smaller than the distance between the feature and the center of other classes, so as to preliminarily achieve the purpose of distinguishing between classes and aggregating shape representations within a class. The calculation formula is shown in formula (8):

[0099]

[0100] in, Represents the loss function after constraining the cluster center, T represents the number of small batch training samples selected from the samples at each iteration, represents the i-th sample The features calculated by the convolutional neural network CNN are Indicates the yth i The center vector of the class sample, c j Represents the class label y i The sample center vectors of different other class labels j, m represents the boundary value that can be manually set, and D(·) represents the square of the Euclidean distance function;

[0101] In step 4.2, we introduce a feature spacing constraint to ensure that the mean distance between the current sample feature and features of the same class is always smaller than the mean distance between the current sample feature and features of samples of different classes. This reduces the distance between features of the same class and improves the compactness within the class, while also shortening the distance between the boundary feature and the corresponding sample center, making it farther away from features of other classes. The feature spacing constraint loss function calculation formula is shown in Equation (9):

[0102]

[0103] in, represents the loss function after constraining the feature spacing, Represents the class label y i In the class except f i Other sample features besides f j Represents the class label y i The features of samples of other classes that belong to different classes, δ(·) represents the operation of taking the extreme value of the sample feature distance; m represents the boundary value that can be manually set. The compact center loss function CCL under the dual constraints is defined as shown in formula (10):

[0104]

[0105] Among them, L CC represents the compact center loss, represents the cluster center constraint function, represents the feature spacing constraint function, and η is a hyperparameter that controls the combination ratio of the above two functions.

[0106] Step 5: Design a 3D joint loss function, introduce cross entropy loss and compact center loss to jointly constrain the 3D feature guidance network;

[0107] The step 5 optimizes the discriminative ability and distribution structure of the features through the joint constraints of cross entropy loss and compact center loss. During the training process, the category center is continuously updated iteratively and gradually converges to a stable cluster center, providing three-dimensional cluster center guidance for subsequent two-dimensional image feature learning. The calculation process is shown in formula (11):

[0108]

[0109] Among them, L 3D is the three-dimensional joint loss, α is a hyperparameter that controls the ratio of cross entropy loss and compact center loss, Represents the cross entropy loss function used to constrain the three-dimensional fusion features, f c Represents a classifier.

[0110] Step 6: Design a precise alignment strategy in the 2D image learning network to continuously narrow the distance between image features and corresponding 3D fusion features and cluster centers;

[0111] The specific steps of step 6 are:

[0112] Step 6.1, introduce instance alignment loss L IL , align the 3D fusion features and image features at the instance level, and minimize the feature distribution differences of the samples by continuously bringing the image and 3D fusion features of the same instance closer, ensuring the consistency of features of different modalities in the shared space. The instance alignment loss is calculated as shown in formula (12):

[0113]

[0114] Where T represents the number of small batch training samples selected from the samples at each iteration, Represents the image features of the i-th sample obtained by the convolutional neural network, represents the 3D fusion feature extracted by the 3D feature guidance network for the i-th sample, and D(·) represents the square of the Euclidean distance function;

[0115] Step 6.2: Introduce the feature alignment loss L of the constrained category center CL , pushing image features toward the trained three-dimensional clustering center to achieve precise alignment between cross-modal embedding features of the same category. This enables the model to gradually optimize the feature distribution in the shared embedding space, reduce the feature distribution differences of samples within the same category, enhance intra-class compactness, and thus make the features of different modalities more consistent in the shared space, improving retrieval performance. The feature alignment loss function is defined as shown in formula (13):

[0116]

[0117] in, Indicates the label is y i The cluster center of the class samples.

[0118] Step 7: Design a cross-modal joint loss function that combines cross entropy loss, 3D joint loss, instance alignment loss, and feature alignment loss to jointly constrain the retrieval framework for progressive optimization of compact center and precise alignment.

[0119] The specific steps of step 7 are:

[0120] In the cross-modal weight space CMS, the cross entropy loss is introduced to improve the classification performance of cross-modal data and enhance the model's ability to recognize categories. 3D , instance alignment loss L IL , feature alignment loss L CL Jointly constrain the model to ensure that the different modal features learned by the model are discriminative and semantically consistent; the cross-modal joint loss function L total The calculation process is shown in formula (14):

[0121]

[0122] in, To constrain the cross entropy loss of image features, β and μ are the control instance alignment loss L IL , feature alignment loss L CL A hyperparameter for the proportion in the total function.

[0123] In step 8, based on the real image features, similarity matching with the three-dimensional fusion features is completed in the metric space to achieve cross-modal retrieval.

[0124] The proposed method was evaluated on two public datasets suitable for cross-domain retrieval tasks, MI3DOR and IM2MN. The results are shown in Tables 1 and 2. The number of point cloud neighbors was set to 20, the number of views was set to 12, and the input image size was set to 224 × 224. The IM2MN dataset covers a wider range of categories than the MI3DOR dataset and contains more easily confused samples, posing a higher challenge.

[0125] Table 1 Comparison of different methods on the MI3DOR dataset

[0126]

[0127] The present invention uses view modality and point cloud modality as guidance information, respectively, and conducts single-modal guidance experiments, which are denoted as MVCNN and DGCNN. As can be seen from the table, although the retrieval performance of MVCNN is close to the level of methods in recent years, the feature alignment effect of single-modal guidance still has certain limitations. On this basis, multimodal fusion guidance experiments based on different network architectures (AlexNet and Resnet50) were further designed. The experimental results show that fusion guidance significantly improves the retrieval performance, and all indicators are basically better than MVCNN and DGCNN. Compared with the existing methods, the multimodal fusion strategy of the present invention effectively makes up for the shortcomings of single-modal guidance, and has achieved significant advantages in key indicators such as FT, ST and ANMRR, which fully verifies the potential advantages and practical value of the proposed method in improving cross-modal retrieval performance.

[0128] Compared to MI3DOR, the IM2MN dataset has more complex cross-modal features and a larger number of categories. To further verify the effectiveness of the proposed method, we conducted a detailed comparison with existing advanced methods on the IM2MN dataset. The specific results are shown in Table 2.

[0129] Table 2 Comparison of different methods on the IM2MN dataset

[0130]

[0131] Experimental results show that the overall performance of MVCNN approaches that of GSCTL, but still falls short of our method, further demonstrating the limitations of single-modal guidance in cross-modal retrieval tasks. In terms of mAP and Prec@50 metrics, our method using ResNet50 excels, achieving the best results, with mAP and Prec@50 reaching 0.790 and 0.658, respectively, demonstrating a significant advantage in retrieval accuracy. In summary, experimental results on the IM2MN dataset confirm the significant performance improvement of our proposed method in cross-modal retrieval, demonstrating its superiority in feature alignment and cross-modal adaptability.

[0132] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A cross-modal 3D model retrieval method with progressive optimization of compact center and precise alignment, characterized by: The method comprises the following steps: Step 1: Design a retrieval framework with compact center and precise alignment progressive optimization. The retrieval framework includes a 3D feature guidance network (3DFGN) and a 2D image learning network (2DGLN). The 3D feature guidance network inputs point cloud data samples into a DGCNN network to extract global point cloud features, and inputs multi-view image samples into an MVCNN network to extract local view features. The 2D image learning network (2DGLN) uses AlexNet and ResNet50 as the backbone networks of the convolutional neural network (CNN) to extract real image features. Step 2: Input the global point cloud features and local view features into the attention perception fusion module to obtain the local point cloud-view complementary features; Step 3: Aggregate the global point cloud features, local point cloud-view complementary features, and global view features in a cascade manner, and send the aggregated joint vector to the fully connected layer to obtain the final 3D fusion features; Step 4: Design a compact center loss to impose dual constraints on the cluster center and feature spacing of the 3D fusion features, thereby improving the intra-class compactness and inter-class separability of the samples. Step 5: Design a three-dimensional joint loss function and introduce cross entropy loss and compact center loss to jointly constrain the three-dimensional feature guidance network 3DFGN; Step 6: Design an accurate alignment strategy in the 2DGLN to shorten the distance between image features and corresponding 3D fusion features and cluster centers. Step 7: Design a cross-modal joint loss function that combines cross entropy loss, 3D joint loss, instance alignment loss, and feature alignment loss to jointly constrain the retrieval framework for progressive optimization of compact center and precise alignment. In step 8, based on the real image features, similarity matching with the three-dimensional fusion features is completed in the metric space to achieve cross-modal retrieval.

2. The cross-modal 3D model retrieval method with compact center and precise alignment progressive optimization according to claim 1 is characterized in that: The specific operations of step 1 are: For point cloud data, the 3D point cloud data samples are first input into the DGCNN network, and the global point cloud features F are extracted through the 3D space transformation network, EdgeConv network layer and pooling operation. P , where the point cloud data is represented as N1 is the number of sampling points; Multi-view image samples The data is obtained by N2 virtual cameras evenly set around the 3D shape with an interval of 30°. The multi-view image samples are input into the MVCNN network, which is based on AlexNet and extracts local view features. The specific extraction process is shown in formula (1): Among them, F P Represents the global point cloud features, F V represents the local view feature, F V′ represents the global view feature after pooling aggregation, f DGCNN (·),f MVCNN (·) represents the point cloud feature extractor and the multi-view feature extractor, respectively, and MaxPooling represents the maximum pooling operation; AlexNet and ResNet50 are used as the backbone network of convolutional neural network CNN to extract feature F I , where the real image data is represented as N2 is the number of images, and the specific extraction process is shown in formula (2): F I =f ImageNet (I) (2) Among them, f ImageNet Represents a real image feature extractor.

3. The cross-modal 3D model retrieval method with compact center and precise alignment progressive optimization according to claim 2 is characterized in that: The specific steps of step 2 are: Step 2.1, calculate the cosine similarity of the two modal features, obtain the relationship score between the point cloud and the view feature, and generate the attention weight coefficient of different view images The specific calculation process is shown in formula (3): Among them, F P Represents the global point cloud features, represents the i-th view feature, S θ represents the cosine cosine function, ξ sig Represents the sigmoid function; Step 2.2: Multiply the view feature by the attention weight coefficient to obtain the multi-view feature based on the attention weight. On this basis, the residual connection method is used to further add the original multi-view feature to obtain the enhanced view feature. The connection process is shown in formula (4): In step 2.3, the relation scores are used to aggregate the enhanced view features with strong correlation to obtain multi-view complementary features of different scales. On this basis, the point cloud and the filtered multi-scale view features are fused hierarchically and iteratively. The calculation method is shown in formula (5): in, represents the complementary features of the kth layer, MLP(·) represents the multi-layer perceptron; Step 2.4: Aggregate the complementary features of each layer through maximum pooling to obtain the final local point cloud-view complementary feature F PV , the calculation method is shown in formula (6):

4. The cross-modal 3D model retrieval method with compact center and precise alignment progressive optimization according to claim 3 is characterized in that: The specific operations of step 3 are: The global point cloud feature F is cascaded P , local point cloud-view complementary feature F PV , global view feature F V′ Aggregated into a joint vector, and then sent to the fully connected layer to obtain the final three-dimensional fusion feature f i 3D , strengthen the synergy between modes, and the calculation method is shown in formula (7): in, Indicates cascade operation, h φ represents a fully connected layer.

5. The cross-modal 3D model retrieval method with compact center and precise alignment progressive optimization according to claim 4 is characterized in that: The specific steps of step 4 are: In step 4.1, the cluster center is constrained by controlling the distance between the 3D fusion feature and the center of the sample of the same type to be always smaller than the distance between the feature and the center of other classes, so as to preliminarily achieve the purpose of distinguishing between classes and aggregating shape representations within a class. The calculation formula is shown in formula (8): in, represents the loss function after constraining the cluster center, T represents the number of small batch training samples selected from the samples at each iteration, and f i 3D represents the i-th sample The features calculated by the convolutional neural network CNN are Indicates the yth i The center vector of the class sample, c j Represents the class label y i The sample center vectors of different other class labels j, m represents the boundary value that can be manually set, and D(·) represents the square of the Euclidean distance function; In step 4.2, feature spacing constraints are introduced to ensure that the mean distance between the current sample feature and the features of the same type of samples is always smaller than the mean distance between the current sample feature and the features of different types of samples. The feature spacing constraint loss function is calculated as shown in formula (9): in, represents the loss function after constraining the feature spacing, Represents the class label y i In the class except f i Other sample features besides f j Represents the class label y i The features of samples of other classes that belong to different classes, δ(·) represents the operation of taking the extreme value of the sample feature distance; m represents the boundary value that can be manually set. The compact center loss function CCL under the dual constraints is defined as shown in formula (10): Among them, L CC represents the compact center loss, represents the cluster center constraint function, represents the feature spacing constraint function, and η is a hyperparameter that controls the combination ratio of the above two functions.

6. The cross-modal 3D model retrieval method with compact center and precise alignment progressive optimization according to claim 5 is characterized in that: The step 5 optimizes the discriminative ability and distribution structure of the features through the joint constraints of cross entropy loss and compact center loss. During the training process, the category center is continuously iteratively updated and gradually converges to a stable cluster center, providing guidance for the three-dimensional cluster center for subsequent two-dimensional image feature learning. The calculation process is shown in formula (11): Among them, L 3D is the three-dimensional joint loss, α is a hyperparameter that controls the ratio of cross entropy loss and compact center loss, Represents the cross entropy loss function used to constrain the three-dimensional fusion features, f c Represents a classifier.

7. The cross-modal 3D model retrieval method with compact center and precise alignment progressive optimization according to claim 6 is characterized in that: The specific steps of step 6 are: Step 6.1, introduce instance alignment loss L IL , align the 3D fusion features and image features at the instance level, and minimize the feature distribution differences of the samples by continuously bringing the image and 3D fusion features of the same instance closer, ensuring the consistency of features of different modalities in the shared space. The instance alignment loss is calculated as shown in formula (12): Where T represents the number of small batch training samples selected from the samples at each iteration, f i 2D represents the image features of the i-th sample obtained by the convolutional neural network, f i 3D represents the 3D fusion feature extracted by the 3D feature guidance network for the i-th sample, and D(·) represents the square of the Euclidean distance function; Step 6.2: Introduce the feature alignment loss L of the constrained category center CL , push the image features to the trained three-dimensional cluster center to achieve accurate alignment between cross-modal embedded features of the same category. The feature alignment loss function is defined as shown in formula (13): in, Indicates the label is y i The cluster center of the class samples.

8. The cross-modal 3D model retrieval method with compact center and precise alignment progressive optimization according to claim 7 is characterized in that: The specific steps of step 7 are: In the cross-modal weight space CMS, the cross entropy loss is introduced to improve the classification performance of cross-modal data and enhance the model's ability to recognize categories. 3D , instance alignment loss L IL , feature alignment loss L CL Jointly constrain the model to ensure that the different modal features learned by the model are discriminative and semantically consistent; the cross-modal joint loss function L total The calculation process is shown in formula (14): in, To constrain the cross entropy loss of image features, β and μ are the control instance alignment loss L IL , feature alignment loss L CL A hyperparameter for the proportion in the total function.

Images