An Unsupervised Multimodal 3D Model Recognition Method

Abstract

This invention discloses an unsupervised multimodal 3D model recognition method, comprising: constructing a mapping-based missing feature generation module to learn the mapping relationship between modalities by exploring consistent semantic information between modalities and generating a representation of missing instances; constructing a dual adaptive cross-modal fusion module, introducing instance-level adaptive weight selection strategy and modality-level adaptive weight selection strategy to effectively alleviate differences between instances and modalities during the fusion process, and introducing a point cloud-view fusion method to obtain a comprehensive 3D model representation; designing a network optimization mechanism based on multiple loss functions, constructing a mapping loss to utilize existing pairwise multi-view and point cloud feature constraints on the learning of the mapping-based missing feature generation module, and constructing a reconstruction loss to promote the updating of instance-level adaptive weights and modality-level adaptive weights; training the network in an end-to-end manner, and finally outputting the recognition result of the 3D model. This invention effectively improves the accuracy of unlabeled multimodal 3D model recognition and promotes the application of 3D models in virtual reality, intelligent manufacturing and other fields.

Description

Technical Field

[0001] This invention relates to the field of 3D vision, and more particularly to an unsupervised multimodal 3D model recognition method. Background Technology

[0002] In recent years, with the widespread deployment of various 3D acquisition devices, the number of multimodal 3D models has increased rapidly. Compared with 2D images, 3D models can better represent 3D objects in the real world and provide more comprehensive spatial geometric information about these objects. Therefore, they have been widely used in many fields, including film creation, game modeling, architectural design, and medical imaging. However, annotating massive amounts of multimodal 3D models is costly and time-consuming. How to efficiently identify massive amounts of multimodal 3D models in an unsupervised manner is worth further exploration.

[0003] Point clouds and multi-view data, as two of the most widely used modalities in 3D models, have attracted extensive attention from researchers. Typically, point clouds and multi-view data are acquired by different sensors. Point clouds are acquired by 3D scanners, while multi-view data is captured by multi-view camera systems. However, the acquired multimodal 3D models are often incomplete, meaning that point clouds or multi-view data from the same 3D model are missing. For example, due to factors such as deployment costs, site limitations, and data corruption, point cloud or multi-view instances may be missing, increasing the difficulty of unsupervised 3D model recognition. As an effective unsupervised 3D model recognition method, clustering can efficiently identify massive amounts of unlabeled multimodal 3D model data, thereby achieving effective partitioning of unlabeled 3D models, and therefore has significant research significance and value.

[0004] In recent years, researchers have proposed many clustering-based methods for incomplete multimodal recognition. These methods typically utilize existing features or instances to infer knowledge of missing modal features or instances, thereby obtaining the discriminative multimodal representation required for clustering, thus enabling effective recognition of multimodal text or images without label dependence. Among them, the method proposed by Lin et al. combines contrastive learning with clustering analysis, enabling the recovery of missing data while exploring consistent data representations. Wen et al. utilized graph embeddings to generate missing data, thereby adaptively reducing the negative impact of missing instances. However, the above methods are mainly used for text or image data, and few works have explored the recognition of multimodal 3D models. For multimodal 3D models, there are significant modal differences between point clouds and multi-view modalities.

[0005] Existing methods suffer from relatively poor recognition accuracy due to their difficulty in effectively mining the 3D characteristics of point clouds and multi-view models. Therefore, exploring a clustering-based method for recognizing incomplete multimodal 3D models is crucial. By effectively mining the 3D characteristics of point clouds and multi-view modalities and jointly utilizing different modalities to obtain a more comprehensive representation of the 3D model, this approach plays a vital role in improving the recognition accuracy of 3D models and achieving efficient segmentation of label-free 3D models. While existing clustering-based incomplete multimodal recognition methods have achieved considerable results in text or image data, few studies have explored incomplete multimodal 3D models. When extending existing recognition methods to multimodal 3D models, these methods neglect the mining of 3D characteristics, resulting in low recognition accuracy. Summary of the Invention

[0006] This invention provides an unsupervised multimodal 3D model recognition method. It effectively explores the 3D characteristics inherent in multimodal 3D models composed of point clouds and multi-view modalities, and jointly utilizes the intrinsic information of different 3D modalities to obtain a more comprehensive description of the 3D model. This allows for robust 3D model recognition results using clustering algorithms. Furthermore, it can optimize the feature learning, missing feature generation, and cross-modal feature fusion processes of point clouds and multi-view modalities without relying on labels, effectively improving the accuracy of unlabeled multimodal 3D model recognition and promoting the application of 3D models in virtual reality, intelligent manufacturing, and other fields. See the description below for details:

[0007] An unsupervised multimodal 3D model recognition method, the method comprising:

[0008] A mapping-based missing feature generation module is constructed. By exploring consistent semantic information between modalities, the mapping relationship between modalities is learned, and a representation of missing instances is generated.

[0009] A dual adaptive cross-modal fusion module is constructed, and an instance-level adaptive weight selection strategy and a modality-level adaptive weight selection strategy are introduced to effectively alleviate the differences between instances and modalities during the fusion process. A point cloud-view fusion method is introduced to obtain a comprehensive 3D model representation.

[0010] We design a network optimization mechanism based on multiple loss functions, construct a mapping loss to leverage the learning of the missing feature generation module based on the existing pairwise multi-view and point cloud feature constraints, and construct a reconstruction loss to facilitate the updating of instance-level adaptive weights and modality-level adaptive weights.

[0011] The network is trained in an end-to-end manner, and the final output is the recognition result of the 3D model without the dependence on labels.

[0012] The mapping relationship of the mapping-based missing feature generation module is as follows:

[0013] By aggregating view information from different perspectives to obtain a more comprehensive multi-view representation, the aggregated multi-view features are input into the multilayer perceptron mapping unit to generate missing point cloud features. The mapping process is as follows:

[0014] F′ P =φ(VP(F) I ))

[0015] Among them, F′ P The generated point cloud features are represented by VP(·), which represents the view pooling operation, and φ(.) represents the MLP mapping function.

[0016] Given the pose information of V views, the extracted point cloud features are projected onto a learnable dynamic plane using basis transformation, orthogonal projection, and normalization to generate missing multi-view features. The mapping process is as follows:

[0017] F′ I =norm(proj(b(F P )))

[0018] Among them, F′ I The generated multi-view features are represented by b(·), which represents the basis transformation operation, proj(·) which represents the orthogonal projection operation, and norm(·) which represents the normalization operation.

[0019] The instance-level adaptive weight selection strategy is as follows: it balances the information content of different instance representations between existing instance representations and generated instance representations, as shown in the following formula:

[0020]

[0021]

[0022] in, and denoted as point cloud and multi-view feature sequences after instance-level adaptation, respectively; μ1 and μ2 represent the adaptive weights used for existing instance representation and generated instance representation, respectively; and r is a coefficient that controls the smoothness between different weights.

[0023] Furthermore, the modality-level adaptive weight selection strategy is used for point cloud modal representation and multi-view modal representation to balance the information content between different modal representations, as shown in the following formula:

[0024]

[0025]

[0026] in, and ν1 and ν2 represent the point cloud and multi-view feature sequences after instance-level and modal-level adaptation, respectively, and represent the adaptive weights used for point cloud modal representation and multi-view modal representation, respectively.

[0027] The point cloud-view fusion method is as follows:

[0028] Point cloud features from the i-th 3D model Copy it V times, then combine it with the feature sequence of V views. Aggregation, represented as:

[0029]

[0030] in, Let represent the fusion representation of the i-th 3D model, Pool(·) represents the max pooling operation, θ(·) represents the linear mapping operation, and c(·) represents the cascading operation.

[0031] Furthermore, the mapping losses for the point cloud mode and the multi-view mode are defined as follows:

[0032]

[0033]

[0034] Where ||·||2 represents the L2 norm, and These represent the existing and generated point cloud features in the i-th 3D model, respectively. and These represent the existing and generated multi-view features in the i-th 3D model, respectively, where N represents the number of 3D models and M represents the number of multi-view features. i,P =0 and M i,I =0 indicates that the point cloud and multi-view instances of the i-th 3D model are missing, respectively. M i,P =1 and M i,I =1 indicates that the multi-view and point cloud instance of the i-th 3D model are complete;

[0035] Reconstruction losses are expressed as:

[0036]

[0037]

[0038] Among them, L CD (·) represents the chamfer distance loss, L MSE (·) indicates the mean square error loss.

[0039] The beneficial effects of the technical solution provided by this invention are:

[0040] (1) This invention effectively reduces the negative impact of missing instances, and uses clustering algorithms to classify massive unlabeled multimodal 3D models, effectively getting rid of the dependence on large-scale manual annotation and improving the recognition accuracy of unlabeled 3D models.

[0041] (2) This invention constructs a mapping-based missing representation generation module, which effectively generates the representation corresponding to the missing modality instance by exploring the mapping relationship between different modalities of the three-dimensional model;

[0042] (3) The present invention constructs a dual adaptive cross-modal fusion module, which obtains a more comprehensive comprehensive representation of the three-dimensional model by fusing representations from different modalities of the three-dimensional model;

[0043] (4) The present invention designs a network optimization mechanism based on multiple loss functions, which effectively constrains the learning process of the constructed modules and helps to learn more discriminative 3D model representations, thereby improving the unsupervised recognition accuracy of the proposed method. Attached Figure Description

[0044] Figure 1 This is a flowchart of an unsupervised multimodal 3D model recognition method. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0046] To overcome the shortcomings of existing technologies, this invention proposes an unsupervised multimodal 3D model recognition method to effectively analyze and recognize massive amounts of unlabeled multimodal 3D models, as detailed below:

[0047] I. Constructing a mapping-based missing feature generation module

[0048] This invention constructs a mapping-based missing feature generation module, which learns reliable mapping relationships between modalities by effectively exploring consistent semantic information between modalities, thereby generating representations of missing instances.

[0049] Specifically, incomplete multimodal 3D model data can be expressed by the formula D = {I, P}, where I = {I...} v}| v＝1,...,V P and V represent the multi-view data of the 3D model and the corresponding point cloud data, respectively, and V represents the number of views. In this embodiment of the invention, V is set to 6. This embodiment of the invention constructs a multimodal feature encoder to extract features from the existing point cloud and multi-view data respectively:

[0050]

[0051]

[0052] in, The mapping function represents the point cloud encoder, which contains E1 convolutional layers, for example, 5 convolutional layers; The mapping function represents the multi-view encoder, which contains E2 convolutional layers, for example, 5 convolutional layers.

[0053] For the mapping process from multi-view modalities to point cloud modalities, in order to fully explore the rich 3D information between views, view information from different perspectives is aggregated to obtain a more comprehensive multi-view representation. Subsequently, the aggregated multi-view features are input into a multilayer perceptron (MLP) mapping unit to generate missing point cloud features. This mapping process can be expressed by the following formula:

[0054] F′ P =φ(VP(F) I ))

[0055] Among them, F′ P The generated point cloud features are represented by VP(·), which represents the view pooling operation, and φ(.) represents the MLP mapping function.

[0056] Simultaneously, for the mapping process from point cloud modalities to multi-view modalities, a dynamic plane projection method is adopted to project the extracted point cloud features onto a predicted plane to obtain the corresponding multi-view features. Specifically, given the pose information of V views, basis transformation, orthogonal projection, and normalization operations are used to project the extracted point cloud features onto a learnable dynamic plane. This mapping process can be expressed by the formula:

[0057] F′ I =norm(proj(b(F P )))

[0058] Among them, F′ I The generated multi-view features are represented by b(·), which represents the basis transformation operation, proj(·) which represents the orthogonal projection operation, and norm(·) which represents the normalization operation.

[0059] II. Constructing a Dual Adaptive Cross-Modal Fusion Module

[0060] Subsequently, this embodiment of the invention constructs a dual adaptive cross-modal fusion module, introducing an instance-level adaptive weight selection strategy and a modality-level adaptive weight selection strategy to effectively alleviate differences between instances and between modalities during the fusion process, thereby obtaining a more comprehensive three-dimensional model representation.

[0061] Specifically, an instance-level adaptive weight selection strategy is proposed for both existing and generated instance representations, thereby effectively balancing the information content of different instance representations. The formula is as follows:

[0062]

[0063]

[0064] in, and represents the point cloud and multi-view feature sequences after instance-level adaptation, respectively; μ1 and μ2 represent the adaptive weights used for existing instance representations and generated instance representations, respectively; and r is a coefficient controlling the smoothness between different weights. In this embodiment of the invention, r is set to 1.0.

[0065] Furthermore, a modality-level adaptive weight selection strategy is proposed for point cloud modality representation and multi-view modality representation, thereby effectively balancing the information content among different modality representations. The formula is as follows:

[0066]

[0067]

[0068] in, and ν1 and ν2 represent the point cloud and multi-view feature sequences after instance-level and modal-level adaptation, respectively, and represent the adaptive weights used for point cloud modal representation and multi-view modal representation, respectively.

[0069] Subsequently, to fully consider the 3D characteristics of different modalities during the fusion process, a point cloud-view fusion method was introduced to obtain a comprehensive 3D model representation. For the point cloud features from the i-th 3D model... Copy it V times, then combine it with the feature sequence of V views. Aggregation, expressed by the formula:

[0070]

[0071] in, Let represent the fusion representation of the i-th 3D model, Pool(·) represents the max pooling operation, θ(·) represents the linear mapping operation, and c(·) represents the cascading operation.

[0072] Finally, the fused 3D model representation is fed into the point cloud and multi-view decoders respectively to obtain the reconstructed point cloud and multi-view, as expressed by the formula:

[0073]

[0074]

[0075] Among them, F FU A fused representation of a 3D model. This represents the reconstructed point cloud data. This represents the reconstructed multi-view data. The mapping function of the point cloud decoder consists of D1 linear layers, for example, containing 5 linear layers; This represents the mapping function of the multi-view decoder, which consists of D2 layers of deconvolutional layers, for example, containing 5 layers of deconvolutional layers.

[0076] III. Designing a Network Optimization Mechanism Based on Multiple Loss Functions

[0077] To obtain reliable multi-view and point cloud features and robust clustering results, this embodiment of the invention designs a network optimization mechanism based on multiple loss functions, and designs mapping loss and reconstruction loss for multi-view and point cloud data respectively.

[0078] First, a mapping loss is constructed to leverage the learning of the missing feature generation module based on mapping, constrained by existing pairwise multi-view and point cloud feature constraints. The mapping losses for the point cloud modality and the multi-view modality are defined as follows:

[0079]

[0080]

[0081] Where ||·||2 represents the L2 norm, and These represent the existing and generated point cloud features in the i-th 3D model, respectively. and Let M represent the existing and generated multi-view features in the i-th 3D model, respectively, and N represent the number of 3D models. i,P =0 and M i,I =0 indicates that the point cloud and multi-view instances of the i-th 3D model are missing, respectively. M i,P =1 and M i,I =1 indicates that the multi-view and point cloud instance of the i-th 3D model are complete.

[0082] Subsequently, a reconstruction loss is constructed to facilitate the updating of instance-level adaptive weights and modality-level adaptive weights. The constructed reconstruction loss is expressed as:

[0083]

[0084]

[0085] Among them, L CD (·) represents the Chamfer Distance loss, L MSE(·) represents the mean square error loss. Both the chamfer distance loss and the mean square error loss are well known in the art and will not be described further here.

[0086] IV. Training an Unsupervised Multimodal 3D Model Recognition Network

[0087] The network training proposed in this embodiment of the invention is divided into two stages, neither of which uses the category labels of the multimodal 3D model. The first stage is the pre-training of the multimodal encoder and decoder. In this stage, in order to obtain a certain feature learning capability, the constructed point cloud and multi-view encoder and decoder are pre-trained using existing instances, and reconstruction loss is used to constrain the training of the encoder and decoder.

[0088] After the pre-training phase, the encoder learns a reliable 3D representation. Subsequently, the entire network is trained jointly in an end-to-end manner. The total loss for network training is:

[0089]

[0090] Wherein, λ represents the weighting parameter between different modes. In this embodiment of the invention, λ is set to 10.0.

[0091] Finally, based on the obtained 3D model fusion representation F FU The clustering algorithm is invoked to obtain the corresponding clustering results. The obtained clustering results are then used to classify massive amounts of label-free multimodal 3D models, thereby achieving effective identification of these models. Furthermore, this result can be effectively applied to fields such as robot vision and virtual reality, providing reliable assistance and support for intelligent sensing devices to identify 3D objects in a scene. In this embodiment of the invention, the k-Means algorithm, which is well-known in the field, is selected as the clustering algorithm.

[0092] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An unsupervised multimodal 3D model recognition method, characterized in that, The method includes: A mapping-based missing feature generation module is constructed. By exploring consistent semantic information between modalities, the mapping relationship between modalities is learned, and a representation of missing instances is generated. A dual adaptive cross-modal fusion module is constructed, and an instance-level adaptive weight selection strategy and a modality-level adaptive weight selection strategy are introduced to alleviate the differences between instances and modalities during the fusion process. A point cloud-view fusion method is introduced to obtain a comprehensive 3D model representation. We design a network optimization mechanism based on multiple loss functions, construct a mapping loss to leverage the learning of the missing feature generation module based on existing pairwise multi-view and point cloud feature constraints, and construct a reconstruction loss to facilitate the updating of instance-level adaptive weights and modality-level adaptive weights. The network is trained in an end-to-end manner, and the final output is the recognition result of the 3D model without the dependence on labels; The instance-level adaptive weight selection strategy is as follows: it balances the information content of different instance representations between existing instance representations and generated instance representations, as shown in the following formula: ； ； in, and These represent the point cloud and multi-view feature sequences after instance-level adaptation, respectively. and These represent the adaptive weights used for existing instance representations and generated instance representations, respectively. It is a coefficient that controls the smoothness between different weights; This represents the generated point cloud features. This represents the generated multi-view features; This indicates that the cloud has the characteristics of a dot cloud. Characteristics representing multi-view data; The modality-level adaptive weight selection strategy is used for point cloud modal representation and multi-view modal representation to balance the information content between different modal representations, as shown in the following formula: ； ； in, and These represent the point cloud and multi-view feature sequences after instance-level and modal-level adaptation, respectively. and These represent the adaptive weights used for point cloud modal representation and multi-view modal representation, respectively. and These represent the point cloud and multi-view feature sequences after instance-level adaptation, respectively; It is a coefficient that controls the smoothness between different weights.

2. The unsupervised multimodal 3D model recognition method according to claim 1, characterized in that, The mapping relationship of the mapping-based missing feature generation module is as follows: By aggregating view information from different perspectives to obtain a more comprehensive multi-view representation, the aggregated multi-view features are input into the multilayer perceptron mapping unit to generate missing point cloud features. The mapping process is as follows: ； in, This represents the generated point cloud features. This indicates a view pooling operation. Represents the MLP mapping function; Given The pose information of each view is used to perform basis transformation, orthogonal projection, and normalization operations to project the extracted point cloud features onto a learnable dynamic plane, generating missing multi-view features. The mapping process is as follows: ； in, This indicates the generated multi-view features. This indicates a basis transformation operation. This indicates an orthographic projection operation. This indicates a normalization operation.

3. The unsupervised multimodal 3D model recognition method according to claim 1, characterized in that, The point cloud-view fusion method is as follows: For those from the Point cloud features of a 3D model Copy it Next, and then with Feature sequence of a view Aggregation, represented as: ； in, Indicates the first A fused representation of three-dimensional models. This represents the maximum pooling operation. Represents a linear mapping operation. This indicates a cascading operation.

4. The unsupervised multimodal 3D model recognition method according to claim 1, characterized in that, The design is based on a network optimization mechanism with multiple loss functions, where the mapping losses for point cloud mode and multi-view mode are defined as follows: ； ； in, express Norm, and They represent the first Existing and generated point cloud features in a 3D model and They represent the first Existing and generated multi-view features in a 3D model Indicates the number of 3D models. and They represent the first The point cloud and multi-view instances of the 3D model are missing. and They represent the first The multi-view and point cloud instances of the 3D model are complete; Reconstruction losses are expressed as: ； ； in, Indicates chamfer distance loss. This represents the mean square error loss.

Images