Three-dimensional model retrieval method and system based on cross-modal fusion of single image
By aligning and fusing features of images and 3D models, a 3D model retrieval method based on cross-modal fusion of single images was developed. This method solves the modal gap problem, improves retrieval accuracy and efficiency, and achieves efficient image-to-3D model retrieval.
Patent Information
- Application Number
- CN202410633848.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-21
- Publication Date
- 2026-07-28
- Estimated Expiration
- 2044-05-21
AI Technical Summary
In existing technologies for 3D model retrieval, the modal gap between images and 3D models leads to insufficient retrieval accuracy and efficiency, and neglects the fusion of information between modalities, making it difficult to effectively utilize readily available images for efficient retrieval.
A 3D model retrieval method based on cross-modal fusion of single images is adopted. By aligning features through a modal fusion network and introducing a data exchange mechanism, the data in the image domain and the 3D model domain are aligned and fused. Contrastive learning is used to improve feature similarity. The image domain network learns the model domain data, and the model domain network learns the image domain data, thereby achieving efficient fusion of cross-modal information.
It improves the accuracy of 3D model retrieval, achieving retrieval accuracy 1%-4% higher than common algorithms on known datasets, simplifies the deployment process, and realizes efficient image-to-3D model retrieval.
Smart Images
Figure CN118484558B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of 3D model retrieval technology, specifically to a 3D model retrieval method and system based on cross-modal fusion of a single image. Background Technology
[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.
[0003] With the maturity of 3D modeling technology and the dramatic increase in the number of 3D models, effectively retrieving and managing these models has become extremely challenging. Traditional model-based 3D model retrieval methods take 3D models as input, but directly obtaining models from real life is difficult.
[0004] Therefore, in recent years, existing methods have tended to use readily available images to retrieve 3D models, greatly improving the accuracy and efficiency of retrieval. However, the modal gap between images and 3D models makes this a challenging task. To address this gap, a common approach in previous work is to process the retrieval task by mapping 3D models and query images to a common embedding space. This involves using a location field to embed 3D models and 2D images into a common low-level representation space, and using metric learning to shorten the distance between images and 3D models of the same instance and widen the distance between images and 3D models of different instances. This improves the feature similarity between images and 3D models of the same instance and reduces the feature similarity between images and 3D models of different instances.
[0005] However, in the previous work mentioned above, the metric function was mostly triple loss, which needs to take into account the problem of mining hard negative samples, and most methods also ignored the information fusion between modalities. Summary of the Invention
[0006] To address the aforementioned issues, this disclosure proposes a method and system for 3D model retrieval based on cross-modal fusion of single images. Through a modal fusion network, features are aligned before fusion based on contrastive learning, bringing images and 3D models of the same instance from two domains closer together and pushing images and 3D models from different domains further apart. Data exchange is introduced by providing additional channels for image data to the rendering view and vice versa, enabling the image domain network to learn data from model domain network data, and vice versa.
[0007] According to some embodiments, the present disclosure adopts the following technical solutions:
[0008] A 3D model retrieval method based on single-image cross-modal fusion includes:
[0009] Retrieve the image and multi-view 3D model set to be queried;
[0010] The image to be queried and a set of 3D models rendered into multiple views are input into a trained cross-modal fusion network based on a single image, and the corresponding 3D model is output.
[0011] The training process of the single-image cross-modal fusion network is as follows: A single image and a 3D model rendered into multiple views are input. A data exchange process is introduced, where image domain data is assigned to additional channels in the 3D model domain with a set probability, and model domain data is assigned to additional channels in the image domain with a set probability. Features are extracted from both the image domain data and the 3D model domain data, and after feature alignment in the image domain network and the 3D model domain network respectively, they are input into the cross-modal network to obtain image domain representation vectors and 3D model domain representation vectors. Attention is introduced between the image domain representation vectors and the 3D model domain representation vectors to fuse them and generate the final fused feature vector. This vector is then projected into a two-dimensional vector through a linear layer, completing the training of the single-image cross-modal fusion network.
[0012] According to some embodiments, the present disclosure adopts the following technical solutions:
[0013] A 3D model retrieval system based on single-image cross-modal fusion includes:
[0014] The data acquisition module is used to acquire the image to be queried and a collection of multi-view 3D models;
[0015] The retrieval module is used to input the image to be queried and a set of 3D models rendered into multiple views into a trained cross-modal fusion network based on a single image, and output the corresponding 3D model retrieved.
[0016] The training process of the single-image cross-modal fusion network is as follows: A single image and a 3D model rendered into multiple views are input. A data exchange process is introduced, where image domain data is assigned to additional channels in the 3D model domain with a set probability, and model domain data is assigned to additional channels in the image domain with a set probability. Features are extracted from both the image domain data and the 3D model domain data, and after feature alignment in the image domain network and the 3D model domain network respectively, they are input into the cross-modal network to obtain image domain representation vectors and 3D model domain representation vectors. Attention is introduced between the image domain representation vectors and the 3D model domain representation vectors to fuse them and generate the final fused feature vector. This vector is then projected into a two-dimensional vector through a linear layer, completing the training of the single-image cross-modal fusion network.
[0017] According to some embodiments, the present disclosure adopts the following technical solutions:
[0018] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned three-dimensional model retrieval method based on cross-modal fusion of a single image.
[0019] According to some embodiments, the present disclosure adopts the following technical solutions:
[0020] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the three-dimensional model retrieval method based on single-image cross-modal fusion.
[0021] Compared with the prior art, the beneficial effects of this disclosure are as follows:
[0022] This disclosure presents a 3D model retrieval method based on single-image cross-modal fusion. It trains a single-image cross-modal fusion network comprising an image domain network, a 3D model domain network, and a cross-modal network. When an input image and a 3D model are used, their similarity can be obtained. The network is trained using the image-model pairing loss (ISM) to output similarity probabilities after modality fusion. To improve modality fusion, feature alignment is performed before fusion based on contrastive learning. Feature alignment brings images and 3D models with the same instances from two domains closer together and pushes them apart from each other. Using contrastive learning in feature alignment can solve the problem of difficult negative sample mining in triplet loss.
[0023] This disclosed method for 3D model retrieval based on single-image cross-modal fusion proposes the concept of data exchange. By exchanging image data with an additional channel of the rendering view and rendering view data with an additional channel of image domain data, the image domain network acquires the ability to learn data from model domain network data, and the model domain network acquires the ability to learn data from image domain data, thus making it more conducive to modal fusion.
[0024] The proposed method for retrieving 3D models based on cross-modal fusion of single images is used for image retrieval of 3D models. Its retrieval accuracy on known datasets is 1%-4% higher than that of most algorithms on common retrieval metrics. The proposed method is efficient, advantageous, simple to implement and deploy. Attached Figure Description
[0025] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.
[0026] Figure 1This is an architecture diagram of a three-dimensional model retrieval method based on cross-modal fusion of a single image, according to an embodiment of this disclosure. Detailed Implementation
[0027] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.
[0028] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.
[0029] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0030] Example 1
[0031] One embodiment of this disclosure provides a 3D model retrieval method based on cross-modal fusion of a single image, which can retrieve a 3D model matching the image using an easily obtainable 2D image. The method includes:
[0032] Step 1: Obtain the image to be queried and the set of multi-view 3D models;
[0033] Step 2: Input the image to be queried and the set of 3D models rendered into multiple views into the trained single-image cross-modal fusion network, and output the corresponding 3D model retrieved.
[0034] Furthermore, the single-image-based cross-modal fusion network structure disclosed herein includes an image domain network, a 3D model domain network, and a cross-modal network. The image domain network of this disclosure consists of a query encoder (QIE) and a key image encoder (KIE). The 3D model domain network consists of a query model rendering view encoder (QSE) and a key model rendering view encoder (KSE). The cross-modal network mainly includes an image encoder (IE), a model encoder (SE), and an attention mechanism network (AN). The image encoder (IE) shares weights with the image domain network (QIE), and the model encoder (SE) shares weights with the 3D model domain network (QSE).
[0035] As one embodiment, the cross-modal fusion network based on a single image trained in this disclosure can fuse information from different modalities and align features between fusion processes using feature alignment modules in both the image domain network and the 3D model domain network, thereby reducing the difficulty of modal fusion. The training process mainly includes:
[0036] Step 1: Input a single RGB image i and a 3D model s rendered into multiple views. This introduces a data exchange process. Since a single RGB image typically has three channels (red, green, and blue), image domain data is assigned to additional channels in the 3D model domain with predetermined probabilities, and vice versa. This data exchange improves the ability to fit data from different modalities. After the exchange module, image i becomes... Model s becomes
[0037] Step 2: Image after data exchange Two different data augmentation methods were used to augment the data, resulting in two types of image domain augmented data. and
[0038] Among them, the two different data augmentation methods are: random cropping and random horizontal or vertical flipping.
[0039] Then, randomly select two views from the rendered multi-view 3D model. and
[0040] Step 3: Enhance the two image domains of the data after data augmentation. and Feature extraction is performed, and the query image encoder and key image encoder of the image domain network are used to obtain the first image domain feature vector and the second image domain feature vector. The first image domain feature vector and the second image domain feature vector are then aligned in the feature alignment module of the image domain network.
[0041] Specifically, the two types of enhanced image domain data and The first image domain feature vector is obtained by encoding the query image encoder (QIE) and the key image encoder (KIE) of the image domain network, respectively. Second image domain feature vector The first and second image domain feature vectors are aligned in the feature alignment module of the image domain network. The encoding process is represented as follows:
[0042]
[0043] in, Used for domain feature alignment.
[0044] Step 4: Randomly select two views from the multi-view data after the 3D model is rendered, and input them into the query model rendering view encoder and the key model rendering view encoder respectively to obtain the feature vector of the first view and the feature vector of the second view. The feature vector of the first view and the feature vector of the second view are aligned in the 3D model domain network.
[0045] Specifically, two randomly selected views Input the query model rendering view encoder QSE and the key model rendering view encoder KSE to obtain the first view feature vector. Second view feature vector This process is represented as:
[0046]
[0047] Among them, the first view feature vector Second view feature vector Used for feature alignment in 3D model domain networks.
[0048] Step 5: After aligning the domain features of the image domain network and the 3D model domain network, input them into the cross-modal network to obtain the image domain representation vector and the 3D code model domain representation vector.
[0049] Specifically, the network loss is calculated through feature alignment and ISM. Feature alignment is based on contrastive learning, and the feature alignment function is:
[0050]
[0051] in, It is the query vector for the image. It is the key vector of the image; where It is the query vector of the model view. is the key vector of the model; N is the number of samples.
[0052] The ISM loss function is:
[0053] l2 = -ylogp
[0054] Where y represents whether the image and the 3D model are the same instance; if they are the same instance, y = 1, otherwise 0. p is the probability vector of whether the output of the attention network is the same instance.
[0055] After feature alignment, the distribution gap between the two domains will be reduced. The next step is to fuse cross-modal networks. Shared weights mean that the weights used in the two networks are the same.
[0056] Step 7: Backpropagate gradient descent to update the network.
[0057] Furthermore, the cross-modal network's image encoder and model encoder extract features from the image domain data and the 3D model domain data, respectively, to obtain image domain representation vectors and 3D model domain representation vectors. These image domain representation vectors and 3D model domain representation vectors are then passed through an attention mechanism network, where attention is applied between the rendered views of the image and the model to generate the final weighted feature vector.
[0058] Specifically, the image domain representation vector v of image i i Generated by the image encoder IE, i.e., v i =IE(i), the three-dimensional model domain representation vector v of model s s Generated by the model encoder SE, i.e., v s =SE(s).
[0059] Image domain representation vector v i and the 3D model domain representation vector v s Then, through the attention mechanism network AN, attention is applied between the image and the rendered view of the model to generate a final weighted fused feature vector.
[0060] Specifically, after generating the fused feature vector, it passes through a linear layer to project it into a two-dimensional vector.
[0061]
[0062] Where fc is a linear neural network, d k Used for scaling.
[0063] Step 6: Calculate the network loss using feature alignment and ISM.
[0064] Specifically,
[0065] Loss = loss1 + loss2
[0066] Step 7: Backpropagation gradient descent to update the network
[0067] As one embodiment, when performing a 3D model retrieval method based on cross-modal fusion of a single image, firstly, image i and 3D model s are processed by the IE and SE of the cross-modal network to extract the feature vector v of the query image. i and the 3D model feature vector v s Secondly, the similarity between the query image i and the 3D model s is obtained through the attention mechanism network AN:
[0068] p = AN(v) i ,v s )
[0069] The above formula can be used to calculate the similarity between the query image and all 3D models, thereby retrieving the most similar 3D model.
[0070] Example 2
[0071] One embodiment of this disclosure provides a 3D model retrieval system based on single-image cross-modal fusion, comprising:
[0072] The data acquisition module is used to acquire the image to be queried and a collection of multi-view 3D models;
[0073] The retrieval module is used to input the image to be queried and a set of 3D models rendered into multiple views into a trained cross-modal fusion network based on a single image, and output the corresponding 3D model retrieved.
[0074] The retrieval module includes image domain networks, model domain networks, cross-modal networks, and feature alignment modules.
[0075] The training process of the single-image cross-modal fusion network is as follows: A single image and a 3D model rendered into multiple views are input. A data exchange process is introduced, where image domain data is assigned to additional channels in the 3D model domain with a set probability, and model domain data is assigned to additional channels in the image domain with a set probability. Features are extracted from both the image domain data and the 3D model domain data, and after feature alignment in the image domain network and the 3D model domain network respectively, they are input into the cross-modal network to obtain image domain representation vectors and 3D code model domain representation vectors. Attention is introduced between the image domain representation vectors and the 3D code model domain representation vectors to fuse them and generate the final fused feature vector. This vector is then projected into a two-dimensional vector through a linear layer, completing the training of the single-image cross-modal fusion network.
[0076] Example 3
[0077] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the 3D model retrieval method based on single-image cross-modal fusion.
[0078] Example 4
[0079] One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the 3D model retrieval method based on single-image cross-modal fusion.
[0080] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0082] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.
Claims
1. A 3D model retrieval method based on cross-modal fusion of a single image, characterized in that, include: Retrieve the image and multi-view 3D model set to be queried; The image to be queried and a set of 3D models rendered into multiple views are input into a trained cross-modal fusion network based on a single image, and the corresponding 3D model is output. The training process of the single-image cross-modal fusion network is as follows: A single image and a 3D model rendered into multiple views are input. A data exchange process is introduced, where image domain data is assigned to additional channels in the 3D model domain with a set probability, and model domain data is assigned to additional channels in the image domain with a set probability. Features are extracted from both the image domain data and the 3D model domain data, and after feature alignment in the image domain network and the 3D model domain network respectively, they are input into the cross-modal network to obtain image domain representation vectors and 3D model domain representation vectors. Attention is introduced between the image domain representation vectors and the 3D model domain representation vectors to fuse them and generate the final fused feature vector. This vector is then projected into a two-dimensional vector through a linear layer, completing the training of the single-image cross-modal fusion network.
2. The 3D model retrieval method based on cross-modal fusion of a single image as described in claim 1, characterized in that, The image domain network consists of a query image encoder and a key image encoder, the 3D model domain network consists of a query model rendering view encoder and a key model rendering view encoder, and the cross-modal network consists of an image encoder, a model encoder, and an attention mechanism network, wherein the image encoder shares weights with the image domain network, and the model encoder shares weights with the 3D model domain network.
3. The 3D model retrieval method based on cross-modal fusion of a single image as described in claim 1, characterized in that, The image domain network and the 3D model domain network align the image domain feature vectors and the 3D model domain feature vectors respectively through a feature alignment module before feature fusion. Feature fusion is then performed after feature alignment.
4. The 3D model retrieval method based on cross-modal fusion of a single image as described in claim 1, characterized in that, After exchanging data between the image domain data and the 3D model domain data, and before extracting features from the image domain data and the 3D model domain data, the image domain data and the 3D model domain data are enhanced. The image domain data after the data exchange is enhanced by two different data enhancement methods to obtain two types of image domain enhanced data. The 3D model domain data is the multi-view data after the 3D model is rendered.
5. The 3D model retrieval method based on cross-modal fusion of a single image as described in claim 4, characterized in that, The two types of image domain augmented data are processed by the query image encoder and the key image encoder of the image domain network to obtain the first image domain feature vector and the second image domain feature vector. The first image domain feature vector and the second image domain feature vector are then aligned in the feature alignment module of the image domain network.
6. The 3D model retrieval method based on cross-modal fusion of a single image as described in claim 4, characterized in that, Two views are randomly selected from the multi-view data after the 3D model is rendered, and input into the query model rendering view encoder and the key model rendering view encoder respectively to obtain the first view feature vector and the second view feature vector. The first view feature vector and the second view feature vector are then aligned in the 3D model domain network.
7. The 3D model retrieval method based on cross-modal fusion of a single image as described in claim 1, characterized in that, The cross-modal network's image encoder and model encoder extract features from the image domain data and the 3D model domain data, respectively, to obtain image domain representation vectors and 3D model domain representation vectors. The image domain representation vectors and 3D model domain representation vectors are then passed through an attention mechanism network, where attention is applied between the rendered views of the image and the model to generate the final weighted feature vector.
8. A 3D model retrieval system based on cross-modal fusion of a single image, characterized in that, include: The data acquisition module is used to acquire the image to be queried and a collection of multi-view 3D models; The retrieval module is used to input the image to be queried and a set of 3D models rendered into multiple views into a trained cross-modal fusion network based on a single image, and output the corresponding 3D model retrieved. The training process of the single-image cross-modal fusion network is as follows: A single image and a 3D model rendered into multiple views are input. A data exchange process is introduced, where image domain data is assigned to additional channels in the 3D model domain with a set probability, and model domain data is assigned to additional channels in the image domain with a set probability. Features are extracted from both the image domain data and the 3D model domain data, and after feature alignment in the image domain network and the 3D model domain network respectively, they are input into the cross-modal network to obtain image domain representation vectors and 3D model domain representation vectors. Attention is introduced between the image domain representation vectors and the 3D model domain representation vectors to fuse them and generate the final fused feature vector. This vector is then projected into a two-dimensional vector through a linear layer, completing the training of the single-image cross-modal fusion network.
9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the three-dimensional model retrieval method based on cross-modal fusion of a single image as described in any one of claims 1-7.
10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the three-dimensional model retrieval method based on single-image cross-modal fusion as described in any one of claims 1-7.