Retrieval model training method and apparatus, retrieval method and apparatus, electronic equipment and storage medium

Abstract

This invention proposes a training method for a 3D model retrieval model, comprising the following steps: rendering 3D models from a 3D model database to obtain corresponding multi-view grayscale images; color-converting the query images in a training batch to obtain corresponding data-augmented images; feeding the data-augmented images and their corresponding mask images into a query image encoder to obtain the query code corresponding to the query image; feeding the multi-view grayscale images corresponding to the 3D models into a rendering encoder to obtain the rendering code corresponding to the multi-view grayscale images of the 3D models; feeding the query code and the rendering code into an attention mechanism module to obtain the specific code of the 3D model for the current query image; and optimizing using a loss function to obtain the 3D model retrieval model.

Description

Technical Field

[0001] This invention relates to the field of geometric processing in computer graphics, and more particularly to a method and apparatus for training a 3D model retrieval model, a 3D model retrieval method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] Multimedia retrieval, including image retrieval and 3D model retrieval, has always been a fundamental and important frontier research topic in computer vision and graphics. One of the research challenges is how to characterize the similarity relationship between retrieved images and 3D models in different modalities. Due to the development of deep learning and 3D model datasets with rich objects and categories, and their wide applications, including scene reconstruction, 3D printing, virtual reality, and e-commerce platforms, the task of 3D model retrieval based on a single real image has recently gained considerable attention. In some studies, the PnP algorithm is used to predict the pose of the 3D model during rendering from the image, and the predicted pose is used to render all 3D models in the database, calculating the similarity between the query image and the rendered image. This method avoids multi-view rendering of the 3D model, directly mapping the 3D model to the space of the 2D query image. In other studies, the 2D query image is transformed into a spatial location field through a network, and the 3D model is simultaneously transformed into the spatial location field through multi-view rendering. Thus, triplet loss is used to constrain the retrieval network in the spatial location field-based representation. In other studies, a conditional coding-based texture synthesis method is used to construct difficult-to-distinguish samples from a 3D model database, based on triplet loss. This method further improves the upper bound of triplet loss in this task by deeply mining the difficult-to-distinguish samples.

[0003] However, it should be noted that retrieval algorithms based on 3D pose prediction require a relatively accurate 3D pose prediction algorithm, which remains an open problem, as obtaining accurate 3D pose directly from images is difficult. Furthermore, in spatial position field-based representation algorithms, the spatial position field is highly sensitive to the model's location, easily leading to overfitting during training. Additionally, in supervised methods based on triplet loss, the construction of triples requires mining hard-to-classify samples. Due to the strong randomness of sample mining, most of the triples constituting the training samples become invalid samples in the later stages of retrieval model training, thus slowing down the convergence speed of the retrieval model and limiting further improvement in retrieval accuracy. For 3D model retrieval, whether it is possible to overcome the triplet loss barrier and further improve retrieval accuracy has become a key research focus. At the same time, existing technologies have unanimously ignored the interference caused by class labels in multi-class datasets on retrieval. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the main objective of this invention is to propose a three-dimensional model retrieval model training method and apparatus, a three-dimensional model retrieval method and apparatus, an electronic device and a storage medium, which can overcome problems such as difficult sample mining, interference of category information on retrieval, and coupling of color and shape features in image-based three-dimensional model retrieval tasks.

[0005] To achieve the above objectives, this invention proposes a three-dimensional model retrieval model training method, comprising:

[0006] Step 1: Render the 3D models in the 3D model database to obtain the corresponding multi-view grayscale images. Step 2: In a small batch of trained query images, perform color conversion on the query image to obtain the corresponding data-augmented image. Step 3: Feed the data-augmented image and its corresponding mask image into the query image encoder to obtain the query code corresponding to the query image. Step 4: Feed the multi-view grayscale image corresponding to the 3D model into the rendering encoder to obtain the rendering code corresponding to the multi-view grayscale image of the 3D model. Step 5: Feed the query code and the rendering code into the attention mechanism module to obtain the specific code of the 3D model for the current query image. Step 6: Optimize using a loss function to obtain the 3D model retrieval model.

[0007] The above-mentioned 3D model retrieval model training method, wherein step 2 includes:

[0008] Step 21: In the training query image mini-batch, randomly select other query images as the source style images for color conversion of the current query image;

[0009] Step 22: Convert the source style image and the target geometric image from RGB space to lαβ space. The color conversion process in space is as follows:

[0010]

[0011] Where lαβ and l'α'β' represent the values ​​of the three channels before and after color conversion, μ and σ represent the channel mean and variance, respectively, and s and t represent the source style image and the target geometric image, respectively;

[0012] Step 23: Convert the color-converted query image from lαβ back to RGB space to obtain the corresponding data-enhanced image.

[0013] The above-mentioned 3D model retrieval model training method, wherein step 3 includes:

[0014] Enhance the image with this data Its corresponding network parameter θ q And the query image qi mask image k i Input the query image encoder to obtain the query image q i Corresponding query code Right now

[0015]

[0016] Among them, f q (·,·,·) represents querying the image encoder.

[0017] The above-mentioned 3D model retrieval model training method, wherein step 4 includes:

[0018] This multi-view grayscale image r i Its corresponding network parameter θ r1 Input the rendering encoder to obtain the multi-view rendered grayscale image r i Corresponding rendering code Right now

[0019]

[0020] Among them, f r1 (·,·) represents the rendering encoder.

[0021] The above-mentioned 3D model retrieval model training method, wherein step 5 includes:

[0022] Encode the rendering The query code and its corresponding network parameters θ r2 Input the attention module to obtain this specific encoding. Right now

[0023]

[0024] Among them, f r2 (·,·,·) represents the attention module.

[0025] The above-mentioned 3D model retrieval model training method, wherein step 6 includes:

[0026] Step 61: Design a loss term for inter-instance supervision. The inter-instance loss term is represented in coded form as follows:

[0027] Where · denotes the inner product dot product, used to calculate the similarity between codes, τ∈R + It is a super parameter representing temperature;

[0028] Step 62: Design a loss term for inter-class supervision. The inter-class loss term is represented in coded form as follows:

[0029]

[0030] Where P(i) represents {j|j∈B\{B i}and y j =y i}, that is, with B i Other data with the same label.

[0031] In the above-described 3D model retrieval model training method, the loss function in step 6 is expressed as follows:

[0032] L total =L inst +β·L cats

[0033] Here, β is a hyperparameter used to control the ratio between the two loss terms.

[0034] To achieve the above objectives, the present invention also proposes a three-dimensional model retrieval model training device, comprising:

[0035] The rendering unit is used to render the 3D models in the 3D model database to obtain the corresponding multi-view grayscale images.

[0036] The color conversion unit is used to convert the query image in a small batch of training query images to obtain the corresponding data augmentation image;

[0037] The query image encoding unit is used to feed the data augmented image and its corresponding mask image into the query image encoder to obtain the query code corresponding to the query image.

[0038] The rendering encoding unit is used to send the multi-view rendering grayscale image corresponding to the 3D model into the rendering encoder to obtain the rendering encoding corresponding to the multi-view rendering grayscale image of the 3D model.

[0039] The attention mechanism unit is used to feed the query code and the rendering code into the attention mechanism module to obtain the specific code of the 3D model for the current query image;

[0040] The model optimization unit is used to optimize the 3D model retrieval model using a loss function.

[0041] To achieve the above objectives, the present invention also proposes a three-dimensional model retrieval method, comprising:

[0042] Step 1: Render the 3D models in the 3D model database to obtain corresponding multi-view grayscale images, and then send these multi-view grayscale images into the rendering encoder to obtain their corresponding rendering codes. Step 2: Send the query image and its corresponding mask image into the query image encoder to obtain the query code corresponding to the query image. Step 3: Send the rendering code and the query code into the attention mechanism module to obtain a specific code for the 3D model for the current query image. Step 4: Calculate the similarity between the specific code and the query code, and select the 3D model with the highest similarity as the final search result.

[0043] To achieve the above objectives, the present invention also proposes a three-dimensional model retrieval device, comprising:

[0044] The rendering encoding unit is used to render the three-dimensional models in the three-dimensional model database to obtain the corresponding multi-view rendering grayscale images, and send the multi-view rendering grayscale images into the rendering encoder to obtain their corresponding rendering codes.

[0045] The query encoding unit is used to feed the query image and its corresponding mask image into the query image encoder to obtain the query code corresponding to the query image.

[0046] The attention mechanism unit is used to feed the rendering code and the query code into the attention mechanism module to obtain a specific code for the 3D model for the current query image;

[0047] The calculation retrieval unit is used to calculate the similarity between the specific code and the query code, and select the 3D model with the highest similarity as the final retrieval result.

[0048] To achieve the above objectives, the present invention also proposes an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, the above-described 3D model retrieval model training method and / or the above-described 3D model retrieval method are implemented.

[0049] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the above-described 3D model retrieval model training method and / or the above-described 3D model retrieval method.

[0050] As can be seen from the above scheme, the advantages of this invention are as follows: This invention introduces a contrastive loss function to replace the triplet loss function, constructs dual contrastive loss functions between instances and between categories, and uses a color conversion module to perform image-based 3D model retrieval tasks. Compared with the traditional triplet loss, the method disclosed in this invention can avoid the construction of triples, especially the mining of hard-to-distinguish samples, thus improving the model convergence speed; compared with supervising the image-based 3D model retrieval task only from the perspective of between instances, the method disclosed in this invention combines the perspective of between categories for contrastive loss supervision, widening the encoding distance between objects of different categories, and reducing the interference caused by models with similar appearances but different categories to the retrieval; compared with ignoring the influence of color on the retrieval model and addressing the requirement of strong data augmentation for contrastive learning, the method disclosed in this invention uses a color conversion module to convert the color style of the query image during training, enabling the retrieval network to learn color-independent features; in summary, the method proposed in this invention greatly improves the accuracy of 3D model retrieval. Attached Figure Description

[0051] Figure 1 This is a flowchart of a three-dimensional model retrieval and model training method according to an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram of a three-dimensional model retrieval model training method according to an embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram illustrating the color conversion effect according to an embodiment of the present invention.

[0054] Figure 4 This is a schematic diagram illustrating the inter-instance and inter-class losses according to an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram illustrating the comparison test results of a retrieval model according to an embodiment of the present invention.

[0056] Figure 6 This is a schematic diagram illustrating the visualization of retrieval results on the ShapeNet dataset according to an embodiment of the present invention.

[0057] Figure 7 This is a schematic diagram showing the contrast between color conversion according to an embodiment of the present invention and traditional color enhancement.

[0058] Figure 8 This is a t-SNE diagram showing the encoded distance between a single query image and the entire 3D model dataset on the Pix3D dataset, according to an embodiment of the present invention.

[0059] Figure 9 This is a block diagram of a three-dimensional model retrieval and model training device according to an embodiment of the present invention.

[0060] Figure 10 This is a flowchart of a three-dimensional model retrieval method according to an embodiment of the present invention.

[0061] Figure 11 This is a block diagram of a three-dimensional model retrieval device according to an embodiment of the present invention.

[0062] Figure 12 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0063] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0064] Given the limitations of existing methods, this invention aims to improve the entire retrieval algorithm from the perspectives of supervision and data augmentation. A key focus of this invention is the construction of a dual contrastive loss function between instances and between categories to replace the triplet loss. As a popular research direction in recent years, contrastive learning uses contrastive loss to ensure that different data augmentations of the same sample have similar encodings, and different samples have different encodings, achieving great success in unsupervised representation learning. Specifically, in this work, this mechanism of contrastive learning coincides with metric learning methods in distinguishing different sample instances; and when distinguishing samples of different categories to reduce the interference of category labels on retrieval, category labels can be introduced into contrastive learning to form supervised contrastive learning, further widening the encoding gap between objects of different categories. Addressing the requirement for strong data augmentation when using contrastive learning, this work introduces color conversion to decouple color and shape features in the retrieved image, allowing the retrieval network to focus on learning color-independent features.

[0065] The purpose of this invention is to overcome the problems of difficult sample mining, interference from category information in retrieval, and coupling of color and shape features in image-based 3D model retrieval tasks. It proposes a single-image-based 3D model retrieval algorithm based on cross-modal contrastive learning, constructing a dual contrastive loss function between instances and between categories to replace the triplet loss, and introducing color conversion to decouple color and shape features in the retrieved image. This invention has the following three key points:

[0066] Key point 1: In the image-based 3D model retrieval task, contrastive loss is introduced to replace triplet loss; the technical effect is that it avoids the mining of hard samples in the construction of negative samples, thereby improving the model convergence speed and retrieval accuracy.

[0067] Key point 2: Introducing inter-category supervision information into the instance-based contrastive loss supervision term; the technical effect is that supervision between categories increases the distance between categories, reduces the interference caused by shapes that are similar in appearance but different in category to the retrieval, thereby improving the retrieval accuracy.

[0068] Key point 3: To address the requirement of strong data augmentation for contrastive learning, a color conversion module was introduced. The technical effect is that the color style of the query image is converted during training to decouple the color and shape features in the retrieval image, allowing the retrieval network to focus more on learning color-independent features, thereby improving retrieval accuracy.

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description, in conjunction with the accompanying drawings, provides a more comprehensive understanding of the single-image 3D model retrieval algorithm proposed in this invention, which utilizes cross-modal contrastive learning. It should be understood that the specific implementation methods described herein are merely illustrative of the invention and are not intended to limit the scope of the invention.

[0070] The objective of this invention is to retrieve, given a single query image and a 3D model database, the 3D model most similar to the object in the query image from the 3D model database. Since 3D models and 2D images are data from different modalities and cannot be directly calculated for similarity, this invention first converts the 3D model into a multi-view single-channel grayscale image corresponding to the 3D model using planar shadow rendering. The task then becomes, given a query image, to retrieve the 3D model most similar to the query image from the image set.

[0071] See Figure 1 and Figure 2 As shown, an embodiment of the present invention proposes a three-dimensional model retrieval model training method 100, including steps S110-S160. Wherein:

[0072] Step S110: Render the 3D model in the 3D model database to obtain the corresponding multi-view grayscale image.

[0073] In this embodiment, there exists a dataset S containing |S| data points, where the i-th data point s i From a query image q i A collection of grayscale images for multi-view rendering i (M perspectives) and a semantic label y i Formally, it can be expressed as:

[0074]

[0075] Step S120: In the training query image mini-batch, the query image is color-converted to obtain the corresponding data augmentation image.

[0076] This invention innovatively introduces a color transfer module during the retrieval model training phase for two main reasons: First, this invention introduces contrast loss in this task, and the training process of contrastive learning requires strong data augmentation. However, traditional data augmentation methods, such as affine transformation, are insufficient for this task because the retrieval network typically learns some color-related features. Second, the retrieval network co-encodes the shape and color in the query image. In other words, the color in the query image will have a certain impact on the retrieval results, and traditional HSV data augmentation for color cannot decouple the encoding of color and shape in the query image.

[0077] See Figure 3 As shown, in order to ensure that the query image q is used during the network model training process... i It can acquire a variable style in each training epoch, and the source of the style of each query image is randomly selected from other query images in the same mini-batch B.

[0078] In this embodiment, the present invention first converts the source style image and the target geometric image from RGB space to lαβ space. Compared with RGB space, lαβ space minimizes the correlation between different color channels and reduces the impact of operations on one channel on other channels. After all query images are converted to lαβ space, the color conversion process in lαβ space is as follows:

[0079]

[0080] lαβ and l'α'β' represent the values ​​of the three channels before and after color conversion, respectively. μ and σ represent the channel mean and variance, respectively. The subscripts s and t represent the source style image and the target geometric image.

[0081] Finally, the color-transformed query image is converted back from the lαβ space to the RGB space, and the color-transformed query image q is... i Noted as data augmented image

[0082] Step S130: The data augmented image and its corresponding mask image are fed into the query image encoder to obtain the query code corresponding to the query image.

[0083] A collection of grayscale images with clean backgrounds rendered from multiple perspectives i The difference is that the query image q i It contains a large amount of redundant background information, which can interfere with the retrieval network's ability to retrieve information from the query image q. iKey features are extracted, such as geometric structure features. To reduce interference from redundant background information, in this embodiment, the present invention obtains the query image q. i The mask image is used to guide the retrieval network to focus its attention on relevant objects in the two-dimensional image. Here, the query image q will be used. i The mask image is denoted as k i .

[0084] The retrieval network contains the query image encoder f q (·,·,·), representing the query image q i The encoder, input data augmentation image Its corresponding network parameter θ q And query image q i mask image k i Get the query image q i query code Right now

[0085]

[0086] Step S140: Send the multi-view grayscale image corresponding to the 3D model into the rendering encoder to obtain the rendering code corresponding to the multi-view grayscale image of the 3D model.

[0087] The search network contains the rendering encoder f r1 (·,·), representing the multi-view rendering grayscale image r i The encoder, input r i Its corresponding network parameter θ r1 Obtain multi-view rendered grayscale image r i Corresponding rendering code Right now

[0088]

[0089] Step S150: The query code and the rendering code are fed into the attention mechanism module to obtain the specific code of the 3D model for the current query image.

[0090] Similarity calculation is closely related to the choice of viewpoint, and rendering encoding... It is a set of codes for multi-view rendering of grayscale images, and it cannot yet be directly compared with the query image q. i query code The similarity is calculated directly. This invention uses an attention mechanism to process the rendering encoding. Merge into a single instance-based, query-specific encoding. That is, the multi-view rendering encoding of the i-th data is relative to the encoding of the merged retrieved image j. The attention mechanism module is represented as f.r2 (·,·,·), input multi-view rendering grayscale image r i Corresponding rendering code Query image q j query code and its corresponding network parameters θ r2 Get a specific code Right now

[0091]

[0092] Step S160: Optimize using a loss function to obtain the 3D model retrieval model.

[0093] After obtaining the query code of the query image Specific encoding of the query-specific 3D model See afterward. Figure 4 As shown, this invention is based on contrastive learning and designs loss terms for supervision between instances and between categories.

[0094] From the perspective of inter-instance relationships, in a small batch B for retrieving image q i q i The positive sample is the multi-view rendering image of its corresponding 3D model. i The remaining multi-view renderings of the 3D model are used as negative samples. Therefore, the loss term between instances is represented in encoded form as follows:

[0095]

[0096] Where · represents the inner product dot product, used to calculate the similarity between codes; τ∈R + It is a super parameter representing temperature.

[0097] From the perspective of inter-category relationships, in a small batch B for the query image q i q i At this point, a positive sample is a 3D model that has the same class label y as its corresponding model. i Multi-view renderings of all other 3D models. Therefore, the inter-class loss term is represented in encoded form as:

[0098]

[0099] Where P(i) represents {j|j∈B\{B i}and y j =y i}, that is, with B iOther data with the same labels. It can be noted that the triplet loss has only one positive sample and one negative sample; the between-instance loss has one positive sample and multiple negative samples; and the between-class loss has multiple positive samples and multiple negative samples.

[0100] Combining the losses between instances and between categories, the final loss function of the retrieval network is expressed as:

[0101] L total =L inst +β·L cats

[0102] Here, β is a hyperparameter used to control the ratio between the two loss terms.

[0103] During testing after the retrieval model training is complete, the input to the query image encoder is no longer processed by the color conversion module and is used directly. The 3D model with the highest similarity to the query image is selected as the retrieval result by calculating the similarity between the query image and all 3D models in the database.

[0104] The retrieval model was trained and tested on three datasets containing real images and 3D models: Pix3D, Comp Cars, and Stanford Cars. For the Pix3D dataset, after removing occluded images, data from four categories—bed, chair, sofa, and table—was selected, totaling 5118 images and 322 3D models. Comp Cars and Stanford Cars only had the car category. Stanford Cars provided 16185 images (8144 training images and 8041 test images) and 134 3D models. Comp Cars provided 5696 images (3798 training images and 1898 test images). Additionally, to test the generalization ability of the retrieval network, 3D model retrieval was also performed on the ShapeNet dataset.

[0105] This invention compares the results with four existing methods, using the retrieval accuracy of Top 1 and Top 10, HAU (mean modified Hausdorff Distance), and IoU (Intersection over Union) as metrics. See the comparison results below. Figure 5 As shown, the method proposed in this invention significantly outperforms existing methods in terms of Top1 and Top10 retrieval accuracy, as well as HAU and IoU, on Pix3D, Stanford Cars, and Comp Cars.

[0106] Furthermore, this invention also provides a visual representation of the search results on the ShapeNet dataset, see [link / reference]. Figure 6 As shown, the first column is the query image, the second column is the ground truth 3D model corresponding to the query image, and the last five columns are the top five similarity results retrieved from the ShapeNet 3D model dataset. It can be seen that the 3D model retrieved by this invention is very similar to the ground truth 3D model of the query image.

[0107] In addition, this invention also explores the functions of internal modules, specifically including the functions of the color conversion module and the inter-category loss function.

[0108] See Figure 7 As shown, this invention compares color conversion with traditional color enhancement such as HSV enhancement. It can be seen that the color conversion module used in this invention can produce a more realistic and natural result, rather than performing random perturbation enhancement, because the module can learn a more realistic color distribution from other source style images.

[0109] See Figure 8 The figure illustrates the impact of the inter-class loss function on the retrieval network. It shows the t-SNE results for the encoded distance between a query image and the entire 3D model dataset on the Pix3D dataset. The left side shows the results without the inter-class loss function, and the right side shows the results with the inter-class loss function. It can be seen that using the inter-class loss function helps the retrieval network increase the distance between different classes and decrease the distance between classes within the same class, thereby reducing cross-class errors in the retrieval results.

[0110] In addition, see Figure 9 As shown, based on the same inventive concept, embodiments of the present invention also propose a three-dimensional model retrieval model training device 200, comprising:

[0111] The rendering unit 210 is used to render the three-dimensional models in the three-dimensional model database to obtain the corresponding multi-view rendering grayscale images.

[0112] Color conversion unit 220 is used to perform color conversion on the query image in a small batch of trained query images to obtain the corresponding data augmentation image;

[0113] The query image encoding unit 230 is used to send the data augmented image and its corresponding mask image into the query image encoder to obtain the query code corresponding to the query image.

[0114] The rendering encoding unit 240 is used to send the multi-view rendering grayscale image corresponding to the 3D model into the rendering encoder to obtain the rendering encoding corresponding to the multi-view rendering grayscale image of the 3D model.

[0115] Attention mechanism unit 250 is used to feed the query code and the rendering code into the attention mechanism module to obtain the specific code of the 3D model for the current query image;

[0116] The model optimization unit 260 is used to optimize using a loss function to obtain the 3D model retrieval model.

[0117] The training of the 3D model retrieval model was completed using the above method. The application of the retrieval model will be explained below. (See also...) Figure 10 As shown, an embodiment of the present invention proposes a three-dimensional model retrieval method 300, comprising:

[0118] Step S310: Render the three-dimensional model in the three-dimensional model database to obtain the corresponding multi-view rendering grayscale image, and send the multi-view rendering grayscale image into the rendering encoder to obtain its corresponding rendering code.

[0119] Step S320: The query image and its corresponding mask image are fed into the query image encoder to obtain the query code corresponding to the query image;

[0120] Step S330: The rendering encoding and the query encoding are fed into the attention mechanism module to obtain the specific encoding of the 3D model for the current query image;

[0121] Step S340: Calculate the similarity between the specific code and the query code, and select the 3D model with the highest similarity as the final retrieval result.

[0122] See Figure 11 As shown, based on the same inventive concept, embodiments of the present invention also propose a three-dimensional model retrieval device 400, comprising:

[0123] The rendering encoding unit 410 is used to render the three-dimensional model in the three-dimensional model database to obtain the corresponding multi-view rendering grayscale image, and send the multi-view rendering grayscale image into the rendering encoder to obtain its corresponding rendering code.

[0124] The query encoding unit 420 is used to send the query image and its corresponding mask image into the query image encoder to obtain the query code corresponding to the query image.

[0125] Attention mechanism unit 430 is used to feed the rendering code and the query code into the attention mechanism module to obtain a specific code of the 3D model for the current query image;

[0126] The calculation retrieval unit 440 is used to calculate the similarity between the specific code and the query code, and select the three-dimensional model with the highest similarity as the final retrieval result.

[0127] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the methods, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] See Figure 12 As shown, based on the same inventive concept, an embodiment of the present invention also proposes an electronic device 500, including: a processor 510 and a memory 520, wherein the memory 520 stores a computer program that can be run on the processor 510, and when the computer program is executed by the processor 510, it implements the three-dimensional model retrieval model training method 100 and / or the three-dimensional model retrieval method 300 of the above embodiments.

[0129] Based on the same inventive concept, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, wherein, when the computer program is executed by a processor, it implements the three-dimensional model retrieval model training method and / or the three-dimensional model retrieval method of the above embodiments.

[0130] In summary, this invention addresses image-based 3D model retrieval by introducing a contrastive loss function instead of a triplet loss function, constructing dual contrastive loss functions between instances and between categories, and incorporating a color conversion module. Compared to traditional triplet loss, the method disclosed in this invention avoids the construction of triples, particularly in identifying hard-to-distinguish samples, thus improving model convergence speed. Compared to supervising image-based 3D model retrieval solely from an instance-to-instance perspective, this invention combines contrastive loss supervision with an inter-category perspective, widening the encoding distance between objects of different categories and reducing interference from models with similar appearances but different categories. Furthermore, compared to ignoring the impact of color on the retrieval model and addressing the need for strong data augmentation in contrastive learning, this invention uses a color conversion module to transform the color style of the query image during training, enabling the retrieval network to learn color-independent features. Combining these three key points, the retrieval accuracy is significantly improved.

[0131] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A three-dimensional model search model training method characterized by comprising: The method comprises the following steps: Step 1, rendering a three-dimensional model in a three-dimensional model database to obtain a corresponding multi-view rendered grayscale image; Step 2, in a small batch of training query images, performing color conversion on the query image to obtain a corresponding data enhanced image; Step 3, inputting the data enhanced image and the corresponding mask image into a query image encoder to obtain a query code corresponding to the query image; Step 4, inputting the multi-view rendered grayscale image corresponding to the three-dimensional model into a rendering encoder to obtain a rendering code corresponding to the multi-view rendered grayscale image of the three-dimensional model; Step 5, inputting the query code and the rendering code into an attention mechanism module to obtain a specific code of the three-dimensional model for the current query image; Step 6, optimizing by using a loss function to obtain a three-dimensional model retrieval model.

2. The three-dimensional model search model training method according to claim 1, wherein The step 2 comprises: Step 21, in the small batch of training query images, randomly selecting other query images as source style images for color conversion; Step 22, convert the source style image and the target geometry image from RGB space to Spatially, the spatial color conversion process is as follows: wherein, and respectively represent the values of the three channels before and after color conversion, and respectively represent the channel mean and variance, and respectively represent the source style image and the target geometry image; Step 23, the color-converted query image is converted from Back to RGB space to obtain the corresponding data enhanced image.

3. The three-dimensional model search model training method according to claim 1, wherein The step 3 comprises: The data augmented image corresponding network parameters and the query image mask image inputting the query image into the query image encoder to obtain the query image corresponding query encoding i.e. wherein, represents a query image encoder.

4. The three-dimensional model search model training method according to claim 3, wherein The step 4 comprises: rendering grayscale image corresponding network parameters inputting the rendered grayscale image corresponding rendering encoding i.e. wherein, represents the rendering encoder.

5. The three-dimensional model search model training method according to claim 4, wherein The step 5 comprises: encoding of the rendering , the query encoding and its corresponding network parameters inputting the attention mechanism module to obtain the specific encoding i.e. wherein, denotes an attention mechanism module.

6. The three-dimensional model search model training method according to claim 5, wherein The step 6 comprises: Step 61, designing an inter-instance loss term for supervision, and the inter-instance loss term is expressed in the form of code: wherein, denotes an inner product multiplication used to compute the similarity between encodings, is a super parameter representing temperature; Step 62, designing an inter-class loss term for supervision, and the inter-class loss term is expressed in the form of code: wherein represents i.e. other data having the same label as 7. The three-dimensional model search model training method according to claim 6, wherein The loss function in the step 6 is expressed as follows: where, is a hyperparameter used to control the ratio between the two loss terms.

8. A three-dimensional model search model training apparatus characterized by comprising: The method comprises the following steps: A rendering unit is configured to render a three-dimensional model in a three-dimensional model database to obtain a corresponding multi-view rendered grayscale image; A color conversion unit is configured to, in a small batch of training query images, perform color conversion on the query image to obtain a corresponding data enhanced image; A query image encoding unit is configured to input the data enhanced image and the corresponding mask image into a query image encoder to obtain a query code corresponding to the query image; A rendering encoding unit is configured to input the multi-view rendered grayscale image corresponding to the three-dimensional model into a rendering encoder to obtain a rendering code corresponding to the multi-view rendered grayscale image of the three-dimensional model; An attention mechanism unit is configured to input the query code and the rendering code into an attention mechanism module to obtain a specific code of the three-dimensional model for the current query image; A model optimization unit is configured to optimize by using a loss function to obtain a three-dimensional model retrieval model.

9. A three-dimensional model search method characterized by comprising: The method comprises the following steps: Step 1, rendering a three-dimensional model in a three-dimensional model database to obtain a corresponding multi-view rendered grayscale image, and inputting the multi-view rendered grayscale image into a rendering encoder to obtain a corresponding rendering code; Step 2, inputting a query image and a corresponding mask image into a query image encoder to obtain a query code corresponding to the query image; Step 3, inputting the rendering code and the query code into an attention mechanism module to obtain a specific code of the three-dimensional model for the current query image; Step 4, calculating the similarity between the specific code and the query code, and selecting the three-dimensional model with the highest similarity as the final retrieval result.

10. A three-dimensional model search device characterized by comprising: The method comprises the following steps: a rendering encoding unit, configured to render a three-dimensional model in a three-dimensional model database to obtain a corresponding multi-view rendered grayscale image, and send the multi-view rendered grayscale image into a rendering encoder to obtain a corresponding rendering encoding of the multi-view rendered grayscale image; a query encoding unit, configured to send a query image and a corresponding mask image of the query image into a query image encoder to obtain a corresponding query encoding of the query image; an attention mechanism unit, configured to send the rendering encoding and the query encoding into an attention mechanism module to obtain a specific encoding of the three-dimensional model for the current query image; a calculation retrieval unit, configured to calculate a similarity between the specific encoding and the query encoding, and select a three-dimensional model with a highest similarity as a final retrieval result.

11. An electronic device, comprising: comprising: a processor and a memory, wherein the memory stores a computer program capable of being executed by the processor, and when the computer program is executed by the processor, the method according to any one of claims 1-7 and / or the method according to claim 9 is implemented.

12. A computer readable storage medium storing a computer program, wherein the computer program comprises program instructions configured to cause a processor to perform the method according to any one of claims 1 to 11. the computer program is executed by the processor, the method according to any one of claims 1-7 and / or the method according to claim 9 is implemented.

Images