Three-dimensional model retrieval method and system based on searching images by images

Abstract

The invention relates to the technical field of three-dimensional model retrieval, in particular to a three-dimensional model retrieval method and system based on searching images by images, and the method comprises the following steps: constructing a model library, carrying out two-dimensional feature image sketch conversion on a to-be-retrieved three-dimensional model, carrying out detail optimization on the two-dimensional feature image sketch, and carrying out retrieval on the two-dimensional feature image sketch. Searching pictures similar to the fine three views of the to-be-searched three-dimensional model in a file system of the model library by using a picture search algorithm, obtaining the similarity of the similar pictures, and screening out the corresponding three-dimensional model in the model library according to the similarity; the three-dimensional model retrieval method and the three-dimensional model retrieval system have the beneficial effects that the required model can be quickly and accurately retrieved from the three-dimensional model library only by simply building the model or searching based on the existing model by a user; by using the method, the accuracy and the retrieval experience of three-dimensional model retrieval are improved, and the efficiency of modeling workers in a three-dimensional scene building process is improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional model retrieval technology, and in particular to a three-dimensional model retrieval method and system based on image-by-image search. Background Art

[0002] Image search algorithm is a content-based image retrieval technology that searches for similar images by analyzing the image content. This technology is widely used in daily life, such as searching for the same product on e-commerce platforms and discovering related content on social media. There are many ways to implement image search, including but not limited to deep learning-based methods, perceptual hashing algorithms, local feature extraction methods, global feature extraction methods, etc. The choice of image search algorithm depends on the specific application scenario and requirements, including retrieval accuracy, speed, system scalability, and hardware resource requirements. With the development of deep learning technology, deep learning-based methods are increasingly valued due to their superior performance.

[0003] Stable Diffusion's img2img (image to image) algorithm is an advanced image generation technique that generates new images based on an initial image and text prompts provided by the user. The core of this technique is to use a diffusion model to generate images through an iterative denoising process, so as to achieve a user-specified visual effect or style transfer while maintaining similarity to the original image. In img2img mode, users can upload an image and provide positive and negative text prompts, and the model will adjust the content and style of the generated image based on these prompts. For example, a user can upload a landscape photo and add the text prompt "oil painting style", and the model will try to convert the landscape photo into an oil painting style image. In addition, users can set some parameters to control the generation process, such as resizing the image, selecting the sampling method, setting the number of denoising steps, the number of batches, etc. Stable Diffusion's img2img algorithm also includes some advanced features such as sketching. Sketching allows users to add or modify content on an existing image and then refine the details of the image through text prompts.

[0004] 3D model retrieval is a technology that searches for similar or specific 3D models to a query in a 3D dataset. Common 3D model retrieval methods include: 1. Retrieval based on geometric features: This method retrieves by analyzing the geometric shape features of the 3D model, such as edges, corners, surface normals, etc.; 2. Retrieval based on shape descriptors: This method uses shape descriptors (such as curvature, diameter, etc.) to describe the 3D model and matches based on these descriptors. 3. Retrieval based on topological structure: This method focuses on the connection relationship and structure of the 3D model and retrieves by analyzing the topological features of the model. 4. Content-based retrieval (CBIR): This is a more advanced method that not only considers geometric and topological features, but may also combine color, texture and semantic information; 5. Keyword-based retrieval: Users can retrieve related 3D models by entering keywords (such as model name, purpose, etc.)

[0005] The 3D model retrieval algorithms currently on the market are very inefficient for 3D scene builders and modelers. In most scenarios, the mainstream 3D model retrieval method still relies on keyword searches. This method not only relies heavily on the worker's understanding of the model library, but also has a very low retrieval efficiency, making it difficult to find the expected model in one go.

[0006] Therefore, how to implement a solution that can more quickly retrieve a model that meets the expectations from a 3D model library has become one of the problems that must be solved when performing model retrieval in the process of building a 3D scene. Summary of the invention

[0007] The purpose of the present invention is to provide a three-dimensional model retrieval method and system based on image search to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a three-dimensional model retrieval method based on image search, the method comprising the following steps:

[0009] a) constructing a model library, which uses a relational database to store data of three-dimensional models, and each data record at least includes a storage address of the three-dimensional model and unique identifiers of the front view, side view and top view of the three-dimensional model;

[0010] b) When uploading a 3D model resource to the model library, adjust the direction of the 3D model through the visual interface and capture the front view, side view and top view, store the 3D model in the object storage, store the three views in the file system, and record the corresponding unique identifier in the relational database;

[0011] c) converting the 3D model to be retrieved into a 2D feature image sketch by adjusting the direction of the 3D model to be retrieved and capturing its three views as a 2D feature sketch;

[0012] d) Optimize the details of the 2D feature image sketch and generate refined three-view images through the img2img function of the diffusion model;

[0013] e) using an image search algorithm to search for images similar to the detailed three-view images of the three-dimensional model to be retrieved in the file system of the model library, and obtaining the similarity of the similar images;

[0014] f) According to the similarity, the corresponding three-dimensional model in the model library is selected, and the comprehensive similarity of each matching three-dimensional model is calculated, and the three-dimensional model with the highest similarity is returned.

[0015] Preferably, in step a), the storage address of the three-dimensional model is the download address of the object storage, the two-dimensional images of the three views are stored in different folders, and the unique identifier is stored as the picture name.

[0016] Preferably, in step b), when uploading the three-dimensional model resource, the three-dimensional model is loaded and rendered in real time through the programming tool library. The user adjusts the direction of the model to determine the front view, and then the program automatically captures the side view and the top view, and stores them in the file system, and records the corresponding unique identifier in the relational database.

[0017] Preferably, in step d), the img2img function of the diffusion model is used to take the three views of the model to be retrieved as input, and the output style of the sketch drawing is controlled by presetting prompt words and LoRA parameter tuning to generate refined three views.

[0018] Preferably, in step f), the comprehensive similarity of each matching three-dimensional model is calculated based on the similarity of the front view, side view and top view, wherein the weight of the front view is relatively large to improve the retrieval effect, and finally the access address of the retrieved three-dimensional model is returned according to the size of the comprehensive similarity.

[0019] A three-dimensional model retrieval system based on image search is applied to a three-dimensional model retrieval method based on image search, the system comprising:

[0020] A model library construction module is used to construct a model library containing a 3D model and its corresponding three-view images, wherein the storage address of the 3D model, the unique identifier of the three-view image and other information are stored in a relational database, the 3D model itself adopts an object storage method, and the three-view image adopts a file system storage method;

[0021] The 3D model upload module provides a visual interface for users to adjust the 3D model direction and determine the front view, then automatically captures the side view and top view, uploads the 3D model and its three-view images to the model library, and records relevant information in the relational database;

[0022] The model processing module to be retrieved is used to convert the three-dimensional model to be retrieved specified by the user into a two-dimensional feature image sketch, including the steps of adjusting the model direction and capturing three views;

[0023] Sketch optimization module, which uses the img2img function of the diffusion model to optimize the details of the two-dimensional feature image sketch and generate refined three-view images;

[0024] The image search retrieval module uses the optimized three-view image as the query image, uses the deep learning-based image search algorithm to retrieve similar images in the model library, and selects the corresponding 3D models based on the similarity;

[0025] The result presentation module is used to calculate the comprehensive similarity of each matching 3D model, sort them by similarity, return the search results, and present them to the user through a visual interface.

[0026] Preferably, in the model library construction module, the relational database contains at least the following fields: the storage address of the three-dimensional model, the unique identifier of the three-dimensional model front view image, the unique identifier of the three-dimensional model side view image, and the unique identifier of the three-dimensional model top view image.

[0027] Preferably, in the 3D model uploading module, after the user adjusts the direction of the 3D model and determines the front view through the visual interface, the system uses a programming tool library, such as Open3D, to load and render the visual 3D model in real time, and automatically capture the side view and top view, and then upload the 3D model and its three-view images to the model library.

[0028] Preferably, in the sketch optimization module, when the img2img function of the diffusion model is used to optimize the details of the two-dimensional feature image sketch, the output style of the sketch drawing is controlled by presetting prompt words and LoRA parameter tuning. The user inputs optional prompt words as needed to generate a detailed three-view drawing of the three-dimensional model that meets the requirements.

[0029] Preferably, in the image-search retrieval module, when searching for similar images in the model library using the image-search algorithm based on deep learning, the front view, side view, and top view are searched separately, and the corresponding three-dimensional models are screened out according to the similarity; when calculating the comprehensive similarity, the weight of the front view is relatively large to improve the retrieval effect.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The 3D model retrieval method and system based on image search proposed in the present invention only require users to quickly and accurately retrieve the required model from the 3D model library through simple model construction or search based on existing models. This method is used to improve the accuracy and retrieval experience of 3D model retrieval, and thereby improve the efficiency of modeling workers in the process of building 3D scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] For example, see Figure 1 The present invention provides a technical solution: a three-dimensional model retrieval method based on image search, the method comprising the following steps:

[0035] 1. Construction of model library

[0036] In order to achieve the purpose of matching the 3D target model in the model library through the 2D image, it is necessary to formulate a method for constructing the model library, which specifically includes the following aspects:

[0037] (1) The data structure of the model library uses a relational database and needs to include at least the following fields: the storage address of the 3D model, the unique identifier of the front view of the 3D model, the unique identifier of the side view of the 3D model, and the unique identifier of the top view of the 3D model. Each relational data corresponds to a model in the model library. It is recommended to use object storage to store the 3D model, and use the download address of the object storage as the storage address of the 3D model. It is recommended to use a file system to store the 2D images of the three views. Images of different views are placed in a separate folder, and the unique identifier is stored as the image name.

[0038] (2) Uploading of 3D model resources: When uploading a 3D model to the model library, a visual interface needs to be provided to help users adjust the front view of the 3D model. Here, a 3D mesh model in ply format is taken as an example. With the help of a programming tool library, such as Python's Open3D tool library, the ply model can be loaded, rendered and visualized in real time. The user needs to adjust the size and direction of the model on the window, determine the front view of the 3D model, and click Upload, and the 3D model will be uploaded to the object storage; the software program captures the front view of the 3D model in the window and converts it into a 2D image, which is stored in the front view file system; then the program rotates the model 90 degrees around the vertical axis (usually the Y axis) so that its side faces the window, and then captures the window content, obtains the side view image of the model and stores it in the file system. This process is called by the program background and is not presented on the display screen; finally, the program resets the model, rotates it 90 degrees around the horizontal axis (usually the X axis) so that its top faces the window, and then captures the window content, obtains the front view image of the model and stores it in the file system, and stores the corresponding three-view image ID in the relational database, thus completing the upload of a 3D model resource.

[0039] 2. Retrieval method of 3D model

[0040] The model retrieval process needs to go through several main steps: converting the 3D model to be retrieved into a 2D feature image sketch, optimizing the details of the 2D feature image sketch, matching similar images in the model library using the image search algorithm, and selecting the corresponding 3D model in the model library based on the feature image similarity. The specific process is as follows:

[0041] (1) Converting the 3D model to be retrieved into a 2D feature image sketch: Modelers can initiate the retrieval of 3D models in two ways: the first is to assemble and modify simple geometric models to build the basic shape of the expected model; the second is to initiate a retrieval of an existing imperfect 3D model in the scene and retrieve a similar 3D model from the model library. Regardless of the method, the subsequent retrieval process is the same. After the user determines the model to be retrieved, adjust the 3D model to face the screen, click on the search button, and the system will obtain the 3D view of the 3D model to be retrieved as the 2D feature sketch of the model to be retrieved according to the 3D view capture method of "Model Library Construction-3D Model Resource Upload".

[0042] (2) Optimization of 2D feature image sketch details: Since the models to be retrieved are often textured and incomplete in shape, the sketch drawing capability in img2img, which is beneficial to the diffusion model, is used to take the three-view image of the model to be retrieved as input. The output style of the sketch drawing is controlled by presetting prompt words + LoRA parameter tuning. The user can then input optional prompt words as needed to generate the detailed three-view image of the 3D model that the user wants.

[0043] (3) The two-dimensional feature image uses the image search algorithm to match similar images in the model library: After using the diffusion model to generate refined three-view images, the front view, side view, and top view are searched in the corresponding file system of the model library. Taking the front view as an example, the front view drawn by the sketch is searched in the front view file system of the model library using the image search algorithm. Since the sketch is generated by generative AI, the drawn image must not exist in the front view file library of the model library. Therefore, it is recommended to use the image search algorithm based on deep learning to search and improve the search effect. After retrieving the topN similar images, the similarity of each similar image is obtained. In this way, the topN similar images of the side view and the top view are retrieved respectively, and then the relational database is queried to find the 3D model corresponding to each similar image in the model library, and the 3D model address corresponding to these three views is obtained.

[0044] (4) Filter out the corresponding 3D models in the model library based on the similarity of the feature images: After obtaining the 3D models corresponding to the similar three-view images in the model library, it is necessary to perform weighted calculation on the retrieved 3D models to obtain the final similar 3D model list:

[0045] Assume that the retrieved 3D model A is matched with the front view of the model to be retrieved, and the similarity is x. At the same time, the side view of the model to be retrieved also matches the side view of the 3D model A, and the similarity is y. The top view of the model to be retrieved does not match the top view of the 3D model A, and the similarity z is 0. Then: The similarity of 3D model A

[0046] F=x*m+y*n+z*q

[0047] Among them, z is counted as 0 because no image is matched;

[0048] m, n, and q represent the weights of the front, side, and top views, respectively. It is generally recommended to design the weight of the front view to be larger to improve the retrieval effect.

[0049] In this way, the comprehensive similarity of each matched 3D model is calculated, and the models are sorted in reverse order according to the similarity. Then, the access address of the retrieved 3D model is returned according to the similarity, and loaded and presented through a visual interface. The user clicks or drags the model to load it into the 3D target scene.

[0050] Through the above method, the complete process from initiating the search of the model to be retrieved from the model library to retrieving the target three-dimensional model is finally completed.

[0051] Embodiment 2, based on embodiment 1, proposes a 3D model retrieval system based on image search, which is applied to a 3D model retrieval method based on image search. The system includes:

[0052] A model library construction module is used to construct a model library including a three-dimensional model and its corresponding three views, wherein the storage address of the three-dimensional model, the unique identifier of the three-view image and other information are stored in a relational database, the three-dimensional model itself adopts an object storage method, and the three-view image adopts a file system storage method, and the relational database contains at least the following fields: the storage address of the three-dimensional model, the unique identifier of the three-dimensional model front view image, the unique identifier of the three-dimensional model side view image, and the unique identifier of the three-dimensional model top view image;

[0053] The 3D model upload module provides a visual interface for users to adjust the direction of the 3D model and determine the front view, then automatically captures the side view and top view, uploads the 3D model and its three-view images to the model library, and records the relevant information in the relational database. After the user adjusts the direction of the 3D model and determines the front view through the visual interface, the system uses a programming tool library, such as Open3D, to load and render the visual 3D model in real time, automatically captures the side view and top view, and then uploads the 3D model and its three-view images to the model library;

[0054] The model processing module to be retrieved is used to convert the three-dimensional model to be retrieved specified by the user into a two-dimensional feature image sketch, including the steps of adjusting the model direction and capturing three views;

[0055] Sketch optimization module, which uses the img2img function of the diffusion model to optimize the details of the 2D feature image sketch and generate refined three-view images. When using the img2img function of the diffusion model to optimize the details of the 2D feature image sketch, the output style of the sketch drawing is controlled by presetting prompt words and LoRA parameter tuning. Users can input optional prompt words as needed to generate refined three-view images of the 3D model that meet their needs;

[0056] The image search retrieval module uses the optimized three-view image as the query image, and uses the deep learning-based image search algorithm to search similar images in the model library, and selects the corresponding three-dimensional models according to the similarity. When searching similar images in the model library using the deep learning-based image search algorithm, the front view, side view, and top view are searched respectively, and the corresponding three-dimensional models are selected according to the similarity; when calculating the comprehensive similarity, the weight of the front view is relatively large to improve the retrieval effect;

[0057] The result presentation module is used to calculate the comprehensive similarity of each matching 3D model, sort them by similarity, return the search results, and present them to the user through a visual interface.

[0058] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional model retrieval method based on image search, characterized in that: The method comprises the following steps: a) constructing a model library, which uses a relational database to store data of three-dimensional models, and each data record at least includes a storage address of the three-dimensional model and unique identifiers of the front view, side view and top view of the three-dimensional model; b) When uploading a 3D model resource to the model library, adjust the direction of the 3D model through the visual interface and capture the front view, side view and top view, store the 3D model in the object storage, store the three views in the file system, and record the corresponding unique identifier in the relational database; c) converting the 3D model to be retrieved into a 2D feature image sketch by adjusting the direction of the 3D model to be retrieved and capturing its three views as a 2D feature sketch; d) Optimize the details of the 2D feature image sketch and generate refined three-view images through the img2img function of the diffusion model; e) using an image search algorithm to search for images similar to the detailed three-view images of the three-dimensional model to be retrieved in the file system of the model library, and obtaining the similarity of the similar images; f) According to the similarity, the corresponding three-dimensional model in the model library is selected, and the comprehensive similarity of each matching three-dimensional model is calculated, and the three-dimensional model with the highest similarity is returned.

2. The three-dimensional model retrieval method based on image search according to claim 1, characterized in that: In step a), the storage address of the three-dimensional model is the download address of the object storage, the two-dimensional images of the three views are stored in different folders, and the unique identifier is stored as the picture name.

3. The three-dimensional model retrieval method based on image search according to claim 1, characterized in that: In step b), when uploading the three-dimensional model resource, the programming tool library loads and renders the visual three-dimensional model in real time. The user adjusts the model direction to determine the front view, and then the program automatically captures the side view and top view and stores them in the file system, and records the corresponding unique identifier in the relational database.

4. The three-dimensional model retrieval method based on image search according to claim 1, characterized in that: In step d), the img2img function of the diffusion model is used to take the three views of the model to be retrieved as input, and the output style of the sketch drawing is controlled by presetting prompt words and tuning LoRA parameters to generate refined three views.

5. The three-dimensional model retrieval method based on image search according to claim 1, characterized in that: In step f), the comprehensive similarity of each matched 3D model is calculated based on the similarity of the front view, side view and top view, where the weight of the front view is relatively large to improve the retrieval effect. Finally, the access address of the retrieved 3D model is returned according to the size of the comprehensive similarity.

6. A three-dimensional model retrieval system based on image search, applied to the three-dimensional model retrieval method based on image search according to any one of claims 1 to 5, characterized in that: The system comprises: A model library construction module is used to construct a model library containing a 3D model and its corresponding three-view images, wherein the storage address of the 3D model, the unique identifier of the three-view image and other information are stored in a relational database, the 3D model itself adopts an object storage method, and the three-view image adopts a file system storage method; The 3D model upload module provides a visual interface for users to adjust the 3D model direction and determine the front view, then automatically captures the side view and top view, uploads the 3D model and its three-view images to the model library, and records relevant information in the relational database; The model processing module to be retrieved is used to convert the three-dimensional model to be retrieved specified by the user into a two-dimensional feature image sketch, including the steps of adjusting the model direction and capturing three views; Sketch optimization module, which uses the img2img function of the diffusion model to optimize the details of the two-dimensional feature image sketch and generate refined three-view images; The image search retrieval module uses the optimized three-view image as the query image, uses the deep learning-based image search algorithm to retrieve similar images in the model library, and selects the corresponding 3D models based on the similarity; The result presentation module is used to calculate the comprehensive similarity of each matching 3D model, sort them by similarity, return the search results, and present them to the user through a visual interface.

7. A three-dimensional model retrieval system based on image search according to claim 6, characterized in that: In the model library construction module, the relational database contains at least the following fields: the storage address of the 3D model, the unique identifier of the front view image of the 3D model, the unique identifier of the side view image of the 3D model, and the unique identifier of the top view image of the 3D model.

8. The three-dimensional model retrieval system based on image search according to claim 6, characterized in that: In the 3D model upload module, after the user adjusts the direction of the 3D model and determines the front view through the visual interface, the system uses a programming tool library, such as Open3D, to load and render the visual 3D model in real time, automatically capture the side view and top view, and then upload the 3D model and its three-view images to the model library.

9. The three-dimensional model retrieval system based on image search according to claim 6, characterized in that: In the sketch optimization module, when the img2img function of the diffusion model is used to optimize the details of the two-dimensional feature image sketch, the output style of the sketch drawing is controlled by presetting prompt words and LoRA parameter tuning. The user enters optional prompt words as needed to generate detailed three-view drawings of the three-dimensional model that meet the requirements.

10. The three-dimensional model retrieval system based on image search according to claim 6, characterized in that: In the image search retrieval module, when searching similar images in the model library using the deep learning-based image search algorithm, the front view, side view, and top view are searched separately, and the corresponding three-dimensional models are screened out based on the similarity; when calculating the comprehensive similarity, the weight of the front view is relatively large to improve the retrieval effect.

Images