Search system and search method

Abstract

To provide a search system and a search method that realize a highly convenient 3D object search.SOLUTION: A search system includes: a three-dimensional object generation unit that generates an input three-dimensional object from an input image; an image generation unit that generates a plurality of images of the input three-dimensional object viewed from a plurality of viewpoints as a search image group; an image feature amount extraction unit that calculates an image feature amount from each image included in the search image group; an image search unit that searches a database where the plurality of images of three-dimensional objects to be searched is registered, the plurality of images being viewed from the plurality of viewpoints, using each of the images included in the search image group as a search query; and an image search result integration unit that integrates search results for each of the images included in the search image group for each three-dimensional object to be searched, and outputs a three-dimensional object similar to the input image among the three-dimensional objects to be searched as a similar three-dimensional object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a search system and a search method for searching for 3D objects. [Background technology]

[0002] With the spread of computer-aided design (CAD) technology and software, the use of 3D (3D) data is becoming more widespread in product design, manufacturing, architecture, construction, and other fields. Because this 3D data is digital, it can be easily stored, creating a need for efficient search of the vast amounts of accumulated 3D data. Furthermore, if it were possible to search for visually similar 3D data using not only the bibliographic information associated with the 3D data but also a single image, such as a sketch or photograph from the idea stage, this would accelerate the use of 3D data, such as by improving work efficiency in the early design stages and searching for related information from on-site photographs.

[0003] Similarity image search technology is known as a method for finding visually similar images from a database. In similarity image search, image features representing color, shape, etc. from an image are calculated as numerical vectors, and images with close distances between the vectors are searched for in the database. In recent years, search accuracy has improved by calculating image features using large datasets and deep learning. However, because the appearance of 3D data varies significantly depending on the viewpoint, it is difficult to search for similar 3D data using a single image as a clue.

[0004] For such 3D data searches, the object search device of Patent Document 1 includes an image feature extraction unit configured with a first neural network that inputs an image and extracts image features, a three-dimensional data feature extraction unit configured with a second neural network that inputs three-dimensional data and extracts three-dimensional data features, a learning unit that extracts image features and three-dimensional data features from images and three-dimensional data of objects obtained from the same individual, respectively, and updates parameters for image feature extraction so as to reduce the difference between the image features and the three-dimensional data features, and a search unit that uses the updated image feature extraction parameters to extract image features of a query image and a gallery image of the object, calculates the similarity of the image features of both images, and searches for the object. This provides an object search device that enables accurate searches when searching for objects with distinctive shapes or unevenness by expressing that information as features using only images.

[0005] On the other hand, with the advancement of deep learning, methods for automatically generating images from simple inputs such as text or simple sketches are rapidly developing. For example, methods for generating images using diffusion models and methods for generating 3D objects using text or images as input are known. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 7196058 Summary of the Invention [Problem to be solved by the invention]

[0007] The prior art disclosed in Patent Document 1 has two main problems. First, training data is required for feature learning of images and 3D data. In this case, new training data is required every time a new type of 3D object or image is added to the database. This is very labor-intensive, and it is not realistic to provide enough training data to cover all possibilities.

[0008] Second, the device in Patent Document 1 has the problem that users cannot check or control intermediate results. Users have no choice but to use the search results directly, making it difficult to examine the basis for which parts of the image and 3D data are similar to the search results, or to provide specific instructions or adjustments to provide feedback to the search query and perform a re-search. For these reasons, there is a need to realize a highly convenient search system that does not require learning and provides search feedback when searching for 3D objects. [Means for solving the problem]

[0009] In order to solve the above problem, one representative search system of the present invention is characterized by comprising: a 3D object generation unit that generates an input 3D object from an input image; an image generation unit that generates a plurality of images of the input 3D object viewed from multiple viewpoints as a set of search images; an image feature extraction unit that calculates image features from each image included in the set of search images; an image search unit that searches a database that registers a plurality of images of a plurality of 3D objects to be searched for viewed from multiple viewpoints, using each of the images included in the set of search images as a search query; and an image search result integration unit that integrates search results for each of the images included in the set of search images for each of the 3D objects to be searched for, and outputs 3D objects from the 3D objects to be searched that are similar to the input image as similar 3D objects. Furthermore, one representative search method of the present invention is characterized in that the search device includes a 3D object generation step of generating an input 3D object from an input image, an image generation step of generating a plurality of images of the input 3D object viewed from multiple viewpoints as a set of search images, an image feature calculation step of calculating image features from each image included in the set of search images, an image search step of searching a database in which a plurality of images viewed from multiple viewpoints of a plurality of 3D objects to be searched are registered, using each of the images included in the set of search images as a search query, and an image search result integration step of integrating search results for each of the images included in the set of search images for each of the 3D objects to be searched, and outputting 3D objects from the 3D objects to be searched that are similar to the input image as similar 3D objects. [Effects of the Invention]

[0010] According to the present invention, a highly convenient 3D object search can be realized. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the configuration of a 3D object generation and search system. [Figure 2] FIG. 1 is a block diagram showing the hardware configuration of a 3D object generation and retrieval system. [Figure 3] This figure shows the structure of the 3D object and image database. [Figure 4] FIG. 1 is a diagram illustrating generation and search of multi-viewpoint images. [Figure 5] 10 is a flowchart showing a database registration process. [Figure 6] 10 is a flowchart showing a database search process. [Figure 7] FIG. 10 is a diagram showing an example of a screen of a 3D object generation and search system. [Figure 8]FIG. 1 is a sequence diagram showing the overall processing of the 3D object generation and search system. [Figure 9] FIG. 10 is a diagram illustrating selection of a representative image by clustering. [Figure 10] 10 is a flowchart showing a representative image selection process by clustering. [Figure 11] FIG. 10 is a diagram illustrating representative image selection using feature amount distribution. [Figure 12] 10 is a flowchart showing a representative image selection process using a feature amount distribution. [Figure 13] FIG. 10 is a diagram illustrating weighting of search results according to camera positions. [Figure 14] 10 is a flowchart showing a process of weighting search results based on camera positions. [Figure 15] FIG. 10 is a diagram illustrating verification of the camera positional relationship of the search results. [Figure 16] 10 is a flowchart showing a process of verifying the camera positional relationship of the search results. [Figure 17] FIG. 1 is a diagram illustrating multimodal / multiobject image generation. [Figure 18] 10 is a flowchart showing a multimodal / multiobject image generation process. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. This embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are used to designate common components in each drawing. [Example]

[0013] In the registration phase, the 3D object generation and search device 104 of this embodiment generates images from multiple viewpoints from an input 3D object, extracts image features from each of the images from the multiple viewpoints, and registers the images in the 3D object and image database 112. In the search phase, it generates a 3D object from the input image, generates images from multiple viewpoints from the 3D object, searches the 3D object and image database 112 for similar images using the image features extracted from each of the images from the multiple viewpoints, and outputs a similar 3D object by integrating the search results. This makes it possible to accurately search for similar 3D objects from a single image.

[0014] FIG. 1 is a block diagram showing an example of the configuration of a 3D object generation and retrieval system 100 according to a first embodiment. Possible use cases for the 3D object generation and retrieval system 100 include, but are not limited to, searching for 3D data used in product design and building construction. The search targets stored in the database do not necessarily have to be 3D data, and may be images only. Each component will be described below.

[0015] The 3D object generation and retrieval system 100 generates images from multiple viewpoints from a 3D object input by a user and registers them in a 3D object and image database 112. It also generates a 3D object from an image input by a user, generates images from multiple viewpoints from the 3D object, searches the 3D object and image database 112 for images similar to each image, and integrates the search results to output a similar 3D object. The 3D object generation and retrieval system 100 is composed of a storage device 101, an input device 102, a display device 103, and a 3D object generation and retrieval device 104.

[0016] The storage device 101 is a storage medium for storing still image and video data or associated bibliographic information, and is configured using a hard disk drive built into a computer, or a storage system connected via a network such as a NAS (Network Attached Storage) or a SAN (Storage Area Network). The storage device 101 may also be a cache memory for temporarily storing data continuously input from a photographing device such as a camera.

[0017] The input device 102 is an input interface such as a mouse, keyboard, or touch device that transmits user operations to the 3D object generation and retrieval device 104. The display device 103 is an output interface such as a liquid crystal display, and is used to display the search results of the 3D object generation and retrieval device 104 and for interactive operations with the user.

[0018] The 3D object generation and search device 104 is a device that performs a registration process of inputting a 3D object to be searched into a database, and a search process of acquiring a 3D object similar to an image input by a user from the database.

[0019] The 3D object generation and search device 104 is composed of a 3D object input unit 105, an image input unit 106, a 3D object generation unit 107, an image generation unit 108, a generated image evaluation unit 109, an image feature extraction unit 110, a 3D object and image registration unit 111, a 3D object and image database 112, an image search unit 113, an image search result integration unit 114, and a display unit 115.

[0020] The database registration process will be explained below, and the details of the process will also be explained in the flowchart of FIG. The database registration process performs processing to make the 3D object read from the storage device 101 searchable. In this process, images are generated by rendering the input 3D object from multiple viewpoints in 3D space, and feature amounts for similar image search are extracted from each image and registered in the 3D object image database 112.

[0021] The 3D object input unit 105 accepts input of 3D object data from the storage device 101 and converts it into a data format used within the 3D object generation and search device 104. Examples of data formats include (1) a point cloud, which is a collection of points in a 3D space; (2) a mesh consisting of vertices, edges, and faces; (3) voxels, which divide a 3D space into small cubes with a certain volume; and (4) Neural Radiance Fields (NeRF), which expresses the color and density corresponding to points in a 3D space. However, any format can be used as long as it can render an image as seen from a specified viewpoint.

[0022] The image generation unit 108 generates one or more images by placing the 3D object acquired by the 3D object input unit 105 in a 3D space and rendering it with a camera positioned at one or more predetermined positions. If necessary, the material, light source, or background of the 3D object may be changed before rendering.

[0023] The image feature extraction unit 110 extracts one or more types of image feature for each of the images generated by the image generation unit 108. Image feature is numerical data that represents the color or shape of an image, and is usually given as fixed-length vector data. The similarity between the two original images can be determined by calculating the Euclidean distance between two vectors, cosine similarity, etc. Extracting multiple types of image feature also makes it possible to determine the similarity from different perspectives in response to user instructions.

[0024] The 3D object / image registration unit 111 links the 3D object acquired by the 3D object input unit 105 with the multi-viewpoint image generated by the image generation unit 108 and whose features have been calculated by the image feature extraction unit 110, and adds the data to a 3D object / image database 112.

[0025] The 3D object / image database 112 stores input 3D objects, generated multi-viewpoint images, and their feature amounts. In response to queries from each section of the 3D object generation / search device 104, the 3D object / image database 112 can search for registered data that meets given conditions or read data with a specified ID. In addition, by calculating the similarity between the feature amounts of a query image and the feature amounts of registered images, it is possible to perform a similar image search in which registered images are sorted and output in descending order of similarity. The structure of the 3D object / image database 112 will be described in detail later with reference to FIG. 3.

[0026] The above is the operation of each unit in the database registration process of the 3D object generation and retrieval device 104. Next, the operation of each unit in the search process of the 3D object generation and retrieval device 104 will be explained. The image generation unit 108 and image feature extraction unit 110 were explained in the registration process, but they are also used in the search process. Details of the process will also be explained in the flowchart of Figure 6.

[0027] In the database search process, search conditions specified by the user via the input device 102 are used to search for images that match the search conditions from the 3D object / image database 112, and the information is presented on the display device 103. By specifying an image, the user can obtain similar 3D objects.

[0028] The image input unit 106 receives input of still image data or video data from the storage device 101 and converts it into a data format used within the image generation and search device 104. For example, if the data received by the image input unit 106 is video data, the image input unit 106 performs video decoding processing to break it down into frames (still image data format). In addition, if the data is stroke information input using a mouse or touch device, the stroke information is drawn on an image.

[0029] The 3D object generation unit 107 automatically generates a 3D object from an image using a 3D object generation model. The 3D object generation model can use any algorithm or trained model. Alternatively, an algorithm capable of generating a 3D object from text may be employed, and text may be used instead of an image.

[0030] The image generation unit 108 generates one or more images by placing the 3D object acquired by the 3D object generation unit 107 in a 3D space and rendering it with cameras positioned at one or more predetermined positions. A method related to the method of generating a 3D object from an image is a method of generating images from multiple different viewpoints from a single image. When using this method, it is possible to obtain output equivalent to that of the 3D object generation unit 107 and the image generation unit 108. However, the user will no longer be able to check the 3D object on the screen or adjust the generated image by changing the camera position or rendering conditions.

[0031] The generated image evaluation unit 109 selects an image to be used for retrieval from the multi-viewpoint images generated by the image generation unit 108. A 3D object automatically generated from a single image can be viewed from various angles, and among these, an image with high generation quality and matching the purpose is selected. The evaluation may be performed directly by a user checking the generated image, or may be performed automatically using a heuristic method or an evaluation model based on machine learning. Here, selecting an image that matches the purpose means, in other words, removing images from the search. Images that clearly show the features of the object are selected, and images that do not show the features of the object are removed. Image selection may be performed at the time of registration or at the time of search.

[0032] The image feature extraction unit 110 calculates image feature quantities of a type designated by the user for all images generated by the image generation unit 108 and selected by the generated image evaluation unit 109. Types of image feature quantities that can be designated by the user include, for example, shape, color, size, and the like.

[0033] The image search unit 113 searches for images registered in the 3D object image database 112 based on the image feature amounts calculated by the image feature amount extraction unit 110. The search results are sorted in descending order of similarity and output, and include information such as the image ID, similarity, and 3D object ID. However, for the same 3D object, only the image with the highest similarity appears in the search results. Since images from multiple viewpoints are used, the search process is performed multiple times, resulting in multiple search results.

[0034] The image search result integration unit 114 integrates multiple search results obtained from the image search unit 113 and outputs them as the final search result. The integration method is, for example, to add up the similarity scores included in each search result for each 3D object. This makes it possible to obtain the similarity of the 3D object as the total value of the similarities of the multiple viewpoint images. Finally, the 3D objects are sorted in descending order of similarity and output.

[0035] The display unit 115 outputs data for displaying the processing results of each unit of the 3D object generation and search device 104 on the display device 103. For example, the display device 103 displays an image input by the user via the image input unit 106, a 3D object generated by the 3D object generation unit 107, a multi-viewpoint image generated by the image generation unit 108, search results for each image obtained by the image search unit 113, similar 3D objects obtained by the image search result integration unit 114, and the like.

[0036] The above is the operation of each unit in the search process of the 3D object generation and search device 104. The registration process and search process of the 3D object generation and search device 104 are performed repeatedly in response to user instructions, and the contents of the 3D object and image database 112 are added and updated successively, and the processing content of each unit that uses the registered data changes accordingly. In addition, if exclusive control of database updates is performed appropriately, multiple users can access and use the database simultaneously.

[0037] Although the above description has been given of the case where the input in the database registration process is a 3D object and the input in the database search process is an image, this system can be used with any combination. For example, an image can be accepted in the registration process, and the generated 3D object can be registered in the database using the 3D object generation unit 107. Another example is that a 3D object can be accepted as a query in the search process, and a search can be performed using rendered images from multiple viewpoints.

[0038] 2 is a block diagram showing an example of the hardware configuration of the 3D object generation and retrieval system 100 of this embodiment. The 3D object generation and retrieval device 104 includes a processor 201 and a storage device 202 connected to each other. The storage device 202 is configured with any type of storage medium. For example, the storage device 202 is configured by combining a semiconductor memory as a main storage device and a hard disk drive or the like as an auxiliary storage device.

[0039] 1 , functional units such as the 3D object input unit 105, image input unit 106, 3D object generation unit 107, image generation unit 108, generated image evaluation unit 109, image feature extraction unit 110, 3D object / image registration unit 111, image search unit 113, image search result integration unit 114, and display unit 115 are realized by the processor 201 executing a processing program 203 stored in the storage device 202. In other words, the processing performed by each functional unit is executed by the processor 201 based on the processing program 203. Furthermore, data in the 3D object / image database 112 is stored in the storage device 202. Note that when the 3D object generation / search system 100 is configured with multiple devices for purposes such as distribution of processing load, the device that includes the 3D object / image database 112 and the device that executes the processing program 203 may be physically different devices connected via a network, or the processing program 203 may be executed simultaneously on multiple devices as long as the consistency of the data recorded in the 3D object / image database 112 can be maintained.

[0040] The 3D object generation and retrieval device 104 further includes a network interface device (NIC) 204 connected to the processor 201. The storage device 101 is assumed to be a NAS or a SAN connected to the 3D object generation and retrieval device 104 via the network interface device 204. Note that the storage device 101 may be included in the storage device 202.

[0041] Figure 3 is an explanatory diagram showing the structure and example data of the 3D object / image database 112 of this embodiment. Note that in this embodiment, the information used by the system does not depend on the data structure and may be expressed in any data structure. While Figure 3 shows an example of a table format, the information can be stored in any data structure appropriately selected from, for example, a table, a list, a database, or a queue.

[0042] The 3D object / image database 112 includes, for example, a 3D object table 300 and an image table 310. The table configuration and field configuration of each table in Figure 3 are merely examples, and additional tables and fields may be added depending on the application. Furthermore, the table configuration may be changed as long as similar information is held.

[0043] The 3D object table 300 is composed of a 3D object ID field 301 and a 3D object data field 302 .

[0044] The 3D object ID field 301 holds an identification number of a 3D object. The 3D object data field 303 holds data of the 3D object. The 3D object is held in any text or binary format, such as a point cloud, mesh, voxel, or NeRF model. As long as the data of the 3D object can be accessed, it may also be a file path on file storage.

[0045] The image table 310 is composed of an image ID field 311 , an image data field 312 , a reference 3D object ID field 313 , an image feature field 314 , and a camera position field 315 .

[0046] The image ID field 311 holds the identification number of a 3D object. The image field 312 holds the binary data of the image. As long as the binary data of the image can be accessed, it may be a file path on file storage. The reference 3D object ID field 313 is a reference to the 3D object from which the image was rendered and holds the 3D object ID managed in the 3D object table 300. The image feature field 314 holds a numerical vector representing features extracted from the image. The camera position field 315 holds information about the position of the camera in 3D space when the image was rendered. For example, if a 3D object is placed at the origin in 3D space and a camera is pointed at the 3D object from a predetermined distance away, the position can be defined by three values: the camera's rotation angle relative to the X, Y, and Z axes. Alternatively, any numerical value may be held as long as it uniquely determines the camera's position in 3D space.

[0047] Fields may be added as needed to the 3D object table 300 and the image table 310. For example, the time the data was registered, tags for the data, etc. may be added to narrow down the search results. Also, multiple fields may be prepared for the feature field 314, and the results of calculations using multiple feature extraction methods may be stored, allowing the user to select which feature to use during a search.

[0048] FIG. 4 is a diagram illustrating an overview of 3D object generation from images and 3D object search using multi-viewpoint images in this embodiment. When a user provides an image as a search query, the input image 401 is converted into a 3D object by the 3D object generation unit 107. A known algorithm can be used to generate the 3D object. The generated 3D object 402 is expanded in 3D space by the image generation unit 108 and converted into multi-viewpoint images by rendering from multiple camera positions. The image feature extraction unit 110 calculates features for each multi-viewpoint image 403, which then becomes a query for image search. The image search unit 113 then obtains search results from a 3D object / image database. The ID below each search result image in the diagram is the ID of the 3D object associated with the image stored in the reference 3D object ID field 313. The number in parentheses indicates the similarity with the query image. The image search result integration unit 114 integrates image search results 404 into search results 405 for each 3D object. The search results 405 for 3D objects are output by sorting them in descending order of total similarity, which is the sum of the similarities of all images included in the image search results 404 for a certain 3D object. When calculating the total value, weighting may be applied to each query. For example, in FIG. 4, the similarity of the search results for query 1 is multiplied by 0.75, the similarity of the search results for query 2 by 0.50, and the similarity of the search results for query 3 by 1.00, and then these are added to the total similarity.

[0049] Figure 5 is a diagram showing the processing flow for database registration. Each step in Figure 5 will be explained below. The 3D object generation and search device 104 determines the type of input data, and if it is a 3D object, it reads it in using the 3D object input unit 105 and then executes step S504, or if it is an image, it reads it in using the image input unit 106 and then executes step S502 (S501).

[0050] The 3D object generation and retrieval device 104 executes step S503 if it is necessary to convert the registered data into 3D data, and executes step S504 if it is not (S502). The 3D object generation unit 107 generates a 3D object from the image input in step S501 (S503).

[0051] The 3D object / image registration unit 111 registers the 3D object input in step S501 or the 3D object generated in step S503 in the 3D object / image database 112. This causes an ID for the 3D object to be issued (S504). However, if it is determined in step S502 that only one image has been input and 3D conversion is not required, the single input image is registered in the database as a pseudo 3D object.

[0052] The image generation unit 108 places the 3D object input in step S501 or the 3D object generated in step S503 in a 3D space and generates multi-viewpoint images by rendering them from cameras at multiple positions (S505). However, if only one image is input in step S502 and 3D conversion is not required, only the single input image is output.

[0053] The 3D object generating and retrieving device 104 executes steps S507 to S508 for each image generated in step S505 (S506). The image feature extraction unit 110 calculates image feature values ​​from the image (S507).

[0054] The 3D object / image registration unit 111 registers the image data and the image feature amount extracted in step S507 in the 3D object / image database 112. At this time, the 3D object ID issued in step S504 is registered as a 3D object reference ID (S508). If the processing has been performed for all images, the 3D object generating and searching device 104 ends the database registration processing (S509).

[0055] Figure 6 shows the process flow for database search. Each step in Figure 6 will be explained below. The 3D object generation and search device 104 determines the type of input data, and if it is a 3D object, it reads it in using the 3D object input unit 105 and then executes step S604, and if it is an image, it reads it in using the image input unit 106 and then executes step S602 (S601).

[0056] The 3D object generation and retrieval device 104 executes step S603 if a search is to be performed using multiple viewpoint images, and executes step S605 if not (S602). Note that whether or not to perform a search using multiple viewpoint images can be specified by the user.

[0057] The 3D object generation unit 107 generates a 3D object from the image input in step S601 (S603). The image generation unit 108 places the 3D object input in step S601 or the 3D object generated in step S603 in a 3D space and generates multi-viewpoint images by rendering them from cameras at multiple positions (S604).

[0058] The 3D object generating and retrieving device 104 executes steps S606 to S607 for each image generated in step S604 (S605).

[0059] The image feature extraction unit 110 calculates image feature values ​​from the image (S606). The image search unit 113 searches the 3D object / image database 112 for similar images using the image feature amount calculated in step S607 as a query (S607).

[0060] If the search process has been executed for all images, the 3D object generating and searching device 104 executes step S609 (S608). The image search result integration unit 114 calculates the total similarity for each 3D object for each image search result obtained in step S607, sorts and outputs the total similarities in descending order, and ends the database search process (S609).

[0061] FIG. 7 is a diagram showing an example of the configuration of an operation screen for performing database search processing in the 3D object generation and search device 104 of Example 1. The 3D object generation and search device 104 displays the processing results on the display device 103. The user uses the input device 102 to convey operation information to the 3D object generation and search device 104, using a mouse cursor 701 or the like displayed on the screen. The screen is made up of a query load button 702, an input image display field 703, a 3D object generation button 704, a 3D object display field 705, a search button 706, an image search result display field 707, and a 3D object search result field 708. The example of the screen configuration is just an example, and the screen may be configured by freely arranging these elements.

[0062] When the user clicks the query load button 702 and selects data stored in the storage device 101, the query is loaded into the 3D object generation and search device 104. If the query is an image, it is displayed in the input image display field 703. If the query is a 3D object, it is displayed in the 3D object display field 705. When the user clicks the 3D object generation button while an image is displayed in the input image display field 703, a 3D object is generated from the input image and displayed in the 3D object display field 705. In the 3D object display field 705, the user can adjust the camera position to view the 3D object from any angle. The user can also specify the camera position used for image generation and adjust rendering conditions such as the light source. When the search button 706 is clicked, multiple viewpoint images are generated from the 3D object, and a similar image search is performed using each image as a query. The results are displayed in the image search result display field 707. The user can check the search results and adjust the weights used to integrate the search results. The integrated image search results for each 3D object are displayed in the 3D object search result display field 708.

[0063] FIG. 8 is a sequence diagram showing the process of database registration and database search in the 3D object generation and search device 104 of the first embodiment. FIG. 8 specifically shows the processing sequence between the user 800, the storage device 101, the computer 801, and the 3D object and image database 112 in each process of the 3D object generation and search system 100 described above. The sequence in FIG. 8 is roughly divided into the database registration process (S810) described in FIG. 5 and the database search process (S830) described in FIG. 6, and these sequences are repeatedly executed in response to a request from the user 800. Note that the computer 801 is a computer that realizes the 3D object generation and search device 104. Each step in FIG. 8 will be described below.

[0064] When the user 800 issues a data registration request (S811), a series of processes related to database registration is initiated in the computer 801. The computer 801 requests input data from the storage device 101 (S812), and the storage device 101 returns the input data (S813). If the input data is an image, the computer 801 generates a 3D object (S814) and generates a multi-viewpoint image from the input or generated 3D object (S815). The computer 801 extracts features from the image (S817), associates them with the 3D object, and requests registration in the 3D object / image database 112 (S818). The 3D object / image database 112 registers the data and issues a registration completion notice and an ID (S819). The computer 801 repeatedly executes the registration process for each image (S816), and when all images have been registered, notifies the user 800 of the registration completion (S820).

[0065] When the user 800 issues a data search request (S831), a series of processes related to a database search is initiated in the computer 801. The computer 801 requests query data from the storage device 101 (S832), and the storage device 101 returns the query data (S833). If the input data is an image, the computer 801 generates a 3D object (S834) and generates multi-viewpoint images from the input or generated 3D object (S835). The computer 801 extracts features from the image (S837) and performs a similar image search on the 3D object / image database 112 (S838). The 3D object / image database 112 returns image search results including the reference 3D object ID and the similarity (S839). The computer 801 repeatedly performs a search process for each image (S836), and calculates the similarity for each 3D object by integrating the similarity of the search results for each image (S840), and sorts the search results for the 3D objects in order of similarity and presents them to the user 800 (S841). [Example]

[0066] The 3D object generation and search device 104 of the first embodiment generates a 3D object from an image, generates images from multiple viewpoints, and searches for them, thereby obtaining a similar 3D object from the 3D object and image database 112. However, since it is necessary to store images from multiple viewpoints in the 3D object and image database 112, there are problems such as an increase in the size of the database and a long search time due to the need to perform multiple searches using images from multiple viewpoints. The 3D object generation and search device 104 of the second embodiment reduces the number of images used for registration or search by automatically selecting a representative image that is effective for search when generating images from multiple viewpoints from a 3D object. This reduces the size of the database and speeds up searches.

[0067] FIG. 9 is a diagram illustrating the selection of a representative image by clustering in the 3D object generation and retrieval device 104 of the second embodiment. Similar to the first embodiment, a multi-viewpoint image 902 is generated from an input or generated 3D object 901. Next, image features are extracted from the generated images, and clustering is performed in a feature space (903). The clustering process is a process of classifying images with similar features into groups. A clustering algorithm such as the k-means algorithm can be used. The k-means algorithm classifies data points in a dataset into k predefined clusters. First, the algorithm randomly selects k cluster centers (centroids) from the dataset. Next, each data point is assigned to a cluster based on the nearest centroid, and the position of each centroid is updated based on the average position of the assigned data points. This process of assignment and centroid update is repeated until the cluster assignment remains unchanged or a predetermined number of iterations is reached.

[0068] Based on the above clustering 903, a predetermined number of representative images are selected from each group of grouped multi-viewpoint images. For example, the selection method may be to select the image closest to the center of the cluster. As a result, a smaller number of representative images 904 than the original multi-viewpoint images 902 are obtained. Because the thinned images are similar to the representative image, it is expected that there will be no significant impact on search accuracy. Alternatively, the representative image may be selected based on the angle of the viewpoint. In this case, the closer the angle of the input image is to the angle of the input image, the more accurately the original features remain, making it more suitable as a representative image. Furthermore, if the algorithm used to generate the 3D object can output reliability, an image with a high reliability may be selected as the representative image.

[0069] 10 is a diagram showing a processing flow of representative image selection by clustering in the 3D object generating and searching device 104 of the embodiment 2. Each step in FIG. 10 will be described below. The image generation unit 108 places a 3D object in a 3D space and generates multi-viewpoint images by rendering the object from cameras at multiple positions (S1001).

[0070] The 3D object generating and retrieving device 104 executes step S1003 for each image generated in step S1001 (S1002). The image feature extraction unit 110 calculates image feature values ​​from the image (S1003). When the feature amounts of all images have been extracted, the 3D object generation and retrieval device 104 executes step S1005 (S1004). The generated image evaluation unit 109 performs clustering on the feature amounts extracted in step S1003 (S1005). Any method such as the k-means algorithm can be used for the clustering.

[0071] The 3D object generation and retrieval device 104 executes step S1007 for each cluster obtained in step S1005 (S1006). The generated image evaluation unit 109 selects a predetermined number of representative images from the cluster (S1007), for example, an image with a feature quantity closest to the central vector of the cluster. When the representative images have been selected from all the clusters, the 3D object generation and retrieval device 104 combines them and outputs them as a representative multi-viewpoint image of the 3D object, and ends the process (S1008).

[0072] The 3D object generating and searching device 104 of the second embodiment can also select a representative image using the feature distribution of a reference image database in addition to the clustering result. FIG. 11 is a diagram illustrating the selection of a representative image using a feature distribution by the 3D object generation and search device 104 of the second embodiment. As described in FIG. 9, clustering can be used to select a representative image from multiple viewpoint images (1101). Next, a feature distribution around the feature of the representative image is acquired from a reference image database in which multiple viewpoint images of numerous 3D objects are evenly registered (1102). If the representative image exists in a dense region in the feature space, it is likely to be an image with a common appearance shared by many 3D objects, making it difficult to obtain an image of the desired 3D object through similar image search. On the other hand, if the representative image exists in a sparse space, it is likely to be an image with a unique appearance, making it easier to obtain the desired 3D object through similar image search. Therefore, by narrowing down the representative images to those existing in a sparse region (1103), the number of representative images associated with a 3D object can be reduced, further reducing the database size during registration and the computational cost during search.

[0073] 12 is a diagram showing a processing flow of representative image selection using feature amount distribution in the 3D object generation and retrieval device 104 according to the second embodiment. Each step in FIG. 12 will be described below.

[0074] The image generation unit 108 places a 3D object in a 3D space and generates multi-viewpoint images by rendering the object from cameras at multiple positions (S1201). The 3D object generating and retrieving device 104 executes step S1206 for each image generated in step S1201 (S1202).

[0075] The image feature extraction unit 110 calculates image feature values ​​from the image (S1203). The image search unit 113 uses the feature extracted in step S1203 as a query to retrieve images with a predetermined similarity or higher from the 3D object / image database 112 (S1204). At this time, since the feature distribution changes every time new data is registered, a reference image table may be held in the 3D object / image database 112 in addition to the registration image table 310, and this table may be used as the search target.

[0076] If the number of similar images obtained in step S1204 is equal to or greater than a predetermined number, the generated image evaluation unit 109 determines that the image is a common image existing in a dense space and discards it, otherwise it executes step S1206 (S1205). The generated image evaluation unit 109 determines that the image to be evaluated exists in a sparse space and therefore is an image with unique characteristics, and adds it to the representative image (S1206). The 3D object generating and searching device 104 ends the process if evaluation based on the feature amount distribution has been performed on all images from multiple viewpoints (S1207). The 3D object generating and searching device 104 of the second embodiment may select a representative image using either clustering or feature distribution, or may select both. [Example]

[0077] The 3D object generation and retrieval device 104 of the first embodiment performs a simple addition of similarities of the same 3D objects or a weighted addition using a weight specified by a user when integrating search results of multiple viewpoint images. The 3D object generation and retrieval device 104 of the third embodiment automatically sets the weight of the search results using the positional relationship of the cameras and integrates the search results. Furthermore, the 3D object generation and retrieval device 104 of the third embodiment can verify the integration process of the search results by using camera position information for the 3D objects attached to the search result images. This can improve the accuracy of the search results of 3D objects.

[0078] FIG. 13 is a diagram illustrating weighting of search results based on camera position. The 3D object generation and search device 104 can obtain a 3D object from an input image using a 3D object generation model. For a field of view equivalent to that of the input image, appropriate feedback is applied during the generation process, resulting in high accuracy for the corresponding portion of the generated 3D model. On the other hand, for portions not included in the input image, the accuracy of 3D generation for those portions is low because random numbers and guesswork are more likely to be used. Therefore, when the generated 3D object is placed in 3D space, the camera position of the input image is estimated, and the amount of movement or rotation of the multi-viewpoint images from the camera position of the input image is calculated. Images with large amounts of movement or rotation are considered to have low reliability, and a relatively low weight is set when integrating search results.

[0079] 14 is a diagram showing a processing flow for weighting search results according to camera positions in the 3D object generation and search device 104 of Example 3. Each step in FIG. 14 will be described below. The 3D object generation unit 107 generates a 3D object from an input image (S1401).

[0080] The 3D object generation unit 107 estimates the camera position of the input image by examining from which position rendering will produce an image similar to the input image when the 3D object is placed in the 3D space (S1402). The camera position of the input image may be determined in the 3D object generation algorithm, so this information may be output from the algorithm.

[0081] The image generation unit 108 generates images of multiple viewpoints from the 3D object generated in step S1401 (S1403). The 3D object generating and retrieving device 104 executes steps S1405 to S1407 for each image generated in step S1403 (S1404). The image feature extraction unit 110 calculates image feature values ​​from the image (S1405). The image search unit 113 searches for similar images from the 3D object / image database using the image feature amount calculated in step S1405 (S1406).

[0082] The image search result integration unit 114 calculates the amount of camera movement from the camera position of the search query image and the camera position of the input image acquired in step S1402, and determines the weight of the search results according to the amount of movement (S1407).

[0083] When the search process has been executed for all images, the 3D object generating and searching device 104 executes step S1409 (S1408). The image search result integration unit 114 calculates the similarity for each 3D object using the image search results obtained in step S1406 and the weights for the search results determined in step S1407, sorts the total similarities in descending order, outputs the results, and terminates the processing (S1409).

[0084] 15 is a diagram for explaining verification of the camera position relationship between a query and search results. In this example, images at camera positions 1, 2, and 3 are generated from the query 3D object (1501). Similar images are obtained from the 3D object / image database 112 using each image as a query. At this time, the value of the camera position field 315 is obtained from the image table 310 in addition to the value of the ID field 313 of the reference 3D object (1502).

[0085] The image search result integration unit 114 compiles the image search results for each 3D object, verifying whether the positional relationship in 3D space of the query image set matches the positional relationship in 3D space of the search result image set. For example, for 3D object 1 (1503), images taken at camera positions 1, 3, and 4 appear in the search results. Images at positions different from the query happen to be similar, and it is highly likely that the 3D structures are different. On the other hand, for 3D object 2 (1504), images at the same positional relationship as the query appear in the search results, and it is highly likely that the 3D structures are closer. In this way, adjusting the total similarity using the identity of the image positional relationship improves search accuracy.

[0086] 16 is a diagram showing the processing flow for verifying the camera positional relationship between a query and search results. Each step in FIG. 16 will be described below. The 3D object generation and search device 104 searches through multiple viewpoint images, integrates the image search results, and outputs the search results for the 3D object (S1601). This corresponds to the database search process in FIG.

[0087] The image search result integration unit 114 acquires the positional relationship in the 3D space of the multi-viewpoint images of the query (S1602). This can be easily acquired from the camera parameters used by the image generation unit 108 to render the 3D object. The 3D object generation and retrieval device 104 executes steps S1604 to S1605 for each of the 3D object retrieval results acquired in step S1601 (S1603).

[0088] The image search result integration unit 114 acquires camera position information from the image search results related to the target 3D object, and calculates the degree of coincidence between the position of the query image and the position of the search result image (S1604). The search result integration unit 114 updates the value of the total similarity calculated in step S1601 with the degree of match in the positional relationship calculated in step S1604 (S1605). For example, if the degree of match is 50%, the value obtained by multiplying the total similarity by 0.5 is set as the updated total similarity. If the positional relationship has been verified and the total similarity has been updated for all 3D objects, the 3D object generating and searching device 104 ends the process (S1606). [Example]

[0089] The 3D object generation and retrieval device 104 of the first embodiment generates an image for search by rendering a 3D object of a search query from multiple viewpoints in a 3D space. By handling the search query in a 3D space, the image for search can be processed in various ways other than changing the viewpoint. The 3D object generation and retrieval device 104 of the fourth embodiment renders an object placed in a 3D space using multiple methods, and generates a high-quality image from an image generation model that uses a multimodal image as input. Furthermore, by placing multiple 3D objects in a 3D space, it becomes possible to search for a composite object. As a result, search accuracy and search flexibility are improved.

[0090] FIG. 17 is a diagram illustrating query image generation using a multimodal / multi-object algorithm by the 3D object generation and retrieval device 104 of the fourth embodiment. The process in the upper part 1700 of the diagram is query image generation using a multimodal image. After a search query 3D object 1701 is placed in 3D space, a general 3D rendering engine can output not only a normal image but also a multimodal image (1702). For example, it is possible to obtain a contour image that leaves only the contour lines of the object, a mask image that fills the object area with a single color, a distance image that represents the position from the camera in grayscale, and so on. A method for generating a new, high-quality image from an existing image using an image generation model is already known. Also, a method is known in which the texture and details of an object can be changed while maintaining its general shape by providing a contour image, a mask image, and a distance image as control information. Using such a method, a high-quality generated image can be obtained (1703). In other words, search accuracy can be improved by generating a multimodal image from a 3D object generated using a predetermined algorithm, generating an image from the multimodal image, and using the image generated from the multimodal image for search.

[0091] The process 1710 at the bottom of Figure 17 is the generation of a query image using multiple objects. A typical 3D rendering engine can place multiple 3D objects in 3D space at any position and rotate them as desired. In this example, 3D objects 1701, 1711, and 1712 are used as input to create a 3D space in which multiple objects are placed (1713). In a multi-object search where multiple objects are arranged, the search accuracy is improved because the relationship with other objects can also be taken into consideration when searching.

[0092] Note that either the arrangement of multi-objects or the image generation process using multi-modal images may be applied, or the image generation process using multi-modal images may be performed on a 3D space in which multiple objects are arranged.

[0093] 18 is a diagram showing a processing flow of multimodal / multi-object query image generation by the 3D object generating and retrieving device 104 of Example 4. Each step in FIG. 18 will be described below. The image generation unit 108 places one or more objects in a 3D space (S1801). The user can specify the position and rotation of each object in the 3D space using the input device 102. The 3D object generating and searching device 104 executes steps S1803 to S1808 for each camera position for generating multiple images (S1803).

[0094] The image generation unit 108 executes step S1805 if processing using a multimodal image is to be performed, and executes step S1804 if not (S1804). The image generation unit 108 performs rendering using the normal image (S1804). The image generation unit 108 renders a multimodal image required for image conversion (S1805). For example, it can render a contour image, a mask image, a distance image, and the like.

[0095] The image generation unit 108 acquires parameters to be given to the image generation model (S1806). For example, in a method of generating a new image by giving a multimodal image and a text prompt, the text is acquired as a parameter.

[0096] The image generation unit 108 inputs the multimodal image acquired in step S1805 and the parameters acquired in step S1806 into an image generation model, thereby converting the image into a high-quality image (S1807). If image generation has been performed for all camera positions, the 3D object generating and searching device 104 ends the process (S1808).

[0097] As described above, the search system disclosed in the embodiments includes a 3D object generation unit 107 that generates an input 3D object from an input image, an image generation unit 108 that generates a plurality of images of the input 3D object viewed from multiple viewpoints as a set of search images, an image feature extraction unit 110 that calculates image features from each image included in the set of search images, an image search unit 113 that searches a database in which a plurality of images viewed from multiple viewpoints of a plurality of 3D objects to be searched are registered, using each of the images included in the set of search images as a search query, and an image search result integration unit 114 that integrates search results for each of the images included in the set of search images for each of the 3D objects to be searched, and outputs 3D objects from the 3D objects to be searched that are similar to the input image as similar 3D objects. Therefore, a 3D object can be generated from an input image provided by the user, multiple viewpoint images can be generated from the generated 3D object, a database can be searched using the multiple images as queries, and multiple search results can be integrated to output a 3D object that is similar in appearance to the input image. By using multiple viewpoint images, it is possible to obtain more accurate search results than when searching from only a single viewpoint. Also, there is no need to learn features tailored to the object. Furthermore, users can provide feedback to the search by viewing and editing the generated 3D objects in 3D space, viewing and processing the images used in the search, and controlling the weighting of the search results. This makes it possible to realize a highly convenient search system that does not require learning and allows search feedback when searching for 3D objects.

[0098] Specifically, images registered in the database are associated with identification information of the 3D object to be searched, and the image search unit 113 calculates the similarity to the search query for multiple images that are search results for the search query, and the image search result integration unit 114 adds up the similarities of images associated with identification information of the same 3D object from the search results of each image included in the group of search images, and determines this as the similarity of the 3D object. This process allows the search system to easily determine the similarity of 3D objects.

[0099] In addition, when summing up the similarities of images associated with the identification information of the same 3D object, the image search result integration unit 114 sums up the similarities with a predetermined weight, and the weight is determined according to the difference between the viewpoint of each image included in the search image group and the viewpoint of the input image. Therefore, it is possible to search for a 3D object generated from an input image, taking into account the viewpoint from which the accuracy is highest.

[0100] The disclosed search system further includes a display unit 115 that displays the input image and images included in the group of search images, and an input unit (102) that accepts a weight specification for the images included in the group of search images, and the image search result integration unit 114 adds up the similarities of images associated with the identification information of the same 3D object by applying the specified weight. This allows the user to check the 3D object generated from the input image and perform a search by considering from which viewpoint the features become most apparent.

[0101] The disclosed search system further includes a registration unit (111) that registers images in the database, wherein the 3D object generation unit 107 generates a 3D object as the search target when it receives an image as the search target, and the image generation unit 108 uses the 3D object received as the search target or a 3D object generated from the image received as the search target to generate a plurality of images of the 3D object viewed from a plurality of viewpoints as a group of images for registration, and the registration unit associates the group of images generated by the image generation unit 108 with identification information of the 3D object as the search target and registers them in the database. This allows 3D objects to be generated from images and images from multiple viewpoints to be registered in a database.

[0102] The database also associates 3D object identification information with 3D object data indicating the 3D object to be searched, and associates 3D object identification information, image feature amounts, and viewpoint positions with image data indicating the image to be searched. This allows for the relationship between data relating to 3D objects and image data generated from the 3D objects to be managed collectively.

[0103] As another example, the image search unit 113 searches for representative images among the images registered in the database, and the representative images are selected for each cluster obtained by dividing the feature space. This configuration avoids searching for images with overlapping features, enabling efficient searches.

[0104] As another example, the image search unit 113 searches for a representative image among the images registered in the database, and the representative image is an image that exists in an area in the feature space where the density is less than a threshold value. This configuration enables efficient searches by searching for images in which the characteristics of 3D objects are apparent.

[0105] In addition, the image search result integration unit 114 modifies the similarity of the 3D object to be searched based on the degree of match between the combination of viewpoints in the group of search images and the combination of viewpoints in images associated with identification information of the same 3D object. In this configuration, by taking into consideration the combination of viewpoints, highly accurate searches are possible.

[0106] Furthermore, the image generation unit 108 generates a multimodal image from the input 3D object, and outputs the image generated from the multimodal image as an image included in a group of images for search. In this configuration, multiple features are extracted from a multimodal image, an image is constructed from the extracted features, and the constructed image is used as a search query, thereby improving search accuracy.

[0107] Furthermore, the 3D object generation unit 107 combines a plurality of 3D objects, and the image generation unit 108 generates a plurality of images of the combination of the plurality of 3D objects viewed from a plurality of viewpoints as a search image group. This configuration allows for highly accurate searches that take into account relationships with other objects.

[0108] The embodiments of the present invention have been described above. The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. Furthermore, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0109] 100: Search system, 101: Storage device, 102: Input device, 103: Display device, 104: Search device, 105: 3D object input unit, 106: Image input unit, 107: 3D object generation unit, 108: Image generation unit, 109: Generated image evaluation unit, 110: Image feature extraction unit, 111: Image registration unit, 112: Image database, 113: Image search unit, 114: Image search result integration unit, 115: Display unit, 201: Processor, 202: Storage device, 203: Processing program, 204: Network interface device, 300: 3D object table

Claims

1. a 3D object generator that generates an input 3D object from an input image; an image generation unit that generates a plurality of images of the input 3D object viewed from a plurality of viewpoints as a search image group; an image feature extraction unit that calculates image feature values ​​from each image included in the search image group; an image search unit that searches a database in which multiple images of multiple 3D objects to be searched are registered, the multiple images being viewed from multiple viewpoints, using each of the images included in the search image group as a search query; an image search result integration unit that integrates search results for each of the images included in the search image group for each 3D object to be searched, and outputs 3D objects similar to the input image from among the 3D objects to be searched as similar 3D objects; A search system comprising:

2. The search system according to claim 1, The images registered in the database are associated with identification information of the 3D object to be searched, the image search unit calculates a similarity to the search query for a plurality of images that are search results for the search query; The image search result integration unit sums up the similarities of images associated with the identification information of the same 3D object from each search result of the images included in the search image group, and determines the similarity as the similarity of the 3D object.

3. 3. The search system according to claim 2, the image search result integration unit adds up the similarities of the images associated with the identification information of the same 3D object with a predetermined weight, A search system characterized in that the weight is determined according to a difference between a viewpoint of each image included in the search image group and a viewpoint of the input image.

4. 3. The search system according to claim 2, a display unit that displays the input image and images included in a search image group; an input unit that receives a weight specification for an image included in the search image group; The image search result integration unit adds up the similarities of images associated with the identification information of the same 3D object by assigning a specified weight to the sum.

5. The search system according to claim 1, a registration unit that registers an image in the database, the 3D object generation unit generates a 3D object as a search target when an image is received as a search target; the image generation unit uses a 3D object accepted as a search target or a 3D object generated from an image accepted as a search target to generate a plurality of images of the 3D object viewed from a plurality of viewpoints as an image group for registration; The registration unit associates the group of images generated by the image generation unit with identification information of the 3D object to be searched and registers the images in the database.

6. The search system according to claim 1, The database associates 3D object identification information with 3D object data indicating the 3D object to be searched, and associates 3D object identification information, image features, and viewpoint positions with image data indicating the image to be searched.

7. The search system according to claim 1, the image search unit searches for a representative image among the images registered in the database; A search system characterized in that the representative images are selected for each cluster obtained by dividing a feature space.

8. The search system according to claim 1, the image search unit searches for a representative image among the images registered in the database; A search system characterized in that the representative image is an image that exists in an area in feature space where the density is less than a threshold value.

9. 3. The search system according to claim 2, The image search result integration unit modifies the similarity of the 3D object to be searched based on the degree of match between the combination of viewpoints in the group of search images and the combination of viewpoints in images associated with identification information of the same 3D object.

10. The search system according to claim 1, The search system is characterized in that the image generation unit generates a multimodal image from the input 3D object and outputs the image generated from the multimodal image as an image to be included in a group of images for search.

11. The search system according to claim 1, The 3D object generation unit combines a plurality of 3D objects, The search system is characterized in that the image generation unit generates a plurality of images of the combination of the plurality of 3D objects viewed from a plurality of viewpoints as a group of images for search.

12. The search device a 3D object generation step of generating an input 3D object from the input image; an image generation step of generating a plurality of images of the input 3D object viewed from a plurality of viewpoints as a search image group; an image feature amount calculation step of calculating an image feature amount from each image included in the search image group; an image search step of searching a database in which a plurality of images of a plurality of 3D objects to be searched are registered, the images being viewed from a plurality of viewpoints, using each of the images included in the search image group as a search query; an image search result integration step of integrating search results for each of the images included in the search image group for each 3D object to be searched, and outputting 3D objects similar to the input image among the 3D objects to be searched as similar 3D objects; A search method comprising:

Images