3D retrieval device and 3D retrieval method

Abstract

To provide a technique capable of improving retrieval accuracy of 3D information desired by a user.SOLUTION: A 3D retrieval device 1 that retrieves 3D includes an arithmetic unit 3 and a storage unit 4, and the arithmetic unit 3 extracts a 3D feature vector and a metadata feature vector from 3D and metadata added to the 3D, respectively, generates a fusion vector by fusing the 3D feature vector and the metadata feature vector, stores the fusion vector in the storage unit 4 as a dataset, and extracts a fusion vector having a high degree of association with a fusion vector of 3D serving as a retrieval query from the dataset.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a 3D search device and a 3D search method. [Background technology]

[0002] In recent years, the use of 3D data, such as design data from 3D CAD (Computer Aided Design) software and 3D point cloud data from LiDAR (Light Detection And Ranging) sensors, is expected to grow not only in the traditional fields of design and maintenance but also in various fields such as the metaverse and medicine. Demand for 3D data is expected to continue to increase in the future.

[0003] To utilize 3D data, technology is needed to search for necessary 3D data from the vast amount of data being accumulated and extract the necessary information. For example, in the field of design, there is a need for technology that can search for 3D CAD data with shapes similar to a design to support non-experts and prevent mistakes and rework in downstream development stages. Also, in the field of maintenance, there is a need for technology that can search for 3D data of the causative parts in order to respond to system malfunctions.

[0004] Patent Document 1 discloses a technology capable of searching for 3D data. The technology in Patent Document 1 takes a 3D model as input and rotates and / or translates it so that the principal axis direction of the 3D model coincides with the coordinate axes of a system coordinate system. The 3D model is projected in both the positive and negative directions of each coordinate axis to generate multiple 2D images, which are then compared with the 2D images stored in a database to search for 3D models with similar shapes.

[0005] Furthermore, in Non-Patent Document 1, 2D images from multiple viewpoints are first generated using rendering, which generates 2D images from a 3D model. The 2D images capture the geometric features of the 3D model from a certain viewpoint. Using the 2D images as input, a machine learning model is trained so that models with similar geometric features output similar feature vectors. By matching 3D models with similar feature vectors during inference, other 3D models with similar shapes can be extracted from a database. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-80796 [Non-patent literature]

[0007] [Non-Patent Document 1] H, Su., S, Maji., E, Kalogerakis., E, Learned-Miller., “Multi-view Convolutional Neural Networks for 3D Shape Recognition”, International Conference on Computer Vision, 2015. Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the above Patent Document 1 and Non-Patent Document 1, 3D data is searched using only the similarity of the 3D shape as a clue.

[0009] In the fields of design and maintenance mentioned above, it is desirable to search for 3D data that is similar not only in the 3D shape of parts, but also in properties such as function, material, and dimensions. However, conventional search techniques that rely only on the similarity of 3D shapes cannot meet this demand, and users are therefore unable to search for the 3D data they desire.

[0010] Therefore, an object of the present invention is to provide a technique that can improve the accuracy of searching for 3D data desired by a user. [Means for solving the problem]

[0011] In order to solve the above problem, one representative 3D search device of the present invention is a 3D search device for searching 3D data, which is equipped with a calculation unit and a memory unit, and the calculation unit extracts a 3D feature vector and a metadata feature vector from the 3D data and metadata attached to the 3D data, respectively, fuses the 3D feature vector and the metadata feature vector to generate a fusion vector, stores the fusion vector as a dataset in the memory unit, and extracts from the dataset a fusion vector that is highly relevant to the fusion vector of the 3D data that is the search query. [Effects of the Invention]

[0012] According to the present invention, it is possible to improve the accuracy of searching for 3D data desired by a user.

[0013] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of the configuration of a 3D search device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a process executed by a calculation unit. [Figure 3] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer to which a 3D search device is applied. [Figure 4] 10 is a flowchart illustrating an example of a data set creation process. [Figure 5] 10 is a flowchart illustrating an example of a 3D search process. [Figure 6] 10 is a flowchart illustrating an example of a text feature selection process performed by a text feature selection unit. [Figure 7] 10 is a flowchart illustrating an example of feature amount fusion processing by a feature amount fusion unit. [Figure 8] 10 is a flowchart illustrating an example of a search process performed by a search unit. [Figure 9] FIG. 10 is a diagram illustrating a search process performed by a search unit. [Figure 10] 10A and 10B are diagrams showing examples of display screens displayed by a 3D display unit and a similarity display unit. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment will be described with reference to the drawings. [Example]

[0016] The 3D search device of this embodiment provides a method for searching for and displaying 3D data that is similar in shape to the 3D data and has characteristics related to the instructions in the metadata and prompt, using 3D data, metadata attached to the 3D data, and a prompt indicating a user's instructions as input.

[0017] It should be noted that this embodiment is merely an example for realizing the present embodiment and does not limit the technical scope of the present embodiment. The same reference numerals are used to designate common components in each drawing.

[0018] First, the configuration of the 3D search device of this embodiment will be described with reference to FIG.

[0019] FIG. 1 is a block diagram showing an example of the configuration of a 3D search device according to this embodiment.

[0020] The 3D search device 1 includes an input unit 2, a calculation unit 3, a storage unit 4, and a display unit 5.

[0021] The input unit 2 includes a 3D input unit 101, a prompt receiving unit 102, and a metadata input unit 103, and mainly has the function of reading input data.

[0022] The 3D input unit 101 converts 3D data such as 3D CAD, 3D mesh, and 3D point cloud data into a format that is easy to handle in the calculation unit 3, and stores the converted data in the memory 202 (FIG. 3) of the 3D search device 1.

[0023] The prompt receiving unit 102 reads the text data of the prompt entered by the user and temporarily stores it in the memory 202. The user can specify the properties of the part they want to search for by writing them in the text data of the prompt. For example, by specifying the material or dimensions of the part, it is possible to extract and search for information related to the prompt from metadata, which will be described later.

[0024] The metadata input unit 103 reads the text data of the metadata attached to the 3D data and stores it in the memory 202. The metadata is information about the properties of the parts of the 3D data. The metadata stores various attributes such as material, dimensions, weight, part name, function, label, etc. The attributes that are stored vary depending on the 3D data.

[0025] The calculation unit 3 includes a token generation unit 104 , a two-dimensional image acquisition unit 105 , a text feature extraction unit 106 , an image feature extraction unit 107 , a text feature selection unit 108 , a search unit 109 , and a feature fusion unit 110 .

[0026] Here, the processing executed by the calculation unit 3 will be described with reference to FIG.

[0027] FIG. 2 is a diagram illustrating the processing executed by the calculation unit 3. As shown in FIG.

[0028] As shown in FIG. 2, the calculation unit 3 processes the 3D data 40, metadata 41, and prompt 42 acquired from the input unit 2 as a 3D rendering image and text data.

[0029] First, using a base model 43, a 3D feature vector 44, a metadata feature vector 45, and a prompt feature vector 46 are extracted from 3D data 40, metadata 41, and a prompt 42, respectively.

[0030] The base model 43 may be, for example, CLIP (Contrastive Language-Image Pre-Training) that can express the relationship between language and images, or a model that is trained with large-scale training data on the scale of hundreds of millions and can recognize various general objects. If a base model that can directly express the relationship between language and 3D can be constructed, it is desirable to utilize such a base model because it allows feature vectors to be extracted directly from 3D data, thereby reducing the number of processing steps.

[0031] However, it is not easy to prepare a large-scale dataset showing the relationship between 3D and language that can be used to train a foundational model, and the increased computational cost of handling 3D data is also a major challenge.

[0032] For this reason, in this example, the 3D data is first converted into a 2D image using 3D rendering. By applying a pre-trained foundation model that can represent the relationship between language and images to the converted 2D image, it is possible to utilize a model trained on an existing ultra-large training dataset.

[0033] The text feature selection unit 108 is used to extract the metadata feature vector 45 that is highly related to the prompt feature vector 46 and update the metadata feature vector 45 .

[0034] The updated metadata feature vector 48 is fused with the 3D feature vector in the feature fusion unit 110, and finally a fusion vector 50 is generated that combines the 3D feature and the metadata feature.

[0035] Returning to FIG. 1, each functional unit of the calculation unit 3 will be described.

[0036] The 2D image acquisition unit 105 outputs multiple 2D images by applying a rendering process that generates 2D data from 3D data. 3D rendering generates 2D data by viewing 3D data from a specific viewpoint and distance. To capture 3D geometric features, it is important to generate 2D images that allow the overall shape to be recognized.

[0037] For example, 2D images may be generated from the front, side, and top viewpoints, such as a three-dimensional view, or from random viewpoints in a 360-degree range centered on the target 3D object. The user can also define the angle of view, light source position, and number of 2D images rendered from 3D data. The output image data is converted into an easy-to-handle format by the image feature extraction unit 107.

[0038] The image feature extraction unit 107 extracts a 3D feature vector 44 from the rendered image. The feature extraction can use a base model 43 or the like. By using a trained image encoder of the base model 43, it is possible to convert the 3D feature vector into a one-dimensional feature vector.

[0039] The image encoder may use a network architecture such as Residual Neural Networks (ResNet), which consists of multiple layers with residual connections to blocks that extract features by layering convolutional layers, batch regularization, and activation functions on the input image, or Vision Transformer, which does not use convolutional layers but extracts features by inputting an image subset and performs global recognition of the entire image using a self-attention mechanism.

[0040] These trained models are used to recognize objects contained in the 3D rendering image, and features related to the shape of the 3D object are converted into a 3D feature vector 44. Since the 3D feature vector is extracted from the 3D rendering image, it captures 3D geometric features. The 3D feature vector 44 extracted by the image feature extraction unit 107 becomes input data for the feature fusion unit 110.

[0041] The token generator 104 generates tokens from text data (prompts and metadata) to serve as input data for the language-based model. The role of the token generator 104 is to divide sentences and characters (text data) obtained from metadata and prompts entered by the user into vocabulary (tokens) and convert them into a format that can be input to a model used by the text feature extractor 106, which will be described later.

[0042] The token generation unit 104 uses a tokenizer associated with a trained language encoder, which will be described later, to divide the sentence given as input into words, and generates tokens by assigning IDs to each word.

[0043] The text feature extraction unit 106 receives the tokens generated by the token generation unit 104 as input and extracts a metadata feature vector 45 and a prompt feature vector 46 as text features relating to the metadata and prompt.

[0044] To extract features, a language-based model trained on a large dataset or a pre-trained language encoder from an existing foundation model such as CLIP can be used. The language encoder can be something like BERT (Bidirectional Encoder Representations from Transformers). BERT is based on the Transformer, and takes tokens representing words in a sentence as input, and processes the features by using a self-attention mechanism to understand how each word is related to all other words. In this way, the language encoder can convert the meaning of the entire context into a one-dimensional feature vector.

[0045] The extracted metadata feature vector 45 and prompt feature vector 46 are input to the text feature selection unit 108 .

[0046] The text feature selection unit 108 automatically weights metadata information that the user wants to emphasize in a search based on a prompt specified by the user. The user can specify multiple prompts based on the characteristics of parts that should be considered during a search, such as "Material: ____, Function: ____, Dimensions: ____."

[0047] When a user searches, they have knowledge about the part they are searching for, so utilizing this knowledge helps improve search accuracy.

[0048] If the user does not input a prompt, the text feature selection unit 108 does not weight the information, and the information included in the metadata is treated uniformly. In this case, features related to all information included in the metadata 41 are used by the feature fusion unit 110, which will be described later.

[0049] The feature amount fusion unit 110 fuses the 3D feature vector 44 and the metadata feature vector 48 to generate a fusion vector 50. The fusion vector 50 is input to the search unit 109 and the feature amount storage unit 111 in the storage unit 4.

[0050] The search unit 109 extracts feature vectors that are highly relevant to the search query from among the feature vectors in the database 112 of the storage unit 4. The search query uses any one of the fusion vector 50, the 3D feature vector 44, and the metadata feature vector 48.

[0051] The storage unit 4 includes a feature amount storage unit 111 and a database 112 .

[0052] The feature storage unit 111 stores the fusion vector 50 generated by the feature fusion unit 110, the 3D feature vector 44 extracted by the image feature extraction unit 107, and the metadata feature vector 45 extracted by the text feature extraction unit 106 in a database 112 together with the input 3D data 40. In addition, the 3D data 40 stored in the database 112 is assigned an ID that identifies each piece of 3D data 40.

[0053] These feature vectors are extracted by preparing a dataset of 3D data 40 and metadata 41 before searching and applying feature extraction processing by the calculation unit 3, and are stored in the database 112. Each feature vector in the database 112 is used for matching during a search by the search unit 109. The dataset creation process for saving each feature vector in the database 112 will be described later with reference to FIG. 4.

[0054] The display unit 5 includes a similarity display unit 113 and a 3D display unit 114, and has the function of displaying search results.

[0055] The 3D display unit 114 displays, on the display screen of the output unit 206 as search results, 3D data linked to feature vectors that are highly relevant to the search query and that have been extracted by the search in the search unit 109.

[0056] The similarity display unit 113 displays, as a numerical value, the similarity of the feature vector between the search query and the 3D data displayed by the 3D display unit 114. The numerical value is calculated using a cosine similarity that measures the angle between vectors, and can indicate the degree of association between the search query and the 3D data in the database in the range of 0 to 1.

[0057] The user can check the search results on the display screen displayed by the display unit 5, and if the desired results are not output, can modify the prompt 42 to change the search results. Details of the display screen will be described later with reference to FIG.

[0058] FIG. 3 is a block diagram showing an example of the hardware configuration of a computer to which the 3D search device 1 is applied.

[0059] The calculator 200 is a server or computer configured by interconnecting a CPU (Central Processing Unit) 201, a memory 202, a storage device 203, a communication unit 204, an input unit 205, and an output unit 206 via a bus 207.

[0060] The CPU 201 is a central processing unit that executes a program stored in the memory 202 (or the storage device 203) to implement necessary functions.

[0061] Memory 202 is a main storage device used by CPU 201 when executing processing, and is composed of volatile memory elements such as RAM (Random Access Memory) or ROM (Read Only Memory), and is used to permanently store programs required for processing and temporarily store data required for processing.

[0062] The storage device 203 is an auxiliary storage device for storing input data provided to the CPU 201 and output data output from the CPU 201, and is configured with non-volatile storage elements such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The database 112 is stored in the storage device 203. Data is input to and output from the storage device 203 via the memory 202 under the control of the CPU 201.

[0063] The communication unit 204 is an interface used by the computer 200 to communicate with external devices, and is configured by a NIC (Network Interface Card) etc. The communication unit 204 is connected to a network (e.g., the Internet) and communicates with external devices via the network.

[0064] The input unit 205 is an interface that accepts input from a user (operator), and is configured with a keyboard, a touch panel, a card reader, a voice input device, or the like.

[0065] The output unit 206 is an interface that outputs data to the operator, and is composed of a display, a speaker, a printer, etc. The display screen displayed by the display unit 5 is displayed on a liquid crystal display or the like.

[0066] The bus 207 is an internal communication path of the computer 200 .

[0067] In this embodiment, the input unit 2, the calculation unit 3, the memory unit 4, and the display unit 5 are executed on one or more computers 200 having a hardware configuration as illustrated in FIG. 2, thereby realizing each of the processes described below.

[0068] Next, the data set creation process executed by the 3D search device of this embodiment will be described with reference to FIG.

[0069] FIG. 4 is a flowchart showing an example of the data set creation process.

[0070] The 3D input unit 101 and the metadata input unit 103 respectively accept input of the 3D data 40 and the text data of the metadata 41 added to the 3D data 40 (step S301), and store them in the memory 202.

[0071] The two-dimensional image acquisition unit 105 outputs multiple two-dimensional images by applying a rendering process that generates 2D data from 3D data, and the image feature extraction unit 107 extracts 3D feature vectors 44 from the rendered two-dimensional images (step S302).

[0072] Next, the token generation unit 104 generates tokens from the metadata 41 to serve as input data for the language-based model, and the text feature extraction unit 106 extracts a metadata feature vector 45 using the tokens generated by the token generation unit 104 as input (step S303).

[0073] Next, the feature amount fusion unit 110 executes the feature amount fusion process of FIG. 7, which will be described later, to fuse the 3D feature vector 44 and the metadata feature vector 45, and outputs the fusion vector 50 (step S304).

[0074] Next, the feature storage unit 111 stores the fusion vector 50 generated by the feature fusion unit 110, the 3D feature vector 44 extracted by the image feature extraction unit 107, and the metadata feature vector 48 extracted by the text feature extraction unit 106 in the database 112 as search keys (step S305), and then terminates this processing.

[0075] According to the process of FIG. 4, a data set of fusion vectors 50, 3D feature vectors 44, and metadata feature vectors 48 can be created and used for 3D searching using the fusion vectors.

[0076] Next, the 3D search process executed by the 3D search device of this embodiment will be described with reference to FIG.

[0077] FIG. 5 is a flowchart showing an example of the 3D search process.

[0078] The 3D input unit 101, the prompt receiving unit 102, and the metadata input unit 103 each receive input of 3D data 40, a prompt 42 entered by the user, and metadata 41 assigned to the 3D data 40 (step S401), and store them in memory 202.

[0079] The two-dimensional image acquisition unit 105 outputs multiple two-dimensional images by applying a rendering process to generate 2D data from the 3D data 40, and the image feature extraction unit 107 extracts 3D feature vectors 44 from the rendered two-dimensional images (step S402).

[0080] Next, the token generation unit 104 generates tokens from the prompt 42 and the metadata 41, which serve as input data for the language-based model, and the text feature extraction unit 106 uses the tokens generated by the token generation unit 104 as input to extract a metadata feature vector 45 and a prompt feature vector 46 (step S403).

[0081] Next, the text feature selection unit 108 executes the text feature selection process shown in FIG. 6, which will be described later, to automatically weight the metadata feature vector 45 that the user wants to emphasize in the search using the prompt feature vector 46 of the prompt 42 specified by the user, and outputs the weighted metadata feature vector 48 (step S404).

[0082] Next, the feature amount fusion unit 110 executes the feature amount fusion process of FIG. 7, which will be described later, to fuse the 3D feature vector 44 and the metadata feature vector 48, and outputs the fusion vector 50 (step S405).

[0083] Next, the search unit 109 executes a search process shown in FIG. 8, which will be described later, to extract feature vectors highly relevant to the search query from the feature vectors in the database 112 (step S406).

[0084] Next, the 3D display unit 114 displays, on the output unit 206 as a search result, the 3D data associated with the feature vector that is highly relevant to the search query and that has been extracted by the search unit 109 (step S407).

[0085] Next, the similarity display unit 113 displays the similarity of the feature vector of the 3D data displayed as the search result with the search query as a numerical value (step S408), and ends this process.

[0086] According to the process of Figure 5, a 3D search is performed using a fusion vector 50, which is a fusion of the 3D feature vector 44 and the metadata feature vector 48, as a search query, so that 3D data of parts with similar properties can be searched for, improving the accuracy of searching for the 3D data desired by the user.

[0087] Furthermore, since the metadata feature vector 45 that the user wants to emphasize in the search is automatically weighted by the prompt feature vector 46 of the prompt 42 specified by the user, the accuracy of the search for the 3D data desired by the user can be further improved.

[0088] FIG. 6 is a flowchart showing an example of the text feature selection process of the text feature selection unit 108.

[0089] First, the text feature selection unit 108 projects the prompt feature vector 46 and the metadata feature vector 45 into a latent space of the same dimension (step S11). If the prompt feature vector 46 and the metadata feature vector 45 use the same language encoder and have already output feature vectors of the same dimension, this step may be omitted.

[0090] Next, the text feature quantity selection unit 108 calculates the inter-vector similarity between the feature vectors of the prompt feature vector 46 and the metadata feature vector 45 (step S12).

[0091] In this embodiment, a self-attention mechanism is used to take an inner product using the text information specified by the user in the prompt 42 as a query and the text data included in the metadata 41 as a key, and a similarity matrix indicating the degree of association between the text features of the prompt 42 and the metadata 41 is calculated.

[0092] If prompt data cannot be obtained, the similarity matrix is ​​treated as a unit matrix so as not to affect the processing described below.

[0093] Next, the text feature selection unit 108 weights the metadata features by taking the inner product of the inter-vector similarity matrix indicating the degree of association and the metadata feature vector, updates the metadata feature vector (step S13), and ends this process. The updated metadata feature vector and 3D feature vector are input to the feature fusion unit 110.

[0094] According to the process of Figure 6, the text feature selection unit 108 generates a metadata feature vector 48 that emphasizes the features of the metadata 41 that the user wants to emphasize in a 3D search, based on the prompt 42 entered by the user, thereby improving the accuracy of searches for the 3D data desired by the user.

[0095] FIG. 7 is a flowchart showing an example of the feature fusion process performed by the feature fusion unit 110.

[0096] Since foundational models such as CLIP are trained to match language and image features, it is possible to compare feature vectors obtained from images and text. However, because these foundational models are trained on RGB images of general objects, there is a large gap between feature vectors obtained from 3D rendering images and feature vectors from text, making them uncomparable.

[0097] In this embodiment, the 3D feature vector 44 and the metadata feature vector 48 are converted into vector representations that can be compared in the latent space, and a fusion vector containing both feature quantities is generated.

[0098] First, the feature fusion unit 110 projects the 3D feature vector 44 into a latent space of the same dimension as the metadata feature vector 48 so that it can be compared with the metadata feature vector 48 (step S21). For the projection, a model trained to project the 3D feature vector 44 into the same latent space as the metadata feature vector is used, using a two-layer neural network consisting of a linear combination layer and an activation function such as ReLU (Rectified Linear Unit).

[0099] To train the network parameters, a method such as LoRA (Low-Rank Adaptation) can be used, which allows tuning of a trained model with a small number of parameters. This allows a small dataset to be prepared in advance and a model that can project the 3D feature vectors 44 and the metadata feature vectors 48 into a comparable latent space to be trained. The trained model is used to extract comparable vectors in the same latent space when building the database 112 (described later) and when searching for 3D data.

[0100] Next, the feature fusion unit 110 calculates the inter-vector similarity between the 3D feature vectors 44 and the metadata feature vectors 48 in order to fuse the 3D feature vectors 44 and the metadata feature vectors 48 that are generated by the above-mentioned processing and can be compared in the same latent space (step S22).

[0101] The vector similarity can be calculated using a self-attention mechanism and indicates the degree of association between metadata features and 3D features. Since fusing metadata features that are completely unrelated to 3D features may reduce search accuracy, it is necessary to fuse only related features.

[0102] In this embodiment, the feature fusion unit 110 uses a self-attention mechanism to fuse the 3D and metadata features by taking the inner product of the metadata feature vector 48 and a matrix indicating the inter-vector similarity between the 3D feature vector 44 and the metadata feature vector 48, thereby generating a fusion vector 50 (step S23), and then terminates this process.

[0103] The fusion vector 50 generated in this process is input to the search unit 109 .

[0104] The fusion vector 50 generated in step S23 further fuses the metadata feature vector 48 selected at the user prompt with the 3D feature vector 44, and utilizes the features of both the 3D data 40 and the metadata 41 for search. This makes it possible to distinguish between 3D data that have similar geometric shapes but different properties, which was difficult to do with searches that utilize only one modality, such as text or 3D data, in conventional technology.

[0105] In addition, by using the relevance between the 3D feature vector 44 and the metadata feature vector 48 during fusion, if the metadata information contains features that are unrelated to the target 3D data 40, the relevance will be low, and the fusion vector 50 can be generated by automatically omitting features with low relevance.

[0106] According to the processing of FIG. 7, the feature fusion unit 110 fuses the 3D and metadata features by taking the inner product of the metadata feature vector 48 and a matrix indicating the inter-vector similarity between the 3D feature vector 44 and the metadata feature vector 48, thereby generating a fusion vector 50, thereby improving the accuracy of searches for the 3D data desired by the user.

[0107] FIG. 8 is a flowchart showing an example of the search process of the search unit 109.

[0108] First, the search unit 109 selects a feature vector to be used as a search query 60 (FIG. 9) (step S31). The search query 60 can be selected based on information that the user wants to emphasize in the search from among the fusion vector 50 generated by the feature fusion unit 110, the 3D feature vector 44 of the 3D rendering image extracted by the image feature extraction unit 107, and the metadata feature vector 48 extracted by the text feature selection unit 108.

[0109] When a 3D feature vector 44 is selected as a search query 60, a search can be performed based on the geometric information of the 3D object. When a metadata feature vector 48 is selected as a search query, a search can be performed based on the properties of the 3D object contained in the metadata 41. When a fusion vector 50 is selected as a search query, a search can be performed based on both the shape and properties of the 3D data 40.

[0110] Here, the search process of the search unit 109 will be described with reference to FIG.

[0111] FIG. 9 is a diagram illustrating the search process of the search unit 109.

[0112] The feature vectors 61 stored in the database 112 include the 3D feature vectors 44, metadata feature vectors 45, and fusion vectors 50 described above.

[0113] The search query 60 corresponds to one of the 3D feature vectors 44, metadata feature vectors 48, and fusion vectors 50 obtained from the 3D data 40, metadata 41, and prompts 42 entered during the search, respectively, and is compared with each feature vector 61 stored in the database 112.

[0114] Returning to FIG. 8, the search unit 109 calculates the inter-vector similarity between the search query 60 and all feature vectors 61 in the database 112 in order to extract feature vectors 61 that are highly relevant to the search query 60 from among the feature vectors 61 stored in the database 112 (step S32).

[0115] Next, the search unit 109 extracts the feature vectors 61 having high vector similarity calculated in step S32 from the feature vectors 61 in the database 112 (step S33), and ends this process.

[0116] Here, the number of extracted feature vectors 61 is set in advance according to the number of search results that the user wants to display.

[0117] Furthermore, the extracted feature vector 61 is linked to an ID that identifies the 3D data 40 in the database 112 that corresponds to the feature vector 61.

[0118] The process of step S33 may use an approximate nearest neighbor search library that can perform a vector search at high speed.

[0119] According to the processing of Figure 8, the search unit 109 performs a 3D search using a search query 60 selected from the 3D feature vector 44, the metadata feature vector 48, and the fusion vector 50, thereby improving the search accuracy for the 3D data desired by the user.

[0120] FIG. 10 is a diagram showing an example of a display screen displayed by the 3D display unit 114 and the similarity display unit 113. As shown in FIG.

[0121] The display screen 70 is displayed on, for example, a liquid crystal display provided in the 3D search device 1.

[0122] The display screen 70 in FIG. 10 includes an input data display area 71, an input prompt display area 73, a 3D search result display area 72, and a metadata information display area 74.

[0123] The input data display area 71 displays 3D data 40 input by the user. The user inputs 3D data 40 having a similar shape to the 3D object they wish to search for in the 3D search device 1, for example, via the input unit 205. The 3D data 40 can be read as input in data formats such as 3D CAD data, 3D mesh data, and 3D point cloud data. The 3D display unit 114 displays the received 3D data 40 on a browser using a 3D viewer.

[0124] In addition, by dragging the displayed 3D data 40 on the input data display area 71 with the mouse, the 3D data 40 can be viewed from various viewpoints.

[0125] The display screen 70 also has functions for zooming in and out, allowing the user to check the entire 3D data 40 displayed in the input data display area 71 as well as details.

[0126] The input prompt display area 73 displays the prompt 42 entered by the user. The prompt 42 indicates an instruction for the user to select information that the user wants to emphasize in the metadata 41 in a search by the 3D search device 1.

[0127] For example, if a user wants to search with emphasis on material and dimensions, he or she executes the command by entering information about material and dimensions as prompt 42. In this way, information about material and dimensions can be automatically extracted from metadata 41, and search results that reflect the intent of prompt 42 can be obtained.

[0128] The 3D search result display area 72 displays search results for the 3D data 40 displayed in the input data display area 71 and the prompt 42 displayed in the input prompt display area 73. That is, the search unit 109 compares the search query 60 with feature vectors 61 in the database 112 and displays the 3D data 40 corresponding to the feature vector 61 with the highest similarity. In FIG. 10 , the 3D search result display area 72 displays, as an example, three search results in descending order of similarity.

[0129] The 3D search result display area 72 displays each 3D data 40 on the browser in a 3D viewer, similar to the input data display area 71. Therefore, by dragging the 3D data 40 displayed in the 3D search result display area 72 with the mouse, the search result 3D data 40 can be viewed from various viewpoints and scales.

[0130] The 3D search result display area 72 displays a similarity 75 between the search query 60 and the feature vector of the 3D data 40 displayed in the 3D search result display area 72. The similarity 75 is calculated by the search unit 109 in step S32 of FIG.

[0131] The metadata information display area 74 displays information about the metadata 41 that has a high degree of contribution to the similarity with the search query 60 for each piece of 3D data in the search results displayed in the 3D search result display area 72. This allows the user to confirm which piece of information in the metadata 41 contributed to the search results and to what degree of relevance. The relevance is calculated using the inter-vector similarity calculated by the text feature selection unit 108. A numerical value of the relevance may also be displayed together with the information about the metadata 41 displayed in the metadata information display area 74.

[0132] The user can check the display screen 70 and change the prompt 42 to change the referenced metadata 41, thereby obtaining different search results for the same 3D data 40.

[0133] The prompt 42 may be changed by the user manually modifying the input text.

[0134] In addition, by selecting any 3D data from the 3D data displayed in the 3D search result display area 72, the prompt 42 can be automatically tuned to search for 3D data having similar shapes and properties to the selected 3D data.

[0135] In this case, based on the information of the metadata 41 that contributes highly to the similarity of the selected 3D data to the search query 60, a loss function that measures the similarity of the feature vectors is used to calculate a prompt 42 that is suitable for searching for 3D data that has similar shapes and properties to the selected 3D data.

[0136] According to the display screen 70 in FIG. 10, the user can search for desired 3D data through intuitive operations.

[0137] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations. [Explanation of symbols]

[0138] 1...3D search device, 2...input unit, 3...calculation unit, 4...storage unit, 5...display unit, 101...3D input unit, 102...prompt reception unit, 103...metadata input unit, 104...token generation unit, 105...2D image acquisition unit, 106...text feature extraction unit, 107...image feature extraction unit, 108...text feature selection unit, 109...search unit, 110...feature fusion unit, 111...feature storage unit, 112...database, 113...similarity display unit, 114...3D display unit, 40...3D data, 41...metadata, 42...prompt, 43...basic model, 44...3D feature vector, 45...metadata feature vector, 46...prompt feature vector, 48...metadata feature vector, 50...fusion vector, 60...search query, 61...feature vector, 70...display screen, 71...input data display area, 72...3D search result display area

Claims

1. In a 3D search device for searching 3D data, A calculation unit and a storage unit are provided, The calculation unit extracting a 3D feature vector and a metadata feature vector from the 3D data and the metadata added to the 3D data, respectively; fusing the 3D feature vector and the metadata feature vector to generate a fusion vector; storing the fusion vector as a data set in the storage unit; A 3D search device that extracts, from the dataset, the fusion vector that is highly relevant to the fusion vector of the 3D data that is a search query.

2. The 3D search device according to claim 1, The calculation unit Extract a prompt feature vector from the prompt; calculating a vector similarity between the prompt feature vector and the metadata feature vector; A 3D search device that extracts, from the metadata feature vectors, metadata feature vectors that are highly relevant to the prompt feature vector.

3. 3. The 3D search device according to claim 2, The calculation unit A 3D search device that weights the metadata feature vector by calculating an inner product of the inter-vector similarity and the metadata feature vector.

4. 3. The 3D search device according to claim 2, The calculation unit A 3D search device that projects the 3D feature vector and the metadata feature vector into a latent space of the same dimension to fuse the 3D feature vector and the metadata feature vector.

5. 5. The 3D search device according to claim 4, calculating a vector similarity between the 3D feature vector and the metadata feature vector; A 3D search device that fuses the 3D feature vector and the metadata feature vector by taking an inner product of a matrix indicating the inter-vector similarity and the metadata feature vector.

6. 3. The 3D search device according to claim 2, The storage unit A 3D search device that stores the 3D feature vector, the metadata feature vector, and the fusion vector as a dataset.

7. 7. The 3D search device according to claim 6, The calculation unit A 3D search device that extracts from the dataset any of the 3D feature vectors, the metadata feature vectors, and the fusion vectors that are highly relevant to any of the 3D feature vectors, the metadata feature vectors, and the fusion vectors of the 3D data that is a search query.

8. 3. The 3D search device according to claim 2, further comprising a display unit for displaying search results; The display unit A 3D search device that displays the 3D data corresponding to one or more fusion vectors extracted from the dataset.

9. 9. The 3D search device according to claim 8, The display unit A 3D search device that displays the metadata of the extracted fusion vector that is highly relevant to the fusion vector of the 3D data that is the search query.

10. 9. The 3D search device according to claim 8, The calculation unit Accepting one selection from the 3D data displayed on the display unit; A 3D search device that calculates a prompt for searching for 3D data similar to the selected 3D data based on the metadata that is highly relevant to the selected 3D data.

11. In a 3D search method for a 3D search device for searching 3D data, The 3D search device includes a calculation unit and a storage unit, The calculation unit extracting a 3D feature vector and a metadata feature vector from the 3D data and the metadata added to the 3D data, respectively; fusing the 3D feature vector and the metadata feature vector to generate a fusion vector; storing the fusion vector as a data set in the storage unit; A 3D search method that extracts, from the dataset, a fusion vector that is highly relevant to the fusion vector of 3D data that is a search query.

Images