3D model search device and 3D model search method

Abstract

This technology enables high-speed processing of 3D models even as the amount of data increases. [Solution] The 3D model search device comprises: a search query receiving unit that receives search queries indicated by a 3D shape including boundary representations; a whole shape embedding unit that calculates a whole vector for the entire 3D shape of the search query; a partial shape embedding unit that calculates partial vectors for partial shapes which are solid, face, and edge units of the boundary representation; and a search unit that uses the whole vector and partial vectors to search for parts with similar shapes as similar parts from predetermined part information by vector search.

Description

Technical Field

[0005] ,

[0001] The present invention relates to a three-dimensional model search device and a three-dimensional model search method.

Background Art

[0002] Searching for three-dimensional shape models is an effective means of organizing them when there are a large number of three-dimensional models. In particular, searches based on similar shapes and text are effective in preventing the remodeling of three-dimensional models and reusing existing models. In the manufacturing and construction industries, products are represented as three-dimensional models by three-dimensional computer-aided design (hereinafter referred to as 3D (Dimension) CAD (Computer Aided Design)) and are used in CAE (Computer Aided Engineering) simulations, control, manufacturing plans, etc.

[0003] If the search for three-dimensional shape models can be realized, manufacturers can organize a large number of 3D CAD model assets created so far, prevent the redesign of similar parts, and extract and effectively utilize knowledge such as manufacturing methods and defects associated with similar parts based on the shape search results.

[0004] In 3D CAD, designers create the three-dimensional shape of a product on a computer using techniques including solid modeling and parametric modeling. In 3D CAD software, the three-dimensional shape is generally represented by a Brep (Boundary REPresentation) that describes the solid, surface, edge, point of the shape and its topological information (connection relationship information). Hereinafter, the three-dimensional shape created by the above-mentioned 3D CAD software is called a CAD model. Generally, in Brep, it is possible to accurately represent curved surfaces such as spherical surfaces and the topology such as the connection of surfaces and edges.

[0005] Other methods for representing 3D shapes include using meshes and point clouds. Meshes approximate and represent 3D shapes as a collection of triangular patches, and are the most commonly used method for visualization because rendering can be done quickly using dedicated hardware. However, meshes do not contain information about curved surfaces or topology.

[0006] A point cloud is a method of representing a shape as a collection of points that hold attribute information such as color. Point clouds are used for 3D shape measurement data using sensors, and because of their simple data structure, they are often used as input for machine learning to recognize 3D shapes. The recognition results of point clouds by machine learning models can also be used for searching for 3D shapes. On the other hand, since point clouds approximate shapes with sparse points, point cloud data does not contain information about surfaces or topology. Also, if you try to represent a detailed shape, the number of points required can increase exponentially.

[0007] As described above, while using approximate representations of 3D shapes such as meshes and point clouds allows for efficient processing in specific applications, compared to 3D representations using Brep, which are mainly used in manufacturing, there is a lack of topology information (patchwork (edge) information) and surface information, which can make them inefficient for design applications in manufacturing.

[0008] There are two use cases for searching 3D shapes: one where you search for shape records stored in a database using text or shape queries, and another where you search for partial shapes, including those contained within larger shapes such as scenes or assemblies, using text or shape queries.

[0009] The former method involves, for example, converting the part shapes stored in each record of a database into point clouds, converting the shapes into vectors using deep learning-based methods such as PointNet, and then searching for similar shapes through vector search.

[0010] Patent Document 1 discloses a search method for partial shapes, which involves training a one-shot detection model using pairs of training 3D scenes converted to point clouds and objects to search for 3D query shapes including CAD models, and outputting boundary volumes and shape similarity scores by combining representation descriptors of query objects and scenes. [Prior art documents] [Patent Documents]

[0011] [Patent Document 1] Japanese Patent Publication No. 2022-183068 [Overview of the Initiative] [Problems that the invention aims to solve]

[0012] However, the technology described in Patent Document 1 suffers from a high computational load when there are a large number of scenes to be searched. Since it is necessary to perform neural network calculations for object detection in each scene, the computational load is proportional to the number of scenes, making searching computationally difficult as the number of scenes increases. The objective of the present invention is to enable high-speed 3D model searching even when the amount of data increases. [Means for solving the problem]

[0013] The present invention includes several means for solving at least some of the above problems, and an example thereof is as follows: A three-dimensional model search device according to one aspect of the present invention that solves the above problems includes: a search query receiving unit that receives a search query indicated by a three-dimensional shape including a boundary representation; a whole shape embedding unit that calculates a whole vector that covers the entire three-dimensional shape of the search query; a partial shape embedding unit that calculates partial vectors that cover partial shapes which are solid, face and edge units of the boundary representation; and a search unit that uses the whole vector and the partial vectors to search for parts with similar shapes as similar parts from predetermined part information by vector search. [Effects of the Invention]

[0014] According to the present invention, even when the amount of data for showing the shape in detail increases, a three-dimensional model search that can be processed at high speed becomes possible. Problems, configurations, and effects other than those described above will be clarified by the description of the mode for carrying out the following invention.

Brief Description of Drawings

[0015] [Figure 1] It is a diagram showing a configuration example of a design information search system. [Figure 2] It is a diagram showing a configuration example of CAD shape information. [Figure 3] It is a diagram showing a configuration example of embedding information. [Figure 4] It is a diagram showing an example of a flowchart of search processing. [Figure 5] It is a diagram showing an example of a flowchart of embedding calculation processing. [Figure 6] It is a diagram showing an example image of embedding calculation processing. [Figure 7] It is a diagram showing an example image of data used in search processing. [Figure 8] It is a diagram showing an example of a display screen of search results. [Figure 9] It is a diagram showing an example of the hardware configuration of a design information search device.

Mode for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments are examples for explaining the present invention, and for the sake of clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise limited, each component may be singular or plural.

[0017] In the drawings, the positions, sizes, shapes, ranges, etc. of the respective components shown may not represent the actual positions, sizes, shapes, ranges, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0018] As examples of various types of information, descriptions may be made using expressions such as "table", "list", "queue", etc., but the various types of information may be represented by data structures other than these. For example, various types of information such as "XX table", "XX list", "XX queue" may be referred to as "XX information". When explaining identification information, expressions such as "identification information", "identifier", "name", "ID", "number", etc. are used, but these are mutually replaceable. Also, the identification information described by these expressions is represented by symbols, numerical values, natural language, or a combination thereof, etc. in the embodiments, but the identification information may be in other forms.

[0019] When there are a plurality of components having the same or similar functions, they may be described by attaching different subscripts to the same reference numeral. Also, when it is not necessary to distinguish these plurality of components, the subscripts may be omitted in the description.

[0020] In embodiments, processing performed by executing a program may be described. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs processing defined by the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The main entity performing the processing by executing the program may be an arithmetic unit, and may include dedicated circuits that perform specific processing. Here, dedicated circuits include, for example, FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), CPLDs (Complex Programmable Logic Devices), etc.

[0021] The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in some embodiments, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0022] Furthermore, while the present invention is typically implemented by an information processing device, it may also be implemented as a platform having the functions of the present invention.

[0023] In this embodiment, the design information retrieval system 1 searches for parts with similar shapes as similar parts from predetermined part information using vector search.

[0024] The technology described in Patent Document 1 above processes the 3D CAD model after converting it to a point cloud. Therefore, when the shape is large and complex, or when the part shape to be found is small relative to the entire model, the information may not be represented in the point cloud. In addition, point cloud-based methods such as PointNet are known to have lower accuracy than Brep-based methods such as UVNet in part type classification, and from the perspective of the accuracy of deep learning models, there are disadvantages to recognition targeting CAD models. In particular, it is not possible to directly present solids, faces, and edges created in 3D CAD as search results. To present them, it is necessary to separately compare which entity of the CAD model from which the point cloud was generated is related to the search results. Also, because the output is a bounding volume, it is possible that neighboring parts with low correlation to the query shape may be included in the results.

[0025] Furthermore, the technology described in Patent Document 1 above creates a detection model using a dataset with pre-annotated shapes. In the case of creating a general shape search system, designing and annotating class labels is a costly process. Moreover, since it is often difficult to anticipate search queries, it is not easy to design appropriate class labels and prepare a corresponding search dataset.

[0026] Therefore, in order to solve the above problems, in the embodiment of the present invention, the 3D CAD model is not converted to a format such as a point cloud, and the similarity of the shapes is evaluated by a deep learning model that takes geometric information consisting of parametric surface and curve information contained in Brep and topology information consisting of information such as the connections between faces and edges as input.

[0027] The design information retrieval system 1 can also achieve the following effects: By using Brep to compare similarities, information loss in topology (patchwork (edge) information) and surface information, which can occur when using point clouds or meshes, is reduced. Furthermore, even when the shape of the target object is complex or large, it is possible to accurately represent the shape with a relatively small data size, thus improving the accuracy of similarity judgments.

[0028] Because Brep retains information on solids, faces, and edges, it is possible to search for information on parts and solids included in the assembly, as well as information on small faces such as fillets, thus preventing detection omissions.

[0029] Because it does not use detection models such as object classifiers or segmentation for searching, it does not require the existence of training data such as pairs of shape models with defined class labels and sub-shapes, allowing for the preparation of a search system at a low cost. The cost of creating a shape search system can be reduced by utilizing learning methods that use unlabeled data, such as self-supervised learning using contrastive learning.

[0030] Furthermore, multimodal learning using comparative learning allows the search queries to be extended to other data formats, including text.

[0031] Furthermore, in the embodiment of the present invention, assuming the search for partial shapes from a large number of stored assembly CAD shapes, the embedding vector of the entire shape such as an assembly, and the embedding vector of the partial shape including the subassembly, part, solid, face, and edge to be searched are stored in a database, and vector indexes are constructed for each using approximate nearest neighbor method or the like. In addition, to enable consideration of the topology of the shape, a topology database is constructed that includes topology information such as the connection relationships of faces and edges of Breps, in association with the above database. Strictly speaking, in this embodiment, Brep can be any boundary representation (a method of representing a shape with multiple independent faces), and does not refer to any specific standard or rule. Also, in this embodiment, boundary representations may be referred to as Breps, or Breps may be referred to as boundary representations.

[0032] Then, when shape information is provided as a search query, the embedding vectors for the overall shape and the embedding vectors for the partial shape are calculated for the shape of the search query, and the above database is searched using the obtained overall and partial shape embedding vectors. The search for overall shape vectors and the search for partial shape vectors can be performed sequentially, by first narrowing down the search by searching for overall shape vectors and then searching for partial shape vectors. Alternatively, it is also possible to process the search for overall shape vectors and partial shape vectors in parallel.

[0033] Furthermore, in the embodiment of the present invention, the results of the search for the overall shape vector and the search for the partial shape vector are integrated to narrow down the candidate overall shapes that include more similar partial shapes, and then the search results are re-ranked using the similarity between the topology information. To determine the similarity between the topology information, the topology information such as faces and edges of parts that had high similarity as a result of the shape search is matched with the topology information of the shape of the search query, the instance segmentation of the search target shape in the overall shape and the similarity score of the topology information are calculated, and the search results are re-ranked based on that score. In the embodiment of the present invention, since the search process uses vector search, it is possible to process even if the amount of data to be searched increases.

[0034] Furthermore, applying the design information retrieval system 1 according to this embodiment to the manufacturing industry enables highly accurate and high-speed shape searches of existing design information using a search method based on boundary representation and vector search. In addition, the preparation of the search system is simplified, and it becomes possible to search using multimodal information including not only shape but also text as search queries. As a result, various man-hour reduction effects can be expected, mainly in the design process, such as promoting reuse of designs when designing 3D CAD parts, improving the accuracy of checking design shapes using recognition of shape features, and reducing man-hours in design, production design, and manual creation by reusing related information knowledge through shape search.

[0035] Figure 1 shows an example of the configuration of a design information retrieval system. The design information retrieval system 1 is composed of an information processing device. This information processing device is a design information retrieval device 100, which includes a storage unit 110, a processing unit 120, an input unit 130, and a display unit 140. In particular, the design information retrieval device 100 can be described as a 3D model retrieval device.

[0036] First, the memory unit 110 stores CAD shape information 111, embedding information 112, and query history information 113. The processing unit 120 includes a search query receiving unit 121, an overall shape embedding unit 122, a partial shape embedding unit 123, a search unit 124, a reranking unit 125, and a screen generation unit 126.

[0037] Figure 2 shows an example of the structure of CAD shape information. The CAD shape information 111 is the component configuration information of a product and includes a product ID 111a, a component ID 111b, shape information 111c, and topology information 111d. Specifically, the product ID 111a contains information that distinguishes the product from other products. The component ID 111b contains information that distinguishes the components that make up the product from other components. The shape information 111c stores general CAD data files or reference information to CAD data files as information that identifies the shape of the product. The topology information 111d is information that maps faces to CAD shapes as nodes and edges to CAD shapes as links.

[0038] Figure 3 shows an example of the structure of embedding information. The embedding information 112 is vectorized information of a product and includes a product ID 112a, a part ID 112b, a partial shape 112c, and vector information 112d. Specifically, if the vector information 112d is a vector corresponding to the entire product, the vector information 112d is associated with the product ID 112a; if the vector information 112d is a vector corresponding to a part that makes up the product, the vector information 112d is associated with the combination of product ID 112a and part ID 112b; and if the vector information 112d is a vector corresponding to a partial shape (solid, face, edge), the vector information 112d is associated with the partial shape 112c and stored in the embedding information 112. The vector information 112d is feature information of a product or part calculated by an existing predetermined method and is generally treated as a matrix. Furthermore, the vector information 112d is information that makes it possible to calculate similarity by determining the inter-vector distance, such as the dot product. Therefore, the CAD shape information 111 and the embedding information 112 can be described as a database that stores each of the partial shapes, including faces, edges, and solids included in the three-dimensional shape, as records associated with the partial shape embedding.

[0039] The embedding information 112 is preferably a vector database, indexed to allow for fast retrieval of similar vectors using methods such as approximate nearest neighbor or locality-sensitive hashing.

[0040] The search query receiving unit 121 receives search queries that are represented by a three-dimensional shape including boundary representations. The search query receiving unit 121 also extracts topology information from the boundary representation information of the CAD shape of the received search query.

[0041] The overall shape embedding unit 122 calculates an overall vector that covers the entire three-dimensional shape of the search query.

[0042] The partial shape embedding unit 123 calculates partial vectors for the partial shape, which is the unit of solids, faces, and edges in the boundary representation.

[0043] The search unit 124 uses the whole vector and partial vectors to perform a vector search to find parts with similar shapes as similar parts from predetermined part information (for example, CAD shape information 111).

[0044] The reranking unit 125 matches the subgraphs of the topology graph of Brep between the search query and the similar parts found by the search unit 124, and reranks the search results from the search unit 124.

[0045] The screen generation unit 126 generates a screen that highlights the parts of the list of three-dimensional shapes of similar parts found by the search unit 124 that match the search query for each three-dimensional shape.

[0046] Alternatively, the search query receiving unit 121 may accept text input instead of a three-dimensional shape as a search query, the overall shape embedding unit 122 may use a pre-trained model that has been trained to output an overall vector when it receives text as input to calculate an overall vector from the text received by the search query receiving unit 121, and the partial shape embedding unit 123 may use a pre-trained model that has been trained to output partial vectors when it receives text as input to calculate partial vectors from the text received by the search query receiving unit 121.

[0047] Alternatively, the search query receiving unit 121 may accept input text related to metadata, including product ID, part ID, material, etc., as search conditions for the search query, and the search unit 124 may narrow down the search to parts with a high degree of relevance to the metadata related to the text from predetermined part information (for example, CAD shape information 111) and perform the search.

[0048] The input unit 130 accepts various instructions from the operator. The input content includes search queries. Specifically, the input unit 130 accepts 3D shapes containing Brep information and search queries via input devices such as a mouse and keyboard. For example, the input unit 130 accepts operations such as selecting local files or files on a server as input for 3D shapes. However, it is not limited to this; the input unit 130 can also accept programmatic instructions and communications via an API (Application Protocol Interface). The 3D CAD model format includes intermediate formats such as STEP and Parasolid, as well as native formats.

[0049] Furthermore, when the input unit 130 receives the shape of a search query, it may be possible to use either a method in which the operator specifies the entire assembly or part contained in the file, or a method in which the operator selects multiple subassemblies, solids, faces, or edges as partial shapes from the assembly or part by clicking or other operations.

[0050] The display unit 140 outputs (displays) a screen that highlights the parts of each three-dimensional shape that match the search query, based on the list of three-dimensional shapes of similar parts that were searched for by the search unit 124 and generated by the screen generation unit 126.

[0051] Furthermore, the display unit 140 displays the output via a display device such as various displays. For this reason, the input unit 130 and the display unit 140 may be integrated, such as a touch panel. In addition, the input unit 130 and the display unit 140 may be configured in separate housings from the design information retrieval device 100. In this case, the input unit 130 and the display unit 140 may be implemented by separate terminal devices, and the input and output information may be transferred via a network. The network may be, for example, a LAN (Local Area Network), WAN (Wide Area Network), VPN (Virtual Private Network), a communication network using public lines such as the Internet in part or in whole, a mobile phone communication network, or a network that is a combination of these. The network may also be a wireless communication network such as Wi-Fi (registered trademark) or 5G (Generation). Furthermore, the information processing device constituting the design information retrieval system 1 is not limited to one unit, and there may be more than one unit for purposes such as redundancy and load balancing. For example, a client terminal handling the search query reception unit 121, screen generation unit 126, input unit 130, and display unit 140, an embedding calculation server handling the overall shape embedding unit 122 and partial shape embedding unit 123, a search server handling the search unit 124 and rerank unit 125, and a database server holding CAD shape information 111, embedding information 112, and query history information 113 may be connected to each other via a network and cooperate to form a design information retrieval system 1. However, in this embodiment, for the sake of simplicity, it will be described as a single information processing device.

[0052] Figure 4 shows an example of a search process flowchart. The search process starts when the operator gives a start command via the input unit 130.

[0053] First, the search query receiving unit 121 receives CAD information including Brep information as a query (step S001). Specifically, the search query receiving unit 121 receives CAD information including Brep information for a part of the shape of the part to be searched (a sub-assembly, part, a partial shape consisting of faces, edges, etc.) or the whole as a search query. Alternatively, the search query receiving unit 121 accepts text input instead of a 3D shape as a search query. Or, the search query receiving unit 121 accepts text input related to metadata along with the 3D shape as a search query.

[0054] Then, the search query receiving unit 121 extracts topology information from the Brep information of the CAD shape of the received search query (step S002). Specifically, the search query receiving unit 121 extracts topology information from the Brep information, where faces are treated as nodes and edges as links, and information that has been mapped to the CAD shape.

[0055] Then, the overall shape embedding unit 122 performs embedding calculation processing on the entire CAD shape of the search query (step S003). Specifically, the overall shape embedding unit 122 performs the embedding calculation processing described later on the entire three-dimensional shape of the search query.

[0056] Then, the partial shape embedding unit 123 performs embedding calculation processing for the CAD shape portion of the search query (step S004). Specifically, the partial shape embedding unit 123 targets the three-dimensional shape portion of the search query (sub-assemblies, parts, faces, edges, etc.) and performs the embedding calculation processing described later. Note that the embedding calculation processing described later has already been performed in step S003, so it does not need to be performed again, and it is sufficient to identify the partial embedding.

[0057] Then, the search unit 124 performs matching processing with existing embedding information (step S005). Specifically, the search unit 124 uses the embedding information 112 related to the existing part and the overall vector and partial vector of the search query to search and extract parts with similar shapes as similar parts from the CAD shape information 111 using vector search. For example, in the vector search, the search unit 124 searches for multiple (K items: K is a natural number) with high similarity indices, including cosine similarity, for each of the overall vector and partial vector of the search query.

[0058] The search unit 124 retrieves the top few (K1: K1 is a natural number) CAD models with similar overall vectors and their similarity scores for the overall vector of the search query. For the sub-vectors of the search query, it searches for faces and edges with similar vector information 112d registered in the embedding information 112 for each sub-vector corresponding to a subassembly, solid, face, edge, etc. of the shape of the search query, and retrieves the top few (K2: K2 is a natural number) similar subassemblies, solids, faces, edges, etc. and their similarity scores. Here, the search using sub-vectors targets all shapes included in the embedding information 112. However, if the search computation load is too high or the capacity of the search resources is insufficient, a sequential search targeting only the K1 shapes narrowed down by the overall vector search may be performed.

[0059] The search unit 124 then integrates the results of the matching process with existing embedding information to identify the top K similar parts. In this integration, the search unit 124 calculates a composite index of the similarity score of the search results for the overall vector and the similarity score of the search results for the sub-vectors. The composite index can be a linear sum of the similarity of the overall vector and the average of the similarity of the sub-shape vectors of the shapes with the highest similarity for each search query shape, or a simple weighted average. However, it is not limited to this, and the composite index may also be calculated using a neural network that combines linear transformations and nonlinear functions with the similarity of the overall vector and sub-vectors as input.

[0060] Then, the reranking unit 125 searches for CAD shape information 111 for the top matching parts (K pieces) and obtains topology information 111d (step S006).

[0061] Then, the reranking unit 125 performs a reranking (sorting) process on the top-ranking parts (K parts) in order of topology similarity, using the embedding information as the basis for calculation (step S007). Specifically, in order to compare topology information, the reranking unit 125 describes the shape of Brep as a face adjacency graph in which faces are nodes and edges are edges, and calculates the similarity of the subvectors by determining the node matching rate and the edge matching rate. The reranking unit 125 then calculates the topology similarity by quantifying the degree of agreement between the shape of the search query and the nodes and edges of the face adjacency graph of the subshapes of the CAD models of the top-ranking parts.

[0062] The reranking unit 125 uses subgraph matching to quantify the similarity between graphs for matching face-adjacent graphs, but is not limited to this method. For example, the reranking unit 125 can employ methods to perform node and edge matching as an optimal transport problem using a greedy algorithm or Hungarian algorithm and evaluate the degree of match. Alternatively, the reranking unit 125 can employ a method to evaluate graph similarity using a graph neural network. Furthermore, when comparing topologies, it is possible to exclude face nodes that have a small contribution to the overall shape, such as fillets and chamfers, from the comparison by removing them from both the shape of the search query and the top-ranking matching components, or by reducing their weight during evaluation, thereby evaluating without giving much weight to differences in fillets and chamfers.

[0063] The reranking unit 125 then obtains a similarity score between the shape of the search query, which includes topology information, and the partial shape of the CAD of the existing part, and reranks the search results based on this score.

[0064] Then, the screen generation unit 126 generates screen information to present the top K results of the comparison with the search query based on the reranking results (step S008). Specifically, the screen generation unit 126 receives a list of partial shapes corresponding to the shapes of the CAD models of the search results reranked by the reranking unit 125 and the shapes of the search queries, and processes it in accordance with the screen layout information to be displayed.

[0065] Then, the display unit 140 displays the screen generated by the screen generation unit 126 (step S009).

[0066] Then, if a registration instruction is given, the search unit 124 stores the CAD information of the search query as a new part in the CAD shape information 111 (step S010).

[0067] The above is an example of a search process flowchart. According to this example, it becomes possible to perform high-speed 3D model searches even when the amount of data used to describe the shape in detail increases.

[0068] Figure 5 shows an example flowchart of the embedding calculation process. The embedding calculation process starts in step S003 of the search process.

[0069] First, the overall shape embedding unit 122 receives CAD information including Brep information as the target shape (step S101). The target shape refers to the shape for which embedding is calculated. Therefore, it does not matter whether it is the shape of the search query or the shape of a part included in the CAD shape information 111.

[0070] The overall shape embedding unit 122 then extracts geometric and topological information contained in the Brep information of the received CAD information and generates graph structure data (step S102). Specifically, the overall shape embedding unit 122 forms a surface adjacency graph from the extracted topological information, with the faces of the Brep as nodes and the edges as edges. It then creates and links feature quantities to each node and edge from information such as the area, curvature, type, and coordinates of the faces obtained from the extracted geometric information, as well as information such as the length, concavity, and orientation of the edges, thereby generating graph structure data.

[0071] Furthermore, in addition to face-adjacent graphs, graph structure data that represents information including edge loops, solids, and assembly trees as graphs may also be adopted, resulting in heterogeneous graphs. By constructing a graph neural network that takes such graph structure data as input and training it using learning methods such as symmetric learning with the InfoNCE loss function, it becomes possible to obtain vectors (corresponding to subvectors) for each node and edge from this graph structure data, as well as a neural network that calculates graph embeddings (corresponding to the overall vector) using graph pooling and Set2Set. Possible graph neural networks include message-passing neural networks that can utilize edge information, and graph convolutional networks targeting heterogeneous graphs.

[0072] Furthermore, regarding graph embedding, it is possible to combine text and image metadata not included in Brep information, encoded into vectors, to create an embedding vector. Additionally, during comparative learning, it is possible to train a network that obtains embedding from text information instead of graphs by replacing the image encoder of CLIP (Contrastive Language-Image Pre-training), a machine learning model that processes images and text simultaneously, with this graph neural network, and training both text and graph information together. In this way, even when the search query contains only text information, embedding can be calculated, enabling shape search processing.

[0073] Then, the overall shape embedding unit 122 calculates the overall vector information of the target shape using the graph structure data (step S103). Specifically, the overall shape embedding unit 122 obtains the overall vector using the entire graph structure data generated in step S102. In this process, if the aforementioned graph neural network, which takes graph structure data represented as a heterogeneous graph as input, is used, it is also possible to calculate the overall vector using graph pooling or Set2Set.

[0074] Then, the partial shape embedding unit 123 calculates partial vector information for the solids, faces, and edges that constitute the Brep using the graph structure data (step S104). Specifically, for the solids, faces, and edges included in the graph structure data generated in step S102, the partial shape embedding unit 123 uses the feature vectors of the nodes and edges as the corresponding partial shape embeddings for the faces and edges, and obtains vectors using the graph embedding for each connected component of the graph as the partial shape embedding for the solid. In this process, it is possible to calculate partial vectors by using the aforementioned graph neural network, which takes graph structure data represented as a heterogeneous graph as input.

[0075] Then, the overall shape embedding unit 122 stores the vector information of the target shape as query history in the query history information 113, or as embedding information in the embedding information 112 if the target shape is not a search query (step S105).

[0076] The above is an example of a flowchart for embedding calculation. According to this example of embedding calculation, it is possible to obtain overall vector information and partial vector information using CAD information including Brep information.

[0077] The neural networks used to calculate overall shape embedding and partial shape embedding may be the same, or specialized networks may be used for each.

[0078] Figure 6 shows an example of the embedding calculation process. In the embedding calculation process 200, the input CAD shape or text information is encoded using a shape encoder to obtain the overall embedding. In addition, by encoding using a partial shape encoder, partial embedding can be obtained for each part of the input CAD shape or text information, such as faces and edges.

[0079] Figure 7 shows an example of the data used in the search process. As example 300, we assume a pair 301 of a search query shape consisting of two faces and a CAD shape with a similar sub-shape. The overall embedding 302 represents each of these shapes as a single multidimensional vector, and can be compared using cosine similarity, etc.

[0080] Partial embedding 303 is represented by a graph in which faces are nodes and edges are edges for each pair of the search query shape and CAD shape 301, and then a vector is obtained for each of these nodes and edges to form a subvector. In Example 300, nodes that are judged to be similar as a result of comparing the subvectors of the nodes are connected by arrows.

[0081] The whole / partial comparison 304 is an example that further incorporates topology information, comparing the entire graph, which includes topology information of the search query shape, with the information of a subgraph containing multiple nodes and edges that are similar in CAD shape. The matching process in step S005 of the search process is realized by comparing the whole embedding 302 and the partial embedding 303. Furthermore, the reranking process in step S007 is realized by determining the similarity of the topologies through the subgraph-level comparison in the whole / partial comparison 304.

[0082] Figure 8 shows an example of a search results display screen. Example 400 of the search results display screen includes an input area 401 for specifying the type of search query, a query input area 402 for specifying or uploading a shape, an input field 403 for additional information to narrow down the search target, and a search button 404 for receiving a command to start the search process. If the type of search query in input area 401 is set to "Shape", then the query input area 402 can be used to specify a 3D shape or its CAD data containing Brep information via file or reference. If the type of search query in input area 401 is set to "Text", then the query input area 402 can be used to input free text.

[0083] The additional information input field 403 accepts metadata such as material name, name, and product name. Upon receiving text related to the metadata, the search unit 124 narrows down the CAD shape information 111 to parts with a high degree of relevance to the metadata related to the text and performs a search.

[0084] When input is received via the search button 404, the search process is executed, and the screen generated in step S009 is output to the results display 410. In the example in Figure 8, the results display 410 shows, as an example of the search results, the CAD shape name, a preview of the CAD shape with the matched partial shapes highlighted (highlighted) according to the partial shape similarity, the similarity score, and the number of similar partial shapes. Clicking on the highlighted part in the preview allows you to view the corresponding partial shape similarity value. This partial shape similarity is calculated when reranking is performed in step S007 of the search process.

[0085] [Explanation of Hardware Configuration] Figure 9 shows an example of the hardware configuration of an information processing device that constitutes a design information retrieval system. The information processing device 900 that constitutes the design information retrieval system 1 can be realized as a general-purpose computer or a network system equipped with multiple such computers, comprising a processor (e.g., CPU: Central Processing Unit, or GPU: Graphics Processing Unit) 901, hardware memory 902 such as RAM (Random Access Memory), storage 903 such as a hard disk drive (HDD) or SSD (Solid State Drive), a reader 905 that reads information from a portable storage medium 904 such as a CD (Compact Disk) or DVD (Digital Versatile Disk), an input device 906 such as a keyboard, mouse, barcode reader, or touch panel, an output device 907 such as a display, and a communication device 908 that communicates with other computers via a communication network such as a LAN or the Internet. Note that the reader 905 may be capable of writing as well as reading portable storage media.

[0086] The processor 901 performs various processes by executing predetermined programs loaded from the storage 903 into the memory 902. These programs are, for example, application programs that can be executed on an OS (Operating System) program. These programs may be installed in the storage 903 from a portable storage medium 904 via a reader 905, or they may be downloaded from a network via a communication device 908 and executed by the processor 901.

[0087] For example, the processing unit 120, search query receiving unit 121, overall shape embedding unit 122, partial shape embedding unit 123, search unit 124, reranking unit 125, and screen generation unit 126 can be implemented by loading a program stored in storage 903 into memory 902 and executing it with processor 901. The input unit 130 and display unit 140 can be implemented by processor 901 using input device 906, output device 907, and communication device 908. The storage unit 110 can be implemented by processor 901 using memory 902 or storage 903.

[0088] The above describes the design information retrieval system 1 according to the embodiment. The design information retrieval system 1 enables high-speed 3D model retrieval even when the amount of data increases. The present invention is not limited to the above-described embodiment and includes various modifications. For example, the above-described embodiment is described in detail to make the present invention easy to understand and is not necessarily limited to having all the described configurations. It is possible to replace some of the configurations of the embodiment with other configurations, and it is also possible to add configurations of other embodiments to the configuration of the embodiment. Furthermore, it is possible to delete some of the configurations of the embodiment.

[0089] Each of the above-mentioned parts, configurations, functions, and processing units may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above-mentioned parts, configurations, and functions may be implemented in software by having the processor interpret and execute programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in memory, a recording device such as a hard disk, or a recording medium such as an IC card, SD card, or DVD.

[0090] It should be noted that the control lines and information lines in the embodiments described above are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. In practice, it can be assumed that almost all components are interconnected. The present invention has now been described, focusing on its embodiments. [Explanation of symbols]

[0091] 1: Design information retrieval system, 100: Design information retrieval device, 110: Storage unit, 111: CAD shape information, 112: Embedding information, 113: Query history information, 120: Processing unit, 121: Search query reception unit, 122: Overall shape embedding unit, 123: Partial shape embedding unit, 124: Search unit, 125: Rerank unit, 126: Screen generation unit, 130: Input unit, 140: Display unit.

Claims

1. A search query receiving unit that accepts search queries represented by three-dimensional shapes including boundary representations, A whole shape embedding unit calculates a whole vector that covers the entire three-dimensional shape of the search query, A partial shape embedding unit calculates partial vectors targeting the partial shape, which is the solid, face, and edge unit of the boundary representation, A search unit that uses the overall vector and the partial vector to search for parts with similar shapes as similar parts from predetermined part information by vector search, A 3D model search device equipped with the following features.

2. A 3D model search device according to claim 1, A reranking unit performs matching between the search query and the similar components found by the search unit, and the subgraphs of the topology graph of the boundary representation, and reranks the search results from the search unit. A 3D model search device equipped with the following features.

3. A 3D model search device according to claim 1, A screen generation unit generates a screen that highlights the portion of each three-dimensional shape that matches the search query in the list of three-dimensional shapes of similar parts found by the search unit. A 3D model search device equipped with the following features.

4. A 3D model search device according to claim 1, The search query receiving unit accepts text input instead of the three-dimensional shape input. The overall shape embedding unit calculates the overall vector from the text received by the search query receiving unit using a model that has been trained to output the overall vector when it receives text as input. The partial shape embedding unit calculates the partial vector from the text received by the search query receiving unit using a model that has been trained to output the partial vector when it receives text as input. A three-dimensional model search device characterized by the following features.

5. A 3D model search device according to claim 1, The aforementioned predetermined part information includes a database that stores each of the partial shapes, including faces, edges, and solids included in the three-dimensional shape, as a record associated with the partial shape embedding. A three-dimensional model search device characterized by the following features.

6. A 3D model search device according to claim 1, The aforementioned search query receiving unit accepts input of text related to metadata, including product name or material, as search conditions. The search unit performs the search by narrowing down the parts from the predetermined part information to those parts that have a high degree of relevance to the metadata related to the text. A three-dimensional model search device characterized by the following features.

7. A method for searching for a 3D model using an information processing device, The information processing device comprises a storage unit for storing predetermined component information and a processing unit, The aforementioned processing unit, A search query acceptance step that accepts search queries represented by a three-dimensional shape including boundary representations, A whole shape embedding step that calculates a whole vector that covers the entire three-dimensional shape of the search query, A sub-shape embedding step calculates sub-vectors targeting sub-shapes which are units of solids, faces, and edges of the boundary representation, A search step in which parts with similar shapes are searched as similar parts from predetermined part information using the overall vector and the partial vector, A method for searching for a 3D model, characterized by performing the following steps.

Images