Three-dimensional model search device and three-dimensional model search method

Abstract

Provided is technology for enabling a three-dimensional model search that can be processed at high speed even when the amount of data increases.　This three-dimensional model search device comprises: a search query accepting unit that accepts a search query indicated by a three-dimensional shape including a boundary representation; an overall shape embedding unit that calculates an overall vector targeting the entire three-dimensional shape of the search query; a partial shape embedding unit that calculates partial vectors targeting partial shapes that are units of solids, surfaces, and edges of the boundary representation; and a search unit that employs a vector search using the overall vector and the partial vectors to search for components having a similar shape, as similar components, from predetermined component information.

Description

Stereo model search device and stereo model search method

[0001] The present invention relates to a stereo model search device and a stereo model search method. The present invention claims the priority of Japanese Patent Application No. 2024-201743 filed on November 19, 2024. For designated countries where incorporation by reference is permitted, the content described in that application is incorporated into the present application by reference.

[0002] Searching for 3D shape models is an effective means of organizing them when there are a large number of 3D models. In particular, searches based on similar shapes and text are effective for preventing remodeling of 3D models and reusing existing models. In the manufacturing and construction industries, products are represented as 3D models by 3D computer-aided design (hereinafter referred to as 3D (Dimension) CAD (Computer Aided Design)) and are used for CAE (Computer Aided Engineering) simulations, control, manufacturing plans, etc.

[0003] If the search for 3D shape models can be realized, it will be possible for manufacturing companies to organize a large number of 3D CAD model assets they have created so far, prevent the redesign of similar parts, and extract and effectively utilize knowledge such as manufacturing methods and defects associated with similar parts based on the shape search results.

[0004] In 3D CAD, designers create the 3D shape of a product on a computer using techniques including solid modeling and parametric modeling. In 3D CAD software, this 3D shape is generally represented by a Brep (Boundary REPresentation) that describes the solid, surface, edge, point of the shape and its topological information (connection relationship information). Hereinafter, the 3D shape created by the above-mentioned 3D CAD software is called a CAD model. Generally, in Brep, it is possible to accurately represent curved surfaces such as spherical surfaces and the topology such as the connection of surfaces and edges.

[0005] Other methods for representing three-dimensional shapes include using meshes and point clouds. Meshes approximate and represent three-dimensional shapes as a collection of triangular patches, and are the most commonly used representation method for visualization purposes because rendering can be achieved quickly with dedicated hardware. On the other hand, meshes do not contain information about curved surfaces or topology.

[0006] A point cloud is a method of representing a shape as a collection of points that hold attribute information such as color. Point clouds are used for 3D shape measurement data using sensors, and because of their simple data structure, they are often used as input for machine learning to recognize 3D shapes. The recognition results of point clouds by machine learning models can also be used for searching for 3D shapes. On the other hand, since point clouds approximate shapes with sparse points, point cloud data does not contain information about surfaces or topology. Also, if you try to represent a detailed shape, the number of points required can increase exponentially.

[0007] As described above, while using approximate representations of three-dimensional shapes such as meshes and point clouds allows for efficient processing in specific applications, compared to three-dimensional representations using BREP, which are mainly used in manufacturing, there is a lack of topology information (patchwork (edge) information) and surface information, which can make them inefficient for design purposes in manufacturing.

[0008] There are two use cases for searching 3D shapes: one where you search for shape records stored in a database using text or shape queries, and another where you search for partial shapes, including those contained within larger shapes such as scenes or assemblies, using text or shape queries.

[0009] The former method involves, for example, converting the part shapes stored in each record of a database into point clouds, converting the shapes into vectors using a deep learning-based method such as PointNet, and then searching for similar shapes through vector search.

[0010] Patent Document 1 discloses a search method for partial shapes, which involves training a one-shot detection model using pairs of training 3D scenes converted into point clouds and objects to search for 3D query shapes including CAD models, and outputting boundary volumes and shape similarity scores by combining the representation descriptors of the query object and the scene.

[0011] Japanese Patent Publication No. 2022-183068

[0012] However, the technology described in Patent Document 1 suffers from a high computational load when there are a large number of scenes to be searched. Since it is necessary to perform neural network calculations for object detection in each scene, the computational load is proportional to the number of scenes, making searching computationally difficult as the number of scenes increases. The objective of the present invention is to enable high-speed 3D model searching even when the amount of data increases.

[0013] The present invention includes several means for solving at least some of the above problems, and an example thereof is as follows: A three-dimensional model search device according to one aspect of the present invention that solves the above problems includes: a search query receiving unit that receives a search query indicated by a three-dimensional shape including a boundary representation; a whole shape embedding unit that calculates a whole vector that covers the entire three-dimensional shape of the search query; a partial shape embedding unit that calculates partial vectors that cover partial shapes which are solid, face and edge units of the boundary representation; and a search unit that uses the whole vector and the partial vectors to search for parts with similar shapes as similar parts from predetermined part information by vector search.

[0014] According to the present invention, it becomes possible to perform high-speed 3D model searching even when the amount of data used to describe the shape in detail increases. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments for carrying out the invention.

[0015] This figure shows an example of the configuration of a design information retrieval system. This figure shows an example of the configuration of CAD shape information. This figure shows an example of the configuration of embedding information. This figure shows an example of a flowchart for the search process. This figure shows an example of a flowchart for the embedding calculation process. This figure shows an example of an image of the embedding calculation process. This figure shows an example of an image of the data used in the search process. This figure shows an example of a display screen for search results. This figure shows an example of the hardware configuration of a design information retrieval device.

[0016] Embodiments of the present invention will be described below with reference to the drawings. The embodiments are illustrative examples for explaining the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0017] The positions, sizes, shapes, and ranges of the components shown in the drawings may not represent their actual positions, sizes, shapes, and ranges in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, and ranges disclosed in the drawings.

[0018] Examples of various types of information may be described using expressions such as "table," "list," and "queue," but these types of information may also be represented by data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may be referred to as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable. Furthermore, the identification information described using these expressions may be represented using symbols, numbers, natural language, or combinations thereof in the embodiment, but the identification information may also be in other formats.

[0019] When there are multiple components with the same or similar function, they may be described using the same symbol but with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0020] In embodiments, processing performed by executing a program may be described. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs processing defined by the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The main entity performing the processing by executing the program may be an arithmetic unit, and may include a dedicated circuit that performs a specific processing. Here, a dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0021] The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in some embodiments, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0022] Furthermore, while the present invention is typically implemented by an information processing device, it may also be implemented as a platform having the functions of the present invention.

[0023] In this embodiment, the design information retrieval system 1 searches for parts with similar shapes as similar parts from predetermined part information using vector search.

[0024] The technology described in Patent Document 1 above performs processing after converting the 3D CAD model into a point cloud. Therefore, when the shape is large and complex, or when the part shape to be found is small relative to the entire model, the information may not be represented in the point cloud. In addition, point cloud-based methods such as PointNet are known to have lower accuracy than Brep-based methods such as UVNet in part type classification, and from the perspective of the accuracy of deep learning models, there are disadvantages to recognition targeting CAD models. In particular, it is not possible to directly present solids, faces, and edges created in 3D CAD as search results. In order to present them, it is necessary to separately compare which entity of the CAD model from which the point cloud was derived the search result relates. Also, since the output is a bounding volume, it is possible that neighboring parts with low correlation to the query shape may be included in the results.

[0025] Furthermore, the technology described in Patent Document 1 above creates a detection model using a dataset with pre-annotated shapes. In the case of creating a general shape search system, designing and annotating class labels is a costly process. Moreover, since it is often difficult to anticipate search queries, it is not easy to design appropriate class labels and prepare a corresponding search dataset.

[0026] Therefore, in order to solve the above problems, in the embodiment of the present invention, the 3D CAD model is not converted into a format such as a point cloud, and the similarity of the shape is evaluated by a deep learning model that takes geometric information consisting of parametric surface and curve information contained in Brep and topology information consisting of information such as the connections between faces and edges as input.

[0027] The design information retrieval system 1 can also achieve the following effects: By comparing similarities using Brep, information loss in topology information (patchwork (edge) information) and surface information is reduced compared to when using point clouds or meshes. Furthermore, even when the shape of the search target is complex or large, it is possible to accurately represent the shape with a relatively small data size, thus improving the accuracy of similarity judgments.

[0028] Because Brep retains information on solids, faces, and edges, it is possible to search for information on parts and solids included in the assembly, as well as information on small faces such as fillets, thus preventing detection omissions.

[0029] - Because it does not use detection models such as object classifiers or segmentation for searching, it does not require the existence of training data such as shape models with defined class labels and pairs of sub-shapes, allowing for the preparation of a search system at low cost. The cost of creating a shape search system can be reduced by utilizing learning methods that use unlabeled data, such as self-supervised learning using contrastive learning.

[0030] Furthermore, multimodal learning using comparative learning allows the search queries to be extended to other data formats, including text.

[0031] Furthermore, in the embodiment of the present invention, assuming the search for partial shapes from a large number of stored assembly CAD shapes, the embedding vector of the entire shape such as an assembly, and the embedding vector of the partial shape including the subassembly, part, solid, face, and edge to be searched are stored in a database, and vector indexes are constructed for each using approximate nearest neighbor method or the like. In addition, to enable consideration of the topology of the shape, a topology database is constructed that includes topology information such as the connection relationships of faces and edges of Brep, in association with the above database. Strictly speaking, in this embodiment, Brep can be any boundary representation (a method of representing a shape with multiple independent faces), and does not refer to any specific standard or rule. Also, in this embodiment, boundary representation may be referred to as Brep, or Brep may be referred to as boundary representation.

[0032] Then, when shape information is provided as a search query, the embedding vectors for the overall shape and the embedding vectors for the partial shape are calculated for the shape of the search query, and the above database is searched using the obtained overall and partial shape embedding vectors. The search for overall shape vectors and the search for partial shape vectors can be performed sequentially, by first narrowing down the search by searching for overall shape vectors and then searching for partial shape vectors. Alternatively, it is also possible to process the search for overall shape vectors and partial shape vectors in parallel.

[0033] Furthermore, in the embodiment of the present invention, the results of the search for the overall shape vector and the search for the partial shape vector are integrated to narrow down the candidate overall shapes that include more similar partial shapes, and then the search results are re-ranked using the similarity between the topology information. To determine the similarity between the topology information, the topology information such as faces and edges of parts that had high similarity as a result of the shape search is matched with the topology information of the shape of the search query, the instance segmentation of the search target shape in the overall shape and the similarity score of the topology information are calculated, and the search results are re-ranked based on that score. In the embodiment of the present invention, since the search process uses vector search, it is possible to process even if the amount of data to be searched increases.

[0034] Furthermore, when the design information retrieval system 1 according to this embodiment is applied to the manufacturing industry, the search method based on boundary representation and vector search enables highly accurate and high-speed shape retrieval of existing design information. In addition, the preparation of the search system is simplified, and it becomes possible to search using multimodal information including not only shape but also text as search queries. As a result, various man-hour reduction effects can be expected, mainly in the design process, such as promoting reuse of designs when designing 3D CAD parts, improving the accuracy of checking design shapes using recognition of shape features, and reducing man-hours in design, production design, and manual creation by reusing related information knowledge through shape retrieval.

[0035] Figure 1 shows an example of the configuration of a design information retrieval system. The design information retrieval system 1 is composed of an information processing device. The information processing device is a design information retrieval device 100, which includes a storage unit 110, a processing unit 120, an input unit 130, and a display unit 140. In particular, the design information retrieval device 100 can be described as a 3D model retrieval device.

[0036] First, the memory unit 110 stores CAD shape information 111, embedding information 112, and query history information 113. The processing unit 120 includes a search query receiving unit 121, an overall shape embedding unit 122, a partial shape embedding unit 123, a search unit 124, a reranking unit 125, and a screen generation unit 126.

[0037] Figure 2 shows an example of the configuration of CAD shape information. The CAD shape information 111 is the component configuration information of a product and includes a product ID 111a, a component ID 111b, shape information 111c, and topology information 111d. Specifically, the product ID 111a contains information that distinguishes the product from other products. The component ID 111b contains information that distinguishes the components that make up the product from other components. The shape information 111c stores a general CAD data file or reference information to a CAD data file as information that identifies the shape of the product. The topology information 111d is information that maps faces to CAD shapes as nodes and edges to CAD shapes as links.

[0038] Figure 3 shows an example of the structure of embedding information. The embedding information 112 is vectorized information of a product and includes a product ID 112a, a part ID 112b, a partial shape 112c, and vector information 112d. Specifically, if the vector information 112d is a vector corresponding to the entire product, the vector information 112d is associated with the product ID 112a; if the vector information 112d is a vector corresponding to a part that makes up the product, the vector information 112d is associated with the combination of product ID 112a and part ID 112b; and if the vector information 112d is a vector corresponding to a partial shape (solid, face, edge), the vector information 112d is associated with the partial shape 112c and stored in the embedding information 112. The vector information 112d is feature information of a product or part calculated by an existing predetermined method and is generally treated as a matrix. Furthermore, the vector information 112d is information that makes it possible to calculate similarity by determining the inter-vector distance such as the dot product. Therefore, the CAD shape information 111 and the embedding information 112 can be described as a database that stores each of the partial shapes, including faces, edges, and solids included in the three-dimensional shape, as records associated with the partial shape embedding.

[0039] The embedding information 112 is preferably a vector database, indexed to allow for high-speed retrieval of similar vectors using methods such as approximate nearest neighbor or locality-sensitive hashing.

[0040] The search query receiving unit 121 receives search queries that are represented by a three-dimensional shape including boundary representations. The search query receiving unit 121 also extracts topology information from the boundary representation information of the CAD shape of the received search query.

[0041] The overall shape embedding unit 122 calculates an overall vector that targets the entire three-dimensional shape of the search query.

[0042] The partial shape embedding unit 123 calculates partial vectors for the partial shape, which is the unit of solids, faces, and edges in the boundary representation.

[0043] The search unit 124 uses the overall vector and the partial vector to search for parts with similar shapes by vector search as similar parts from predetermined part information (for example, CAD shape information 111).

[0044] The rerank unit 125 performs matching between the search query and the partial graph of the topological graph of the Brep of the similar parts searched by the search unit 124, and reranks the search results by the search unit 124.

[0045] The screen generation unit 126 generates a screen that highlights the portions of the list of three-dimensional shapes of the similar parts searched by the search unit 124 that match the search query for each three-dimensional shape.

[0046] Alternatively, the search query reception unit 121 may receive an input of text instead of a three-dimensional shape as a search query, and the overall shape embedding unit 122 calculates an overall vector from the text received by the search query reception unit 121 using a learned model that is learned to output an overall vector when receiving the text as an input, and the partial shape embedding unit 123 calculates a partial vector from the text received by the search query reception unit 121 using a learned model that is learned to output a partial vector when receiving the text as an input.

[0047] Alternatively, the search query reception unit 121 may receive an input of text related to metadata including product ID, part ID, material, etc. as search conditions of the search query, and the search unit 124 may narrow down and search for parts with a high degree of relevance to the metadata related to the text from predetermined part information (for example, CAD shape information 111).

[0048] The input unit 130 receives various instructions from the operator. The input content includes a search query. Specifically, the input unit 130 receives a three-dimensional shape including Brep information and a search query input via an input device such as a mouse or keyboard. For example, as an input of the three-dimensional shape, the input unit 130 receives operations such as selection of a local file or a file on a server. However, it is not limited to this, and the input unit 130 can also receive inputs such as instructions and communications by a program via an API (Application Protocol Interface). Note that the 3D CAD model format includes intermediate formats such as STEP and Parasolid, or native formats, etc.

[0049] In addition, when receiving the shape of the search query, the input unit 130 may adopt either a method of designating an assembly or an entire part included in a file as the operator's input method, or a method of allowing the operator to select a plurality of sub-assemblies, solids, surfaces, or edges as partial shapes by operations such as clicking from the assembly or part.

[0050] The display unit 140 outputs (displays) a screen that highlights the matching parts of each three-dimensional shape with the search query for the list of three-dimensional shapes of similar parts searched by the search unit 124 and generated by the screen generation unit 126.

[0051] Furthermore, the display unit 140 displays the output via a display device such as various types of displays. For this reason, the input unit 130 and the display unit 140 may be integrated into a single unit, such as a touch panel. In addition, the input unit 130 and the display unit 140 may be configured in a separate housing from the design information retrieval device 100. In this case, the input unit 130 and the display unit 140 may be implemented by a separate terminal device, and the input information and output information may be transferred via a network. The network may be, for example, a LAN (Local Area Network), WAN (Wide Area Network), VPN (Virtual Private Network), a communication network that uses public telephone lines such as the Internet in part or in whole, a mobile phone communication network, or a network that is a combination of these. The network may also be a wireless communication network such as Wi-Fi (registered trademark) or 5G (Generation). Furthermore, the information processing devices constituting the design information retrieval system 1 are not limited to one unit, and there may be even more, including for purposes such as redundancy and load balancing. For example, a client terminal handling the search query reception unit 121, screen generation unit 126, input unit 130, and display unit 140, an embedding calculation server handling the overall shape embedding unit 122 and partial shape embedding unit 123, a search server handling the search unit 124 and rerank unit 125, and a database server holding CAD shape information 111, embedding information 112, and query history information 113 may be connected to each other via a network and cooperate to form the design information retrieval system 1. However, in this embodiment, for the sake of simplicity, it will be described as a single information processing device.

[0052] Figure 4 shows an example of a search process flowchart. The search process starts when the operator gives a start command via the input unit 130.

[0053] First, the search query receiving unit 121 receives CAD information including Brep information as a query (step S001). Specifically, the search query receiving unit 121 receives CAD information including Brep information for a part (sub-assembly, part, partial shape consisting of faces, edges, etc.) or the whole of the shape of the part to be searched as a search query. Alternatively, the search query receiving unit 121 accepts text input instead of a 3D shape as a search query. Or, the search query receiving unit 121 accepts text input related to metadata along with the 3D shape as a search query.

[0054] Then, the search query receiving unit 121 extracts topology information from the Brep information of the CAD shape of the received search query (step S002). Specifically, the search query receiving unit 121 extracts information that has been mapped to the CAD shape, with faces as nodes and edges as links, from the Brep information as topology information.

[0055] Then, the overall shape embedding unit 122 performs embedding calculation processing on the entire CAD shape of the search query (step S003). Specifically, the overall shape embedding unit 122 performs the embedding calculation processing described later on the entire three-dimensional shape of the search query.

[0056] Then, the partial shape embedding unit 123 performs embedding calculation processing for the CAD shape portion of the search query (step S004). Specifically, the partial shape embedding unit 123 targets the three-dimensional shape portion of the search query (sub-assemblies, parts, partial shapes consisting of faces, edges, etc.) and performs the embedding calculation processing described later. Note that the embedding calculation processing described later has already been performed in step S003, so it does not need to be performed again, and it is sufficient to identify the partial embedding.

[0057] Then, the search unit 124 performs matching processing with existing embedding information (step S005). Specifically, the search unit 124 uses the embedding information 112 relating to existing parts and the overall vector and partial vector of the search query to search and extract parts with similar shapes as similar parts from the CAD shape information 111 using vector search. For example, in the vector search, the search unit 124 searches for multiple (K items: K is a natural number) with high similarity indices, including cosine similarity, for each of the overall vector and partial vector of the search query.

[0058] The search unit 124 retrieves the top few (K1: K1 is a natural number) CAD models with similar overall vectors and their similarity scores for the overall vector of the search query. For the sub-vectors of the search query, it searches for faces and edges with similar vector information 112d registered in the embedding information 112 for each sub-vector corresponding to a subassembly, solid, face, edge, etc. of the shape of the search query, and retrieves the top few (K2: K2 is a natural number) similar subassemblies, solids, faces, edges, etc. and their similarity scores. Here, the search using sub-vectors targets all shapes included in the embedding information 112. However, if the search computation load is too high or the capacity of the search resources is insufficient, a sequential search targeting only the K1 shapes narrowed down by the overall vector search may be performed.

[0059] The search unit 124 then integrates the results of the matching process with existing embedding information to identify the top K similar parts. In this integration, the search unit 124 calculates a composite index of the similarity score of the search results for the overall vector and the similarity score of the search results for the sub-vectors. The composite index can be a linear sum of the similarity of the overall vector and the average of the similarity of the sub-shape vectors of the shapes with the highest similarity for each search query shape, or a simple weighted average. However, it is not limited to this, and the composite index may also be calculated using a neural network that combines linear transformations and nonlinear functions with the similarity of the overall vector and sub-vectors as input.

[0060] Then, the reranking unit 125 searches for CAD shape information 111 for the top matching parts (K pieces) and obtains topology information 111d (step S006).

[0061] Then, the reranking unit 125 performs a reranking (sorting) process on the top-ranking parts (K parts) in order of topological similarity, using the embedding information as the basis for calculation (step S007). Specifically, in order to compare topological information, the reranking unit 125 describes the shape of Brep as a face adjacency graph with faces as nodes and edges as edges, including the topological information, and then calculates the similarity of the subvectors by determining the node matching rate and the edge matching rate. The reranking unit 125 then calculates the topological similarity by quantifying the degree of agreement between the shape of the search query and the nodes and edges of the face adjacency graph of the subshapes of the CAD models of the top-ranking parts.

[0062] The reranking unit 125 uses a method of quantifying the similarity between graphs by performing subgraph matching for matching face-adjacent graphs, but is not limited to this. For example, the reranking unit 125 can employ a method of matching nodes and edges using a greedy algorithm or Hungarian algorithm as an optimal transport problem and evaluating the degree of match. Alternatively, the reranking unit 125 can employ a method of evaluating graph similarity using a graph neural network. Furthermore, when comparing topologies, it is possible to exclude face nodes that have a small contribution to the overall shape, such as fillets and chamfers, from the comparison by removing them from both the shape of the search query and the top-ranking matching components, or by reducing their weight during evaluation, thereby evaluating without giving much weight to differences in fillets and chamfers.

[0063] The reranking unit 125 then obtains a score of similarity between the shape of the search query, which includes topology information, and the CAD partial shape of an existing part, and reranks the search results based on this score.

[0064] Then, the screen generation unit 126 generates screen information to present the top K results of the comparison with the search query based on the reranking results (step S008). Specifically, the screen generation unit 126 receives a list of partial shapes corresponding to the shapes of the CAD models of the reranked search results and the search queries from the reranking unit 125, and processes them in accordance with the screen layout information to be displayed.

[0065] Then, the display unit 140 displays the screen generated by the screen generation unit 126 (step S009).

[0066] Then, if a registration instruction is given, the search unit 124 stores the CAD information of the search query as a new part in the CAD shape information 111 (step S010).

[0067] The above is an example of a search process flowchart. According to this example, it becomes possible to perform high-speed 3D model searches even when the amount of data used to describe the shape in detail increases.

[0068] Figure 5 is an example of a flowchart for the embedding calculation process. The embedding calculation process starts in step S003 of the search process.

[0069] First, the overall shape embedding unit 122 receives CAD information including Brep information as the target shape (step S101). The target shape refers to the shape for which embedding is calculated. Therefore, it does not matter whether it is the shape of the search query or the shape of a part included in the CAD shape information 111.

[0070] Then, the overall shape embedding unit 122 extracts geometric and topological information contained in the Brep information of the received CAD information and generates graph structure data (step S102). Specifically, the overall shape embedding unit 122 forms a surface adjacency graph from the extracted topological information, with Brep faces as nodes and edges as edges, and generates graph structure data by creating and linking feature quantities from information such as surface area, curvature, type, and coordinates obtained from the extracted geometric information, as well as information such as edge length, concavity, and orientation, to each node and edge.

[0071] Furthermore, in addition to face-adjacent graphs, graph structure data that represents information including edge loops, solids, and assembly trees as graphs may also be adopted, resulting in heterogeneous graphs. By constructing a graph neural network that takes such graph structure data as input and training it using learning methods such as symmetric learning with an InfoNCE loss function, it becomes possible to obtain vectors (corresponding to subvectors) for each node and edge from this graph structure data, as well as a neural network that calculates graph embedding (corresponding to the overall vector) using graph pooling or Set2Set. Possible graph neural networks include message-passing neural networks that can utilize edge information, and graph convolutional networks targeting heterogeneous graphs.

[0072] Furthermore, regarding graph embedding, it is possible to combine text and image metadata not included in the Brep information, encoded into vectors, to create an embedding vector. Additionally, during comparative learning, it is possible to train a network that obtains embedding from text information instead of graphs by replacing the image encoder of CLIP (Contrastive Language-Image Pre-training), a machine learning model that processes images and text simultaneously, with this graph neural network, and training both text and graph information together. In this way, even when the search query contains only text information, embedding can be calculated, enabling shape search processing.

[0073] Then, the overall shape embedding unit 122 calculates the overall vector information of the target shape using the graph structure data (step S103). Specifically, the overall shape embedding unit 122 obtains the overall vector using the entire graph structure data generated in step S102. In this process, if the aforementioned graph neural network, which takes graph structure data represented as a heterogeneous graph as input, is used, it is also possible to calculate the overall vector using graph pooling or Set2Set.

[0074] Then, the partial shape embedding unit 123 calculates partial vector information for the solids, faces, and edges that constitute Brep using the graph structure data (step S104). Specifically, for the solids, faces, and edges included in the graph structure data generated in step S102, the partial shape embedding unit 123 uses the feature vectors of the nodes and edges as the corresponding partial shape embeddings for the faces and edges, and obtains vectors using the graph embedding for each connected component of the graph as the partial shape embedding for the solid. In this process, it is possible to calculate partial vectors by using the aforementioned graph neural network, which takes graph structure data represented as a heterogeneous graph as input.

[0075] Then, the overall shape embedding unit 122 stores the vector information of the target shape as query history in the query history information 113, or as embedding information in the embedding information 112 if the target shape is not a search query (step S105).

[0076] The above is an example of a flowchart for embedding calculation. According to this example of embedding calculation, it is possible to obtain overall vector information and partial vector information using CAD information including Brep information.

[0077] The neural networks used to calculate overall shape embedding and partial shape embedding may be the same, or specialized networks may be used for each.

[0078] Figure 6 shows an example of the embedding calculation process. In the embedding calculation process 200, the input CAD shape or text information is encoded using a shape encoder to obtain the overall embedding. In addition, by encoding using a partial shape encoder, partial embedding can be obtained for each part of the input CAD shape or text information, such as faces and edges.

[0079] Figure 7 shows an example of the data used in the search process. As example 300, we assume a pair 301 of a search query shape consisting of two faces and a CAD shape with a similar sub-shape. The overall embedding 302 represents each of these shapes as a single multidimensional vector, and can be compared using cosine similarity, etc.

[0080] Partial embedding 303 is represented by a graph in which faces are nodes and edges are edges for each pair 301 of the shape of the search query and the CAD shape, and then a vector is obtained for each of these nodes and edges to form a partial vector. In Example 300, nodes that are judged to be similar as a result of comparing the partial vectors of the nodes are connected by arrows.

[0081] The whole / partial comparison 304 further incorporates topology information, comparing the entire graph, which includes topology information of the search query shape, with the information of a subgraph containing multiple nodes and edges similar in CAD shape. The matching process in step S005 of the search process is realized by comparing the whole embedding 302 and the partial embedding 303. Furthermore, the reranking process in step S007 is realized by determining the similarity of the topologies through the subgraph-level comparison in the whole / partial comparison 304.

[0082] Figure 8 shows an example of a search results display screen. Example 400 of the search results display screen includes an input area 401 for specifying the type of search query, a query input area 402 for specifying or uploading a shape, an input field 403 for additional information to narrow down the search target, and a search button 404 for receiving a command to start the search process. If the type of search query in input area 401 is set to "shape", then the query input area 402 can be used to specify and input a 3D shape or its CAD data including Brep information from a file or by referencing. If the type of search query in input area 401 is set to "text", then the query input area 402 can be used to input free text.

[0083] The additional information input field 403 accepts metadata such as material name, name, and product name. Upon receiving text related to the metadata, the search unit 124 narrows down the CAD shape information 111 to parts with a high degree of relevance to the metadata related to the text and performs a search.

[0084] When input is received by the search button 404, the search process is executed, and the screen generated in step S009 is output to the results display 410. In the example in Figure 8, the results display 410 shows, as an example of the search results, the CAD shape name, a preview of the CAD shape with the matched partial shapes highlighted (highlighted) according to the partial shape similarity, the similarity score, and the number of similar partial shapes. Clicking on the highlighted part in the preview allows you to view the corresponding partial shape similarity value. This partial shape similarity is calculated when reranking is performed in step S007 of the search process.

[0085] [Explanation of Hardware Configuration] Figure 9 shows an example of the hardware configuration of the information processing device that constitutes the design information retrieval system. The information processing device 900 that constitutes the design information retrieval system 1 includes a processor (for example, CPU: Central Processing Unit, or GPU: Graphics Processing Unit) 901, hardware memory 902 such as RAM (Random Access Memory), storage 903 such as a hard disk drive (HDD) or SSD (Solid State Drive), and CD (Compact Disk) or DVD (Digital Versatile). This can be implemented as a general-purpose computer or a network system comprising multiple such computers, which includes a reader 905 that reads information from a portable storage medium 904 such as a disk, an input device 906 such as a keyboard, mouse, barcode reader, or touch panel, an output device 907 such as a display, and a communication device 908 that communicates with other computers via a communication network such as a LAN or the Internet. The reader 905 may be capable of not only reading but also writing to the portable storage medium.

[0086] The processor 901 performs various processes by executing predetermined programs loaded from the storage 903 into the memory 902. These programs are, for example, application programs that can be executed on an OS (Operating System) program. These programs may be installed in the storage 903 from a portable storage medium 904 via a reader 905, or they may be downloaded from a network via a communication device 908 and executed by the processor 901.

[0087] For example, the processing unit 120, search query receiving unit 121, overall shape embedding unit 122, partial shape embedding unit 123, search unit 124, reranking unit 125, and screen generation unit 126 can be implemented by loading a program stored in storage 903 into memory 902 and executing it with processor 901. The input unit 130 and display unit 140 can be implemented by processor 901 utilizing input device 906, output device 907, and communication device 908. The storage unit 110 can be implemented by processor 901 utilizing memory 902 or storage 903.

[0088] The above describes the design information retrieval system 1 according to the embodiment. The design information retrieval system 1 enables high-speed 3D model retrieval even when the amount of data increases. The present invention is not limited to the above-described embodiment and includes various modifications. For example, the above-described embodiment is described in detail to explain the present invention in an easy-to-understand manner and is not necessarily limited to having all the described configurations. It is possible to replace some of the configurations of the embodiment with other configurations, and it is also possible to add configurations of other embodiments to the configuration of the embodiment. Furthermore, it is possible to delete some of the configurations of the embodiment.

[0089] Each of the above-mentioned parts, configurations, functions, and processing units may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above-mentioned parts, configurations, and functions may be implemented in software by having the processor interpret and execute programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in memory, a recording device such as a hard disk, or a recording medium such as an IC card, SD card, or DVD.

[0090] It should be noted that the control lines and information lines in the embodiments described above are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. In practice, it can be assumed that almost all components are interconnected. The present invention has now been described, focusing on its embodiments.

[0091] 1: Design information retrieval system, 100: Design information retrieval device, 110: Storage unit, 111: CAD shape information, 112: Embedding information, 113: Query history information, 120: Processing unit, 121: Search query reception unit, 122: Overall shape embedding unit, 123: Partial shape embedding unit, 124: Search unit, 125: Rerank unit, 126: Screen generation unit, 130: Input unit, 140: Display unit.

Claims

1. A three-dimensional model search device comprising: a search query receiving unit that receives a search query indicated by a three-dimensional shape including boundary representations; a whole shape embedding unit that calculates a whole vector for the entire three-dimensional shape of the search query; a partial shape embedding unit that calculates partial vectors for partial shapes which are solid, face, and edge units of the boundary representation; and a search unit that uses the whole vector and the partial vectors to search for parts with similar shapes as similar parts from predetermined part information by vector search.

2. A three-dimensional model search device according to claim 1, comprising: a reranking unit that performs matching between the sub-vector and the sub-graph of the topology graph of the boundary representation between the search query and the similar parts found by the search unit, and reranks the search results from the search unit.

3. A three-dimensional model search device according to claim 1, comprising: a screen generation unit that generates a screen that highlights the portion of each three-dimensional shape that matches the search query in the list of three-dimensional shapes of similar parts found by the search unit.

4. A three-dimensional model search device according to claim 1, characterized in that: the search query receiving unit accepts text input instead of the three-dimensional shape; the overall shape embedding unit calculates the overall vector from the text received by the search query receiving unit using a model trained to output the overall vector when text is received as input; and the partial shape embedding unit calculates the partial vector from the text received by the search query receiving unit using a model trained to output the partial vector when text is received as input.

5. A three-dimensional model search device according to claim 1, characterized in that the predetermined part information includes a database that stores each of the partial shapes, including faces, edges, and solids included in the three-dimensional shape, as records associated with partial shape embedding.

6. A three-dimensional model search device according to claim 1, wherein the search query receiving unit receives input of text related to metadata including a product name or material as a search condition, and the search unit performs the search by narrowing down the parts from predetermined part information to parts with a high degree of relevance to the metadata related to the text.

7. A method for searching a three-dimensional model using an information processing device, wherein the information processing device comprises a storage unit for storing predetermined part information and a processing unit, and the processing unit performs: a search query reception step for receiving a search query indicated by a three-dimensional shape including a boundary representation; a whole shape embedding step for calculating a whole vector for the entire three-dimensional shape of the search query; a partial shape embedding step for calculating partial vectors for partial shapes which are solid, face, and edge units of the boundary representation; and a search step for searching for parts with similar shapes as similar parts from predetermined part information by vector search using the whole vector and the partial vectors.

Images