Multi-mode-based model segmentation method and segmentation model training method

Abstract

The invention provides a multimodal-based model segmentation method and a segmentation model training method, and the method comprises the steps: obtaining a part model, and generating a graph embedding representation of the part model based on the part model; obtaining prompt information, and generating embedded representation of the prompt information based on the prompt information; and based on the embedded representation of the prompt information and the graph embedded representation of the component model, segmenting the component model by using the segmentation model to obtain a target surface adjacent sub-graph of the component model. According to the technical scheme, model segmentation can be carried out based on the multi-modal data, the segmentation result oriented to part semantics is obtained, the segmentation result is more accurate, and the segmentation efficiency is higher.

Description

Technical Field

[0001] The present application relates to the field of model segmentation technology, and specifically to a multimodal based model segmentation method and a segmentation model training method. Background Art

[0002] Segmentation technology aims to decompose complex Computer Aided Design (CAD) models into smaller and more meaningful parts for further analysis, processing or application. Boundary Representation (B-Rep) model is the main method to describe complex three-dimensional geometric shapes and is widely used in CAD and other fields. B-Rep model segmentation has important application value in improving product design quality, shortening development cycle and reducing manufacturing cost. However, there are still some problems in segmentation of B-Rep model.

[0003] The most representative CAD model segmentation model for B-Rep is BRep-Net. Its segmentation result is essentially to classify the model surface (such as plane, groove surface, etc.), and classify the patches in different areas into specific engineering prior categories. The result is usually discontinuous, and this method cannot effectively identify and segment the engineering semantics formed by continuous part surfaces. Summary of the invention

[0004] In order to solve the above problems, the embodiments of the present application provide a multimodal model segmentation method and a segmentation model training method.

[0005] In a first aspect, an embodiment of the present application provides a multimodal model segmentation method, comprising: obtaining a component model, and generating a graph embedding representation of the component model based on the component model; obtaining prompt information, and generating an embedding representation of the prompt information based on the prompt information; based on the embedding representation of the prompt information and the graph embedding representation of the component model, using a segmentation model to segment the component model to obtain a target surface adjacency subgraph of the component model.

[0006] In combination with the first aspect, a graph embedding representation of the component model is generated based on the component model, including: constructing a topological structure and extracting geometric features of the component model to obtain a face adjacency graph of the component model; and using a neural network to process the face adjacency graph to obtain a graph embedding representation of the component model.

[0007] In combination with the first aspect, the prompt information includes text information and / or image information, and generating an embedded representation of the prompt information based on the prompt information includes: inputting the text information and / or image information into an embedded representation generation model to obtain an embedded representation of the text information and / or an embedded representation of the image information.

[0008] In a second aspect, an embodiment of the present application provides a segmentation model training method, comprising: obtaining an initial segmentation model; obtaining training samples, the training samples comprising multiple sample component models; based on each sample component model, determining a graph embedding representation of the sample component model; based on each sample component model, determining a view embedding representation of multiple substructures of the sample component model; based on the graph embedding representation of the sample component model and the view embedding representation of the multiple substructures of the corresponding sample component model, training the initial segmentation model to obtain a segmentation model, the segmentation model being used to segment the above-mentioned component model.

[0009] In combination with the second aspect, the graph embedding representation of the sample component model and the view embedding representation of multiple substructures corresponding to the sample component model are used to train the initial segmentation model to obtain the segmentation model, including: based on the graph embedding representation of the sample component model and the view embedding representation of multiple substructures corresponding to the sample component model, using the initial segmentation model to determine multiple face adjacency subgraphs; obtaining multiple predicted substructures corresponding to the multiple face adjacency subgraphs; determining the view embedding representation of the multiple predicted substructures; based on the view embedding representation of the multiple substructures and the view embedding representation of the multiple predicted substructures, training the initial segmentation model to obtain the segmentation model.

[0010] In combination with the second aspect, determining view embedding representations of multiple predicted substructures includes: adjusting views of the predicted substructures and determining rendering views of the predicted substructures; and determining view embedding representations of the predicted substructures based on the rendering views of the predicted substructures.

[0011] In combination with the second aspect, based on the view embedding representations of multiple substructures and the view embedding representations of multiple predicted substructures, an initial segmentation model is trained to obtain a segmentation model, including: calculating a loss function based on the view embedding representations of multiple substructures and the view embedding representations of multiple predicted substructures; based on the loss function, adjusting model parameters by a gradient descent method until convergence to obtain a segmentation model.

[0012] In combination with the second aspect, based on each sample component model, a graph embedding representation of the sample component model is generated, including: constructing the topological structure and extracting the geometric features of the sample component model to obtain the face adjacency graph of the sample component model; using a neural network to process the face adjacency graph to obtain the graph embedding representation of the sample component model.

[0013] In combination with the second aspect, based on each sample component model, the view embedding representation of multiple substructures of the sample component model is determined, including: based on the sample component model, determining multiple substructures of the sample component model; determining rendering views of the multiple substructures; and determining view embedding representations of the multiple substructures based on the rendering views of the multiple substructures.

[0014] In combination with the second aspect, based on the sample component model, multiple substructures of the sample component model are determined, including: parsing the sample component model to determine the current shape of the sample component model; if the current shape of the sample component model is an assembly, determining all subshapes of the current shape, and determining the subshapes as substructures; if the current shape of the sample component model is not an assembly, determining the current shape as a substructure.

[0015] In the third aspect, an embodiment of the present application provides a multimodal model segmentation device, including a generation module and a segmentation module; the generation module is used to obtain a component model, and generate a graph embedding representation of the component model based on the component model; the generation module is also used to obtain prompt information, and generate an embedding representation of the prompt information based on the prompt information; the segmentation module is used to segment the component model using a segmentation model based on the embedding representation of the prompt information and the graph embedding representation of the component model, to obtain a target surface adjacency subgraph of the component model.

[0016] In a fourth aspect, an embodiment of the present application provides a segmentation model training device, comprising an acquisition module, a determination module and a training module; the acquisition module is used to acquire an initial segmentation model; the acquisition module is also used to acquire training samples, and the training samples include multiple sample component models; the determination module is used to determine the graph embedding representation of the sample component model based on each sample component model; the determination module is also used to determine the view embedding representation of multiple substructures of the sample component model based on each sample component model; the training module is used to train the initial segmentation model based on the graph embedding representation of the sample component model and the view embedding representation of the multiple substructures of the corresponding sample component model to obtain a segmentation model, and the segmentation model is used to segment the above-mentioned component model.

[0017] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory, the memory being connected to the processor, the memory being used to store a computer program, and when the computer program is executed by the processor, the above-mentioned multimodal-based model segmentation method or the above-mentioned segmentation model training method is implemented.

[0018] In a sixth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned multimodal-based model segmentation method or the above-mentioned segmentation model training method is implemented.

[0019] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when executed by a processor, implement the above-mentioned multimodal-based model segmentation method or the above-mentioned segmentation model training method.

[0020] Through the above technical solution, when segmenting a component model (for example, a component B-Rep model), the component model is segmented in combination with prompt information (for example, text information, image information), that is, the model is segmented based on multimodal data to obtain a segmentation result oriented to the part semantics (that is, a face adjacency subgraph that characterizes the surface of a continuous part). On the one hand, multimodal data can provide multi-faceted data, making the segmentation result more accurate and the segmentation efficiency higher; on the other hand, it can improve the flexibility of model segmentation. In addition, when training the segmentation model, the method is self-supervised. The method can mine the intrinsic hierarchical information of the sample component model, obtain the substructure of the sample component model, and form a "pseudo-label" implementation. It does not need to provide the true value of the segmentation result label, which alleviates the problem of severe shortage of B-Rep model segmentation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of a model segmentation method provided in one embodiment of the present application.

[0022] Figure 2 It is a flowchart of a method for generating a graph embedding representation of a component model provided in one embodiment of the present application.

[0023] Figure 3 It is a schematic diagram of the process of segmenting a component model provided by an embodiment of the present application.

[0024] Figure 4 It is a flowchart of a model segmentation method provided in one embodiment of the present application.

[0025] Figure 5 It is a flowchart of a method for training an initial segmentation model provided in an embodiment of the present application.

[0026] Figure 6 It is a structural diagram of an initial segmentation model provided in an embodiment of the present application.

[0027] Figure 7 It is a schematic diagram of the process of training a segmentation model provided in one embodiment of the present application.

[0028] Figure 8 It is a structural block diagram of a model segmentation device provided by an exemplary embodiment of the present application.

[0029] Fig. 9 It is a structural diagram of a segmentation model training device provided in one embodiment of the present application.

[0030] Fig.10 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In the first aspect, the embodiment of the present application provides a model segmentation method based on multimodality, including: obtaining a component model, and generating a graph embedding representation of the component model based on the component model; obtaining prompt information, and generating an embedding representation of the prompt information based on the prompt information; based on the embedding representation of the prompt information and the graph embedding representation of the component model, using a segmentation model to segment the component model to obtain a target face adjacency subgraph of the component model. In the embodiment of the present application, the component model refers to a model used to describe the three-dimensional geometric shape of the component in computer-aided design. Here, the component includes an assembly composed of a part or a plurality of parts. The graph embedding representation of the component model refers to the embedding representation of the face adjacency graph of the component model. Among them, the embedding representation of the face adjacency graph is a set of vectors that can be recognized by the segmentation model. The face adjacency graph (FAG) is a model used to describe the adjacency relationship between three-dimensional entities or two-dimensional planar geographic elements in spatial data structures and graphical representations. The face adjacency graph is a graph structure in which each node represents a face and each edge represents the adjacency relationship between two faces. A face adjacency subgraph (FAS) is a subset of a face adjacency graph that retains some nodes and edges in the original face adjacency graph that are related to a specific task or analysis. A face adjacency subgraph is a portion extracted from a face adjacency graph that focuses on a specific region or feature, so as to better understand and / or analyze the spatial relationship and attribute characteristics between specific regions or features. In an embodiment of the present application, when a component model (e.g., a component B-Rep model) is segmented, the component model is segmented in combination with prompt information (e.g., text information, image information), that is, model segmentation is performed based on multimodal data to obtain a segmentation result oriented to part semantics (i.e., a face adjacency subgraph that characterizes a continuous part surface). On the one hand, multimodal data can provide multi-faceted data, making the segmentation result more accurate and the segmentation efficiency higher; on the other hand, it can improve the flexibility of model segmentation.

[0033] Figure 1 FIG. 1 is a flow chart of a model segmentation method provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps.

[0034] Step S110: acquiring a component model, and generating a graph embedding representation of the component model based on the component model.

[0035] In the embodiment of the present application, the component model may include a part model, or a model of an assembly composed of multiple parts. A component model refers to a model used to describe the three-dimensional geometric shape of a component in computer-aided design. The component model in the embodiment of the present application is a B-Rep model.

[0036] In an embodiment of the present application, generating a graph embedding representation of a component model includes: constructing a topological structure and extracting geometric features of the component model to obtain a face adjacency graph of the component model; processing the face adjacency graph using a neural network to obtain a graph embedding representation of the component model (i.e., an embedding representation of the face adjacency graph). The graph embedding representation includes a set of vectors that can be recognized and processed by the segmentation model. For more description of generating a graph embedding representation of a component model, please refer to the present application. Figure 2 The description in will not be repeated here.

[0037] Step S120: Obtain prompt information, and generate an embedded representation of the prompt information based on the prompt information.

[0038] In an embodiment of the present application, the prompt information includes text information and / or image information. Optionally, the text information and / or image information is input into an embedding representation generation model to obtain an embedding representation of the text information and / or an embedding representation of the image information, for example, an embedding representation of the text information and / or an embedding representation of the image information in the same feature space is obtained. The embedding representation generation model includes a contrastive language-image pre-training model (Contrastive Language-Image Pre-training, CLIP), or other embedding representation generation models. Specifically, if the prompt information is text information, the embedding representation generation model can be a CLIP Transformer-based Text Encoder; if the prompt information is image information, the embedding representation generation model can be a CLIP ViT-B / 16. The embedded representation of the prompt information includes a set of vectors that can be recognized and processed by a segmentation model.

[0039] Step S130 , based on the embedded representation of the prompt information and the graph embedded representation of the component model, the component model is segmented using a segmentation model to obtain a target surface adjacent subgraph of the component model.

[0040] The embedded representation of the prompt information and the graph embedded representation of the component model are input into the segmentation model to output the target surface adjacent subgraph. In an embodiment of the present application, the segmentation result includes multiple target surface adjacent subgraphs. The segmentation model in the embodiment of the present application includes Segment Anything Model (SAM). In an embodiment of the present application, the SAM architecture is extended to adapt the B-Rep model through training. For a description of the training method of the segmentation model, please refer to the present application Figure 4 The description in will not be repeated here.

[0041] In an embodiment of the present application, when a component model (for example, a component B-Rep model) is segmented, the component model is segmented in combination with prompt information (for example, text information, image information), that is, model segmentation is performed based on multimodal data to obtain a segmentation result oriented to part semantics (that is, a face adjacency subgraph that characterizes a continuous part surface). On the one hand, multimodal data can provide multi-faceted data to make the segmentation result more accurate and the segmentation efficiency higher; on the other hand, it can improve the flexibility of model segmentation.

[0042] Figure 2 FIG. 1 is a flow chart of a method for generating a graph embedding representation of a component model provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps.

[0043] Step S210 , constructing a topological structure and extracting geometric features of the component model to obtain a face adjacency graph of the component model.

[0044] First, the component model is topologically constructed. The component model in the embodiment of the present application is a B-Rep model. For each face, by traversing each half-edge of the current face; for each half-edge, find its dual half-edge; according to the face connected by the dual half-edge, determine the connection relationship between the current face and its adjacent faces, and construct a face adjacency graph, that is: current face → half-edge → dual half-edge → adjacent face, and obtain a B-Rep face adjacency graph G(V,E), where the vertex V represents the face in the B-rep, and the edge E encodes the connectivity between the faces.

[0045] Secondly, geometric features are extracted from the component model, including:

[0046] Surface geometric feature extraction. The surface geometric feature information is constructed using a UV grid sampling method. Specifically, after the surface is parameterized, the current two-dimensional parameter domain interval [u min ,u max ]×[v min ,v max ] to the geometry domain The mapping of (i.e., the parameter surface S(u,v)), where [u min ,u max ]× represents a two-dimensional real number. Then, for the current two-dimensional parameter domain interval [u min ,u max ]×[v min ,v max ] respectively perform uniform sampling in two dimensions, so that the two-dimensional parameter domain interval [u min ,u max ]×[vmin ,v max ] is discretized into a two-dimensional grid, multiple discrete points in front of the current grid are obtained, and the sampling step size is determined: and Among them, M init -1, N init -1 indicates the number of sampling steps in each dimension, M init -1 and N init -1 can be the same or different. For each discrete point, calculate the local features of the discrete point, that is: the three-dimensional absolute coordinates of the discrete point (u i ,v j ), the unit normal vector of the discrete point pointing to the outside And the pruning mask, with 1 and 0 representing the sampling points in the visible area and the pruning area respectively; according to the local features of the discrete points, they are spliced ​​according to the channels to encode the surface geometric features.

[0047] Edge geometry feature extraction. The U grid sampling method is used to construct edge geometry feature information. Specifically, after edge parameterization, the parameter domain interval of the current edge is realized. min ,u max ] to the geometry domain The mapping of is (parameter curve C(u)), where represents a one-dimensional real number. The parameterized current line is uniformly sampled so that the parameter domain interval [u min ,u max ] discretize to obtain multiple discrete points of the current edge. For each discrete point, calculate the local feature of the discrete point, that is, the three-dimensional absolute point coordinates C(u i ) and the unit tangent vector C′ u (u i ). According to the local features of discrete points, they are concatenated channel by channel to encode edge geometric features.

[0048] Step S220, using a neural network to process the face adjacency graph to obtain a graph embedding representation of the component model.

[0049] In an embodiment of the present application, the neural network includes UV-Net. UV-Net is a neural network architecture and representation that can directly operate the boundary representation (B-Rep) data of the CAD model. The topological structure and geometric features of the face adjacency graph are input into UV-Net to obtain a graph embedding representation of the component model, that is, an embedding representation of the face adjacency graph.

[0050] Figure 3 Schematic diagram of the process of segmenting a component model provided by an embodiment of the present application. Figure 3As shown, the process includes: UV parameterization of the component model (i.e., B-Rep model) to obtain a face adjacency graph; then, the face adjacency graph is processed using a neural network (UV-Net) to obtain a graph embedding representation of the component model. The process also includes: obtaining text information and / or image information. It can be understood that text information and image information can be obtained at the same time, or only text information can be obtained, or only image information can be obtained. After obtaining the text information, CLIP Transformed-Based Text Encoder is used to generate an embedding representation of the text information (i.e., text embedding); after obtaining the image information, CLIP ViT-B / 16 is used to generate an embedding representation of the image information (i.e., view embedding). The text embedding and / or view embedding, as well as the graph embedding of the component model are input into the segmentation model (i.e., SA-BR model) to obtain a face adjacency subgraph.

[0051] In a second aspect, the present application provides a segmentation model training method, which trains an initial segmentation model, optimizes model parameters, and obtains a segmentation model, so that the trained segmentation model is adapted to the segmentation of the B-Rep model. The trained segmentation model is used in the above method to segment the component model.

[0052] Figure 4 FIG. 1 is a flow chart of a model segmentation method provided in an embodiment of the present application. Figure 4 As shown, the method includes the following steps.

[0053] Step S410, obtaining an initial segmentation model.

[0054] In an embodiment of the present application, the segmentation model includes SAM, and SAM includes a feature fusion module and a fused feature decoding module. Optionally, during the training process, the feature fusion module performs four steps: calculating the self-attention of the view embedding representation to update the view embedding representation (i.e., the view embedding vector of each substructure); calculating the cross-attention from the view embedding representation to the graph embedding representation to update the view embedding representation; updating the view embedding representation using a point-by-point multi-layer perceptron (MLP); calculating the cross-attention from the graph embedding representation to the view embedding representation to update the graph embedding representation (i.e., the embedding vector of each face). The fused feature decoding module performs three steps: calculating the cross-attention from the view embedding representation to the graph embedding representation to obtain the final view embedding representation, inputting the final view embedding representation into the point-by-point multi-layer perceptron to obtain a node segmentation mask; for each mask, extracting the mask node set and index set, and constructing the subgraph edge set.

[0055] Step S420, obtaining training samples.

[0056] Optionally, the training sample includes a plurality of sample component models. The sample component model includes a part model, an assembly model, etc. In the embodiment of the present application, the sample component model is a B-Rep model.

[0057] Step S430: Based on each sample component model, determine the graph embedding representation of the sample component model.

[0058] In the embodiment of the present application, the topological structure of the sample component model is constructed and the geometric features are extracted to obtain the face adjacency graph of the sample component model; the face adjacency graph is processed by a neural network to obtain a graph embedding representation of the sample component model. For example, the graph embedding representation of the sample component model is: in, represents the embedding vector of the jth face, and |V| represents the number of faces of the sample component model. For the method of determining the graph embedding representation of the sample component model, please refer to the method of determining the graph embedding representation of the component model (i.e. Figure 2 ), which will not be described here.

[0059] Step S440: Based on each sample component model, determine the view embedding representation of multiple substructures of the sample component model. In the present application, the view embedding representation refers to the embedding representation of the views of multiple substructures, which includes a set of vectors identifiable by the segmentation model.

[0060] In an embodiment of the present application, first, based on the sample component model, multiple substructures of the sample component model are determined. Specifically, the sample component model is parsed to determine the current shape of the sample component model. For example, the ShapeTool tool of OpenCasCade is used to parse the model to determine the current shape of the sample component model. If the current shape of the sample component model is an assembly, all subshapes of the current shape are determined, and the subshapes are determined as substructures. Among them, the subshapes may include direct subshapes, or recursively divided multi-level subshapes. If the current shape of the sample component model is not an assembly, the current shape is directly determined as a substructure.

[0061] Then, the rendering views of the multiple substructures are determined. Specifically, for each substructure, the view is adjusted to the substructure, and the views in the x, y, and z directions are used as the rendering views of the substructure. For example, using the FreeCAD Python macro to adjust the view to the substructure, the views in the x, y, and z directions are used as the rendering views of the substructure {I x ,I y ,I z}.

[0062] Finally, based on the rendered views of the multiple substructures, view embedding representations of the multiple substructures are determined.

[0063] For any substructure rendered view I i , extract the view average embedding vector using an embedding representation generation model (e.g., CLIP ViT B / 16, CLIP ViT B / 32) in They are the view embedding vectors in the x, y, and z directions respectively. Then we get the view embedding representation of all substructures in represents the embedding vector of the i-th substructure, and |S| represents the number of substructures.

[0064] Step S450: Based on the graph embedding representation of the sample component model and the view embedding representation of the multiple substructures corresponding to the sample component model, an initial segmentation model is trained to obtain a segmentation model.

[0065] Optionally, based on the graph embedding representation of the sample component model and the view embedding representation of the multiple substructures corresponding to the sample component model, the initial segmentation model is used to determine multiple face adjacency subgraphs; multiple predicted substructures corresponding to the multiple face adjacency subgraphs are obtained; the view embedding representations of the multiple predicted substructures are determined; based on the view embedding representations of the multiple substructures and the view embedding representations of the multiple predicted substructures, the initial segmentation model is trained to obtain a segmentation model. For more descriptions on training the initial segmentation model, please refer to this application Figure 5 , I will not go into details here.

[0066] In the embodiment of the present application, the sample component model is parsed, the internal hierarchical information of the sample component model (B-Rep model) is mined, the substructure of the sample component model is determined, and a "pseudo-label" is implemented. In addition, the initial segmentation model is trained using the views of multiple substructures of the sample component model, and there is no need to provide the true value of the segmentation result label, which alleviates the problem of the lack of segmentation data of the B-Rep model, and the method can achieve self-supervision and improve the segmentation accuracy and precision of the trained segmentation model.

[0067] Figure 5 FIG. 1 is a flow chart of a method for training an initial segmentation model provided in an embodiment of the present application. Figure 5 As shown, the method includes the following steps.

[0068] Step S510 , based on the graph embedding representation of the sample component model and the view embedding representation of the multiple substructures corresponding to the sample component model, determine multiple face adjacency subgraphs using the initial segmentation model.

[0069] In the embodiment of the present application, the graph of the sample component model is embedded in the representation ( Figure 6 ) and the view embedding representations of multiple substructures of the corresponding sample component model ( Figure 6The hint embedding in ( ) is input into the initial segmentation model and processed by the feature fusion module and the fused feature decoding module of the initial segmentation model.

[0070] like Figure 6 As shown, first, the graph of the sample component model is embedded into and the view embedding representation of multiple substructures of the corresponding sample component model As the input of the feature fusion module, the following four steps are performed in the feature fusion model:

[0071] (1) Calculate the self-attention of the view embedding representation to update the view embedding representation (i.e. Figure 6 ), for each view embedding vector,

[0072]

[0073] in, represents the view embedding vector of the i-th substructure, i=1,2,…,|S|, They are the learnable query weight matrix, key weight matrix, and value weight matrix, LayerNorm(·) represents layer normalization, and softmax(·) represents the softmax activation function;

[0074] (2) Calculate the cross attention from the view embedding representation to the graph embedding representation and update the view embedding representation (i.e. Figure 6 , for each view embedding vector,

[0075]

[0076] in, represents the view embedding vector of the i-th substructure, i=1,2,…,|S|,E g Graph embedding representation representing sample component models;

[0077] (3) Use point-by-point multilayer perceptrons to update the view embedding vectors of multiple substructures (i.e. Figure 6 In the multi-layer perceptron, for each view embedding vector,

[0078]

[0079] in, represents the view embedding vector of the i-th substructure, i = 1, 2, …, |S|, MLP(·) represents the use of point-by-point multilayer perceptron to update the view embedding vector of each substructure;

[0080] (4) Calculate the cross attention from the graph embedding representation to the view embedding representation and update the graph embedding representation (i.e. Figure 6 Graph-to-word attention in ), for each graph embedding vector,

[0081]

[0082] in, Represents the embedding vector of the jth face, j = 1, 2, …, |V|.

[0083] It can be understood that self-attention, cross-attention and MLP all include residual connections, layer normalization, and a dropout of 0.1 during training. These are conventional settings in this field and will not be described here.

[0084] Continue to refer Figure 6 , embed the updated graph into and view embedding representation Input the fusion feature decoding module, which performs the following steps:

[0085] (1) Calculate the cross attention from the view embedding representation to the graph embedding representation to obtain the final view embedding representation ( Figure 6 , for each view embedding vector,

[0086]

[0087] (2) The final view embedding representation is input into a point-by-point multilayer perceptron to obtain |S| node segmentation masks; Where m i (i=1,2,..,|S|) identifies the classification result of the i-th node (e.g., foreground or background, marked with 1 or 0 respectively);

[0088] (3) For each mask m i , extract the mask node set Index Set Construct subgraph edge set based on index set Thus, the face adjacency subgraph is obtained.

[0089] Step S520, obtaining multiple predicted substructures corresponding to multiple face adjacency subgraphs.

[0090] Optionally, multiple predicted substructures are determined according to the face adjacency subgraph, that is, each face adjacency subgraph corresponds to a predicted substructure.

[0091] Step S530: determining view embedding representations of multiple predicted substructures.

[0092] In an embodiment of the present application, the view of the predicted substructure is adjusted, and the rendering view of the predicted substructure is determined. For example, the view of the predicted substructure is adjusted using "Fit Selection", and the views in the x, y, and z directions are used as the rendering view of the predicted substructure.

[0093] Based on the rendered views of the predicted substructures, the view embedding representation of the predicted substructure is determined. For example, for each rendered view, CLIP ViT B / 16 or CLIP ViT B / 32 is used to extract the average embedding vector of the rendered view, thereby obtaining the view embedding representation E of all predicted substructures. render .

[0094] Step S540: Based on the view embedding representations of the multiple substructures and the view embedding representations of the multiple predicted substructures, an initial segmentation model is trained to obtain a segmentation model.

[0095] Optionally, a loss function is calculated based on the view embedding representations of the multiple substructures and the view embedding representations of the multiple predicted substructures. Exemplarily, the loss function is as follows:

[0096]

[0097] Among them, cosine_similarity(E render ,E img ) represents the similarity between the view embedding representations of multiple substructures and the view embedding representations of multiple predicted substructures, Represents the view embedding representation of the i-th predicted substructure.

[0098] Based on the loss function, the model parameters are adjusted by gradient descent until convergence (i.e. Get the trained segmentation model.

[0099] Figure 7 Schematic diagram of the process of training a segmentation model provided in an embodiment of the present application. Figure 7 As shown in the figure, the process includes: UV parameterization of the component model (i.e., B-Rep model) to obtain a face adjacency graph; then using a neural network (UV-Net) to process the face adjacency graph to obtain a graph embedding representation of the component model. The process also includes: using OpenCasCade to parse the component model to determine multiple substructures; performing sparse perspective rendering on multiple substructures to determine sparse views of the substructures; and using CLIP ViT-B / 16 to determine the view embedding representation. Based on the graph embedding representation and the corresponding view embedding representation, the initial segmentation model (SA-BR) is trained until the model parameters converge to obtain a trained segmentation model.

[0100] Figure 8 is a structural block diagram of a model segmentation device provided by an exemplary embodiment of the present application. Figure 8 As shown, the model segmentation device 800 includes a generation module 810 and a segmentation module 820 .

[0101] The generation module 810 is used to obtain a component model and generate a graph embedding representation of the component model based on the component model.

[0102] Optionally, the generation module 810 is also used to construct a topological structure and extract geometric features of the component model to obtain a face adjacency graph of the component model; and use a neural network to process the face adjacency graph to obtain a graph embedding representation of the component model.

[0103] The generating module 810 is further used to obtain prompt information and generate an embedded representation of the prompt information based on the prompt information.

[0104] Optionally, the generation module 810 is further configured to input text information and / or image information into an embedding representation generation model to obtain an embedding representation of the text information and / or an embedding representation of the image information.

[0105] The segmentation module 820 is used to segment the component model using the segmentation model based on the embedded representation of the prompt information and the graph embedded representation of the component model, so as to obtain the target surface adjacent subgraph of the component model.

[0106] The specific working principle and benefits of the model segmentation device provided in the embodiment of the present application are similar to the specific working principle and benefits of the model segmentation method provided in the embodiment of the present application, and will not be repeated here.

[0107] Fig. 9 Schematic diagram of the structure of a segmentation model training device provided in one embodiment of the present application. Fig. 9 As shown, the segmentation model training device includes an acquisition module 910, a determination module 920 and a training module 930.

[0108] The acquisition module 910 is used to acquire an initial segmentation model.

[0109] The acquisition module 910 is further used to acquire training samples, where the training samples include multiple sample component models.

[0110] The determination module 920 is used to determine the graph embedding representation of the sample component model based on each sample component model.

[0111] Optionally, the determination module 920 is further used to construct a topological structure and extract geometric features of the sample component model to obtain a face adjacency graph of the sample component model; and process the face adjacency graph using a neural network to obtain a graph embedding representation of the sample component model.

[0112] The determination module 920 is further configured to determine, based on each sample component model, view embedding representations of multiple substructures of the sample component model.

[0113] The determination module 920 is further used to determine multiple substructures of the sample component model based on the sample component model; determine rendering views of the multiple substructures; and determine view embedding representations of the multiple substructures based on the rendering views of the multiple substructures.

[0114] The training module 930 is used to train the initial segmentation model to obtain a segmentation model based on the graph embedding representation of the sample component model and the view embedding representation of multiple substructures corresponding to the sample component model.

[0115] Optionally, the training module 930 is also used to determine multiple face adjacency subgraphs using the initial segmentation model based on the graph embedding representation of the sample component model and the view embedding representation of multiple substructures corresponding to the sample component model; obtain multiple predicted substructures corresponding to the multiple face adjacency subgraphs; determine the view embedding representation of the multiple predicted substructures; and train the initial segmentation model based on the view embedding representation of the multiple substructures and the view embedding representation of the multiple predicted substructures to obtain a segmentation model.

[0116] Optionally, the training module 930 is further used to adjust the view of the predicted substructure and determine the rendering view of the predicted substructure; based on the rendering view of the predicted substructure, determine the view embedding representation of the predicted substructure.

[0117] The specific working principle and benefits of the segmentation model training device provided in the embodiment of the present application are similar to the specific working principle and benefits of the segmentation model training method provided in the embodiment of the present application, and will not be repeated here.

[0118] Below, reference Fig.10 To describe an electronic device according to an embodiment of the present application. Fig.10 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0119] like Fig.10 As shown, the electronic device 1000 includes one or more processors 1001 and a memory 1002 .

[0120] The processor 1001 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0121] The memory 1002 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1001 may run the program instructions to implement the multimodal model segmentation method or segmentation model training method and / or other desired functions of the various embodiments of the present application described above. Various contents such as graph embedding representation, embedded representation of prompt information, view embedding representation, etc. may also be stored in the computer-readable storage medium.

[0122] In one example, the electronic device 1000 may further include: an input device 1003 and an output device 1004 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0123] The input device 1003 may include, for example, a keyboard, a mouse, etc.

[0124] The output device 1004 can output various information to the outside, including face adjacency subgraphs, etc. The output device 1004 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0125] Of course, to simplify, Fig.10 Only some of the components related to the present application in the electronic device 1000 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 1000 may also include any other appropriate components.

[0126] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the multimodal model segmentation method or segmentation model training method described above in this specification according to various embodiments of the present application.

[0127] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0128] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the multimodal model segmentation method or segmentation model training method according to various embodiments of the present application described above in this specification.

[0129] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0130] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0131] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0132] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0133] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0134] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A multimodal model segmentation method, characterized in that: include: Acquire a component model, and generate a graph embedding representation of the component model based on the component model; Acquire prompt information, and generate an embedded representation of the prompt information based on the prompt information; Based on the embedded representation of the prompt information and the graph embedded representation of the component model, the component model is segmented using a segmentation model to obtain a target surface adjacency subgraph of the component model.

2. The multimodal model segmentation method according to claim 1, characterized in that: The step of generating a graph embedding representation of the component model based on the component model comprises: Performing topological structure construction and geometric feature extraction on the component model to obtain a face adjacency graph of the component model; The face adjacency graph is processed using a neural network to obtain a graph embedding representation of the component model.

3. The multimodal model segmentation method according to claim 1, characterized in that: The prompt information includes text information and / or image information, and generating the embedded representation of the prompt information based on the prompt information includes: The text information and / or image information is input into an embedding representation generation model to obtain an embedding representation of the text information and / or an embedding representation of the image information.

4. A segmentation model training method, characterized in that: include: Obtain an initial segmentation model; Obtaining a training sample, where the training sample includes a plurality of sample component models; Based on each of the sample component models, determining a graph embedding representation of the sample component model; Based on each of the sample component models, determining view embedding representations of a plurality of substructures of the sample component model; Based on the graph embedding representation of the sample component model and the view embedding representation of the multiple substructures corresponding to the sample component model, the initial segmentation model is trained to obtain a segmentation model, and the segmentation model is used to segment the component model described in any one of claims 1 to 3.

5. The segmentation model training method according to claim 4, characterized in that: The graph embedding representation of the sample component model and the view embedding representation of the multiple substructures corresponding to the sample component model are used to train the initial segmentation model to obtain a segmentation model, including: Determining a plurality of face adjacency subgraphs using the initial segmentation model based on the graph embedding representation of the sample component model and the view embedding representations of the plurality of substructures corresponding to the sample component model; Obtaining a plurality of predicted substructures corresponding to the plurality of face adjacency subgraphs; determining view embedding representations of the plurality of predicted substructures; Based on the view embedding representations of the multiple substructures and the view embedding representations of the multiple predicted substructures, the initial segmentation model is trained to obtain the segmentation model.

6. The segmentation model training method according to claim 5, characterized in that: The determining of view embedding representations of the plurality of predicted substructures comprises: adjusting a view of the predicted substructure and determining a rendering view of the predicted substructure; Based on the rendered view of the predicted substructure, a view embedding representation of the predicted substructure is determined.

7. The segmentation model training method according to claim 5, characterized in that: The step of training the initial segmentation model based on the view embedding representations of the multiple substructures and the view embedding representations of the multiple predicted substructures to obtain the segmentation model comprises: Calculating a loss function based on the view embedding representations of the multiple substructures and the view embedding representations of the multiple predicted substructures; Based on the loss function, the model parameters are adjusted by the gradient descent method until convergence, thereby obtaining the segmentation model.

8. The segmentation model training method according to claim 4, characterized in that: The step of generating a graph embedding representation of the sample component model based on each of the sample component models comprises: Constructing a topological structure and extracting geometric features of the sample component model to obtain a face adjacency graph of the sample component model; The face adjacency graph is processed using a neural network to obtain a graph embedding representation of the sample component model.

9. The segmentation model training method according to claim 4, characterized in that: The step of determining, based on each of the sample component models, view embedding representations of a plurality of substructures of the sample component model comprises: Based on the sample component model, determining a plurality of substructures of the sample component model; determining rendering views of the plurality of substructures; Based on the rendered views of the plurality of substructures, view embedding representations of the plurality of substructures are determined.

10. The segmentation model training method according to claim 9, characterized in that: The step of determining a plurality of substructures of the sample component model based on the sample component model comprises: parsing the sample component model to determine the current shape of the sample component model; If the current shape of the sample component model is an assembly, all sub-shapes of the current shape are determined, and the sub-shapes are determined as the sub-structures; If the current shape of the sample component model is not an assembly, the current shape is determined as the substructure.

Images