A text and sketch based multi-modal CAD model retrieval method and device

Abstract

The application discloses a kind of multi-modal CAD model retrieval method and device based on text and sketch, through to CAD model dataset model is rendered to multiple views, text automatic labeling is realized using visual large model, and it is converted into sketch style by sketch generator;B-Rep model encoder is constructed, geometry and topological information is extracted from B-Rep data structure, and feature coding is completed using graph neural network;Sketch-text encoder is constructed, and sketch is converted to text;Multi-stage training strategy is used, first realize three modal alignment training of model network, CLIP text network and CLIP visual network;Text+sketch feature and CAD model alignment training are carried out again;Finally, retrieval reasoning supporting text and sketch input is realized.The application does not need to use model as retrieval input, can directly use more convenient and easy text and sketch to complete retrieval, significantly improve retrieval usability and engineering practicability.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and computer-aided design, deep learning and multimodal retrieval technology, and specifically to a method and apparatus for multimodal CAD model retrieval based on text and sketches. Background Technology

[0002] With the deep penetration of information technology into the manufacturing industry, digital model-driven product development and manufacturing has become the industry mainstream. CAD models, as the core information carrier for product development, are constructed based on boundary representation (B-Rep) structures, containing precise geometric shapes and topological relationships, and carrying rich design concepts, process parameters, and manufacturing knowledge. Rapidly retrieving target models from massive amounts of existing CAD models and achieving efficient reuse of design knowledge is a key path to shortening product development cycles, improving development quality and production efficiency, and is of great significance to the intelligent upgrading of modern industrial enterprises.

[0003] Existing content-based CAD model retrieval methods primarily accept inputs including the 3D model itself, 2D views, hand-drawn sketches, and text descriptions. Among these, text descriptions and sketches offer advantages such as low acquisition costs and low barriers to entry. Text can accurately express design intent and functional requirements, while sketches can intuitively present shape features; combining the two provides a more comprehensive fit for user search scenarios. The core objective of text- and sketch-based CAD model retrieval is to establish semantic relationships between multimodal inputs and CAD models. For user-provided text descriptions or sketch queries, the similarity between the model and the query is quantified and ranked for output, enabling rapid location of the target model.

[0004] However, existing technologies still face two major bottlenecks: First, there is the problem of modal heterogeneity. Text and sketches belong to two-dimensional modalities, while the geometric and topological information of CAD models based on B-Rep belongs to three-dimensional structured data. The feature spaces of different modal data differ significantly, making cross-modal semantic alignment difficult. Second, there is insufficient fine-grained retrieval capability. CAD models have obvious characteristics of large intra-class differences and uneven inter-class distribution. Traditional coarse-grained retrieval can only return models of the same type, which cannot accurately match users' specific needs for detailed shapes and functional attributes. The retrieval accuracy is difficult to meet the actual engineering application scenarios.

[0005] Therefore, overcoming the barriers of multimodal heterogeneity and constructing accurate semantic mappings between text, sketches, and CAD models to improve fine-grained retrieval performance has become a key research focus and challenge in the field of CAD model retrieval. Solving this problem can not only drive the iterative upgrade of CAD model retrieval technology but also accelerate the reuse of design knowledge, providing core technical support for fields such as intelligent manufacturing and product innovation design. Based on this, this invention proposes a method and apparatus for multimodal CAD model retrieval based on text and sketches, integrating multimodal learning and cross-modal feature alignment strategies. Summary of the Invention

[0006] This invention relates to a multimodal CAD model retrieval method and apparatus based on text and sketches. The purpose of this invention is to solve problems such as low retrieval efficiency, high computational cost, and expensive database tagging, thereby realizing a CAD model retrieval system supporting the industrial design and manufacturing sectors.

[0007] The objective of this invention is achieved through the following technical solution: Firstly, this invention provides a multimodal CAD model retrieval method and apparatus based on text and sketches, comprising the following steps:

[0008] Step 1: Obtain the CAD model, generate text annotations using the visual big language big model, and render it into a sketch using the sketch generation network, thereby constructing a CAD model dataset containing text annotations and sketches.

[0009] Step 2: Based on the geometric attribute adjacency graph, extract geometric and topological information from the boundary representation B-Rep data structure of the CAD model in the CAD model dataset;

[0010] Step 3: Construct a B-Rep model encoder based on graph neural network to encode the geometric and topological information from Step 2, thereby extracting the features of the CAD model.

[0011] Step 4: Construct a sketch-to-text encoder, which includes a sketch network and a sketch-to-text transformation network, to convert sketches into text representations and fuse them with text, thereby achieving hybrid feature extraction of "text + sketch".

[0012] Step 5, CAD model multimodal alignment training: The first stage aligns the B-Rep model encoder and the CLIP visual encoder; the second stage performs trimodal alignment based on the pre-trained B-Rep model encoder and combines the CLIP text encoder and CLIP visual encoder, and fine-tunes each network.

[0013] Step 6, Sketch-Text Hybrid Input Training: The first stage constructs a triplet alignment sketch network consisting of sketches, positive sample views, and negative sample views; the second stage simplifies the annotation text and achieves "text + sketch" modal alignment with the CAD model based on the pre-trained sketch network, CLIP visual encoder, and sketch-text conversion network.

[0014] Step 7: Based on the CLIP visual encoder, sketch encoder and sketch-to-text conversion network trained by alignment, input text and sketches to perform CAD model retrieval.

[0015] Further, in step one, a dataset with text annotations and sketches is constructed. Specifically, using an existing public dataset, multi-view projection rendering of the 3D model is performed: the model is scaled down to a unit sphere, and cameras are placed at n points uniformly sampled along the 30-degree latitude direction on the ring to obtain n projected views of the CAD model; for generating text annotations, customized text annotation prompts are used for different datasets, and the text prompts and rendered multi-views are input into a visual large language model to complete automated text annotation; for generating sketches, a set of rendered views is input into a view angle selection network for selection, and the selected views are input into a sketch generation network to obtain sketches corresponding to the CAD model.

[0016] Further, in step two, the geometric and topological information embedded in the B-Rep data structure is fully extracted based on the geometric attribute adjacency graph: First, faces and edges are discretized into UV meshes with a fixed step size. Geometric features are sampled for each face, including the coordinates, normal vector, and visibility of each mesh point. Geometric features are sampled for each edge, including the coordinates, tangent vector, normal vector of the left adjacent surface, and normal vector of the right adjacent surface. Second, face attribute features are extracted, including the one-hot code of the face type (plane, cylinder, cone, sphere, torus, revolution, extrusion, offset, or others). The topological information of the B-Rep model is represented by the following parameters: code, area, centroid coordinates, whether it is a rational B-spline surface, and edge attribute features including one-hot codes representing the edge geometry type (circular, closed, elliptical, straight, hyperbola, parabola, Bezier, irrational B-spline, rational B-spline, offset, or others), length, and one-hot codes representing the edge convexity (concave, convex, or smooth). Finally, a Face Adjacency Graph (FAG) is used to represent the topological information of the B-Rep model. In the FAG, nodes correspond to faces of the 3D model, and the connecting edges between nodes represent edges.

[0017] Furthermore, in step three, the B-Rep model network is a feature encoding network specifically adapted to the B-Rep data structure of CAD models. Its core function is to transform the extracted geometric and topological structured information into model feature representations that possess both semantic expressiveness and discriminative power, providing support for subsequent cross-modal alignment training and retrieval tasks. This network adopts a modular design, mainly composed of three core components: a geometric information encoder, a topological information encoder, and a multi-task feature encoder. Each component has a clearly defined function and achieves information exchange and fusion through a joint learning mechanism, ensuring that the output model features can comprehensively and accurately represent the core structural characteristics of the CAD model.

[0018] The geometric information encoder encodes the UV mesh data and geometric attribute data in B-Rep, uses CNN to embed the mesh data of faces and edges respectively, and uses multilayer perceptron (MLP) to embed the geometric attribute data of faces and edges respectively. Then, the mesh embeddings corresponding to faces and edges are concatenated with the attribute embeddings (CONCAT) to form the node and edge feature vectors of FAG, thus forming the graphic data structure.

[0019] The topology information encoder architecture comprises five encoding modules and four gating networks. Four of the encoding modules are task-specific (corresponding to the four subsequent tasks) and one is a task-shared encoding module. Each encoding module consists of a GNN block. The GNN block comprises three cascaded graph encoders and one linear layer, forming the core of the initial feature encoding. The internal graph encoder sequentially includes graph convolution (Graph Conv), the Mish activation function, graph normalization (GraphNorm), and droppath. In the graph convolution layer, residual connections and gating mechanisms are combined to aggregate and update graph information. Based on the original features of a node, it is weighted and combined with the aggregated information of neighboring nodes using the aggregation function. The original features are then preserved through residual connections, ultimately obtaining the updated features of the node. The linear transformation result of the node's own features and the features of neighboring nodes is combined with the Sigmoid activation function to obtain the weights used for weighting. The GNN block uses multiple aggregators to enhance the learning performance of the graph network, mainly four aggregation functions: Mean, Std, Max, and Min. It then uses a graph encoder and linear layers to obtain surface-level features. Another branch takes in node features and aggregates them into graph-level global features through graph attention convolution (GAT Conv), pooling layers, and linear layers. Finally, it concatenates the node features with the graph-level global features to obtain an embedding that combines fine-grained surface-level features with graph-level global features, representing all the information of the graph.

[0020] In the topology information encoder, shared and task-specific knowledge is selectively fused through information routing constructed by a gating network. Specifically, the task-specific encoding module consists of GNN blocks and outputs task-specific encoded features; the shared encoding module, also composed of GNN blocks, outputs shared encoded features with the same dimensionality as the task-specific features; the gating network takes the original geometric and topological information as input, passes it sequentially through convolution, linear layers, and the softmax activation function, and outputs a selected feature (Selector); the Selector is then multiplied element-wise with both the task-specific features and the shared features. This yields the weighted bi-branch features. The two weighted features are then added element by element. The final output task embedding.

[0021] The multi-task feature encoder receives topological feature vectors from the topology information encoder, covering classification task features, instance grouping task features, instance relationship prediction task features, and model feature task features. These vectors correspond to four core tasks in the CAD model: classification information for each face, clustering information indicating whether a face is an instance, topological relationship prediction information between instances, and global model feature generation. Based on the gating network of the topology information encoder, an information routing link is constructed to accurately distribute geometric features and topological features (including shared topological knowledge and task-specific topological knowledge) to the corresponding MLPs, establishing semantic links between the features of each sub-task: Face classification task features are processed by the MLP to output face-level classification information features; face classification information features and instance grouping task features are combined, and processed by the MLP to output face-level clustering features of feature instances; face classification information features, face-level clustering features of feature instances, and instance relationship prediction task features are combined to complete link prediction and relationship classification between feature instances, outputting instance grouping topological features; face classification information features, face-level clustering features of feature instances, output instance grouping topological features, and model feature task features are combined, pooled, and mapped by the MLP to generate a unified CAD model feature. The model feature vector combines completeness (covering all dimensions of core information such as geometric shape, topology, face type, and instance relationships) and discriminativeness (accurately capturing fine-grained differences between different CAD models), directly adapting to the cross-modal alignment training of the subsequent B-Rep model network and CLIP vision and text networks.

[0022] Furthermore, in step four, the sketch encoder constructs a sketch-to-text encoder. The sketch network employs a multi-scale ViT network, and through a moving window design, multiple independent and non-overlapping windows interact with each other to perform self-attention operations, thereby obtaining a larger receptive field to meet the needs of sketch input. The sketch-to-text converter network structure is a 3-layer MLP with ReLU, taking sketch feature vectors as input and outputting sketch-to-text tokens. The sketch-to-text tokens and text tokens are weighted and combined to obtain a mixed retrieval input of sketch and text.

[0023] Further, in step five, the CAD model-visual multimodal alignment training uses the projected view rendered in step one to input all views into the CLIP visual encoder to generate feature vectors, which are then output as final visual feature vectors after average pooling. The geometric and topological information of the B-Rep model extracted in step two is used as input, and the model feature vector is output through the B-Rep model encoder generated in step three. Cosine loss is used to train the visual feature vector and the model feature vector: based on the cosine similarity between the model output feature vector and the image feature vector, a difference operation is performed with 1 to obtain the loss value reflecting the difference in vector similarity. During training, the CLIP visual encoder parameters are frozen, and the B-Rep model encoder parameters are trained.

[0024] Furthermore, CAD model-visual-text multimodal alignment training is performed for each CAD model in the dataset. Create a view Text tags and model triples Training is performed using contrastive loss: the visual feature extraction method is the same as described in step five, and the model feature extraction method is also the same as described in step five. The text description generated in step one is input into the text encoder of the pre-trained CLIP large model to generate text features. Through the above methods, view features, text features, and model features are obtained. The contrastive loss function is used for training. Based on the similarity score of the positive feature pairs, a normalized comparison is made with the similarity scores of all samples in the same batch. The similarity distribution is adjusted by combining the learnable temperature parameter τ, and finally, the contrastive loss that measures the matching degree of a single modality pair is obtained. Based on the contrastive loss between each pair of text, view, and model modalities, and the hyperparameter... , , The weighted combination yields the total loss for multimodal joint training; during training, all parameters are fine-tuned.

[0025] Furthermore, in step six, the first training phase of the sketch-text mixed input training, for each CAD model in the dataset... Create a positive sample view Negative sample view and sketch triples The sketch encoder is trained using a triplet loss: for sketches, the sketch encoder is input to generate sketch feature vectors; for views, the view encoder trained in step four is input to generate view feature vectors for positive and negative samples. The difference between the view features of positive sample pairs and sketch features is compared with the difference between the view features of negative sample pairs and sketch features, and a boundary threshold is used for constraint to finally obtain a loss value that ensures effective differentiation between positive and negative sample features. During training, only the sketch encoder is trained, and the parameters of the CLIP text encoder are frozen.

[0026] Training phase two, for each CAD model in the dataset Create a model and sketch binary pairs The text and sketch features are aligned with the CAD model using contrastive loss training: A sketch is input, and a sketch encoder and sketch-to-text conversion network output sketch-to-text tokens. The text annotations generated in step one are rewritten into short text prompts, mainly consisting of brief shape and feature descriptions, and passed to a text tokenizer to output text tokens. The sketch-to-text tokens and the text tokens are combined and passed to a frozen CLIP text encoder to obtain the final combined query vector. The corresponding CAD model is input into the B-Rep model encoder, trained using a contrastive loss function. Based on the similarity score of the corresponding feature pairs, a normalized comparison is performed with the similarity scores of all samples in the same batch. A learnable temperature parameter τ is used to adjust the similarity distribution, ultimately obtaining a contrastive loss that measures the degree of matching between the two modal pairs. During training, only the parameters of the sketch encoder and sketch-to-text conversion network are trained; the remaining parameters remain frozen.

[0027] Further, in step seven, a CAD model feature vector library is first pre-generated: geometric and topological information of CAD models in the database is extracted, input into the trained B-Rep model feature encoder to obtain CAD model feature vectors, and stored. Secondly, the CLIP text encoder, sketch encoder, and sketch-to-text converter network are integrated to construct a multimodal retrieval inference architecture, supporting joint retrieval of text-only, sketch-only, and text-to-sketch bimodal modes: User text queries are received, and text feature vectors are generated through CLIP text network encoding; user sketch queries are received, and sketches are input into the sketch encoder to convert to text tokens, which are then passed to the CLIP text encoder to generate sketch feature vectors; user text and sketch queries are received, and the tokens for optional text queries are combined with the text tokens converted from sketches by the sketch encoder, and then passed to the CLIP text encoder to obtain the final combined query vector; a cosine similarity metric algorithm is used to calculate the matching degree between query features and database model feature vectors, and candidate CAD models are output in order of similarity score. Simultaneously, an approximate nearest neighbor retrieval index is configured to achieve fast retrieval under large-scale data.

[0028] Secondly, the present invention also provides a multimodal CAD model retrieval device based on text and sketches, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the multimodal CAD model retrieval method based on text and sketches.

[0029] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned multimodal CAD model retrieval method based on text and sketches.

[0030] The beneficial effects of this invention are: it supports text, sketch, and bimodal retrieval, significantly reducing the barrier to entry; it automates dataset annotation and sketch generation, reducing construction costs; the innovative B-Rep model encoder achieves deep fusion of geometric and topological features, improving feature discrimination; multi-stage multimodal alignment training addresses the semantic gap, improving matching accuracy; multi-level optimization reduces computational costs and improves retrieval efficiency, adapting to large-scale model libraries; it enhances system robustness and scenario adaptability, promotes CAD model reuse, and assists in industrial digital transformation, possessing both technological innovation and practical application value. Attached Figure Description

[0031] Figure 1 This is a flowchart of a multimodal CAD model retrieval method based on text and sketches, as described in one embodiment of the present invention.

[0032] Figure 2 This invention relates to a method for rendering multiple views of a CAD model in a multimodal CAD model retrieval method based on text and sketches, as described in one embodiment of the present invention.

[0033] Figure 3 This invention provides an automatic text annotation method for CAD datasets, which is a multimodal CAD model retrieval method based on text and sketches, as one embodiment of the present invention.

[0034] Figure 4 This invention provides a method for generating CAD model sketches based on a multimodal CAD model retrieval method using text and sketches, as an embodiment of the present invention.

[0035] Figure 5 This is a B-Rep model encoder architecture diagram of a multimodal CAD model retrieval method based on text and sketches, according to one embodiment of the present invention.

[0036] Figure 6 This is a structural diagram of the geometric information encoder in the B-Rep model encoder of a multimodal CAD model retrieval method based on text and sketches, according to one embodiment of the present invention.

[0037] Figure 7 This is a GNN block structure diagram in a B-Rep model encoder of a multimodal CAD model retrieval method based on text and sketches, according to one embodiment of the present invention.

[0038] Figure 8 This is a structural diagram of the topology information encoder in the B-Rep model encoder of a multimodal CAD model retrieval method based on text and sketches, as shown in one embodiment of the present invention.

[0039] Figure 9This is a multi-task feature encoding map in the B-Rep model encoder of a multimodal CAD model retrieval method based on text and sketches, as described in one embodiment of the present invention.

[0040] Figure 10 This invention provides a model-view modality alignment training method for a multimodal CAD model retrieval method based on text and sketches, as described in one embodiment of the present invention.

[0041] Figure 11 This invention provides a model-view-text three-modal alignment training method for a multimodal CAD model retrieval method based on text and sketches, as described in one embodiment of the present invention.

[0042] Figure 12 This is a sketch-view modality alignment training method for a multimodal CAD model retrieval method based on text and sketches, as described in one embodiment of the present invention.

[0043] Figure 13 This invention provides a sketch + text and model alignment training method for a multimodal CAD model retrieval method based on text and sketches, as described in one embodiment of the present invention.

[0044] Figure 14 This is a sketch and text-based multimodal CAD model retrieval method according to one embodiment of the present invention, which includes the reasoning process for retrieval of multimodal CAD models based on text and sketches.

[0045] Figure 15 This is a structural diagram of a multimodal CAD model retrieval device based on text and sketches, according to one embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the invention.

[0047] This invention provides a CAD model retrieval method that supports both text and sketch input, significantly improving the accuracy and convenience of CAD model retrieval and making it more suitable for industrial scenarios. This invention utilizes automated text annotation, multi-view sketch generation, B-Rep geometric topology joint encoding, multimodal alignment training, and text-sketch hybrid retrieval to achieve high-precision CAD model retrieval without requiring original model input, relying solely on text descriptions and hand-drawn sketches.

[0048] The advantages of this invention include: (a) more convenient retrieval method, no need to provide a complete CAD model, the query can be completed by only text description and hand-drawn sketch, reducing the user threshold and improving the ease of use of retrieval; (b) adopting automated text annotation and multi-view sketch generation, reducing manual annotation costs, improving the efficiency of dataset construction, and adapting to the retrieval needs of large-scale CAD model libraries; (c) based on the geometric and topological joint encoding of B-Rep structure, combined with graph neural network to achieve fine-grained feature extraction, which significantly improves the expressive power and discriminative power of model features compared with traditional methods that only rely on geometric or shallow features; (d) through two-stage multimodal alignment training, the text, vision and model three-modal feature space is unified, effectively reducing modal differences and improving cross-modal matching accuracy.

[0049] Example 1

[0050] All documents mentioned in this invention are incorporated herein by reference as if each document were individually incorporated by reference. Furthermore, it should be understood that after reading the foregoing teachings of this invention, those skilled in the art can make various alterations or modifications to this invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0051] Figure 1 This is a flowchart of a multimodal CAD model retrieval method based on text and sketches, according to one embodiment of the present invention. The method includes the following steps:

[0052] Step 101: Construct a CAD model dataset containing text annotations and sketches;

[0053] Step 102: Extract the geometric and topological information of the CAD model;

[0054] Step 103, construct the B-Rep model encoder;

[0055] Step 104: Pre-train the B-Rep model network to align the model features with the view features;

[0056] Step 105: Use the CLIP model and B-Rep model network to perform joint training of model-view-text;

[0057] Step 106: Construct the sketch network and the sketch-to-text transformation network;

[0058] Step 107, Pre-train the sketch encoder;

[0059] Step 108: Training to align text and sketch features with the CAD model;

[0060] Step 109: Implement CAD model retrieval and reasoning based on text and sketch inputs.

[0061] Specifically, in one embodiment, the text- and sketch-based multimodal CAD model retrieval method includes the following steps:

[0062] Step one involves constructing a dataset containing text annotations and sketches. Specifically, existing publicly available CAD model datasets such as Fusion 360 Gallery Assembly Dataset, MFTRCAD, and FabWave are used. A Blender script is employed to perform multi-view projection rendering of the 3D model: the model is scaled to a unit sphere, and along a 30-degree latitude direction on the sphere, six points are uniformly sampled on a ring, with cameras positioned to obtain six projected views of the CAD model. The rendering viewpoints are as follows: Figure 2 As shown.

[0063] For generating text annotations, customized text annotation prompts are used for different datasets. For example, the Fusion 360Gallery Assembly Dataset only has CAD models without shape labels, so the prompts mainly guide the visual language model to observe the main shape features of the model. MFTRCAD has model feature label information, so the view rendering assigns prominent colors to the feature faces, renders a set of multiple views for each feature, and then describes the feature names of these prominently colored faces in the text annotation prompts, guiding the visual language model to observe the size and position of these features in the model, outputting feature information description text. Finally, all feature information description text and multiple views are added to the text prompts, guiding the visual language model to summarize the overall shape information of the model and the names and positions of all features. FabWave has model class names, so the text annotation prompts mainly guide the visual language model to observe the main shape features of the model, combining the model class names to summarize text annotations containing functional and shape information. The text annotation process and related text prompts are as follows: Figure 3 As shown.

[0064] For generating sketches, a set of rendered views is input into a view angle selection network. This network is pre-trained on the CADSketchNet dataset, enabling it to select images that best approximate the perspective of a hand-drawn sketch. The selected views are then input into the pre-trained sketch generator PhotoSketch to obtain the sketch corresponding to the CAD model. The sketch rendering pipeline is as follows: Figure 4 As shown.

[0065] Step two: Fully extract the geometric and topological information embedded in the B-Rep data structure based on the geometric attribute adjacency graph: First, discretize the faces and edges into UV meshes with a fixed step size, and sample each face as... The feature representation, where, The number of grid sampling points in the U direction. This represents the number of grid sampling points in the V direction. It includes the coordinates (3D), normal vector (3D), and visibility (1D) of each grid point, with sampling performed for each edge. The feature representation includes the coordinates (3D), tangent (3D), normal vectors of the left and right adjacent surfaces (3D) for each grid point; secondly, the attribute features of faces and edges are extracted, with face attributes represented as 14D feature vectors, including one-hot codes for face types (plane, cylinder, cone, sphere, torus, revolution, extrusion, offset, or others). The B-Rep model is represented by a 15D feature vector, including a one-hot code (9D), area (1D), centroid coordinates (3D), whether it is a rational B-spline surface (1D), and edge attributes (11D). This includes a one-hot code (11D) representing the edge geometry type (circular, closed, elliptical, straight, hyperbola, parabola, Bezier, irrational B-spline, rational B-spline, offset, or others), length (1D), and convexity (3D) representing the edge convexity (concave, convex, or smooth). Finally, a graphical data structure called a Face Adjacency Graph (FAG) is used to represent the topological information of the B-Rep model. In the FAG, nodes correspond to faces of the model, and the connecting edges between nodes represent edges. The FAG is constructed from the B-Rep model via the Python OCC API and stored as an undirected graph data structure. Node features include face grid point parameters and face attributes, while connecting edge features include edge grid point parameters and edge attributes.

[0066] Step 3: Construct a B-Rep model encoder adapted to the B-Rep data structure of the CAD model, transforming the extracted geometric and topological structured information into model feature representations that possess both semantic expressiveness and discriminative power. The overall architecture is as follows: Figure 5 As shown, the whole adopts a modular design, mainly composed of three core components: geometric information encoder, topological information encoder and multi-task feature encoder. Each component has a clear function and achieves information exchange and fusion through a joint learning mechanism.

[0067] The geometric information encoder encodes the UV mesh data and geometric attribute data in B-Rep. The architecture of the geometric information encoder is as follows: Figure 6 As shown, this network utilizes three 2D CNNs (Convolutional Neural Networks), Avg-Pool (Average Pooling), and FC (Fully Connected Layers) to extract data from the surface grid data. Surface mesh features ( (To determine the number of faces), three 1D CNNs, Avg-Pool, and FC were used to extract from the edge grid data. edge mesh features ( (The number of edges), using a 2-layer MLP to embed the geometric attribute data of faces and edges respectively. surface attribute features and The edge attribute features are then obtained, and the face mesh features and face attribute features are concatenated into a single feature. The surface features are used as the node features of the FAG, and the edge mesh features and edge attribute features are concatenated into a single feature. The edge features are used as the edge features of FAG to form a graph data structure.

[0068] The topology information encoder employs an architecture comprising five encoding modules and four gating networks. The entire network architecture has four core tasks: classification information for each face of the CAD model, clustering information to determine if a face is an instance, prediction of topological relationships between instances, and global model feature generation. Each task has a task-specific encoding module. Therefore, four of the encoding modules are task-specific, and the remaining one is a task-shared encoding module. Each encoding module consists of a GNN block, which comprises multiple cascaded graph encoders and one linear layer, such as... Figure 7 As shown. The internal graph encoder sequentially includes Graph Conv (graph convolution), the activation function Mish, GraphNorm (graph normalization), and DropPath (random path discarding to prevent overfitting). In the graph convolution layer, residual connections and gating mechanisms are combined to aggregate and update graph information. Based on the original features of nodes, the information is weighted and combined with the information of neighboring nodes after aggregation by the aggregation function, and then the original features are preserved through residual connections. This can be represented as follows:

[0069]

[0070]

[0071] in and Representing nodes respectively Features before and after the update. Represents a node The set of adjacent nodes. Represents a node and The edges between them. It is an aggregate function. These are learnable weight parameters. The activation function is Sigmoid. The aggregation function is used during message passing between neighboring nodes. The choice of aggregation function has a decisive impact on the performance of GNN. Using multiple aggregators can further enhance the learning effect of graph networks. The GNN block uses four aggregation functions: Mean, Std, Max, and Min. After learning through multiple graph encoders, the features of each node are obtained, and through Linear, the basic encoded features at the surface level are obtained. Another branch takes input node features and uses GAT Conv (Graph Attention Convolution), Pooling, and Linear layers to globally aggregate the patch-level features, compressing them into graph-level global features with a dimension of 1×64D. The node features are then concatenated with the graph-level global features to obtain an embedding that incorporates all information from both the fine-grained patch-level and graph-level global feature representations, with a shape of... .

[0072] In the topology information encoder, shared and task-specific knowledge is selectively fused through information routing constructed by a gating network. The collaborative process between the task-specific coding module and the task-sharing coding module of the topology information encoder is as follows: Figure 8 As shown. Specifically: The task-specific encoding module consists of GNN blocks, outputting task-specific encoded features with the shape of... The shared encoding module, also composed of GNN blocks, outputs shared encoded features with dimensions consistent with the task-specific features. The gating network takes the original geometric and topological information as input, passes it sequentially through GCN Conv, Linear, and Softmax, and outputs a selection feature (Selector) with the shape... The selector is multiplied element-wise with the specific task features and shared features respectively. This yields the weighted bi-branch features. The two weighted features are then added element by element. The final output shape is Task embedding. Based on the characteristics of each task, each encoding module, together with its corresponding gating network, shares the encoder to output classification task features, instance grouping task features, instance relationship prediction task features, and model feature task features.

[0073] Figure 9 This is a diagram of the multi-task feature encoder structure. The multi-task feature encoder receives topological feature vectors from the topology information encoder, which output features for classification tasks, instance grouping tasks, instance relationship prediction tasks, and model features. It then constructs information routing links based on the gating network of the topology information encoder, accurately distributing geometric features and topological features (including shared topological knowledge and task-specific topological knowledge) to the corresponding MLPs, thus establishing semantic links between the features of each sub-task. For example, the model surface classification task features are processed by the MLP to output surface-level classification information features, with the shape of... The splicing surface classification information features and instance grouping task features are processed by MLP to output the surface-level clustering features of the feature instances, with the shape being... The system combines surface classification information features, surface-level clustering features of feature instances, and instance relationship prediction task features to complete link prediction and relationship classification between feature instances, outputting instance grouping topological features with the shape of... The splicing surface classification information features, surface-level clustering features of feature instances, output instance grouping topological features, and model feature task features are mapped through Pooling and MLP to generate a unified CAD model feature vector with the shape of... .

[0074] Step four involves constructing a sketch-to-text encoder. The sketch network employs a multi-scale ViT network, the swin transformer. This network utilizes a moving window design, allowing multiple independent, non-overlapping windows to interact and perform self-attention operations, thereby achieving a larger receptive field to suit sketch input needs. The sketch-to-text converter network structure is a 3-layer MLP with ReLU, taking sketch feature vectors as input and outputting sketch-to-text tokens. Combining these tokens with the text tokens yields a mixed retrieval input of sketch and text.

[0075] Step 5: Perform CAD model-visual multimodal alignment training, the training method is as follows: Figure 10 As shown. Because the Fusion360 Gallery Assembly Dataset contains a large amount of data and rich shape information, this dataset was chosen for training. Six projected views were rendered using two models from this dataset. The CLIP model used in this embodiment is FG-CLIP, which has learned visual knowledge highly aligned with fine-grained text. Therefore, all views were input into the CLIP visual encoder to generate m×6×512D feature vectors. Where m is the batch size. The output after average pooling is an m×512D visual feature vector. To obtain view features :

[0076]

[0077] CLIP-ImageEncoder represents the CLIP visual encoder. The view is represented. The geometric and topological information of the B-Rep model extracted in step two is used as input, and the B-Rep model encoder generated in step three outputs an m×512D model feature vector. :

[0078]

[0079] in, The model is represented by the cosine loss training method, as shown in the formula:

[0080]

[0081] During training, the parameters of the CLIP image encoder are kept frozen, while the parameters of the B-Rep model encoder are trained.

[0082] Step 6: Perform CAD model-visual-text multimodal alignment training, the training method is as follows: Figure 11 As shown. A subset of models from the Fusion 360 Gallery Assembly Dataset, MFTRCAD, and FabWave datasets were selected and trained together on all three datasets. For each CAD model in the datasets... Create a view Text tags and model triples Then, these triples are used for training: the visual feature extraction method is the same as in step five, and the model feature extraction method is the same as in step five. The text description generated in step one is input into the text encoder of the pre-trained CLIP large model to generate text features.

[0083]

[0084] CLIP-TextEncoder represents the CLIP text encoder, which outputs an m×512D model feature vector. View features are obtained through the above method. Text features and model features Training is performed using a contrastive loss function, and the contrastive loss between each pair of modes is calculated as follows:

[0085]

[0086] in and Representing two modes, This represents the pairs in each training batch. This represents other training samples in the batch. A learnable temperature parameter is used. Adjust the distribution of similarity scores. Minimize for all modality pairs with different coefficients:

[0087]

[0088] in, , , These are hyperparameters, and their values ​​can be adjusted to control the parameter update speed of each network model.

[0089] During training, all parameters are fine-tuned.

[0090] In step seven, the first training phase of the sketch-text mixed input training is conducted as follows: Figure 12 As shown. Select the Fusion 360 Gallery Assembly Dataset dataset, for each CAD model in the dataset. Create a positive sample view Negative sample view and sketch triples For the sketch, input it into the encoder to generate an m×512D vector. For the view, its input is the CLIP visual encoder trained in step four, a vector of m×512D. , The output loss function is:

[0091]

[0092]

[0093] in, The boundary threshold is used. During training, only the sketch encoder is trained, and the parameters of the CLIP visual encoder remain frozen.

[0094] The training phase two of the sketch encoder training is as follows: Figure 13 As shown. Select the FabWave dataset, for each CAD model in the dataset. Create a model and sketch binary pairs The text and sketch features are aligned with the CAD model using contrastive loss: A sketch is input, and a sketch encoder and sketch-to-text conversion network output sketch-to-text tokens. The text annotations generated in step one are rewritten into brief text prompts, mainly consisting of short descriptions of shape and features. These prompts are then passed to the text tokenizer to output text tokens. Convert sketches to text tokens and combine text tokens. This is then passed to the frozen CLIP text encoder to obtain the final combined query vector; the corresponding CAD model is input into the B-Rep model encoder and trained using the contrastive loss function.

[0095]

[0096] in and These represent two different models and a sketch + text modality, respectively. This represents the pairs in each training batch. This represents other training samples in the batch. A learnable temperature parameter is used. Adjust the distribution of similarity scores. During training, only the parameters of the sketch encoder and sketch-to-text conversion network are trained, while the remaining parameters remain frozen.

[0097] Step 7: Implement retrieval inference supporting text and sketch input. First, a CAD model feature vector library is pre-generated: geometric and topological information of CAD models in the database is extracted, input into the trained B-Rep model feature encoder to obtain CAD model feature vectors, and stored. Second, the CLIP text encoder, sketch encoder, and sketch-to-text converter network are integrated to construct a multimodal retrieval inference architecture, supporting joint retrieval of text monomodality, sketch monomodality, and text-to-sketch bimodality. It receives user text queries and generates text feature vectors through CLIP text network encoding; it receives user sketch queries, inputs sketches into sketch encoders to convert them into text tokens, and passes them to CLIP text encoder vectors to generate sketch feature vectors; it receives both user text and sketch queries, combines the optional text query tokens with the sketch encoder-converted text tokens, and passes them to CLIP text encoder to obtain the final combined query vector; a cosine similarity metric algorithm is used to calculate the matching degree between query features and database model feature vectors, and candidate CAD models are output in order of similarity score. Simultaneously, an approximate nearest neighbor retrieval index is configured to achieve fast retrieval under large-scale data. The system interface developed for multimodal CAD model retrieval based on text and sketch input is as follows: Figure 14 As shown.

[0098] Corresponding to the aforementioned embodiment of a multimodal CAD model retrieval method based on text and sketches, the present invention also provides an embodiment of a multimodal CAD model retrieval device based on text and sketches.

[0099] See Figure 15 The present invention provides a multimodal CAD model retrieval device based on text and sketches, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a multimodal CAD model retrieval method based on text and sketches as described in the above embodiment.

[0100] The embodiment of the multimodal CAD model retrieval device based on text and sketches provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 15 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an example of a multimodal CAD model retrieval device based on text and sketches provided by the present invention. (Except for...) Figure 15 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0101] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0102] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0103] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a multimodal CAD model retrieval method based on text and sketches as described in the above embodiments.

[0104] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0105] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned multimodal CAD model retrieval method based on text and sketches.

[0106] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A multimodal CAD model retrieval method based on text and sketches, characterized in that, Includes the following steps: Step 1: Obtain the CAD model, generate text annotations using the visual big language big model, and render it into a sketch using the sketch generation network, thereby constructing a CAD model dataset containing text annotations and sketches. Step 2: Based on the geometric attribute adjacency graph, extract geometric and topological information from the boundary representation B-Rep data structure of the CAD model in the CAD model dataset; Step 3: Construct a B-Rep model encoder based on graph neural networks to encode geometric and topological information and extract CAD model features; Step four involves constructing a sketch-to-text encoder, which includes a sketch network and a sketch-to-text conversion network. The sketch network employs a multi-scale ViT network, where multiple independent and non-overlapping windows interact with each other through self-attention operations by moving windows to obtain sketch feature vectors. The sketch-to-text conversion network is an MLP with ReLU, taking sketch feature vectors as input and outputting sketch-to-text tokens. The sketch-to-text tokens and text tokens are then fused to obtain a mixed retrieval input of sketch and text, achieving mixed feature extraction of "text + sketch". Step 5, CAD model multimodal alignment training: The first stage aligns the B-Rep model encoder with the CLIP visual encoder; The second stage is based on the pre-trained B-Rep model encoder and combined with the CLIP text encoder and CLIP visual encoder to perform trimodal alignment and fine-tune each encoder network. Step 6, Sketch-Text Hybrid Input Training: In the first stage, a triplet containing a sketch, a positive sample view, and a negative sample view is constructed for each CAD model. The sketch is input into the sketch network, and the positive and negative sample views are input into the CLIP visual encoder respectively. The sketch network is trained through the triplet loss to align the sketch network with the CLIP visual encoder. The second stage simplifies the labeled text and achieves "text + sketch" modal alignment with the CAD model based on a pre-trained sketch network, CLIP visual encoder, and sketch-to-text conversion network. Step 7: Based on the CLIP visual encoder, sketch network, and sketch-to-text conversion network trained with alignment, input text and sketches to perform CAD model retrieval.

2. The multimodal CAD model retrieval method based on text and sketches according to claim 1, characterized in that, Step one, the construction of the CAD model dataset, includes: performing multi-view projection rendering on the 3D CAD model based on the publicly available CAD model dataset, scaling the model to a unit sphere, and uniformly sampling. n The camera pose is obtained n Multiple projected views; custom text annotation prompts can be set for different datasets, and the text annotation prompts can be combined with... n Input a projection view into the visual large language model to generate text annotations; input a set of rendered views into the view selection network to filter views, and input the selected views into the sketch transformation network to generate corresponding sketches.

3. The multimodal CAD model retrieval method based on text and sketches according to claim 1, characterized in that, In step two, the extraction of geometric and topological information includes: discretizing the faces and edges into UV meshes and sampling to obtain mesh features, the mesh features including the coordinates of the sampling points and the normal vector; extracting face attribute features, including face type, area and centroid coordinates; extracting edge attribute features, including edge geometry type, length and convexity representation; and using a face adjacency graph (FAG) to represent the B-Rep topological information, where FAG nodes correspond to faces and the connecting edges between nodes correspond to model edges.

4. The multimodal CAD model retrieval method based on text and sketches according to claim 1, characterized in that, In step three, the B-Rep model encoder consists of a geometric information encoder, a topological information encoder, and a multi-task feature encoder: The geometric information encoder encodes the geometric attribute data, uses a convolutional neural network to embed the mesh data of the face and the edge respectively, and uses an MLP to embed the geometric attribute data of the face and the edge respectively. Then, the mesh embeddings corresponding to the face and the edge are concatenated with the attribute embeddings to form the node and edge feature vectors of the FAG, thus forming a graphic data structure. The topology information encoder consists of five coding modules and four gating networks, four of which are task-specific modules and the remaining one is a task-shared module. Each coding module is composed of GNN blocks, which obtain surface-level encoded features and aggregate them into graph-level global features based on graph attention convolution. The joint embedding is obtained by concatenating node features with graph-level global features; The multi-task feature encoder receives the topology information encoder output of each task; it establishes information routing based on the gating network, distributes geometric features and topology features to the corresponding MLP, transmits the features of each subtask in sequence, and concatenates the current task features with the previous set of MLP features as input, outputting features containing task target information; it integrates the current task and the aforementioned task information in the model feature task, and maps them to generate a unified CAD model feature vector.

5. The multimodal CAD model retrieval method based on text and sketches according to claim 1, characterized in that, Step five, the multimodal alignment training of the CAD model includes: In the first stage, the projected view rendered by the large visual language and large model in step one is used to input all views into the CLIP visual encoder to generate visual feature vectors. Geometric and topological information is used as input, and the B-Rep model encoder outputs CAD model feature vectors. The visual feature vectors and model feature vectors are trained by performing a difference operation between the cosine similarity between the model output feature vector and the image feature vector and 1 to obtain a loss value that reflects the difference in vector similarity. During training, the CLIP visual encoder parameters are frozen, and the B-Rep model encoder parameters are trained. The second stage uses a centralized CAD model, multiple views, and text annotations for training. The model feature extraction and visual feature extraction methods are consistent with those in the first stage. Text annotations are input into the pre-trained CLIP large model's text encoder to generate text features. View features, text features, and model features are trained. Based on the similarity scores of opposite feature pairs, a normalized comparison is made with the similarity scores of all samples in the same batch. The similarity distribution is adjusted by incorporating a learnable temperature parameter to finally obtain the contrast loss that measures the matching degree of the bimodal pairs. Based on the contrast losses between each pair of text, view, and model modalities, a weighted fusion is performed to obtain the total loss for multimodal joint training. During training, all parameters are fine-tuned.

6. The multimodal CAD model retrieval method based on text and sketches according to claim 1, characterized in that, Step six, the sketch-text mixed input training includes: Training Phase 1: Sketch input generates sketch feature vectors in the sketch network; positive and negative sample views are input into the CLIP visual encoder respectively, generating view feature vectors for both positive and negative samples. The difference between the view features of the positive sample pair and the sketch features is compared with the similarity between the negative sample pair and the sketch features. Constraints are applied using boundary thresholds to obtain a loss value that ensures effective differentiation between positive and negative sample features. During training, only the sketch network is trained, and the parameters of the CLIP text encoder remain frozen. Training Phase Two: For each CAD model, construct a pair containing a sketch and the model. The sketch is processed by the sketch network and the sketch-to-text conversion network to output sketch-to-text tokens. Simplify the text annotations into text prompts, which are short shape and feature descriptions. These prompts are then passed to the text tokenizer to output text tokens. Combine the sketch-to-text tokens and the text tokens and pass them to the frozen CLIP text encoder to obtain the final combined query vector. Input the corresponding CAD model into the B-Rep model encoder and train it using the contrastive loss function. During training, only the parameters of the sketch network and the sketch-to-text conversion network are trained, while the other parameters remain frozen.

7. The multimodal CAD model retrieval method based on text and sketches according to claim 1, characterized in that, In step seven, a CAD model feature vector library is pre-generated. This involves extracting geometric and topological information from the CAD models in the database and inputting it into the trained B-Rep model encoder to obtain and store the model feature vectors. A multimodal retrieval inference framework integrating a trained and aligned CLIP text encoder and a sketch-text encoder is constructed to support text retrieval, sketch retrieval, and joint text-sketch retrieval. Cosine similarity is used to calculate the matching degree between the query features and the feature vectors in the library, and candidate CAD models are output in order of similarity. An approximate nearest neighbor index is also established to achieve large-scale fast retrieval.

8. A multimodal CAD model retrieval device based on text and sketches, comprising a memory and one or more processors, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements a multimodal CAD model retrieval method based on text and sketches as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a multimodal CAD model retrieval method based on text and sketches as described in any one of claims 1-7.

Images