A reconstruction guided multi-modal cad retrieval method and system

Abstract

The application discloses a kind of reconstruction guided multimodal CAD retrieval method and system, it is related to CAD model retrieval technical field.The method includes: obtaining text description corresponding with CAD model, rendering image and the CAD modeling sequence for representing the modeling process of CAD model;Text description, rendering image and CAD modeling sequence are extracted and fused processing feature, and multiple modal fusion features are obtained;Based on multiple modal fusion features, the reconstruction of CAD modeling sequence is carried out, and the reconstruction result is obtained, and guide constraint is constructed using reconstruction result, and the joint training of multimodal CAD retrieval model is carried out, and the training of multimodal CAD retrieval model is completed;Query information to be retrieved is obtained and matched with candidate CAD modeling sequence, and CAD modeling sequence retrieval result is output.Thereby it can improve the utilization sufficiency of retrieval information and editable reuse feature representation ability, and enhance the distinguishing degree of similar CAD model.

Description

Technical Field

[0001] This application relates to the field of CAD model retrieval technology, and in particular to a reconstruction-guided multimodal CAD retrieval method and system. Background Technology

[0002] With the development of digital industrial product design, platform-based model resources, and intelligent design tools, computer-aided design models are widely used in mechanical design, industrial manufacturing, product reuse, and solution iteration. When developing new products or modifying existing structures, designers typically need to search through a large number of historical CAD models for models that closely match current design requirements. This allows for modification and reuse of existing designs, reducing repetitive modeling work and improving design efficiency. Therefore, how to quickly and accurately retrieve matching models from CAD model libraries based on user-input design requirements has become an important research topic in the fields of CAD data management and intelligent assisted design.

[0003] In related technologies, CAD model retrieval is typically based on information such as keywords, attribute tags, 2D images, 3D geometric features, or text descriptions. These methods generally first extract features from the query information and candidate CAD models, then rank the candidate models based on feature similarity to obtain the retrieval results. However, in actual industrial design scenarios, users' retrieval needs are often not entirely expressed by a single piece of information; information from different sources may have complementary relationships. Furthermore, different CAD models may have similar appearances, similar local structures, or similar semantic descriptions, making reliance solely on conventional feature similarity prone to insufficient discriminative power in the retrieval results. In addition, some retrieval methods focus more on the appearance or overall geometric similarity of the model, neglecting the structural composition, design intent, and parametric features required for the editable and reusable process of CAD models, resulting in retrieved models that fail to accurately meet subsequent editing and reuse requirements.

[0004] Therefore, in intelligent CAD model retrieval, insufficient utilization of retrieval information, inadequate representation of editable and reusable model features, and low distinguishability of similar CAD model retrieval have become urgent problems to be solved. Summary of the Invention

[0005] This application provides a reconstruction-guided multimodal CAD retrieval method and system, aiming to solve the problems of insufficient utilization of retrieval information, insufficient representation of editable and reusable features of models, and low distinguishability of similar CAD models in existing technologies for intelligent CAD model retrieval.

[0006] Firstly, a reconstruction-guided multimodal CAD retrieval method, the method comprising: Obtain multimodal sample data corresponding to the CAD model. The multimodal sample data includes text descriptions, rendered images, and CAD modeling sequences. The CAD modeling sequences are used to characterize the modeling process of the CAD model. Feature extraction and fusion processing are performed on the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features that characterize the CAD model; The CAD modeling sequence is reconstructed based on the multimodal fusion features to obtain the reconstruction result. The reconstruction result is then used to construct guiding constraints and jointly train the multimodal CAD retrieval model to obtain the trained multimodal CAD retrieval model. Obtain the query information to be retrieved, input the query information to be retrieved into the trained multimodal CAD retrieval model, match it with the candidate CAD modeling sequence, and output the corresponding CAD modeling sequence retrieval result.

[0007] Optionally, in the above scheme, obtaining the multimodal sample data corresponding to the CAD model includes: Obtain text description data, rendered image data, and CAD modeling sequence data corresponding to multiple CAD models to obtain initial trimodal data; The initial trimodal data is processed to obtain trimodal standard data with a unified format; Based on the correspondence of the same CAD model, the text description, rendered image and CAD modeling sequence in the trimodal standard data are paired to obtain trimodal paired samples; The multimodal sample data is constructed based on multiple paired trimodal samples.

[0008] Optionally, in the above scheme, constructing the multimodal sample data based on multiple trimodal paired samples includes: The CAD modeling sequence is parsed to obtain multiple modeling steps arranged in sequence; Based on the CAD commands and modeling parameters in each modeling step, generate corresponding command representation information and parameter representation information; According to the order of the multiple modeling steps, the command representation information and the parameter representation information are combined to obtain a serialized modeling representation; The serialized modeling representation is associated with and stored with the corresponding text description and rendered image to obtain the multimodal sample data.

[0009] Optionally, in the above scheme, the step of performing feature extraction and fusion processing on the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features for characterizing the CAD model includes: Text feature extraction is performed on the text description to obtain text modal features; Image feature extraction is performed on the rendered image to obtain image modal features; Sequence feature extraction is performed on the CAD modeling sequence to obtain CAD modal features; The text modal features, image modal features, and CAD modal features are mapped to a unified feature space to obtain the corresponding trimodal embedding features; The three-modal embedding features are fused to obtain the multimodal fused features.

[0010] Optionally, in the above scheme, the step of fusing the three-modal embedding features to obtain the multimodal fused features includes: Based on the feature dimensions of the unified feature space, modality selection information corresponding to the text modality, image modality, and CAD modality is generated respectively; Based on the modality selection information, the unique source of the target modality corresponding to each feature dimension is determined. Based on the source of the target modality, feature values ​​of the corresponding dimension are selected from the three-modal embedding features to obtain dimension-level combined features; The multimodal fusion feature is generated based on the dimensional combination features.

[0011] Optionally, in the above scheme, the step of reconstructing the CAD modeling sequence based on the multimodal fusion features to obtain a reconstruction result, and using the reconstruction result to construct guiding constraints to train the multimodal CAD retrieval model, thereby obtaining a trained multimodal CAD retrieval model, includes: The multimodal fusion features are input into the sequence decoding model to obtain a predicted modeling sequence for the CAD modeling sequence; The prediction sequence representation is determined based on the prediction CAD commands and prediction modeling parameters in the prediction modeling sequence; Based on the difference between the predicted sequence representation and the actual sequence representation corresponding to the CAD modeling sequence, the modeling sequence reconstruction error information is determined; The reconstruction result is obtained based on the predicted modeling sequence and the reconstruction error information of the modeling sequence; Based on the reconstruction results, a reconstruction guidance loss is constructed, and the reconstruction guidance loss is determined as the reconstruction guidance constraint in the guidance constraint. The reconstruction guidance constraint is used to constrain the feature representation space of the multimodal CAD retrieval model. Based on the guiding constraints, the multimodal CAD retrieval model is jointly trained to obtain the trained multimodal CAD retrieval model.

[0012] Optionally, in the above scheme, the guiding constraints may also include difficult negative sample guiding constraints; Before jointly training the multimodal CAD retrieval model based on the guiding constraints to obtain the trained multimodal CAD retrieval model, the method further includes: Query modal features are determined from the text modal features and / or image modal features corresponding to the training samples, and a set of candidate hard negative samples is determined based on the similarity between the query modal features and multiple candidate CAD modal features; Based on the query modal features, the candidate CAD modeling sequences in the difficult negative sample candidate set are reconstructed and verified to obtain the negative sample reconstruction information corresponding to each candidate CAD modeling sequence; Based on the similarity and the negative sample reconstruction information, a target difficult negative sample is determined from the difficult negative sample candidate set; Based on the target difficult negative sample and the negative sample reconstruction information, difficult negative sample processing information is generated; The difficult negative sample guidance constraint is constructed based on the difficult negative sample processing information. The difficult negative sample guidance constraint is used to determine the training weight and / or training constraint of the target difficult negative sample in the joint training of the multimodal CAD retrieval model.

[0013] Optionally, in the above scheme, the step of jointly training the multimodal CAD retrieval model based on the guiding constraints to obtain the trained multimodal CAD retrieval model includes: Based on the matching relationship between samples from different modalities, determine the cross-modal contrastive learning objective; Based on the reconstruction guidance constraints, the reconstruction target of the modeling sequence is determined; Based on the aforementioned difficult negative sample guiding constraints, the target for difficult negative sample mining is determined; The cross-modal contrastive learning objective, the modeling sequence reconstruction objective, and the hard negative sample mining objective are weighted to obtain a joint training objective; Based on the joint training objective, the parameters of the multimodal CAD retrieval model are updated to obtain the trained multimodal CAD retrieval model.

[0014] Optionally, in the above scheme, the step of obtaining the query information to be retrieved, inputting the query information to be retrieved into the trained multimodal CAD retrieval model, matching it with candidate CAD modeling sequences, and outputting the corresponding CAD modeling sequence retrieval results includes: Feature extraction is performed on the candidate CAD modeling sequences to obtain the candidate CAD sequence features; Based on the candidate CAD sequence features, a candidate CAD modeling sequence feature library is constructed; Obtain the query information to be retrieved, and extract query features from the query information to obtain query features. The query information to be retrieved includes text query information and / or image query information. The query features are matched with the candidate CAD sequence features in the candidate CAD modeling sequence feature library to obtain the matching results; The candidate CAD modeling sequences are sorted according to the matching results, and the CAD modeling sequence retrieval results are output.

[0015] Secondly, a reconstruction-guided multimodal CAD retrieval system, the system comprising: The data acquisition module is used to acquire multimodal sample data corresponding to the CAD model. The multimodal sample data includes text descriptions, rendered images, and CAD modeling sequences. The CAD modeling sequences are used to characterize the modeling process of the CAD model. The feature fusion module is used to extract and fuse features from the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features that characterize the CAD model. The reconstruction training module is used to reconstruct the CAD modeling sequence based on the multimodal fusion features, determine model training constraints based on the reconstruction results, train the multimodal CAD retrieval model, and obtain the trained multimodal CAD retrieval model. The retrieval output module is used to obtain the query information to be retrieved, input the query information to be retrieved into the trained multimodal CAD retrieval model, match it with the candidate CAD modeling sequence, and output the corresponding CAD modeling sequence retrieval result.

[0016] Compared with the prior art, this application has at least the following beneficial effects: This application, based on further analysis and research of existing technical problems, recognizes the issues of insufficient utilization of retrieval information, inadequate model editability and reusability, and low distinguishability in CAD model intelligent retrieval. By acquiring multimodal sample data including text descriptions, rendered images, and CAD modeling sequences, the training data of the retrieval model simultaneously covers the semantic description information, visual appearance information, and modeling process information of the CAD model, thus providing a foundation for the model to learn the correspondence between multi-source retrieval information. Furthermore, by performing feature extraction and fusion processing on text descriptions, rendered images, and CAD modeling sequences, multimodal fusion features are obtained to represent CAD models, enabling retrieval information from different sources to work synergistically in the same feature expression, alleviating the problem of insufficient expression of single retrieval information. Based on this, the CAD modeling sequence is reconstructed using multimodal fusion features to obtain reconstruction results, and guiding constraints are constructed using these results to apply multimodal fusion features to CAD models. Joint training of modal CAD retrieval models enables the multimodal CAD retrieval model to learn not only the surface similarity relationship between query information and candidate CAD modeling sequences during training, but also the modeling process, operation sequence, and parametric expression features of CAD models. This enhances the model's ability to represent the editable and reusable features of CAD models. Finally, by inputting the query information to be retrieved into the trained multimodal CAD retrieval model and matching it with candidate CAD modeling sequences to output CAD modeling sequence retrieval results, the retrieval results are geared towards editable CAD modeling sequences rather than just the appearance of the models. Therefore, it can improve the sufficiency of multimodal retrieval information utilization, enhance the retrieval differentiation between similar CAD models, and improve the accuracy and reusability of CAD modeling sequence retrieval results. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a reconstruction-guided multimodal CAD retrieval method provided in this application embodiment; Figure 2 A schematic diagram of a CAD modeling sequence provided in an embodiment of this application; Figure 3 A schematic diagram of the architecture of a multimodal CAD retrieval model provided in an embodiment of this application; Figure 4 This is a schematic diagram of CAD modeling sequence retrieval results under an image query method provided in an embodiment of this application; Figure 5 This is a schematic diagram of CAD modeling sequence retrieval results under a text query method provided in an embodiment of this application; Figure 6 This is a schematic diagram of CAD modeling sequence retrieval results under a text-image joint query method provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] Figure 1 This is a flowchart illustrating a reconstruction-guided multimodal CAD retrieval method provided in an embodiment of this application. Figure 1 As shown, the multimodal CAD retrieval method includes S101 to S104.

[0020] This method can be executed by a server, workstation, CAD model retrieval platform, industrial design data management system, or electronic device with model training and retrieval capabilities. It is used to retrieve CAD modeling sequences that match the query information from candidate CAD modeling sequences based on text query information and / or image query information, enabling designers to perform parametric editing, structural modifications, and design reuse based on the retrieval results.

[0021] S101. Obtain multimodal sample data corresponding to the CAD model. The multimodal sample data includes text descriptions, rendered images, and CAD modeling sequences. The CAD modeling sequences are used to characterize the modeling process of the CAD model.

[0022] When executing S101, multiple CAD models can be retrieved from a CAD model library, design document management system, historical design project database, component resource platform, or manually annotated dataset. For each CAD model, a corresponding text description, rendered image, and CAD modeling sequence can be obtained. The text description can describe the CAD model's category, shape features, structural composition, purpose, local structure, or design requirements; the rendered image can be a two-dimensional image of the CAD model generated from one or more viewpoints; and the CAD modeling sequence can represent the ordered modeling process required to generate the CAD model.

[0023] like Figure 2As shown, a CAD modeling sequence can be understood as a series of ordered modeling operations and their corresponding modeling parameters used to generate a target CAD model. The CAD modeling sequence may include multiple sequentially arranged modeling steps, each of which may include a CAD command and its corresponding modeling parameters. For example, CAD commands may include modeling operations such as sketching, extrusion, rotation, cutting, chamfering, filleting, arraying, and Boolean operations; modeling parameters may include parameters such as dimensions, angles, coordinates, radius, direction, distance, and constraint relationships. Therefore, a CAD modeling sequence can not only represent the final geometric shape of a CAD model, but also its construction process, operation sequence, and parametric design intent.

[0024] S102. Perform feature extraction and fusion processing on the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features for characterizing the CAD model.

[0025] During execution of S102, feature encoding can be performed on the text description, rendered image, and CAD modeling sequence separately. Specifically, a text encoder can be used to extract semantic features from the text description to obtain text modal features; an image encoder can be used to extract visual features from the rendered image to obtain image modal features; and a CAD sequence encoder can be used to extract program structure features from the CAD modeling sequence to obtain CAD modal features. Subsequently, the text modal features, image modal features, and CAD modal features can be mapped to a unified feature space through corresponding projection heads, making the features of different modalities comparable and fusionable.

[0026] Furthermore, the mapped three types of modal features can be fused to obtain multimodal fusion features. These multimodal fusion features comprehensively represent the textual semantic information, visual appearance information, and modeling process information of the CAD model. Through these multimodal fusion features, the multimodal CAD retrieval model can simultaneously utilize multi-source information during subsequent training and retrieval processes, avoiding insufficient representation caused by relying solely on a single modality.

[0027] S103. Based on the multimodal fusion features, the CAD modeling sequence is reconstructed to obtain the reconstruction result. The reconstruction result is used to construct guiding constraints and jointly train the multimodal CAD retrieval model to obtain the trained multimodal CAD retrieval model.

[0028] When executing S103, multimodal fusion features can be input into the sequence decoding model. The sequence decoding model then predicts and reconstructs the corresponding CAD modeling sequence, obtaining a reconstruction result that reflects the difference between the predicted modeling sequence and the actual CAD modeling sequence. The reconstruction result can be used to characterize the degree to which the multimodal fusion features express the modeling operations, modeling parameters, and operation order in the CAD modeling sequence.

[0029] Furthermore, guiding constraints can be constructed using the reconstruction results and introduced into the joint training process of the multimodal CAD retrieval model. Through these guiding constraints, the multimodal CAD retrieval model can learn the modeling process information contained in the CAD modeling sequence while aligning features of different modalities, thus enabling the model to obtain feature representations more suitable for CAD modeling sequence retrieval.

[0030] During joint training, guided constraints can be combined with cross-modal matching-related training objectives to jointly optimize the text encoder, image encoder, CAD sequence encoder, projection head, feature fusion unit, and sequence decoding model. After training, a trained multimodal CAD retrieval model is obtained.

[0031] S104. Obtain the query information to be retrieved, input the query information to be retrieved into the trained multimodal CAD retrieval model, match it with the candidate CAD modeling sequence, and output the corresponding CAD modeling sequence retrieval result.

[0032] When executing S104, feature extraction can be performed on the candidate CAD modeling sequences to obtain candidate CAD sequence features, which are then stored in the candidate CAD modeling sequence feature library. The query information to be retrieved can include text query information and / or image query information. When the input is text query information, text query features can be extracted using a trained text encoder; when the input is image query information, image query features can be extracted using a trained image encoder; when the input includes both text and image query information, text query features and image query features can be extracted separately and then fused to obtain the query features.

[0033] After obtaining the query features, similarity matching can be performed between the query features and the candidate CAD sequence features in the candidate CAD modeling sequence feature library. The candidate CAD modeling sequences are then sorted according to the similarity, and finally, one or more CAD modeling sequences with high similarity are output as the CAD modeling sequence retrieval results. Since the output object is a CAD modeling sequence, the retrieval results can be used for subsequent parametric editing and design reuse.

[0034] This embodiment can incorporate text descriptions, rendered images, and CAD modeling sequences into the same retrieval framework, and use the reconstruction results of CAD modeling sequences to form model training constraints. This allows the model to learn the modeling process information of the CAD model while performing cross-modal semantic alignment, thereby improving the sufficiency of retrieval information utilization and the representation ability of CAD model editable and reusable features. At the same time, since the retrieval results are oriented towards CAD modeling sequences rather than just model appearance, it can improve the distinguishability between similar CAD models, thereby enhancing the accuracy and reusability of CAD modeling sequence retrieval results.

[0035] In one possible embodiment, the method steps shown in S101 can be implemented by S201 to S204, which are described in detail below.

[0036] S201. Obtain the text description data, rendered image data and CAD modeling sequence data corresponding to multiple CAD models respectively to obtain the initial three-modal data.

[0037] When executing S201, the CAD model can be used as the basic object, and text description data, rendered image data, and CAD modeling sequence data can be obtained for each CAD model. Text description data can originate from model names, design specifications, component descriptions, manually annotated content, or naturally generated content; rendered image data can be generated from the CAD model under preset viewpoints, preset lighting conditions, and preset rendering parameters; CAD modeling sequence data can be extracted from the modeling history, parametric design files, operation logs, or procedural modeling scripts of the CAD modeling software. Through this step, initial three-modal data, which has not yet been standardized in format and paired, can be obtained.

[0038] S202. The initial trimodal data is processed to obtain trimodal standard data with a unified format.

[0039] When executing S202, text description data can be cleaned, segmented, have special characters filtered, and have its length truncated or padded to obtain text descriptions with a uniform format. Similarly, rendered image data can be normalized in size, converted in color space, filtered for noise, filtered for viewpoint, or normalized in pixel value to obtain rendered images with a uniform format. Furthermore, CAD modeling sequence data can be normalized in terms of commands, parameters, invalid operations, sequence length, or encoded in terms of modeling commands to obtain CAD modeling sequences with a uniform format. This reduces format differences between data from different sources, providing a unified input for subsequent feature extraction.

[0040] S203. Based on the correspondence of the same CAD model, the text description, rendered image and CAD modeling sequence in the three-modal standard data are paired to obtain three-modal paired samples.

[0041] When executing S203, it is possible to determine whether text descriptions, rendered images, and CAD modeling sequences correspond to the same CAD model based on CAD model identifiers, file identifiers, design project identifiers, or manually established associations. For text descriptions, rendered images, and CAD modeling sequences belonging to the same CAD model, a one-to-one correspondence can be established, resulting in trimodal pairing samples. Each trimodal pairing sample includes mutually corresponding text descriptions, rendered images, and CAD modeling sequences.

[0042] S204. Construct the multimodal sample data based on the multiple trimodal paired samples.

[0043] When executing S204, multiple trimodal paired samples can be divided into training, validation, and test sets, or organized according to CAD model category, structure type, complexity, or application scenario to form multimodal sample data. This multimodal sample data can be used as training data for a multimodal CAD retrieval model, enabling the model to learn the correspondence between text descriptions, rendered images, and CAD modeling sequences.

[0044] This embodiment can standardize and pair text description data, rendered image data, and CAD modeling sequence data from different data sources, ensuring that the three modalities have a clear correspondence at the sample level. This provides a reliable data foundation for subsequent cross-modal feature alignment, reconstruction training, and retrieval matching, and improves the consistency and effectiveness of model training data.

[0045] In one possible embodiment, the method steps shown in S204 can be implemented by S301 to S304, which are described in detail below.

[0046] S301. The CAD modeling sequence is parsed to obtain multiple modeling steps arranged in sequence.

[0047] When executing S301, modeling history information, operation command information, or procedural modeling scripts in the CAD modeling sequence can be read, and the modeling operations within them can be parsed according to the generation order of the CAD model. The multiple modeling steps obtained after parsing are arranged in the actual modeling sequence, with each modeling step representing a modeling operation in the CAD model generation process. This step transforms the CAD modeling sequence from its original record form into a sequence structure that can be encoded and learned by the model.

[0048] In one possible implementation, such as Figure 2 As shown, a CAD modeling sequence can be understood as a set of ordered modeling operations and their corresponding modeling parameters used to generate a target CAD model. For the... The CAD model, the first The text description corresponding to a CAD model can be represented as follows: , No. The rendered image corresponding to a CAD model can be represented as follows: , No. The CAD modeling sequences corresponding to each CAD model can be represented as follows: The CAD modeling sequence consists of multiple sequentially arranged modeling steps, each including a CAD command and its corresponding parameter vector. Therefore, the CAD modeling sequence can not only represent the final geometric form of the CAD model, but also the modeling operations, the order of operations, and the parametric design intent.

[0049] S302. Generate corresponding command representation information and parameter representation information based on the CAD commands and modeling parameters in each modeling step.

[0050] When executing S302, the CAD commands within each modeling step can be identified and converted into command representation information. For example, different types of CAD commands can be categorized, vectorized, or embedded. Simultaneously, the modeling parameters corresponding to the CAD command can be extracted and processed through numerical normalization, discretization encoding, vectorization, or parameter masking to obtain parameter representation information. For cases where different CAD commands require varying numbers of parameters, padding, masking, or parameter field normalization can be used to ensure a unified data representation format for each modeling step.

[0051] S303. According to the order of the multiple modeling steps, the command representation information and the parameter representation information are combined to obtain a serialized modeling representation.

[0052] When executing S303, the command and parameter representation information corresponding to each modeling step can be combined into a modeling step representation. Then, multiple modeling step representations are arranged sequentially according to the order of the modeling steps to obtain a serialized modeling representation. This serialized modeling representation retains both the CAD commands and their parameters, as well as the sequential relationship between different modeling steps, thus reflecting the construction process of the CAD model and the parametric design intent.

[0053] S304. The serialized modeling representation is associated with and stored with the corresponding text description and rendered image to obtain the multimodal sample data.

[0054] When executing S304, the serialized modeling representation can be used as a standardized representation of the CAD modeling sequence and stored in association with the text description and rendered image corresponding to the same CAD model. This association can be implemented using sample indexes, model identifiers, or database records. Thus, each multimodal sample can include a text description, a rendered image, and a serialized modeling representation for subsequent model training.

[0055] This embodiment can refine the CAD modeling sequence into an ordered serialized modeling representation composed of CAD commands and modeling parameters, transforming the CAD modeling process from raw operation records into coded, reconstructable, and comparable program structure data, thereby enhancing the model's ability to learn from the editable and reusable features, construction logic, and parametric design intent of the CAD model.

[0056] In one possible embodiment, the method steps shown in S102 can be implemented by S401 to S405, which are described in detail below.

[0057] S401. Extract text features from the text description to obtain text modal features.

[0058] When executing S401, the text description can be input into a text encoder, which extracts semantic information from the text description. The text encoder can employ a Transformer coding network, a recurrent neural network, a pre-trained language model, or other coding models capable of processing natural language text. Text modal features can be used to characterize information such as the categorical semantics, structural description, functional description, and design requirements of the CAD model.

[0059] S402. Extract image features from the rendered image to obtain image modal features.

[0060] When executing S402, the rendered image can be input into the image encoder, which extracts the visual appearance information of the CAD model. The image encoder can employ convolutional neural networks, visual Transformer networks, multi-view image coding networks, or other image feature extraction models. Image modal features can be used to characterize the outline, local structure, spatial morphology, and visual similarity of the CAD model.

[0061] S403. Extract sequence features from the CAD modeling sequence to obtain CAD modal features.

[0062] When executing S403, a CAD modeling sequence or its serialized modeling representation can be input into a CAD sequence encoder, which extracts information about the CAD modeling process. The CAD sequence encoder can employ a Transformer encoding network, a sequence modeling network, or a graph-sequence hybrid encoding network. CAD modal features can be used to characterize the modeling commands, modeling parameters, operation sequence, and construction logic of the CAD model.

[0063] S404. Map the text modal features, the image modal features, and the CAD modal features to a unified feature space to obtain the corresponding trimodal embedding features.

[0064] When executing S404, text modal features, image modal features, and CAD modal features can be input into the corresponding feature mapping layer or projection head, respectively. This maps modal features with different dimensions and distributions into a unified feature space, resulting in text embedding features, image embedding features, and CAD embedding features. The three modal embedding features in the unified feature space can have the same embedding dimension and can undergo similarity calculation and fusion processing.

[0065] In one possible implementation, the modal index is recorded. Where t represents the text modality, v represents the image modality, and s represents the CAD modality. The corresponding inputs can be uniformly represented as: ,in, This represents the input of the i-th sample in mode m. This represents the text description corresponding to the i-th sample. This represents the rendered image corresponding to the i-th sample. This represents the CAD modeling sequence corresponding to the i-th sample.

[0066] Text encoders can be used separately. Image encoder and CAD sequence encoder Feature extraction is performed on the data of each modality to obtain the initial feature representation of each modality:

[0067] in, Represents the real number field. This represents the dimension of the encoder output features. This represents the initial feature representation of the i-th sample in mode m. This represents a sequence encoder with m modes.

[0068] Furthermore, the initial features of each mode are represented. Input the corresponding projection head respectively This yields the embedding representation in the unified feature space:

[0069] in, Represents the embedding dimension in the unified feature space. Let represent the embedding representation of the i-th sample in modality m. Through the above processing, the features corresponding to text descriptions, rendered images, and CAD modeling sequences can be mapped to the same feature space, providing a foundation for subsequent multimodal fusion and cross-modal matching.

[0070] S405. The three-modal embedding features are fused to obtain the multimodal fused features.

[0071] When executing S405, text embedding features, image embedding features, and CAD embedding features can be fused according to preset fusion rules. Fusion methods can include dimensionality-level fusion, weighted fusion, concatenation fusion, attention fusion, or a combination of at least two of these methods. Through fusion processing, multimodal fused features for comprehensively representing CAD models can be generated.

[0072] like Figure 3 As shown, the multimodal CAD retrieval model can include a text encoder, an image encoder, a CAD sequence encoder, a projection head, a feature fusion unit, and a sequence decoding model. The text encoder is used to extract features from text descriptions, the image encoder is used to extract features from rendered images, and the CAD sequence encoder is used to extract features from CAD modeling sequences. After each modal feature is mapped to a unified feature space by its corresponding projection head, the feature fusion unit generates multimodal fused features. The sequence decoding model reconstructs the CAD modeling sequence based on the multimodal fused features and uses the reconstruction results as constraints for model training.

[0073] This embodiment can extract features of CAD models from three perspectives: textual semantics, visual appearance, and modeling process. It also eliminates representation differences between different modalities by using a unified feature space mapping, enabling multimodal information to be fused and compared in the same feature space. This improves the ability to collaboratively utilize multi-source information during CAD model retrieval.

[0074] In one possible embodiment, the method steps shown in S405 can be implemented by S501 to S504, which are described in detail below.

[0075] S501. Based on the feature dimensions of the unified feature space, generate modality selection information corresponding to the text modality, image modality, and CAD modality, respectively.

[0076] When executing S501, modality selection information corresponding to the text modality, image modality, and CAD modality can be generated based on the number of dimensions of the embedded features in the unified feature space. Modality selection information can be represented as a binary mask vector, a dimension selection identifier, or a modality source index. For each feature dimension, the modality selection information can be used to indicate which modality—text modality, image modality, or CAD modality—should provide the feature value for that dimension.

[0077] In one possible implementation, to perform dimensionality-level fusion of the embedded features of each modality, binary mask vectors corresponding to the text modality, image modality, and CAD modality can be generated separately: ,in, Represents the embedding dimension in the unified feature space. , , These represent binary mask vectors corresponding to the text modality, image modality, and CAD modality, respectively. The mask vectors can be configured according to a preset mixing ratio. ,in These represent the fusion ratios of text, image, and modeling sequence modalities, respectively, and are sampled while satisfying the following conditions: .

[0078] S502. Based on the modality selection information, determine the unique source of the target modality corresponding to each feature dimension.

[0079] When executing S502, constraints can be applied to the modality selection information corresponding to the text modality, image modality, and CAD modality, ensuring that each feature dimension in the unified feature space corresponds to only one target modality source. In other words, for any feature dimension, its feature value can be provided by one of the text embedding feature, image embedding feature, or CAD embedding feature, rather than by multiple modalities simultaneously. This process avoids information mixing caused by the direct superposition of different modalities on the same dimension.

[0080] To ensure that each dimension in the unified feature space is provided with feature values ​​by only one mode, the mask vector satisfies the following constraints: in, It is a vector consisting entirely of 1s.

[0081] The above constraint means that, in each feature dimension of the unified feature space, only one modality has a valid mask value.

[0082] S503. According to the source of the target modality, select the feature values ​​of the corresponding dimension from the three-modal embedding features to obtain the dimension-level combined features.

[0083] When executing S503, each feature dimension in the unified feature space can be traversed dimension by dimension. Based on the source of the target modality corresponding to that feature dimension, feature values ​​for the corresponding dimension are selected from text embedding features, image embedding features, or CAD embedding features. The feature values ​​selected for each feature dimension are combined in dimensional order to obtain the dimension-level combined features. These dimension-level combined features originate from different modalities in different dimensions, thus enabling the fusion of trimodal information at the feature dimension level.

[0084] S504. Generate the multimodal fusion feature based on the dimensional combination feature.

[0085] When executing S504, dimensional combined features can be used as multimodal fusion features, or the dimensional combined features can be further normalized, linearly mapped, nonlinearly transformed, or attention-enhanced to obtain the final multimodal fusion features.

[0086] Furthermore, the random mask is used to analyze text modal features. Image modal features and CAD modal features By performing dimensional combination, we obtain the fused multimodal representation: ,in, This represents element-wise multiplication. This represents the multimodal fusion feature corresponding to the i-th sample. Through this method, the multimodal fusion feature can incorporate information from text description, rendered image, and CAD modeling sequence across different feature dimensions, thereby achieving the collaborative expression of trimodal information.

[0087] This embodiment can select feature values ​​of different modalities in a unified feature space on a dimensional basis, so that complementary information from text description, rendered image and CAD modeling sequence can be introduced into the same fusion feature; at the same time, since each feature dimension uniquely corresponds to a target modality source, it can reduce the interference caused by direct mixing of different modal features, thereby improving the expression stability and information complementarity of multimodal fusion features.

[0088] In one possible embodiment, the method steps shown in S103 can be implemented by S601 to S606, which are described in detail below.

[0089] S601. Input the multimodal fusion features into the sequence decoding model to obtain a predicted modeling sequence for the CAD modeling sequence.

[0090] When executing S601, multimodal fusion features can be used as conditional input to the sequence decoding model. The sequence decoding model can employ a Transformer Decoder. This model can progressively generate predictive modeling sequences based on the multimodal fusion features. At each prediction time, the sequence decoding model can predict the CAD commands and modeling parameters corresponding to the current modeling step based on the generated preceding modeling steps and the multimodal fusion features.

[0091] For example, for a CAD modeling sequence, the sequence decoding model can first predict the first CAD command and its modeling parameters, and then, based on the first prediction result, continue to predict the second CAD command and its modeling parameters, until a complete predictive modeling sequence is generated. Thus, the predictive modeling sequence can reflect the CAD modeling process recovered by the model based on multimodal fusion features.

[0092] In one possible implementation, the fused multimodal representation can be... Input TransformerDecoder to generate the CAD modeling sequence corresponding to the i-th sample. Autoregressive reconstruction is performed. The CAD modeling sequence consists of multiple sequentially arranged modeling steps, each of which includes CAD commands. and its corresponding parameter vector .

[0093] The Transformer Decoder models the conditional probability of CAD modeling sequences as follows:

[0094] in, This represents the length of the i-th CAD modeling sequence. Further, in the... At each modeling step, the joint conditional probability of the CAD command and its parameters is expressed as:

[0095] in, This represents the dimension of the parameter vector. Indicates the first The k-th parameter in each modeling step.

[0096] S602. Determine the prediction sequence representation based on the prediction CAD commands and prediction modeling parameters in the prediction modeling sequence.

[0097] When executing S602, the predictive modeling steps in the predictive modeling sequence can be organized. The predictive CAD commands and predictive modeling parameters in each predictive modeling step are combined into a corresponding predictive step representation, and then combined in the predictive order to obtain a predictive sequence representation. This predictive sequence representation can maintain the same data structure as the real sequence representation corresponding to the real CAD modeling sequence, so as to perform difference calculations.

[0098] S603. Determine the modeling sequence reconstruction error information based on the difference between the predicted sequence representation and the actual sequence representation corresponding to the CAD modeling sequence.

[0099] When executing S603, the predicted sequence representation can be compared with the true sequence representation to determine the modeling sequence reconstruction error information. The modeling sequence reconstruction error information may include CAD command prediction error, modeling parameter prediction error, sequence length error, or modeling step sequence error.

[0100] Furthermore, the reconstruction error of the program under given conditions can be defined as the length-normalized mask negative log-likelihood:

[0101] in, This represents the reconstruction error of the CAD modeling sequence S under conditional feature z. Indicates the length of the CAD modeling sequence. Indicates the first The CAD commands corresponding to each modeling step Indicates the first The k-th parameter in each modeling step.

[0102] This reconstruction error can be used to measure the ability of multimodal fusion features to represent commands, parameters, and operation sequences in a CAD modeling sequence.

[0103] S604. Based on the predicted modeling sequence and the modeling sequence reconstruction error information, the reconstruction result is obtained.

[0104] When executing S604, the predicted modeling sequence and the modeling sequence reconstruction error information can be used together as the reconstruction result of the CAD modeling sequence. The predicted modeling sequence characterizes the CAD modeling process recovered by the sequence decoding model based on multimodal fusion features, while the modeling sequence reconstruction error information characterizes the degree of deviation between the predicted modeling sequence and the actual CAD modeling sequence.

[0105] Specifically, the reconstruction results can include predicted CAD commands, predicted modeling parameters, predicted modeling step sequence, and reconstruction error information determined by the difference between the predicted sequence representation and the true sequence representation. By using the predicted modeling sequence and the reconstruction error information of the modeling sequence together as the reconstruction results, the ability of multimodal fusion features to express commands, parameters, and operation sequences in the CAD modeling sequence can be reflected from two aspects: "reconstruction content" and "reconstruction deviation."

[0106] S605. Construct a reconstruction guidance loss based on the reconstruction result, and determine the reconstruction guidance loss as the reconstruction guidance constraint in the guidance constraint. The reconstruction guidance constraint is used to constrain the feature representation space of the multimodal CAD retrieval model.

[0107] When executing S605, a reconstruction guidance loss can be constructed based on the reconstruction error information of the modeling sequence in the reconstruction results. The reconstruction guidance loss is used to measure the reconstruction deviation of the multimodal fusion features from the real CAD modeling sequence. When the difference between the predicted modeling sequence and the real CAD modeling sequence is large, the reconstruction guidance loss is large; when the difference between the predicted modeling sequence and the real CAD modeling sequence is small, the reconstruction guidance loss is small.

[0108] In one possible implementation, the reconstruction error information of the modeling sequence can be used as the reconstruction guidance loss, or the reconstruction guidance loss can be generated based on the reconstruction error information of the modeling sequence. The reconstruction guidance loss can be used as the reconstruction guidance constraint in the guidance constraint to constrain the feature representation space of the multimodal CAD retrieval model, so that the corresponding feature representation can not only express the cross-modal correspondence between text description, rendered image and CAD modeling sequence, but also retain the modeling command, modeling parameter and operation sequence information in the CAD modeling sequence.

[0109] By using the above-mentioned reconstruction guidance constraints, the multimodal fusion features can be constrained by the information of the CAD modeling process during training, avoiding the model from only learning appearance similarity or semantic similarity and ignoring the parametric construction logic reflected in the CAD modeling sequence.

[0110] S606. Based on the guiding constraints, the multimodal CAD retrieval model is jointly trained to obtain the trained multimodal CAD retrieval model.

[0111] When executing S606, the guiding constraints can be introduced into the joint training process of the multimodal CAD retrieval model. The joint training can simultaneously apply to the text encoder, image encoder, CAD sequence encoder, projection head, feature fusion unit, and sequence decoding model, so that the model is guided by the CAD modeling sequence reconstruction results while learning the matching relationships between different modalities.

[0112] Specifically, the feature representation space of the multimodal CAD retrieval model can be adjusted based on the guiding constraints to make the model's output feature representation more suitable for CAD modeling sequence retrieval. During training, when the reconstruction guiding constraints indicate that the multimodal fusion features cannot reconstruct the corresponding CAD modeling sequence well, parameter updates can be used to enhance the model's expression of modeling commands, modeling parameters, and operation sequences. When the reconstruction bias decreases, it indicates that the model's feature representation can more fully reflect the information of the CAD modeling process.

[0113] After multiple rounds of joint training, a multimodal CAD retrieval model can be obtained when the model meets the preset convergence conditions, reaches the preset number of training rounds, or the retrieval performance meets the preset requirements. Through this training method, the model can obtain feature representations that take into account both cross-modal matching capabilities and the expressive capabilities of the CAD modeling process, thereby improving the accuracy and editable reusability of subsequent CAD modeling sequence retrieval results.

[0114] In one possible embodiment, the guiding constraint further includes a difficult negative sample guiding constraint.

[0115] Prior to the method steps shown in S606, the method further includes Sa1 to Sa6, which are described in detail below.

[0116] Sa1. Determine the query modality features from the text modality features and / or image modality features corresponding to the training samples, and determine the candidate set of difficult negative samples based on the similarity between the query modality features and multiple candidate CAD modality features.

[0117] When executing Sa1, text modal features from the training samples can be used as query modal features, image modal features can be used as query modal features, or text modal features and image modal features can be fused together as query modal features. Candidate CAD modal features can be CAD modal features corresponding to multiple CAD modeling sequences in the same training batch.

[0118] The similarity between a query modal feature and multiple candidate CAD modal features can be calculated, such as cosine similarity, dot product similarity, or normalized feature similarity. For each query modal feature, the CAD modeling sequences corresponding to several candidate CAD modal features with high similarity (excluding its matching positive samples) can be identified as a set of difficult negative samples. The candidate CAD modeling sequences in this set are close to the query modal feature in the embedding space, and therefore are easily misclassified as matching samples by the model.

[0119] In one possible implementation, for query modal features Modal features of CAD ,in First, we can calculate the similarity matrix between the query modality and the CAD modality: ;in, This represents the normalized feature. For each query feature... From the CAD samples other than their matching positive samples, select the top samples with the highest similarity. Each sample constitutes a difficult negative sample candidate set: .

[0120] Sa2. Based on the query modal features, the candidate CAD modeling sequences in the difficult negative sample candidate set are reconstructed and verified to obtain the negative sample reconstruction information corresponding to each candidate CAD modeling sequence.

[0121] When executing S702, the query modal features can be used as input conditions, and the sequence decoding model can be used to reconstruct and verify the candidate CAD modeling sequences in the difficult negative sample candidate set. Specifically, the reconstruction error of the candidate CAD modeling sequence under the query modal feature conditions can be calculated as negative sample reconstruction information. The negative sample reconstruction information can be used to reflect the program structure differences between the candidate CAD modeling sequence and the target CAD modeling process corresponding to the query modal features.

[0122] In one possible implementation, for each CAD sample in the candidate set The reconstruction error of the difficult negative sample candidate under the query conditions is calculated using the frozen decoder: .

[0123] in, This represents the CAD modeling sequence corresponding to the j-th candidate CAD sample. Let represent the features of the i-th query sample under query mode q.

[0124] Sa3. Based on the similarity and the negative sample reconstruction information, determine the target difficult negative sample from the difficult negative sample candidate set.

[0125] When executing S703, the embedding space similarity between the candidate CAD modeling sequence and the query modal features, as well as the reconstruction information of the negative samples corresponding to the candidate CAD modeling sequence, can be comprehensively considered. If a candidate CAD modeling sequence has a high similarity to the query modal features, but its reconstruction error under the query conditions is large, it indicates that the candidate CAD modeling sequence is easily confused with positive samples in the embedding space, but its CAD modeling program structure is inconsistent with the real CAD modeling sequence corresponding to the query. Such samples can be identified as target hard negative samples.

[0126] In one possible implementation, to simultaneously consider embedding space similarity and program structure inconsistency, the candidate sets can be processed separately. The similarity and reconstruction energy within the range are normalized using Min-Max to obtain: Then, the final difficult negative sample is selected based on the normalized similarity and reconstruction energy. .

[0127] Therefore, negative samples that are similar to the query features in the embedding space but are inconsistent with the corresponding samples in the CAD modeling program structure can be preferentially selected as target difficult negative samples.

[0128] Sa4. Based on the target difficult negative sample and the negative sample reconstruction information, generate difficult negative sample processing information.

[0129] When executing Sa4, difficult negative sample processing information can be generated based on at least one of the following: the similarity to the target difficult negative sample, the negative sample reconstruction information, and the normalized reconstruction energy. The difficult negative sample processing information may include the target difficult negative sample identifier, the degree of reconstruction difference corresponding to the target difficult negative sample, the similarity information between the target difficult negative sample and the query modality features, the training weight reference information and / or training constraint reference information corresponding to the target difficult negative sample.

[0130] Specifically, if the target difficult negative sample has a high similarity to the query modality features, it indicates that the target difficult negative sample is likely to be misclassified as a matching sample in the feature representation space. If the negative sample reconstruction information corresponding to the target difficult negative sample is large, it indicates that there is a significant difference between the target difficult negative sample and the query sample in terms of CAD modeling program structure. Therefore, the similarity of the target difficult negative sample and the negative sample reconstruction information can be combined to generate difficult negative sample processing information for subsequent joint training, enabling the model to focus on samples that are similar in appearance or semantics but have inconsistent modeling programs during training.

[0131] Through the above processing, the information on difficult negative samples can not only reflect the degree of confusion of the target difficult negative sample in the embedding space, but also reflect the structural differences in the CAD modeling program between the target difficult negative sample and the query sample, thus providing a basis for the subsequent construction of guiding constraints for difficult negative samples.

[0132] Sa5. Construct the difficult negative sample guiding constraint based on the difficult negative sample processing information. The difficult negative sample guiding constraint is used to determine the training weight and / or training constraint of the target difficult negative sample in the joint training of the multimodal CAD retrieval model.

[0133] When executing Sa5, difficult negative sample guidance constraints can be constructed based on the difficult negative sample processing information. These difficult negative sample guidance constraints are used to apply differential training constraints to target difficult negative samples during the joint training of the multimodal CAD retrieval model, enabling the model to enhance its ability to learn the differences between target difficult negative samples and matching positive samples.

[0134] Specifically, the training weights of the target difficult negative sample can be determined based on the negative sample reconstruction information or the degree of reconstruction difference contained in the difficult negative sample processing information. Alternatively, the training constraint strength corresponding to the target difficult negative sample can be determined based on the similarity between the target difficult negative sample and the query modality features, as well as the negative sample reconstruction information. When the target difficult negative sample is relatively close to the query sample in the embedding space but differs significantly in the CAD modeling program structure, the training weights of the target difficult negative sample in joint training can be increased, and / or its corresponding training constraints can be strengthened, further widening the feature distance between the target difficult negative sample and the query sample in the multimodal CAD retrieval model.

[0135] By using the aforementioned challenging negative sample guidance constraints, the multimodal CAD retrieval model can focus on distinguishing CAD modeling sequences that are structurally similar and semantically close but have different modeling procedures during joint training. This reduces the false detection probability between similar CAD models and improves the fine-grained discrimination capability of CAD modeling sequence retrieval.

[0136] In one possible embodiment, the method steps shown in S606 can be implemented by S801 to S805.

[0137] S801. Determine the cross-modal contrastive learning objective based on the matching relationship between samples of different modalities.

[0138] When executing S801, cross-modal positive and negative sample pairs can be constructed based on the matching relationships between text descriptions, rendered images, and CAD modeling sequences corresponding to the same CAD model in the multimodal sample data. For example, text descriptions and CAD modeling sequences corresponding to the same CAD model can form positive sample pairs, as can rendered images and CAD modeling sequences corresponding to the same CAD model. Text descriptions and CAD modeling sequences or rendered images and CAD modeling sequences between different CAD models can form negative sample pairs.

[0139] Contrastive learning loss can be used to bring positive sample pairs closer to each other in a unified feature space and to push negative sample pairs further apart, thus determining the cross-modal contrastive learning objective. Cross-modal contrastive learning objectives can include alignment objectives between text and CAD, image and CAD, CAD and text, CAD and image, and between ordered modal pairs of text and image.

[0140] In one possible implementation, a joint loss function for training the model can be constructed based on multimodal representations, CAD modeling sequence reconstruction results, and hard negative sample mining results. This joint loss function includes at least a cross-modal contrastive learning loss. Modeling sequence reconstruction loss and the loss from difficult negative sample mining based on reconstruction validation. .

[0141] For any ordered mode pair (m,n), where and The contrastive learning loss from mode m to mode n is defined as:

[0142] Where B represents the batch size. Indicates the temperature coefficient. This represents the normalized embedded features. This represents the similarity function.

[0143] S802. Determine the reconstruction target of the modeling sequence based on the reconstruction guidance constraints.

[0144] During execution of S802, the reconstruction target of the modeling sequence can be determined based on the reconstruction guidance constraints. These constraints can be constructed from the reconstruction results of the CAD modeling sequence and are used to constrain the feature representation space of the multimodal CAD retrieval model, enabling the multimodal fusion features to support the reconstruction of the corresponding CAD modeling sequence. Specifically, the reconstruction guidance constraints can be reflected through modeling sequence reconstruction error information, which characterizes the difference between the predicted modeling sequence and the actual CAD modeling sequence. Therefore, the reconstruction target of the modeling sequence can be determined based on the reconstruction guidance constraints, allowing the model to further learn the CAD commands, modeling parameters, and modeling step sequence in the CAD modeling sequence during training.

[0145] The modeling sequence reconstruction loss is defined as:

[0146] Where B represents the batch size. Indicated in the multimodal table Under the condition, for the first CAD modeling sequence Reconstruction error.

[0147] By using the modeling sequence reconstruction loss as the modeling sequence reconstruction target, the multimodal CAD retrieval model can be further constrained by CAD modeling process information in addition to cross-modal alignment, thereby improving the expressive power of feature representation for editable CAD modeling sequences.

[0148] S803. Determine the target for mining difficult negative samples based on the difficult negative sample guiding constraints.

[0149] When executing S803, the target for mining difficult negative samples can be determined based on the difficult negative sample guidance constraint. The difficult negative sample guidance constraint can be constructed based on the target difficult negative sample, the similarity corresponding to the target difficult negative sample, and the negative sample reconstruction information, and is used to enhance the ability of the multimodal CAD retrieval model to distinguish easily confused negative samples during joint training.

[0150] Specifically, when the target difficult negative sample and the query modality features have a high similarity in the embedding space, but it is determined from the negative sample reconstruction information that there is a difference between it and the query sample in the CAD modeling program structure, the penalty weight and / or training constraint strength of the target difficult negative sample in the training process can be improved by the difficult negative sample guided constraint.

[0151] The loss for mining difficult negative samples based on reconstruction validation is defined as:

[0152] in, This represents the interval hyperparameter. This is the weighting adjustment coefficient. This indicates the similarity between the query modal features and the final hard negative sample. This indicates the similarity between the query modal features and the matching positive samples. This represents the normalized reconstruction energy corresponding to the final difficult negative sample. The larger the normalized reconstruction energy, the more significant the difference between the target difficult negative sample and the query sample in the CAD modeling program structure, and therefore it can be assigned a larger penalty weight during training.

[0153] By using the hard negative sample mining loss based on reconstruction verification as the hard negative sample mining target, the multimodal CAD retrieval model can focus on distinguishing candidate CAD modeling sequences that are similar to the query sample in appearance or semantics but have inconsistent CAD modeling program structures, thereby improving the fine-grained retrieval and discrimination ability between similar CAD models.

[0154] S804. The cross-modal contrastive learning objective, the modeling sequence reconstruction objective, and the difficult negative sample mining objective are weighted to obtain a joint training objective.

[0155] When executing S804, weight coefficients can be set for the cross-modal contrastive learning objective, the modeling sequence reconstruction objective, and the hard negative sample mining objective, and the three training objectives are weighted and summed to obtain a joint training objective. The weight coefficients can be preset or adjusted according to the training stage, the number of samples, the model convergence status, or the validation set retrieval effect. Through the joint training objective, the model can simultaneously meet the training requirements of cross-modal semantic alignment, CAD modeling sequence reconstruction, and hard negative sample distinguishability.

[0156] Furthermore, the contrastive learning loss, modeling sequence reconstruction loss, and hard negative sample mining loss can be weighted and summed to construct a joint training objective function:

[0157] in, and These are weighting coefficients used to adjust the contribution of each loss term to model training.

[0158] S805. Based on the joint training objective, update the parameters of the multimodal CAD retrieval model to obtain the trained multimodal CAD retrieval model.

[0159] When executing S805, multimodal sample data can be input into the multimodal CAD retrieval model in batches, and gradient backpropagation is performed based on the joint training objective to update the parameters of the text encoder, image encoder, CAD sequence encoder, projection head, feature fusion unit, and sequence decoding model. Parameter updates can be implemented using stochastic gradient descent, Adam, AdamW, or other optimizers.

[0160] Furthermore, training samples from the multimodal CAD sample set can be input into the model in batches, based on the constructed joint training objective function. The gradient backpropagation algorithm is used to jointly optimize the parameters of the text encoder, image encoder, CAD sequence encoder, projection head, and Transformer Decoder.

[0161] Let the model parameter set be In each training iteration, parameter updates are achieved by minimizing the joint training objective function, and its optimization objective is expressed as:

[0162] During parameter updates, an optimizer can be used to adjust the parameters based on the joint training objective function. The model is iteratively updated until it converges or reaches the preset number of training rounds, thus obtaining a trained multimodal CAD retrieval model.

[0163] This embodiment can incorporate cross-modal contrastive learning, CAD modeling sequence reconstruction, and difficult negative sample mining into the same training objective. This enables the model to establish semantic correspondences between text descriptions, rendered images, and CAD modeling sequences, learn program structure information in CAD modeling sequences, and enhance its ability to distinguish easily confused negative samples, thereby improving the overall retrieval accuracy and robustness of the multimodal CAD retrieval model.

[0164] In one possible embodiment, the method steps shown in S104 can be implemented by S901 to S905, which are described in detail below.

[0165] S901. Extract features from candidate CAD modeling sequences to obtain candidate CAD sequence features.

[0166] When executing S901, each candidate CAD modeling sequence from the candidate library can be input into the trained CAD sequence encoder and mapped to a unified feature space through the corresponding projection head to obtain the candidate CAD sequence features corresponding to each candidate CAD modeling sequence. The candidate CAD modeling sequences can come from enterprise CAD model libraries, parts libraries, historical design project libraries, or public CAD datasets.

[0167] S902. Construct a candidate CAD modeling sequence feature library based on the candidate CAD sequence features.

[0168] When executing S902, candidate CAD sequence features and their corresponding CAD modeling sequence identifiers, CAD model identifiers, file paths, category tags, or metadata can be associated and stored to obtain a candidate CAD modeling sequence feature library. This library can be deployed in a vector database, a retrieval index, or a dedicated feature retrieval system. To improve retrieval efficiency, an approximate nearest neighbor index can be created for the candidate CAD sequence features.

[0169] S903. Obtain the query information to be retrieved, and extract query features from the query information to obtain query features. The query information to be retrieved includes text query information and / or image query information.

[0170] When executing S903, users can input text query information, such as natural language descriptions of the target part's structure, shape, purpose, or local features; users can also input image query information, such as product renderings, sketches, reference images, or renderings of existing CAD models. When the query information includes text, text query features can be extracted using a trained text encoder; when the query information includes image, image query features can be extracted using a trained image encoder; when the query information includes both text and image, text query features and image query features can be extracted separately, and query features can be obtained through weighted summation, concatenation, dimensionality-level fusion, or attention fusion.

[0171] S904. Match the query features with the candidate CAD sequence features in the candidate CAD modeling sequence feature library to obtain the matching results.

[0172] When executing S904, the similarity between the query features and the candidate CAD sequence features can be calculated to obtain the matching results. Similarity can be calculated using cosine similarity, dot product similarity, Euclidean distance transformed score, or other vector matching methods. The matching results can include a similarity score between each candidate CAD modeling sequence and the query information to be retrieved.

[0173] S905. Sort the candidate CAD modeling sequences according to the matching results and output the CAD modeling sequence retrieval results.

[0174] When executing S905, candidate CAD modeling sequences can be sorted from highest to lowest similarity score, and one or more top-ranked CAD modeling sequences can be output as CAD modeling sequence retrieval results. The retrieval results may include the CAD modeling sequence, corresponding CAD model preview, model identifier, modeling step information, modeling parameter information, and similarity score. Designers can select a target CAD modeling sequence based on the retrieval results and perform parametric editing and design reuse based on that CAD modeling sequence.

[0175] In one possible embodiment, such as Figure 4 As shown, when the query information to be retrieved is image query information, the image query information can be input into the trained image encoder and the corresponding projection head to obtain image query features; then, the image query features are matched with the candidate CAD sequence features in the candidate CAD modeling sequence feature library for similarity, and the CAD modeling sequence retrieval results under the image query method are output in descending order of similarity.

[0176] In another possible embodiment, such as Figure 5As shown, when the query information to be retrieved is text query information, the text query information can be input into the trained text encoder and the corresponding projection head to obtain text query features; then, the text query features are matched with the candidate CAD sequence features in the candidate CAD modeling sequence feature library for similarity, and the CAD modeling sequence retrieval results under the text query method are output in descending order of similarity.

[0177] In yet another possible embodiment, such as Figure 6 As shown, when the query information to be retrieved includes both text query information and image query information, text query features and image query features can be extracted separately, and the text query features and image query features can be fused to obtain joint query features. Subsequently, the joint query features are matched with the candidate CAD sequence features in the candidate CAD modeling sequence feature library for similarity, and the CAD modeling sequence retrieval results under the text-image joint query method are output according to the similarity from high to low.

[0178] This embodiment can pre-build a feature library of candidate CAD modeling sequences, and in the retrieval stage, convert text query information and / or image query information into query features and match and sort them with the candidate CAD sequence features, thereby realizing rapid retrieval from multiple query inputs to editable CAD modeling sequences; this method can improve retrieval efficiency and make the retrieval results easier to modify and reuse in subsequent CAD models.

[0179] In one embodiment, a reconstruction-guided multimodal CAD retrieval system is provided, comprising: The data acquisition module is used to acquire multimodal sample data corresponding to the CAD model. The multimodal sample data includes text descriptions, rendered images, and CAD modeling sequences. The CAD modeling sequences are used to characterize the modeling process of the CAD model. The feature fusion module is used to extract and fuse features from the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features that characterize the CAD model. The reconstruction training module is used to reconstruct the CAD modeling sequence based on the multimodal fusion features, determine model training constraints based on the reconstruction results, train the multimodal CAD retrieval model, and obtain the trained multimodal CAD retrieval model. The retrieval output module is used to obtain the query information to be retrieved, input the query information to be retrieved into the trained multimodal CAD retrieval model, match it with the candidate CAD modeling sequence, and output the corresponding CAD modeling sequence retrieval result.

[0180] The specific implementation details of each module can be found in the above description of the limitations of the reconstruction-guided multimodal CAD retrieval method, and will not be repeated here.

[0181] In one possible embodiment, such as Figure 3 As shown, this embodiment also provides a reconstruction-guided multimodal CAD retrieval system. This system can be deployed on a server, cloud platform, industrial design data management system, CAD model retrieval platform, or local workstation. The system includes a data acquisition module, a feature fusion module, a reconstruction training module, and a retrieval output module.

[0182] The data acquisition module is used to acquire multimodal sample data corresponding to the CAD model. This multimodal sample data includes text descriptions, rendered images, and CAD modeling sequences, where the CAD modeling sequences characterize the modeling process of the CAD model. Specifically, the data acquisition module can acquire text description data, rendered image data, and CAD modeling sequence data corresponding to the CAD model from a CAD model library, design file library, or training dataset. It then performs format processing, standardization, and pairing processing on this data to form multimodal sample data. The data acquisition module can also parse the CAD modeling sequences to obtain a serialized modeling representation composed of CAD commands and modeling parameters.

[0183] The feature fusion module is used to extract and fuse features from the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features representing the CAD model. Specifically, the feature fusion module may include a text feature extraction unit, an image feature extraction unit, a CAD sequence feature extraction unit, a feature mapping unit, and a fusion unit. The text feature extraction unit outputs text modal features; the image feature extraction unit outputs image modal features; the CAD sequence feature extraction unit outputs CAD modal features; the feature mapping unit maps different modal features to a unified feature space; and the fusion unit fuses the three-modal embedding features in the unified feature space to obtain multimodal fusion features. In one possible implementation, the fusion unit can select feature values ​​of corresponding dimensions from the three-modal embedding features based on the unique target modality source corresponding to each feature dimension to obtain dimension-level combined features.

[0184] The reconstruction training module is used to reconstruct the CAD modeling sequence based on the multimodal fusion features, determine model training constraints based on the reconstruction results, and train the multimodal CAD retrieval model to obtain the trained multimodal CAD retrieval model. Specifically, the reconstruction training module may include a sequence decoding unit, a reconstruction error determination unit, a hard negative sample processing unit, and a joint training unit. The sequence decoding unit can predict the CAD modeling sequence based on the multimodal fusion features; the reconstruction error determination unit can determine the modeling sequence reconstruction error information based on the difference between the predicted modeling sequence and the real CAD modeling sequence; the hard negative sample processing unit can determine the hard negative sample candidate set based on the similarity between the query modality features and the candidate CAD modality features, and determine the target hard negative sample based on reconstruction verification; the joint training unit can train the multimodal CAD retrieval model based on the cross-modal contrastive learning objective, the modeling sequence reconstruction objective, and the hard negative sample mining objective.

[0185] The retrieval output module is used to acquire the query information to be retrieved, input the query information into the trained multimodal CAD retrieval model, match it with candidate CAD modeling sequences, and output the corresponding CAD modeling sequence retrieval results. Specifically, the retrieval output module can extract features from candidate CAD modeling sequences and construct a candidate CAD modeling sequence feature library; after receiving text query information and / or image query information, it extracts query features and performs similarity matching between the query features and candidate CAD sequence features in the candidate CAD modeling sequence feature library, and outputs the sorted CAD modeling sequence retrieval results based on the matching results.

[0186] The reconstruction-guided multimodal CAD retrieval system provided in this embodiment can form a reliable trimodal sample base through the data acquisition module, obtain multimodal fusion features that integrate text semantics, visual appearance, and modeling process through the feature fusion module, optimize the retrieval model through the reconstruction training module using the reconstruction results of CAD modeling sequence and information on handling difficult negative samples, and achieve matching output of text query information and / or image query information to candidate CAD modeling sequences through the retrieval output module, thereby improving the accuracy, fine-grained discrimination ability, and CAD model reuse efficiency of multimodal CAD retrieval.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A reconstruction-guided multimodal CAD retrieval method, characterized in that, The method includes: Obtain multimodal sample data corresponding to the CAD model. The multimodal sample data includes text descriptions, rendered images, and CAD modeling sequences. The CAD modeling sequences are used to characterize the modeling process of the CAD model. Feature extraction and fusion processing are performed on the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features that characterize the CAD model; The CAD modeling sequence is reconstructed based on the multimodal fusion features to obtain the reconstruction result. The reconstruction result is then used to construct guiding constraints and jointly train the multimodal CAD retrieval model to obtain the trained multimodal CAD retrieval model. Obtain the query information to be retrieved, input the query information to be retrieved into the trained multimodal CAD retrieval model, match it with the candidate CAD modeling sequence, and output the corresponding CAD modeling sequence retrieval result.

2. The reconstruction-guided multimodal CAD retrieval method according to claim 1, characterized in that, The acquisition of multimodal sample data corresponding to the CAD model includes: Obtain text description data, rendered image data, and CAD modeling sequence data corresponding to multiple CAD models to obtain initial trimodal data; The initial trimodal data is processed to obtain trimodal standard data with a unified format; Based on the correspondence of the same CAD model, the text description, rendered image and CAD modeling sequence in the trimodal standard data are paired to obtain trimodal paired samples; The multimodal sample data is constructed based on multiple paired trimodal samples.

3. The reconstruction-guided multimodal CAD retrieval method according to claim 2, characterized in that, The step of constructing the multimodal sample data based on multiple paired trimodal samples includes: The CAD modeling sequence is parsed to obtain multiple modeling steps arranged in sequence; Based on the CAD commands and modeling parameters in each modeling step, generate corresponding command representation information and parameter representation information; According to the order of the multiple modeling steps, the command representation information and the parameter representation information are combined to obtain a serialized modeling representation; The serialized modeling representation is associated with and stored with the corresponding text description and rendered image to obtain the multimodal sample data.

4. The reconstruction-guided multimodal CAD retrieval method according to claim 1, characterized in that, The step of extracting and fusing features from the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features for characterizing the CAD model includes: Text feature extraction is performed on the text description to obtain text modal features; Image feature extraction is performed on the rendered image to obtain image modal features; Sequence feature extraction is performed on the CAD modeling sequence to obtain CAD modal features; The text modal features, image modal features, and CAD modal features are mapped to a unified feature space to obtain the corresponding trimodal embedding features; The three-modal embedding features are fused to obtain the multimodal fused features.

5. The reconstruction-guided multimodal CAD retrieval method according to claim 4, characterized in that, The process of fusing the three-modal embedding features to obtain the multimodal fused features includes: Based on the feature dimensions of the unified feature space, modality selection information corresponding to the text modality, image modality, and CAD modality is generated respectively; Based on the modality selection information, the unique source of the target modality corresponding to each feature dimension is determined. Based on the source of the target modality, feature values ​​of the corresponding dimension are selected from the three-modal embedding features to obtain dimension-level combined features; The multimodal fusion feature is generated based on the dimensional combination features.

6. The reconstruction-guided multimodal CAD retrieval method according to claim 5, characterized in that, The process of reconstructing the CAD modeling sequence based on the multimodal fusion features to obtain a reconstruction result, and using the reconstruction result to construct guiding constraints to train the multimodal CAD retrieval model, resulting in a trained multimodal CAD retrieval model, includes: The multimodal fusion features are input into the sequence decoding model to obtain a predicted modeling sequence for the CAD modeling sequence; The prediction sequence representation is determined based on the prediction CAD commands and prediction modeling parameters in the prediction modeling sequence; Based on the difference between the predicted sequence representation and the actual sequence representation corresponding to the CAD modeling sequence, the modeling sequence reconstruction error information is determined; The reconstruction result is obtained based on the predicted modeling sequence and the reconstruction error information of the modeling sequence; Based on the reconstruction results, a reconstruction guidance loss is constructed, and the reconstruction guidance loss is determined as the reconstruction guidance constraint in the guidance constraint. The reconstruction guidance constraint is used to constrain the feature representation space of the multimodal CAD retrieval model. Based on the guiding constraints, the multimodal CAD retrieval model is jointly trained to obtain the trained multimodal CAD retrieval model.

7. The reconstruction-guided multimodal CAD retrieval method according to claim 6, characterized in that, The guiding constraints also include difficult negative sample guiding constraints; Before jointly training the multimodal CAD retrieval model based on the guiding constraints to obtain the trained multimodal CAD retrieval model, the method further includes: Query modal features are determined from the text modal features and / or image modal features corresponding to the training samples, and a set of candidate hard negative samples is determined based on the similarity between the query modal features and multiple candidate CAD modal features; Based on the query modal features, the candidate CAD modeling sequences in the difficult negative sample candidate set are reconstructed and verified to obtain the negative sample reconstruction information corresponding to each candidate CAD modeling sequence; Based on the similarity and the negative sample reconstruction information, a target difficult negative sample is determined from the difficult negative sample candidate set; Based on the target difficult negative sample and the negative sample reconstruction information, difficult negative sample processing information is generated; The difficult negative sample guidance constraint is constructed based on the difficult negative sample processing information. The difficult negative sample guidance constraint is used to determine the training weight and / or training constraint of the target difficult negative sample in the joint training of the multimodal CAD retrieval model.

8. The reconstruction-guided multimodal CAD retrieval method according to claim 7, characterized in that, The step of jointly training the multimodal CAD retrieval model based on the guiding constraints to obtain the trained multimodal CAD retrieval model includes: Based on the matching relationship between samples from different modalities, determine the cross-modal contrastive learning objective; Based on the reconstruction guidance constraints, the reconstruction target of the modeling sequence is determined; Based on the aforementioned difficult negative sample guiding constraints, the target for difficult negative sample mining is determined; The cross-modal contrastive learning objective, the modeling sequence reconstruction objective, and the hard negative sample mining objective are weighted to obtain a joint training objective; Based on the joint training objective, the parameters of the multimodal CAD retrieval model are updated to obtain the trained multimodal CAD retrieval model.

9. The reconstruction-guided multimodal CAD retrieval method according to claim 8, characterized in that, The process of obtaining the query information to be retrieved, inputting the query information into the trained multimodal CAD retrieval model, matching it with candidate CAD modeling sequences, and outputting the corresponding CAD modeling sequence retrieval results includes: Feature extraction is performed on the candidate CAD modeling sequences to obtain the candidate CAD sequence features; Based on the candidate CAD sequence features, a candidate CAD modeling sequence feature library is constructed; Obtain the query information to be retrieved, and extract query features from the query information to obtain query features. The query information to be retrieved includes text query information and / or image query information. The query features are matched with the candidate CAD sequence features in the candidate CAD modeling sequence feature library to obtain the matching results; The candidate CAD modeling sequences are sorted according to the matching results, and the CAD modeling sequence retrieval results are output.

10. A reconstruction-guided multimodal CAD retrieval system, characterized in that, The system includes: The data acquisition module is used to acquire multimodal sample data corresponding to the CAD model. The multimodal sample data includes text descriptions, rendered images, and CAD modeling sequences. The CAD modeling sequences are used to characterize the modeling process of the CAD model. The feature fusion module is used to extract and fuse features from the text description, the rendered image, and the CAD modeling sequence to obtain multimodal fusion features that characterize the CAD model. The reconstruction training module is used to reconstruct the CAD modeling sequence based on the multimodal fusion features, determine model training constraints based on the reconstruction results, train the multimodal CAD retrieval model, and obtain the trained multimodal CAD retrieval model. The retrieval output module is used to obtain the query information to be retrieved, input the query information to be retrieved into the trained multimodal CAD retrieval model, match it with the candidate CAD modeling sequence, and output the corresponding CAD modeling sequence retrieval result.

Images