Assembly model generation method and diffusion model training method based on large language model

Abstract

The invention provides an assembly model generation method based on a large language model and a diffusion model training method. The assembly model generation method based on the large language model comprises the steps that text description of a target assembly structure is acquired; based on the text description of the target assembly structure, determining part information and an assembly process of the target assembly structure; based on the part information, utilizing a diffusion model to generate a plurality of target part models; and generating a target assembly model of the target assembly structure by using the large language model based on the plurality of target part models and the assembly process. According to the technical scheme, the assembly relation of the assembly model can be kept in the generation result, the generation precision and details of the assembly model can be ensured, and a large-scale refined assembly model can be constructed.

Description

Technical Field

[0001] The present application relates to the technical field of computer-aided design, and in particular to an assembly model generation method and a diffusion model training method based on a large language model. Background Art

[0002] In the field of Computer Aided Design (CAD), CAD assembly models play a vital role in the functional realization and structural integrity of products. CAD assembly models simulate the dynamic interaction of various components in a computer environment, and can predict and solve problems that may arise in the production process, especially in aerospace, automotive simulation manufacturing and high-tech industries. However, building complex assembly models usually requires a high level of expertise and technical proficiency, and the cost is relatively high.

[0003] Boundary Representation (B-Rep) is a geometric representation method that is widely used in CAD model design because it can accurately describe complex geometric shapes. Currently, most complex CAD assembly models generated by B-Rep are based on point-line-surface representation, which cannot contain assembly relationships and has poor accuracy and details on large-scale models.

[0004] JoinABLe automates the generation of assembly models by learning the underlying B-Rep model joint information, mining assembly knowledge from existing CAD files without manual labeling or additional user input. However, this method cannot generate assembly models end-to-end and assembly correctness is difficult to generalize.

[0005] BRep-Gen represents another advanced technology path by using diffusion model for B-Rep model generation. This method converts data through structured latent geometry representation, encoding complex B-Rep models into tree data structures, thus achieving high flexibility and accuracy in 3D modeling. Nevertheless, this technology still faces problems such as the accumulation of errors in the generation process caused by topological complexity when dealing with large assembly models with complex topology structures in accurately capturing and reconstructing complex joints and interface relationships. Summary of the invention

[0006] In order to solve the above problems, the embodiments of the present application provide an assembly model generation method and a diffusion model training method based on a large language model.

[0007] In a first aspect, an embodiment of the present application provides an assembly model generation method based on a large language model, comprising: obtaining a text description of a target assembly structure; determining part information and an assembly process of the target assembly structure based on the text description of the target assembly structure; generating multiple target part models based on the part information using a diffusion model; and generating a target assembly model of the target assembly structure based on the multiple target part models and assembly processes using a large language model.

[0008] In combination with the first aspect, based on multiple target part models and assembly processes, a target assembly model of a target assembly structure is generated using a large language model, including: based on the assembly process, using a large language model to generate automated assembly instructions; based on the automated assembly instructions and multiple target part models, generating a target assembly model of a target assembly structure.

[0009] In combination with the first aspect, a target assembly model of a target assembly structure is generated based on automated assembly instructions and multiple target part models, including: based on the automated assembly instructions and the multiple target part models, using modeling software to generate an assembly model and obtain an isometric view of the assembly model; based on the isometric view and a text description of the target assembly structure, determining a visual-text difference measure between the isometric view and the text description of the target assembly structure; if the visual-text difference measure between the isometric view and the text description of the target assembly structure is greater than or equal to the target accuracy, determining the assembly model as the target assembly model of the target assembly structure.

[0010] In combination with the first aspect, if the visual-textual difference measure between the isometric view and the textual description of the target assembly structure is less than the target accuracy, the textual description of the assembly model is determined; and based on the textual description of the assembly model, the automated assembly instructions are updated using a large language model, and then the step of returning to the automated assembly instructions and the target part model, using modeling software to generate the assembly model and obtain the isometric view of the assembly model.

[0011] In combination with the first aspect, determining a visual-textual difference metric between an isometric view and a textual description of a target assembly structure includes: inputting the isometric view and the textual description of the target assembly structure into a visual question answering model to obtain a visual-textual difference metric between the isometric view and the textual description of the target assembly structure; determining a textual description of an assembly model includes: inputting the isometric view into a description extraction model to obtain a textual description of the assembly model.

[0012] In combination with the first aspect, a target assembly model of a target assembly structure is generated based on automated assembly instructions and multiple target part models, including: if an error occurs when running the automated assembly instructions, relevant information of the error is sent to a large language model, and the automated assembly instructions are updated using the large language model; based on the updated automated assembly instructions and multiple target part models, a target assembly model of the target assembly structure is generated.

[0013] In combination with the first aspect, before the step of generating multiple target part models based on the part information using the diffusion model, the method further includes: embedding the part information using the embedding generation model to obtain an embedded representation of the part information.

[0014] In a second aspect, an embodiment of the present application provides a diffusion model training method, including: obtaining a part data set, the part data set including several types of small sample part models and corresponding sample part text descriptions; constructing a redundant hierarchical tree representation of an initial diffusion model; based on several types of small sample part models and corresponding sample part text descriptions, fine-tuning the initial diffusion model to obtain a diffusion model, wherein the diffusion model is used to generate the above-mentioned target part model.

[0015] In combination with the second aspect, the initial diffusion model includes a multilayer perceptron. Based on several types of small sample part models and corresponding sample part text descriptions, the initial diffusion model is fine-tuned to obtain a diffusion model, including: training the multilayer perceptron through DreamBooth so that the loss function meets the preset conditions, and the loss function is:

[0016]

[0017] Where ω(t) represents the weight coefficient of the t-th diffusion step under the original conditions, ω'(t) represents the weight coefficient of the t-th diffusion step under additional conditions, Represents a pair ∈, t, c, ω CFG The mathematical expectation of the noise estimation result ∈ is given by ω CFG Control: ∈ = ω CFG ∈ φ (f, t, c)+(1-ω CFG )·∈ φ (f, t).

[0018] In conjunction with the second aspect, the initial diffusion model includes a BRep-Gen model.

[0019] In a third aspect, an embodiment of the present application provides an assembly model generation device, including: an acquisition module, a determination module and a generation module; the acquisition module is used to acquire a text description of a target assembly structure; the determination module is used to determine the part information and assembly process of the target assembly structure based on the text description of the target assembly structure; the generation module is used to generate multiple target part models based on the part information using a diffusion model; the generation module is also used to generate a target assembly model of the target assembly structure based on the target part model and the assembly process using a large language model.

[0020] In a fourth aspect, an embodiment of the present application provides a diffusion model training device, comprising: an acquisition module, a construction module and a training module; the acquisition module is used to acquire a part data set, the part data set comprising several classes of small sample part models and corresponding sample part text descriptions; the construction module is used to construct a redundant hierarchical tree representation of an initial diffusion model; the training module is used to fine-tune the initial diffusion model based on several classes of small sample part models and corresponding sample part text descriptions to obtain a diffusion model, wherein the diffusion model is used to generate the above-mentioned target part model.

[0021] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory, the memory being connected to the processor, the memory being used to store a computer program, and when the computer program is executed by the processor, the above-mentioned assembly model generation method based on a large language model or diffusion model training method is implemented.

[0022] In a sixth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned assembly model generation method or diffusion model training method based on a large language model is implemented.

[0023] In a seventh aspect, an embodiment of the present application provides a computer program product, including computer program instructions, which, when executed by a processor, implement the above-mentioned assembly model generation method or diffusion model training method based on a large language model.

[0024] Through the above technical solution, the target assembly model of the target part is generated in two stages: first, the target part model is generated using the diffusion model, and then the target part is assembled according to the assembly process (or assembly relationship) using the large language model to generate the target assembly model. This method can maintain the assembly relationship of the assembly model in the generated result, and can ensure the generation accuracy and details of the assembly model, and can build a large-scale refined assembly model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of an assembly model generation method provided in an embodiment of the present application.

[0026] Figure 2It is a flowchart of a method for generating a target assembly model of a target assembly structure provided by an embodiment of the present application.

[0027] Figure 3 It is a flowchart of a method for generating a target assembly model of a target assembly structure provided by an embodiment of the present application.

[0028] Figure 4 It is a flowchart of a diffusion model training method provided in an embodiment of the present application.

[0029] Figure 5 It is a structural schematic diagram of a target assembly model for generating a target assembly structure provided by an embodiment of the present application.

[0030] Figure 6 It is a structural schematic diagram of an assembly model generating device provided in one embodiment of the present application.

[0031] Figure 7 It is a structural schematic diagram of a diffusion model training device provided in one embodiment of the present application.

[0032] Figure 8 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In view of the technical problems mentioned in the background technology, in the first aspect, the embodiment of the present application provides a method for generating an assembly model based on a large language model. In the embodiment of the present application, the assembly model refers to a CAD assembly model, which can represent the structure assembled from multiple parts and the assembly relationship between multiple parts. The assembly model can simulate the dynamic interaction of each part in a computer environment, and can predict and solve problems that may arise in the production process.

[0035] Figure 1 FIG. 1 is a flow chart of an assembly model generation method provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps.

[0036] Step S110, obtaining a text description of the target assembly structure.

[0037] In the embodiment of the present application, the text description of the target assembly structure includes information that can describe or represent the target assembly structure, such as the name, category, structure description, function description, etc. of the target assembly structure.

[0038] Step S120 , determining part information and assembly process of the target assembly structure based on the text description of the target assembly structure.

[0039] In the embodiment of the present application, the text description of the target assembly structure is input into the Large Language Model (LLM) to output part information and assembly process. Optionally, the part information includes part name, part type, number of parts, etc. The assembly process includes assembly relationships between parts, assembly steps, etc.

[0040] Step S130: Generate multiple target part models using a diffusion model based on the part information.

[0041] Optionally, before step S130, the part information is embedded using an embedding representation generation model to obtain an embedded representation (i.e., a vector) of the part information. The embedding representation generation model includes a contrastive language-image pre-training model (CLIP) or other embedding representation generation models. After obtaining the embedded representation of the part information, the embedded representation of the part information is input into a diffusion model, and a plurality of target part models are generated using the diffusion model. In an embodiment of the present application, the diffusion model is a diffusion model trained using small sample data. For more descriptions on training diffusion models, please refer to the present application. Figure 4 The relevant description in will not be repeated here.

[0042] Step S140 , based on the multiple target part models and assembly processes, a target assembly model of the target assembly structure is generated using the large language model.

[0043] Optionally, based on the assembly process, the large language model is used to generate the automated assembly instructions. In an embodiment of the present application, the automated assembly instructions include Python macros that can be executed on modeling software (eg, CAD software).

[0044] Based on the automated assembly instruction and the multiple target part models, a target assembly model of the target assembly structure is generated. The automated assembly instruction is executed on the modeling software (e.g., CAD software), and the multiple target part models are assembled to obtain the target assembly model of the target assembly structure. For more description on generating the target assembly model of the target assembly structure, please refer to the present application. Figure 2 and Figure 3 The description is not repeated here.

[0045] In an embodiment of the present application, a diffusion model is used to generate a target part model, and then based on the target part model and the assembly process, a large language model is used to generate a target assembly model of a target assembly structure. That is, in an embodiment of the present application, a target assembly model of a target part is generated in two stages: first, a target part model is generated (determined by a diffusion model), and then the target part is assembled according to the assembly process (or assembly relationship) using a large language model to generate a target assembly model. This method can maintain the assembly relationship of the assembly model in the generated result, and can ensure the generation accuracy and details of the assembly model, and can build a large-scale refined assembly model.

[0046] Figure 2 FIG. 1 is a flow chart of a method for generating a target assembly model of a target assembly structure provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps.

[0047] Step S210: Generate automated assembly instructions based on the assembly process using a large language model.

[0048] In an embodiment of the present application, the assembly process is input into a large language model to generate automated assembly instructions. Optionally, the automated assembly instructions include Python macros that can be executed on modeling software (eg, CAD software).

[0049] Optionally, the text description of the target assembly structure is input into the large language model to determine the part information and assembly process (i.e. Figure 1 After step S120 in the above process, the large language model can directly generate automated assembly instructions according to the assembly process.

[0050] Step S220 , generating a target assembly model of a target assembly structure based on the automated assembly instruction and a plurality of target part models.

[0051] Optionally, the automated assembly instruction is executed on the modeling software (e.g., CAD software), the target part model is assembled, and the target assembly model of the target assembly structure is obtained. Optionally, if an error occurs when running the automated assembly instruction, the relevant information of the error is sent to the large language model, and the automated assembly instruction is updated using the large language model; based on the updated automated assembly instruction and the plurality of target part models, the target assembly model of the target assembly structure is generated.

[0052] Optionally, after the automated assembly instruction is executed to generate the assembly model, the generated assembly model is directly used as the target assembly model of the target assembly structure. In some embodiments, in order to ensure the accuracy of the assembly model, during the process of generating the target assembly model based on the automated assembly instruction and the target part model, the generated assembly model is verified and / or adjusted, such as Figure 3 shown.

[0053] Figure 3 FIG. 1 is a flow chart of a method for generating a target assembly model of a target assembly structure provided by an embodiment of the present application. Figure 3 As shown, the method includes the following steps.

[0054] Step S310 , based on the automated assembly instructions and the multiple target part models, using modeling software to generate an assembly model and obtain an isometric view of the assembly model.

[0055] In an embodiment of the present application, the automated assembly instruction is executed on the modeling software, and the target part model is assembled to obtain the assembly model. The isometric view of the assembly model is captured on the modeling software. Optionally, in an embodiment of the present application, PyAutoGUI is used to control the keyboard and mouse to automatically execute the steps of the automated assembly instruction, that is, it is only necessary to open the automated assembly instruction (i.e., Python macro) to execute the automated assembly instruction.

[0056] Step S320 : determining a visual-textual difference metric between the isometric view and the textual description of the target assembly structure based on the isometric view and the textual description of the target assembly structure.

[0057] Optionally, the text description of the isometric view and the target assembly structure is input into a visual question answering model (Visual Question Answering, VQA) to obtain a visual-text difference measure between the isometric view and the text description of the target assembly structure. Specifically, the visual question answering model converts the text description of the isometric view and the target assembly structure into vectors in the latent space, respectively, and then compares them to obtain the visual-text difference measure. In an embodiment of the present application, the visual question answering model includes CLIP-FlanT5-XL or other visual question answering models. CLIP-FlanT5-XL is a powerful solution that combines OpenAI's CLIP model and Google's Flan-T5-XL model to achieve multimodal (image and text) tasks.

[0058] Step S330, whether the visual-text difference measure between the isometric view and the text description of the target assembly structure is less than the target accuracy. Here, the target accuracy is a hyperparameter, which is a manually set value. If the visual-text difference measure between the isometric view and the text description of the target assembly structure is greater than or equal to the target accuracy, step S360 is executed to determine the assembly model as the target assembly model of the target assembly structure. That is, the generated assembly model meets the accuracy requirement.

[0059] If the visual-text difference measure between the isometric view and the text description of the target assembly structure is less than the target accuracy, step S340 is executed to determine the text description of the assembly model.

[0060] Specifically, the isometric view is input into the description extraction model to obtain a text description of the assembly model. In an embodiment of the present application, the description extraction model includes a Bootstrapping Language-Image Pre-training (BLIP) model or other models capable of converting images into text.

[0061] Step S350 , based on the text description of the assembly model, the large language model is used to update the automated assembly instructions.

[0062] The text description of the assembly model and the automated assembly instructions are input into the large language model, and the large language model is used to determine the difference between the text description of the assembly model and the text description of the target assembly structure, and based on the difference between the text description of the assembly model and the text description of the target assembly structure, the automated assembly instructions are updated, that is, new automated assembly instructions are generated.

[0063] After the automatic assembly instruction is updated, the process returns to step S310 and repeats the above steps until the score between the isometric view and the text description of the target assembly structure is greater than or equal to the target accuracy, thereby obtaining a target assembly model of the target assembly structure.

[0064] In an embodiment of the present application, the generated assembly model is verified, and the automated assembly instructions are adjusted using a large language model. The assembly model generated by this method has higher accuracy.

[0065] In a second aspect, the present application provides a diffusion model training method. The trained diffusion model is used to generate the above-mentioned target part model.

[0066] Figure 4 FIG. 1 is a flow chart of a diffusion model training method provided by an embodiment of the present application. Figure 4 As shown, the method includes the following steps.

[0067] Step S410, obtaining a part data set.

[0068] In an embodiment of the present application, the part data set includes several types of small sample part models and corresponding sample part text descriptions, and each type of small sample part model is 10-50 model files in STEP / IGES format.

[0069] Step S420: construct a redundant hierarchical tree representation of the initial diffusion model.

[0070] In an embodiment of the present application, the initial diffusion model is a boundary representation (B-Rep) model. The geometry and topology of the B-Rep model are unified into a hierarchical tree with a fixed graph topology to determine the input and output dimensions of the diffusion model. Therefore, a redundant hierarchical tree representation is constructed in which node features encode geometric information, and repeated nodes (i.e., nodes with almost the same features during the generation process) implicitly encode topological information. Among them, the redundant hierarchical tree representation represents a complete CAD entity with the root node, and the redundant hierarchical tree is divided into three levels from top to bottom: faces, edges, and vertices. For ease of understanding, an embodiment of the present application describes a method for constructing a redundant hierarchical tree representation. It can be understood that this description does not constitute a limitation on the present application. Specifically, constructing a redundant hierarchical tree representation includes the following steps.

[0071] First, geometric encoding is performed. Face nodes, edge nodes, or vertex nodes encode their geometric information into node features, including: global position, such as bounding box parameters or points; local shape details, such as latent space encoding. Specifically, it includes:

[0072] (1) Surface feature encoding F = [F z ,F p ].

[0073] Geometric features F z It is constructed based on UV grid sampling. Specifically, after the surface is parameterized, the two-dimensional parameter domain interval [u min ,u max ]×[v min ,v max ] to the geometry domain The mapping of (i.e., the parameter surface S(u,v)), where represents a two-dimensional real number. Then, for the current two-dimensional parameter domain interval [u min ,u max ]×[v min ,v max ] respectively perform uniform sampling in two dimensions, so that the two-dimensional parameter domain interval [u min ,u max ]×[v min ,v max ] is discretized into a two-dimensional grid, multiple discrete points in front of the current grid are obtained, and the sampling step size is determined: and Where M and N represent the number of sampling steps in each dimension, and M and N can be the same or different. The three-dimensional (3D) coordinates of all discrete points are connected to encode shape features The variational autoencoders (VAE) are used to further transform F s Compressed into latent space code F z .

[0074] Position feature F p =[x1,y1,z1,x2,y2,z2],F p is an axis-aligned bounding box that encloses all discrete points and encodes the two discrete points at the lower left and upper right corners. In addition, F p The two sets of 3D coordinates are normalized by mapping them to [-1,1]3 through an affine transformation.

[0075] (2) Edge feature E = [E zv ,E p ]coding.

[0076] Geometric features zv The U grid sampling method is used to construct it. Specifically, after the edge is parameterized, the parameter domain interval of the current edge is realized [u min ,u max ] to the geometry domain The mapping of is (parameter curve C(u)), where represents a one-dimensional real number. The parameterized current line is uniformly sampled so that the parameter domain interval [u min ,u max ] discretizes, and obtains And use VAE to convert E s Compressed into E z , E z The vertex 3D coordinates are further concatenated to form E zv .

[0077] Position feature E p Defined as the bounding box parameters surrounding the point.

[0078] Secondly, topological redundancy coding is performed. Topological redundancy coding defines the topological relationship between faces, edges, and vertices, and determines the input and output dimensions of the diffusion model by filling the number of faces and the variables around the faces in the B-Rep model to a fixed maximum length. Specifically, it includes:

[0079] (1) Construct a B-Rep model hierarchy tree. The hierarchy tree is divided into three levels from top to bottom: face, edge, and vertex, encoding face-edge-face and edge-vertex-edge adjacency relationships, where shared edges are replicated so that each parent face obtains a copy of the edge geometry as a child node. Similarly, shared vertices are replicated as child nodes of their parent edges.

[0080] (2) Determine the maximum branching factor. The maximum branching factor determines the maximum number of child nodes that the root node and the face node can reach. Randomly copy nodes until the maximum number of child nodes is reached.

[0081] Step S430: fine-tune the initial diffusion model based on several types of small sample part models and corresponding sample part text descriptions to obtain a diffusion model. The fine-tune diffusion model is used to generate a target part model based on the part information.

[0082] In the embodiment of the present application, low rank matrix fine-tuning (Low Rank Adaptation, LoRA) is used to add 4 up and down sampling multi-layer perceptrons to each bypass of the face position denoiser, face geometry denoiser, edge position denoiser and edge geometry denoiser of the original B-Rep model, that is, the initial diffusion model includes a multi-layer perceptron, and the initial diffusion model (that is, the B-Rep model) is fine-tuned based on the sample part text description and the small sample part model to obtain the diffusion model. In the embodiment of the present application, the face position denoiser, face geometry denoiser, edge position denoiser and edge geometry denoiser are U-Net under the spatial diffusion model (Latent Diffusion Model, LDM). In the embodiment of the present application, the model parameters are as follows.

[0083] In the embodiment of the present application, the noise estimator ∈ φ for:

[0084] ∈ φ (f,t,c)=UNet(f,t,c)+MLP(f,t,c),

[0085] in, is the current topological and geometric embedding representation, c is the sample part text description embedding representation obtained by the CLIP text encoder, t is the current diffusion step, and T is the total number of diffusion steps.

[0086] Based on the parts data set, each U-Net is fine-tuned with a small sample. Specifically, the multi-layer perceptron is trained through DreamBooth so that the loss function meets the preset conditions. The loss function is the category prior loss, and the loss function is:

[0087]

[0088] Where ω(t) represents the weight coefficient of the t-th diffusion step under the original conditions, and ω'(t) represents the weight coefficient of the t-th diffusion step under additional conditions. Represents a pair ∈, t, c, ω CFG The mathematical expectation of the noise estimation result ∈ is given by the text CFG Scale (Classier Free Guidance Scale) ω CFG Control: ∈ = ω CFG ∈ φ (f,t,c)+(1-ω CFG )·∈ φ (f,t).

[0089] In an embodiment of the present application, a diffusion model is trained using a small sample part data set, and there is no need for a complete model as training or fine-tuning data, which reduces the requirement for the number of training samples. When the trained diffusion model is used to generate a target part model, the generation accuracy is higher.

[0090] Figure 5 Schematic diagram of the structure of a target assembly model for generating a target assembly structure provided by an embodiment of the present application. Figure 5 As shown, the specific process includes the following:

[0091] (1) Given a text description y of the target assembly structure goal ;

[0092] (2) The text description of the target assembly structure y goal Input the large language model LLM, the large language model generates the type and number of parts Assembly process assy ;

[0093] (3) Types and quantities of parts given by LLM Use CLIP text encoder to get text description embedding representation, input the text description embedding representation into the (trained) diffusion model, and use the diffusion model to generate the target part model (B-Rep part model);

[0094] (4) Assembly process y given by LLM assy , use LLM to generate automated assembly instructions (e.g., Python macros) that can be executed on CAD software (e.g., FreeCAD software) marco );

[0095] (5) Execute Python macro y on FreeCAD software marco , if any error is encountered when running the macro, execute step (6), otherwise execute step (7);

[0096] (6) Pass the error-related information to LLM to correct the error and obtain the new Python macro y marco , proceed to step (5);

[0097] (7) When executing Python macro y marco After that, the assembly model is generated and the isometric view x1 of the assembly model is captured;

[0098] (8) Clip-FlanT5-XL is used as the VQA model to calculate the text description y of the target assembly structure goal Visual-text difference measure (VQAScore) S between isometric view x1VQA ;

[0099] (9) If S VQA ≥γ (γ is a hyperparameter representing the target accuracy), then execute step (13), otherwise execute step (10);

[0100] (10) Input the isometric view into BLIP2 to obtain the text description y of the assembly model j ;

[0101] (11) The text description y of the assembly model j , Python macro y macr o Input LLM; LLM finds the text description of the assembly model and the text description of the target assembly structure y goal and generates a modified Python macro y macro , proceed to step (5);

[0102] (12) Save the assembly model as the final complex assembly model (i.e., the target assembly model).

[0103] Figure 6 : is a structural block diagram of an assembly model generation device provided by an embodiment of the present application. Figure 6 As shown, the assembly model generating device 600 includes an acquisition module 610 , a determination module 620 and a generation module 630 .

[0104] The acquisition module 610 is used to acquire a text description of the target assembly structure.

[0105] The determination module 620 is used to determine the part information and assembly process of the target assembly structure based on the text description of the target assembly structure.

[0106] The generation module 630 is used to generate multiple target part models based on the part information using the diffusion model.

[0107] The generation module 630 is further configured to generate a target assembly model of a target assembly structure using a large language model based on multiple target part models and assembly processes.

[0108] Optionally, the generating module 630 is further configured to generate an automated assembly instruction based on the assembly process using a large language model; and to generate a target assembly model of a target assembly structure based on the automated assembly instruction and a plurality of target part models.

[0109] Optionally, the generation module 630 is also used to generate an assembly model and obtain an isometric view of the assembly model using modeling software based on the automated assembly instructions and the target part model; determine a visual-text difference measure between the isometric view and the text description of the target assembly structure based on the isometric view and the text description of the target assembly structure; if the visual-text difference measure between the isometric view and the text description of the target assembly structure is greater than or equal to the target accuracy, the assembly model is determined as the target assembly model of the target assembly structure.

[0110] Optionally, the generation module 630 is also used to determine the text description of the assembly model if the visual-text difference measure between the isometric view and the text description of the target assembly structure is less than the target accuracy; based on the text description of the assembly model, update the automated assembly instructions using the large language model, and then return to the step of generating the assembly model based on the automated assembly instructions and the target part model using the modeling software and obtaining the isometric view of the assembly model.

[0111] Optionally, the generation module 630 is also used to send relevant information of the error to the large language model if an error occurs when running the automated assembly instruction, and use the large language model to update the automated assembly instruction; based on the updated automated assembly instruction and multiple target part models, generate a target assembly model of the target assembly structure.

[0112] Optionally, the generation module 630 is further used to perform pre-embedding processing on the part information using the embedding generation model to obtain an embedded representation of the part information.

[0113] Optionally, the generating module 630 is further configured to input the isometric view and the text description of the target assembly structure into the visual question answering model to obtain a visual-text difference metric between the isometric view and the text description of the target assembly structure.

[0114] Optionally, the generating module 630 is further configured to input the isometric view into the description extraction model to obtain a text description of the assembly model.

[0115] The specific working principle and benefits of the assembly model generation device provided in the embodiment of the present application are similar to the specific working principle and benefits of the assembly model generation method provided in the embodiment of the present application, and will not be repeated here.

[0116] Figure 7 Schematic diagram of the structure of a diffusion model training device provided by an embodiment of the present application. Figure 7 As shown, the diffusion model training device includes an acquisition module 710 , a construction module 720 and a training module 730 .

[0117] The acquisition module 710 is used to acquire a part data set, which includes several types of small sample part models and corresponding sample part text descriptions.

[0118] The construction module 720 is used to construct a redundant hierarchical tree representation of the initial diffusion model.

[0119] The training module 730 is used to fine-tune the initial diffusion model based on several types of small sample part models and corresponding sample part text descriptions to obtain a diffusion model.

[0120] Optionally, the training module 730 is used to train the multilayer perceptron through DreamBooth so that the loss function meets preset conditions.

[0121] The specific working principle and benefits of the diffusion model training device provided in the embodiment of the present application are similar to the specific working principle and benefits of the diffusion model training method provided in the embodiment of the present application, and will not be repeated here.

[0122] Below, reference Figure 8 To describe an electronic device according to an embodiment of the present application. Figure 8 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0123] like Figure 8 As shown, the electronic device 800 includes one or more processors 801 and a memory 802 .

[0124] The processor 801 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0125] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may run the program instructions to implement the assembly model generation method based on a large language model or the diffusion model training method and / or other desired functions of the various embodiments of the present application described above. Various contents such as text descriptions including target assembly structures, target part models, etc. may also be stored in the computer-readable storage medium.

[0126] In one example, the electronic device 800 may further include: an input device 803 and an output device 804, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0127] The input device 803 may include, for example, a keyboard, a mouse, etc.

[0128] The output device 804 can output various information to the outside, including the target assembly model of the target assembly structure, the diffusion model, etc. The output device 804 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0129] Of course, to simplify, Figure 8 Only some of the components related to the present application in the electronic device 800 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 800 may also include any other appropriate components.

[0130] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the assembly model generation or diffusion model training method based on a large language model according to various embodiments of the present application described above in this specification.

[0131] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0132] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the assembly model generation or diffusion model training method based on a large language model according to various embodiments of the present application described above in this specification.

[0133] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0135] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0136] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0137] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0138] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating an assembly model based on a large language model, characterized in that: include: Get the text description of the target assembly structure; Determining part information and assembly process of the target assembly structure based on the text description of the target assembly structure; Based on the part information, a plurality of target part models are generated using a diffusion model; Based on the multiple target part models and the assembly process, a target assembly model of the target assembly structure is generated using a large language model.

2. The assembly model generation method based on a large language model according to claim 1, characterized in that: The step of generating a target assembly model of the target assembly structure by using a large language model based on the multiple target part models and the assembly process includes: Based on the assembly process, generating automated assembly instructions using the large language model; A target assembly model of the target assembly structure is generated based on the automated assembly instruction and the plurality of target part models.

3. The assembly model generation method based on a large language model according to claim 2, characterized in that: The step of generating a target assembly model of the target assembly structure based on the automated assembly instruction and the plurality of target part models comprises: Based on the automated assembly instructions and the plurality of target part models, using modeling software to generate an assembly model and obtain an isometric view of the assembly model; determining a visual-textual difference metric between the isometric view and the textual description of the target assembly structure based on the isometric view and the textual description of the target assembly structure; If a visual-text difference metric between the isometric view and the text description of the target assembly structure is greater than or equal to a target accuracy, the assembly model is determined as a target assembly model of the target assembly structure.

4. The assembly model generation method based on a large language model according to claim 3 is characterized in that: If a visual-textual difference metric between the isometric view and the textual description of the target assembly structure is less than the target accuracy, determining a textual description of the assembly model; as well as Based on the text description of the assembly model, the large language model is used to update the automated assembly instructions, and then the step of returning to the step of generating an assembly model based on the automated assembly instructions and the target part model using modeling software and obtaining an isometric view of the assembly model.

5. The assembly model generation method based on a large language model according to claim 4, characterized in that: Determining a visual-textual difference metric between the isometric view and the textual description of the target assembly structure includes: Inputting the isometric view and the text description of the target assembly structure into a visual question answering model to obtain a visual-text difference measure between the isometric view and the text description of the target assembly structure; Determining the text description of the assembly model includes: inputting the isometric view into a description extraction model to obtain the text description of the assembly model.

6. The assembly model generation method based on a large language model according to claim 2, characterized in that: The generating the target assembly model of the target assembly structure based on the automated assembly instruction and the plurality of target part models includes: if an error occurs when running the automated assembly instruction, sending relevant information of the error to the large language model, and updating the automated assembly instruction using the large language model; A target assembly model of the target assembly structure is generated based on the updated automated assembly instructions and the plurality of target part models.

7. The assembly model generation method based on a large language model according to claim 1, characterized in that: Before the step of generating a plurality of target part models by using a diffusion model based on the part information, the method further comprises: The part information is embedded using an embedding generation model to obtain an embedded representation of the part information.

8. A diffusion model training method, characterized in that: include: Acquire a part data set, wherein the part data set includes several types of small sample part models and corresponding sample part text descriptions; Constructing a redundant hierarchical tree representation of the initial diffusion model; Based on the several types of small sample part models and the corresponding text descriptions of the sample parts, the initial diffusion model is fine-tuned to obtain a diffusion model, wherein the diffusion model is used to generate the target part model described in any one of claims 1 to 7.

9. The diffusion model training method according to claim 8, characterized in that: The initial diffusion model includes a multi-layer perceptron, and fine-tuning the initial diffusion model based on the several types of small sample part models and the corresponding sample part text descriptions to obtain a diffusion model includes: The multilayer perceptron is trained by DreamBooth so that the loss function meets the preset conditions. The loss function is: Where ω(t) represents the weight coefficient of the t-th diffusion step under the original conditions, ω'(t) represents the weight coefficient of the t-th diffusion step under additional conditions, Represents the pair ∈,t,c,ω CFG The mathematical expectation of the noise estimation result ∈ is given by ω CFG Control: ∈ω CFG ∈ φ (f,t,c)+(1-ω CFG )·∈ φ (f,t).

10. The diffusion model training method according to claim 8, characterized in that: The initial diffusion model includes the BRep-Gen model.

Images