Three-dimensional model generation and editing system and method based on differentiatable instruction nodes

Abstract

The invention discloses a three-dimensional model generation and editing method and system based on differentiatable instruction nodes, and the system comprises a model input module, an instruction diagram generation module, an instruction sequence extraction module, a model reconstruction module and a model editing module, and the model input module carries out the preprocessing according to a three-dimensional model of a to-be-generated instruction sequence; the instruction diagram generation module constructs differentiable instruction nodes and an instruction diagram in sequence according to the preprocessed three-dimensional model, and optimizes the instruction diagram; an instruction sequence extraction module extracts instruction parameters and constructs an instruction sequence according to the optimized instruction diagram; the model reconstruction module regenerates a three-dimensional model according to the instruction sequence; and the model editing module realizes the non-destructive editing of the model by manually adjusting parameters in the instruction sequence through an interactive editing tool. According to the method, differentiable instruction nodes and instruction diagrams can be constructed from three-dimensional models (such as a CAD model, a grid model and a point cloud model) of any object, an instruction sequence is extracted and generated, and efficient three-dimensional model reconstruction and editing are performed by using the instruction sequence.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a method and system for generating and editing 3D models based on differentiable instruction nodes. Background Art

[0002] The characteristics and deficiencies of existing 3D model generation technologies are that there are significant limitations in understanding the intentions and creative thinking of human designers, resulting in obstacles to effective communication and collaboration between humans and machines in 3D content creation. The root cause of this problem is that the 3D representation forms (such as meshes, point clouds, Neural Radiance Fields (NeRF), and Gaussian splashes) targeted by existing 3D model generation technologies are usually low-rank and redundant particle representations, which are inconsistent with the forms used in the human design workflow, making it difficult for users to perform personalized and intuitive editing on the generated content, bringing great inconvenience to 3D content design. Existing 3D model generation technologies often have noise, incompleteness, or geometric distortion. The geometric structures of the generated models are usually not precise enough, and at the same time, they cannot reconstruct any type of object. Summary of the Invention

[0003] In view of the above deficiencies of the existing technology, the present invention proposes a method and system for generating and editing 3D models based on differentiable instruction nodes, which can construct differentiable instruction nodes and instruction graphs from 3D models of arbitrary objects (such as CAD models, mesh models, point cloud models, etc.), extract generation instruction sequences, and use the instruction sequences for efficient 3D model reconstruction and editing.

[0004] The present invention is implemented through the following technical solutions:

[0005] The present invention relates to a system for generating and editing 3D models based on differentiable instruction nodes, including: a model input module, an instruction graph generation module, an instruction sequence extraction module, a model reconstruction module, and a model editing module, wherein: the model input module preprocesses the 3D model for which an instruction sequence is to be generated; the instruction graph generation module constructs differentiable instruction nodes and an instruction graph in sequence according to the preprocessed 3D model, and optimizes the instruction graph; the instruction sequence extraction module extracts instruction parameters and constructs an instruction sequence according to the optimized instruction graph; the model reconstruction module regenerates a 3D model according to the instruction sequence; the model editing module performs non-destructive editing on the model by manually adjusting the parameters in the instruction sequence through an interactive editing tool.

[0006] The present invention relates to a method for generating and editing 3D models based on the above system, including:

[0007] Step 1: Preprocess the 3D model for which an instruction sequence is to be generated.

[0008] Step 2: Construct differentiable instruction nodes for the 3D model, and initialize the instruction parameters and differentiability of each instruction node, that is, determine the continuous parameter data type and discrete parameters of each instruction node.

[0009] Step 3: Stack the instruction nodes, set probability weights for each instruction node, and construct an instruction graph.

[0010] Step 4: Optimize the instruction graph, calculate and minimize the objective function, so as to update all parameters of the instruction graph.

[0011] Step 5: Extract the instruction parameters from the optimized instruction graph, retain the branch with the highest probability, and construct an instruction sequence.

[0012] Step 6: Regenerate the 3D model according to the instruction sequence.

[0013] Step 7: Adjust the instruction parameters or add or subtract instructions to obtain an edited instruction sequence. Technical Effects

[0014] The present invention uses a method of adopting differentiable instruction nodes to construct geometric modeling instructions for the model based on probability. Compared with the prior art, the present invention can generate a modeling instruction sequence aligned with human 3D designers, thereby supporting intuitive editing by users. Designers can make adjustments while maintaining the original design intent, ensuring the integrity of the design, and the 3D model reconstructed based on this modeling instruction sequence is geometrically accurate and noise-free. Brief Description of the Drawings

[0015] Figure 1 It is a schematic diagram of the system of the present invention;

[0016] Figure 2 It is a schematic diagram of the system of the present invention;

[0017] Figure 3 It is a schematic structural diagram of constructing an instruction node;

[0018] Figure 4 It is a schematic structural diagram of constructing an instruction graph;

[0019] Figure 5 It is a schematic diagram of extracting an instruction sequence and reconstructing a 3D model;

[0020] Figure 6 It is a schematic diagram of editing an instruction sequence. Detailed Embodiment

[0021] Such as Figure 1As shown in the figure, this embodiment relates to a three-dimensional model generation and editing system based on differentiable instruction nodes, including: a model input module, an instruction graph generation module, an instruction sequence extraction module, a model reconstruction module, and a model editing module, where: the model input module preprocesses the three-dimensional model for which an instruction sequence is to be generated; the instruction graph generation module sequentially constructs differentiable instruction nodes and an instruction graph based on the preprocessed three-dimensional model, and optimizes the instruction graph; the instruction sequence extraction module extracts instruction parameters and constructs an instruction sequence based on the optimized instruction graph; the model reconstruction module regenerates a three-dimensional model based on the instruction sequence; and the model editing module edits the model by manually adjusting the parameters in the instruction sequence through an interactive editing tool.

[0022] The described instruction graph generation module system includes: an instruction node construction unit, an instruction graph construction unit, and an instruction graph optimization unit, where: the instruction node construction unit initializes the instruction differentiability of the instruction parameters of 14 preset instructions to construct instruction nodes; the instruction graph construction unit sets probability weights for each instruction node based on the constructed instruction nodes, and superimposes the instruction nodes to obtain an instruction graph; the instruction graph optimization unit calculates an objective function based on the instruction graph and the input model and optimizes the instruction graph.

[0023] The described instruction sequence extraction module includes: a weight comparison unit, a branch extraction unit, and an instruction node extraction unit, where: the weight comparison unit searches for the branch with the largest weight in each instruction node and the instruction nodes in the instruction graph with weights higher than a preset threshold; the branch extraction unit extracts the branch with the largest weight in each instruction node; and the instruction node extraction unit extracts the instruction nodes in the instruction graph with weights higher than a preset threshold.

[0024] As Figure 2 shown in the figure, this embodiment relates to a method for generating a three-dimensional model based on differentiable instruction nodes using the above system, including the following steps:

[0025] Step 1: Select an arbitrary three-dimensional model for which an instruction sequence is to be generated and import it into the system. After using it as the initial input data, convert the model into a point cloud format.

[0026] The described arbitrary three-dimensional model includes, but is not limited to, a geometric model created through CAD software or a point cloud model obtained through a scanning device.

[0027] Step 2: Construct differentiable instruction nodes: Treat each instruction node as a geometric transformation function. Using the current state (such as geometric data and topological structure) of the three-dimensional model received in Step 1 as the input, after applying the instruction, return the modified state of the three-dimensional model, thereby obtaining the instruction parameter information and differentiability information of instruction nodes with different functions.

[0028] The instruction nodes described in this embodiment include: instruction nodes such as Subdivide, Extrude, Inset, Bevel, SimpleDeform, Boolean, Mirror, Loopcut, KnifeCut, BridgeEdgeLoops, Rotate, Scale, Transform, and VertexMove.

[0029] The described instruction parameter information includes:

[0030] As Figure 3 shown, the described differentiability information includes:

[0031] a) For instruction nodes with only continuous parameters Differentiability is achieved through automatic differentiation, where: is a certain type of instruction node, x is the model state of the input node, is a vector of continuous parameters, and these parameters are regarded as continuous variables.

[0032] For example: The Subdivide Node receives a continuous parameter representing the subdivision level as input and outputs a mesh with the corresponding surface refinement level.

[0033] b) For instruction nodes with only discrete parameters It is constructed to include a branching mechanism. Each value of the discrete parameter corresponds to a branch, and by applying the Softmax function, it is ensured that the branch with the highest probability is selected during the inference process, thus maintaining differentiability, where: d is the discrete parameter, and d i is a certain value of the discrete parameter.

[0034] For example: The Bevel Node has branches corresponding to different numbers of segments (such as 1, 2, or 3 segments), and each branch is assigned a probability.

[0035] c) For instruction nodes with both continuous parameters and discrete parameters Then, through the branching mechanism, different values of the discrete parameter are used, and each branch has independent continuous parameter values.

[0036] For example: The continuous parameters of the Extrude Node are the length, width, and direction of the face extrusion, and the discrete parameter is the index of the extruded face.

[0037] d) For instruction nodes without parameters Only perform corresponding type transformations on the input model.

[0038] For example: bridging loop edges.

[0039] Step 3. Construct a differentiable instruction graph by stacking instruction nodes: As Figure 4 shown, use the output of each instruction node as the input of the next node: where: x is the initial state (a cube with side length 1), y is the output state of the instruction graph, and the node weight ω i is the probability of the existence of this node. That is, when the probability is low, the instruction graph tends to skip this node.

[0040] The differentiability of the described differentiable instruction graph depends on the differentiability of its individual nodes, enabling the calculation of the gradient of the entire graph using the chain rule.

[0041] Step 4. Optimize the differentiable instruction graph: Using the minimization of geometric error as the objective function, calculate the gradient using backpropagation and update the node parameters and node weights. Specifically: where: S1 is the point cloud obtained from the input model in Step 1, and S2 is the 3D model output by the instruction graph.

[0042] Step 5. Extract instruction parameters and construct an instruction sequence: After removing the nodes in the differentiable instruction graph optimized in Step 4 with probability weights lower than the preset instruction weight threshold, select the branch with the highest probability for each node in the instruction nodes with only discrete parameters and the instruction nodes with both continuous and discrete parameters, and construct an instruction sequence including the detailed parameters of each instruction.

[0043] For example: the length of face extrusion, the angle of rotation, or the type of Boolean instruction.

[0044] The described instruction sequence is preferably stored in an editable format for easy subsequent modification and reuse.

[0045] Step 6. Reconstruct the 3D model: According to the instruction sequence extracted in Step 5, gradually construct the geometric shape and topological structure of the 3D model by sequentially executing each instruction in the instruction sequence, and generate a 3D model that can be directly used for rendering, 3D printing, or other downstream applications.

[0046] Preferably, edit the generated 3D model, including modifying the geometric shape, adjusting the instruction parameters, or adding new instruction nodes. By adjusting the parameters in the instruction sequence, human designer-oriented editing is achieved to ensure the integrity and design intent of the model during the modification process.

[0047] After specific actual experiments, under the setting where the number of instruction layers is 25 and the instruction weight threshold = 0.2, the instruction sequence obtained by inputting any 3D model is as follows Figure 5 shown. The results obtained by editing the 3D model by adjusting the instruction parameters, adding or deleting instructions are as follows Figure 6 shown. The setting environment of the above embodiments is a Linux server configured with 64GB of memory space, a GTX 1080 graphics card, and an Intel i7-8700K CPU. In this embodiment, the chamfer distance and normal consistency in 3D model reconstruction on the 3D model dataset are used as objective evaluation indicators to evaluate the effect of 3D model reconstruction. The lower the chamfer distance and the higher the normal consistency, the closer the 3D model reconstructed by the instruction is to the input 3D model, that is, the generated instruction sequence can better restore the input 3D model. As shown in Table 2, this method is compared with the prior art MeshAnythingV2 and Point2CAD.

[0048] Table 2 Index / Method MeshAnythingV2 Point2CAD This method Chamfer distance <![CDATA[2.857*10 -2 > <![CDATA[9.349*10 -2 > <![CDATA[0.823*10 -2 > Normal vector consistency 0.843 0.757 0.961

[0049] As shown in Table 2, MeshAnythingV2 generates the final 3D model by generating each mesh patch one by one, resulting in error accumulation and large geometric errors. Point2CAD performs pre-segmentation processing on the input 3D model, fits B-rep surfaces for each segmentation module, and is highly dependent on the segmentation results and cannot adaptively adjust for complex 3D model inputs.

[0050] In summary, compared with the prior art methods, the present invention includes 14 types of instruction function types, and through the differentiability of all instruction nodes and the branch mechanism, the optimal instruction and corresponding instruction parameters can be automatically obtained at each step, achieving adaptability to any 3D model. Thus, while obtaining a modeling instruction sequence that highly matches the input 3D model, the reconstruction error is significantly reduced.

[0051] The above specific embodiments can be locally adjusted by those skilled in the art in different ways without departing from the principles and purposes of the present invention. The protection scope of the present invention is subject to the claims and is not limited by the above specific embodiments, and all implementation solutions within its scope are subject to the constraints of the present invention.

Claims

1. A three-dimensional model generation and editing system based on differentiable instruction nodes, characterized in that, Including: A model input module, an instruction graph generation module, an instruction sequence extraction module, a model reconstruction module, and a model editing module. Among them: The model input module performs preprocessing on the three-dimensional model for which the instruction sequence is to be generated; the instruction graph generation module constructs differentiable instruction nodes and an instruction graph in sequence based on the preprocessed three-dimensional model, and optimizes the instruction graph; the instruction sequence extraction module extracts instruction parameters and constructs an instruction sequence based on the optimized instruction graph; the model reconstruction module regenerates a three-dimensional model according to the instruction sequence; the model editing module performs non-destructive editing on the model by manually adjusting the parameters in the instruction sequence through an interactive editing tool.

2. The 3D model generation and editing system based on differentiable instruction nodes according to claim 1, characterized in that, The described instruction graph generation module system includes: an instruction node construction unit, an instruction graph construction unit, and an instruction graph optimization unit. Among them: The instruction node construction unit initializes the instruction differentiability of the instruction parameters of 14 preset instructions to construct instruction nodes; the instruction graph construction unit sets probability weights for each instruction node according to the constructed instruction nodes, and superimposes the instruction nodes to obtain an instruction graph; the instruction graph optimization unit calculates an objective function based on the instruction graph and the input model and optimizes the instruction graph.

3. The three-dimensional model generation and editing system based on differentiable instruction nodes according to claim 1, characterized in that, The described instruction sequence extraction module includes: a weight comparison unit, a branch extraction unit, and an instruction node extraction unit. Among them: The weight comparison unit searches for the branch with the largest weight in each instruction node and the instruction nodes in the instruction graph with weights higher than a preset threshold; the branch extraction unit extracts the branch with the largest weight in each instruction node; the instruction node extraction unit extracts the instruction nodes in the instruction graph with weights higher than a preset threshold.

4. A method for generating and editing a three-dimensional model based on differentiable instruction nodes of the system according to any one of claims 1-3, characterized in that, Including: Step 1: Perform preprocessing on the three-dimensional model for which the instruction sequence is to be generated; Step 2: Construct differentiable instruction nodes for the three-dimensional model, and initialize the instruction parameters and differentiability of each instruction node, that is, determine the continuous parameter data type and discrete parameters of each instruction node; Step 3: Superimpose the instruction nodes, set probability weights for each instruction node, and construct an instruction graph; Step 4: Optimize the instruction graph, calculate and minimize the objective function, so as to update all parameters of the instruction graph; Step 5: Extract instruction parameters from the optimized instruction graph, retain the branch with the largest probability, and construct an instruction sequence; Step 6: Regenerate a three-dimensional model according to the instruction sequence.

5. The three-dimensional model generation and editing method according to claim 4, characterized in that, The described instruction nodes include: surface subdivision, face extrusion, face interpolation, chamfering, simple deformation, boolean, mirror, loop cut, cut, bridge loop edge, rotation, scale, translation, and vertex displacement.

6. The three-dimensional model generation and editing method according to claim 5, characterized in that, The described instruction parameter information includes: The continuous parameter of surface subdivision is: the number of subdivision layers and there are no discrete parameters; The continuous parameters of face extrusion are: length, width, direction, and the discrete parameter is: the index of the extruded face; The continuous parameter of face interpolation is: width and there are no discrete parameters; Chamfering has no continuous parameters and the discrete parameter is: the number of segments; The continuous parameter of simple deformation is: angle, and the discrete parameter is: the deformation axis; Boolean has no continuous parameters and the discrete parameter is: boolean type; Mirror has no continuous parameters and no discrete parameters; Loop cut has no continuous parameters and the discrete parameter is: the cutting edge; Cutting has no continuous parameters and the discrete parameter is: cutting edge; Bridging loop edges have no continuous parameters and no discrete parameters; The continuous parameter of rotation is: angle and has no discrete parameters; The continuous parameter of scaling is: scaling factor and has no discrete parameters; The continuous parameter of translation is: translation coefficient and has no discrete parameters; The continuous parameter of vertex displacement is: displacement vector and has no discrete parameters.

7. The three-dimensional model generation and editing method according to claim 5 or 6, characterized in that The said differentiability information includes: a) For instruction nodes with only continuous parameters Differentiability is achieved through automatic differentiation, where: is an instruction node of a certain type, x is the model state of the input node, is a vector of continuous parameters, and these parameters are regarded as continuous variables; b) For an instruction node with only discrete parameters is constructed to include a branching mechanism, where each value of the discrete parameter corresponds to a branch, and by applying the Softmax function, it ensures that the branch with the highest probability is selected during the inference process, thereby maintaining differentiability, where: d is the discrete parameter, and d i is a certain value of the discrete parameter; c) For an instruction node that has both continuous parameters and discrete parameters The branch mechanism is used for different values of the discrete parameters, and each branch has independent values of the continuous parameters; d) For instruction nodes without parameters Only perform corresponding type conversions on the input model.

8. The three-dimensional model generation and editing method according to claim 4, characterized in that The optimization described in Step 4 means: taking the minimization of geometric error as the objective function, using backpropagation to calculate the gradient and update the node parameters and node weights, specifically: where: S1 is the point cloud obtained from the input model in Step 1, and S2 is the 3D model output by the instruction graph.

9. The three-dimensional model generation and editing method according to claim 4, characterized in that The editing of the instruction sequence is achieved by adjusting the instruction parameters or adding or subtracting instructions.

Images