Progressive sketch model reconstruction method based on template

Abstract

The invention discloses a progressive sketch model reconstruction method based on a template. The method comprises the following steps: constructing a sketch data set containing multiple modeling operation semantics; a sketch classification module is constructed and trained, and modeling operation categories of sketches are identified; constructing and training a parameter extraction module, and calculating sketch parameters corresponding to different types of modeling operations; the sketch data set, the sketch classification module and the parameter extraction module are integrated into CAD software, an interactive CAD modeling system is constructed, and after the system identifies modeling operation categories of input sketches, the system calls the corresponding parameter extraction module to calculate corresponding sketch parameters, or directly calls sketch parameters of other modeling templates to construct a sketch model. And carrying out modeling operation so as to realize CAD model reconstruction based on the freehand sketch. According to the method, based on the constructed sketch data set, more modeling operations can be achieved, a good classification effect is achieved for the free-hand sketches with sparse and diverse operation semantics, and a sketch analysis algorithm and a sketch reconstruction algorithm are simplified.

Description

Technical Field

[0001] The present invention belongs to the field of computer graphics and relates to a template-based progressive sketch model reconstruction method. Background Art

[0002] In the industrial design process, sketching is the initial stage for designers to express their design intentions, featuring speed and flexibility. The later 3D modeling stage instantiates the design intentions and requires professional software operation skills, presenting a relatively high threshold. Sketch-Based Modeling (SBM) aims to reduce the complexity of the conversion from sketches to 3D models through automated techniques, improve design efficiency, and enable users without a background to design products on their own.

[0003] Early methods in the SBM field relied on the analysis of wireframe diagrams from a fixed perspective. Limited by the limitations of geometric reasoning, they were difficult to handle complex shapes. After the rise of deep learning, methods based on convolutional neural networks have improved 3D reconstruction capabilities, but the generated results are mostly non-editable static meshes such as voxels and point clouds, and their generalization capabilities are limited, only applicable to specific categories such as chairs and cars. Most existing deep learning models generate meshes lacking a parametric structure, such as the stretching and rotation features in CAD, and key parameters such as dimensions and curvatures cannot be directly modified. The existing technology is the method of sketch-based progressive generation modeling (Sketch2CAD), where users can interactively generate very complex parametric models. However, the existing methods are only limited to simple four modeling methods, restricting the diversity and practicality of the generated models and unable to meet the generation requirements of complex geometric structures in industrial design. Moreover, classification and segmentation rely on the modeling information generated in the previous steps as assistance, such as the depth map and normal map of existing models. Although it increases the accuracy of classification and segmentation, it increases the difficulty of dataset production and network training, and also limits the generalization ability of the model, unable to directly process single-view sketch inputs, and the parsing parameters of classification and segmentation often show blurred segmentation due to changes in sketch styles and degrees of abstraction and scribbling. Summary of the Invention

[0004] To solve the above technical problems existing in the prior art, the present invention proposes a template-based progressive sketch model reconstruction method, and its specific technical solutions are as follows:

[0005] A template-based progressive sketch model reconstruction method includes the following steps:

[0006] Step 1: Construct a sketch dataset containing various modeling operation semantics;

[0007] Step 2: Construct and train a sketch classification module to identify the modeling operation categories of sketches;

[0008] Step 3: Construct and train a parameter extraction module to extract sketch parameters corresponding to different categories of modeling operations;

[0009] Step 4: Integrate the sketched dataset, the sketch classification module, and the parameter extraction module into CAD software to construct an interactive CAD modeling system. After the system identifies the category of the modeling operation of the input sketch, it calls the corresponding parameter extraction module to calculate the corresponding sketch parameters for modeling operations, thereby realizing the reconstruction of the CAD model based on the hand-drawn sketch.

[0010] Further, the specific steps of Step 1 include:

[0011] Step 1.1: Design a model instantiation module to automatically generate the corresponding CAD model based on the given operation category and given modeling parameters;

[0012] Step 1.2: Design a sketch generation module to load the CAD model generated in Step 1.1 and generate multi-view projection diagrams, and use the hash algorithm to remove duplicate images;

[0013] Step 1.3: Use the CLIPasso model to perform hand-drawn stylization on the generated images, retaining geometric structures and semantic information.

[0014] Further, in Step 1.2, the sketch generation module loads the model through pythonocc, normalizes the model, configures the camera parameters, and then performs image rendering to generate multi-view projection diagrams of the corresponding model.

[0015] Further, the specific steps of Step 1.3 include:

[0016] Step 1.3.1: Use the image encoder of CLIP-ViT to extract the input image features and generate a saliency map;

[0017] Step 1.3.2: Take the saliency map as a probability distribution and randomly sample n points from it as the starting positions of each stroke;

[0018] Step 1.3.3: Convert each stroke parameter into an image representation through a differentiable rasterizer.

[0019] Further, in Step 1.3.3, use CLIP to extract the feature vectors between the original image and the generated sketch, calculate the semantic loss and geometric loss between the two, and through the backpropagation algorithm, calculate the gradient of the loss function with respect to the stroke parameters, and use the optimizer to update the stroke parameters to minimize the loss function and achieve image stylization;

[0020] Among them, the semantic loss is calculated as follows:

[0021] L semantic= dist(CLIP(I), CLIP(I g )),

[0022] where I is the original image, and I g is the generated image. CLIP is the CLIP encoder that encodes high-dimensional semantic features. dist calculates the cosine distance between two features, and the expression is as follows:

[0023]

[0024] The geometric loss is calculated as follows:

[0025]

[0026] where CLIP l is the output of the activation layer of the l-th layer of CLIP, because the shallow features of the CLIP encoder can capture geometric features better.

[0027] Furthermore, the sketch classification module adopts the ResNet18 network architecture and is trained using the cross-entropy loss function.

[0028] Furthermore, the parameter extraction module includes:

[0029] A continuous parameter prediction network that uses ResNet18 as the backbone network and outputs the prediction results of continuous parameters by connecting the regression layer. The loss function of the continuous parameter prediction network is calculated as follows:

[0030]

[0031] C is the number of continuous parameters, O c,s is the predicted numerical value of the continuous parameter, s is the n-th sketch data, and [δ c,s == 1] is a binary indicator function that indicates whether a certain continuous parameter exists in the sketches of this category;

[0032] A discrete parameter prediction network that uses ResNet18 as the backbone network and outputs the prediction results of discrete parameters representing the number of sides of the sketch by connecting the classification layer. The loss function of the discrete parameter prediction network is calculated as follows:

[0033]

[0034] Prob() calculates the prediction probability of the network for the sketch s and the discrete variable r, is the ground truth.

[0035] Further, step 4 is specifically as follows: The interactive CAD modeling system receives the hand-drawn sketch drawn by the user, identifies the modeling operation category of the sketch through the sketch classification module, then calls the corresponding parameter extraction module, calculates the discrete parameters and continuous parameters of the input sketch, and displays the top 5 modeling operation category examples through the interactive modeling interface of the system. Finally, a CAD model is generated according to the user's selection or default configuration.

[0036] Further, in the interactive modeling interface, if the user does not perform a selection operation, a CAD model with the highest probability is automatically generated according to the calculated discrete parameters and continuous parameters; if the user selects other modeling templates, that is, other modeling operations, the discrete parameters and continuous parameters in the selected template are called to generate a CAD model.

[0037] Further, the interactive modeling interface supports the user to move, rotate, and resize the CAD model through gestures.

[0038] Beneficial effects: The inventive method constructs a sketch dataset containing various modeling operation semantics, enabling more modeling operations; the proposed robust sketch classification algorithm module achieves a better classification effect for hand-drawn sketches with sparse and diverse operation semantics; and, the template-based parameter extraction algorithm does not require relying on an existing model as an auxiliary during the sketch analysis process, nor does it require post-processing of the segmentation results, thus simplifying the sketch parsing algorithm and the reconstruction algorithm. Description of the Drawings

[0039] Figure 1 is a flowchart of a template-based progressive sketch model reconstruction method according to an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of the process of the model instantiation module automatically generating a CAD model according to an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of the process of the sketch generation module generating an image according to an embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of the structure of the continuous parameter prediction network according to an embodiment of the present invention;

[0043] Figure 5 is a schematic diagram of the structure of the discrete parameter prediction network according to an embodiment of the present invention. Detailed Embodiments

[0044] In order to make the objectives, technical solutions, and technical effects of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings of the specification and embodiments.

[0045] This embodiment discloses a template-based progressive sketch model reconstruction method, which specifically includes the following steps:

[0046] Step 1: Construct a sketch dataset containing various modeling operation semantics, including:

[0047] Step 1.1: Design an algorithm module for automatically generating CAD models based on given operation categories and given modeling parameters, abbreviated as the model instantiation module. Randomly sample modeling parameters for each operation category and automatically generate corresponding geometric bodies, including stretching, sweeping, helical lines, spheres, pyramids, and cones, as Figure 2 shown, the corresponding modeling parameters are as follows:

[0048] Stretching: Sketch shape, sketch size, stretching height;

[0049] Sweeping: Sketch shape, sketch size, control points of the sweeping path;

[0050] Helical line: Sketch shape, sketch size, major diameter of the helical line, pitch, number of turns;

[0051] Sphere: Sphere radius;

[0052] Pyramid: Sketch shape, sketch size and height;

[0053] Cone: Sketch shape, sketch size and height.

[0054] Among them, the sketch shape can be characterized by sketch variables.

[0055] Step 1.2: As Figure 3 shown, design a sketch generation module. For the already generated CAD model, load the model based on pythonocc, normalize the model, set the camera parameters and then render the image to generate multi-view projection maps of the corresponding model, and remove duplicates from the generated images based on the hash algorithm.

[0056] Step 1.3: Use the CLIPasso model to perform hand-drawn stylization on the above-generated images while retaining the geometric structure and semantic information of the original images to generate data more in line with the hand-drawn style. Specifically, it includes:

[0057] Step 1.3.1: The CLIPasso model uses the image encoder of CLIP-ViT to extract the features of the input image and generates a saliency map based on these features.

[0058] Step 1.3.2: Use the saliency map as a probability distribution and randomly sample n points from it as the starting positions of each stroke.

[0059] Step 1.3.3: With the help of a differentiable rasterizer, convert each stroke parameter into a continuous and differentiable image representation, which can be optimized through backpropagation. Use CLIP to extract the feature vectors between the original image and the generated sketch, calculate the semantic loss and geometric loss between the two, through the backpropagation algorithm, calculate the gradient of the loss function with respect to the stroke parameters, and use the optimizer to update the stroke parameters to minimize the loss function and achieve image stylization.

[0060] Among them, the semantic loss is calculated as follows:

[0061] L semantic = dist(CLIP(I), CLIP(I g ))

[0062] In the formula, I is the original image, and I g is the generated image. CLIP is the CLIP encoder, which encodes high-dimensional semantic features. dist calculates the cosine distance between the two features, and the expression is as follows:

[0063]

[0064] The geometric loss is calculated as follows:

[0065]

[0066] In the formula, CLIP l is the output of the activation layer of the l-th layer of CLIP, because the shallow features of the CLIP encoder can capture geometric features better.

[0067] Step 2: Build a sketch classification module to identify the modeling operation categories of the sketches.

[0068] Train a model for modeling operation classification for the above-generated sketch dataset. The model adopts the ResNet18 network architecture and is trained using the cross-entropy loss function.

[0069] Step 3: For different categories of modeling operations, train a parameter extraction module, which consists of a continuous parameter prediction network and a discrete parameter prediction network. The skeletons of both networks adopt ResNet18. As Figure 4 shown, the continuous parameter prediction network is finally connected to a regression layer to output the prediction results of continuous parameters. The loss function of the continuous parameter prediction network is calculated as follows:

[0070]

[0071] C is the number of continuous parameters, O c,s is the predicted numerical value of the continuous parameter, s is the n-th sketch data, [δ c,s==1] is a binary indicator function that indicates whether a certain continuous parameter exists in the sketch of this category.

[0072] As Figure 5 shown, the discrete parameter prediction network is finally connected to a classification layer to output the prediction result of the discrete parameter characterizing the number of sketch edges. The loss function of the discrete parameter prediction network is calculated as follows:

[0073]

[0074] Prob() calculates the prediction probability of the network for the sketch s and the discrete variable r, is the true value.

[0075] Step 4: Integrate the above modules into CAD software, build an interactive modeling interface, and construct a hand-drawn interactive CAD modeling system.

[0076] The specific implementation process is as follows: The user draws a sketch to be modeled in the interaction interface. The system identifies the modeling operation according to the sketch classification algorithm, calls the corresponding parameter extraction module according to the modeling operation category with the highest probability, calculates the corresponding discrete parameters and continuous parameters, and displays examples of the top 5 modeling operation categories; if the user does not perform a selection operation, the most likely CAD model is automatically generated in the display interface; if the user selects other modeling templates, that is, other modeling operations, the discrete parameters and continuous parameters in the selected template are called and a CAD model is generated. The user can move, rotate, and resize the model using gestures.

[0077] In summary, the method of the present invention constructs a sketch dataset containing various modeling operation semantics, such as modeling operations like helix and complex sweep, as well as geometric primitives including cylinders, cones, spheres, etc., to achieve more modeling operations; proposes a robust sketch classification algorithm module to achieve good classification results for hand-drawn sketches with sparse and diverse operation semantics; and, a template-based parameter extraction algorithm that does not require relying on existing models as assistance during the sketch analysis process and does not require post-processing of the segmentation results, thereby simplifying the sketch parsing algorithm and the reconstruction algorithm; also proposes a CAD instantiation algorithm and a hand-drawn interactive CAD modeling system including the above algorithms.

[0078] The above is only the preferred implementation case of the present invention and does not impose any formal restrictions on the present invention. Although the implementation process of the present invention has been described in detail above, for those familiar with the field, they can still modify the technical solutions recorded in the foregoing examples or make equivalent replacements for some of the technical features. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A template-based progressive sketch model reconstruction method, characterized in that It includes the following steps: Step 1: Construct a sketch dataset containing the semantics of various modeling operations; Step 2: Construct and train a sketch classification module to identify the modeling operation categories of sketches; Step 3: Construct and train a template-based parameter extraction module to calculate the sketch parameters corresponding to different categories of modeling operations; Step 4: Integrate the sketch dataset, the sketch classification module, and the parameter extraction module into CAD software to construct an interactive CAD modeling system. After the system identifies the modeling operation category of the input sketch, it calls the corresponding parameter extraction module to calculate the corresponding sketch parameters for modeling operations, thereby realizing the reconstruction of the CAD model based on the hand-drawn sketch.

2. The method according to claim 1, characterized in that, The specific content of Step 1 includes: Step 1.1: Design a model instantiation module to automatically generate the corresponding CAD model based on the given operation category and given modeling parameters; Step 1.2: Design a sketch generation module to load the CAD model generated in Step 1.1 and generate multi-view projection diagrams, and use the hash algorithm to remove duplicate images; Step 1.3: Use the CLIPasso model to perform hand-drawn style processing on the generated images, retaining geometric structures and semantic information.

3. The method according to claim 2, wherein In Step 1.2, the sketch generation module loads the model through pythonocc, normalizes the model, configures the camera parameters, and then performs image rendering to generate multi-view projection diagrams of the corresponding model.

4. The method according to claim 2, characterized in that The specific content of Step 1.3 includes: Step 1.3.1: Use the image encoder of CLIP-ViT to extract the input image features and generate a saliency map; Step 1.3.2: Use the saliency map as a probability distribution, and randomly sample n points from it as the starting positions of each stroke; Step 1.3.3: Convert each stroke parameter into an image representation through a differentiable rasterizer.

5. The method according to claim 4, characterized in that, In Step 1.3.3, use CLIP to extract the feature vectors between the original image and the generated sketch, calculate the semantic loss and geometric loss between them, and through the backpropagation algorithm, calculate the gradient of the loss function with respect to the stroke parameters, and use the optimizer to update the stroke parameters to minimize the loss function and achieve image stylization; Among them, the semantic loss is calculated as follows: L semantic = dist(CLIP(I), CLIP(I g )) where I is the original image, and I g is the generated image, CLIP is the CLIP encoder that encodes high-dimensional semantic features, and dist calculates the cosine distance between two features, with the expression as follows: The geometric loss is calculated as follows: where CLIP l is the output of the activation layer of the l-th layer of CLIP.

6. The method according to claim 1, wherein The sketch classification module adopts the ResNet18 network architecture and is trained using the cross-entropy loss function.

7. The method according to claim 1, characterized in that, The parameter extraction module includes: A continuous parameter prediction network, which uses ResNet18 as the backbone network and outputs the prediction results of continuous parameters by connecting the regression layer; the loss function of the continuous parameter prediction network is calculated as follows: C is the number of continuous parameters, O c,s is the predicted value of the continuous parameter, s is the nth sketch data, [δ c,s == 1] is a binary indicator function expressing whether a certain continuous parameter exists in the sketch of this category; A discrete parameter prediction network, which uses ResNet18 as the backbone network and outputs the prediction results of discrete parameters representing the number of sketch edges by connecting the classification layer; the loss function of the discrete parameter prediction network is calculated as follows: Prob() computes the predicted probability of the network for sketch s and discrete variable r, which is the true value.

8. The method according to claim 7, characterized in that The specific content of step 4 is as follows: The interactive CAD modeling system receives the hand-drawn sketches drawn by the user, identifies the modeling operation categories of the sketches through the sketch classification module, then calls the corresponding parameter extraction module to calculate the discrete parameters and continuous parameters of the input sketches, and displays the top 5 modeling operation category examples through the interactive modeling interface of the system. Finally, a CAD model is generated according to the user's selection or default configuration.

9. The method according to claim 8, characterized in that, In the interactive modeling interface, if the user does not perform a selection operation, a CAD model with the highest probability is automatically generated according to the calculated discrete parameters and continuous parameters; if the user selects other modeling templates, that is, other modeling operations, the discrete parameters and continuous parameters in the selected template are called to generate a CAD model.

10. The method according to claim 8, wherein The interactive modeling interface supports the user to move, rotate, and resize the CAD model through gestures.

Images