A method and system for generating a parametric three-dimensional model based on a hand-drawn sketch

Abstract

The application discloses a parameterized three-dimensional model generation method and system based on hand-drawn sketches, and the system is composed of four modules, including: a user sketch input module, which acquires real-time sketch input of a user; a sketch feature extraction module, which carries out instance segmentation on the sketch drawn by the user, provides an operation category for modeling, and extracts key features in the sketch; a geometric parameter calculation module, which carries out regression calculation on normalized parameters according to the segmented image result; and a three-dimensional model generation module, which generates a modeling operation sequence according to the corresponding modeling operation type and modeling parameters generated by the above operation, and inputs the modeling software, so as to generate a parameterized three-dimensional model.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and computer-aided design, and in particular to a method and system for generating parametric 3D models based on hand-drawn sketches. Background Technology

[0002] Sketching and 3D modeling are two crucial steps in industrial design. Designers typically begin by creating sketches to express and refine their design intentions; then, the completed sketches are converted into 3D models using parametric modeling software, generating high-precision and editable CAD models. However, using 3D modeling software usually requires operators to undergo a certain level of software training and understand the sequence of object modeling and design processes; whereas creating concise, illustrative sketches requires no training, and anyone can use sketches to express design intentions and concepts.

[0003] Sketch-based modeling (SBM) is a field that studies the automatic generation of 3D models based on sketches. Early research mostly focused on the analysis and prediction of wireframe diagrams from a fixed perspective. After the rise of deep learning, some scholars began to use convolutional neural networks to extract features from images for 3D model reconstruction, such as sketch feature-generated mesh models (Sketch2Mesh, ICCV2021), 3D model reconstruction based on multi-view sketches, and 3D models of hairstyles reconstructed from sketches (DeepSketchHair).

[0004] However, some deep learning-based sketch generation modeling methods can only produce non-editable mesh models and have certain limitations on the generated models. For example, they may require multi-view sketches or be limited to the 3D reconstruction and editing of hair, car, or chair models. Therefore, to address these issues, a sketch generation modeling system capable of generating multiple types of parametric models is needed. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a parametric 3D model generation method and system based on hand-drawn sketches. The aim is to overcome the limitations of existing sketch-based modeling methods in generating limited types of uneditable mesh models. This allows users without 3D software experience to easily and quickly realize their design intentions and effectively edit the generated models interactively.

[0006] This invention is achieved through the following technical solution: a method for generating parametric 3D models based on hand-drawn sketches, characterized in that the method includes the following steps:

[0007] (1) Obtain the user's real-time sketch input, including displaying and obtaining the 3D view of the currently generated model;

[0008] (2) Combine the current 3D view to analyze the user's sketch input and extract the key features in the sketch;

[0009] (3) Generate standardized modeling operation descriptions and geometric parameter calculations based on the segmented sketch results;

[0010] (4) Based on the modeling operation description and geometric parameters obtained in step (3), generate a modeling operation sequence and input it into the CAD modeling software to generate a parametric 3D model.

[0011] Further, step (1) specifically involves obtaining the user's real-time sketch input, thereby obtaining the current 3D model view in real time, and directly sketching on the view. The hand-drawn sketch can be input via mouse or graphics tablet, and can be edited and redrawn.

[0012] Further, step (2) is an instance segmentation network for sketching handwriting. The input of this network is a merged image of the current 3D model view and the user's sketching handwriting, and the output is the pixel-level segmentation result of the user's sketch. This network is used to analyze the modeling behavior category of the user's sketch and the structural features of the sketch content, and to provide input for the subsequent modeling key parameter extraction network.

[0013] Furthermore, the sketching instance segmentation network adopts a Mask R-CNN network architecture to perform instance segmentation of modeling information on the merged image of sketching and 3D view. Instance segmentation can segment and extract the user's sketching and classify the hand-drawn stroke pixels that represent different operation meanings; identify the modeling operation category represented by the hand-drawn sketch, namely, stretching, chamfering, rounding, and additive spiral; for the stretched sketch, extract and identify the category of the bottom outline of the sketch geometry, the top surface outline of the stretched geometry, and the side curves respectively, and perform coarse classification of the geometric features of the bottom outline to distinguish whether its composition is a circle, a regular polygon, an irregular polygon, or an irregular sketch that combines curves and straight lines;

[0014] The Mask R-CNN network architecture consists of two main parts. The first part generates object detection candidate boxes on the image, generates category prediction and size prediction of the candidate boxes, and filters a portion of the candidate boxes. The second part predicts the object mask.

[0015] Furthermore, the Mask R-CNN network architecture is specifically as follows:

[0016] First, the combined image of the 3D view and the user's sketch is input into the Resnet-FPN module in Mask R-CNN to extract the feature layer. Then, the Region Proposal Network (RPN) in Mask R-CNN is used to generate target candidate boxes of different sizes, and the ROIAlign layer is used to project the generated candidate boxes onto the feature map to generate feature maps of different candidate boxes.

[0017] Input the feature map into the category and bounding box prediction branches, perform convolution on the features, generate the category prediction corresponding to the candidate box, as well as the prediction of the candidate box size, and filter some candidate boxes.

[0018] For the mask prediction branch, another ROIAlign layer is used to project and extract the feature maps of a series of candidate boxes generated by the RPN structure.

[0019] Convolution and transpose convolution are performed on the feature map generated by the ROIAlign layer of the mask prediction branch to generate mask predictions for different categories of information.

[0020] The loss function used during training is shown in equation (1), and the total loss is the loss of the candidate box prediction class. classfication Candidate box regression parameter loss box Loss of mask prediction mask Together they constitute.

[0021] Loss = Loss classfication +Loss box +Loss mask (1)

[0022]

[0023]

[0024] In equation (2), Loss classfication For predicting the category loss of candidate boxes, N class This represents the number of samples in a batch. The loss for predicting the class of each candidate box is calculated using softmax cross-entropy; Loss box For the loss of the candidate box regression parameters, equations (3), (4), and (5) are used to calculate the smoothness of the boundary parameters of each candidate box relative to the boundary parameters of the ground truth candidate box. L1 Loss, t i Let be the boundary parameters predicted for the i-th candidate box. These are the true boundary parameters of the i-th candidate box;

[0025]

[0026]

[0027] Loss mask For mask prediction loss, calculate the average binary cross-entropy loss between the predicted mask matrix of the true class and the ground truth mask matrix of that class.

[0028] Furthermore, the generation of standardized modeling operation descriptions and geometric parameter calculations specifically involves calculating standardized parameters based on the segmented image results, including the analysis and extraction of sketch 3D coordinate information and regression of key modeling parameters.

[0029] The 3D coordinate information analysis and extraction of the sketch involves classifying the shape and detecting feature points based on the bottom contour extracted from the sketch segmentation results, and obtaining the sketch coordinates through the inverse of the projection matrix. Figure 2 The projected ray of the feature point is used to calculate the intersection point between the ray and the generated model, which is used as the three-dimensional coordinate of the feature point. According to the result of sketch segmentation, if the modeling category of the sketch is an extrusion of an irregular shape sketch, that is, the extruded shape is neither a circle nor a regular polygon, but an irregular polygon or a geometric figure composed of polylines and curves, then it is necessary to first analyze the composition of the sketch, then extract the feature points according to the composition, and obtain the three-dimensional coordinates of the feature points, so as to generate a parametric sketch and perform the extrusion operation.

[0030] The regression of key modeling parameters involves constructing an image feature regression network. The input consists of the segmentation result image from the sketch handwriting instance segmentation network and the camera parameters of the current viewpoint. The output is the key parameters of the modeling operation. The feature extraction network needs to be trained specifically for different modeling operations, and the output is also determined by the type of modeling operation. If the segmentation network shows that the modeling operation is to stretch a regular polygon, the key parameters are the number of polygons and the ratio of the stretch height to the side length of the polygon. If the segmentation network shows that the modeling operation is to stretch a circle, the key parameters are the ratio of the stretch height to the radius of the circle. If the segmentation network shows that the modeling operation is a spiral, the key parameters are the number of spiral turns and the ratio of the pitch to the radius.

[0031] Specifically, the regression of key modeling parameters involves training different feature extraction networks for different modeling operation categories. The network architecture employs multiple convolutional and fully connected layers, with camera parameters also used as input to the fully connected layers. The network uses L2 loss as its loss function, calculating the loss between predicted and true parameter values, λ||θ||. 2 λ is the regularization term, λ is the penalty coefficient, and θ is the network weight:

[0032]

[0033] Different parameter types were designed for different types of modeling operations. For example, for extruding regular polygons, the number of side lengths n and the ratio of the height to the side length of the regular polygon (h / L) need to be predicted. For cylinders or cones with different sized circles on the top and bottom surfaces, the parameters to be predicted are the ratio of the height to the radius of the bottom surface (h / r1) and the ratio of the radii of the top and bottom circles (r2 / r1). For irregular sketch extrusion operations, the ratio of the extrusion distance to the circumcircle of the sketch (h / r) needs to be predicted. For operations related to generating spirals, i.e., adding / removing spirals, the ratio of the spiral spacing to the radius of the bottom surface d / R and the number of spiral turns n need to be predicted.

[0034] Specifically, the process of analyzing and obtaining the sketched three-dimensional coordinate information is as follows:

[0035] First, the bottom contour image extracted from the sketch segmentation results is preprocessed, namely binarized, and the contour is dilated to restore the broken bottom contour to a closed and complete state.

[0036] Extract the contours from the processed image and iterate through the contours to find the outer contour.

[0037] Based on the segmentation results of the sketch instance, targeted feature point detection is performed according to the category of the bottom contour. If the shape of the bottom is a regular geometric shape, i.e., a regular polygon, a circle, or an irregular polygon, then corner point detection is performed to extract the polygon vertices, or circle detection is used to calculate the center and radius of the circle. If the classification result is a complex geometric shape, i.e. a combination of curves, line segments, and arcs, then the next feature point extraction step is performed.

[0038] Based on the segmentation results, if the bottom contour detection is an irregular sketch, a segmentation-fitting operation is required. First, the curvature of the points in the contour is calculated, and the points corresponding to the local maxima of the curve curvature are used as segmentation points. The sketch is segmented, and different segmentation results are fitted and classified as straight lines, arcs, and B-spline curves. The category information and corresponding feature points are saved.

[0039] Based on sketch category recognition and feature point extraction, the feature points of the segmented 2D sketch can be obtained. Subsequently, the inverse of the projection matrix is ​​used to obtain the final sketch. Figure 2 The projected ray of the feature point is calculated, and the intersection point between the ray and the generated model is obtained to obtain the three-dimensional position of the feature point, thereby obtaining the base plane or geometry on which the current modeling operation is based, as well as the three-dimensional coordinates of the feature point.

[0040] Specifically, the 3D model generation is an instantiation of modeling operations. It is based on the segmentation output results extracted from the sketch features, which can obtain the modeling operation category corresponding to the sketch, namely, extrusion, chamfering and rounding, and additive spiral.

[0041] Based on the analysis of the 3D coordinate information of the sketch, the positioning information for different modeling operations is obtained; for chamfering and filleting operations, the two endpoints of the chamfering and filleting sketch outline are obtained, and the chamfered edge and the size of the chamfer are located; for the additive spiral, the positioning of the spiral is obtained: center, radius and starting plane; for the extrusion operation, the extrusion reference plane and the 3D information of the extruded sketch are obtained, so as to reconstruct the sketch in 3D space.

[0042] The parameters for specific modeling operations are obtained by regression analysis of key modeling parameters. For example, in the extrusion modeling operation, after the bottom sketch is determined, the actual extrusion distance is calculated based on the ratio of the extrusion distance obtained by regression analysis of key modeling parameters to the sketch side length or radius, thereby generating a complete modeling instruction.

[0043] Based on the corresponding modeling operations and modeling parameters generated by the above operations, a modeling operation sequence is generated and input into 3D modeling software that supports modeling operation sequence files to generate a parametric 3D model.

[0044] This invention also provides a parametric 3D model generation system based on hand-drawn sketches, the system comprising the following modules:

[0045] User sketch input module: can acquire real-time sketch input from users, including displaying and acquiring 3D views of the currently generated model;

[0046] Sketch feature extraction module: Combines the current 3D view to analyze the user's sketch input and extracts the key features in the sketch;

[0047] Geometric parameter calculation module: Generates standardized modeling operation descriptions and calculates geometric parameters based on the segmented sketch results;

[0048] 3D Model Generation Module: Based on the modeling operation description and geometric parameters obtained from the geometric parameter calculation module, it generates a modeling operation sequence, inputs it into CAD modeling software, and generates a parametric 3D model.

[0049] The beneficial effects of this invention are as follows:

[0050] Compared to existing methods that generate mesh models from sketches, which can only produce limited types of non-editable mesh models, this invention generates parametric CAD models by analyzing user sketches. This facilitates subsequent model editing, and users can interactively modify the model in real time, such as adding chamfers, fillets, and new geometric shapes. This invention enables users without 3D software experience to easily and quickly realize their design intentions and effectively interactively edit the generated models. Attached Figure Description

[0051] Figure 1A flowchart of a parametric 3D model generation system based on hand-drawn sketches provided in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the user sketch input module interface provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of a sketch feature extraction module provided in an embodiment of the present invention;

[0054] Figure 4 This invention provides a regression module for key parameters in modeling based on segmentation results, as provided in an embodiment of the invention.

[0055] Figure 5 The effect of the 3D model generation module in this embodiment of the invention - stretched circular image;

[0056] Figure 6 The effect of the 3D model generation module in this embodiment of the invention - stretched polygon image;

[0057] Figure 7 The effect of the 3D model generation module in this embodiment of the invention - rounded corners;

[0058] Figure 8 The image shows the effect of the 3D model generation module in this embodiment of the invention - a beveled view. Detailed Implementation

[0059] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0060] This invention provides a parametric 3D model generation system based on hand-drawn sketches.

[0061] like Figure 1 As shown, the present invention includes the following modules:

[0062] (1) User sketch input module, which obtains the user's real-time sketch input;

[0063] (2) Sketch feature extraction module: segment the user's sketch into instances, provide the operation categories for modeling, and extract key features from the sketch;

[0064] (3) Geometric parameter calculation module, which performs regression calculation of normalized parameters based on the segmented image results;

[0065] (4) The 3D model generation module generates a modeling operation sequence based on the corresponding modeling operation type and modeling parameters generated by the above operations, and inputs it into the modeling software to generate a parameterized 3D model.

[0066] S1, such as Figure 2 In the user sketch input module shown, users can obtain the current 3D model view in real time through the sketch input module and draw directly on the view. Hand-drawn sketches can be input through a mouse or a graphics tablet and can be edited and redrawn.

[0067] S2, such as Figure 3 The sketch feature extraction module shown is an instance segmentation network for sketch handwriting. The network input is a merged image of the current 3D model view and the user's sketch handwriting, and the output is the pixel-level segmentation result of the user's sketch. It can be used to analyze the modeling behavior category of the user's sketch, as well as the structural features of the sketch content, and to provide input for the subsequent modeling key parameter extraction network.

[0068] S21. The sketch handwriting instance segmentation network adopts the Mask R-CNN network architecture to perform instance segmentation of modeling information from the merged image of sketch handwriting and 3D view. Instance segmentation can segment and extract the user's sketch handwriting and classify the handwriting pixels representing different operation meanings. It identifies the modeling operation category represented by the hand-drawn sketch, such as: extrusion, chamfering, rounding, and additive spiral, etc. Furthermore, for extruded sketches, it extracts and identifies the categories of the bottom outline of the sketch geometry, the top surface outline of the extruded geometry, and the side curves, and performs coarse classification of the geometric features of the bottom outline to distinguish whether its composition is a circle, a regular polygon, an irregular polygon, or an irregular sketch that combines curves and straight lines.

[0069] The S22 Mask R-CNN network architecture mainly consists of two parts. The first part generates object detection candidate boxes on the image, predicts the category and size of the candidate boxes, and filters a portion of the candidate boxes, such as... Figure 3 The left branch is shown; the other part is the prediction of the target mask, such as... Figure 3 As shown in the branch on the right.

[0070] like Figure 3 As shown, the specific details of the network architecture are as follows:

[0071] S221. First, input the combined image of the 3D view and the user's sketch into the Resnet-FPN module in Mask R-CNN to extract the feature layer; use the Region Proposal Network (RPN) in Mask R-CNN to generate target candidate boxes of different sizes, and use the ROIAlign layer to project the generated candidate boxes onto the feature map to generate feature maps of different candidate boxes.

[0072] S222. Input the feature map into the category and bounding box prediction branches, perform a series of convolutions on the features, generate category predictions for candidate boxes and predictions for candidate box sizes, and filter out a portion of the candidate boxes; for example... Figure 3 As shown in the lower left corner, taking the stretching operation as an example, this part of the network can perform rough target detection on the top surface, side curves, and bottom contour of the stretched sketch.

[0073] S223. For the mask prediction branch, another ROIAlign layer is used to project and extract the feature maps of a series of candidate boxes generated by the RPN structure.

[0074] S224. Perform convolution and transpose convolution on the feature map generated by the ROIAlign layer of the mask branch to generate mask predictions for different categories of information; such as... Figure 3 As shown in the lower right corner, taking the stretching operation as an example, this part of the network can perform pixel-level segmentation and classification of the top surface, side curves, and bottom contour of the stretched sketch.

[0075] The loss function used during training is shown in equation (1), and the total loss is the loss of the candidate box prediction class. classfication Candidate box regression parameter loss box Loss of mask prediction mask Together they constitute.

[0076] Loss = Loss classfication +Loss box +Loss mask (1)

[0077]

[0078]

[0079] (2) In the formula, Loss classfication For predicting the category loss of candidate boxes, N class This represents the number of samples in a batch. The loss for predicting the class of each candidate box is calculated using softmax cross-entropy; Loss box For the loss of the candidate box regression parameters, equations (3), (4), and (5) are used to calculate the smoothness of the boundary parameters of each candidate box relative to the boundary parameters of the true candidate box. L1 Loss, t i Let be the boundary parameters predicted for the i-th candidate box. These are the true boundary parameters of the i-th candidate box;

[0080]

[0081]

[0082] Loss mask For mask prediction loss, calculate the average binary cross-entropy loss between the predicted mask matrix of the true class and the ground truth mask matrix of that class.

[0083] S3, the geometric parameter calculation module needs to calculate normalized parameters based on the segmented image results, such as... Figure 1 As shown, this module includes a module for analyzing and extracting 3D coordinate information from sketching, and a module for regressing key modeling parameters.

[0084] S31, such as Figure 4 As shown, the key parameter regression module for modeling constructs an image feature regression network. The inputs are the segmentation result image from the sketch handwriting instance segmentation network and the camera parameters of the current viewpoint. The output is the key parameters of the modeling operation. The key parameter regression module trains different feature extraction networks for different modeling operation categories. The network architecture employs multiple convolutional layers and fully connected layers, and the camera parameters are also used as input to the fully connected layers. The network uses L2 loss as the loss function, calculating the loss between the predicted parameter values ​​and the true parameter values, λ||θ||. 2 λ is the regularization term, λ is the penalty coefficient, and θ is the network weight:

[0085]

[0086] Different parameter types were designed for different types of modeling operations, such as: 1) For extruding regular polygons, the number of side lengths n and the ratio of the height to the side length of the regular polygon (h / L) need to be predicted; for cylinders or cones with different sized circles on the top and bottom surfaces, the parameters to be predicted are the ratio of the height to the radius of the bottom surface (h / r1) and the ratio of the radii of the top and bottom circles (r2 / r1); for irregular sketch extrusion operations, the ratio of the extrusion distance to the circumcircle of the sketch (h / r) needs to be predicted; for operations related to generating spirals, such as adding / removing spirals, the ratio of the spiral spacing to the radius of the bottom surface d / R and the number of spiral turns n need to be predicted.

[0087] S32, the sketch 3D coordinate information analysis and extraction module, based on the bottom contour extracted from the sketch segmentation results, performs shape classification and feature point detection, and obtains the sketch coordinate information through the inverse of the projection matrix. Figure 2The projected rays of the feature points are used to calculate the intersection points between the rays and the generated model, which serve as the 3D coordinates of the feature points. Based on the sketch segmentation results, if the sketch modeling category is an extrusion of an irregular shape sketch (i.e., the extruded shape is neither a circle nor a regular polygon, but an irregular polygon or a geometric figure composed of polylines and curves), then it is necessary to first analyze the composition of the sketch, then extract feature points based on the composition, and obtain the 3D coordinates of the feature points, thereby generating a parametric sketch and performing the extrusion operation.

[0088] The specific process of the sketch 3D coordinate information analysis and acquisition module is as follows:

[0089] S321. First, the bottom contour image extracted from the sketch segmentation result is preprocessed: binarized, and the contour is dilated to restore the broken bottom contour to a closed and complete state.

[0090] S322. Extract the contours of the processed image and traverse the contours to find the outer contour.

[0091] S323. Based on the results of sketch instance segmentation, perform targeted feature point detection according to the category of the bottom contour. If the shape of the bottom is a regular geometric shape: regular polygon, circle, irregular polygon, then perform corner point detection to extract polygon vertices, or use circle detection to calculate the center and radius of the circle; if the classification result is a complex geometric shape: a combination of curve, line segment, and arc, then perform the feature point extraction step in 4).

[0092] S324. Based on the segmentation results, if the bottom contour detection is an irregular sketch, a segmentation and fitting operation is required. First, the curvature of the points in the contour is calculated, and the points corresponding to the local maxima of the curve curvature are used as segmentation points. The sketch is segmented, and different segmentation results are fitted and classified as straight lines, arcs, and B-spline curves. The category information and corresponding feature points are saved.

[0093] S325. Based on the above sketch category recognition and feature point extraction steps, the feature points of the segmented 2D sketch can be obtained. Subsequently, the inverse of the projection matrix is ​​used to obtain the sketch's feature points. Figure 2 The projected ray of the feature point is calculated, and the intersection point between the ray and the generated model is obtained to obtain the three-dimensional position of the feature point, thereby obtaining the base plane or geometry on which the current modeling operation is based, as well as the three-dimensional coordinates of the feature point.

[0094] S4, the instantiation of the 3D model generation module for performing modeling operations.

[0095] Based on the segmentation output of the sketch feature extraction module, the corresponding modeling operation category can be obtained, such as extrusion, chamfering / rounding, and additive spiral. The sketch 3D coordinate information analysis module obtains the positioning information for different modeling operations; for chamfering and rounding operations, it obtains the two endpoints of the chamfer / rounding sketch outline and locates the chamfered edge and its size; for additive spiral, it obtains the spiral's positioning: center, radius, and starting plane; for extrusion, it obtains the extrusion reference plane and the 3D information of the extruded sketch, thus reconstructing the sketch in 3D space. The modeling key parameter regression module obtains the parameters for specific modeling operations. For example, in the extrusion modeling operation, after determining the bottom sketch, the actual extrusion distance is calculated based on the ratio of the extrusion distance obtained from the modeling key parameter regression module to the sketch's side length or radius, thereby generating a complete modeling instruction. Based on the corresponding modeling operations and modeling parameters generated above, a modeling operation sequence is generated and input into 3D modeling software that supports modeling operation sequence files to generate a parameterized 3D model. Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown, the effects of the geometry generated by the 3D model generation module when the user draws a sketch representing modeling operations such as extruding a circle, extruding a regular geometry, chamfering, and rounding corners are demonstrated.

[0096] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0097] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for generating parametric 3D models based on hand-drawn sketches, characterized in that, The method includes the following steps: (1) Obtain the user's real-time sketch input, including displaying and obtaining the 3D view of the currently generated model; (2) Combine the current 3D view to analyze the user's sketch input and extract the key features in the sketch; Step (2) is an instance segmentation network for sketching handwriting. The input of this network is a merged image of the current 3D model view and the user's sketching handwriting. The output is the pixel-level segmentation result of the user's sketching. This network is used to analyze the modeling behavior category of the user's sketching and the structural features of the sketching content, and to provide input for the subsequent modeling key parameter extraction network. The sketching instance segmentation network adopts the Mask R-CNN network architecture to perform instance segmentation of modeling information on the merged image of sketching and 3D view. Instance segmentation can segment and extract the user's sketching and classify the hand-drawn stroke pixels that represent different operation meanings; identify the modeling operation category represented by the hand-drawn sketch, namely, extrusion, chamfering, rounding, and additive spiral; for extruded sketches, extract and identify the category of the bottom outline of the sketch geometry, the top surface outline of the extruded geometry, and the side curves respectively, and perform coarse classification of the geometric features of the bottom outline to distinguish whether it is composed of circles, regular polygons, irregular polygons, or irregular sketches that combine curves and straight lines. The Mask R-CNN network architecture consists of two main parts. The first part generates object detection candidate boxes on the image, generates category prediction and size prediction for the candidate boxes, and filters some candidate boxes. The second part predicts the object mask. (3) Generate standardized modeling operation descriptions and geometric parameter calculations based on the segmented sketch results; Specifically, the generation of standardized modeling operation descriptions and geometric parameter calculations requires the calculation of standardized parameters based on the segmented image results, including the analysis and extraction of sketch 3D coordinate information and regression of key modeling parameters; The 3D coordinate information analysis and extraction of the sketch involves classifying the shape and detecting feature points based on the bottom contour extracted from the sketch segmentation results, and obtaining the projection rays of the 2D feature points of the sketch through the inverse of the projection matrix. The intersection points between the rays and the generated model are then calculated as the 3D coordinates of the feature points. Based on the sketch segmentation results, if the modeling category of the sketch is an extrusion of an irregular shape sketch, that is, the extruded shape is neither a circle nor a regular polygon, but an irregular polygon or a geometric figure composed of polylines and curves, then it is necessary to first analyze the composition of the sketch, then extract feature points based on the composition, and obtain the 3D coordinates of the feature points, thereby generating a parametric sketch and performing the extrusion operation. The key parameter regression for modeling involves constructing an image feature regression network. The inputs are the segmentation result image from the sketch handwriting instance segmentation network and the camera parameters from the current viewpoint. The output is the key parameters of the modeling operation. The feature extraction network needs to be trained specifically for different modeling operations, and the output is determined by the type of modeling operation. If the segmentation network indicates that the modeling operation is extruding a regular polygon, the key parameters are the number of polygons and the ratio of the extrusion height to the polygon's side length. If the segmentation network indicates that the modeling operation is extruding a circle, the key parameters are the ratio of the extrusion height to the circle's radius. If the segmentation network indicates that the modeling operation is a spiral, the key parameters are the number of spiral turns and the ratio of the pitch to the radius. The key parameter regression for modeling involves training different feature extraction networks for different types of modeling operations. The network architecture uses multiple convolutional layers and fully connected layers, and the camera parameters are also used as input to the fully connected layers. The network loss function uses L2 loss to calculate the loss between the predicted parameter values ​​and the true parameter values. For regularization terms, Penalty coefficient, Network weights: （6） Different parameter types were designed for different types of modeling operations. For example, for extruding regular polygons, the number of side lengths n and the ratio of the height to the side length of the regular polygon (h / L) need to be predicted. For cylinders or cones with different sized circles on the top and bottom surfaces, the parameters to be predicted are the ratio of the height to the radius of the bottom surface (h / r1) and the ratio of the radii of the top and bottom circles (r2 / r1). For irregular sketch extrusion operations, the ratio of the extrusion distance to the circumcircle of the sketch (h / r) needs to be predicted. For operations related to generating spirals, i.e., adding / removing spirals, the ratio of the spiral spacing to the bottom radius d / R and the number of spiral turns n need to be predicted. (4) Based on the modeling operation description and geometric parameters obtained in step (3), generate a modeling operation sequence and input it into the CAD modeling software to generate a parametric 3D model.

2. The method for generating a parametric 3D model based on hand-drawn sketches according to claim 1, characterized in that, The specific steps (1) are as follows: obtain the user's real-time sketch input, then obtain the current three-dimensional model view in real time, and directly draw sketches on the view. The hand-drawn sketches can be input by mouse or graphics tablet, and can be edited and redrawn.

3. The method for generating a parametric 3D model based on hand-drawn sketches according to claim 1, characterized in that, The specific architecture of the Mask R-CNN network is as follows: (5.1) First, input the combined image of the 3D view and the user's sketch into the Resnet-FPN module in Mask R-CNN to extract the feature layer; use the Region Proposal Network (RPN) in Mask R-CNN to generate target candidate boxes of different sizes, and use the ROIAlign layer to project the generated candidate boxes onto the feature map to generate feature maps of different candidate boxes. (5.2) Input the feature map into the category and bounding box prediction branches, perform convolution on the features, generate the category prediction corresponding to the candidate box and the prediction of the candidate box size, and filter some candidate boxes; (5.3) For the mask prediction branch, another ROIAlign layer is used to project and extract the feature maps of a series of candidate boxes generated by the RPN structure; (5.4) Perform convolution and transpose convolution on the feature map generated by the ROIAlign layer of the mask prediction branch to generate mask predictions for different categories of information; The loss function used during training is shown in equation (1), and the total loss is the loss from predicting the candidate box category. Candidate box regression parameter loss and mask prediction loss Together they constitute: （1） （2） （3） In equation (2), Predict the category loss for candidate boxes. This represents the number of samples in a batch. The loss for predicting the category of each candidate box is calculated using softmax cross-entropy; For the loss of the candidate box regression parameters, equations (3), (4), and (5) are used to calculate the boundary parameters of each candidate box and the boundary parameters of the ground truth candidate box. loss, Let be the boundary parameters predicted for the i-th candidate box. These are the true boundary parameters of the i-th candidate box; （4） （5） For mask prediction loss, calculate the average binary cross-entropy loss between the predicted mask matrix of the true class and the ground truth mask matrix of that class.

4. The method for generating a parametric 3D model based on a hand-drawn sketch according to claim 1, wherein the process of analyzing and obtaining the 3D coordinate information of the sketch is as follows: (8.1) First, the bottom contour image extracted from the sketch segmentation result is preprocessed; that is, binarized and the contour is dilated to restore the broken bottom contour to a closed and complete state. (8.2) Extract the contours of the processed image and traverse the contours to find the outer contour; (8.3) Based on the results of sketch instance segmentation, perform targeted feature point detection according to the category of bottom contour. If the shape of the bottom is a regular geometric shape, i.e., a regular polygon, a circle, or an irregular polygon, then perform corner point detection to extract the polygon vertices, or use circle detection to calculate the center and radius of the circle; if the classification result is a complex geometric shape, i.e. a combination of curves, line segments, and arcs, then proceed to the next feature point extraction step. (8.4) Based on the results of instance segmentation, if the bottom contour detection is an irregular sketch, it is necessary to perform segmentation and fitting operations first; firstly, the curvature of the points in the contour is calculated, and the points corresponding to the local maxima of the curve curvature are used as segmentation points to segment the sketch. For different segmentation results, straight lines, arcs, and B-spline curves are fitted and classified, and the category information and corresponding feature points are saved. (8.5) Based on sketch category recognition and feature point extraction, the feature points of the segmented two-dimensional sketch can be obtained. Then, by inverting the projection matrix, the projection rays of the two-dimensional feature points of the sketch can be obtained. The intersection points between the rays and the generated model can be calculated to obtain the three-dimensional position of the feature points, thereby obtaining the base plane or geometry on which the current modeling operation is based, as well as the three-dimensional coordinates of the feature points.

5. The method for generating a parametric 3D model based on a hand-drawn sketch according to claim 1, characterized in that, The instantiation of the modeling operation for generating the 3D model is specifically as follows: Based on the segmentation output of the sketch feature extraction, the modeling operation category corresponding to the sketch can be obtained, namely, extrusion, chamfering and rounding, and additive spiral. Based on the analysis of the 3D coordinate information of the sketch, the positioning information for different modeling operations is obtained; for chamfering and filleting operations, the two endpoints of the chamfering and filleting sketch outline are obtained, and the chamfered edge and the size of the chamfer are located; for the additive spiral, the positioning of the spiral is obtained: center, radius and starting plane; for the extrusion operation, the extrusion reference plane and the 3D information of the extruded sketch are obtained, so as to reconstruct the sketch in 3D space. Based on the regression of key modeling parameters, the parameters for specific modeling operations are obtained, namely the extrusion modeling operation. After the bottom sketch is determined, the actual extrusion distance is calculated based on the ratio of the extrusion distance obtained from the regression of key modeling parameters to the sketch side length or radius, thereby generating a complete modeling instruction. Based on the modeling operations and parameters that generate complete modeling instructions, a modeling operation sequence is generated and input into 3D modeling software that supports modeling operation sequence files to generate a parametric 3D model.

6. A system for generating a parametric 3D model based on a hand-drawn sketch as described in any one of claims 1-5, characterized in that, The system includes the following modules: User sketch input module: can acquire real-time sketch input from the user, including displaying and retrieving the three-dimensional representation of the currently generated model. 3D view; Sketch feature extraction module: Combines the current 3D view to analyze the user's sketch input and extracts the key features in the sketch; Geometric parameter calculation module: Generates standardized modeling operation descriptions and calculates geometric parameters based on the segmented sketch results; 3D Model Generation Module: Based on the modeling operation description and geometric parameters obtained from the geometric parameter calculation module, it generates a modeling operation sequence, inputs it into CAD modeling software, and generates a parametric 3D model.

Images