Method and system for generating a three-dimensional model based on a hand-drawn sketch

Abstract

The application discloses a three-dimensional model generation method and system based on hand-drawn sketches, extracts features from hand-drawn sketches, divides hand-drawn strokes into different component structures, extracts component structure features as sketch local features, fuses the sketch local features with extracted sketch global features, introduces the fused sketch features as conditions through a three-dimensional model rough generation network, generates a rough voxel three-dimensional model, introduces the fused sketch features as conditions through a three-dimensional model fine generation network, constructs a three-dimensional voxel mask based on a voxel result of the rough voxel three-dimensional model, guides the three-dimensional model fine generation network to intercept a part corresponding to the voxel mask from the sketch features, and generates a fine voxel three-dimensional model. A new data set 3DCM Sketch is made, a model is trained, and a mapping relationship between a cartoon model hand-drawn sketch and a three-dimensional model is established.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional modeling, and particularly relates to a three-dimensional model generation method and system based on hand-drawn sketches. BACKGROUND

[0002] In the field of design, the stroke outlines in sketches convey important features of three-dimensional shapes, such as figure and background boundaries, surface curvatures, and occlusion conditions. In addition, because sketch lines are relatively simple, designers often use them in the initial stage of character design and object modeling, so sketches are a crucial part of three-dimensional modeling and industrial design.

[0003] However, the process of converting sketches into three-dimensional models is both time-consuming and tedious. Therefore, in the field of deep learning, research has begun on the implementation of reconstruction from sketches to three-dimensional models based on convolutional neural networks. This greatly reduces the threshold for three-dimensional modeling, and users do not need to have experience in three-dimensional modeling to use sketches to design three-dimensional models.

[0004] However, in existing technologies, there are still many problems in the reconstruction from sketches to three-dimensional models based on neural networks. Because sketches are sparse and abstract in the modeling process, especially in model categories with high shape degrees of freedom (such as cartoon animal models), there are often problems of mutual occlusion and partial overlap between parts, which leads to the failure of some part features to be correctly identified, resulting in modeling errors or confusion. Therefore, how to improve the accuracy of three-dimensional model generation from sketches and solve the problem of part modeling errors or confusion caused by overlapping sketch lines is of great academic and engineering value. SUMMARY

[0005] To solve the problems of the prior art and achieve the purpose of explicitly extracting semantic information from sketches to guide three-dimensional reconstruction, avoid confusion of reconstructed model parts, and improve modeling quality and accuracy, the present application adopts the following technical solutions:

[0006] The three-dimensional model generation method based on hand-drawn sketches comprises the following steps:

[0007] Step 1: Obtain a hand-drawn sketch.

[0008] Step 2: Extract sketch features; construct and train a semantic segmentation network to perform semantic segmentation on the sketch, divide the hand-drawn strokes into different part structures, construct a sketch feature extraction network and a feature fusion layer, extract part structure features as local features of the sketch, and fuse the local features of the sketch with the extracted global features of the sketch through the feature fusion layer.

[0009] Step 3: Roughly generate a three-dimensional model; construct a three-dimensional model rough generation network, introduce the fused sketch features as a condition, and generate a rough voxel three-dimensional model.

[0010] Step 4, fine generation of three-dimensional model, constructing a fine generation network of three-dimensional model, introducing the fused sketch features as conditions, constructing a three-dimensional voxel mask based on the voxel results of step 3, guiding the fine generation network of three-dimensional model to intercept the part corresponding to the voxel mask from the sketch features, and generating a fine voxel three-dimensional model.

[0011] Further, in step 3, the input and output of the three-dimensional model rough generation network are both low-dimensional sketch features, and the output result indicates whether there is a voxel in the space by 0 and 1; in step 4, the input and output of the three-dimensional model fine generation network are high-dimensional sketch features, and in the sampling stage, based on the low-dimensional sketch feature voxel three-dimensional model of step 3, the high-dimensional sketch feature is up-sampled to a three-dimensional voxel mask composed of 0 and 1, guiding the three-dimensional model fine generation network to intercept the part corresponding to the voxel mask from the high-dimensional sketch feature Gaussian noise.

[0012] Further, in step 2, the sketch is classified at the pixel level; in step 3, cross-attention mechanism is used to introduce the fused sketch features at the lowest and highest pixel layers of the three-dimensional model rough generation network, as conditions to guide the prediction process of the three-dimensional model rough generation network; in step 4, since the shape range of the rough model is clear, in order to enable the sketch and the three-dimensional model to interact in the deep feature layer, cross-attention mechanism is used to introduce the fused sketch features only in the low pixel layer of the three-dimensional model fine generation network, as conditions to guide the prediction process of the three-dimensional model fine generation network.

[0013] Further, in step 3, the three-dimensional model rough generation network is a voxel diffusion model based on UNet, and cross-attention mechanism is used to introduce the fused sketch features at the lowest and highest pixel layers of the UNet structure, as conditions to guide the diffusion model prediction process; in step 4, the three-dimensional model fine generation network is a directed distance field SDF diffusion model based on UNet, and cross-attention mechanism is used to introduce the fused sketch features only in the low pixel layer of the UNet structure, as conditions to guide the diffusion model prediction process.

[0014] Further, in step 1, the hand-drawn sketch is preprocessed, the bounding box of the strokes in the hand-drawn sketch is detected, the complete sketch stroke area in the hand-drawn sketch is intercepted, and a standardized sketch is obtained.

[0015] Further, in step 2, the semantic segmentation network adopts a sketch graph convolution network SketchGCN architecture with a graph convolution and a global branch network structure to extract features, and performs pixel-level classification on the sketch; the sketch feature extraction network adopts a pre-trained Visual Transformer model to extract sketch global features and sketch multi-layer local features from the complete sketch and the component sketch respectively; and the feature fusion layer fuses the sketch global features and the sketch multi-layer local features to obtain the final sketch features.

[0016] Further, in step 4, the three-dimensional model fine generation network is a directed distance field SDF diffusion model, each element represents the directed distance field SDF value of the corresponding space position, and the output discrete directed distance field SDF value represents a three-dimensional model, which is reconstructed into a surface through a voxel reconstruction Marching Cubes algorithm and converted into a displayable geometric patch model.

[0017] The three-dimensional model generation system based on a hand-drawn sketch includes a sketch hand-drawing input module, a sketch feature extraction and fusion module, a three-dimensional model rough generation module, and a three-dimensional model fine generation module.

[0018] The sketch hand-drawing input module is configured to obtain a hand-drawn sketch.

[0019] The sketch feature extraction and fusion module performs semantic segmentation on the sketch by constructing and training a semantic segmentation network, divides the hand-drawing strokes into different component structures, extracts component structure features as sketch local features through a sketch feature extraction network and a feature fusion layer, and fuses the sketch local features with extracted sketch global features through the feature fusion layer.

[0020] The three-dimensional model rough generation module generates a rough voxel three-dimensional model by constructing a three-dimensional model rough generation network and introducing the fused sketch features as conditions.

[0021] The three-dimensional model fine generation module generates a fine voxel three-dimensional model by constructing a three-dimensional model fine generation network, introducing the fused sketch features as conditions, constructing a three-dimensional voxel mask based on the voxel result of the three-dimensional model rough generation module, and guiding the three-dimensional model fine generation network to extract the part corresponding to the voxel mask from the sketch features.

[0022] Further, in the three-dimensional model rough generation module, the input and output of the three-dimensional model rough generation network are both low-dimensional sketch features, and the output result indicates whether the space has a voxel through 0 and 1; in the three-dimensional model fine generation module, the input and output of the three-dimensional model fine generation network are high-dimensional sketch features, and in the sampling stage, based on the voxel three-dimensional model of the low-dimensional sketch feature of the three-dimensional model rough generation module, up-sampling is performed to high-dimensional sketch features to serve as a three-dimensional voxel mask composed of 0 and 1 to guide the cutting of the part corresponding to the voxel mask from the Gaussian noise of the high-dimensional sketch feature.

[0023] Further, in the sketch feature extraction and fusion module, the sketch is classified at the pixel level; in the three-dimensional model rough generation module, the cross attention mechanism is used to introduce the fused sketch feature at the lowest and highest pixel layers of the three-dimensional model rough generation network to serve as a condition to guide the prediction process of the three-dimensional model rough generation network; in the three-dimensional model fine generation module, since the shape range of the model is rough, in order to enable the sketch and the three-dimensional model to interact in the deep feature layer, the cross attention mechanism is used to introduce the fused sketch feature only at the low pixel layer of the three-dimensional model fine generation network to serve as a condition to guide the prediction process of the three-dimensional model fine generation network.

[0024] The advantages and beneficial effects of the present application are as follows:

[0025] The three-dimensional model generation method and system based on a hand-drawn sketch of the present application can solve the three-dimensional model modeling error or confusion problem caused by the overlapping of sketch lines by performing semantic segmentation and feature extraction and fusion on the sketch, explicitly extracting the semantic information in the sketch to guide the three-dimensional reconstruction process, and introducing the global and local features of the sketch in the rough and refined modeling stages, so as to realize high-precision three-dimensional reconstruction of a hand-drawn sketch. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 FIG. 1 is a structural schematic diagram of the three-dimensional model generation system based on a hand-drawn sketch of the present application.

[0027] Figure 2 FIG. 2 is a flowchart of the three-dimensional model generation method based on a hand-drawn sketch of the present application.

[0028] Figure 3 FIG. 3 is a sketch feature extraction module schematic diagram of the present application.

[0029] Figure 4 FIG. 4 is a three-dimensional model rough generation module schematic diagram of the present application.

[0030] Figure 5 FIG. 5 is a three-dimensional model fine generation module schematic diagram of the present application. DETAILED DESCRIPTION

[0031] The specific embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended for illustration and explanation of the present application and are not intended to limit the present application.

[0032] The present application constructs a 3DCM Sketch (3D Cartoon Model sketch) dataset. The dataset contains multi-category 3D cartoon animal models of various shapes and postures, including 12 common categories such as elephants, rabbits, bears, etc., and about 100 3D models of different actions and component sizes for each category. Since for cartoon 3D models, the front direction is the most complete expression of user's creative intention, for each model, only the sketch is randomly sampled at the camera rotation angle of 0±10°. To ensure the robustness of the trained model, the number of samples is 20. For sketch sampling, the model is rendered from the camera view and projected as a grayscale picture, and the contour edge of the grayscale picture is extracted through the OpenCV-Python library to form a line sketch. The overall line of the sketch obtained by extraction is simple and smooth, so the sketch constructed in this way is very similar to hand-drawn sketches.

[0033] The present application provides a parameterized 3D model generation system based on hand-drawn sketches, as shown in Figure 1 The system includes the following modules:

[0034] Sketch hand-drawing input module: for obtaining user sketch drawing input;

[0035] Sketch feature extraction and fusion module: for segmenting the input sketch and extracting component features respectively, and fusing local and global features;

[0036] 3D model rough generation module: for predicting a voxel 3D model;

[0037] 3D model fine generation module: for generating a refined 3D shape according to the voxel 3D model.

[0038] The present application provides a parameterized 3D model generation method based on hand-drawn sketches, as shown in Figure 2 The specific steps include:

[0039] Step 1, obtain the user's hand-drawn sketch input and process it into a standard picture format;

[0040] Specifically, the hand-drawn sketch is exported as a.png picture format, and the complete sketch stroke area in the sketch is intercepted by detecting the bounding box of the strokes in the sketch and scaled to a picture of size 224*224 pixels.

[0041] Step 2, sketch feature extraction, as shown in Figure 3As shown, the semantic segmentation network is constructed and trained to perform semantic segmentation on the input sketch, and a sketch feature extraction network and a feature fusion layer are constructed to extract component features, i.e., local features, and fuse the sketch local features with global features.

[0042] The semantic segmentation network adopts a SketchGCN (Sketch Graph Convolutional Networks) architecture that extracts features using a graph convolution and a global branch network structure, performs pixel-level classification on the sketch, and divides the hand-drawn strokes into different component structures, which are stored as data structures of 224*224*3.

[0043] The sketch feature extraction network adopts a pre-trained Visual Transformer model to extract global features and multi-layer local features from the complete sketch and component sketch.

[0044] The feature fusion layer obtains the final sketch features by fusing the global features and multi-layer local features.

[0045] Step 3, rough generation of three-dimensional model, as shown in Figure 4 The three-dimensional model rough generation network is constructed, and the fused sketch features are introduced as conditions to generate a rough voxel three-dimensional model.

[0046] The three-dimensional model rough generation network is a voxel diffusion model based on UNet, and the input and output dimensions are both 64*64*64. The output result is represented by 0 and 1 to indicate whether there is a voxel in the space. To enable the sketch and three-dimensional model to interact at both deep and shallow feature layers to obtain more comprehensive feature guidance in model generation, the cross-attention mechanism is used to introduce the fused sketch features as conditions to guide the diffusion model prediction process at the lowest and highest pixel layers of the UNet structure. During the model training phase, the 3DCM Sketch dataset is processed to construct a fused sketch feature-voxel three-dimensional model data dictionary as training data. During the model sampling phase, starting from 64*64*64 Gaussian noise, a rough voxel three-dimensional model is gradually generated through 50 iteration steps.

[0047] Step 4, fine generation of three-dimensional model, as shown in Figure 5 The three-dimensional model fine generation network is constructed, and the fused sketch features are introduced as conditions to generate a fine voxel three-dimensional model.

[0048] The fine-grained 3D model generation network is a UNet-based SDF (Signed Distance Fields) diffusion model with input and output dimensions of 128*128*128, where each element represents the SDF value of the corresponding spatial location. Unlike the coarse-grained 3D model generation network, since the approximate shape range of the model is already defined, to enable feature interaction between the sketch and the 3D model at deeper feature layers, a cross-attention mechanism is used only in the low-pixel layers of the UNet structure to introduce fused sketch features as a condition to guide the diffusion model's prediction process. During model training, the 3DCM Sketch dataset is processed to construct a 3D model data dictionary that integrates sketch features, voxel 3D models, and SDF value sampling, which serves as the training data. During the sampling phase, the 64*64*64 voxel 3D model obtained in step 3 is upsampled to 128*128*128, serving as a 3D voxel mask composed of 0s and 1s. This mask guides the extraction of the corresponding portion from the 128*128*128 Gaussian noise, gradually generating a refined 3D model represented by SDF values ​​through 50 iterations. The output discrete SDF-represented 3D model is then used to reconstruct its surface using the Marching Cubes algorithm, converting it into a displayable geometric patch model.

[0049] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a three-dimensional model based on a hand-drawn sketch, characterized by Comprising the following steps: Step 1, obtaining a hand-drawn sketch; Step 2, sketch feature extraction; construct and train a semantic segmentation network, perform semantic segmentation on the sketch, and segment the hand-drawn strokes into different component structures; construct a sketch feature extraction network and a feature fusion layer to extract component structure features as local sketch features; and fuse the local sketch features with the extracted global sketch features through the feature fusion layer; Step 3, rough generation of a three-dimensional model, constructing a three-dimensional model rough generation network, introducing the fused sketch features as conditions, and generating a rough voxel three-dimensional model; In the lowest and highest pixel layers of the three-dimensional model rough generation network, the cross-attention mechanism is used to introduce the fused sketch features as conditions to guide the prediction process of the three-dimensional model rough generation network; The three-dimensional model rough generation network is a voxel diffusion model based on UNet, and in the lowest and highest pixel layers of the UNet structure, the cross-attention mechanism is used to introduce the fused sketch features as conditions to guide the prediction process of the diffusion model; Step 4, fine generation of three-dimensional model, constructing a fine generation network of three-dimensional model, introducing the fused sketch features as conditions, Based on the voxel results of step 3, a three-dimensional voxel mask is constructed to guide The fine generation network of three-dimensional model extracts the part corresponding to the voxel mask from the sketch features to generate a fine voxel three-dimensional model. Only in the low pixel layer of the three-dimensional model fine generation network, the cross-attention mechanism is used to introduce the fused sketch features as conditions to guide the prediction process of the three-dimensional model fine generation network; The three-dimensional model fine generation network is based on UNet Directed distance field SDF The diffusion model only uses the sketch features in the low pixel layer of the UNet structure as a condition to guide the prediction process of the diffusion model.

2. The hand-drawn sketch-based three-dimensional model generation method of claim 1, wherein: In step 3, the input and output of the three-dimensional model rough generation network are low-dimensional sketch features, and the output result is represented by 0 and 1 to indicate whether there is a voxel in the space; in step 4, the input and output of the three-dimensional model fine generation network are high-dimensional sketch features, and in the sampling stage, based on the low-dimensional sketch feature voxel three-dimensional model of step 3, upsample to high-dimensional sketch features, as a three-dimensional voxel mask composed of 0 and 1, to guide the extraction of the corresponding part of the voxel mask from the high-dimensional sketch feature Gaussian noise.

3. The hand-drawn sketch based 3D model generation method of claim 1, wherein: In step 1, the hand-drawn sketch is preprocessed by detecting the bounding box of the strokes in the hand-drawn sketch, and the complete sketch stroke area in the hand-drawn sketch is extracted to obtain a standardized sketch. 4.The hand-drawn sketch based 3D model generation method of claim 1, wherein: In step 2, the semantic segmentation network uses a graph convolution and global branch network structure to extract features, and the sketch graph convolution network architecture classifies the sketch at the pixel level; the sketch feature extraction network uses a pre-trained Visual Transformer model to extract global sketch features and multi-layer local sketch features from the complete sketch and component sketch respectively; The feature fusion layer obtains the final sketch features by concatenating and fusing the global sketch features and the multi-layer local sketch features.

5. The hand-drawn sketch-based three-dimensional model generation method of claim 1, wherein: In step 4, Directed distance field SDF The three-dimensional model fine generation network is Directed a diffusion model, each element representing a corresponding spatial position Distance field SDF value, Three-dimensional model represented by the directed distance field SDF value, The output discrete Conversion reconstruct the surface by a voxel reconstruction algorithm, 6. A hand-drawn sketch-based three-dimensional model generation system using the hand-drawn sketch-based three-dimensional model generation method of claim 1, comprising a sketch hand-drawing input module, a sketch feature extraction and fusion module, a three-dimensional model rough generation module, and a three-dimensional model fine generation module, characterized in that: is a geometric patch model. The sketch hand-drawing input module is used to obtain a hand-drawn sketch; ​ The sketch feature extraction and fusion module, by constructing and training a semantic segmentation network, performs semantic segmentation on the sketch, divides the hand-drawn strokes into different component structures, extracts component structure features as sketch local features by constructing a sketch feature extraction network and a feature fusion layer, and fuses the sketch local features with the extracted sketch global features through the feature fusion layer; The three-dimensional model rough generation module, by constructing a three-dimensional model rough generation network, introduces the fused sketch features as conditions to generate a rough voxel three-dimensional model; The three-dimensional model fine generation module introduces the fused sketch features as conditions by constructing a three-dimensional model fine generation network, Based on the voxel result, a three-dimensional voxel mask is constructed to guide The three-dimensional model coarse generation module In the three-dimensional model rough generation module, the three-dimensional model rough generation network inputs and outputs are both low-dimensional sketch features, and the output result indicates whether there is a voxel in the space through 0 and 1; The three-dimensional model fine generation network extracts a part corresponding to a voxel mask from the sketch features to generate a fine voxel three-dimensional model.

7. The system of claim 6, wherein: In the three-dimensional model fine generation module, the three-dimensional model fine generation network inputs and outputs are high-dimensional sketch features, and in the sampling stage, based on the voxel three-dimensional model of the low-dimensional sketch features of the three-dimensional model rough generation module, up-sampling is performed to high-dimensional sketch features as a three-dimensional voxel mask composed of 0 and 1 to guide the extraction of the corresponding part of the voxel mask from the Gaussian noise of the high-dimensional sketch features. In the sketch feature extraction and fusion module, pixel-level classification is performed on the sketch; in the three-dimensional model rough generation module, at the lowest and highest pixel layers of the three-dimensional model rough generation network, the fused sketch features are introduced as conditions to guide the prediction process of the three-dimensional model rough generation network through a cross-attention mechanism; and in the three-dimensional model fine generation module, only at the low pixel layer of the three-dimensional model fine generation network, the fused sketch features are introduced as conditions to guide the prediction process of the three-dimensional model fine generation network through a cross-attention mechanism.

8. The system of claim 6, wherein: ​

Images