Freehand sketch-based three-dimensional model generation method and system

Abstract

The invention discloses a three-dimensional model generation method and system based on a freehand sketch, and the method comprises the steps: carrying out the feature extraction of the freehand sketch, segmenting a freehand stroke into different part structures, extracting the part structure features as the local features of the sketch, fusing the local features of the sketch with the extracted global features of the sketch, and carrying out the rough generation of a network through a three-dimensional model, generating a rough voxel three-dimensional model by introducing fused sketch features as a condition, and constructing a three-dimensional voxel mask based on a voxel result of the rough voxel three-dimensional model through a three-dimensional model fine generation network and introducing the fused sketch features as a condition; and guiding the three-dimensional model fine generation network to intercept a part corresponding to the voxel mask from the sketch features, and generating a fine voxel three-dimensional model. A new data set 3DCM Sketch is made to train a model, so that a mapping relation between a cartoon model freehand sketch and a three-dimensional model is established.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional modeling, and in particular relates to a method and system for generating a three-dimensional model based on a hand-drawn sketch. Background Art

[0002] In the field of design, the stroke outlines in sketches convey important features of three-dimensional shapes, such as the boundary between figures and backgrounds, surface curvature, and occlusion. In addition, because sketch lines are relatively simple, designers often use them in the initial stages of character design and object modeling, so sketches are a vital part in three-dimensional modeling and industrial design.

[0003] However, the process of converting sketches into 3D models is time-consuming and cumbersome. Therefore, in the field of deep learning, research has begun on the reconstruction of sketches into 3D models based on convolutional neural networks. This has greatly lowered the threshold for 3D modeling, and users do not need to have 3D modeling experience to use sketches to design 3D models.

[0004] However, in the existing technology, there are still many problems in the process of reconstructing 3D models from sketches based on neural networks. In the modeling process, sketches are sparse and abstract, especially in models with high shape freedom (such as cartoon animal models), parts often overlap and overlap each other, resulting in some parts features not being correctly identified, causing modeling failure. Therefore, how to improve the accuracy of 3D models generated from sketches and solve part modeling errors or confusion caused by overlapping sketch lines is of great academic and engineering value. Summary of the invention

[0005] In order to solve the shortcomings of the prior art, explicitly extract semantic information from the sketch to guide the 3D reconstruction, avoid confusion of reconstructed model parts, and improve the modeling quality and accuracy, the present invention adopts the following technical solutions: The method for generating a three-dimensional model based on a hand-drawn sketch comprises the following steps: Step 1, obtain a hand-drawn sketch; Step 2: Sketch feature extraction: construct and train a semantic segmentation network to perform semantic segmentation on the sketch, segment the hand-drawn strokes into different component structures, construct a sketch feature extraction network and a feature fusion layer, extract component structure features as sketch local features, and fuse the sketch local features with the extracted sketch global features through the feature fusion layer; Step 3: roughly generate the 3D model, build a rough 3D model generation network, introduce the fused sketch features as conditions, and generate a rough voxel 3D model; Step 4, fine generation of the 3D model, building a fine generation network for the 3D model, introducing the fused sketch features as conditions, building a 3D voxel mask based on the voxel results of step 3, guiding the fine generation network of the 3D model to cut out the part corresponding to the voxel mask from the sketch features, and generating a fine voxel 3D model.

[0006] Furthermore, in the step 3, the input and output of the rough generation network of the three-dimensional model are both low-dimensional sketch features, and the output result is represented by 0 and 1 whether there are voxels in the space; in the step 4, the input and output of the fine generation network of the three-dimensional model are high-dimensional sketch features. In the sampling stage, the voxel three-dimensional model based on the low-dimensional sketch features of step 3 is upsampled to high-dimensional sketch features as a three-dimensional voxel mask composed of 0 and 1, which guides the extraction of the part corresponding to the voxel mask from the Gaussian noise of the high-dimensional sketch features.

[0007] Furthermore, in the step 2, the sketch is classified at the pixel level; in the step 3, the cross-attention mechanism is used to introduce fused sketch features in the lowest and highest pixel layers of the rough generation network of the three-dimensional model, as conditions to guide the prediction process of the rough generation network of the three-dimensional model; in the step 4, since the rough shape range of the model has been clarified, in order to enable the sketch and the three-dimensional model to interact with each other in the deep feature layer, the cross-attention mechanism is used to introduce fused sketch features only in the low-pixel layer of the fine generation network of the three-dimensional model, as conditions to guide the prediction process of the fine generation network of the three-dimensional model.

[0008] Furthermore, in the step 3, the rough generation network of the three-dimensional model is a voxel diffusion model based on UNet, and the cross-attention mechanism is used to introduce fused sketch features in the lowest and highest pixel layers of the UNet structure as a conditional guided diffusion model prediction process; in the step 4, the fine generation network of the three-dimensional model is a directed distance field SDF diffusion model based on UNet, and the cross-attention mechanism is used to introduce fused sketch features only in the low-pixel layer of the UNet structure as a conditional guided diffusion model prediction process.

[0009] Furthermore, in step 1, the hand-drawn sketch is preprocessed by detecting the bounding box of the strokes in the hand-drawn sketch and capturing the complete sketch stroke area in the hand-drawn sketch to obtain a standardized sketch.

[0010] Furthermore, in step 2, the semantic segmentation network uses a sketch graph convolution network SketchGCN architecture that uses graph convolution and a global branch network structure to extract features and classify sketches at the pixel level; the sketch feature extraction network uses a pre-trained Visual Transformer model to extract sketch global features and sketch multi-layer local features for the complete sketch and component sketch, respectively; the feature fusion layer obtains the final sketch features by splicing and fusing the sketch global features with the sketch multi-layer local features.

[0011] Furthermore, in step 4, the three-dimensional model fine generation network is a signed distance field SDF diffusion model, each element represents the signed distance field SDF value of the corresponding spatial position, and the three-dimensional model represented by the output discrete signed distance field SDF value is reconstructed through the voxel reconstruction Marching Cubes algorithm and converted into a displayable geometric patch model.

[0012] A 3D model generation system based on hand-drawn sketches, including a sketch hand-drawn input module, a sketch feature extraction and fusion module, a 3D model rough generation module and a 3D model fine generation module; The sketch hand-drawing input module is used to obtain the hand-drawn sketch; The sketch feature extraction and fusion module constructs and trains a semantic segmentation network to perform semantic segmentation on the sketch, segmenting the hand-drawn strokes into different component structures, extracting component structure features as sketch local features by constructing a sketch feature extraction network and a feature fusion layer, and fusion of the sketch local features with the extracted sketch global features through the feature fusion layer; The three-dimensional model rough generation module generates a rough voxel three-dimensional model by constructing a three-dimensional model rough generation network and introducing the fused sketch features as conditions; The three-dimensional model fine generation module constructs a three-dimensional model fine generation network, introduces the fused sketch features as conditions, constructs a three-dimensional voxel mask based on the voxel results of the three-dimensional model rough generation module, and guides the three-dimensional model fine generation network to intercept the part corresponding to the voxel mask from the sketch features to generate a fine voxel three-dimensional model.

[0013] Furthermore, in the rough generation module of the three-dimensional model, the input and output of the rough generation network of the three-dimensional model are both low-dimensional sketch features, and the output result is represented by 0 and 1 whether there are voxels in the space; in the fine generation module of the three-dimensional model, the input and output of the fine generation network of the three-dimensional model are high-dimensional sketch features. In the sampling stage, the voxel three-dimensional model based on the low-dimensional sketch features of the rough generation module of the three-dimensional model is upsampled to high-dimensional sketch features as a three-dimensional voxel mask composed of 0 and 1, which guides the extraction of the part corresponding to the voxel mask from the Gaussian noise of the high-dimensional sketch features.

[0014] Furthermore, in the sketch feature extraction and fusion module, the sketch is classified at the pixel level; in the 3D model rough generation module, in the lowest and highest pixel layers of the 3D model rough generation network, a cross-attention mechanism is used to introduce fused sketch features as conditions to guide the 3D model rough generation network prediction process; in the 3D model fine generation module, since the rough shape range of the model is already clear, in order to enable the sketch and the 3D model to interact with each other in the deep feature layer, a cross-attention mechanism is used to introduce fused sketch features only in the low-pixel layer of the 3D model fine generation network as conditions to guide the 3D model fine generation network prediction process.

[0015] The advantages and beneficial effects of the present invention are: The hand-drawn sketch-based 3D model generation method and system of the present invention performs semantic segmentation, feature extraction and fusion on the sketch, explicitly extracts semantic information in the sketch to guide the 3D reconstruction process, and introduces the global and local features of the sketch in the rough and detailed modeling stages to solve the 3D modeling errors or confusion problems caused by overlapping sketch lines, thereby achieving high-precision 3D reconstruction of hand-drawn sketches. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural schematic diagram of the three-dimensional model generation system based on hand-drawn sketches of the present invention.

[0017] Figure 2 The present invention is a flowchart of a method for generating a three-dimensional model based on a hand-drawn sketch.

[0018] Figure 3 It is a schematic diagram of the sketch feature extraction module of the present invention.

[0019] Figure 4 It is a schematic diagram of a rough generation module of a three-dimensional model of the present invention.

[0020] Figure 5 It is a schematic diagram of a three-dimensional model fine generation module of the present invention. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0022] The present invention constructs a 3DCM Sketch (3D Cartoon Model sketch) dataset. The dataset contains multi-category 3D cartoon animal models of various shapes and postures, including 12 common categories such as elephants, rabbits, bears, etc., and each category has about 100 3D models with different actions and component sizes. Since for cartoon 3D models, the frontal orientation is the angle that can most fully express the user's creative intention, for each model, sketches are randomly sampled only in the viewing angle direction of its camera rotation angle of 0±10°. In order to ensure the robustness of the training model, the number of samples is 20. For sketch sampling, the model is rendered from the camera perspective and projected as a grayscale image. The contour edges of the grayscale image are extracted through the OpenCV-Python library to form a line sketch. The overall lines of the sketch obtained by extraction are simple and smooth, so the sketch constructed in this way is very similar to a hand-drawn sketch.

[0023] The present invention provides a parameterized three-dimensional model generation system based on hand-drawn sketches, such as Figure 1 As shown, the system includes the following modules: Sketch hand-drawing input module: used to obtain user sketch input; Sketch feature extraction and fusion module: used to segment the input sketch and extract component features separately, and fuse local and global features; 3D model rough generation module: used to predict voxel 3D model; 3D model refinement module: Generates refined 3D shapes based on voxel 3D models.

[0024] The present invention provides a method for generating a parameterized three-dimensional model based on a hand-drawn sketch. Figure 2 As shown, the specific steps include: Step 1: Obtain the user's hand-drawn sketch input and process it into a standard image format; Specifically, the hand-drawn sketch is exported to a .png image format, and the complete sketch stroke area in the sketch is captured by detecting the bounding box of the strokes in the sketch and scaled to an image of 224*224 pixels.

[0025] Step 2, sketch feature extraction, such as Figure 3 As shown, a semantic segmentation network is constructed and trained to perform semantic segmentation on the input sketch, and a sketch feature extraction network and a feature fusion layer are constructed to extract component features, i.e., local features, respectively, and fuse the sketch local features with the global features.

[0026] Among them, the semantic segmentation network adopts the SketchGCN (Sketch Graph Convolutional Networks) architecture that uses graph convolution and global branch network structure to extract features, classify sketches at the pixel level, and divide hand-drawn strokes into different component structures, which are stored as 224*224*3 data structures.

[0027] The sketch feature extraction network uses a pre-trained Visual Transformer model to extract global features and multi-layer local features for complete sketches and component sketches respectively.

[0028] The feature fusion layer obtains the final sketch features by splicing and fusing global features with multiple layers of local features.

[0029] Step 3: Roughly generate the 3D model, such as Figure 4 As shown, a rough generation network of a 3D model is constructed, and the fused sketch features are introduced as conditions to generate a rough voxel 3D model.

[0030] Among them, the rough generation network of the 3D model is a voxel diffusion model based on UNet, whose input and output dimensions are both 64*64*64, and the output results use 0 and 1 to indicate whether there are voxels in the space. In order to enable the sketch and the 3D model to interact in both the deep and shallow feature layers to obtain more comprehensive feature guidance in model generation, the cross-attention mechanism is used to introduce the fused sketch features in the lowest and highest pixel layers of the UNet structure as conditions to guide the diffusion model prediction process. In the model training stage, by processing the 3DCM Sketch dataset, a fused sketch feature-voxel 3D model data dictionary is constructed as training data. In the model sampling stage, starting from 64*64*64 Gaussian noise, a rough voxel 3D model is gradually generated through 50 iterations.

[0031] Step 4: 3D model generation, such as Figure 5 As shown, a 3D model fine generation network is constructed, and the fused sketch features are introduced as conditions to generate a fine voxel 3D model.

[0032] Among them, the fine generation network of the 3D model is a SDF (Signed Distance Fields) diffusion model based on UNet, whose input and output dimensions are both 128*128*128, and each element represents the SDF value of the corresponding spatial position. Unlike the rough generation network of the 3D model, since the rough shape range of the model is clear, in order to make the sketch and the 3D model interact with each other in the deep feature layer, only in the low-pixel layer of the UNet structure, the cross-attention mechanism is used to introduce the fused sketch features as conditions to guide the diffusion model prediction process. In the model training stage, the 3DCM Sketch dataset is processed to construct a fused sketch feature-voxel 3D model-SDF value sampling 3D model data dictionary as training data. In the sampling stage, the voxel 3D model with a dimension of 64*64*64 obtained in step 3 is upsampled to 128*128*128 as a 3D voxel mask composed of 0 and 1, which guides the interception of the corresponding part of the voxel mask from the 128*128*128 Gaussian noise, and gradually generates a fine 3D model represented by the SDF value through 50 iterations. The output discrete SDF-represented 3D model is reconstructed through the voxel reconstruction Marching Cubes algorithm and converted into a displayable geometric patch model.

[0033] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D model generation method based on hand-drawn sketches, characterized in that The steps include: Step 1, obtain a hand-drawn sketch; Step 2: Sketch feature extraction: construct and train a semantic segmentation network to perform semantic segmentation on the sketch, segment the hand-drawn strokes into different component structures, construct a sketch feature extraction network and a feature fusion layer, extract component structure features as sketch local features, and fuse the sketch local features with the extracted sketch global features through the feature fusion layer; Step 3: roughly generate the 3D model, build a rough 3D model generation network, introduce the fused sketch features as conditions, and generate a rough voxel 3D model; Step 4, fine generation of the 3D model, building a fine generation network for the 3D model, introducing the fused sketch features as conditions, building a 3D voxel mask based on the voxel results of step 3, guiding the fine generation network of the 3D model to cut out the part corresponding to the voxel mask from the sketch features, and generating a fine voxel 3D model.

2. The method for generating a three-dimensional model based on a hand-drawn sketch according to claim 1, characterized in that: In the step 3, the input and output of the rough generation network of the three-dimensional model are both low-dimensional sketch features, and the output result is represented by 0 and 1 to indicate whether there are voxels in the space; in the step 4, the input and output of the fine generation network of the three-dimensional model are high-dimensional sketch features. In the sampling stage, the voxel three-dimensional model based on the low-dimensional sketch features of step 3 is upsampled to high-dimensional sketch features as a three-dimensional voxel mask composed of 0 and 1, which guides the extraction of the part corresponding to the voxel mask from the Gaussian noise of the high-dimensional sketch features.

3. The method for generating a three-dimensional model based on a hand-drawn sketch according to claim 1, characterized in that: In the step 2, the sketch is classified at the pixel level; in the step 3, the cross-attention mechanism is used to introduce the fused sketch features in the lowest and highest pixel layers of the rough generation network of the three-dimensional model, as conditions to guide the prediction process of the rough generation network of the three-dimensional model; in the step 4, the cross-attention mechanism is used to introduce the fused sketch features only in the low-pixel layer of the fine generation network of the three-dimensional model, as conditions to guide the prediction process of the fine generation network of the three-dimensional model.

4. The method for generating a three-dimensional model based on a hand-drawn sketch according to claim 3, characterized in that: In the step 3, the rough generation network of the three-dimensional model is a voxel diffusion model based on UNet. In the lowest and highest pixel layers of the UNet structure, a cross-attention mechanism is used to introduce fused sketch features as a conditional guide diffusion model prediction process; in the step 4, the fine generation network of the three-dimensional model is a directed distance field SDF diffusion model based on UNet. Only in the low pixel layer of the UNet structure, a cross-attention mechanism is used to introduce fused sketch features as a conditional guide diffusion model prediction process.

5. The method for generating a three-dimensional model based on a hand-drawn sketch according to claim 1, characterized in that: In the step 1, the hand-drawn sketch is preprocessed by detecting the bounding box of the strokes in the hand-drawn sketch and intercepting the complete sketch stroke area in the hand-drawn sketch to obtain a standardized sketch.

6. The method for generating a three-dimensional model based on a hand-drawn sketch according to claim 1, characterized in that: In step 2, the semantic segmentation network uses a sketch graph convolution network architecture that extracts features using graph convolution and a global branch network structure to perform pixel-level classification on the sketch; the sketch feature extraction network uses a pre-trained Visual Transformer model to extract sketch global features and sketch multi-layer local features for the complete sketch and component sketch respectively; The feature fusion layer obtains the final sketch features by splicing and fusing the sketch global features with the sketch's multi-layer local features.

7. The method for generating a three-dimensional model based on a hand-drawn sketch according to claim 1, characterized in that: In step 4, the three-dimensional model fine generation network is a signed distance field SDF diffusion model, each element represents the signed distance field SDF value of the corresponding spatial position, and the three-dimensional model represented by the output discrete signed distance field SDF value is reconstructed through a voxel reconstruction algorithm and converted into a geometric patch model.

8. A 3D model generation system based on hand-drawn sketches, comprising a sketch hand-drawn input module, a sketch feature extraction and fusion module, a 3D model rough generation module and a 3D model fine generation module, characterized in that: The sketch hand-drawing input module is used to obtain the hand-drawn sketch; The sketch feature extraction and fusion module constructs and trains a semantic segmentation network to perform semantic segmentation on the sketch, segmenting the hand-drawn strokes into different component structures, extracting component structure features as sketch local features by constructing a sketch feature extraction network and a feature fusion layer, and fusion of the sketch local features with the extracted sketch global features through the feature fusion layer; The three-dimensional model rough generation module generates a rough voxel three-dimensional model by constructing a three-dimensional model rough generation network and introducing the fused sketch features as conditions; The three-dimensional model fine generation module constructs a three-dimensional model fine generation network, introduces the fused sketch features as conditions, constructs a three-dimensional voxel mask based on the voxel results of the three-dimensional model rough generation module, and guides the three-dimensional model fine generation network to intercept the part corresponding to the voxel mask from the sketch features to generate a fine voxel three-dimensional model.

9. The three-dimensional model generation system based on hand-drawn sketches according to claim 8, characterized in that: In the three-dimensional model rough generation module, the input and output of the three-dimensional model rough generation network are both low-dimensional sketch features, and the output result is represented by 0 and 1 to indicate whether there is a voxel in the space; In the three-dimensional model fine generation module, the input and output of the three-dimensional model fine generation network are high-dimensional sketch features. In the sampling stage, the voxel three-dimensional model based on the low-dimensional sketch features of the three-dimensional model rough generation module is upsampled to high-dimensional sketch features as a three-dimensional voxel mask composed of 0 and 1, which guides the extraction of the part corresponding to the voxel mask from the Gaussian noise of the high-dimensional sketch features.

10. The three-dimensional model generation system based on hand-drawn sketches according to claim 8, characterized in that: In the sketch feature extraction and fusion module, the sketch is classified at the pixel level; in the 3D model rough generation module, the cross-attention mechanism is used to introduce fused sketch features in the lowest and highest pixel layers of the 3D model rough generation network, which are used as conditions to guide the 3D model rough generation network prediction process; in the 3D model fine generation module, the cross-attention mechanism is used to introduce fused sketch features only in the low-pixel layer of the 3D model fine generation network, which are used as conditions to guide the 3D model fine generation network prediction process.

Images