Color three-dimensional model generation method and system based on sketch and text
Through the integration of sketch segmentation and text features, combined with cross attention and joint diffusion model, the problem of understanding errors in sketch and text conversion in the existing technology is solved, and high-precision color three-dimensional model generation and semantic segmentation are realized. Model editing.
Patent Information
- Application Number
- CN202411216112.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-31
AI Technical Summary
The prior art has errors in generating understanding when converting sketches or text into three-dimensional models and lack of perception of semantic structures, making it difficult to effectively combine sketches and text to improve the integrity and clarity of feature expression of the three-dimensional model.
Through the integration of sketch segmentation and text features, the cross attention mechanism and joint diffusion model are adopted, combined with the pre-trained Visual Transformer and BERT models, a three-dimensional model with color and component classification is generated to achieve deep interaction and feature guidance between sketches and text information.
It improves the high-precision reconstruction of 3D models, reduces information comprehension errors, supports model editing, and implements semantic segmentation in color 3D model generation.
Smart Images
Figure CN119228997A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of three-dimensional modeling, and in particular relates to a method and system for generating a color three-dimensional model based on sketches and texts. Background Art
[0002] In the field of design, the strokes in the sketch can represent the basic features of the 3D model, such as the shape and structure, and is an important guide for 3D modeling. Text, as a design idea refined and extracted by humans, can also provide basic feature descriptions for 3D models and guide the reconstruction of 3D models through concise information.
[0003] With the rise of deep learning, research on conditional 3D reconstruction based on convolutional neural networks has gradually become a hot topic. However, due to the sparsity and abstractness of sketches and the ambiguity and roughness of text, it is still a challenge to convert sketches or texts into correct 3D models using deep learning methods. In the existing technology, there are still problems such as generation and understanding errors when reconstructing 3D images from sketches or texts based on neural networks. In addition, there is a lack of perception of the semantic structure of 3D shapes.
[0004] Therefore, how to combine sketches and texts and introduce them into the neural network of 3D reconstruction to enhance the completeness and clarity of the feature expression of the 3D model is the key to improving the modeling effect of the 3D model and is also of great academic and engineering value. Summary of the invention
[0005] In order to solve the shortcomings of the prior art, the present invention adopts the following technical solutions to achieve the purpose of guiding 3D reconstruction by extracting semantic information from sketches and fusing text features, and calculating color and component classification information while generating point clouds:
[0006] The method for generating a color 3D model based on sketch and text includes the following steps:
[0007] Step 1: Get a hand-drawn sketch;
[0008] Step 2: Sketch segmentation and text feature fusion, segment the hand-drawn sketch by components and extract features, and fuse the hand-drawn sketch features, sketch component features and extracted text features;
[0009] Step 3: Construct a 3D model generation network, and introduce the fusion features of the sketch and text obtained in step 2 into the cross-attention mechanism it adopts, and train the 3D model generation network to generate a 3D model with color and component classification.
[0010] Furthermore, in the step 1, the boundaries around the complete sketch stroke area in the sketch are detected and captured, and the image is normalized, and the surrounding blank areas are scaled and supplemented to a picture with a size of 224*224 pixels.
[0011] Furthermore, in step 2, a FCN (Fully Convolutional Networks) model with ResNet-50 as the network backbone is trained to perform semantic segmentation on the input sketch and mark different parts in the sketch.
[0012] Furthermore, in step 2, the sketch feature extraction network uses a pre-trained Visual Transformer model to extract global features and multi-layer local features for the complete sketch and sketch components, respectively, and realizes feature fusion through feature splicing.
[0013] Furthermore, in step 2, the text features are extracted by pre-training the BERT model to improve the performance of natural language processing tasks based on deep bidirectional context information.
[0014] Furthermore, in step 2, the hand-drawn sketch features, sketch component features and text features are fused through a multi-head attention mechanism.
[0015] Furthermore, in the step 3, a joint diffusion model based on UNet is constructed, and point cloud shape, color and component classification are generated by synchronous training; in the diffusion model, a classifier-free guidance method is introduced, and an unconditional generation model and a conditional generation model are trained at the same time, and the three-dimensional model reconstruction result is inferred by a linear combination of the two to improve the quality and diversity of the generated model; in order to enable the sketch and text information to interact with the three-dimensional model at each feature layer, so that the diffusion process can be deeply guided by features, at each layer of the UNet structure, a projection-based cross-attention mechanism is used to introduce the sketch and text fusion features output by step 2, so that the fusion features interact with the corresponding projection position point cloud features under the camera perspective.
[0016] Furthermore, in step 3, the loss function of the constructed diffusion model is set to:
[0017]
[0018] in, represents the expected function, ∈ and t represent the sampling noise and time step respectively, x0 represents the point cloud point, c0 represents the point cloud color, s0 represents the point cloud classification, x t ,c t ,s t represents the corresponding noisy vector, C represents the feature condition of sketch and text fusion, and f represents the UNet network.
[0019] Furthermore, in step 3, in the sampling stage, starting from Gaussian noise, a point cloud three-dimensional model of joint features is gradually generated through iterative training.
[0020] A color 3D model generation system based on sketch and text, including a sketch hand-drawn input module, a sketch segmentation and text feature fusion module, and a 3D model generation module;
[0021] The sketch hand-drawing input module is used to obtain the hand-drawn sketch;
[0022] The sketch segmentation and text feature fusion module divides the input sketch into parts and extracts features, and fuses the hand-drawn sketch features, sketch part features and extracted text features;
[0023] The three-dimensional model generation module constructs a three-dimensional model generation network, and introduces the fusion features of the sketch and text obtained by the sketch segmentation and text feature fusion module into the cross-attention mechanism it adopts, so as to train the three-dimensional model generation network to generate a three-dimensional model with color and component classification.
[0024] The advantages and beneficial effects of the present invention are:
[0025] The sketch-and-text-based color 3D model generation method and system of the present invention perform semantic segmentation on the sketch and fuse it with text features, explicitly extract semantic information from the sketch, and guide the 3D reconstruction process with text assistance, thereby reducing fuzzy information brought by the sketch and text, and solving the problem of modeling errors caused by information misunderstanding in 3D modeling, thereby achieving high-precision 3D reconstruction of hand-drawn sketches, and achieving 3D semantic segmentation of the model while generating the color 3D model, providing support for extended tasks such as model editing. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a schematic diagram of the structure of the sketch- and text-based color 3D model generation system of the present invention.
[0027] Figure 2 The present invention is a flowchart of the method for generating a color three-dimensional model based on sketches and texts.
[0028] Figure 3 It is a schematic diagram of the sketch segmentation and text feature fusion module of the present invention;
[0029] Figure 4 It is a schematic diagram of a three-dimensional model generation module of the present invention. DETAILED DESCRIPTION
[0030] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.
[0031] The present invention provides a color 3D model generation system based on sketch and text, such as Figure 1As shown, the system includes the following modules:
[0032] Sketch hand-drawing input module: used to obtain user sketch input;
[0033] Sketch segmentation and text feature fusion module: Segment the input sketch by parts and extract features, and fuse sketch and text features;
[0034] 3D model generation module: used to generate point cloud 3D shapes with color and component classification results.
[0035] The present invention provides a method for generating a color 3D model based on sketches and texts, such as Figure 2 As shown, the specific steps include:
[0036] Step 1: Obtain the user's hand-drawn sketch input and process it into a standard image format;
[0037] Specifically, the hand-drawn sketch is exported to .png image format, the boundaries around the complete sketch stroke area in the sketch are detected and captured, and then the surrounding blank areas are scaled and supplemented to an image with a size of 224*224 pixels.
[0038] Step 2: Sketch segmentation and text feature fusion, such as Figure 3 As shown, the input sketch is segmented by parts and features are extracted, and the sketch and text features are fused;
[0039] Specifically, a semantic segmentation network is constructed and trained to perform semantic segmentation on the input sketch, text features are extracted through a text feature extraction network, and the sketch features are fused with the text features.
[0040] Among them, the semantic segmentation network adopts the FCN model with ResNet-50 as the network backbone to classify the sketch at the pixel level, thereby dividing the sketch strokes into different component structures and storing them as 224*224*3 data structures respectively, providing the semantic information expressed in the sketch for 3D reconstruction and synchronous component classification.
[0041] The sketch feature extraction network uses a pre-trained Visual Transformer model to extract global features and multi-layer local features for the complete sketch and component sketch respectively, and realizes feature fusion through feature splicing.
[0042] The text feature extraction network adopts the pre-trained BERT model, which improves the performance of natural language processing tasks through deep bidirectional context information.
[0043] The sketch and text feature fusion part is realized through a multi-head attention mechanism. The text feature is the query Q, and the sketch fusion feature is the key K and the value V, which provides a text and sketch fusion feature vector as a condition for the subsequent 3D reconstruction task.
[0044] Step 3: Generate a 3D model. Figure 4 As shown, a three-dimensional model with color and component classification is generated by a three-dimensional model generation network.
[0045] Specifically, a 3D model generation network is constructed, and the fused sketch and text features are introduced as conditions to generate point cloud 3D models and the color and component classification of the corresponding point clouds.
[0046] Among them, the 3D model generation network is a joint diffusion model based on UNet, which is divided into three parts: point cloud diffusion, appearance diffusion, and component classification diffusion. In the diffusion model, a classifier-free guidance method is introduced, and an unconditional generation model and a conditional generation model are trained at the same time. The 3D model reconstruction results are inferred through a linear combination of the two to improve the quality and diversity of the generation model. In order to enable the sketch and text information to interact with the 3D model at all feature layers, so that the diffusion process can be deeply guided by features, the cross-attention mechanism based on projection is used to introduce the fused sketch and text features in all pixel layers of the three-part diffused UNet structure as the conditional guidance diffusion model prediction process, so that the fused features interact with the corresponding projection position point cloud features under the camera perspective.
[0047] The loss function of the constructed diffusion model is set as:
[0048]
[0049] in, represents the expected function, ∈ and t represent the sampling noise and time step respectively, x0 represents the point cloud point, c0 represents the point cloud color, s0 represents the point cloud classification, x t ,c t ,s t represents the corresponding noisy vector, C represents the feature condition of sketch and text fusion, and f represents the UNet network.
[0050] In the model training stage, the sketch and text fusion feature-3D model data dictionary is constructed as training data. In the model sampling stage, starting from Gaussian noise, a colored point cloud 3D model and a segmented 3D model are gradually generated through 50 iterations.
[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for generating a color 3D model based on sketches and texts, characterized in that The steps include: Step 1: Get a hand-drawn sketch; Step 2: Sketch segmentation and text feature fusion, segment the hand-drawn sketch by components and extract features, and fuse the hand-drawn sketch features, sketch component features and extracted text features; Step 3: Construct a 3D model generation network, and introduce the fusion features of the sketch and text obtained in step 2 into the cross-attention mechanism it adopts, and train the 3D model generation network to generate a 3D model with color and component classification.
2. The method for generating a color 3D model based on sketch and text according to claim 1, characterized in that: In the step 1, the boundaries of the complete sketch stroke area in the sketch are detected and captured, and image normalization is performed.
3. The method for generating a color 3D model based on sketch and text according to claim 1, characterized in that: In the step 2, the FCN model with ResNet as the network backbone is trained to perform semantic segmentation on the input sketch and mark different parts in the sketch.
4. The method for generating a color 3D model based on sketch and text according to claim 1, characterized in that: In step 2, the sketch feature extraction network uses a pre-trained Visual Transformer model to extract global features and multi-layer local features from the complete sketch and sketch components, respectively, and realizes feature fusion through feature splicing.
5. The method for generating a color 3D model based on sketch and text according to claim 1, characterized in that: In step 2, text features are extracted based on deep bidirectional context information by pre-training the BERT model.
6. The method for generating a color 3D model based on sketch and text according to claim 1, characterized in that: In step 2, the hand-drawn sketch features, sketch component features and text features are fused through a multi-head attention mechanism.
7. The method for generating a color 3D model based on sketch and text according to claim 1, characterized in that: In the step 3, a joint diffusion model based on UNet is constructed, and point cloud shape, color and component classification are generated by synchronous training; in the diffusion model, a classifier-free guidance method is introduced, and an unconditional generation model and a conditional generation model are trained simultaneously, and the three-dimensional model reconstruction result is inferred by a linear combination of the two; in each layer of the UNet structure, a cross-attention mechanism based on projection is used to introduce the sketch and text fusion features output by step 2, so that the fusion features interact with the corresponding projection position point cloud features under the camera perspective.
8. The method for generating a color 3D model based on sketch and text according to claim 7, characterized in that: In step 3, the loss function of the diffusion model is set to: in, represents the expected function, ∈ and t represent the sampling noise and time step respectively, x0 represents the point cloud point, c0 represents the point cloud color, s0 represents the point cloud classification, x t ,c t ,s t represents the corresponding noisy vector, C represents the feature condition of sketch and text fusion, and f represents the UNet network.
9. The method for generating a color 3D model based on sketch and text according to claim 1, characterized in that: In step 3, in the sampling stage, starting from Gaussian noise, a point cloud three-dimensional model of joint features is gradually generated through iterative training.
10. A color 3D model generation system based on sketch and text, comprising a sketch hand-drawn input module, a sketch segmentation and text feature fusion module and a 3D model generation module, characterized in that: The sketch hand-drawing input module is used to obtain the hand-drawn sketch; The sketch segmentation and text feature fusion module divides the input sketch into parts and extracts features, and fuses the hand-drawn sketch features, sketch part features and extracted text features; The three-dimensional model generation module constructs a three-dimensional model generation network, and introduces the fusion features of the sketch and text obtained by the sketch segmentation and text feature fusion module into the cross-attention mechanism it adopts, so as to train the three-dimensional model generation network to generate a three-dimensional model with color and component classification.