Three-dimensional model construction method and system, and related device

Through multi-stage sketch processing and pre-training neural networks, the complex and time-consuming problems of existing three-dimensional modeling methods are solved, and the ability of non-professional users to create high-quality 3D models is realized, which improves efficiency and accuracy, and is suitable for multiple application fields.

WO2024230843A1PCT designated stage expired Publication Date: 2024-11-14MOXIN (HUZHOU) TECH CO LTD

Patent Information

Application Number
PCT/CN2024/092742
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2024-05-11
Publication Date
2024-11-14

Smart Images

  • Figure CN2024092742_14112024_PF_FP_ABST
    Figure CN2024092742_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention relates to the technical fields of computer graphics and computer-aided graphic designs. Provided are a sketch-based three-dimensional model construction method and system, and a related device. The three-dimensional model construction method comprises: step one, receiving a user input on an interactive interface; step two, converting the user input into one or more sketches, and then presenting same on the interactive interface; step three, a user selecting an appropriate sketch from the interactive interface; and step four, generating a three-dimensional model by using the sketch. By means of converting a user input into a sketch which is relatively easily edited, and then converting the sketch into a three-dimensional model, the sketch is used as an intermediary between the three-dimensional model and other data modalities, such that the performance consumption by the rendering of an interactive interface can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

A three-dimensional model construction method, system and related equipment Technical Field

[0001] The present invention relates to the technical field of computer graphics and computer-aided graphic design, and in particular to a sketch-based three-dimensional model construction method, system and related equipment. Background Art

[0002] In recent years, the demand for 3D content has increased significantly, driven by the rise of virtual and augmented reality technologies and the demand for immersive experiences in various fields, including entertainment, education, and product design. However, creating professional 3D models remains a challenge for everyone. The widely adopted 3D modeling method using computer-aided design (CAD) software is a challenging and time-consuming process that requires a high level of expertise. CAD software typically requires users to have extensive knowledge of 3D modeling techniques and the ability to use complex interfaces and tools. This creates a barrier for many potential creators, who may have great ideas but lack the necessary skills to bring them to life.

[0003] Therefore, there is a need for a more convenient and intuitive 3D modeling method and system that enables anyone to easily create high-quality 3D models without the need for extensive training or expertise. Some recent work has achieved AI 3D modeling using only text-based input, but it may be difficult for novice users to accurately describe the 3D model they want in detail, especially when it comes to complex or abstract shapes. Although there is currently 3D modeling based on image input, it is difficult for image input to represent the true intention of the desired model, especially the perspective ambiguity contained in a single photo, and bitmaps are difficult to edit. Users sometimes need to adjust the image multiple times, which takes a lot of time to obtain the desired model. How to use an intermediate data representation generated by 3D content that is easy for users to handle and edit to promote the customization and smoothness of the 3D modeling process has become a technical problem that needs to be solved urgently.

[0004] In fact, compared to images or text, sketches contain important spatial relationships and contextual information, conveying the shape, size, and position of objects. Therefore, sketches may be a good input for AI 3D modeling. However, how to make good use of this modality of input and integrate it into the existing 3D modeling pipeline has become another technical problem that needs to be solved urgently.

[0005] However, modeling with sketches differs from using text and single images. Sketches are inherently sparse and fuzzy, lack color information, and often represent only simplified versions of predetermined objects with boundaries. Existing approaches use precise line drawings from multiple views or employ step-by-step multi-view sketching workflows, but these approaches are not user-friendly and can be time-consuming. While there have been some advances in utilizing single-view sketches and encoder-decoder deep neural networks to ease the 3D modeling process, these approaches still face significant challenges in obtaining high-fidelity 3D models due to the inherent abstractness and fuzziness of sketches, the fact that a single sketch can generate a variety of 3D models, and the significant domain gap between sketches and 3D shapes. Furthermore, existing deep learning methods require large amounts of data, yet currently lack a large number of training data pairs for hand-drawn sketches and 3D models. This lack of data also limits the output effectiveness of existing learning-based methods. Therefore, those skilled in the art are in need of designing a method that can obtain relatively accurate 3D models without requiring a large amount of training data pairs.

[0006] Summary of the Invention

[0007] This paper proposes a multi-stage 3D model construction method. This method uses sketches as an intermediate medium, converting various inputs into sketches. These sketches are then modified to achieve the user's desired construction input. This input is then used for 3D modeling, enabling full user customization and integrating into existing 3D modeling pipelines, such as those based on text or image input. To compensate for the information deficit caused by the sparseness and abstractness of sketches, a pre-trained neural network aligned to the same latent space across multiple modalities is used as additional input, facilitating the generation of sketches and final 3D models that are more detailed and more realistic.

[0008] Since the sketch itself does not carry color information, after the sketch is used to generate a 3D model, a colored / specific style 3D model is generated based on the user's text description, which can be better utilized in various fields such as games and design.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] The present invention provides a sketch-based three-dimensional model construction method for processing multi-stage sketches. The three-dimensional model construction method includes:

[0011] Step 1: Receive user input on an interactive interface. The user input may be a reference image, a text description, a voice description, a partially missing sketch, a semantically segmented image, or an interactive questionnaire. The interactive questionnaire may include one or more questions that guide the user to describe the desired 3D model's features from different angles, such as "category" or "shape."

[0012] The second step is to convert the user input into one or more sketches and present them on the interactive interface;

[0013] In the third step, the user selects the appropriate sketch in the interactive interface;

[0014] The fourth step is to use the sketch to generate a three-dimensional model.

[0015] Furthermore, the method further includes the user editing the generated sketch in the interactive interface. If the user is not satisfied with the generated sketch, the user returns to the first step and re-enters the sketch;

[0016] Editing is done using the artboard brush or sketching using text guides.

[0017] Furthermore, the method also includes coloring and / or stylizing the three-dimensional model by receiving text, image or voice description provided by the user.

[0018] Furthermore, the 3D model construction process displays the generated results in real time; and when the 3D model is colored and / or stylized, the user interface displays the progress in real time.

[0019] Furthermore, the generated 3D model is supervised to ensure that the 3D model matches the user input as closely as possible. The specific supervision process is as follows:

[0020] A neural network is used to map the rendered outline of the 3D model into a vector A, and another neural network is used to map the text, image, or audio based on the sketch into another vector B. The similarity between vectors A and B is calculated, and the neural network is trained to minimize the similarity so that the rendered outline and the text, image, or audio description match as closely as possible.

[0021] Among them, the neural network is a pre-trained encoder aligned to the same latent space under multiple modalities.

[0022] In addition, the 3D model can be optimized after it is generated.

[0023] Furthermore, in the first step, the user input includes a description of the category and shape of the 3D model.

[0024] Furthermore, in the second step, the user input is converted through a modal conversion neural network or one or more sketches are generated by image extraction. The specific modal conversion neural network can be converted through a neural network with a VQ-VAE structure, a generative adversarial model, or a diffusion model.

[0025] Furthermore, the specific steps for generating a 3D model using a sketch are as follows:

[0026] (5.1) Receive a sketch, where the sketch can be directly converted or edited by the user.

[0027] (5.2) Extract features from the sketch through the encoder to obtain a compressed vector Z;

[0028] (5.3) The decoder uses vector Z to reconstruct or generate a three-dimensional model

[0029] Furthermore, when a sketch generates a three-dimensional model, the user input is used as additional information input to optimize the generated three-dimensional model through a pre-trained neural network aligned to the same latent space in multiple modalities, and the multiple modalities at least include the modality corresponding to the user input information and the picture or sketch modality. In addition to the above method, the use of sketches to generate three-dimensional models can also use the projection image or point cloud of the existing three-dimensional model in the database to retrieve the closest three-dimensional model in the database based on similarity. If no close three-dimensional model is retrieved in the database, it is generated using the above method. The specific method is to project the sketch input by the user, and calculate the similarity between the obtained projection image and the projection images of the three-dimensional model rendered at different angles in the database. The similarity calculation index can be mIoU, Dice distance, structural similarity factor PCC, etc.

[0030] Furthermore, step (5.1) also includes receiving a category description of the sketch, where the category of the sketch is manually specified by the user or obtained by a neural network sketch classifier through a pre-training method.

[0031] Furthermore, the category description is an image description rendered according to the three-dimensional model.

[0032] Furthermore, in step (5.3), the decoder generates a three-dimensional model based on a specific template, where the template is a general template or a three-dimensional model template designed or obtained for a certain category.

[0033] Furthermore, the similarity and / or symmetry and / or surface smoothness between the 3D model and the sketch are optimized through a loss function.

[0034] Furthermore, the three-dimensional model is colored by creating a new color value for each vertex of the three-dimensional model, then rendering the three-dimensional model with the color value, comparing the rendering result with the coloring description input by the user, optimizing the color value through the comparison result, and obtaining a three-dimensional model colored in accordance with the user input.

[0035] Furthermore, the specific steps for coloring the three-dimensional model are as follows:

[0036] (6.11) receiving a user-generated textual, visual, or spoken description of the three-dimensional model, wherein the colored description is in the form of a descriptive text describing the three-dimensional model corresponding to the sketch, a user-desired style diagram, or a user-speaking or machine-generated spoken description of the sketch; wherein the colored description may be a previously input modal content as a prompt to generate a colored three-dimensional model, or a user-specified new prompt to generate a colored three-dimensional model;

[0037] (6.12) The colored description is generated by aligning the pre-trained neural network in multiple modalities to the same latent space to generate a vector A1 in a high-dimensional space;

[0038] (6.13) storing each vertex of the generated three-dimensional model in a colored neural radiance field by performing a position encoding, and then assigning a color vector to each vertex position of the three-dimensional model; the colored neural radiance field is a multi-layer perceptron that contains color information;

[0039] (6.14) Add the color vector of each vertex position of the three-dimensional model to the generated three-dimensional model;

[0040] (6.15) Optimize the colored neural radiation field with color vectors;

[0041] The specific steps of the optimization are: (6.151) using a differentiable renderer to render and obtain a colored rendering;

[0042] (6.152) Encode the colored rendering into a vector B1 through a pre-trained neural network (image encoder) aligned to the same latent space under multiple modalities;

[0043] (6.153) Calculate the similarity between vector B1 and vector A1, and train the neural network to minimize the similarity so that the colored rendering and the model colored description match as closely as possible;

[0044] (6.154) The parameters of the neural network are optimized through the loss function L1, and the loss function L1 is calculated as: L1 = 1-Cos<A1,B1> , where Cos<A1,B1> is the similarity between B1 and vector A1.

[0045] (6.16) The optimized colored neural radiation field is converted back into a three-dimensional model, thereby obtaining a colored three-dimensional model.

[0046] Furthermore, in step (6.13), Fourier feature maps are used for position encoding to learn high-frequency function insertion, and then the position encoding of each vertex is passed to the colored neural radiation field, and each vertex of the three-dimensional model is stored in a colored neural radiation field.

[0047] Furthermore, the generated 3D model is rendered in a uniformly sampled interval, and the similarity with the user input is calculated. The perspective with the highest similarity is selected as the anchor perspective. Then, a Gaussian distribution centered on the anchor perspective is used to render multiple perspectives from randomly sampled views, and the 3D model is optimized through multiple perspectives.

[0048] Furthermore, the three-dimensional model is stylized by creating a new displacement value for each vertex of the three-dimensional model, then rendering the three-dimensional model with the displacement value, comparing the rendering result with the stylized description input by the user, optimizing the displacement value through the comparison result, and obtaining a stylized three-dimensional model that conforms to the user input.

[0049] Furthermore, in the sixth step, the specific steps of stylizing the 3D model are as follows:

[0050] (6.21) receiving a user's stylized description of the three-dimensional model in the form of a textual description of the three-dimensional model corresponding to the sketch, an image of the style desired by the user, or a user-speaking or machine-generated voice description of the sketch; the stylized description may be a previously input modal content as a prompt to generate a three-dimensional model of a specific style, or a user-specified new prompt to generate a three-dimensional model of a specific style;

[0051] (6.22) The stylized description is generated into a high-dimensional space vector A2 by aligning the pre-trained neural network in multiple modalities to the same latent space;

[0052] (6.23) storing each vertex of the three-dimensional model generated by the sketch in a stylized neural radiance field by performing a position encoding, and then assigning a displacement vector relative to the normal to each vertex position of the three-dimensional model; the stylized neural radiance field is a multi-layer perceptron that contains displacement information;

[0053] (6.24) Add the normal displacement of each vertex position of the three-dimensional model to the generated three-dimensional model;

[0054] (6.25) Optimize the stylized neural radiation field with displacement vectors. The specific steps are:

[0055] (6.251) Use a differentiable renderer to render and obtain a stylized rendering;

[0056] (6.252) The stylized rendering is passed through a pre-trained neural network (image encoder) aligned to the same latent space under multiple modalities to generate a vector B2;

[0057] (6.253) Calculate the similarity between vector B2 and vector A2, and minimize the similarity by training the neural network so that the stylized rendering and the model stylized description match as closely as possible;

[0058] (6.254) The parameters of the neural network are optimized through the loss function L2, and the loss function L2 is calculated as: L2 = 1-Cos<A2,B2> , where Cos<A2,B2> is the similarity between B2 and vector A2.

[0059] (6.26) The optimized stylized neural radiation field is converted back into a three-dimensional model, thereby obtaining a stylized three-dimensional model.

[0060] Furthermore, the construction method further includes:

[0061] Editing specific parts or areas of a colored and / or stylized 3D model. The editing may include:

[0062] Use the anchor points on the 3D model surface to drag and move the entire part;

[0063] And / or, performing operations such as overall scaling, stretching, and translation using a selected area on the surface of the three-dimensional model, wherein the area can be selected by a selection box.

[0064] In the sketch-to-3D model stage, this application uses a pre-trained neural network aligned to the same latent space in multiple modalities as additional information input to generate a 3D model (using a batch of models to optimize the same neural network), and then uses the same pre-trained neural network aligned to the same latent space in multiple modalities to generate a model with a specific color (coloring processing) and style (stylization processing) (single model optimization). Using the same network and generalization-testing optimization dual-stage, the resulting model can be made more in line with user needs. The present invention also provides a sketch-based 3D model system, the system comprising:

[0065] The sketch generation module is a modality conversion neural network used to convert user input into sketches;

[0066] Sketch encoding module, used to encode the sketch into a vector;

[0067] A decoding module is used to generate a three-dimensional model by converting the vector into a position of a triangular face or a point cloud or a displacement based on a specific template;

[0068] A rendering module, used for rendering the generated three-dimensional model into a contour image of a given perspective;

[0069] A supervision module is used to train a neural network to make the contour map match the user input as closely as possible, where the user input is a reference image, a text description, a voice description, a partially missing sketch, a semantically segmented image, or an interactive questionnaire.

[0070] Furthermore, the system further comprises a coloring module for coloring the three-dimensional model by the user based on the coloring description of the three-dimensional model in text, image or voice;

[0071] And / or, a stylization module is used to color the 3D model based on the user's stylized description of the 3D model in text, image or voice.

[0072] Furthermore, the coloring module specifically includes:

[0073] A coloring prompt encoding module is used to receive coloring prompts in text, image, or voice, and encode the coloring prompts into a vector A1 in a high-dimensional space;

[0074] The colored position encoding module encodes the generated 3D model into a colored neural radiation field;

[0075] A coloring renderer is used to render the colored three-dimensional model to obtain a colored rendering;

[0076] A color supervision module, which is a pre-trained neural network (encoder) aligned to the same latent space under multiple modalities. It encodes the colored rendering into a vector B1 in a high-dimensional space. Because the color supervision module and the color hint encoding module are pre-trained to align to the same latent space under multiple modalities, the vectors A1 and B1 are in the same vector space. In this vector space, the similarity between vectors A1 and B1 is calculated. The neural network is trained to minimize the similarity, so that the colored rendering and the color hint of the text, image, or speech are matched as closely as possible, achieving the desired model color distribution.

[0077] Among them, the colored neural radiation field includes a multi-layer perceptron, which stores the color vector of each vertex position in the three-dimensional model, inputs the vertex position information, and outputs the color vector of the vertex position.

[0078] Furthermore, the stylization module specifically includes:

[0079] A stylized hint encoding module is used to receive stylized hints of text, image, or speech, and encode the stylized hints into a vector A2 in a high-dimensional space;

[0080] Stylized position encoding module, which encodes the generated 3D model into a stylized neural radiation field;

[0081] A stylized renderer is used to render the stylized 3D model to obtain a stylized rendering;

[0082] The stylization supervision module is a pre-trained neural network (encoder) aligned to the same latent space under multiple modalities. It encodes the stylized rendering into a vector B2 in a high-dimensional space. Because the colorization supervision module and the colorization hint encoding module are pre-trained to align to the same latent space under multiple modalities, vectors A2 and B2 are in the same vector space. In this vector space, the similarity between vectors A2 and B2 is calculated. The neural network is trained to minimize the similarity, ensuring that the stylized rendering matches the stylized hint of the text, image, or speech as closely as possible.

[0083] The stylized neural radiation field includes a multi-layer perceptron, which stores the displacement of each vertex position in the three-dimensional model, inputs vertex position information, and outputs the displacement of the vertex position relative to the normal.

[0084] Furthermore, the specific implementation process of the supervision module is as follows:

[0085] The supervision module is one or more neural networks, which are encoders in a multimodal model that have been pre-trained on text-image pairs or speech-image pairs. One neural network maps the rendered contour image into a vector A, and another neural network maps the text, image, or audio based on the sketch into another vector B. The supervision module maps the text and / or picture and / or speech input to a shared feature space, calculates the similarity between vector A and vector B in this vector space, and minimizes the similarity by training the neural network so that the contour image rendered by the rendering module matches the text, image, or speech description as much as possible.

[0086] The present invention also provides a parts processing method, the parts processing method comprising:

[0087] The first step is to obtain a sketch by using a modal conversion neural network or image extraction.

[0088] In the second step, the sketch uses the above method to construct a three-dimensional model;

[0089] The third step is to input the generated 3D model into the parser and convert it into machine-readable instructions;

[0090] The fourth step is that the processing device performs processing according to machine-readable instructions.

[0091] Furthermore, the specific steps of obtaining the photo by image extraction are:

[0092] First, the photos were smoothed using a Gaussian filter to reduce noise and improve image quality;

[0093] Then, calculate the gradient and gradient direction of the image;

[0094] Next, the image is processed with non-maximum suppression to eliminate non-edge pixels and retain only some thin lines as candidate strokes;

[0095] Finally, the edge detection process is completed by applying high and low thresholds and connecting the edges to obtain a processed generated sketch.

[0096] Furthermore, after obtaining the sketch, the sketch is normalized to a fixed size and placed at the center of the added image.

[0097] The present invention also provides a parts processing device, which is used for the above-mentioned processing method, including a motion component, an actuator, and a controller; the controller receives user input and runs the above-mentioned three-dimensional model construction method to obtain a three-dimensional model. The combination of the motion component and the actuator can be a 3D printer or a CNC engraving machine.

[0098] The present invention also provides an electronic terminal, which executes the above-mentioned three-dimensional model construction method.

[0099] A post-editing method for generating a three-dimensional model based on a sketch, characterized in that the post-editing method comprises:

[0100] (1) Selecting a corresponding area of ​​the three-dimensional model according to a region prompt given by the user, wherein the prompt is text, voice or picture;

[0101] (2) Editing the corresponding area of ​​the selected three-dimensional model, wherein the editing includes overall stretching, deleting, and smoothing operations on the triangular faces or points in the area.

[0102] Specifically, the selected area is edited, and the editing may include:

[0103] (2.1) Use the anchor points on the 3D model surface to drag and move the entire part;

[0104] And / or, (2.2) using the selected area on the surface of the three-dimensional model to perform operations such as overall scaling, stretching, and translation.

[0105] Specifically, the specific process of step (1) is:

[0106] (11) receiving a user's text, image, or voice description of a region of the three-dimensional model, wherein the region description is in the form of a descriptive text of the three-dimensional model corresponding to the drawn sketch, a style map expected by the user, or a user-read or machine-generated voice description of the sketch;

[0107] (12) The region description is generated into a high-dimensional space vector A3 by aligning the pre-trained neural network in multiple modalities to the same latent space;

[0108] (13) storing each vertex of the generated three-dimensional model in a regional neural radiance field by performing a position encoding, and then assigning a probability p to the position of each vertex of the three-dimensional model; the regional neural radiance field is a multi-layer perceptron containing the probability p; wherein the probability p refers to the probability that the vertex belongs to the region;

[0109] (14) The color value corresponding to the probability p of each vertex position of the three-dimensional model is added to the model.

[0110] Specifically, the color of the vertices of the three-dimensional model is a pure color, that is, p=1 or 0,

[0111] Alternatively, the colors of the vertices of the three-dimensional model are two colors mixed according to probability, the two colors are the foreground color and the background color, and the color mixing method is:

[0112] p*(High)+(1-p)*Grey;

[0113] Among them, High represents a color;

[0114] Grey represents another color.

[0115] (15) Optimizing the regional neural radiation field with probability p;

[0116] The optimization process is specifically as follows:

[0117] (15.151) Render using a differentiable renderer to obtain a colored rendering;

[0118] (15.152) Encode the colored rendering into a vector B3 using a pre-trained neural network (image encoder) aligned to the same latent space under multiple modalities;

[0119] (15.153) Calculate the similarity between vector B3 and vector A3, and train the neural network to minimize the similarity so that the colored rendering and the model colored description match as closely as possible;

[0120] (15.154) The parameters of the neural network are optimized through the loss function L3, and the loss function L3 is calculated as: L1 = 1-Cos<A3,B3> , where Cos<A3,B3> is the similarity between B3 and vector A3.

[0121] (15.16) The optimized colored neural radiation field is converted back into a three-dimensional model, thereby obtaining a colored three-dimensional model.

[0122] (15.17) The connected vertex set or connected facet set whose color or probability is greater than a certain threshold (such as p>0.5) in the obtained model is taken out, and the area selected by the algorithm is obtained.

[0123] Furthermore, in step (15.13), Fourier feature maps are used for position encoding to learn high-frequency function insertion, and then the position encoding of each vertex is passed to the colored neural radiation field, and each vertex of the three-dimensional model is stored in a colored neural radiation field.

[0124] Furthermore, the generated 3D model is rendered in a uniformly sampled interval, and the similarity with the user input is calculated. The perspective with the highest similarity is selected as the anchor perspective. Then, a Gaussian distribution centered on the anchor perspective is used to render multiple perspectives from randomly sampled views, and the 3D model is optimized through multiple perspectives.

[0125] Alternatively, the specific process of step (1) is:

[0126] (16) The three-dimensional model is rendered from multiple angles, and then semantic segmentation or box detection is performed on the rendered image based on the region description input by the user. The detection results from different perspectives are fused into the three-dimensional model, and the connected regions with a probability greater than a certain threshold after fusion are taken out as the selected regions specified by the user.

[0127] Alternatively, the specific process of step (1) is to first use a model segmentation algorithm, such as GraphCut, to obtain several selection candidates, then use any method (11)-(16) to obtain the selection area R, and select the n areas that include the most R in the selection candidates as the user-specified selection area, where n is a number greater than or equal to 1.

[0128] The present invention has the following beneficial effects:

[0129] (1) The present invention provides an efficient and economical 3D modeling method that can fully meet the user's customized needs by converting the prompt information input by the user into a sketch and then converting the sketch into a 3D model;

[0130] (2) The sketch used as a transition mode in the present invention is generated by a computer. It is different from the hand-drawn sketch in the general sense. The hand-drawn sketch usually has a lot of redundant strokes or lacks strokes and is very abstract. The difference in painting styles of different people leads to a huge domain gap. The sketch in the present invention is a "simple edge map" generated by a computer. It can more accurately depict the key features of the geometric shape of the object and is a compact representation. Such a modeling method of converting through a transition mode can reduce the performance consumption of the rendering of the interactive interface. In addition, since the computing resources required to convert the modalities such as pictures, texts, and voices into two-dimensional sketches are usually less than the computing resources required to convert them into three-dimensional models, using sketches as an intermediary process can reduce the number of times that users consume a lot of computing power to generate three-dimensional models in order to meet their needs, which helps to improve the concurrent capacity of the system.

[0131] (3) In the sketch generation model stage, the present invention uses a pre-trained neural network aligned to the same latent space under multiple modalities as additional information input to compensate for the information deficiency caused by the sparseness and abstractness of the sketch. Compared with the direct application of existing 3D modeling methods based on pictures or text, the additional information input strategy adopted by the present invention helps to generate sketches with more details and more consistent with real objects;

[0132] (4) Although the sketch itself does not contain color information, the present invention uses a pre-trained neural network aligned to the same latent space under multiple modalities to generate a three-dimensional model of a specific color and style based on the user's text description, making it possible to generate colorful three-dimensional models required in various fields such as games and design; the present invention adopts a two-stage strategy of model generation and then coloring, which can achieve real-time modeling adjustment in the model generation stage to improve user experience, and optimize each model separately in the coloring / stylization stage to achieve more accurate and diverse model content creation.

[0133] (5) The present invention provides a strategy for automatically selecting areas for editing after generating a model, enabling users to further freely edit the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0134] FIG1 is a schematic diagram of a module flow chart taking a lamp as an example in Example 1.

[0135] FIG2 is a schematic diagram of a three-dimensional model obtained using the method in Example 1.

[0136] FIG3 is a three-dimensional model for generating additional text prompts.

[0137] FIG4 is a three-dimensional model generated without additional text prompts.

[0138] FIG5 is a simple schematic diagram of the system in Example 2.

[0139] FIG6 is a comparison diagram of three-dimensional models generated with and without a supervision module.

[0140] [Corrected 27.06.2024 according to Rule 26] In Figure 1, Input, 17;

[0141] [Corrected 27.06.2024 in accordance with Rule 26] Input Sketch, 1;

[0142] [Corrected 27.06.2024 according to Rule 26] Text Prompt T, 2;

[0143] [Corrected 27.06.2024 according to Rule 26] Pre-trained multimodal model image encoder (CLIP Image Encoder), 3;

[0144] [Corrected 27.06.2024 according to Rule 26] Pre-trained multimodal model text encoder (CLIP Text Encoder), 4;

[0145] [Corrected 27.06.2024 according to Rule 26] Sketch encoding module (Encoder E), 5;

[0146] [Corrected 27.06.2024 according to Rule 26] Sketch decoding module (Decoder D), 6;

[0147] [Corrected 27.06.2024 in accordance with Rule 26] Supervision Module (CLIP Loss 1), 7;

[0148] [Corrected 27.06.2024 according to Rule 26] Generated model (Mesh), 8;

[0149] [Corrected 27.06.2024 according to Rule 26] Positional Embedding, 9;

[0150] [Corrected 27.06.2024 according to Rule 26] One neural radiation field (Ns), 10;

[0151] [Corrected 27.06.2024 according to Rule 26] A neural radiation field (Nc) storing color information, 11;

[0152] [Corrected 27.06.2024 according to Rule 26] A neural radiation field (Nd) storing style information, 12;

[0153] [Corrected 27.06.2024 in accordance with Rule 26] Displacement, 13;

[0154] [Corrected 27.06.2024 according to Rule 26] color, 14;

[0155] [Corrected 27.06.2024 according to Rule 26] Three-dimensional model after coloring and stylization (Stylized Mesh), 15; [0155.1] [Corrected 27.06.2024 according to Rule 26] Stylized loss function (CLIP Loss 2), 16; [0155.2] [Corrected 27.06.2024 according to Rule 26] Differentiable Renderer (Differentiable Renderer), 18. DETAILED DESCRIPTION

[0156] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be pointed out that the embodiments are only specific explanations of the invention and should not be regarded as limitations of the invention. The purpose of the embodiments is to enable those skilled in the art to better understand and reproduce the technical solutions of the present invention. The scope of protection of the present invention shall still be based on the scope defined by the claims.

[0157] Example 1

[0158] As shown in FIG1 , this embodiment provides a sketch-based 3D model construction method for processing multi-stage sketches. The 3D model construction method includes:

[0159] Step 1: Receive user input on an interactive interface. The user input may be a reference image, a text description, a voice description, a partially missing sketch, a semantically segmented image, or an interactive questionnaire. The interactive questionnaire may include one or more questions that guide the user to describe the characteristics of the desired 3D model from different angles, including the type and shape of the 3D model.

[0160] In the second step, the user input is converted through a modal conversion neural network or generated into one or more sketches using image extraction and then presented on the interactive interface; the modal conversion neural network can be specifically converted through a neural network with a VQ-VAE structure, a generative adversarial model, or a diffusion model.

[0161] In the third step, the user selects the appropriate sketch in the interactive interface;

[0162] In the fourth step, the user edits the sketch in the interactive interface. If the user is not satisfied with the generated sketch, he or she can return to the first step and re-enter it. This step is optional. The editing method can be edited using the sketchpad brush or text-guided sketch editing.

[0163] The fifth step is to generate a 3D model using the sketch. This step is optional. The projected image or point cloud of the existing 3D model in the database can be used to retrieve the closest 3D model in the database based on similarity. Specifically, the sketch input by the user is projected, and the similarity between the obtained projected image and the projection images of the 3D model rendered at different angles in the database is calculated. The similarity calculation indicators can be mIoU, Dice distance, structural similarity factor PCC, etc.

[0164] When no similar 3D model can be retrieved from the database, a 3D model is generated using the following method.

[0165] The specific steps to generate a 3D model using a sketch are:

[0166] (5.1) Receive a sketch, which can be a directly converted sketch or a sketch edited by the user. A text, image, or voice description of the sketch can also be input. Specifically, the sketch can be a textual description of the user's desired sketch, such as "ALEGO-style airplane"; a human or machine-generated voice describing the sketch; or a picture of the desired style, such as a picture of a Lego airplane. Alternatively, the sketch can be a computer-generated description of the sketch, which can be generated from a fixed template, such as "a [object]"; and a description of the sketch's category can be included to reduce the network search domain for generating three-dimensional models from the sketch, thereby obtaining the desired result more quickly. For example, if the user wants the sketch to describe the class of an airplane, the textual description T is "an airplane." The class can be manually specified by the user or obtained by a neural network sketch classifier through a pre-training method. Specifically, the neural network sketch classifier can be composed of the following structure: a sketch encoder, including a down-sampling convolutional neural network, a classifier, including several fully connected neural networks (MLPs), and an activation function between each layer, such as a ReLU activation function; the neural network sketch classifier is trained on categories with category identifiers and corresponding sketches. By specifying the sketch category, the network search domain for generating a three-dimensional model from a sketch can be reduced, so that the desired result can be obtained faster.

[0167] As a preferred technical solution, the description can be explained based on the image rendered by the three-dimensional model through the renderer. For example, if the objects rendered by the rendering module are all black outlines, the description of the sketch can be "a black [object]" (a black [name of the category to which the sketch belongs]); if the object rendered by the rendering module is a gray projection image, the description of the sketch can be "a grey [object]" (a gray [name of the category to which the sketch belongs]).

[0168] (5.2) extracting features from the sketch using an encoder to obtain a compressed vector Z; the encoder may be a convolutional neural network including one or more convolutional layers, or the encoder portion of a residual neural network with a ResNet structure;

[0169] (5.3) The decoder uses the vector Z to reconstruct or generate a three-dimensional model to obtain a three-dimensional model; the decoder D can be a multi-layer perceptron with n outputs, where the n outputs are the positions of each vertex or facet, or it can generate a model based on a specific template. The template can be a general template, such as a spherical three-dimensional model, or a three-dimensional model template designed or obtained for a certain category. For example, if the user wants the hand-drawn sketch model to be a car, then the three-dimensional model of a car already in the dataset can be used as the template. Alternatively, if the user wants the hand-drawn sketch model to be a human body, then the average shape of all human body data in the dataset can be used as the template.

[0170] Preferably, key point extraction can be performed on the universal template. This involves extracting key control points that influence shape variation from all corner points of the model. This can be achieved through manual annotation or by automatically learning the model within the dataset and selecting the K points with the largest displacement range. During the learning process, regression can be used to determine the spatial displacement range of the control points within each template. This range distribution is then applied to the decoder as a conditional constraint, forcing the decoder to produce more natural results without generating excessively variable, abnormal results.

[0171] (5.4) Optimizing the generated three-dimensional model using a pre-trained neural network aligned to the same latent space in multiple modalities as additional information input, wherein the multiple modalities include at least the modality corresponding to the user input information and the image or sketch modality, specifically:

[0172] (5.41) Rendering the 3D model using a differentiable renderer to obtain a rendering image, and calculating the similarity (IoU) between the rendering image and the sketch; the rendering image is a given perspective, and the 3D model can obtain a perspective pi (i=1, 2, ...V). As a preferred technical solution, the sampling of the rotation angle of the 3D model can adopt Gaussian distribution sampling or average sampling; the differentiable renderer can be a SoftRas renderer;

[0173] (5.42) The rendering image is converted into a feature vector Z1 through an encoder E1, and the user input is converted into a feature vector Z2 through another encoder E2. The similarity Cos between the feature vector Z1 and the feature vector Z2 is calculated.<Z1,Z2> ; The cosine similarity factor Cos is used for similarity calculation. If there is more than one rendering image P, the same operation is performed on each image Pi, and the obtained cosine similarity factors are averaged to obtain the Cos value. Encoder E1 and encoder E2 come from encoders in a multimodal model that has been pre-trained on a large number of text and image pairs. When the input T is text, E2 is a text encoder, and when the input T is a picture, E2 is a picture encoder. Large-scale pre-training of E1 and E2 can map text and / or picture inputs to a shared feature space, calculate the similarity in this shared feature space, and minimize the similarity by training the neural network, so that the result rendered by the rendering module can match the text description T or the image input I as much as possible.

[0174] The multimodal model can be an existing CLIP (Contrastive Language-Image Pretraining) model or a GLIP (Grounded Language-Image Pretraining) model. Taking the CLIP model as an example, it is pre-trained using paired image-text pairs (for example, if the input is a picture of a dog, the corresponding text also indicates that it is a dog). These text and image are respectively output through the CLIP text encoder and the CLIP image encoder. Contrastive learning is then performed on these output text features and image features, and the learned text encoder and image encoder share the same feature space.

[0175] During the rendering process, multi-view sampling is performed. The selection of the view can be uniform sampling within a range of view angles or Gaussian distribution sampling. In addition, the image can be rotated and randomly cropped. The optimizer can be an Adam optimizer or an SGD optimizer.

[0176] (5.43) The similarity calculated in step (5.42) and step (5.41) is used to obtain the loss function L, and the similarity Cos in step (5.42) is used to obtain the loss function L.<Z1,Z2> Take the negative sign and add one to the similarity IoU in step (5.41) for weighted addition to get the loss function L. The specific expression is: L = l(1-Cos<Z1,Z2> ) + kIoU, where l and k are manually set parameters, with l = k = 0.1. By inputting a large number of samples and using an optimizer to adjust the parameters of the encoder E, this loss function is minimized. Preferably, training can be performed using data pairs from only one item class, or a mixture of multiple item classes.

[0177] You can use data pairs from only one item class for training, or you can use a mixture of multiple item classes for training.

[0178] As a preferred technical solution, the loss function L also includes a Laplacian Flatten loss.

[0179] As a preferred technical solution, the loss function L also includes a model symmetry loss Lvsym, and the calculation formula of the symmetric loss is:

[0180] Among them, v i are the vertices for generating the three-dimensional model,

[0181] v j It is to generate the j-th vertex of the three-dimensional model;

[0182] N is the total number of vertices in the generated 3D model

[0183] T is the transformation to generate the 3D model

[0184] n is the unit vector along the normal direction; I is the sketch.

[0185] As a preferred technical solution, the loss function L also includes a model symmetry loss LIsym, which is as follows:

[0186] in, is the view in step (5.41), It is a rendering of the projection rendering of the three-dimensional model after the transformation T;

[0187] γ() i,k It is a manually set hyperparameter, which can also be left unset;

[0188] j represents the horizontal coordinate of the rendering of the three-dimensional model; k represents the vertical coordinate of the rendering of the three-dimensional model;

[0189] m represents the total number of given view angles.

[0190] In some preferred embodiments, the generated 3D model is rendered in a uniformly sampled interval, and the similarity is calculated. The perspective with the highest similarity is selected as the anchor perspective, and then a Gaussian distribution centered on the anchor perspective is used to render multiple perspectives from randomly sampled views.

[0191] Step 6: Coloring and / or stylizing the three-dimensional model by receiving a text, image, or voice description provided by the user;

[0192] The 3D model building process displays the generated results in real time; and when the 3D model is colored and / or stylized, the user interface displays the progress in real time.

[0193] Coloring a 3D model is done by creating a color value for each vertex of the 3D model, then rendering the 3D model with the color value using a differentiable renderer, comparing the rendering result with the coloring description input by the user, and optimizing the color value based on the comparison result to obtain a 3D model that matches the coloring input by the user. The specific steps for coloring a 3D model are as follows:

[0194] (6.11) receiving a user-generated textual, image-based, or voice-based description of the three-dimensional model, wherein the coloring description may be a previously input modal content as a prompt to generate a colored three-dimensional model, or a user-specified new coloring prompt to generate a colored three-dimensional model; the coloring prompt may be a descriptive text of the three-dimensional model corresponding to the sketch, such as "A yellow airplane"; or a user-desired style image, such as a picture of a yellow airplane or a yellow image; or a user-sent or machine-generated voice description of the sketch;

[0195] (6.12) Encode the colored description into a high-dimensional space vector A1 using a pre-trained neural network (encoder) aligned to the same latent space under multiple modalities; if the model colored description is a sketch text input, use a pre-trained text encoder to encode the text into a high-dimensional space vector; if the model colored description is a style image, use a pre-trained image encoder to encode the image into a high-dimensional space vector; if the model colored description is a spoken description of the sketch, use a pre-trained speech encoder to encode the speech into a high-dimensional space vector;

[0196] (6.13) storing each vertex of the generated three-dimensional model in a colored neural radiance field by performing a position encoding, and then assigning a color vector to each vertex position of the three-dimensional model; the colored neural radiance field is a multi-layer perceptron containing color information; preferably, the color vector can be described by a combination of [R, G, B] or a color combination of [H, S, V];

[0197] (6.14) Add the color vector of each vertex position of the three-dimensional model to the generated three-dimensional model;

[0198] (6.15) Optimize the colored neural radiation field with color vectors;

[0199] The specific steps of optimization are: (6.151) Use a differentiable renderer to render and obtain a colored rendering;

[0200] (6.152) Encode the colored rendering into a vector B1 through a pre-trained neural network (image encoder) aligned to the same latent space under multiple modalities;

[0201] (6.153) Calculate the similarity between vector B1 and vector A1, and minimize the similarity by training the neural network so that the colored rendering and the model colored description match as much as possible; wherein the image encoder and the text encoder pre-trained in step (6.12) are pre-trained together, and these two encoders are from encoders in a multimodal model that has been pre-trained on a large number of text and image pairs, and map the text and / or image input to a shared feature space, and calculate the similarity Cos in this shared feature space.<A1,B1> By training a neural network to minimize similarity, the colored rendering and the model's stylized description can be made to match as closely as possible. The multimodal model can be an existing CLIP (Contrastive Language-Image Pretraining) model or a GLIP (Grounded Language-Image Pretraining) model. Taking the CLIP model as an example, it is pre-trained using paired image-text pairs (for example, if the input is a picture of a dog, the corresponding text also indicates that it is a dog). These text and image are respectively output through the CLIP text encoder and the CLIP image encoder. Comparative learning is then performed on these output text features and image features, and the learned text encoder and image encoder share the same feature space.

[0202] (6.154) The parameters of the neural network are optimized through the loss function L1, and the loss function L1 is calculated as: L1 = 1-Cos<A1,B1> , where Cos<A1,B1> is the similarity between B1 and vector A1.

[0203] (6.16) The optimized colored neural radiation field is converted back into a three-dimensional model, thereby obtaining a colored three-dimensional model.

[0204] Preferably, in step (6.13), Fourier feature mapping is used for position encoding to learn high-frequency function insertion, and then the position encoding of each vertex is passed to the colored neural radiation field, and each vertex of the three-dimensional model is stored in a colored neural radiation field.

[0205] The three-dimensional model is stylized by creating a new displacement value for each vertex of the three-dimensional model, and then rendering the three-dimensional model with the displacement value with a differentiable renderer. The rendering result is compared with the stylized description input by the user, and the displacement value is optimized based on the comparison result to obtain a stylized three-dimensional model that conforms to the user input.

[0206] The specific steps for stylizing a 3D model are:

[0207] (6.21) receiving a user's stylized description of the three-dimensional model in text, image, or voice, wherein the stylized description may be a previously input modal content as a prompt to generate a three-dimensional model of a specific style, or a user-specified new stylized prompt to generate a three-dimensional model of a specific style; the stylized prompt may be in the form of a descriptive text of the three-dimensional model corresponding to the sketch, such as "A yellow LEGO-style airplane"; or a user-desired image of the desired style, such as a picture of a LEGO airplane; or a user-recited or machine-generated voice description of the sketch;

[0208] (6.22) Encode the stylized description into a high-dimensional space vector A2 using a pre-trained neural network (encoder) aligned to the same latent space under multiple modalities; if the model stylized description is a sketch text input, use a pre-trained text encoder to encode the text into a high-dimensional space vector; if the model stylized description is a style image, use a pre-trained image encoder to encode the image into a high-dimensional space vector; if the model stylized description is a speech description of the sketch, use a pre-trained speech encoder to encode the speech into a high-dimensional space vector;

[0209] (6.23) storing each vertex of the three-dimensional model generated by the sketch in a stylized neural radiance field by performing a position encoding, and then assigning a displacement vector relative to the normal to each vertex position of the three-dimensional model; the stylized neural radiance field is a multi-layer perceptron that contains displacement information;

[0210] (6.24) Add the normal displacement of each vertex position of the three-dimensional model to the generated three-dimensional model,

[0211] (6.25) Optimize the stylized neural radiation field with displacement vectors. The specific steps are:

[0212] (6.251) Rendering using a differentiable renderer to obtain a stylized rendering image, wherein the differentiable renderer may be a SoftRas renderer;

[0213] (6.252) Encode the stylized rendering into a vector B2 through a pre-trained neural network (image encoder) aligned to the same latent space under multiple modalities;

[0214] (6.253) Calculate the similarity between vector B2 and vector A2, and minimize the similarity by training the neural network so that the stylized rendering and the model stylized description match as much as possible; wherein the image encoder and the text encoder pre-trained in step (6.22) are pre-trained together, and these two encoders are from encoders in a multimodal model that has been pre-trained on a large number of text and image pairs, and map the text and / or image input to a shared feature space, and calculate the similarity Cos in the shared feature space.<A2,B2> By training a neural network to minimize similarity, the stylized rendering and the model's stylized description can be made to match as closely as possible. The multimodal model can be an existing CLIP (Contrastive Language-Image Pretraining) model or a GLIP (Grounded Language-Image Pre-training) model. Taking the CLIP model as an example, it is pre-trained using paired image-text pairs (for example, if the input is a picture of a dog, the corresponding text also indicates that it is a dog). These texts and images are respectively output through the CLIP text encoder and the CLIP image encoder. Comparative learning is then performed on these output text features and image features, and the learned text encoder and image encoder share the same feature space.

[0215] (6.254) The parameters of the neural network are optimized through the loss function L2, and the loss function L2 is calculated as: L2 = 1-Cos<A2,B2> , where Cos<A2,B2> is the similarity between B2 and vector A2.

[0216] (6.26) The optimized stylized neural radiation field is converted back into a three-dimensional model, thereby obtaining a stylized three-dimensional model.

[0217] In the seventh step, the 3D model is optimized (this step is optional). The similarity and / or symmetry and / or surface smoothness of the 3D model and the sketch are optimized through a loss function. The loss function can be Laplacian Loss and / or flatten loss.

[0218] In addition, the generation method further includes supervising the generated 3D model so that the 3D model matches the user input as closely as possible. The specific supervision process is as follows:

[0219] A neural network is used to map the rendered outline of the 3D model into a vector A, and another neural network is used to map the text, image, or audio based on the sketch into another vector B. The similarity between vectors A and B is calculated, and the neural network is trained to minimize the similarity so that the rendered outline and the text, image, or audio description match as closely as possible.

[0220] Among them, the neural network is a pre-trained encoder aligned to the same latent space under multiple modalities.

[0221] In some preferred embodiments, the construction method further comprises:

[0222] Editing specific parts or areas of a colored and / or stylized 3D model. The editing may include:

[0223] Use the anchor points on the 3D model surface to drag and move the entire part;

[0224] And / or, performing operations such as overall scaling, stretching, and translation using a selected area on the surface of the three-dimensional model, wherein the area can be selected by a selection box.

[0225] In the sketch-to-3D model stage, this application uses a pre-trained neural network aligned to the same latent space under multiple modalities as additional information input to generate a 3D model (using a batch of models to optimize the same neural network), and then uses the same pre-trained neural network to generate a model of a specific color (coloring processing) and style (stylization processing) (single model optimization). Using the same network and a dual-stage generalization-test optimization can make the resulting model more in line with user needs.

[0226] Furthermore, after obtaining the sketch, a more simplified key simplified graph can be further extracted. The extraction algorithm can also be to extract lines and corner points separately, and use the corner points to enhance the generation process.

[0227] Specifically, the extraction process first uses a stacked hourglass network as the backbone network to extract feature maps corresponding to line data represented as a line field L^n and endpoint candidates P^m. The predicted line segments and endpoints are bound together to form an initialized wireframe. Specifically, in L^n, the nearest endpoint candidates are found for the two endpoint candidates P^m of the line segment candidate, denoted as x1 and x2. Similarly, the nearest endpoint candidate x1 is found for endpoint candidate y1, and the nearest endpoint candidate x2 is found for endpoint candidate y2. The squared Euclidean distance between them is then calculated, denoted as δ1 and δ2. The maximum distance is defined as δ = max(δ1, δ2). A smaller distance indicates a higher quality line segment L^n (greater likelihood of collinearity). At the same time, a threshold ε is used to select high-quality line segment candidates with a binding distance δ less than this threshold. Finally, a new set of endpoint-enhanced line segment candidates L^n = (x1, x2, y1, y2) is generated.

[0228] Since the points x1 and x2 on the line segment are considered "background" points, they will not be verified even after binding. Therefore, line of interest (LOI) pooling is introduced to verify the line segment candidate by connecting the entire line segment candidate with data evidence. The sampling function Ψt is used to map the background points to the points on the line segment as follows:

[0229] Ψt(X)=(1-t)·x1+t·x2

[0230] Where X is the mapping point corresponding to x1 and x2, and t is a value ∈ [0,1].

[0231] The model maintains three sets of sampling points: (1) the two endpoints {y1, y2}; (2) the midpoint Ψt(X) between x1 and x2; and (3) the midpoint Ψt(Y) between y1 and y2. By decoupling the endpoints and midpoints, the model gains geometric concepts when validating candidates, capturing the co-occurrence relationship between line segment candidates and node-guided inferred line segment candidates. After the above operations, the coordinates of the point L^n are updated.

[0232] Furthermore, the sketch conversion module may also be:

[0233] Lines and corners are extracted separately, and the corners are used to enhance the generation process.

[0234] Specifically, the extraction process first uses a stacked hourglass network as the backbone network to extract feature maps corresponding to line data represented as a line field L^n and endpoint candidates P^m. The predicted line segments and endpoints are bound together to form an initialized wireframe. Specifically, in L^n, the nearest endpoint candidates are found for the two endpoint candidates P^m of the line segment candidate, denoted as x1 and x2. Similarly, the nearest endpoint candidate x1 is found for endpoint candidate y1, and the nearest endpoint candidate x2 is found for endpoint candidate y2. The squared Euclidean distance between them is then calculated, denoted as δ1 and δ2. The maximum distance is defined as δ = max(δ1, δ2). A smaller distance indicates a higher quality line segment L^n (greater likelihood of collinearity). At the same time, a threshold ε is used to select high-quality line segment candidates with a binding distance δ less than this threshold. Finally, a new set of endpoint-enhanced line segment candidates L^n = (x1, x2, y1, y2) is generated.

[0235] Since the points x1 and x2 on the line segment are considered "background" points, they will not be verified even after binding. Therefore, line of interest (LOI) pooling is introduced to verify the line segment candidate by connecting the entire line segment candidate with data evidence. The sampling function Ψt is used to map the background points to the points on the line segment as follows:

[0236] Ψt(X)=(1-t)·x1+t·x2

[0237] Where X is the mapping point corresponding to x1 and x2, and t is a value ∈ [0,1].

[0238] The model maintains three sets of sampling points: (1) the two endpoints {y1, y2}; (2) the midpoint Ψt(X) between x1 and x2; and (3) the midpoint Ψt(Y) between y1 and y2. By decoupling the endpoints and midpoints, the model gains geometric concepts when validating candidates, capturing the co-occurrence relationship between line segment candidates and node-guided inferred line segment candidates. After the above operations, the coordinates of the point L^n are updated.

[0239] As shown in Figure 2, this is a schematic diagram of using this method to generate a three-dimensional model based on a sketch and text input, where a represents the input sketch, b represents the generated three-dimensional model, c represents the three-dimensional model after coloring and stylization, and d represents the text description.

[0240] As shown in Figure 3, a1 is a user-provided or generated sketch and text description, and a2-a4 are schematic diagrams of the generated 3D model from different angles. As shown in Figure 4, b1 is a user-provided or generated sketch, and b2-b4 are schematic diagrams of the generated 3D model from different angles. It can be clearly seen that the generated model results in b2-b4 have spikes and texture shift. Through the true measurement of the generated 3D model, the true measurement without text description is 0.2962, while the true measurement with text description is 0.3044. The true measurement is obtained by calculating the CLIP score and testing on the ShapeNet-Synthetic dataset. It can be clearly concluded that using edge sketches as intermediate representations and providing additional descriptions can improve the accuracy of user creation.

[0241] This method can generate a colorful / specific style three-dimensional model based on the user's text description and sketch, which can be better utilized in various fields such as games and design. This application converts the user input into a sketch that is easier to edit, and then converts the sketch into a three-dimensional model. The sketch is used as an intermediate intermediary between the three-dimensional model and other data modalities. The editability of the user modeling can be improved by introducing sketches, and the user can more easily confirm and modify his intentions to generate a three-dimensional model that better meets his intentions. Sketches can also reduce the performance consumption of the rendering of the interactive interface; in addition, since the computing resources required to convert modalities such as pictures, text, and voice into two-dimensional sketches are usually less than the computing resources required to convert into three-dimensional models, using sketches as an intermediary process can reduce the repeated generation of three-dimensional models that consumes a lot of computing power, which helps to improve the concurrent capacity of the system.

[0242] Example 2

[0243] As shown in FIG5 , this embodiment provides a sketch-based 3D model system, which includes:

[0244] The sketch generation module is a modal conversion neural network used to convert user input into a sketch; the sketch generation module includes an image generation network; the image generation network is a neural network with a VQ-VAE structure or an adversarial generation model or a diffusion model.

[0245] When the image generation network is a VQ-VAE neural network, the specific process of generating a sketch is as follows:

[0246] First, K1 vectors are set as a queryable encoding table. A randomly initialized image is input and passed through the encoder CNN to obtain N1 intermediate representations. Then, the nearest neighbor algorithm is used to query the encoding table for the vector most similar to these N1 intermediate representations. At the same time, an autoregressive model is trained to convert the input information into the representations in the encoding table. During prediction, the autoregressive model outputs the intermediate vector corresponding to the input and places the similar vector queried in the encoding table in the corresponding position. The decoder reconstructs the sketch using the obtained intermediate representations.

[0247] The autoregressive model may be a neural network with a Transformer structure.

[0248] When the image generation network is an adversarial generation model, the specific process of generating a sketch is:

[0249] First, K2 vectors are set as a queryable encoding table. A randomly initialized image is input and passed through the encoder CNN to obtain N2 intermediate representations. Then, the nearest neighbor algorithm is used to query the encoding table for the vector most similar to these N2 intermediate representations. At the same time, the BERT model or the CLIP model pre-trained on text-image data is used to compare the similarity between the generated image features and the specified text, image, or speech by calculating the cosine loss. The patch (block) of each image is judged and trained using a GAN loss. Finally, the intermediate vector corresponding to the input is output, and the N2 similar vectors queried in the encoding table are placed in the corresponding position. The decoder reconstructs the sketch using the obtained intermediate representation.

[0250] Among them, the encoder CNN decides based on the category. If the input is text, a text encoder is used; if the input is an image, an image encoder is used; if the input is speech, a speech encoder is used.

[0251] When the image generation network is a diffusion model, the specific process of generating a sketch is:

[0252] The diffusion model includes a forward diffusion model and a reverse denoising model;

[0253] The forward diffusion process is used to convert the image into random noise;

[0254] The reverse denoising process is used to restore the random noise image to a complete image;

[0255] Specifically, a text encoder, image encoder, or speech encoder is sequentially fed into a forward diffusion model and a backward denoising model. The image vector is then passed through an image decoder (VAE decoder) to generate a sketch. The forward diffusion model and the backward denoising model are pre-trained on a large number of images. The text encoder can be a Transformer-based CLIP text encoder model, which maps text into a vector space; the image encoder maps images of the desired style into a vector space; and the speech encoder maps speech signals into a vector space.

[0256] The user input may be a sketch directly, which may be drawn by the user on a touch screen, or a sketch drawn by the user on paper obtained by photographing or scanning, or a contour map obtained by a digital image processing method through contour extraction from a color or grayscale image. The specific digital image processing method may be contour extraction using the Canny operator or training a neural network model for image to contour map.

[0257] User input can also be a textual description of the user's desired sketch, such as "A LEGO-style airplane"; a human- or machine-generated voice describing the sketch; or an image of the desired style, such as a picture of a LEGO airplane. User input comes from a computer-generated description of the sketch, which can be generated from a fixed template, such as "a [object]." For example, if the user wants the sketch to describe the class of an airplane, the text description T is "an airplane." The class can be manually specified by the user or obtained by a neural network sketch classifier through a pre-training method. Specifically, the neural network sketch classifier can be composed of the following structure: a sketch encoder including a down-sampled convolutional neural network, a classifier including several fully connected neural networks (MLPs), and an activation function between each layer, such as a ReLU activation function; the sketch classifier is trained on categories with category identifiers and corresponding sketches. By specifying the sketch category, the network search domain for generating a three-dimensional model from the sketch can be reduced, thereby obtaining the desired result more quickly. As a preferred technical solution, the description can be designed according to the image rendered by the renderer. For example, if the objects rendered by the rendering module are all black outlines, the description of the sketch can be "a black [object]" (a black [category name to which the sketch belongs]); if the objects rendered by the rendering module are gray projection images, the description of the sketch can be "a grey [object]" (a gray [category name to which the sketch belongs]).

[0258] A sketch encoding module, used to encode the sketch into a vector; wherein the sketch encoding module is a neural network, which can be a convolutional neural network;

[0259] A decoding module is used to generate a three-dimensional model from the vector by generating the position of a triangular face or point cloud or a displacement based on a specific template. The decoding module is a neural network, a convolutional neural network, or a multi-layer perceptron (MLP). The specific template can be a general template, such as a spherical three-dimensional model, or a general template for the category to which the hand-drawn sketch belongs. For example, if the user wants the hand-drawn sketch to be a car, a 3D model of a car in the dataset can be used as the template. Alternatively, if the user wants the hand-drawn sketch to be a human body, the average shape of all human body data in the dataset can be used as the template.

[0260] A rendering module, for rendering the generated three-dimensional model into a contour image at a given perspective; wherein the given perspective is a perspective of the sketch; wherein the rendering module is a SoftRas renderer;

[0261] The supervision module is used to train the neural network to match the user input as closely as possible, where the user input can be a reference image, a text description, a voice description, a partially missing sketch, a semantically segmented image, or an interactive questionnaire. The specific implementation process of the supervision module is as follows:

[0262] The supervisory module is one or more neural networks, which are encoders in a multimodal model pre-trained on text-image pairs or speech-image pairs. One neural network maps the rendered outline image into a vector A, and another neural network maps the text, image, or audio based on the sketch into another vector B. The supervisory module maps the text and / or image and / or speech input into a shared feature space, calculates the similarity between vector A and vector B in this vector space, and trains the neural network to minimize the similarity so that the outline image rendered by the rendering module matches the text, image, or speech description as closely as possible. The text input is a text description based on the sketch, which can be entered by the user or automatically generated according to a template, such as "a[object]" ("a[name of the category to which the sketch belongs]"). For example, if the user wants the sketch to describe the class of an airplane, the text description T is "an airplane" (an airplane).

[0263] Among them, the text, image or audio description of the sketch can be designed according to the image rendered by the renderer. For example, if the objects rendered by the rendering module are all black outlines, the description of the sketch can be "ablack[object]" (a black [category name to which the sketch belongs]).

[0264] The multimodal model can be an existing CLIP (Contrastive Language-Image Pretraining) model or a GLIP (Grounded Language-Image Pretraining) model. Taking the CLIP model as an example, it is pre-trained using paired image-text pairs (for example, if the input is a picture of a dog, the corresponding text also indicates that it is a dog). These text and image are respectively output by the CLIP text encoder and the CLIP image encoder, and then comparative learning is performed on these output text features and image features. The learned text encoder and image encoder share the same feature space. The multimodal model can also be an ImageBind model, which can convert speech signals into vectors that share the same vector space with the image.

[0265] The system further includes a coloring module and / or a stylization module.

[0266] A coloring module is used for the user to color the 3D model based on the text, image or voice description of the 3D model. The coloring module specifically includes:

[0267] A coloring prompt encoding module is used to receive coloring prompts in text, image, or voice, and encode the coloring prompts into a vector A1 in a high-dimensional space;

[0268] The colored position encoding module encodes the generated 3D model into a colored neural radiation field;

[0269] A coloring renderer is used to render the colored three-dimensional model to obtain a colored rendering;

[0270] The color supervision module is a pre-trained neural network (encoder) aligned to the same latent space under multiple modalities. The color rendering image is encoded into a vector B1 in a high-dimensional space. Since the color supervision module and the color prompt encoding module are pre-trained to align to the same latent space under multiple modalities, the vector A1 and the vector B1 are in the same vector space. In this vector space, the similarity between the vector A1 and the vector B1 is calculated. The similarity is minimized through neural network training so that the color rendering image and the color prompt of the text, image or voice are matched as much as possible to achieve the desired model color distribution. The color neural radiation field includes a multi-layer perceptron.

[0271] The colored neural radiation field stores the color vector of each vertex position in the three-dimensional model, inputs the vertex position information, and outputs the color vector of the vertex position. The color vector can be described by a combination of [R, G, B] or a color combination of [H, S, V].

[0272] A stylization module is used to colorize a 3D model based on a stylized description of the 3D model in text, image, or voice. The stylization module specifically includes:

[0273] A stylized hint encoding module is used to receive stylized hints of text, image, or speech, and encode the stylized hints into a vector A2 in a high-dimensional space;

[0274] Stylized position encoding module, which encodes the generated 3D model into a stylized neural radiation field;

[0275] A stylized renderer is used to render the stylized 3D model to obtain a stylized rendering;

[0276] A stylized supervision module is a pre-trained neural network (encoder) aligned to the same latent space under multiple modalities. It encodes the stylized rendering into a vector B2 in a high-dimensional space. Since the coloring supervision module and the coloring hint encoding module are pre-trained to align to the same latent space under multiple modalities, the vector A2 and the vector B2 are in the same vector space. In this vector space, the similarity between vector A2 and vector B2 is calculated. The neural network is trained to minimize the similarity so that the stylized rendering and the stylized hints of the text, image, or voice are matched as closely as possible to achieve the desired model shape style. Specifically, the shape style may be "anime style", "abstract style", and other styles. The stylized neural radiation field includes a multi-layer perceptron. The stylized neural radiation field stores the displacement of each vertex position in the three-dimensional model, inputs vertex position information, and outputs the displacement of the vertex position relative to the normal.

[0277] As shown in Figure 6, in order to verify the effect of the supervision module, the present invention is used to compare the three-dimensional models generated with and without the supervision module. In the figure, e represents the input sketch, f represents the generated three-dimensional model, g represents the three-dimensional model generated without the supervision module, and h represents the three-dimensional model generated with the supervision module. It can be clearly seen from the figure that when the supervision model is set, the final generated model is closer to the real three-dimensional model.

[0278] Example 3

[0279] This embodiment also provides a parts processing method, the parts processing method comprising:

[0280] The first step is to use a modal conversion neural network to convert or extract the image to obtain a sketch;

[0281] In the second step, the sketch uses the above method to construct a three-dimensional model;

[0282] The third step is to input the generated 3D model into the parser and convert it into machine-readable instructions;

[0283] The fourth step is that the processing device performs processing according to machine-readable instructions.

[0284] In addition, the generated 3D model can be edited and modified before the third step.

[0285] The steps of obtaining a sketch from a photo by image extraction are as follows:

[0286] First, the photos were smoothed using a Gaussian filter to reduce noise and improve image quality;

[0287] Then, calculate the gradient and gradient direction of the image;

[0288] Next, the image is processed with non-maximum suppression (NMS) to eliminate non-edge pixels and retain only some thin lines as candidate strokes, which can better extract key information.

[0289] Finally, the edge detection process is completed by applying high and low thresholds and connecting the edges to obtain a processed sketch. Alternatively, the sketch can be generated by passing the photo through a pre-trained image-to-sketch neural network.

[0290] Optionally, an angle is selected or entered while sketching, or the angle predicted by the viewfinder is submitted to the user for confirmation.

[0291] Optionally, after the sketch is drawn, a pre-trained sketch-text retrieval algorithm is used to output a text prompt corresponding to the sketch for the user's reference.

[0292] Optionally, after drawing the sketch, normalize it to a fixed size and place it in the center of the image to facilitate the network to generate results.

[0293] Optionally, normalize the thickness of the lines after obtaining the sketch.

[0294] Optionally, after obtaining the sketch, it is converted into sequence information.

[0295] Example 4

[0296] This embodiment provides a parts processing device, which is used for the above-mentioned processing method, including a motion component, an actuator, and a controller; the controller receives user input and runs the three-dimensional model construction method in Example 1 to obtain a three-dimensional model. The combination of the motion component and the actuator can be a 3D printer or a CNC engraving machine.

[0297] Example 5

[0298] This embodiment provides an electronic terminal, which executes the three-dimensional model construction method in Example 1.

[0299] Example 6

[0300] This embodiment provides a post-editing method for generating a three-dimensional model based on a sketch, characterized in that the post-editing method includes:

[0301] (1) Automatically select the corresponding area of ​​the 3D model based on a region prompt algorithm given by the user, where the prompt is text, voice or picture;

[0302] (2) Editing the corresponding area of ​​the automatically selected three-dimensional model, wherein the editing includes overall stretching, deleting, and smoothing operations on the triangular faces or points in the area.

[0303] The specific process of step (1) is:

[0304] (11) receiving a user's text, image, or voice description of a region of the three-dimensional model, wherein the region description is in the form of a descriptive text of the three-dimensional model corresponding to the drawn sketch, a style map expected by the user, or a user-read or machine-generated voice description of the sketch;

[0305] (12) The region description is generated into a high-dimensional space vector A3 by aligning the pre-trained neural network in multiple modalities to the same latent space;

[0306] (13) storing each vertex of the generated three-dimensional model in a regional neural radiance field by performing a position encoding, and then assigning a probability p to each vertex position of the three-dimensional model; the regional neural radiance field is a multi-layer perceptron containing the probability p; wherein the probability p refers to the probability that the vertex belongs to the user-specified region;

[0307] (14) Add the probability p of each vertex position of the three-dimensional model to the generated three-dimensional model;

[0308] (15) Optimizing the regional neural radiation field with probability p;

[0309] (16) The optimized regional neural radiation field is converted back into a three-dimensional model, thereby obtaining a three-dimensional model of regional processing.

[0310] Use Fourier feature maps for position encoding, then pass the position encoding of each vertex to the regional neural radiation field, and store each vertex of the three-dimensional model in a regional neural radiation field;

[0311] The colors of the vertices of the three-dimensional model are pure colors or color mixtures, and the color mixing method is:

[0312] p*(High)+(1-p)*Grey;

[0313] Among them, High represents a color;

[0314] Grey represents another color.

[0315] Edit the selected area, which may include:

[0316] Use the anchor points on the 3D model surface to drag and move the entire part;

[0317] And / or, use the selected area on the 3D model surface to perform operations such as overall scaling, stretching, and translation.

[0318] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

Claims

1. A sketch-based 3D model construction method, It is characterized in that For processing a multi-stage sketch, the three-dimensional model construction method includes: Step 1: receiving user input in an interactive interface, wherein the user input is a reference image, a text description, a voice description, a partially missing sketch, a semantically segmented image, or an interactive questionnaire; The second step is to convert the user input into one or more sketches and present them on the interactive interface; In the third step, the user selects a suitable sketch in the interactive interface; The fourth step is to use the sketch to generate a three-dimensional model.

2. A method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that It also includes, the user editing the generated sketch in the interactive interface, and if the user is not satisfied with the generated sketch, returning to the first step to re-enter; The editing method is to use the artboard brush to edit, or use the text guide for sketch editing.

3. The method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that The method also includes coloring and / or stylizing the three-dimensional model by receiving a text, image or voice description provided by a user.

4. A method for constructing a three-dimensional model based on a sketch according to claim 3, It is characterized in that The 3D model building process displays the generated results in real time; and when the 3D model is colored and / or stylized, the user interface displays the progress in real time.

5. The method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that It also includes supervising the generated 3D model to make the 3D model match the user input as much as possible. The specific supervision process is as follows: Mapping the rendered contour of the 3D model into a vector A through a neural network, and mapping the text, image, or audio based on the sketch into another vector B through another neural network; calculating the similarity between vector A and vector B, and minimizing the similarity through neural network training so that the rendered contour and the text, image, or voice description match as closely as possible; The neural network is a pre-trained encoder that is aligned to the same latent space under multiple modalities.

6. The method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that In the first step, user input includes a description of the category and shape of the 3D model.

7. The method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that In the second step, the user input is transformed through a modal transfer neural network or image extraction to generate one or more sketches.

8. A method for constructing a three-dimensional model based on a sketch according to claim 1 or 2, It is characterized in that The specific steps to generate a 3D model using a sketch are: (5.1) Receive the sketch; (5.2) Extract features from the sketch through the encoder to obtain a compressed vector Z; (5.3) The decoder uses the vector Z to reconstruct or generate a three-dimensional model to obtain a three-dimensional model.

9. A method for constructing a three-dimensional model based on a sketch according to claim 1 or 8, It is characterized in that When a sketch generates a three-dimensional model, the user input is used as additional information input to optimize the generated three-dimensional model through a pre-trained neural network aligned to the same latent space in multiple modalities, and the multiple modalities at least include the modality corresponding to the user input information and the picture or sketch modality. Alternatively, an automatically generated description content is optimized by a pre-trained neural network aligned to the same latent space in multiple modalities as additional information input to generate a three-dimensional model, wherein the multiple modalities at least include a modality corresponding to the user input information and a picture or sketch modality. The automatically generated description is based on the user input or based on the generated sketch.

10. The method for constructing a three-dimensional model based on a sketch according to claim 9, It is characterized in that (5.1) also includes receiving a category description of the sketch, where the category of the sketch is manually specified by the user or obtained by a neural network sketch classifier through a pre-training method.

11. The method for constructing a three-dimensional model based on a sketch according to claim 9, It is characterized in that This class describes an image description rendered from a three-dimensional model.

12. The method for constructing a three-dimensional model based on a sketch according to claim 9, It is characterized in that In (5.3), the decoder generates a three-dimensional model based on a specific template, where the template is a general template or a three-dimensional model template designed or obtained for a certain category.

13. The method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that In the fourth step, the projection image or point cloud of the existing three-dimensional model in the database is used to retrieve the closest three-dimensional model in the database based on similarity.

14. The method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that The similarity and / or symmetry and / or surface smoothness between the 3D model and the sketch are optimized via a loss function.

15. The method for constructing a three-dimensional model based on a sketch according to claim 3, It is characterized in that Coloring a three-dimensional model is done by creating a new color value for each vertex of the three-dimensional model, then rendering the three-dimensional model with the color value, comparing the rendering result with the coloring description input by the user, optimizing the color value through the comparison result, and obtaining a three-dimensional model that meets the coloring process input by the user.

16. A method for constructing a three-dimensional model based on a sketch according to claim 15, It is characterized in that The specific steps for coloring the 3D model are: (6.11) receiving a textual, image or voice-based description of the three-dimensional model from a user, wherein the color-based description is in the form of a descriptive text of the three-dimensional model corresponding to the sketch, a style map of the user's desired style, or a voice description of the sketch read aloud by the user or generated by a machine; (6.12) The colored description is generated by aligning the pre-trained neural network in multiple modalities to the same latent space to generate a vector A1 in a high-dimensional space; (6.13) storing each vertex of the generated three-dimensional model in a colored neural radiation field by performing a position encoding, and then assigning a color vector to each vertex position of the three-dimensional model; the colored neural radiation field is a multi-layer perceptron containing color information; (6.14) Add the color vector of each vertex position of the three-dimensional model to the generated three-dimensional model; (6.15) Optimize the colored neural radiation field with color vector; (6.16) The optimized colored neural radiation field is converted back into a three-dimensional model, thereby obtaining a colored three-dimensional model.

17. A method for constructing a three-dimensional model based on a sketch according to claim 16, It is characterized in that In step (6.13), Fourier feature maps are used for position encoding, and then the position encoding of each vertex is passed to the colored neural radiation field, and each vertex of the three-dimensional model is stored in a colored neural radiation field.

18. The method for constructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that The generated 3D model is rendered in a uniformly sampled interval, and the similarity with the user input is calculated. The perspective with the highest similarity is selected as the anchor perspective. Then, a Gaussian distribution centered on the anchor perspective is used to render multiple perspectives from randomly sampled views, and the 3D model is optimized through multiple perspectives.

19. The method for constructing a three-dimensional model based on a sketch according to claim 3, It is characterized in that The three-dimensional model is stylized by creating a displacement value for each vertex of the three-dimensional model, then rendering the three-dimensional model with the displacement value, comparing the rendering result with the stylized description input by the user, optimizing the displacement value through the comparison result, and obtaining a stylized three-dimensional model that conforms to the user input.

20. The method for constructing a three-dimensional model based on a sketch according to claim 19, It is characterized in that The specific steps for stylizing a 3D model are: (6.21) receiving a user's text, image, or voice stylized description of the three-dimensional model, wherein the stylized description is in the form of a descriptive text of the three-dimensional model corresponding to the drawn sketch, a picture of the style expected by the user, or a voice description of the sketch read aloud by the user or generated by a machine; (6.22) The stylized description is generated into a vector A2 in a high-dimensional space by aligning the pre-trained neural network in multiple modalities to the same latent space; (6.23) storing each vertex of the three-dimensional model generated by the sketch in a stylized neural radiance field by performing a position encoding, and then assigning a displacement vector relative to the normal to each vertex position of the three-dimensional model; the stylized neural radiance field is a multi-layer perceptron that contains displacement information; (6.24) Add the normal displacement of each vertex position of the three-dimensional model to the generated three-dimensional model; (6.25) Optimizing stylized neural radiation fields with displacement vectors; (6.26) The optimized stylized neural radiation field is converted back into a three-dimensional model, thereby obtaining a stylized three-dimensional model.

21. The method for constructing a three-dimensional model based on a sketch according to claim 3, It is characterized in that The construction method also includes: Editing specific parts or areas of the colored and / or stylized 3D model, the editing may include: Use anchor points on the 3D model surface to drag and move the entire part; And / or, use the selected area on the surface of the 3D model to perform operations such as overall scaling, stretching, and translation.

22. A sketch-based 3D model system, It is characterized in that The system is applicable to the three-dimensional model construction method according to any one of claims 1 to 21, and the system comprises: The sketch generation module is a modality transfer neural network that converts user input into sketches; Sketch encoding module, used to encode the sketch into a vector; A decoding module, used for generating a three-dimensional model by converting the vector into a position of a triangular face or a point cloud or a displacement based on a specific template; A rendering module, used for rendering the generated three-dimensional model into a contour image of a given viewing angle; A supervision module is used to train a neural network to make the contour map match the user input as closely as possible; The user input is a reference image, a text description, a voice description, a partially missing sketch, a semantically segmented image, or an interactive questionnaire.

23. A sketch-based 3D model system according to claim 22, It is characterized in that The system further comprises a coloring module, which is used for the user to color the three-dimensional model according to the coloring description of the text, image or voice of the three-dimensional model; And / or, a stylization module is used to colorize the three-dimensional model based on the user's stylized description of the three-dimensional model in text, image or voice.

24. A sketch-based 3D model system according to claim 22, It is characterized in that The coloring module specifically includes: A coloring prompt encoding module, used for receiving a coloring prompt of text, image or voice, and encoding the coloring prompt into a vector A1 in a high-dimensional space; A colored position encoding module encodes the generated 3D model into a colored neural radiation field; A coloring renderer is used to render the colored three-dimensional model to obtain a colored rendering; A coloring supervision module, which is a pre-trained neural network aligned to the same latent space under multiple modalities, encodes the coloring rendering into a vector B1 in a high-dimensional space, calculates the similarity between vector A1 and vector B1, and minimizes the similarity through neural network training, so that the coloring rendering and the coloring prompt of the text, image or voice are matched as much as possible to achieve the desired model color distribution; Among them, the colored neural radiation field includes a multi-layer perceptron, and the colored neural radiation field stores the color vector of each vertex position in the three-dimensional model, inputs the vertex position information, and outputs the color vector of the vertex position.

25. The sketch-based 3D model system according to claim 22, It is characterized in that The stylization module specifically includes: A stylized hint encoding module, used for receiving stylized hints of text, image or speech, and encoding the stylized hints into a vector A2 in a high-dimensional space; The stylized position encoding module encodes the generated 3D model into a stylized neural radiation field; A stylized renderer is used to render the stylized 3D model to obtain a stylized rendering; A stylized supervision module, which is a pre-trained neural network aligned to the same latent space under multiple modalities, encodes the stylized rendering into a vector B2 in a high-dimensional space, calculates the similarity between vector A2 and vector B2, and trains the neural network to minimize the similarity so that the stylized rendering matches the stylized prompt of the text, image, or speech as closely as possible; The stylized neural radiation field includes a multi-layer perceptron, and the stylized neural radiation field stores the displacement vector of each vertex position in the three-dimensional model, inputs the vertex position information, and outputs the displacement of the vertex position relative to the normal.

26. A sketch-based 3D model system according to claim 22, It is characterized in that The specific implementation process of the supervision module is as follows: The supervision module is one or more neural networks, which are encoders in a multimodal model that have been pre-trained for text-image pairs or speech-image pairs, wherein one neural network maps the rendered contour image into a vector A, and another neural network maps the sketch-based text or image or audio into another vector B; the supervision module maps the text and / or picture and / or speech input to a shared feature space, in which the similarity between vector A and vector B is calculated by training the neural network to minimize the similarity, so that the contour image rendered by the rendering module matches the text or image or speech description as much as possible.

27. A parts processing method, It is characterized in that The parts processing method comprises: The first step is to use a modal conversion neural network to convert or extract the sketch using an image. In the second step, the sketch constructs a three-dimensional model using the method described in any one of claims 1 to 21; The third step is to input the generated 3D model into the parser and convert it into machine-readable instructions; The fourth step is that the processing device performs processing according to the machine-readable instructions.

28. A parts processing method according to claim 27, It is characterized in that in, The specific steps of obtaining a sketch through image extraction are: First, the photos were smoothed using a Gaussian filter to reduce noise and improve image quality; Then, calculate the gradient and gradient direction of the image; Next, the image is processed with non-maximum suppression to eliminate non-edge pixels and retain only some thin lines as candidate strokes; Finally, the edge detection process is completed by applying high and low thresholds and connecting the edges to obtain a processed generated sketch.

29. A parts processing method according to claim 27, It is characterized in that After obtaining the sketch, it is normalized to a fixed size and placed at the center of the image.

30. A parts processing device, It is characterized in that The device is used for the processing method described in any one of claims 27-29, and includes a motion component, an actuator, and a controller; the controller receives user input and runs the three-dimensional model construction method described in any one of claims 1-21 to obtain a three-dimensional model.

31. An electronic terminal, It is characterized in that The electronic terminal executes the three-dimensional model building method described in any one of claims 1-21.

32. A post-editing method for generating a three-dimensional model based on a sketch. It is characterized in that The post-editing method comprises: Selecting a corresponding area of ​​the three-dimensional model according to an area prompt given by a user, wherein the prompt is text, voice or picture; (2) Editing the corresponding area of ​​the selected three-dimensional model, wherein the editing includes overall stretching, deleting, and smoothing operations on the triangular faces or points in the area. The specific process of step (1) is as follows: (11) receiving a text, image, or voice description of a region of the three-dimensional model from a user, wherein the region description is in the form of a descriptive text of the three-dimensional model corresponding to the drawn sketch, a style map expected by the user, or a voice description of the sketch read aloud by the user or generated by a machine; (12) The region description is generated into a vector A3 in a high-dimensional space by aligning the pre-trained neural network in multiple modalities to the same latent space; (13) storing each vertex of the generated three-dimensional model in a regional neural radiation field by performing a position encoding, and then assigning a probability p to each vertex position of the three-dimensional model; the regional neural radiation field is a multi-layer perceptron containing the probability p; wherein the probability p refers to the probability that the vertex belongs to the region; (14) adding the probability p of each vertex position of the three-dimensional model to the generated three-dimensional model; (15) Optimizing the regional neural radiation field with probability p; (16) The optimized regional neural radiation field is converted back into a three-dimensional model, thereby obtaining a three-dimensional model of regional processing. Use Fourier feature map to perform position encoding, then pass the position encoding of each vertex to the regional neural radiation field, and store each vertex of the three-dimensional model in a regional neural radiation field; The color of the vertices of the three-dimensional model is a pure color or a color mixture, and the color mixing method is: p*(High)+(1-p)*Grey; Among them, High represents a color; Grey indicates another color; Edit the selected area, which may include: Use anchor points on the 3D model surface to drag and move the entire part; And / or, use the selected area on the surface of the 3D model to perform operations such as overall scaling, stretching, and translation.

Citation Information

Patent Citations

  • Animation data organization method based on sketch interaction

    CN108305306A

  • Method and system for 3D modeling based on irregular-shaped sketch

    KR102410047B1

  • Sketch selection for rendering 3D model avatar

    US20160328886A1

  • Sketch Completion Using Machine Learning

    US20190370617A1

Cited By

  • Three-dimensional model editing method and device based on multiple modes, equipment and medium

    CN119540504A

  • Three-dimensional model editing method and device, storage medium and program product

    CN119672276A

  • Complex scene image generation system and method based on freehand sketch

    CN119887972A

  • Industrial model generation method and industrial model generation device

    CN119939687A

  • Zero-sample generalization three-dimensional object reconstruction method based on semantic guidance

    CN120047615A