User interaction method for high-resolution sketch generation model, and processing method and apparatus

Abstract

A user interaction method for a high-resolution sketch generation model. The method comprises the following steps: step one, an interactive interface receiving a user input, wherein the user input can be a sketch S drawn by a user on a touch-control screen; step two, software converting the sketch drawn by the user into a rough three-dimensional model and presenting same on a user interface, the user confirming a presented effect, and if the presented effect does not satisfy the requirements of the user, returning to step one; step three, on the basis of the rough three-dimensional model, an algorithm performing post-processing optimization on the model, a post-processing optimization process comprising refining or optimizing model details, or refining or optimizing or generating a model texture; step four, presenting the optimized model on the user interface, wherein step three and step four can be executed multiple times until the user is satisfied. By means of three-dimensional model display in an intermediate form, the present invention enables a user to view a rough modeling result more quickly, reduces the consumption of computing resources, and also facilitates the improvement of the user experience.

Description

A user interaction method, processing method and device for high-resolution sketch generation model Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to an interactive three-dimensional model generation method. Background Art

[0002] The rapid development of portable displays and AR / VR has created a huge demand for 3D content. Computer-aided design (CAD) methods require users to master complex CAD software commands and be able to parse shapes into sequential commands, limiting their adoption among professional users. These limitations drive the need for alternative approaches that can open the door to 3D modeling for the masses. Sketch-based 3D modeling offers a potential solution, as sketching is one of the most natural ways for us humans to express ideas. Most existing work either requires precise line drawings from multiple views or applies step-by-step workflows that require strategic knowledge, making it difficult for novice users. Other work has proposed retrieval-based approaches from existing models, but these methods lack customizability. Other existing methods for reconstructing 3D models from a single sketch use an encoder-decoder architecture to compress the input sketch into a coarse representation (latent code) that captures information such as semantic categories and conceptual shape. A decoder then recovers the 3D shape by calculating offsets for a given number of points or vertices. However, due to the significant domain gap between the sketch and 3D shape domains and the limited resolution of point or vertex representations, these methods struggle to capture intricate details.

[0003] Summary of the Invention

[0004] The present invention aims to provide a method for producing a complete three-dimensional model with only a single hand-drawn drawing, which is not limited by resolution. The present invention also proposes to realize user-defined three-dimensional model editing through post-processing iterative editing.

[0005] To this end, the present invention adopts the following technical solutions:

[0006] A user interaction method for generating a high-resolution sketch model comprises the following steps:

[0007] In the first step, the interactive interface receives user input, which can be a sketch S drawn by the user on the touch screen.

[0008] In the second step, the software converts the sketch drawn by the user into a rough 3D model and presents it on the user interface. The user confirms the presented effect. If it does not meet the user's needs, the software returns to the first step.

[0009] In the third step, based on the rough three-dimensional model, the algorithm performs post-processing optimization on the model. The post-processing optimization process includes refining or optimizing model details, or refining, optimizing or generating model textures.

[0010] The fourth step is to present the optimized model on the user interface. Steps 3 and 4 can be performed multiple times until the user is satisfied.

[0011] Through the intermediate form of 3D model display, users can see the rough modeling results more quickly, save computing resources, and help improve user experience.

[0012] Preferably, in the first step, the user input may be a 3D sketch drawn in a three-dimensional space using an AR / VR device.

[0013] Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning,

[0014] Alternatively, the user input may be a contour map obtained by a digital image processing method through contour extraction from a color or grayscale image. The digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for image to contour map.

[0015] Preferably, in the second step, specifically, the rough three-dimensional model can be a sparse point cloud, an occupancy function that only describes the shell surface, a three-dimensional model represented by a rough triangular patch, a three-dimensional model represented by voxels, a three-dimensional model represented by a neural radiation field, or a representation of T-SDF (truncated signed distance field), SDF (signed distance field), or UDF (unsigned distance field).

[0016] Preferably, in the third step, the model post-processing optimization method may be:

[0017] 3.1 Convert the input into a deformable tetrahedral mesh signed distance field (SDF), where each vertex of the tetrahedron is an SDF value.

[0018] 3.2 Subdivide the deformable tetrahedron mesh obtained in the first step by adding new points at the midpoint of each edge to increase the resolution. The SDF value of the new vertex is obtained by averaging the SDF values ​​on the edge. This step can be performed multiple times to obtain the desired resolution.

[0019] 3.3 Convert the encoded SDF into an explicit triangle mesh using the Marching Tetrahedra algorithm;

[0020] Alternatively, the post-processing optimization method may be to use a Loop Subdivision method to subdivide the model input in a triangular face format, and this step may be performed multiple times to obtain an ideal resolution;

[0021] Alternatively, the post-processing optimization method may be to perform broken surface repair in step 3.5.

[0022] Alternatively, the post-processing optimization method can be,

[0023] 3.6 Optimize using a neural network that first encodes the input into 3D features;

[0024] 3.7 Use a fully connected network MLP to predict the SDF value of each vertex in the initial deformable tetrahedral mesh. The fully connected network also outputs a feature vector f for surface refinement in the volume subdivision stage;

[0025] 3.8 Iteratively refine the surface and subdivide the tetrahedral mesh; specifically, we first determine the surface tetrahedron Tsurf based on the current SDF value, construct a graph corresponding to the vertices and edges in the tetrahedron SDF representation, and use a graph convolutional network to predict the position offset Δvi and the SDF difference Δs(vi) for each vertex i. After the surface is refined, a volume subdivision step is performed. In this process, an optimizer is used to minimize the L2 Chamfer distance and the normal vector consistency loss. The optimizer can be an Adam optimizer.

[0026] Alternatively, the post-processing optimization method may be that the surface refinement step specifically includes:

[0027] 3.9 Build a new graph on the extracted mesh and use GCN to predict the updated position of each vertex and the parameter alpha for Loop Subdivision;

[0028] 3.10 Using a pre-trained discriminator, first randomly select a high curvature vertex v in the target mesh and calculate the true signed distance field S1 in a voxelized area near v; similarly, calculate the signed distance field of the predicted surface mesh M at the same position to obtain S2, and feed S1 or S2 together with the feature vector in position v into the discriminator for judgment; the discriminator can be a discriminator of DECOR-GAN, and the judgment loss function can be the adversarial loss of LSGAN.

[0029] Alternatively, the post-processing optimization method can be,

[0030] 3.11 Preprocess the input, which can be any step from step 3.1 to step 3.10;

[0031] 3.12 Use a U-Net-style Diffusion network to optimize the input, initialize the input with Gaussian noise, and denoise it using the DDPM sampling strategy.

[0032] Alternatively, the post-processing optimization method can be,

[0033] 3.13 Convert the input to SDF and directly use the color field optimized from the rough stage to initialize the volume texture field;

[0034] 3.14 We use a differentiable renderer to render the extracted surface mesh into a high-resolution image;

[0035] 3.15 Optimize the SDF value of each vertex using high-resolution SDS gradients through backpropagation;

[0036] 3.16 When rendering a 3D model represented by a mesh into an image, we also keep track of the 3D coordinates of each corresponding pixel projection, which will be used to query the color in the corresponding texture field for joint optimization;

[0037] 3.17 As an optional technical solution, when rendering the mesh, increase the focal length to magnify object details, which is a key step in recovering high-frequency details; retain the pre-trained environment map from the coarse optimization stage, and use differentiable anti-aliasing methods to composite the rendered background with the foreground objects;

[0038] 3.18 To encourage surface smoothness, a normative term is used to normalize the angular differences between adjacent faces on the mesh;

[0039] In addition, the present invention also provides a method for converting a sketch into a three-dimensional model, including beautifying the sketch, comprising the following steps:

[0040] In the first step, the interactive interface receives user input, which can be a sketch S drawn by the user on the touch screen.

[0041] The user input may be a 3D sketch drawn in a three-dimensional space using an AR / VR device,

[0042] Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning,

[0043] Alternatively, the user input may be a contour map obtained by performing contour extraction on a color or grayscale image using a digital image processing method, wherein the digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for converting an image to a contour map;

[0044] Step 2: The algorithm beautifies the sketch drawn by the user in real time and presents the beautified sketch on the interactive interface. The beautification can also be performed when the user completes the sketch. Specifically, the beautification may include the following steps:

[0045] 2.1 Extract the area where the strokes are located,

[0046] 2.2 The strokes are synthesized into corresponding partial references through retrieval and interpolation,

[0047] 2.3 Use bounding boxes of different regions to optimize the complete beautified sketch;

[0048] In the third step, the software converts the sketch drawn by the user into a rough 3D model and presents it on the user interface. The user confirms the presented effect. If it does not meet the user's needs, the software returns to the first step.

[0049] Specifically, the rough three-dimensional model can be a sparse point cloud, an occupancy function that only describes the shell surface, a three-dimensional model represented by a rough triangular patch, a three-dimensional model represented by voxels, a three-dimensional model represented by a neural radiation field, or a representation of a T-SDF (truncated signed distance field), an SDF (signed distance field), or an UDF (unsigned distance field);

[0050] In the fourth step, based on the rough three-dimensional model, the algorithm performs post-processing optimization on the model, wherein the post-processing optimization process includes refining or optimizing model details, or refining, optimizing or generating model textures;

[0051] The fifth step is to present the optimized model on the user interface. Steps 3 and 4 can be performed multiple times until the user is satisfied.

[0052] In addition, the present invention also provides a user interaction method for continuously creating and editing a three-dimensional model based on a sketch, comprising the following steps:

[0053] In the first step, the interactive interface receives user input, which can be a sketch S drawn by the user on the touch screen.

[0054] Alternatively, the user input may be a 3D sketch drawn in three-dimensional space using an AR / VR device.

[0055] Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning,

[0056] Alternatively, the user input may be a contour map obtained by performing contour extraction on a color or grayscale image using a digital image processing method, wherein the digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for converting an image to a contour map;

[0057] As an optional input, the user can enter the viewpoint corresponding to the sketch, such as normal view, oblique view, etc.

[0058] In the second step, the software converts the user's sketch into a 3D model, which the user can view on the interface. The conversion method can be the aforementioned method of generating a model from a high-resolution sketch, or other methods of converting sketches to 3D models, such as a search-based method that searches the sketch against contour images with corresponding 3D models in a database and selects the closest 3D model.

[0059] In the third step, the user rotates the 3D model generated in the third step to an angle V on the interface for editing. If the user does not rotate, the default angle V is the viewpoint of the sketch when the user draws it;

[0060] Step 4: Use a renderer to render the model at angle V to obtain a rendering image R, and then use an edge extraction algorithm to extract the edges of the rendering image into a sketch S2. The edge extraction algorithm can be a Canny operator or a neural network model trained for image-to-contour mapping. The sketch is displayed on the user interface;

[0061] In the fifth step, the user edits the sketch S2 in the fourth step on the user interface to obtain a sketch S3. The editing includes adding some elements, adjusting the positions of some elements, or scaling some elements.

[0062] Alternatively, the user adds a sketch to the rendering R in the fourth step on the user interface to obtain sketch editing information S3. The sketch can be one or more strokes, a closed new structure or a structure that needs to be subtracted, or a complete redrawn sketch.

[0063] The sixth step is an optional step, in which the user adds a new description information on the user interface. The description information can be input in the form of text, voice, gesture, etc.

[0064] Step 7: Based on the sketch S3 and the 3D model of Step 2 and the optional description information of Step 6 as input, the 3D model is edited using a model post-processing iterative editing method to obtain a new 3D model.

[0065] Alternatively, based on the sketch S3 and the 3D model in the second step as input, the 3D model is edited using a model post-processing iterative editing method to obtain a new 3D model;

[0066] In the eighth step, the user views the new three-dimensional model obtained in the seventh step in the interactive interface. If not satisfied, the user can return to the first or third step and start again.

[0067] Preferably, the model iterative editing method in step 7 may be:

[0068] Image-based optimization:

[0069] 7.1 Using Sketch to 3D Model: Use a pre-trained image-to-image translation network to convert sketch S3 into a projection of a 3D model. The projection of the 3D model is an image where each pixel has a value of 0 or 1. When the point can be projected onto the surface, it is assigned a value of 1, and otherwise it is 0.

[0070] 7.2 Use a differentiable renderer to render the sketch in step 6 into a projection of a three-dimensional model. The projection of the three-dimensional model is an image where each pixel has a value of 0 or 1. When the point can be projected onto the surface, the value is assigned to 1, and otherwise it is assigned to 0.

[0071] 7.3 Compare the projections in 7.1 with those in 7.2. The optimizer changes the 3D model to minimize the error between the two projections.

[0072] Alternatively, the iterative editing method for the model in step 7 can be:

[0073] Optimization based on SDF value

[0074] 7.4 Take the surface points of the outer contour area of ​​the model and use a differentiable renderer;

[0075] 7.5 Obtain surface points of the sketch. The surface points may be obtained by emitting rays from the four boundaries of the image and retaining only the first black pixel hit by the ray;

[0076] 7.6 Take the images from 7.4 and 7.5 and optimize the model using the bidirectional 2D Chamfer loss. The optimization can be performed on the model generated in step 2 or on intermediate variables generated during the generation process in step 2, such as the model's latent code.

[0077] 7.7 As a preferred step (optional step), the optimization process is normalized to obtain normal values, and during the optimization process, the surface portion projected near the sketch should match the sketch, while the other portions should maintain their original normals and boundaries;

[0078] Alternatively, the iterative editing method for the model in step 7 can be:

[0079] NeRF field-based optimization

[0080] 7.8 uses an iterative algorithm. In each iteration, a view is randomly sampled and the 3D object is rendered from that view. The new density mass added to the model under the original NeRF representation should occupy the area specified by the sketch. We do this by rendering the object mask for all scribble views. Then, the object mask value of the scribble area is maximized by minimizing the following loss:

[0081] Use rendered images to compute SDS loss to semantically align textual cues;

[0082] 7.9 In addition, we compute the distance between a point and the sketch to preserve the NeRF before the basic edit by constraining the density and color output of Fe to be similar to Fo outside the scribble area. Finally, we define a loss function using the object mask rendering from the sketch's perspective; this loss ensures that the object mask occupies the edited area.

[0083] Alternatively, the iterative editing method for the model in step 7 can be:

[0084] Optimization based on SPAGHETTI representation

[0085] 7.10 Use an image encoder to convert the input sketch into a visual embedding;

[0086] 7.11 uses a Transformer decoder to map a set of learned partial queries and the visual embeddings obtained in 7.10 into the multi-part latent space of SPAGHETTI. The latent representation received as input by SPAGHETTI is mapped to a set of m-dimensional vectors representing a Gaussian mixture model (GMM), i.e., each vector corresponds to a 3D Gaussian distribution;

[0087] 7.12 The user selects an unsatisfactory part from the output shape; the selected part has a corresponding latent vector in the GMM latent space, and a refinement network outputs a set of vectors that replace the selected vectors under the adjustment of the latent vectors of the non-selected parts; finally, the SPAGHETTI shape generator uses the new vectors to generate the improved shape again; the refinement network can be a bidirectional Transformer encoder network, which receives a set of latent vectors so that the corresponding selected part vectors are masked (i.e., set to zero); it outputs a refined vector corresponding to the selected part; as a preferred technical solution, 5%-40% of the input vectors are masked, and the network needs to rely on the unmasked vector parts to output the complete shape.

[0088] The present invention also provides an immersive user interaction method for converting a sketch into a three-dimensional model, comprising the following steps:

[0089] The first step is to collect environmental information and present the environmental information on the interactive interface. The environmental information can be RGB panoramic information;

[0090] Alternatively, the environmental information is a normal map of a three-dimensional reconstruction of the scene or a preview of the scene after three-dimensional reconstruction. The three-dimensional reconstruction of the scene can be obtained by multi-view reconstruction or other deep learning methods, or can be directly read by a laser radar, ultrasonic sensor, etc.;

[0091] In the second step, the software receives user sketch input, and the user draws a single-view sketch on the layer of environment information on the interactive interface;

[0092] As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network;

[0093] In the third step, the software converts the sketch drawn by the user into a 3D model. The conversion method can be the aforementioned method of generating a model from a high-resolution sketch, or other methods of converting sketches into 3D models.

[0094] In the fourth step, the three-dimensional model obtained in the third step and the environmental information in the first step are presented together on the interactive interface. The presentation position is the position and angle of the user's hand-drawn sketch, and the user can view the obtained three-dimensional model on the interface.

[0095] Step 5: Based on the visualization results of step 4, edit the model obtained in step 3. The editing is based on the geometric relationship between the 3D reconstruction of the scene and the model obtained in step 3. The user can adjust parameters including position and size, as well as components of the model.

[0096] The sixth step is an optional step. During the adjustment process, if there is positional interference between the generated model and the result of the existing 3D reconstruction of the environment, a prompt will be given on the interactive interface.

[0097] The present invention also provides a user interaction method for converting a sketch into a three-dimensional model for immersive object editing, comprising the following steps:

[0098] The first step is to collect information about objects in the environment using a collection device and present it on an interactive interface. This information can be a normal map of the object's 3D reconstruction or a preview of the reconstructed object model, such as a mesh rendering or a textured 3D object rendering. The object's 3D reconstruction can be obtained using multi-view 3D reconstruction or other deep learning methods, or directly read using a lidar or ultrasonic sensor. The collection device can be an electronic terminal equipped with an image sensor, depth sensor, or spectral sensor, such as an AR / VR device, mobile phone, or tablet.

[0099] In the second step, the software receives the user's sketch input, and the user draws a single-view sketch on the object information layer on the interactive interface;

[0100] As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network;

[0101] The sketch may be one or more strokes, or a closed new structure or a structure to be subtracted, or a complete sketch drawn from scratch;

[0102] As a preferred technical solution, during the drawing process, when the strokes come into contact with the physical 3D model, the contact points are marked and presented to the user;

[0103] In the third step, the software uses the sketch drawn by the user in the second step and the 3D model of the object in the first step as input and converts it into a complete 3D model of the object after modification based on the original object;

[0104] Step 4: Display the 3D model obtained in step 3 on the interactive interface. The display position and angle are the same as the user's hand-drawn sketch. The user can view the obtained 3D model on the interface.

[0105] Preferably, the user can select the physical 3D model to be overlaid and displayed on the 3D model after the sketch editing;

[0106] Step 5: Based on the visualization results of step 4, edit the model obtained in step 3. The editing is based on the geometric relationship between the 3D reconstruction of the scene and the model obtained in step 3. The user can adjust parameters including position and size, as well as components of the model.

[0107] The sixth step is optional. During the adjustment process, if there is any positional interference between the generated model and the existing 3D reconstruction results of the environment, a prompt will be given on the interactive interface;

[0108] The above steps can be repeated until the user is satisfied with the final model.

[0109] The present invention also provides a user interactive method for converting a sketch into a three-dimensional model after self-constructing an object and editing it, which is characterized by comprising the following steps:

[0110] In the first step, the user manually constructs a three-dimensional model of the real physical world, and the three-dimensional model of the real physical world is obtained through sensors;

[0111] Manually construct a three-dimensional object in a physical space by stacking building blocks with circuits through addition and subtraction, and the three-dimensional object can also be directly obtained and displayed in a digital space. The building blocks can be block structures that can mesh or magnetically attract each other, or basic geometric bodies containing meshing structures or magnetically attracting structures, and each building block has a circuit structure.

[0112] Specifically, the circuit structures can communicate with each other, transfer identity information, and transmit it to a main controller. The main controller determines the relative positions of the building blocks in the physical space through the identity identification codes of different building blocks and the topological structure obtained by mutual communication, and presents it in the virtual three-dimensional space;

[0113] Alternatively, the circuit structure is a passive device composed of resistors and / or capacitors and / or inductors. The main controller (base) has a power supply. The main controller (base) reads the changes in resistance, capacitance, or inductance in real time, obtains the relative position of the building block in the physical space through the changes in resistance and / or capacitance at time t and t+1, and presents it in the virtual three-dimensional space.

[0114] The second step is to present the item information on the interactive interface. The item information is a normal map of the 3D reconstruction of the item or a preview of the 3D reconstructed item model, such as a mesh rendering or a textured 3D object rendering. The 3D object reconstruction can be obtained by multi-view 3D reconstruction or other deep learning methods, or can be directly read by a lidar, ultrasonic sensor, etc. The acquisition device can be an electronic terminal with an image sensor, depth sensor, or spectral sensor, such as an AR / VR device, mobile phone, or tablet computer.

[0115] In the third step, the software receives the user's sketch input, and the user draws a single-view sketch on the object information layer on the interactive interface;

[0116] As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network;

[0117] The sketch may be one or more strokes, or a closed new structure or a structure to be subtracted, or a complete sketch drawn from scratch;

[0118] As a preferred technical solution, during the drawing process, when the strokes come into contact with the physical 3D model, the contact points are marked and presented to the user;

[0119] In the fourth step, the software uses the sketch drawn by the user in the second step and the 3D model of the object in the first step as input and converts it into a complete 3D model of the object after modification based on the original object;

[0120] Step 5: Present the 3D model obtained in step 3 on the interactive interface. The presentation position and angle are the same as the user's hand-drawn sketch, and the user can view the obtained 3D model on the interface.

[0121] Preferably, the user can select the physical 3D model to be overlaid and displayed on the 3D model after the sketch editing;

[0122] Step 6: Based on the visualization results of step 4, edit the model obtained in step 3. The editing is based on the geometric relationship between the 3D reconstruction results of the scene and the model obtained in step 3. The user can adjust parameters including position and size, as well as components of the model.

[0123] The seventh step is optional. During the adjustment process, if there is any positional interference between the generated model and the existing 3D reconstruction results of the environment, a prompt will be given on the interactive interface.

[0124] The above steps can be repeated until the user is satisfied with the final model.

[0125] On this basis, the present invention also provides a workpiece processing method, comprising the following steps:

[0126] The first step is to obtain a customized 3D model using the above method;

[0127] The second step is to input the 3D model into the parser and convert it into machine-readable instructions;

[0128] In the third step, the processing device performs processing according to machine-readable instructions.

[0129] On this basis, the present invention also provides a workpiece processing device, which is characterized in that it is configured to process the workpiece according to the above method.

[0130] On this basis, the present invention further provides an electronic terminal, comprising an input component, an output component and a processing component, configured to be used in the above method.

[0131] The beneficial effects of the present invention are: the present invention can display the three-dimensional model in an intermediate form, allowing users to see the rough modeling results more quickly, saving computing resources and helping to improve user experience. DETAILED DESCRIPTION

[0132] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0133] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0134] Example 1

[0135] A user-interactive approach to high-resolution sketch generation models:

[0136] In the first step, the interactive interface receives user input, which can be a sketch S drawn by the user on the touch screen.

[0137] Alternatively, the user input may be a 3D sketch drawn in three-dimensional space using an AR / VR device.

[0138] Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning,

[0139] Alternatively, the user input may be a contour map obtained by performing contour extraction on a color or grayscale image using a digital image processing method, wherein the digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for converting an image to a contour map;

[0140] In the second step, the software converts the sketch drawn by the user into a rough 3D model and presents it on the user interface. The user confirms the presented effect. If it does not meet the user's needs, the software returns to the first step.

[0141] Specifically, the rough three-dimensional model can be a sparse point cloud, an occupancy function that only describes the shell surface, a three-dimensional model represented by a rough triangular patch, a three-dimensional model represented by voxels, a three-dimensional model represented by a neural radiation field, or a representation of a T-SDF (truncated signed distance field), an SDF (signed distance field), or an UDF (unsigned distance field);

[0142] In the third step, based on the rough three-dimensional model, the algorithm performs post-processing optimization on the model. The post-processing optimization process includes refining or optimizing model details, or refining, optimizing or generating model textures.

[0143] The fourth step is to present the optimized model on the user interface. Steps 3 and 4 can be performed multiple times until the user is satisfied.

[0144] Through the intermediate form of 3D model display, users can see the rough modeling results more quickly, save computing resources, and help improve user experience.

[0145] Specifically, in the third step, the model post-processing optimization method may be:

[0146] 3.1 Convert the input into a signed distance field (SDF) of a deformable tetrahedral mesh, where each vertex of the tetrahedron is an SDF value;

[0147] 3.2 Subdivide the deformable tetrahedron mesh obtained in the first step by adding new points at the midpoints of each edge to increase the resolution. The SDF values ​​of these new vertices are obtained by averaging the SDF values ​​along the edges. This step can be performed multiple times to achieve the desired resolution.

[0148] 3.3 Convert the encoded SDF into an explicit triangle mesh using the Marching Tetrahedra algorithm.

[0149] Alternatively, the post-processing optimization method may be:

[0150] 3.4 Use the Loop Subdivision method to subdivide the model input in triangular patch format. This step can be performed multiple times to obtain the ideal resolution.

[0151] Alternatively, the post-processing optimization method may be:

[0152] 3.5 Repair the broken surface

[0153] Alternatively, the post-processing optimization method may be:

[0154] 3.6 Optimize using a neural network that first encodes the input into 3D features

[0155] 3.7 uses a fully connected network MLP to predict the SDF value for each vertex in the initial deformable tetrahedral mesh. The fully connected network also outputs a feature vector f for surface refinement in the volume subdivision stage.

[0156] 3.8 Iteratively refine the surface and subdivide the tetrahedral mesh. Specifically, we first determine the surface tetrahedron Tsurf based on the current SDF value. Construct a graph corresponding to the vertices and edges in the tetrahedron SDF representation. Use a graph convolutional network to predict the position offset Δvi and the SDF difference Δs(vi) for each vertex i. After surface refinement, we perform a volume subdivision step. In this process, an optimizer is used to minimize the L2 Chamfer distance and the normal vector consistency loss. The optimizer can be an Adam optimizer.

[0157] 3.9 Surface refinement step: Build a new graph on the extracted mesh and use GCN to predict the updated position of each vertex and the parameter alpha for Loop Subdivision.

[0158] 3.10 Using a pre-trained discriminator, first randomly select a high curvature vertex v in the target mesh and calculate the true signed distance field S1 in a voxelized region near v. Similarly, the signed distance field of the predicted surface mesh M is calculated at the same location to obtain S2, and S1 or S2 is fed into the discriminator together with the feature vector at location v for judgment. The discriminator can be a discriminator of DECOR-GAN, and the judgment loss function can be the adversarial loss of LSGAN.

[0159] Alternatively, the post-processing optimization method may be:

[0160] 3.11 Preprocess the input, which can be any step from step 3.1 to step 3.10.

[0161] 3.12 Use a U-Net-style Diffusion network to optimize the input, initialize the input with Gaussian noise, and denoise it using the DDPM sampling strategy.

[0162] Alternatively, the post-processing optimization method may be:

[0163] 3.13 Convert the input to SDF. We also initialize the volume texture field directly using the color field optimized from the coarse stage.

[0164] 3.14 We render the extracted surface mesh into a high-resolution image using a differentiable renderer.

[0165] 3.15 Optimize the SDF value of each vertex using high-resolution SDS gradients through backpropagation.

[0166] 3.16 When rendering a 3D model represented by a mesh into an image, we also keep track of the 3D coordinates of each corresponding pixel projection, which will be used to query the color in the corresponding texture field for joint optimization.

[0167] 3.17 As an optional technical solution, when rendering the mesh, we increase the focal length to magnify the object details, which is a key step in recovering high-frequency details. We retain the pre-trained environment map from the coarse optimization stage and use differentiable anti-aliasing methods to synthesize the rendered background with the foreground objects.

[0168] 3.18 To encourage surface smoothness, a gauge term is used to normalize the angular differences between adjacent faces on the mesh.

[0169] Example 2

[0170] A method for converting a sketch into a 3D model including beautifying the sketch:

[0171] In the first step, the interactive interface receives user input, which can be a sketch S drawn by the user on the touch screen.

[0172] Alternatively, the user input may be a 3D sketch drawn in three-dimensional space using an AR / VR device.

[0173] Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning,

[0174] Alternatively, the user input may be a contour map obtained by a digital image processing method through contour extraction from a color or grayscale image. The digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for image to contour map.

[0175] Step 2: The algorithm beautifies the sketch drawn by the user in real time and presents the beautified sketch on the interactive interface.

[0176] The beautification can also be performed when the user completes the sketch. Specifically, the beautification may include the following steps:

[0177] 2.1 Extracting the area where the strokes are located

[0178] 2.2 Synthesize the strokes into corresponding partial references through retrieval and interpolation

[0179] 2.3 Using bounding boxes of different regions to optimize the complete beautified sketch

[0180] In the third step, the software converts the sketch drawn by the user into a rough three-dimensional model and presents it on the user interface. The user confirms the presented effect. If it does not meet the user's needs, the process returns to the first step.

[0181] Specifically, the rough three-dimensional model can be a sparse point cloud, an occupancy function that only describes the shell surface, a three-dimensional model represented by a rough triangular patch, a three-dimensional model represented by voxels, a three-dimensional model represented by a neural radiation field, or a representation of T-SDF (truncated signed distance field), SDF (signed distance field), or UDF (unsigned distance field).

[0182] In the fourth step, based on the rough three-dimensional model, the algorithm performs post-processing optimization on the model. The post-processing optimization process includes refining or optimizing model details, or refining, optimizing or generating model textures.

[0183] The fifth step is to present the optimized model on the user interface. Steps 3 and 4 can be performed multiple times until the user is satisfied.

[0184] Through the intermediate form of 3D model display, users can see the rough modeling results more quickly, save computing resources, and help improve user experience.

[0185] Example 3

[0186] A user interaction method for continuously creating and editing a three-dimensional model based on a sketch, comprising the following steps:

[0187] In the first step, the interactive interface receives user input, which can be a sketch S drawn by the user on the touch screen.

[0188] Alternatively, the user input may be a 3D sketch drawn in three-dimensional space using an AR / VR device.

[0189] Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning,

[0190] Alternatively, the user input may be a contour map obtained by a digital image processing method through contour extraction from a color or grayscale image. The digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for image to contour map.

[0191] As an optional input, the user can enter the viewpoint corresponding to the sketch, such as normal view, oblique view, etc.

[0192] In the second step, the software converts the user's sketch into a 3D model, which the user can view on the interface. This conversion method can be the aforementioned method for generating a model from a high-resolution sketch, or other methods for converting sketches to 3D models, such as a search-based method that compares the sketch with outlines of corresponding 3D models in a database and selects the closest 3D model.

[0193] In the third step, the user rotates the three-dimensional model generated in the third step to an angle V on the interface for editing. If the user does not rotate, the default angle V is the viewpoint of the sketch drawn by the user.

[0194] In the fourth step, a renderer is used to render the model at angle V to obtain a rendering image R. Then, an edge extraction algorithm is used to extract the edges of the rendering image into a sketch S2. The edge extraction algorithm can be a Canny operator or a neural network model trained for image-to-contour mapping. The sketch is displayed on the user interface.

[0195] In step 5, the user edits the sketch S2 in step 4 on the user interface to obtain a sketch S3. The editing includes adding some elements, adjusting the positions of some elements, or scaling some elements.

[0196] Alternatively, the user adds a sketch to the rendering R in step 4 on the user interface to obtain sketch editing information S3. The sketch can be one or more strokes, a closed new structure or a structure to be subtracted, or a complete redrawn sketch.

[0197] The sixth step is an optional step, in which the user adds a new description information on the user interface. The description information can be text input, voice input, gesture input, etc.

[0198] In the seventh step, based on the sketch S3 and the three-dimensional model of the second step and the optional description information of the sixth step as input, the three-dimensional model is edited using the model post-processing iterative editing method to obtain a new three-dimensional model.

[0199] Alternatively, based on the sketch S3 and the 3D model of the second step as input, the 3D model is edited using a model post-processing iterative editing method to obtain a new 3D model.

[0200] In the eighth step, the user views the new three-dimensional model obtained in the seventh step in the interactive interface. If not satisfied, the user can return to the first or third step and start again.

[0201] Specifically, the model iterative editing method in step 7 can be:

[0202] Image-based optimization:

[0203] 7.1 Using Sketch to 3D Model A pre-trained image-to-image translation network is used to convert the sketch S3 into a projection of a 3D model. The projection of the 3D model is an image where each pixel has a value of 0 or 1. When the point can be projected onto the surface, it is assigned a value of 1, and otherwise it is 0.

[0204] 7.2 Use a differentiable renderer to render the sketch in step 6 into a projection of a three-dimensional model. The projection of the three-dimensional model is an image in which each pixel has a value of 0 or 1. When the point can be projected onto the surface, it is assigned a value of 1, and otherwise it is 0.

[0205] 7.3 compares the projection of 7.1 with the projection of 7.2. The optimizer changes the three-dimensional model so that the error between the two projections is small enough.

[0206] Alternatively, the iterative editing method for the model in step 7 can be:

[0207] Optimization based on SDF value

[0208] 7.4 Take the surface points of the outer contour area of ​​the model and use a differentiable renderer;

[0209] 7.5 Obtain the surface points of the sketch. The surface points may be obtained by emitting rays from the four boundaries of the image and retaining only the first black pixel hit by the ray.

[0210] 7.6 Take the images from 7.4 and 7.5 and optimize the model using the bidirectional 2D Chamfer loss. The optimization can be performed on the model generated in step 2 or on intermediate variables generated during the generation process in step 2, such as the model's latent code.

[0211] 7.7 As a preferred step (optional step), the optimization process is normalized, the normal value is obtained, and during the optimization process, the surface part projected near the sketch should match the sketch, while the other parts should maintain their original normals and boundaries.

[0212] Alternatively, optimization based on NeRF fields

[0213] 7.8 uses an iterative algorithm. In each iteration, a view is randomly sampled and the 3D object is rendered from that view. The new density mass added to the original NeRF representation of the model should occupy the area specified by the sketch. We achieve this by rendering the object mask for all scribble viewpoints.

[0214] We then maximize the object mask value of the scribble region by minimizing the following loss:

[0215] We use rendered images to compute the SDS loss to semantically align textual cues.

[0216] 7.9 In addition, we compute the distance between a point and the sketch to preserve the NeRF before the basic edit by constraining the density and color output of Fe to be similar to Fo outside the scribble area. Finally, we define a loss function using the object mask rendering from the sketch's perspective. This loss ensures that the object mask occupies the edited area.

[0217] Alternatively, an optimization based on the SPAGHETTI representation

[0218] 7.10 Use an image encoder to convert the input sketch into a visual embedding.

[0219] 7.11 uses a Transformer decoder to map a set of learned partial queries and the visual embeddings obtained in 7.10 into the multi-part latent space of SPAGHETTI. The latent representations received as input by SPAGHETTI are mapped to a set of m-dimensional vectors representing a Gaussian mixture model (GMM), i.e., each vector corresponds to a 3D Gaussian distribution.

[0220] 7.12 The user selects unsatisfactory parts from the output shape. The selected parts have corresponding latent vectors in the GMM latent space, and a refinement network outputs a set of vectors that replace the selected vectors, conditioned by the latent vectors of the non-selected parts. Finally, the SPAGHETTI shape generator uses the new vectors to generate the improved shape again. The refinement network can be a bidirectional Transformer encoder network that receives the set of latent vectors so that the corresponding selected part vectors are masked (i.e., set to zero). It outputs a refined vector corresponding to the selected part.

[0221] As a preferred technical solution, 5%-40% of the input vectors are masked, and the network needs to rely on the unmasked vector parts to output the complete shape.

[0222] Example 4

[0223] An immersive user interaction method for converting sketches into three-dimensional models includes the following steps:

[0224] The first step is to collect environmental information and present the environmental information on the interactive interface. The environmental information can be RGB panoramic information;

[0225] Alternatively, the environmental information is a normal map of a three-dimensional reconstruction of the scene or a preview of the scene after three-dimensional reconstruction. The three-dimensional reconstruction of the scene can be obtained by multi-view reconstruction or other deep learning methods, or can be directly read by a lidar, ultrasonic sensor, etc.

[0226] In the second step, the software receives user sketch input, and the user draws a single-view sketch on the layer of environmental information on the interactive interface.

[0227] As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network.

[0228] In the third step, the software converts the sketch drawn by the user into a three-dimensional model. The conversion method can be the aforementioned method of generating a model from a high-resolution sketch, or other methods of converting sketches into three-dimensional models.

[0229] In the fourth step, the 3D model obtained in the third step and the environmental information from the first step are presented on the interactive interface at the same position and angle as the user's hand-drawn sketch. The user can view the resulting 3D model on the interface.

[0230] The fifth step is to edit the model obtained in the third step based on the visualization results of the fourth step. The editing is based on the geometric relationship between the three-dimensional reconstruction results of the scene and the model obtained in the third step. The user can adjust parameters including position, size, etc., and can also adjust the components of the model.

[0231] The sixth step is an optional step. During the adjustment process, if there is positional interference between the generated model and the result of the existing 3D reconstruction of the environment, a prompt will be given on the interactive interface.

[0232] Specifically, the pseudo codes corresponding to the first to fifth steps are as follows:

[0233] in,

[0234] mesh of the target scene 3D reconstructed scene model

[0235] Camera pose of rendered pictures The camera pose of the current viewing scene

[0236] Image rendered in specified camera pose Sketch drawing pose

[0237] Freehand sketch masked on render

[0238] Resize

[0239] Crop interception area

[0240] Sketch-to-model Sketch conversion to 3D model

[0241] compute_translation calculates translation

[0242] Compute_rotation calculates rotation

[0243] Transform

[0244] Merge.

[0245] Example 5

[0246] A user interaction method for converting sketches into three-dimensional models for immersive object editing includes the following steps:

[0247] In the first step, a collection device collects information about objects in the environment and displays it on an interactive interface. This information can be a normal map of the object's 3D reconstruction or a preview of the reconstructed object model, such as a mesh rendering or a textured 3D object rendering. The 3D object reconstruction can be obtained using multi-view 3D reconstruction or other deep learning methods, or directly read using a LiDAR or ultrasonic sensor. The collection device can be an electronic terminal equipped with an image sensor, depth sensor, or spectral sensor, such as an AR / VR device, mobile phone, or tablet.

[0248] In the second step, the software receives the user's sketch input, and the user hand-draws a single-view sketch on the object information layer on the interactive interface.

[0249] As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network.

[0250] The sketch can be one or more strokes, a closed new structure or a structure to be subtracted, or a complete sketch drawn from scratch.

[0251] As a preferred technical solution, during the drawing process, when a stroke contacts a physical three-dimensional model, the contact points are marked and presented to the user.

[0252] In the third step, the software uses the sketch drawn by the user in the second step and the three-dimensional model of the object in the first step as input, and converts it into a complete three-dimensional model of the object after modification based on the original object.

[0253] The fourth step is to present the 3D model obtained in the third step on the interactive interface at the same position and angle as the user's hand-drawn sketch. The user can view the obtained 3D model on the interface.

[0254] Preferably, the user can select the physical 3D model to be overlaid and displayed on the 3D model edited from the sketch.

[0255] The fifth step is to edit the model obtained in the third step based on the visualization results of the fourth step. The editing is based on the geometric relationship between the three-dimensional reconstruction results of the scene and the model obtained in the third step. The user can adjust parameters including position, size, etc., and can also adjust the components of the model.

[0256] The sixth step is an optional step. During the adjustment process, if there is positional interference between the generated model and the result of the existing 3D reconstruction of the environment, a prompt will be given on the interactive interface.

[0257] The above steps can be repeated until the user is satisfied with the final model.

[0258] Example 6

[0259] A user interactive method for converting sketches into three-dimensional models by self-constructing objects and editing them, characterized by:

[0260] In the first step, the user manually constructs a three-dimensional model of the real physical world, and the three-dimensional model of the real physical world is obtained through sensors;

[0261] A three-dimensional object is manually constructed in physical space by stacking building blocks with circuits through addition and subtraction. This three-dimensional object can also be directly obtained and displayed in digital space. The building blocks can be block-like structures that can mesh or attract each other, or basic geometric shapes that contain meshing or magnetic structures. Each building block contains a circuit structure.

[0262] Specifically, the circuit structures can communicate with each other, transmit identity information, and pass it to a main controller. The main controller determines the relative positions of the building blocks in the physical space through the identity identification codes of different building blocks and the topological structure obtained by mutual communication, and presents it in the virtual three-dimensional space.

[0263] Alternatively, the circuit structure is a passive device composed of resistors and / or capacitors and / or inductors. There is a power supply in the main controller (base). The main controller (base) reads the changes in resistance, capacitance or inductance in real time, obtains the relative position of the building block in the physical space through the changes in resistance and / or capacitance at time t and t+1, and presents it in the virtual three-dimensional space.

[0264] The second step is to present the item information on the interactive interface. This item information can be a normal map of the 3D reconstruction of the item or a preview of the reconstructed item model, such as a mesh rendering or a textured 3D object rendering. The 3D object reconstruction can be obtained using multi-view 3D reconstruction or other deep learning methods, or directly read using a lidar, ultrasonic sensor, or other sensor. The acquisition device can be an electronic terminal equipped with an image sensor, depth sensor, or spectral sensor, such as an AR / VR device, mobile phone, or tablet.

[0265] In the third step, the software receives the user's sketch input, and the user hand-draws a single-view sketch on the object information layer on the interactive interface.

[0266] As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network.

[0267] The sketch can be one or more strokes, a closed new structure or a structure to be subtracted, or a complete sketch drawn from scratch.

[0268] As a preferred technical solution, during the drawing process, when a stroke contacts a physical three-dimensional model, the contact points are marked and presented to the user.

[0269] In the fourth step, the software uses the sketch drawn by the user in the second step and the three-dimensional model of the object in the first step as input, and converts it into a complete three-dimensional model of the object after modification based on the original object.

[0270] Step 5: Display the 3D model obtained in step 3 on the interactive interface at the same position and angle as the user's hand-drawn sketch. The user can view the resulting 3D model on the interface.

[0271] Preferably, the user can select the physical 3D model to be overlaid and displayed on the 3D model edited from the sketch.

[0272] The sixth step is to edit the model obtained in the third step based on the visualization results of the fourth step. The editing is based on the geometric relationship between the three-dimensional reconstruction results of the scene and the model obtained in the third step. The user can adjust parameters including position, size, etc., and can also adjust the components of the model.

[0273] The seventh step is an optional step. During the adjustment process, if there is positional interference between the generated model and the result of the existing 3D reconstruction of the environment, a prompt will be given on the interactive interface.

[0274] The above steps can be repeated until the user is satisfied with the final model.

[0275] Example 7

[0276] A workpiece processing method comprises the following steps:

[0277] The first step is to obtain a customized three-dimensional model file using the method described in any one of embodiments 1-6 or any combination thereof;

[0278] The second step is to input the 3D model file into the parser and convert it into machine-readable instructions;

[0279] In the third step, the processing device performs processing according to machine-readable instructions.

[0280] Example 8

[0281] A workpiece processing device is configured to process a workpiece according to the method described in Example 7.

[0282] Example 9

[0283] An electronic terminal includes an input component, an output component, and a processing component, and is configured to execute the method described in any one of embodiments 1-6.

Claims

1. A user interaction method for generating a high-resolution sketch model, characterized in that: The following steps are involved: In the first step, the interactive interface receives user input, which may be a sketch S drawn by the user on the touch screen. In the second step, the software converts the sketch drawn by the user into a rough 3D model and presents it on the user interface. The user confirms the presented effect. If it does not meet the user's needs, it returns to the first step. In the third step, based on the rough three-dimensional model, the algorithm performs post-processing optimization on the model, and the post-processing optimization process includes refining or optimizing model details, or refining or optimizing or generating model textures. The fourth step is to present the optimized model on the user interface. The third and fourth steps can be performed multiple times until the user is satisfied.

2. The user interaction method for generating a high-resolution sketch model according to claim 1, characterized in that: In the first step, the user input may be a 3D sketch drawn in a three-dimensional space using an AR / VR device. Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning, Alternatively, the user input may be a contour map obtained by a digital image processing method through contour extraction from a color or grayscale image. The digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for image to contour map.

3. The user interaction method for generating a high-resolution sketch model according to claim 1, characterized in that In the second step, specifically, the rough three-dimensional model can be a sparse point cloud, an occupancy function that only describes the shell surface, a three-dimensional model represented by a rough triangular patch, a three-dimensional model represented by voxels, a three-dimensional model represented by a neural radiation field, or a representation of T-SDF (truncated signed distance field), SDF (signed distance field), or UDF (unsigned distance field).

4. The user interaction method for generating a high-resolution sketch model according to claim 1, characterized in that In the third step, the model post-processing optimization method may be: 3.1 Convert the input into a deformable tetrahedral mesh signed distance field (SDF), where each vertex of the tetrahedron is an SDF value. 3.2 Subdivide the deformable tetrahedral mesh obtained in the first step, and increase the resolution by adding new points at the midpoint of each edge. The SDF value of the new vertex is obtained by averaging the SDF values ​​on the edge. This step can be performed multiple times to obtain the desired resolution. 3.3 Convert the encoded SDF into an explicit triangle mesh using the Marching Tetrahedra algorithm; 5. A user interaction method for generating a model from a high-resolution sketch according to claim 4, characterized in that the post-processing optimization method can be to use a Loop Subdivision method to subdivide the model input in a triangular face format, and this step can be performed multiple times to obtain an ideal resolution; Alternatively, the post-processing optimization method may be: 3.5 to perform broken surface repair.

5. The user interaction method for generating a high-resolution sketch model according to claim 4, characterized in that: The post-processing optimization method may be, 3.6 Optimize using a neural network that first encodes the input into 3D features; 3.7 Use a fully connected network MLP to predict the SDF value of each vertex in the initial deformable tetrahedral mesh. The fully connected network also outputs a feature vector f for surface refinement in the volume subdivision stage; 3.8 Iteratively refine the surface and subdivide the tetrahedral mesh; specifically, we first determine the surface tetrahedron Tsurf based on the current SDF value, construct a graph corresponding to the vertices and edges in the tetrahedron SDF representation, and use a graph convolutional network to predict the position offset Δvi and the SDF difference Δs(vi) for each vertex i. After the surface is refined, a volume subdivision step is performed. In this process, an optimizer is used to minimize the L2 Chamfer distance and the normal vector consistency loss. The optimizer can be an Adam optimizer.

6. The user interaction method for generating a high-resolution sketch model according to claim 5, characterized in that: The post-processing optimization method may be that the surface refinement step specifically includes: 3.9 Build a new graph on the extracted mesh and use GCN to predict the updated position of each vertex and the parameter alpha for Loop Subdivision; 3.10 Using a pre-trained discriminator, first randomly select a high curvature vertex v in the target mesh, and calculate the true signed distance field S1 in a voxelized area near v; similarly, calculate the signed distance field of the predicted surface mesh M at the same position to obtain S2, and feed S1 or S2 together with the feature vector in position v into the discriminator for judgment; the discriminator can be a DECOR-GAN discriminator, and the judgment loss function can be the adversarial loss of LSGAN.

7. A user interaction method for generating a high-resolution sketch model according to any one of claims 4 to 6, characterized in that: The post-processing optimization method may be, 3.11 Preprocess the input, and the preprocessing may be any step of steps 3.1-3.10; 3.12 Use a U-Net-style Diffusion network to optimize the input, initialize the input with Gaussian noise, and denoise it using the DDPM sampling strategy.

8. The user interaction method for generating a high-resolution sketch model according to claim 7, characterized in that: The post-processing optimization method may be, 3.13 Convert the input to SDF and directly use the color field optimized from the rough stage to initialize the volume texture field; 3.14 We use a differentiable renderer to render the extracted surface mesh into a high-resolution image; 3.15 Optimize the SDF value of each vertex using high-resolution SDS gradients through back-propagation; 3.16 When rendering the 3D model represented by the mesh into an image, we also keep track of the 3D coordinates of each corresponding pixel projection, which will be used to query the color in the corresponding texture field for joint optimization; 3.17 As an optional technical solution, when rendering the mesh, increase the focal length to magnify the object details, which is a key step in recovering high-frequency details; retain the pre-trained environment map from the coarse stage of optimization, and use a differentiable anti-aliasing method to synthesize the rendered background with the foreground object; 3.18 To encourage surface smoothness, a normative term is used to normalize the angle differences between adjacent faces on the mesh; 8. A method for converting a sketch into a three-dimensional model, comprising beautifying the sketch, characterized in that it comprises the following steps: In the first step, the interactive interface receives user input, which may be a sketch S drawn by the user on the touch screen. The user input may be a 3D sketch drawn in a three-dimensional space using an AR / VR device. Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning, Alternatively, the user input may be a contour map obtained by a digital image processing method for contour extraction from a color or grayscale image, wherein the digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for converting an image to a contour map; Step 2: The algorithm beautifies the sketch drawn by the user in real time and presents the beautified sketch on the interactive interface. The beautification can also be performed when the user completes the sketch drawing. Specifically, the beautification may include the following steps: 2.1 Extract the area where the strokes are located, 2.2 The strokes are synthesized into corresponding partial references through retrieval and interpolation, 2.3 Use bounding boxes of different regions to optimize the complete beautified sketch; In the third step, the software converts the sketch drawn by the user into a rough 3D model and presents it on the user interface. The user confirms the presented effect. If it does not meet the user's needs, it returns to the first step; Specifically, the rough three-dimensional model may be a sparse point cloud, an occupancy function that only describes the shell surface, a three-dimensional model represented by a rough triangular patch, a three-dimensional model represented by voxels, a three-dimensional model represented by a neural radiation field, or a representation of a T-SDF (truncated signed distance field), an SDF (signed distance field), or an UDF (unsigned distance field); Step 4: Based on the rough three-dimensional model, the algorithm performs post-processing optimization on the model, wherein the post-processing optimization process includes refining or optimizing model details, or refining or optimizing or generating model textures; The fifth step is to present the optimized model on the user interface. The third and fourth steps can be performed multiple times until the user is satisfied.

9. A user interaction method for continuous creation and editing of a three-dimensional model based on a sketch, characterized in that: The following steps are involved: In the first step, the interactive interface receives user input, which may be a sketch S drawn by the user on the touch screen. Alternatively, the user input may be a 3D sketch drawn in a three-dimensional space using an AR / VR device. Alternatively, the user input may be a sketch drawn by the user on paper obtained by photographing or scanning, Alternatively, the user input may be a contour map obtained by a digital image processing method for contour extraction from a color or grayscale image, wherein the digital image processing method may be contour extraction using a Canny operator, or may be training a neural network model for converting an image to a contour map; As an optional input, the user can enter the viewpoint corresponding to the sketch, such as front view, oblique view, etc. In the second step, the software converts the sketch drawn by the user into a 3D model, and the user can view the obtained 3D model on the interface. The conversion method can be the aforementioned method of generating a model from a high-resolution sketch, or other methods of converting a sketch into a 3D model, such as a search-based method, which searches the sketch and the contour map with the corresponding 3D model in the database to select the closest 3D model; In the third step, the user rotates the three-dimensional model generated in the third step to an angle V for editing on the interface. If the user does not rotate, the viewpoint of the sketch drawn by the user is used as the angle V by default; Step 4: Use a renderer to render the model at angle V to obtain a rendering image R, and then use an edge extraction algorithm to extract the edge of the rendering image into a sketch S2. The edge extraction algorithm can be a Canny operator or a neural network model trained for image to contour map. The sketch is displayed on the user interface; In the fifth step, the user edits the sketch S2 in the fourth step on the user interface to obtain a sketch S3, wherein the editing includes adding some elements, adjusting the positions of some elements, or scaling some elements; Alternatively, the user adds a sketch to the rendering R in the fourth step on the user interface to obtain sketch editing information S3. The sketch may be one or more strokes, or a closed new structure or a structure to be subtracted, or a redrawn complete sketch. The sixth step is an optional step, in which the user adds a new description information on the user interface, and the description information can be text input, voice input, gesture input, etc.; Step 7, based on the sketch S3 and the 3D model of the second step and the optional description information of the sixth step as input, the 3D model is edited using a model post-processing iterative editing method to obtain a new 3D model; Alternatively, based on the sketch S3 and the three-dimensional model in the second step as input, the three-dimensional model is edited using a model post-processing iterative editing method to obtain a new three-dimensional model; In step 8, the user views the new 3D model obtained in step 7 in the interactive interface. If the user is not satisfied, he or she can return to step 1 or step 3 and start over.

10. The user interaction method for continuous creation and editing of a three-dimensional model based on a sketch according to claim 9, characterized in that: The iterative editing method of the model in step 7 can be: Image-based optimization: 7.1 Using Sketch to 3D Model Use a pre-trained image-to-image translation network to convert the sketch S3 into a projection of a 3D model. The projection of the 3D model is an image where each pixel has a value of 0 or 1. When the point can be projected onto the surface, it is assigned a value of 1, and otherwise it is assigned a value of 0. 7.2 Use a differentiable renderer to render the sketch in step 6 into a projection of a three-dimensional model, where the projection of the three-dimensional model is an image where each pixel has a value of 0 or 1. When the point can be projected onto the surface, the value is 1, and otherwise it is 0. 7.3 compares the projection of 7.1 with the projection of 7.

2. The optimizer changes the three-dimensional model so that the error between the two projections is small enough.

11. The user interaction method for continuous creation and editing of a three-dimensional model based on a sketch according to claim 9, characterized in that: The iterative editing method of the model in step 7 can be: Optimization based on SDF value 7.4 Take the surface points of the outer contour area of ​​the model and use a differentiable renderer; 7.5 Take the surface points of the sketch. The surface points may be taken by emitting rays from the four boundaries of the image and retaining only the first black pixel hit by the ray; 7.6 Take the pictures in 7.4 and 7.5, and use the bidirectional 2D Chamfer loss to optimize the model. The optimization may be the model generated in the second step, or the intermediate variables generated in the second step, such as the latent code of the model. 7.7 As a preferred step (optional step), the optimization process is normalized to obtain normal values, and during the optimization process, the surface portion projected near the sketch should match the sketch, while the other portions should maintain their original normals and boundaries; 11. The user interactive method for continuous creation and editing of a three-dimensional model based on a sketch according to claim 9, wherein the model iterative editing method in the seventh step can be: NeRF field-based optimization 7.8 adopts an iterative algorithm. In each iteration, a view is randomly sampled and the 3D object is rendered from that view; such that the new density mass added to the model under the original NeRF representation should occupy the area specified by the sketch. We do this by rendering the object mask for all scribble views; then, we maximize the object mask value of the scribble area by minimizing the following loss: Use rendered images to compute the SDS loss to semantically align text cues; 7.9 In addition, we compute the distance between a point and the sketch to preserve the NeRF before the basic edit, by constraining the density and color output of Fe to be similar to Fo outside the scribble area. Finally, we define a loss function using the object mask rendering from the sketch’s perspective; this loss ensures that the object mask occupies the edited area.

12. The user interaction method for continuous creation and editing of a three-dimensional model based on a sketch according to claim 9, characterized in that: The iterative editing method of the model in step 7 can be: Optimization based on SPAGHETTI representation 7.10 Use an image encoder to convert the input sketch into a visual embedding; 7.11 uses a Transformer decoder to map a set of learned partial queries and the visual embeddings obtained in 7.10 to the multi-part latent space of SPAGHETTI. The latent representation received as input by SPAGHETTI is mapped to a set of m-dimensional vectors representing a Gaussian mixture model (GMM), i.e., each vector corresponds to a 3D Gaussian distribution; 7.12 The user selects an unsatisfactory part from the output shape; the selected part has a corresponding latent vector in the GMM latent space, and a refinement network outputs a set of vectors that replace the selected vectors under the adjustment of the latent vectors of the non-selected parts; finally, the SPAGHETTI shape generator uses the new vectors to generate the improved shape again; the refinement network can be a bidirectional Transformer encoder network, which receives a set of latent vectors so that the corresponding selected part vectors are masked (i.e., set to zero); it outputs a refined vector corresponding to the selected part; as a preferred technical solution, 5%-40% of the input vectors are masked, and the network needs to rely on the vector parts that are not masked to output the complete shape.

13. An immersive user interaction method for converting sketches to three-dimensional models, characterized in that: The following steps are involved: The first step is to collect environmental information and present the environmental information on an interactive interface. The environmental information may be RGB panoramic information taken; Alternatively, the environmental information is a normal map of a three-dimensional reconstruction of a scene or a preview of a scene after three-dimensional reconstruction. The three-dimensional reconstruction of the scene can be obtained by multi-view reconstruction or other deep learning methods, or can be directly read by a laser radar, an ultrasonic sensor, etc.; In the second step, the software receives the user's sketch input, and the user draws a single-view sketch on the layer of environmental information on the interactive interface; As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network; In the third step, the software converts the sketch drawn by the user into a three-dimensional model. The conversion method can be the above-mentioned method of generating a model from a high-resolution sketch, or other methods of converting a sketch into a three-dimensional model; The fourth step is to present the 3D model obtained in the third step and the environmental information in the first step on the interactive interface. The position is the location and angle of the user's hand-drawn sketch, and the user can view the resulting 3D model on the interface. Step 5: Based on the visualization result of step 4, the model obtained in step 3 is edited. The editing is based on the geometric relationship between the scene 3D reconstruction result and the model obtained in step 3. The user can adjust parameters including position, size, etc., and can also adjust components of the model. The sixth step is an optional step. During the adjustment process, if there is position interference between the generated model and the result of the 3D reconstruction of the existing environment, a prompt will be given on the interactive interface.

14. A user interaction method for converting sketches to three-dimensional models for immersive object editing, characterized in that: The following steps are involved: The first step is to collect information about objects in the environment through a collection device and present the information on the interactive interface. The object information is a normal map of the object's 3D reconstruction or a preview of the object model after 3D reconstruction, such as mesh rendering or textured 3D object rendering. The object's 3D reconstruction can be obtained by multi-view 3D reconstruction or other deep learning methods, or can be directly read by a laser radar, ultrasonic sensor, etc. The collection device can be an electronic terminal with an image sensor / depth sensor / spectral sensor, such as an AR / VR device, a mobile phone, or a tablet computer; In the second step, the software receives the user's sketch input, and the user draws a single-view sketch on the object information layer on the interactive interface; As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network; The sketch may be one or more strokes, or a closed new structure or a structure to be subtracted, or a complete sketch drawn from scratch; As a preferred technical solution, during the drawing process, when the strokes come into contact with the physical three-dimensional model, the contact points are marked and presented to the user; In the third step, the software uses the sketch drawn by the user in the second step and the 3D model of the object in the first step as input and converts it into a complete 3D model of the object after modification based on the original object; Step 4: Present the 3D model obtained in step 3 on the interactive interface. The presentation position is the position and angle of the user's hand-drawn sketch. The user can view the obtained 3D model on the interface; Preferably, the user can select the physical 3D model to be overlaid and displayed on the 3D model edited by the sketch; Step 5: Based on the visualization result of step 4, the model obtained in step 3 is edited. The editing is based on the geometric relationship between the scene 3D reconstruction result and the model obtained in step 3. The user can adjust parameters including position, size, etc., and can also adjust components of the model. The sixth step is an optional step. During the adjustment process, if the generated model and the existing environment 3D reconstruction are different, If there is position interference, a prompt will be given on the interactive interface; The above steps can be repeated until the user is satisfied with the final model.

15. A user interactive method for converting a sketch into a three-dimensional model by editing a self-built object, characterized in that: The following steps are involved: In the first step, the user manually constructs a three-dimensional model of the real physical world, and the three-dimensional model of the real physical world is obtained through a sensor; Manually construct a three-dimensional object in a physical space by stacking building blocks with circuits through addition and subtraction, and the three-dimensional object can also be directly obtained and displayed in a digital space, wherein the building blocks can be block structures that can mesh or attract each other, or basic geometric bodies that contain meshing structures or magnetic structures, and each building block has a circuit structure; Specifically, the circuit structures can communicate with each other, transfer identity information, and transfer it to a main controller, and the main controller determines the relative positions of the building blocks in the physical space through the identity identification codes of different building blocks and the topological structure obtained by mutual communication, and presents it in the virtual three-dimensional space; Alternatively, the circuit structure is a passive device composed of resistors and / or capacitors and / or inductors. The main controller (base) has a power supply. The main controller (base) reads the change of the resistor or capacitor or inductor in real time, obtains the relative position of the building block in the physical space through the change of the resistor and / or capacitor at time t and t+1, and presents it in the virtual three-dimensional space. The second step is to present the item information on the interactive interface. The item information is a normal map of the 3D reconstruction of the item or a preview of the item model after 3D reconstruction, such as mesh rendering or 3D object rendering with texture. The 3D reconstruction of the object can be obtained by multi-view 3D reconstruction or other deep learning methods, or can be directly read by a laser radar, ultrasonic sensor, etc. The acquisition device can be an electronic terminal with an image sensor / depth sensor / spectral sensor, such as an AR / VR device, a mobile phone or a tablet computer. In the third step, the software receives the user's sketch input, and the user draws a single-view sketch on the object information layer on the interactive interface; As an optional input, the user can enter the pose of the sketch, or the pose can be predicted by a neural network; The sketch may be one or more strokes, or a closed new structure or a structure to be subtracted, or a complete sketch drawn from scratch; As a preferred technical solution, during the drawing process, when the strokes come into contact with the physical three-dimensional model, the contact points are marked and presented to the user; In the fourth step, the software uses the sketch drawn by the user in the second step and the three-dimensional model of the object in the first step as input and converts it into a complete three-dimensional model of the object after modification based on the original object; Step 5: Present the 3D model obtained in step 3 on the interactive interface. The presentation position is the position and angle of the user's hand-drawn sketch. The user can view the obtained 3D model on the interface. Preferably, the user can select the physical 3D model to be overlaid and displayed on the 3D model edited by the sketch; Step 6: Based on the visualization result of step 4, the model obtained in step 3 is edited. The editing is based on the geometric relationship between the scene 3D reconstruction result and the model obtained in step 3. The user can adjust parameters including position, size, etc., and can also adjust components of the model. The seventh step is an optional step. During the adjustment process, if there is a positional interference between the generated model and the result of the existing 3D reconstruction of the environment, a prompt will be given on the interactive interface; The above steps can be repeated until the user is satisfied with the final model.

16. A workpiece processing method, characterized in that: The following steps are involved: The first step is to obtain a customized 3D model using the method described in any one of claims 1 to 15; The second step is to input the 3D model into a parser and convert it into machine-readable instructions; In the third step, the processing device performs processing according to machine-readable instructions.

17. A workpiece processing device, characterized in that: The method is configured to machine a workpiece according to claim 16.

18. An electronic terminal, comprising an input component, an output component and a processing component, characterized in that: The method is configured to perform the method according to any one of claims 1 to 15.

Images