Method, device and equipment for converting three-dimensional sketch into three-dimensional model and storage medium

Abstract

This application provides a method, apparatus, device, and storage medium for converting 3D sketches into 3D models, relating to the field of image processing technology. The method includes: systematically processing the acquired 3D sketch to obtain a binary voxel sketch with a unified data format; performing multi-view projection and feature encoding on the binary voxel sketch to obtain conditional latent vectors encoded with the sketch's geometric features; generating a 3D model based on the conditional latent vectors using a conditional diffusion model; and converting the generated 3D model into an editable polygonal mesh. This application can significantly improve the quality and controllability of model generation from 3D sketches, reduce dependence on input integrity, and maintain the editability of the output model, providing efficient technical support for conceptual design, rapid prototyping, and human-computer collaborative modeling.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for converting a three-dimensional sketch into a three-dimensional model. Background Technology

[0002] In the field of 3D content creation and collaborative design, the ability to quickly convert hand-drawn 3D sketches into editable 3D models is a technology with significant application value. Designers often use 3D sketches to express initial ideas during the conceptualization stage. These sketches typically feature sparse strokes, incomplete geometric information, and ambiguous spatial structures. How to automatically generate structurally sound and surface-continuous 3D models from these imprecise inputs has long been a challenge in the fields of computer graphics and computer-aided design.

[0003] Traditional methods typically employ rule-based reconstruction algorithms or geometry-optimized deformation methods. These methods rely on predefined geometric primitives or template libraries, requiring high integrity and regularity in the input sketch, and struggle to handle freely drawn, unstructured strokes. With the rise of deep learning, some researchers have attempted to use supervised voxel prediction networks to directly regress 3D occupancy information. However, these methods have high requirements for training data and limited generalization ability to fuzzy inputs, often resulting in models lacking structural diversity and local detail. When the input sketch contains only sparse lines or contains spatial ambiguity, existing techniques struggle to generate coherent 3D shapes that match the design intent.

[0004] Conditional generative models, especially diffusion models, that have emerged in recent years have demonstrated the powerful potential to generate complete data from partial conditional information. However, in the specific task of 3D sketch conversion, how to effectively extract the geometric features of the sketch and use them as generation conditions, how to maintain the continuity of strokes and the consistency of 3D structure, and how to enable the generation process to support multiple candidate results to adapt to the iterative characteristics of collaborative design remain unsolved technical problems. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for converting three-dimensional sketches into three-dimensional models, in order to solve the problem in the prior art of generating coherent structural models from sparse and incomplete three-dimensional sketches.

[0006] Specifically, this application first regularizes the input 3D sketch into a unified binary voxel representation. Through geometric normalization and stroke continuity preservation, it eliminates the scale ambiguity and sampling discreteness of the original sketch, providing a standardized data format for subsequent feature extraction. Second, this application employs a strategy combining multi-view projection and a pre-trained image encoder to extract the 2D projection features of the sketch from multiple perspectives. Multi-view fusion is then performed through a self-attention mechanism to generate a compact and expressive conditional latent vector, which effectively encodes the global geometric contour and local structural information of the sketch. Third, guided by this conditional latent vector, this application constructs a 3D generative network based on a conditional diffusion model. Through a learning paradigm of forward noise addition and inverse denoising, the model can gradually recover a 3D voxel model matching the sketch conditions from Gaussian noise. Furthermore, it supports obtaining multiple candidate outputs by changing the random seed, meeting the diverse needs of collaborative design scenarios. Finally, this application converts the generated voxel model into a signed distance field and uses the traveling cube algorithm to extract a smooth triangular mesh, achieving a smooth conversion from discrete voxels to an editable polygon model.

[0007] In a first aspect, this application provides a method for converting a 3D sketch into a 3D model, the method comprising: The acquired 3D sketches are systematically processed to obtain binary voxel sketches with a unified data format; The binary voxel sketch is subjected to multi-view projection and feature encoding to obtain a conditional latent vector encoded with the sketch's geometric features; Based on the conditional latent vector, a 3D model is generated using a conditional diffusion model, and the generated 3D model is converted into an editable polygonal mesh.

[0008] In one possible design, the acquired 3D sketches are systematically processed to obtain binary voxel sketches with a unified data format, specifically including: The acquired 3D sketch is represented as a sequence of 3D points. : in, The number of strokes. For the first The number of sampling points in a single stroke. Representing three-dimensional real space, For the first Strokes The Middle The three-dimensional coordinates of each sampling point; Calculate the maximum distance along the X, Y, and Z axes of the stroke. The calculation formula is: in, and These represent the maximum and minimum values ​​of all stroke points on the X-axis. and These represent the maximum and minimum values ​​of all stroke points on the Y-axis. and These represent the maximum and minimum values ​​of all stroke points on the X-axis. It is a function for maximizing the value; Based on the maximum distance The normalized 3D point coordinates are calculated using the following formula: in, For the original three-dimensional points, For normalized three-dimensional points, Center of the sketch; Construct a regular 3D voxel mesh, represented as ,in The resolution of the regular 3D voxel mesh is {0,1}, which means that the voxel values ​​are only 0 or 1. Initially, all voxel values ​​are set to 0. For a normalized 3D point, the corresponding voxel index is calculated using the following formula. in, For voxel indexing, This indicates the floor function. , and Represents the normalized three-dimensional point coordinates; If the calculated voxel index falls within the valid range of the regular 3D voxel grid, then the voxel grid indexed as... The voxel value is set to 1; During the voxel mesh construction process, for any adjacent points in a stroke... Linear interpolation is performed to calculate the Euclidean distance between adjacent point pairs, and linear interpolation is carried out with a fixed step size to obtain the coordinates of the interpolated points as follows: in, These are interpolation parameters. These are the coordinates of the interpolation points; Map all the interpolated 3D points onto a regular 3D voxel grid to obtain a binary voxel sketch.

[0009] In one possible design, the binary voxel sketch is subjected to multi-view projection and feature encoding to obtain a conditional latent vector encoding the sketch's geometric features, specifically including: The binary voxel sketch is projected onto multiple two-dimensional views to generate a binary projected image for each viewpoint. , and These are the height and width of the image, respectively; Each projected image The image is fed into a pre-trained CLIP image encoder to extract high-dimensional semantic representations. , serving as view features from various perspectives; A self-attention-based feature fusion module is used to adaptively aggregate view features from each perspective to obtain conditional latent vectors.

[0010] In one possible design, a self-attention-based feature fusion module is used to adaptively aggregate view features from various perspectives to obtain conditional latent vectors, specifically including: Stack the view features from each perspective into a matrix. X ; Based on the matrix X Constructing a query matrix through linear mapping Key matrix Sum matrix : in, The weight matrix is ​​a learnable matrix. For the real number space, The dimension of the attention head; Calculate attention weights The calculation formula is: in, For normalized exponential functions, This represents the product of the query matrix and the transpose of the key matrix. This is the scaling factor; Based on attention weights , obtain fusion features ;in , , and These represent the first fusion feature, the second fusion feature, and the third fusion feature, respectively. The first, second, and third fusion features are concatenated along the channel dimension: in This represents a vector concatenation operation. Indicates the features after splicing The spliced ​​features The input is processed by a multilayer perceptron for nonlinear fusion, resulting in a single global condition vector. For the global condition vector The final normalization yields conditional latent vectors that encode the geometric features of the sketch multi-view. .

[0011] In one possible design, a 3D model is generated based on the conditional latent vector using a conditional diffusion model, specifically including: The target 3D model is represented as a voxel-occupied mesh, which serves as the generation target for the conditional diffusion model; The conditional diffusion model is obtained through training and employs a continuous-time denoised diffusion model. The forward diffusion process is a gradual diffusion towards the true 3D representation. Adding Gaussian noise to the input, we can obtain the result at any time. perturbation samples : in, Standard Gaussian noise, This represents a Gaussian distribution with a mean of 0 and a covariance of I. For continuous time variables, Indicates a uniform distribution on the range of 0 to 1. It is a noise scheduling function that monotonically decreases from 1 to 0; The inverse process of the conditional diffusion model is completed by a conditional denoising network, which uses the conditional latent vector. As a global guiding condition, its input includes perturbation samples. Continuous time variables and the conditional latent vector It adopts a multi-level encoder-decoder structure based on 3D convolution, and achieves multi-scale feature fusion through residual connections and skip connections, outputting surface occupancy prediction; The inference phase initializes with Gaussian noise on a voxel grid of a set resolution, in the conditional latent vector. Guided by the pre-trained conditional denoising network, a progressive denoising sampling strategy is used to gradually obtain the surface occupancy prediction. Voxels with predicted values ​​greater than a preset threshold are retained to obtain a three-dimensional model.

[0012] In one possible design, the conditional diffusion model employs the following loss function during training: in, For loss function, Represents the mathematical expectation. For the Euclidean norm, The model's previous prediction estimate.

[0013] In one possible design, the generated 3D model is converted into an editable polygonal mesh, specifically including: Calculate the distance from each interior point in the 3D model to the nearest boundary. Distance from external point to nearest boundary Construct a signed distance field : in, This indicates that the point is located inside the object. Indicates that it is located on the outside. Corresponding object surface; Extract the signed distance field The zero-level isosurface is used to obtain an editable polygonal mesh.

[0014] Secondly, this application provides an apparatus for converting a three-dimensional sketch into a three-dimensional model, the apparatus comprising: The sketch voxelization module is configured to systematically process the acquired 3D sketches to obtain binary voxel sketches with a unified data format. The sketch voxel processing module is configured to perform multi-view projection and feature encoding on the binary voxel sketch to obtain a conditional latent vector encoded with the sketch's geometric features. The 3D model generation module is configured to generate a 3D model based on the conditional latent vector using a conditional diffusion model, and convert the generated 3D model into an editable polygonal mesh.

[0015] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the method for converting a three-dimensional sketch into a three-dimensional model as described in the first aspect and various possible designs of the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method for converting a three-dimensional sketch into a three-dimensional model as described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method for converting a three-dimensional sketch into a three-dimensional model as described in the first aspect and various possible designs of the first aspect.

[0018] The method, apparatus, device, and storage medium for converting 3D sketches into 3D models provided in this application have at least the following beneficial effects: 1) This application effectively eliminates the scale ambiguity and stroke sampling discreteness problems existing in the original sketch by regularizing the input 3D sketch into a binary voxel sketch with a unified data format, and combining geometric normalization and stroke continuity preservation processing. Specifically, geometric normalization scales all stroke points to a unit cube while preserving boundary margins, ensuring that sketch inputs of different sizes can be aligned to a unified voxel space; stroke continuity preservation fills the gaps between adjacent points through linear interpolation, avoiding voxel representation breaks caused by sparse sampling. This processing enables the subsequent condition extraction stage to obtain a structurally complete geometric contour, improving robustness to freehand sketches.

[0019] 2) This application employs a strategy of multi-view projection and self-attention feature fusion. The 3D voxel sketch is projected into multiple 2D views. A pre-trained CLIP image encoder is used to extract high-dimensional semantic features, and then a self-attention mechanism adaptively aggregates the geometric information from different perspectives to generate compact conditional latent vectors. Compared to methods that directly process 3D voxels or use single-view conditions, multi-view projection significantly reduces computational complexity while leveraging the mature pre-trained representation capabilities in the image domain. Self-attention fusion captures the complementary relationships between views, avoiding redundancy caused by simple stitching, and enabling the generated conditions to more accurately reflect the 3D structure of the sketch.

[0020] 3) This application uses a conditional diffusion model as its core generative framework. Through a learning paradigm of forward noise addition and inverse denoising, the model can progressively recover a 3D voxel model that matches the sketch conditions from Gaussian noise. This process does not rely on predefined geometric primitives, can adapt to diverse sketch inputs, and generate 3D shapes with complex topology and local details. Compared with traditional regression-based voxel prediction networks, the diffusion model is more robust to fuzzy inputs and can generate structurally coherent and natural models, effectively solving the problem of generating complete 3D structures from sparse strokes in existing technologies.

[0021] 4) This application supports obtaining multiple candidate 3D models by changing the random seed during the inference stage, providing diverse outputs for design exploration in collaborative conceptualization scenarios. Designers can quickly generate multiple different but reasonable model variants under the same sketch conditions, select the result that best matches the design intent, or make further modifications and iterations, significantly improving the efficiency and flexibility of human-computer collaborative modeling.

[0022] 5) This application converts the generated 3D model from voxel representation to a signed distance field, and then extracts triangular meshes using the traveling cube algorithm, thus solving the surface roughness problem caused by the stepped voxel boundaries. The signed distance field can accurately describe the distance information from the object's interior and exterior to the surface. Combined with the mesh generated by the traveling cube algorithm, it has smooth curved surfaces and editable polygonal structures, which can be directly used for downstream 3D rendering, modification, or printing applications, improving the practical usability of the output model. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] Figure 1 This application provides an overall flowchart of a method for converting a 3D sketch into a 3D model. Figure 2 A flowchart illustrating the implementation of a method for converting a 3D sketch into a 3D model, provided in this application embodiment; Figure 3 A flowchart of the three-dimensional sketch voxelization stage provided in the embodiments of this application; Figure 4 A flowchart of the sketch voxel processing stage provided in the embodiments of this application; Figure 5 A flowchart of the content generation stage provided for embodiments of this application; Figure 6 This is a structural diagram of the apparatus for converting a three-dimensional sketch into a three-dimensional model provided in an embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] The collection, storage, use, processing, transmission, provision, and disclosure of relevant data and information in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0029] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0030] This application provides a method for converting a 3D sketch into a 3D model, such as... Figure 1 The diagram shows the overall flowchart of the method for converting a 3D sketch into a 3D model. The method first performs 3D sketch voxelization on the input 3D sketch to obtain sketch voxels. Then, the sketch voxels are input into the Sketch Voxel Processing stage, where they are projected to generate three views. The pre-trained encoder in the Vision Transformer extracts the latent vectors corresponding to each view. These latent vectors are then fed into the Self-Attention Fusion module, where they pass through an attention mechanism, weighted combination, fusion module, and nonlinear projection to obtain a fused vector. This fused vector is then used as a condition input to the Model Generation stage. In the Model Generation stage, Gaussian noise is used as the initial input, and a U-Net is used to generate candidate voxels. The candidate voxel results are then processed by voxel-to-SDF to obtain candidate signed distance field results. The generated candidate signed distance field results support editing operations and can be fed back to the sketch voxel stage to realize iterative optimization and modification of the 3D model.

[0031] This embodiment designs a processing flow based on generative AI, which can convert 3D sketches into 3D models. The method in this application is built upon a conditional diffusion model. This model was chosen because it can generate diverse and structurally coherent outputs from fuzzy or incomplete sketch inputs; this type of input is common in collaborative brainstorming scenarios. The implementation process of this application's method consists of three core stages: 3D sketch voxelization, sketch voxel processing, and content generation. Specifically, as follows... Figure 2 As shown, the method for converting a 3D sketch into a 3D model includes the following steps S10 to S30, which correspond to the three core stages mentioned above.

[0032] S10: Systematize the acquired 3D sketches to obtain binary voxel sketches with a unified data format.

[0033] Step S10 converts the user's hand-drawn 3D sketch into a regularized voxel representation, eliminating issues such as scale differences, sampling discreteness, and stroke discontinuities in the original sketch, thus providing a standardized input format for subsequent feature extraction. Specifically, by defining the data structure of strokes, geometric normalization, voxel mesh mapping, and stroke interpolation filling, the sketch retains a rough topological and structural outline in the 3D voxel space.

[0034] In some embodiments, such as Figure 3 As shown, step S10 specifically includes the following sub-steps S101-S106.

[0035] S101: Define the input representation for the input 3D sketch.

[0036] The input 3D sketch consists of a set of strokes, each stroke representing a sequence of 3D points arranged in the order they were drawn, specifically as follows: in, The number of strokes. For the first The number of sampling points in a single stroke. Representing three-dimensional real space, For the first Strokes The Middle The three-dimensional coordinates of each sampling point.

[0037] S102: Perform geometric normalization on the 3D sketch.

[0038] First, calculate the maximum distances along the X, Y, and Z axes of the stroke. The calculation formula is: in, and These represent the maximum and minimum values ​​of all stroke points on the X-axis. and These represent the maximum and minimum values ​​of all stroke points on the Y-axis. and These represent the maximum and minimum values ​​of all stroke points on the X-axis. It is a function for maximizing the value.

[0039] Then calculate the center of the sketch. And by the longest side, i.e., the maximum distance All 3D points are uniformly scaled to within a unit cube, and multiplied by a scaling factor of 0.8 to avoid boundary clipping and retain boundary margins. The normalized 3D point coordinates are specifically represented as follows: in, For the original three-dimensional points, These are the normalized three-dimensional points.

[0040] S103: Construct a regular 3D voxel mesh.

[0041] Build resolution A regular three-dimensional voxel mesh, in this embodiment The value is set to 32. Initially, all voxel values ​​in the voxel mesh are set to 0. Specifically: in, It is a voxel grid, where {0,1} indicates that the voxel value is only 0 or 1.

[0042] S104: Maps the original 3D points to a voxel mesh.

[0043] For each normalized original 3D point The corresponding voxel index is calculated based on its normalized coordinates, specifically as follows: in, For voxel indexing, This indicates the floor function. , and This represents the normalized coordinates of a three-dimensional point.

[0044] If the calculated voxel index All meet That is, indexing in the voxel grid The voxel value is set to 1.

[0045] S105: Linear interpolation processing that preserves stroke continuity.

[0046] During the voxel mesh construction process, in order to maintain the continuity of strokes, any pair of adjacent points in the stroke ( Perform linear interpolation.

[0047] First, calculate the Euclidean distance between adjacent pairs of points. The formula is as follows: in, The distance is Euclidean.

[0048] Then perform linear interpolation with a fixed step size of 1. The coordinates of the interpolation points are specifically represented as follows: in, These are interpolation parameters. These are the coordinates of the interpolation point.

[0049] Map all interpolated 3D points to a voxel mesh using the same voxel indexing method as the original 3D points. If the index satisfies... If so, the voxel value of the corresponding index will be set to 1.

[0050] S106: Output a binary voxel sketch.

[0051] After completing the voxelization process for all original and interpolation points, the final output is a binary voxel sketch, specifically represented as follows: in, This is a binary voxel sketch.

[0052] Through the above sub-steps, S10 regularizes the messy 3D strokes into a well-structured binary voxel mesh, providing a unified geometric benchmark for subsequent condition extraction.

[0053] S20: Perform multi-view projection and feature encoding on the binary voxel sketch to obtain conditional latent vectors that encode the geometric features of the sketch.

[0054] The goal of step S20 is to convert the discrete voxel sketch into a compact semantic representation that can be used for conditional diffusion generation. This embodiment employs a scheme of multi-view projection, CLIP encoding, and feature fusion to extract geometric information from the sketch from different perspectives.

[0055] In some embodiments, such as Figure 4 As shown, step S20 specifically includes the following sub-steps S201-S203.

[0056] S201: Multi-view projection.

[0057] The binary voxel sketch Project onto multiple 2D views to generate a binary projected image for each viewpoint.

[0058] Because of direct 3D encoding of voxels is computationally expensive, and 2D visual encoders have mature pre-trained representation capabilities. Therefore, in this embodiment, 3D voxels are first projected onto multiple 2D views. This embodiment selects three viewing directions evenly distributed around the vertical axis, spaced approximately 120° apart, corresponding to the left front view, right front view, and rear side view, respectively. This setup can better cover the front, side, and rear features of the model structure, avoiding occlusion ambiguity caused by single-view conditions.

[0059] Given a voxel grid Define the corresponding camera extrinsic matrix for each viewpoint. ,in In this embodiment, the coordinates of the occupies the voxel center are first transformed to the camera coordinate system, and then orthogonal projection is performed. Since the input is essentially a wireframe sketch rather than a solid block, this embodiment uses binary occupies the voxel instead of depth rendering to highlight the structural outline.

[0060] Specifically, a projected image is generated for each viewpoint: in, and These are the height and width of the image, respectively. This embodiment serves as a viewpoint index. m Choose 1, 2, 3. .

[0061] S202: CLIP image coding for feature extraction.

[0062] Each projected image Feed into the pre-trained CLIP image encoder Extracting high-dimensional semantic representations : in, For the first m Feature vectors from each perspective This represents a 1024-dimensional real vector space.

[0063] Therefore, for the three perspectives, this embodiment can obtain a feature set. : in, These represent high-dimensional semantic representations or view features of the left front view, right front view, and rear view, respectively.

[0064] S203: Multi-view feature fusion based on self-attention.

[0065] This embodiment uses a self-attention-based feature fusion module to adaptively aggregate the features of three views, specifically including the following steps S2031-S2035.

[0066] S2031: Stack the three view features into a matrix .

[0067] S2032: Constructing a query matrix using a linear mapping Key matrix Sum matrix Specifically, it is expressed as: in, The weight matrix is ​​a learnable matrix. For the real number space, For the dimension of attention head.

[0068] S2033: Calculate attention weights A and the characteristics after fusion Attention weights A The calculation formula is: in, For normalized exponential functions, This represents the product of the query matrix and the transpose of the key matrix. This is the scaling factor.

[0069] Features after fusion The calculation formula is: .in, , , and These represent the first fusion feature, the second fusion feature, and the third fusion feature, respectively.

[0070] S2034: Calculate the global condition vector based on the fused features.

[0071] Preserve the independent representation of each perspective after the interaction, and then concatenate the three along the channel dimension: in This indicates a vector concatenation operation.

[0072] Finally, the concatenated features are input into a multilayer perceptron (MLP) for nonlinear fusion to obtain a single global conditional vector: in, This is the global condition vector.

[0073] S2035: Perform final normalization on the global condition vector to obtain the individual conditional latent vector. It encodes the multi-view geometric features of the user's sketch, which are used to guide subsequent 3D generation.

[0074] Through the above sub-steps, S20 transforms the 3D sketch voxel into a compact 1024-dimensional conditional latent vector, which carries the global and local geometric information of the sketch and can effectively guide the subsequent conditional diffusion generation.

[0075] S30: Based on conditional latent vectors, a 3D model is generated using a conditional diffusion model, and the generated 3D model is converted into an editable polygonal mesh.

[0076] The purpose of step S30 is to progressively reconstruct a 3D model that matches the sketch conditions from the noise using a conditional diffusion model, and to convert the discrete voxel representation into a smooth, editable triangular mesh. The conditional diffusion model learns the inverse process from a simple noise distribution to the target voxel distribution, enabling it to adapt to fuzzy and incomplete inputs and supporting multiple candidate generation. Subsequently, signed distance field transformation and the Marching Cubes algorithm are used to eliminate voxel block artifacts, resulting in a high-quality mesh model.

[0077] In some embodiments, such as Figure 5 As shown, step S30 specifically includes the following sub-steps S301-S306.

[0078] S301: Define the generation target and forward diffusion process.

[0079] The target 3D model is represented as a voxel-occupied mesh, which serves as the generation target for the conditional diffusion model. Specifically, this voxel-occupied mesh can be represented as... ;in, , It is a true three-dimensional voxel representation.

[0080] The conditional diffusion model is obtained through training and employs a continuous-time denoised diffusion model. The forward diffusion process involves progressively moving towards the true 3D representation. Adding Gaussian noise to the input, we can obtain the result at any time. t perturbation samples : in, Standard Gaussian noise, This represents a Gaussian distribution with a mean of 0 and a covariance of I. For continuous time variables, Indicates a uniform distribution on the range of 0 to 1. It is a noise scheduling function that monotonically decreases from 1 to 0.

[0081] S302: Construct a conditional denoising network.

[0082] The inverse process of the conditional diffusion model is accomplished by a conditional denoising network. The conditional denoising network uses conditional latent vectors... For global guiding conditions, the input includes a noisy voxel grid (i.e., perturbed samples). ), continuous time variables t and conditional latent vectors This conditional denoising network employs a multi-level encoder-decoder structure based on 3D convolution, achieving multi-scale feature fusion through residual connections and skip connections, and finally outputting surface occupancy predictions.

[0083] In a specific example, the conditional denoising network uses U-Net as the main body, and the input is noisy data. The output is a voxel mesh, showing surface occupancy values. This conditional denoising network uses the standard 3D convolutional U-Net and consists of four layers: The number of channels are 64, 128, 256, and 512 respectively; each layer consists of two... The system consists of convolutional ResNet blocks, with two additional ResNet blocks added at the bottleneck, and finally mapped to the surface occupancy prediction through a convolutional layer.

[0084] S303: Training objective.

[0085] The training objective of this application is not to directly predict noise, but rather to employ... The target for reconstruction-based denoising. The basic form is: That is, to let the network sift through noisy samples Directly restore the original clean sample 1 Building upon this, a self-conditioning mechanism was added. The network then became... ,in It is the result of the model's previous prediction. estimate.

[0086] Specifically, this embodiment employs a self-conditioning mechanism. Reconstructive denoising objectives enable conditional denoising networks The input also includes noisy samples. The last prediction estimate Time variables t and conditional latent vectors The output is the original clean sample. The predicted value, and the corresponding loss are specifically expressed as: in, For loss function, Represents the mathematical expectation. It is the Euclidean norm.

[0087] S304: Generate a 3D model during the inference phase.

[0088] In the reasoning phase, at 32 3 Initialized with Gaussian noise on a voxel mesh, in the conditional latent vector Guided by the pre-trained conditional denoising network, a progressive denoising sampling strategy (such as DDPM sampling strategy) is used to gradually obtain surface occupancy predictions. Voxels with predicted values ​​greater than a preset threshold (such as 0.5) are retained to obtain a three-dimensional model, which is represented as a voxel occupancy grid.

[0089] In addition, multiple candidate 3D models can be obtained by repeatedly sampling with different random seeds to meet the diverse design exploration needs in collaborative design scenarios.

[0090] S305: Voxel model converted to signed distance field.

[0091] To address the surface roughness issue caused by the stepped voxel boundaries, this embodiment calculates the Euclidean distance transformation to obtain the distances from the voxel's interior and exterior to the nearest surface boundary. By subtracting the exterior distance from the interior distance, we construct a signed distance field (SDF).

[0092] Specifically, first calculate the distance from a point inside the voxel to the nearest boundary. Distance from external point to nearest boundary Then construct a signed distance field Specifically, it is expressed as: in, This indicates that the point is located inside the object. Indicates that it is located on the outside. Corresponding to the surface of the object.

[0093] S306: Extract editable polygon mesh.

[0094] Signed range fields are extracted using the Marching Cubes algorithm. The zero-level isosurface is used to obtain an editable triangular mesh. Specifically, it is expressed as: in, This indicates the operation of the Marching Cubes algorithm in extracting zero-level isosurfaces. This is the final editable polygon mesh output. It can be directly used in downstream 3D rendering, editing, or printing applications.

[0095] Through steps S10 to S30 described above, this embodiment of the application achieves end-to-end automatic conversion from 3D sketches to editable 3D models. This method fully utilizes the generative capabilities of conditional diffusion models and the semantic representation capabilities of multi-view feature fusion, enabling the generation of structurally coherent and detail-rich candidate models even when sketches are blurry or incomplete. It also supports iterative optimization, significantly improving the efficiency and flexibility of 3D content creation.

[0096] This application also provides an apparatus for converting a 3D sketch into a 3D model, used to implement the methods described in any of the above embodiments, such as... Figure 6 As shown, the device includes: The sketch voxelization module 601 is configured to perform systematic processing on the acquired 3D sketches to obtain binary voxel sketches with a unified data format. The sketch voxel processing module 602 is configured to perform multi-view projection and feature encoding on the binary voxel sketch to obtain a conditional latent vector encoded with the sketch's geometric features. The 3D model generation module 603 is configured to generate a 3D model based on the conditional latent vector using a conditional diffusion model, and convert the generated 3D model into an editable polygonal mesh.

[0097] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0098] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0100] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0101] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the technical solution of the method for converting a three-dimensional sketch into a three-dimensional model as described above.

[0102] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the method for converting three-dimensional sketches to three-dimensional models described in the above embodiments.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0104] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0105] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0106] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0107] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0108] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0109] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0110] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0111] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0112] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method of three-dimensional sketch-to-three-dimensional model, characterized by, The method includes: The acquired 3D sketches are systematically processed to obtain binary voxel sketches with a unified data format; The binary voxel sketch is subjected to multi-view projection and feature encoding to obtain a conditional latent vector encoded with the sketch's geometric features; Based on the conditional latent vector, a 3D model is generated using a conditional diffusion model, and the generated 3D model is converted into an editable polygonal mesh.

2. The method according to claim 1, characterized in that, The acquired 3D sketches are systematically processed to obtain binary voxel sketches with a unified data format, specifically including: The acquired 3D sketch is represented as a sequence of 3D points. : in, The number of strokes. For the first The number of sampling points in a single stroke. Representing three-dimensional real space, For the first Strokes The Middle The three-dimensional coordinates of each sampling point; Calculate the maximum distance along the X, Y, and Z axes of the stroke. The calculation formula is: in, and These represent the maximum and minimum values ​​of all stroke points on the X-axis. and These represent the maximum and minimum values ​​of all stroke points on the Y-axis. and These represent the maximum and minimum values ​​of all stroke points on the X-axis. It is a function for maximizing the value; Based on the maximum distance The normalized 3D point coordinates are calculated using the following formula: in, For the original three-dimensional points, For normalized three-dimensional points, Center of the sketch; Construct a regular 3D voxel mesh, represented as ,in The resolution of the regular 3D voxel mesh is {0,1}, which means that the voxel values ​​are only 0 or 1. Initially, all voxel values ​​are set to 0. For a normalized 3D point, the corresponding voxel index is calculated using the following formula. in, For voxel indexing, This indicates the floor function. , and Represents the normalized three-dimensional point coordinates; If the calculated voxel index falls within the valid range of the regular 3D voxel grid, then the voxel grid indexed as... The voxel value is set to 1; During the voxel mesh construction process, for any adjacent points in a stroke... Linear interpolation is performed to calculate the Euclidean distance between adjacent point pairs, and linear interpolation is carried out with a fixed step size to obtain the coordinates of the interpolated points as follows: in, These are interpolation parameters. These are the coordinates of the interpolation points; Map all the interpolated 3D points onto a regular 3D voxel grid to obtain a binary voxel sketch.

3. The method according to claim 2, characterized in that, The binary voxel sketch is subjected to multi-view projection and feature encoding to obtain a conditional latent vector encoding the sketch's geometric features, specifically including: The binary voxel sketch is projected onto multiple two-dimensional views to generate a binary projected image for each viewpoint. , and These are the height and width of the image, respectively; Each projected image The image is fed into a pre-trained CLIP image encoder to extract high-dimensional semantic representations. , serving as view features from various perspectives; A self-attention-based feature fusion module is used to adaptively aggregate view features from each perspective to obtain conditional latent vectors.

4. The method according to claim 3, characterized in that, A self-attention-based feature fusion module is used to adaptively aggregate view features from various perspectives to obtain conditional latent vectors, specifically including: Stack the view features from each perspective into a matrix. X ; Based on the matrix X Constructing a query matrix through linear mapping Key matrix Sum matrix : in, The weight matrix is ​​a learnable matrix. For the real number space, The dimension of the attention head; Calculate attention weights The calculation formula is: in, For normalized exponential functions, This represents the product of the query matrix and the transpose of the key matrix. This is the scaling factor; Based on attention weights , obtain fusion features ;in , , and These represent the first fusion feature, the second fusion feature, and the third fusion feature, respectively. The first, second, and third fusion features are concatenated along the channel dimension: in This represents a vector concatenation operation. Indicates the features after splicing The spliced ​​features The input is processed by a multilayer perceptron for nonlinear fusion, resulting in a single global condition vector. ; For the global condition vector The final normalization yields conditional latent vectors that encode the geometric features of the sketch multi-view. .

5. The method according to claim 1, characterized in that, Based on the conditional latent vector, a 3D model is generated using a conditional diffusion model, specifically including: The target 3D model is represented as a voxel-occupied mesh, which serves as the generation target for the conditional diffusion model; The conditional diffusion model is obtained through training and employs a continuous-time denoised diffusion model. The forward diffusion process is a gradual diffusion towards the true 3D representation. Adding Gaussian noise to the input, we can obtain the result at any time. perturbation samples : in, Standard Gaussian noise, This represents a Gaussian distribution with a mean of 0 and a covariance of I. For continuous time variables, Indicates a uniform distribution on the range of 0 to 1. It is a noise scheduling function that monotonically decreases from 1 to 0; The inverse process of the conditional diffusion model is completed by a conditional denoising network, which uses the conditional latent vector. As a global guiding condition, its input includes perturbation samples. Continuous time variables and the conditional latent vector It adopts a multi-level encoder-decoder structure based on 3D convolution, and achieves multi-scale feature fusion through residual connections and skip connections, outputting surface occupancy prediction; The inference phase initializes with Gaussian noise on a voxel grid of a set resolution, in the conditional latent vector. Guided by the pre-trained conditional denoising network, a progressive denoising sampling strategy is used to gradually obtain the surface occupancy prediction. Voxels with predicted values ​​greater than a preset threshold are retained to obtain a three-dimensional model.

6. The method according to claim 5, characterized in that, The conditional diffusion model uses the following loss function during training: in, For loss function, Represents the mathematical expectation. For the Euclidean norm, The model's previous prediction estimate.

7. The method according to claim 1, characterized in that, The generated 3D model is converted into an editable polygonal mesh, specifically including: Calculate the distance from each interior point in the 3D model to the nearest boundary. Distance from external point to nearest boundary Construct a signed distance field : in, This indicates that the point is located inside the object. Indicates that it is located on the outside. Corresponding object surface; Extract the signed distance field The zero-level isosurface is used to obtain an editable polygonal mesh.

8. A device for converting a three-dimensional sketch into a three-dimensional model, characterized in that, The device includes: The sketch voxelization module is configured to systematically process the acquired 3D sketches to obtain binary voxel sketches with a unified data format. The sketch voxel processing module is configured to perform multi-view projection and feature encoding on the binary voxel sketch to obtain a conditional latent vector encoded with the sketch's geometric features. The 3D model generation module is configured to generate a 3D model based on the conditional latent vector using a conditional diffusion model, and convert the generated 3D model into an editable polygonal mesh.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method of converting a three-dimensional sketch into a three-dimensional model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method of converting a three-dimensional sketch into a three-dimensional model as described in any one of claims 1-7.

Images