Method and system for three-dimensional model generation and assembly based on hand-drawn sketches

Abstract

The application provides a hand-drawing sketch-based three-dimensional model generation and assembly method and system, which comprises: a three-dimensional model data set constituting an overall form, being split into three-dimensional models of components, three-dimensional model data sets of the components, and rendering to obtain a sketch; calculating spatial position parameters of component assembly, independently training a directed distance function neural network for each component to realize mapping from a component form sketch to a component three-dimensional model; assembling the three-dimensional models of the components according to the estimated spatial position parameters, and generating a three-dimensional model of the overall form composed of a single surface and having a watertight characteristic by using a Poisson reconstruction algorithm, as a result of the hand-drawing sketch-based three-dimensional model generation and assembly. The application splits a complex three-dimensional model into components with simpler forms and easier representation, improves the product form design efficiency, and ensures the accuracy of scheme verification.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, specifically to a method and system for generating and assembling 3D models based on hand-drawn sketches. Background Technology

[0002] In recent years, product design has been developing towards intelligence and automation, resulting in significant improvements in the efficiency of conceptual design creation, the accuracy of design evaluation, and the quality of generated solutions. In this trend, the application of artificial intelligence technologies, such as image analysis and 3D model reconstruction, has played a crucial role. For example, using image content generation technology to quickly generate a large number of prototype images of chair products that combine general characteristics and typical styles can significantly reduce the creative burden on designers and provide them with sources of inspiration. Using 3D model reconstruction technology to generate corresponding 3D models based on existing product images can reduce the workload of manual modeling and improve the efficiency of the design process.

[0003] However, the application of artificial intelligence (AI) technology in product design currently faces some challenges. For example, improving the performance of generative models requires larger models, more data, and greater computing power, which limits their effectiveness. Therefore, it is necessary to apply general AI technologies with specific improvements tailored to specific domain requirements to achieve results that meet those requirements. Regarding the generation of 3D model designs based on hand-drawn sketches, the representation of 3D models requires significantly more data than 2D images, and the increase in generative model size required to improve the accuracy of 3D models is more pronounced than in the 2D case. This results in low accuracy and poor representation of detailed features in current 3D model generation based on hand-drawn sketches.

[0004] Patent document CN108109190A (application number: CN201711349153.7) discloses a method for generating cartoon animation that integrates hand-drawn style, including the following steps: (1) First, construct a three-dimensional mesh and skeleton subspace model of the character using hand-drawn drawings; (2) Then, combine pose reconstruction and differential domain mesh deformation algorithms to create a set of viewpoint-related models that match the multi-viewpoint hand-drawn drawings; (3) Finally, integrate the hand-drawn style into the input motion sequence through motion annotation and heuristic generalization algorithms, so that the generated animation retains the original features of the motion sequence. However, this patent cannot solve the existing technical problems. Summary of the Invention

[0005] In view of the deficiencies in the existing technology, the purpose of this invention is to provide a method and system for generating and assembling three-dimensional models based on hand-drawn sketches.

[0006] The method for generating and assembling a 3D model based on a hand-drawn sketch provided by the present invention includes:

[0007] Step 1: Obtain 3D model samples belonging to the target product type from the open ShapeNet database to form a 3D model dataset of the overall shape, and use the line drawing rendering tool of the Blender platform to render sketches through perspective projection to form a sketch dataset of the overall shape.

[0008] Step 2: Decompose the overall 3D model sample into 3D models of components, forming a 3D model dataset for each component. Then, use the line art rendering tool of the Blender platform to render sketches of each component through perspective projection, forming an independent morphological sketch dataset for each component.

[0009] Step 3: Use the Blender platform's line art rendering tool to render a sketch of the overall 3D model through parallel projection, and calculate the spatial position parameters of the component assembly from it to form a dataset of spatial position parameters for each component.

[0010] Step 4: Using the 3D model dataset of each component, train a directed distance function neural network independently for each component to realize the mapping from the component shape sketch to the component 3D model. Using the spatial position parameter dataset of the component, train a deep neural network for assembly position parameter estimation to realize the mapping from the overall shape sketch to the spatial position parameters of each component.

[0011] Step 5: During testing, the hand-drawn overall shape sketch is input into the trained deep neural network to obtain the estimated results of the assembly position parameters of each component. The hand-drawn shape sketch of each component is used as an input to the trained directed distance function neural network of each component. The moving cube algorithm is used to process the neural network to obtain the three-dimensional model of each component in the form of triangular facets.

[0012] Step 6: Assemble the 3D models of each component generated during the test according to the estimated spatial position parameters. Use the Poisson reconstruction algorithm to generate a 3D model of the overall shape consisting of a single surface and having watertight properties. This model serves as the result of the 3D model generation and assembly based on the hand-drawn sketches.

[0013] Preferably, step 1 includes: the three-dimensional model dataset for constructing the overall shape, wherein each three-dimensional model sample is represented in the form of triangular facets and stored in an obj format file; and the sketch dataset for constructing the overall shape, wherein during rendering, the camera under the Blender platform faces a predefined fixed direction P1 that reflects the characteristics of the product object.

[0014] Preferably, step 3 includes: calculating the spatial position parameters of the component assembly; when rendering the sketch with parallel projection, the camera under the Blender platform faces a predefined fixed direction P2 that reflects the relative position of each component; using the Hough circle detection algorithm, detecting circles in the sketch rendered by parallel projection that conform to the predefined radius range of each component, and matching each component type according to the relative position and radius of the circle in the sketch image; finally, taking the lower left corner of the sketch image as the origin, uniformly normalizing the pixel coordinates of the center of the circle of each component to the range of [0,1] along the x and y axes, respectively, as the spatial position parameters of each component.

[0015] Preferably, step 4 includes: training the directed distance function neural network, before training, randomly sampling 9000 spatial points of the 3D model of each sample, and calculating the directed distance function value of each sampling point, using the morphological sketch of each sample, the spatial coordinates of each sampling point, and the directed distance function value of each sampling point as training data; the training iteration number is 2000, in each training generation, every 8 training samples are taken as the same batch, and the spatial coordinates of the 9000 sampling points of each sample and the morphological sketch of that sample are used as the input of the directed distance function neural network, the training loss is the average absolute error between the directed distance function value output by the network and the directed distance function value obtained by actual sampling, the initial value of the weight parameters of the fully connected layer is set to 0, the Adam optimizer is used to update the network weights, the learning rate is 0.00005, and the network is built using the PyTorch deep learning framework.

[0016] Preferably, step 4 includes: training a deep neural network for assembly position parameter estimation, wherein the training data for network input is a sketch of the overall shape rendered by perspective projection of the camera facing the P1 direction, and the training data for network output is the normalized result of the spatial position parameters of each component. In each training generation, every 32 training samples are taken as the same batch, the network weights are initialized using the Xavier method, and the network is built using the PyTorch deep learning framework.

[0017] The 3D model generation and assembly system based on hand-drawn sketches provided by the present invention includes:

[0018] Module M1: Obtains 3D model samples belonging to the target product type from the open ShapeNet database, forming a 3D model dataset of the overall shape, and uses the line drawing rendering tool of the Blender platform to render sketches through perspective projection, forming a sketch dataset of the overall shape.

[0019] Module M2: The overall 3D model sample is broken down into 3D models of components, forming a 3D model dataset for each component. The 3D model of each component is then rendered using the line art rendering tool of the Blender platform through perspective projection to obtain sketches, forming an independent morphological sketch dataset for each component.

[0020] Module M3: The overall 3D model is rendered using the Blender platform's line drawing rendering tool through parallel projection to obtain a sketch. From this sketch, the spatial position parameters of the component assembly are calculated, forming a dataset of spatial position parameters for each component.

[0021] Module M4: Using the 3D model dataset of each component, a directed distance function neural network is independently trained for each component to achieve the mapping from component shape sketch to component 3D model. Using the spatial position parameter dataset of the component, a deep neural network for assembly position parameter estimation is trained to achieve the mapping from the overall shape sketch to the spatial position parameters of each component.

[0022] Module M5: During testing, the hand-drawn overall shape sketch is input into the trained deep neural network to obtain the estimated results of the assembly position parameters of each component. The hand-drawn shape sketch of each component is used as an input to the trained directed distance function neural network of each component. The moving cube algorithm is used to process the neural network to obtain a three-dimensional model of each component in the form of triangular facets.

[0023] Module M6: Assemble the 3D models of each component generated during testing according to the estimated spatial position parameters. Use the Poisson reconstruction algorithm to generate a 3D model of the overall shape consisting of a single surface and having watertight properties, as the result of the 3D model generation and assembly based on the hand-drawn sketches.

[0024] Preferably, module M1 includes: the three-dimensional model dataset for constructing the overall shape, wherein each three-dimensional model sample is represented in the form of triangular facets and stored in an obj format file; and the sketch dataset for constructing the overall shape, wherein during rendering, the camera under the Blender platform faces a predefined fixed direction P1 that reflects the characteristics of the product object.

[0025] Preferably, the module M3 includes: calculating the spatial position parameters of the component assembly; when rendering the sketch with parallel projection, the camera under the Blender platform faces a predefined fixed direction P2 that reflects the relative position of each component; using the Hough circle detection algorithm, detecting circles in the sketch rendered by parallel projection that conform to the predefined radius range of each component, and matching each component type according to the relative position and radius of the circle in the sketch image; finally, taking the lower left corner of the sketch image as the origin, uniformly normalizing the pixel coordinates of the center of the circle of each component to the range of [0,1] along the x and y axes, respectively, as the spatial position parameters of each component.

[0026] Preferably, module M4 includes: the training directed distance function neural network, wherein before training, 9000 spatial points are randomly sampled from the 3D model of each sample, and the directed distance function value of each sampling point is calculated. The morphological sketch of each sample, the spatial coordinates of each sampling point, and the directed distance function value of each sampling point are used together as training data. The number of training iterations is 2000. In each training iteration, every 8 training samples are taken as the same batch. The spatial coordinates of the 9000 sampling points of each sample and the morphological sketch of the sample are used together as the input of the directed distance function neural network. The training loss is the average absolute error between the directed distance function value output by the network and the directed distance function value obtained by actual sampling. The initial value of the weight parameters of the fully connected layer is set to 0. The Adam optimizer is used to update the network weights. The learning rate is 0.00005. The network is built using the PyTorch deep learning framework.

[0027] Preferably, the module M4 includes: the deep neural network used for training assembly position parameter estimation, wherein the training data corresponding to the network input is the overall shape sketch rendered by perspective projection of the camera facing the P1 direction, and the training data corresponding to the network output is the normalized result of the spatial position parameters of each component. In each training generation, every 32 training samples are taken as the same batch, the network weights are initialized using the Xavier method, and the network is built using the PyTorch deep learning framework.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The present invention breaks down the overall form of the product object into several components. The components have simpler forms and can achieve better detail feature representation under lower precision conditions. The three-dimensional models of the components are generated independently and then merged as the final three-dimensional model generation result.

[0030] (2) The present invention represents the three-dimensional model of each component based on the finite distance field function. Compared with the traditional voxel or triangular mesh representation method, it further reduces the amount of data and the scale of the generative model, improves the generation effect of the three-dimensional model based on hand-drawn sketches, and avoids significantly increasing the computational cost. Attached Figure Description

[0031] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0032] Figure 1 This is a flowchart of a method for generating and assembling complex 3D models based on hand-drawn sketches, according to the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0034] Example 1:

[0035] like Figure 1This invention provides a method for generating and assembling 3D models based on hand-drawn sketches, comprising: Step 1: Obtaining 3D model samples belonging to the target product type from the open ShapeNet database to form a 3D model dataset of the overall form, and using the line art rendering tool of the Blender platform to render sketches through perspective projection to form a sketch dataset of the overall form; Step 2: Decomposing the 3D model samples of the overall form into 3D models composed of components to form a 3D model dataset of each component, and using the line art rendering tool of the Blender platform to render sketches through perspective projection to form an independent form sketch dataset of each component; Step 3: Using the line art rendering tool of the Blender platform to render sketches through parallel projection to form a sketch of the overall form 3D model, calculating the spatial position parameters of the component assembly from it, and forming a spatial position parameter dataset of each component; Step 4: Using each component... The three-dimensional model dataset is used to independently train a directed distance function neural network for each component, realizing the mapping from component shape sketches to component three-dimensional models. Using the component spatial position parameter dataset, a deep neural network for estimating assembly position parameters is trained, realizing the mapping from the overall shape sketch to the spatial position parameters of each component. Step 5: During testing, the hand-drawn overall shape sketch is input into the trained deep neural network to obtain the estimated assembly position parameters of each component. The hand-drawn shape sketches of each component are used as an input to the trained directed distance function neural network of each component. The moving cube algorithm is used to process the neural network to obtain the three-dimensional model of each component in the form of triangular facets. Step 6: The three-dimensional models of each component generated during testing are assembled according to the estimated spatial position parameters. The Poisson reconstruction algorithm is used to generate a three-dimensional model of the overall shape composed of a single surface and with watertight characteristics, which is the result of the three-dimensional model generation and assembly based on the hand-drawn sketches.

[0036] Step 1 includes: the three-dimensional model dataset for constructing the overall shape, wherein each three-dimensional model sample is represented in the form of triangular facets and stored in an obj format file; and the sketch dataset for constructing the overall shape, wherein during rendering, the camera under the Blender platform faces a predefined fixed direction P1 that reflects the characteristics of the product object.

[0037] Step 3 includes: calculating the spatial position parameters of the component assembly. When rendering the sketch with parallel projection, the camera on the Blender platform faces a predefined fixed direction P2 that reflects the relative positions of each component; using the Hough circle detection algorithm, circles in the sketch rendered by parallel projection that conform to the predefined radius range of each component are detected, and the circles are matched to each component type according to their relative position and radius size in the sketch image. Finally, with the lower left corner of the sketch image as the origin, the pixel coordinates of the center of the circle of each component are uniformly normalized to the range [0,1] along the x and y axes, respectively, as the spatial position parameters of each component.

[0038] Step 4 includes: training the directed distance function neural network. Before training, 9000 spatial points are randomly sampled from the 3D model of each sample, and the directed distance function value of each sampling point is calculated. The morphological sketch of each sample, the spatial coordinates of each sampling point, and the directed distance function value of each sampling point are used as training data. The training iterations are 2000. In each training iteration, every 8 training samples are taken as the same batch. The spatial coordinates of the 9000 sampling points of each sample and the morphological sketch of that sample are used as the input of the directed distance function neural network. The training loss is the average absolute error between the directed distance function value output by the network and the directed distance function value obtained by actual sampling. The initial value of the weight parameters of the fully connected layer is set to 0. The Adam optimizer is used to update the network weights. The learning rate is 0.00005. The network is built using the PyTorch deep learning framework.

[0039] Step 4 includes: training a deep neural network for assembly position parameter estimation. The training data for the network input is a sketch of the overall shape rendered by perspective projection of the camera facing the P1 direction. The training data for the network output is the normalized result of the spatial position parameters of each component. In each training generation, every 32 training samples are taken as the same batch. The network weights are initialized using the Xavier method and built using the PyTorch deep learning framework.

[0040] This invention provides a 3D model generation and assembly system based on hand-drawn sketches, comprising: Module M1: Obtaining 3D model samples belonging to the target product type from the open ShapeNet database to form a 3D model dataset of the overall form, and using the line art rendering tool of the Blender platform to render sketches through perspective projection to form a sketch dataset of the overall form; Module M2: Decomposing the 3D model samples of the overall form into 3D models composed of components to form a 3D model dataset of each component, and using the line art rendering tool of the Blender platform to render sketches through perspective projection to form an independent form sketch dataset of each component; Module M3: Using the line art rendering tool of the Blender platform to render sketches through parallel projection to form a sketch of the overall form 3D model, calculating the spatial position parameters of the component assembly from it, and forming a spatial position parameter dataset of each component; Module M4: Utilizing each part... The system uses a dataset of 3D model data for components. Directed distance function neural networks are trained independently for each component to map from component morphological sketches to 3D models. A deep neural network for estimating assembly position parameters is trained using the component's spatial position parameter dataset to map from the overall morphological sketch to the spatial position parameters of each component. Module M5: During testing, the hand-drawn overall morphological sketch is input into the trained deep neural network to obtain the estimated assembly position parameters of each component. The hand-drawn morphological sketches of each component are used as input to the trained directed distance function neural networks for each component. The moving cube algorithm is used to process the neural network, resulting in a 3D model represented by triangular facets for each component. Module M6: The 3D models of each component generated during testing are assembled according to the estimated spatial position parameters. A Poisson reconstruction algorithm is used to generate a 3D model of the overall morphology consisting of a single surface and possessing watertight properties, serving as the result of the 3D model generation and assembly based on the hand-drawn sketches.

[0041] The module M1 includes: the three-dimensional model dataset for constructing the overall shape, wherein each three-dimensional model sample is represented in the form of triangular facets and stored in an obj format file; and the sketch dataset for constructing the overall shape, wherein during rendering, the camera under the Blender platform faces a predefined fixed direction P1 that reflects the characteristics of the product object.

[0042] The module M3 includes: calculating the spatial position parameters of the component assembly; when rendering the sketch with parallel projection, the camera under the Blender platform faces a predefined fixed direction P2 that reflects the relative position of each component; using the Hough circle detection algorithm, detecting circles in the sketch rendered by parallel projection that conform to the predefined radius range of each component, and matching each component type according to the relative position and radius of the circle in the sketch image; finally, taking the lower left corner of the sketch image as the origin, uniformly normalizing the pixel coordinates of the center of the circle of each component to the range of [0,1] along the x and y axes, respectively, as the spatial position parameters of each component.

[0043] The module M4 includes: the training directed distance function neural network, in which 9000 spatial points are randomly sampled from the 3D model of each sample before training, and the directed distance function value of each sampling point is calculated. The morphological sketch of each sample, the spatial coordinates of each sampling point, and the directed distance function value of each sampling point are used as training data. The training iterations are 2000. In each training iteration, every 8 training samples are taken as the same batch. The spatial coordinates of the 9000 sampling points of each sample and the morphological sketch of that sample are used as the input of the directed distance function neural network. The training loss is the average absolute error between the directed distance function value output by the network and the directed distance function value obtained by actual sampling. The initial value of the weight parameters of the fully connected layer is set to 0. The Adam optimizer is used to update the network weights. The learning rate is 0.00005. The network is built using the PyTorch deep learning framework.

[0044] The module M4 includes: the deep neural network used for training assembly position parameter estimation, the training data corresponding to the network input is the overall shape sketch rendered by perspective projection of the camera facing the P1 direction, and the training data corresponding to the network output is the normalized result of the spatial position parameters of each component. In each training generation, every 32 training samples are taken as the same batch, the network weights are initialized using the Xavier method, and the network is built using the PyTorch deep learning framework.

[0045] The mesh nodes of the 3D model samples obtained from ShapeNet have x / y / z coordinate values ​​within the range of [-1, 1]. Each 3D model sample should include mesh node coordinates, triangle vertex information, node normals, and triangle normals, ignoring texture and color information. When rendering the overall 3D model, the line thickness is set to a fixed value of 0.5.

[0046] The Hough circle detection algorithm is implemented using the OpenCV library. During detection, for multiple circles with a center-to-center distance of less than 20 pixels, the algorithm selects the circle with the largest radius. After running the Hough circle detection algorithm, samples with more or fewer detected circles than the predefined number of parts are discarded, as are samples that cannot be matched one-to-one with each circle according to the predefined radius range to be a part type.

[0047] When training the directed distance function neural network and the deep neural network for assembly position parameter estimation, the pixel values ​​of the input sketch image range from 0 to 255. Before inputting the image into the network, the pixel values ​​are uniformly scaled to [0,1]. The computer vision library OpenCV is used to read and scale the sketch.

[0048] When using the moving cube algorithm to process neural networks, the mesh node and triangular facet information output by the moving cube algorithm is stored in the TriangleMesh class object of the open3d library, and the io module of the open3d library is used to export the generated 3D model as a triangular facet mesh in obj format.

[0049] Example 2:

[0050] The present invention also provides a three-dimensional model generation and assembly system based on hand-drawn sketches. The three-dimensional model generation and assembly system based on hand-drawn sketches can be implemented by executing the process steps of the three-dimensional model generation and assembly method based on hand-drawn sketches. That is, those skilled in the art can understand the three-dimensional model generation and assembly method based on hand-drawn sketches as a preferred embodiment of the three-dimensional model generation and assembly system based on hand-drawn sketches.

[0051] This invention provides a 3D model generation and assembly system based on hand-drawn sketches, including a dataset construction module, an assembly position parameter estimation module, a 3D model generation module for each component, and a complex 3D model assembly module. The dataset construction module includes a 3D model dataset for constructing the overall shape, a dataset of overall shape sketches, rendering line sketches from the overall shape's 3D model samples using parallel projection views, constructing spatial position parameter datasets for each component, splitting the overall shape's 3D model dataset into independent 3D model datasets for each component, and rendering morphological sketch datasets for each component using perspective projection views. The assembly position parameter estimation module trains a deep neural network using the constructed datasets, taking the hand-drawn sketches of the overall shape as input and the spatial position parameters of each component as output. The 3D model generation module for each component represents the 3D model of each component using a directed distance field function implemented by a neural network and visualizes it using a moving cube algorithm. The complex 3D model assembly module spatially combines the generated 3D models of each component and obtains the final 3D model of the overall shape with single-surface watertightness using a Poisson reconstruction algorithm.

[0052] The dataset construction module includes a 3D model dataset for the overall form, a sketch dataset for the overall form, a dataset of spatial position parameters for each component, a 3D model dataset for each component, and a sketch dataset for each component. The 3D model dataset for the overall form should consist entirely of samples belonging to the same product category. Each sample should contain the same type and number of components. Each sample is represented by a triangular mesh and stored in .obj file format. The samples are sourced from the open ShapeNet database and have been manually selected. The sketch dataset for the overall form uses the line art rendering tool built into the Blender platform. The 3D model of the overall form is rendered using perspective projection to obtain sketches, and all sketch images are scaled to 224*224 resolution and stored as single-channel grayscale images. The spatial position parameter dataset for each component is constructed using the line art rendering tool built into the Blender platform. A 3D model sample of the overall form is rendered using a parallel projection view to obtain a sketch. The parallel projection directions are parallel to the x, y, and z axes of the absolute spatial coordinate system. Then, the Hough circle detection algorithm is used to detect and locate circles in the sketch that conform to the predefined radius range of each component. Finally, the center position parameters of the circles of each component are calculated as the actual spatial position parameters. The 3D model dataset for each component is constructed by manually dividing the overall 3D model into multiple independent components. Each component is still represented by a triangular mesh and stored separately in .obj file format. The centroid of each component model is moved to the origin of the absolute spatial coordinate system. The morphological sketch dataset for each component is constructed using the line art rendering tool built into the Blender platform. The 3D model of each component is rendered using a perspective projection view to obtain a morphological sketch. All morphological sketch images are scaled to 56*56 resolution and stored as single-channel grayscale images.

[0053] The main body of the assembly position parameter estimation module is a deep neural network, and its structure includes the following sequentially connected components:

[0054] a) Input layer: The input data is a single-channel grayscale image of the overall shape hand-drawn sketch with a resolution of 224*224 and a bit depth of 8. The dimension of the input data is N*224*224*1, where N is the number of hand-drawn sketches.

[0055] b) Two convolutional layers with a kernel size of 5*5 and a stride of 2. The number of kernels in each convolutional layer is (16, 32) and the activation function is the leaky ReLU function.

[0056] c) Batch normalization layer;

[0057] d) Dropout layer, with a dropout probability of 0.1;

[0058] e) There are 3 convolutional layers with a kernel size of 3*3 and a stride of 2. The number of kernels in each convolutional layer is (64, 64, 128) respectively, and the activation function is the ReLU function.

[0059] f) Batch normalization layer;

[0060] g) Dropout layer, with a dropout probability of 0.1;

[0061] h) Flatten layer, which unfolds the N*7*7*128 feature map output by the last convolutional layer into a one-dimensional vector;

[0062] i) Three fully connected layers, each outputting an intermediate vector of size 1*1024, with softmax as the activation function;

[0063] j) Dropout layer, with a dropout probability of 0.1;

[0064] k) batch normalization layer;

[0065] l) Three fully connected layers, each outputting a 1*512 intermediate vector, with softmax as the activation function;

[0066] m) Fully connected layer with tanh activation function, outputting a vector of size 1*M, where M is the number of spatial location parameters.

[0067] The training loss function of the deep neural network of the assembly position parameter estimation module is the sum of the L2 loss between the estimated and true values ​​of the output M spatial position parameters and the L2 regularization term of the network weights. The Adam optimizer is used to update the network weights. The initial learning rate is 0.0001, and the learning rate is reduced to 0.00002 after 100 training generations.

[0068] The 3D model generation module for each component uses a directed distance field function (SDF) implemented by a neural network to represent the 3D model of each component. SDF(*) maps spatial point coordinates to the directed distance from that spatial point to the surface of the 3D model (+ if the spatial point is outside the surface, - otherwise). The directed distance field function neural network is also a generative model, taking the morphological sketches and spatial point coordinates of each component as input and the directed distance field function value of that spatial point as output. The directed distance field function neural network includes the following sequentially connected components:

[0069] a) Input layer: The input data consists of a single-channel grayscale image of a 56*56 resolution part shape sketch and the coordinates of the spatial point x. The image bit depth is 8, the dimension of the image data is N*56*56*1, and the dimension of the coordinate data is N*3, where N is the number of samples.

[0070] b) Two-branch network layers, including a convolutional branch and a bridging branch. The convolutional branch contains two convolutional layers. Its input is a single-channel grayscale image of the part shape sketch. The convolutional kernel size is 3*3, the stride is 2, and the number of convolutional kernels in each convolutional layer is (16, 16) in sequence. The activation function is the ReLU function. The bridging branch directly takes the coordinates of the input spatial point x as the output.

[0071] c) The concatenation layer expands the 14*14*16 feature map output by the convolutional branch into a 1*3136 one-dimensional vector, and then concatenates it with the output vector of the bridging branch to become a 1*3139 one-dimensional vector.

[0072] d) Three fully connected layers, each outputting an intermediate vector of size 1*1024, with softmax as the activation function;

[0073] e) Batch normalization layer;

[0074] f) Dropout layer, with a dropout probability of 0.1;

[0075] g) Three fully connected layers, each outputting a 1*512 intermediate vector, with softmax as the activation function;

[0076] h) batch normalization layer;

[0077] i) Dropout layer, with a dropout probability of 0.1;

[0078] j) Fully connected layer with a scalar output and tanh activation function.

[0079] The 3D model generation module for each component establishes an independent directed distance field function neural network for each component type. During the training phase, the loss function is the average absolute error between the output directed distance function value and the true value. The Adam optimizer is used to update the network weights with a learning rate of 0.00005. During the testing phase, the morphological sketch sample of the component is used as an input, and the trained directed distance field function neural network is processed by the moving cube algorithm to obtain the 3D model of the component in the form of a triangular mesh. The spatial sampling resolution of the moving cube algorithm is 256*256*256, and the isosurface partitioning threshold is 0.

[0080] The complex 3D model assembly module loads the generated 3D models of each component into a newly created 3D model object, moves the centroid of each component's 3D model to the corresponding position according to the estimated spatial position parameters, and then uses the Poisson reconstruction algorithm to generate a single continuous surface with watertight properties that covers each component. As the final result of the complex 3D model generation and assembly based on hand-drawn sketches, it is still represented by a triangular mesh and stored in obj file format.

[0081] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0082] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for generating and assembling 3D models based on hand-drawn sketches, characterized in that, include: Step 1: Obtain 3D model samples belonging to the target product type from the open ShapeNet database to form a 3D model dataset of the overall shape, and use the line drawing rendering tool of the Blender platform to render sketches through perspective projection to form a sketch dataset of the overall shape. Step 2: Decompose the overall 3D model sample into 3D models of components, forming a 3D model dataset for each component. Then, use the line art rendering tool of the Blender platform to render sketches of each component through perspective projection, forming an independent morphological sketch dataset for each component. Step 3: Use the Blender platform's line art rendering tool to render a sketch of the overall 3D model through parallel projection, and calculate the spatial position parameters of the component assembly from it to form a dataset of spatial position parameters for each component. Step 4: Using the 3D model dataset of each component, train a directed distance function neural network independently for each component to realize the mapping from the component shape sketch to the component 3D model. Using the spatial position parameter dataset of the component, train a deep neural network for assembly position parameter estimation to realize the mapping from the overall shape sketch to the spatial position parameters of each component. Step 5: During testing, the hand-drawn overall shape sketch is input into the trained deep neural network to obtain the estimated results of the assembly position parameters of each component. The hand-drawn shape sketch of each component is used as an input to the trained directed distance function neural network of each component. The moving cube algorithm is used to process the neural network to obtain the three-dimensional model of each component in the form of triangular facets. Step 6: Assemble the 3D models of each component generated during the test according to the estimated spatial position parameters. Use the Poisson reconstruction algorithm to generate a 3D model of the overall shape consisting of a single surface and having watertight properties. This model serves as the result of the 3D model generation and assembly based on the hand-drawn sketches.

2. The method for generating and assembling a 3D model based on hand-drawn sketches according to claim 1, characterized in that, Step 1 includes: constructing a 3D model dataset of the overall shape, wherein each 3D model sample is represented in the form of a triangular facet and stored in an obj format file; constructing a sketch dataset of the overall shape, wherein during rendering, the camera under the Blender platform faces a predefined fixed direction P1 that reflects the characteristics of the product object.

3. The method for generating and assembling a 3D model based on hand-drawn sketches according to claim 1, characterized in that, Step 3 includes: calculating the spatial position parameters of the component assembly; when rendering the sketch with parallel projection, the camera under the Blender platform faces a predefined fixed direction P2 that reflects the relative position of each component; using the Hough circle detection algorithm, detecting circles in the sketch rendered by parallel projection that conform to the predefined radius range of each component, and matching each component type according to the relative position and radius of the circle in the sketch image; finally, taking the lower left corner of the sketch image as the origin, uniformly normalizing the pixel coordinates of the center of the circle of each component to the range of [0,1] along the x and y axes, respectively, as the spatial position parameters of each component.

4. The method for generating and assembling a 3D model based on hand-drawn sketches according to claim 1, characterized in that, Step 4 includes: training the directed distance function neural network. Before training, 9000 spatial points are randomly sampled from the 3D model of each sample, and the directed distance function value of each sampling point is calculated. The morphological sketch of each sample, the spatial coordinates of each sampling point, and the directed distance function value of each sampling point are used as training data. The training iterations are 2000. In each training iteration, every 8 training samples are taken as the same batch. The spatial coordinates of the 9000 sampling points of each sample and the morphological sketch of that sample are used as the input of the directed distance function neural network. The training loss is the average absolute error between the directed distance function value output by the network and the directed distance function value obtained by actual sampling. The initial value of the weight parameters of the fully connected layer is set to 0. The Adam optimizer is used to update the network weights. The learning rate is 0.00005. The network is built using the PyTorch deep learning framework.

5. The method for generating and assembling a 3D model based on hand-drawn sketches according to claim 2, characterized in that, Step 4 includes: training a deep neural network for assembly position parameter estimation. The training data for the network input is a sketch of the overall shape rendered by perspective projection of the camera facing the P1 direction. The training data for the network output is the normalized result of the spatial position parameters of each component. In each training generation, every 32 training samples are taken as the same batch. The network weights are initialized using the Xavier method and built using the PyTorch deep learning framework.

6. A 3D model generation and assembly system based on hand-drawn sketches, characterized in that, include: Module M1: Obtains 3D model samples belonging to the target product type from the open ShapeNet database, forming a 3D model dataset of the overall shape, and uses the line drawing rendering tool of the Blender platform to render sketches through perspective projection, forming a sketch dataset of the overall shape. Module M2: The overall 3D model sample is broken down into 3D models of components, forming a 3D model dataset for each component. The 3D model of each component is then rendered using the line art rendering tool of the Blender platform through perspective projection to obtain sketches, forming an independent morphological sketch dataset for each component. Module M3: The overall 3D model is rendered using the Blender platform's line drawing rendering tool through parallel projection to obtain a sketch. From this sketch, the spatial position parameters of the component assembly are calculated, forming a dataset of spatial position parameters for each component. Module M4: Using the 3D model dataset of each component, a directed distance function neural network is independently trained for each component to achieve the mapping from component shape sketch to component 3D model. Using the spatial position parameter dataset of the component, a deep neural network for assembly position parameter estimation is trained to achieve the mapping from the overall shape sketch to the spatial position parameters of each component. Module M5: During testing, the hand-drawn overall shape sketch is input into the trained deep neural network to obtain the estimated results of the assembly position parameters of each component. The hand-drawn shape sketch of each component is used as an input to the trained directed distance function neural network of each component. The moving cube algorithm is used to process the neural network to obtain a three-dimensional model of each component in the form of triangular facets. Module M6: Assemble the 3D models of each component generated during testing according to the estimated spatial position parameters. Use the Poisson reconstruction algorithm to generate a 3D model of the overall shape consisting of a single surface and having watertight properties, as the result of the 3D model generation and assembly based on the hand-drawn sketches.

7. The 3D model generation and assembly system based on hand-drawn sketches according to claim 6, characterized in that, The module M1 includes: a 3D model dataset for constructing the overall shape, wherein each 3D model sample is represented in the form of triangular facets and stored in an obj format file; and a sketch dataset for constructing the overall shape, wherein during rendering, the camera under the Blender platform faces a predefined fixed direction P1 that reflects the characteristics of the product object.

8. The 3D model generation and assembly system based on hand-drawn sketches according to claim 6, characterized in that, The module M3 includes: calculating the spatial position parameters of the component assembly; when rendering the sketch with parallel projection, the camera under the Blender platform faces a predefined fixed direction P2 that reflects the relative position of each component; using the Hough circle detection algorithm to detect circles in the sketch rendered by parallel projection that conform to the predefined radius range of each component, and matching each component type according to the relative position and radius of the circle in the sketch image; finally, taking the lower left corner of the sketch image as the origin, uniformly normalizing the pixel coordinates of the center of the circle of each component to the range of [0,1] along the x and y axes, respectively, as the spatial position parameters of each component.

9. The 3D model generation and assembly system based on hand-drawn sketches according to claim 6, characterized in that, The module M4 includes: the training directed distance function neural network, in which 9000 spatial points are randomly sampled from the 3D model of each sample before training, and the directed distance function value of each sampling point is calculated. The morphological sketch of each sample, the spatial coordinates of each sampling point, and the directed distance function value of each sampling point are used as training data. The training iterations are 2000. In each training iteration, every 8 training samples are taken as the same batch. The spatial coordinates of the 9000 sampling points of each sample and the morphological sketch of that sample are used as the input of the directed distance function neural network. The training loss is the average absolute error between the directed distance function value output by the network and the directed distance function value obtained by actual sampling. The initial value of the weight parameters of the fully connected layer is set to 0. The Adam optimizer is used to update the network weights. The learning rate is 0.00005. The network is built using the PyTorch deep learning framework.

10. The 3D model generation and assembly system based on hand-drawn sketches according to claim 7, characterized in that, The module M4 includes: the deep neural network used for training assembly position parameter estimation, the training data corresponding to the network input is the overall shape sketch rendered by perspective projection of the camera facing the P1 direction, and the training data corresponding to the network output is the normalized result of the spatial position parameters of each component. In each training generation, every 32 training samples are taken as the same batch, the network weights are initialized using the Xavier method, and the network is built using the PyTorch deep learning framework.

Images