Morphological design method and system for reconstructing three-dimensional models based on sketches

Abstract

The present invention provides a morphological design method and system for reconstructing a three-dimensional model based on a sketch, and belongs to the field of intelligent product concept design. The method includes: an independent dimensionality reduction coding module for morphological forward perspective and backward perspective sketches, a multi-perspective sketch fusion and extraction module, an implicit surface generation module based on overall morphological coding, and an implicit surface triangulation module. The independent dimensionality reduction coding module for the morphological forward perspective and backward perspective sketches is based on a convolutional autoencoder, including an encoding part and a decoding part. After training in an unsupervised manner, the sketch images with a resolution of 256*256 under two perspectives are respectively reduced in dimension and encoded into a one-dimensional vector of 1*512. The present invention supports product designers to quickly establish a visualized three-dimensional model based on a hand-drawn sketch, improves the quality of the detailed features of the three-dimensional model by fusing the front and rear perspective sketches, can significantly improve the efficiency of solution generation and verification in the product concept design stage, and has the advantages of being faster and more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection between product morphology design method and three-dimensional model reconstruction, and in particular, to a morphology design method and system for reconstructing a three-dimensional model based on a sketch. Background Art

[0002] In recent years, product form design has increasingly required designers to produce innovative product forms, and at the same time, it has responded to market demands more and more quickly, shortening the process of product form concept design. Designers need to rely on intelligent and automated auxiliary technologies to improve the efficiency of producing sketches and generating three-dimensional models of form prototypes. Therefore, the current product form design method needs to fully combine intelligent technology, and accelerate and upgrade the traditional product form design process through reliable intelligent technology applications. Considering that the development of intelligent technologies such as three-dimensional reconstruction, image analysis, and content generation cannot meet the complex and diverse needs of the industry, the designer's hand-drawn sketch process is still an irreplaceable key link in the product form design process. Therefore, the current product form design method needs to replace the repetitive and labor-intensive steps in the traditional design process as much as possible through intelligent technology, while giving full play to the designer's experience and decision-making power. Based on the above considerations, the collaboration between artificial intelligence technology and designers will be an effective way to integrate the advantages of both and make up for each other's shortcomings, and it has great development potential in the field of product form design.

[0003] Traditional product form design uses the designer's hand-drawn sketches as the main form of inspiration generation, solution generation, divergence and screening, and in the iterative design process, it is necessary to continuously convert the form design scheme in the form of sketches into a three-dimensional model with detailed features and visualization through computer-aided modeling as the output product form design prototype. Among them, the computer-aided modeling process requires a lot of human participation, and due to the iterative nature of the design process, it is an important task burden for designers.

[0004] As a rapidly developing artificial intelligence technology, 3D reconstruction has the potential to solve the problem of manual modeling in morphological design. 3D reconstruction technology takes random quantities or data containing target features as input and outputs a visualized morphological 3D model. Currently, it mainly uses various neural networks as the main framework and uses voxels, point clouds, or triangular meshes as the output representation of the 3D model. In recent years, an image-based 3D reconstruction technology has flourished. Its input is a 2D image of a certain shape, and its output is a 3D model of the shape. Sketch-based 3D reconstruction uses the sketch of the shape as the input image. This technology is more in line with the demand for rapid 3D modeling in product shape design.

[0005] In recent years, with the development of deep learning, sketch-based 3D reconstruction technology has significantly improved reconstruction efficiency and accuracy, but it still faces some challenges in industrial applications. Among them, the main challenge is that the detailed features of the generated 3D model are difficult to reflect the target characteristics in the input image. However, due to the limitation of computing power and other resources, it is impossible to optimize the detailed features of the 3D model by improving the spatial resolution of the model and the number of parameters of the deep network in the short term.

[0006] Therefore, it is necessary to propose a new technical solution to improve the above technical problems. Summary of the invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a morphological design method and system for reconstructing a three-dimensional model based on a sketch.

[0008] According to a morphological design method for reconstructing a three-dimensional model based on a sketch provided by the present invention, the method comprises the following steps:

[0009] Step S1: construct a forward-view sketch dataset and a backward-view sketch dataset of the morphology, obtain all 3D models under the subcategory corresponding to the specific product morphology in the open ShapeNet dataset, and use a line drawing rendering tool to render each 3D model sample in the forward and backward perspectives to obtain a sketch;

[0010] Step S2: Under the independent dimension reduction encoding units of the morphological forward view and backward view sketches, respectively train the convolutional autoencoders for the morphological forward view sketch and the backward view sketch, the training method is unsupervised, and the loss function is the L2 distance function between the input and output images of the autoencoder;

[0011] Step S3: construct a forward view sketch encoding vector dataset and a backward view sketch encoding vector dataset of the morphology, use the trained convolutional autoencoder for inference, and save the intermediate output encoding vectors of the convolutional autoencoder for the forward view and backward view sketches respectively;

[0012] Step S4: Under the multi-view sketch fusion and extraction unit, use the forward view and backward view sketch encoding vector datasets to train the autoencoder, the training method is unsupervised, and the loss function is the L2 distance function between the concatenation layer and the output layer vector of the autoencoder;

[0013] Step S5: construct an overall morphological coding vector data set, use the trained autoencoder under the multi-view sketch fusion and extraction unit for inference, and save the overall morphological coding vector output by the autoencoder;

[0014] Step S6: construct a morphological signed distance field sampling data set. For each three-dimensional model sample, after translating the centroid to the origin of the spatial coordinates, sample points in the space to calculate their signed distance function values, and save the coordinates of the spatial points and the signed distance function values.

[0015] Step S7: Under the implicit surface generation unit based on the overall morphological coding, the overall morphological coding vector data set and the morphological signed distance field sampling data set are combined to train a fully connected neural network;

[0016] Step S8: Under the triangular meshing unit of the implicit surface, a marching cubes algorithm is used to convert the generated implicit surface into a visual three-dimensional model represented by a triangular face mesh.

[0017] Preferably, in the step S1 of constructing the forward perspective sketch and the backward perspective sketch data sets of the morphology, the three-dimensional model collected from the open data sets on the Internet is represented by a triangular mesh and stored in the obj format. When rendering the line draft on the blender platform, the spatial view adopts a perspective mode. When rendering the forward perspective sketch, the blender camera faces the front of the morphology. When rendering the backward sketch, the blender camera position is mirror-symmetric with the position when rendering the forward sketch relative to the axial plane of the morphology.

[0018] Preferably, in the training convolutional autoencoder described in step S2, the convolution kernel and the weight parameters of the fully connected layer are initialized using the xavier method, and are built using the tensorflow and keras deep learning frameworks, and the variable values ​​of the convolutional autoencoder are fixed after the training is completed;

[0019] The training autoencoder described in step S4 uses the xavier initialization method for the fully connected layer weight parameters and is built using the tensorflow and keras deep learning frameworks.

[0020] Preferably, the step S6 of constructing the morphological signed distance field sampling data set first requires filling holes, repairing non-manifold edges, and repairing self-intersections for each three-dimensional model sample, and then manually screening out samples with a single closed surface, calculating the signed distance function value of the spatial sampling point by the camera ray method, simulating the emitted rays from the spatial point and recording the earliest time when the rays contact the morphological surface, and then converting it into the signed distance function value of the spatial point.

[0021] Preferably, the training of the fully connected neural network described in step S7 has a training iteration number of 100. In each training generation, the N samples are divided into m batches and input into the fully connected neural network for forward and backward conduction to update the network weights. The spatial coordinates of the 20,000 sampling points of each sample are respectively spliced ​​with the overall morphological encoding vector of size 1*512 corresponding to the morphological sample and then input into the fully connected neural network. The training loss is the mean absolute error between the directed distance function value output by the network and the directed distance function value obtained by the actual sampling. The weight parameters of the fully connected layer are initialized using the xavier method, the Adam optimizer, the default learning rate of 0.0001, and the tensorflow and keras deep learning frameworks are used for construction.

[0022] The marching cube algorithm described in step S8 is implemented using the digital image processing package skimage.

[0023] The present invention also provides a morphological design system for reconstructing a three-dimensional model based on a sketch, the system comprising the following modules:

[0024] Module M1: Construct a forward-view sketch dataset and a backward-view sketch dataset of the morphology, obtain all 3D models under the subcategory corresponding to the specific product morphology in the open ShapeNet dataset, and use the line drawing rendering tool to render each 3D model sample in the forward and backward perspectives to obtain a sketch;

[0025] Module M2: Under the independent dimensionality reduction encoding unit of the morphological forward view and backward view sketches, convolutional autoencoders are trained on the morphological forward view sketch and backward view sketch respectively. The training method is unsupervised, and the loss function is the L2 distance function between the input and output images of the autoencoder;

[0026] Module M3: Construct the forward view sketch encoding vector dataset and the backward view sketch encoding vector dataset of the morphology, use the trained convolutional autoencoder for inference, and save the intermediate output encoding vectors of the convolutional autoencoder for the forward view and backward view sketches respectively;

[0027] Module M4: Under the multi-view sketch fusion and extraction unit, the forward view and backward view sketch encoding vector datasets are used to train the autoencoder. The training method is unsupervised, and the loss function is the L2 distance function between the concatenation layer and the output layer vector of the autoencoder.

[0028] Module M5: Construct the overall morphological coding vector dataset, use the trained autoencoder under the multi-view sketch fusion and extraction unit for inference, and save the overall morphological coding vector output by the autoencoder;

[0029] Module M6: Construct a morphological signed distance field sampling data set. For each 3D model sample, after translating the centroid to the origin of the spatial coordinates, sample points in the space to calculate their signed distance function values, and save the coordinates of the spatial points and the signed distance function values.

[0030] Module M7: Under the implicit surface generation unit based on overall morphological coding, a fully connected neural network is trained by combining the overall morphological coding vector dataset and the morphological signed distance field sampling dataset;

[0031] Module M8: Under the triangular meshing unit of the implicit surface, the marching cube algorithm is used to convert the generated implicit surface into a visual 3D model represented by a triangular mesh.

[0032] Preferably, the module M1 described in constructing the forward perspective sketch and backward perspective sketch data sets of the morphology, the three-dimensional model collected from the open data sets on the Internet is represented by a triangular mesh and stored in the obj format. When rendering the line draft on the blender platform, the spatial view adopts a perspective mode. When rendering the forward perspective sketch, the blender camera faces the front of the morphology. When rendering the backward sketch, the blender camera position is mirror-symmetric with the position when rendering the forward sketch relative to the axial plane of the morphology.

[0033] Preferably, the training convolutional autoencoder described in the module M2, the convolution kernel and the fully connected layer weight parameters adopt the xavier initialization system, and are built using the tensorflow and keras deep learning frameworks, and the variable values ​​of the convolutional autoencoder are fixed after the training is completed;

[0034] The training autoencoder described in the module M4 uses the xavier initialization system for the fully connected layer weight parameters and is built using the tensorflow and keras deep learning frameworks.

[0035] Preferably, the module M6 described in constructing the signed distance field sampling data set of the morphology first needs to fill holes, repair non-manifold edges, and repair self-intersections for each three-dimensional model sample, and then manually screen out samples with a single closed surface, calculate the signed distance function value of the spatial sampling point through the camera ray system, simulate the emitted rays from the spatial point and record the earliest time when the rays contact the morphological surface, and then convert it into the signed distance function value of the spatial point.

[0036] Preferably, the training fully connected neural network described in the module M7 has a training iteration number of 100. In each training generation, N samples are divided into m batches and input into the fully connected neural network for forward and reverse conduction to update the network weights. The spatial coordinates of the 20,000 sampling points of each sample are respectively spliced ​​with the overall morphological encoding vector of size 1*512 corresponding to the morphological sample and then input into the fully connected neural network. The training loss is the mean absolute error between the directed distance function value output by the network and the directed distance function value obtained by the actual sampling. The weight parameters of the fully connected layer use the xavier initialization system, the Adam optimizer, the default learning rate of 0.0001, and are built using the tensorflow and keras deep learning frameworks.

[0037] The marching cube algorithm described in the module M8 is implemented using the digital image processing package skimage.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention simplifies and expresses the morphology to a certain extent through the implicit surface represented by the neural network. By compressing and encoding the sketch and embedding the code into the generative neural network, the quality of the detailed features of the three-dimensional model is improved under the existing computing power and other resource conditions. It has important application potential in the field of product morphology design and is of great value to its intelligent development.

[0040] 2. This method supports product designers to quickly build visual 3D models based on hand-drawn sketches. It improves the quality of detailed features of 3D models by integrating front and back perspective sketches, and can significantly improve the efficiency of solution generation and verification in the product concept design stage. Compared with traditional morphological design hand-drawn sketches and manual 3D modeling methods, it has the advantages of being faster and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0042] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0044] Embodiment 1:

[0045] According to a morphological design method for reconstructing a three-dimensional model based on a sketch provided by the present invention, the method comprises the following steps:

[0046] Step S1: Construct a forward-view sketch dataset and a backward-view sketch dataset of the morphology, obtain all 3D models under the subclass corresponding to the specific product morphology in the open ShapeNet dataset, and use the line drawing rendering tool of the blender platform to render each 3D model sample in the forward and backward perspectives to obtain a sketch; construct a forward-view sketch dataset and a backward-view sketch dataset of the morphology, the 3D models collected from the open datasets on the Internet are represented by triangular face meshes and stored in obj format. When rendering the line drawings on the blender platform, the spatial view adopts a perspective mode. When rendering the forward-view sketch, the blender camera faces the front of the morphology. When rendering the backward sketch, the blender camera position is mirror-symmetric with the position when rendering the forward sketch relative to the morphology's medial axis.

[0047] Step S2: Under the independent dimensionality reduction coding units of the morphological forward view and backward view sketches, the convolutional autoencoders are trained for the forward view sketch and the backward view sketch respectively. The training method is unsupervised, and the loss function is the L2 distance function between the input and output images of the autoencoder. The Adam optimizer is used with a learning rate of 0.00005. The convolutional autoencoders are trained, and the convolution kernel and the fully connected layer weight parameters are initialized using the xavier method. The tensorflow and keras deep learning frameworks are used for construction. After the training is completed, the variable values ​​of the convolutional autoencoders are fixed.

[0048] Step S3: Construct a forward-view sketch encoding vector dataset and a backward-view sketch encoding vector dataset of the morphology, use the trained convolutional autoencoder for inference, and save the 1*512 encoding vector of the intermediate output of the convolutional autoencoder for the forward-view and backward-view sketches respectively.

[0049] Step S4: Under the multi-view sketch fusion and extraction unit, the forward view and backward view sketch encoding vector datasets are used to train the autoencoder. The training method is unsupervised, and the loss function is the L2 distance function between the concatenation layer and the output layer vector of the autoencoder. The Adam optimizer is used with a learning rate of 0.0001. The weight parameters of the fully connected layer are initialized using the xavier method for training the autoencoder, and the tensorflow and keras deep learning frameworks are used for construction.

[0050] Step S5: construct an overall morphological coding vector dataset, use the trained autoencoder under the multi-view sketch fusion and extraction unit for inference, and save the overall morphological coding vector of size 1*512 output by the autoencoder.

[0051] Step S6: Construct a morphological signed distance field sampling data set. For each 3D model sample, after translating the centroid to the origin of the spatial coordinate system, sample 20,000 points in space to calculate its signed distance function value, and save the coordinates and signed distance function values ​​of the spatial points. To construct a morphological signed distance field sampling data set, first, for each 3D model sample, fill holes, repair non-manifold edges, and repair self-intersections, then manually select samples with a single closed surface, calculate the signed distance function values ​​of the spatial sampling points through the camera ray method, simulate the rays emitted from the spatial points and record the earliest time when the rays contact the morphological surface, and then convert it into the signed distance function value of the spatial point.

[0052] Step S7: Under the implicit surface generation unit based on overall morphological coding, the overall morphological coding vector data set and the morphological signed distance field sampling data set are combined to train a fully connected neural network; the fully connected neural network is trained with 100 training iterations. In each training generation, the N samples are divided into m batches and input into the fully connected neural network for forward and reverse conduction to update the network weights. The spatial coordinates of the 20,000 sampling points of each sample are respectively concatenated with the overall morphological coding vector of size 1*512 corresponding to the morphological sample and then input into the fully connected neural network. The training loss is the mean absolute error between the directed distance function value output by the network and the directed distance function value obtained by the actual sampling. The weight parameters of the fully connected layer are initialized using the xavier method, the Adam optimizer, the default learning rate of 0.0001, and the tensorflow and keras deep learning frameworks are used for construction.

[0053] Step S8: Under the triangular mesh unit of the implicit surface, the generated implicit surface is converted into a visual three-dimensional model represented by a triangular mesh using a marching cube algorithm; the marching cube algorithm is implemented using the digital image processing package skimage.

[0054] The present invention also provides a morphological design system based on reconstructing three-dimensional models based on sketches. The morphological design system based on reconstructing three-dimensional models based on sketches can be implemented by executing the process steps of the morphological design method based on reconstructing three-dimensional models based on sketches, that is, those skilled in the art can understand the morphological design method based on reconstructing three-dimensional models based on sketches as a preferred implementation of the morphological design system based on reconstructing three-dimensional models based on sketches.

[0055] Embodiment 2:

[0056] The present invention also provides a morphological design system for reconstructing a three-dimensional model based on a sketch, the system comprising the following modules:

[0057] Module M1: Construct the forward-view sketch dataset and the backward-view sketch dataset of the morphology, obtain all the 3D models under the subcategory corresponding to the specific product morphology in the open ShapeNet dataset, and use the line draft rendering tool of the blender platform to render each 3D model sample in the forward and backward perspectives to obtain the sketch; construct the forward-view sketch and the backward-view sketch dataset of the morphology, the 3D models collected from the open datasets on the Internet are represented by triangular face meshes and stored in obj format. When rendering the line draft on the blender platform, the spatial view adopts the perspective mode. When rendering the forward-view sketch, the blender camera faces the front of the morphology. When rendering the backward sketch, the blender camera position is mirror-symmetric with the position when rendering the forward sketch relative to the axial plane of the morphology.

[0058] Module M2: Under the independent dimensionality reduction coding units of the morphological forward view and backward view sketches, the convolutional autoencoders are trained for the forward view sketch and the backward view sketch respectively. The training method is unsupervised, and the loss function is the L2 distance function between the input and output images of the autoencoder. The Adam optimizer is used with a learning rate of 0.00005. The convolutional autoencoders are trained, and the convolution kernel and the fully connected layer weight parameters are initialized using the xavier system. The system is built using the tensorflow and keras deep learning frameworks. After the training is completed, the variable values ​​of the convolutional autoencoders are fixed.

[0059] Module M3: Construct the forward view sketch encoding vector dataset and the backward view sketch encoding vector dataset of the morphology, use the trained convolutional autoencoder for inference, and save the 1*512 size encoding vector of the intermediate output of the convolutional autoencoder for the forward view and backward view sketches respectively.

[0060] Module M4: Under the multi-view sketch fusion and extraction unit, the forward view and backward view sketch encoding vector datasets are used to train the autoencoder. The training method is unsupervised, and the loss function is the L2 distance function between the concatenation layer and the output layer vector of the autoencoder. The Adam optimizer is used with a learning rate of 0.0001. The weight parameters of the fully connected layer of the autoencoder are initialized using the xavier system, and the tensorflow and keras deep learning frameworks are used for construction.

[0061] Module M5: Construct an overall morphological coding vector dataset, use the trained autoencoder under the multi-view sketch fusion and extraction unit for inference, and save the overall morphological coding vector of size 1*512 output by the autoencoder.

[0062] Module M6: Construct a morphological signed distance field sampling data set. For each 3D model sample, after translating the centroid to the origin of the spatial coordinate system, sample 20,000 points in space to calculate its signed distance function value, and save the coordinates and signed distance function values ​​of the spatial points. To construct a morphological signed distance field sampling data set, first fill in the holes, repair the non-manifold edges, and repair the self-intersections for each 3D model sample, then manually select samples with a single closed surface, calculate the signed distance function values ​​of the spatial sampling points through the camera ray system, simulate the emitted rays from the spatial points and record the earliest time when the rays contact the morphological surface, and then convert it into the signed distance function value of the spatial point.

[0063] Module M7: Under the implicit surface generation unit based on overall morphological coding, the overall morphological coding vector data set and the morphological signed distance field sampling data set are combined to train a fully connected neural network; the fully connected neural network is trained with 100 training iterations. In each training generation, the N samples are divided into m batches and input into the fully connected neural network for forward and backward transmission to update the network weights. The spatial coordinates of the 20,000 sampling points of each sample are concatenated with the overall morphological coding vector of size 1*512 corresponding to the morphological sample and then input into the fully connected neural network. The training loss is the mean absolute error between the directed distance function value output by the network and the directed distance function value obtained by the actual sampling. The weight parameters of the fully connected layer use the xavier initialization system, the Adam optimizer, the default learning rate of 0.0001, and the tensorflow and keras deep learning frameworks are used for construction.

[0064] Module M8: Under the triangular mesh unit of the implicit surface, the marching cube algorithm is used to convert the generated implicit surface into a visual three-dimensional model represented by a triangular patch mesh; the marching cube algorithm is implemented using the digital image processing package skimage.

[0065] Embodiment 3:

[0066] A morphological design method for reconstructing a three-dimensional model based on a sketch, comprising an independent dimension reduction coding unit for morphological forward and backward perspective sketches, a multi-perspective sketch fusion and extraction unit, an implicit surface generation unit based on overall morphological coding, and an implicit surface triangulation unit. The method is characterized in that the independent dimension reduction coding unit for the morphological forward and backward perspective sketches comprises two independent convolutional autoencoders, wherein the encoder takes the sketch as input and the one-dimensional vector of the dimension reduction coding as output, the decoder takes the coding vector as input and the reconstructed sketch as output, the multi-perspective sketch fusion and extraction unit fuses the coding of the two perspective sketches in the independent dimension reduction coding unit of the sketch into an overall morphological coding, and then inputs it into the implicit surface generation unit, and finally the implicit surface triangulation unit converts the generated implicit surface into a triangular patch mesh model to provide a visualized morphological design result.

[0067] The independent dimension reduction encoding unit of the morphological forward view and backward view sketches comprises two convolutional autoencoders with identical structures for processing the forward view sketch and the backward view sketch respectively, each convolutional autoencoder is composed of an encoder input layer, a plurality of sequentially stacked convolutional layers, a plurality of sequentially stacked deconvolutional layers and an encoder output layer, including sequentially connected:

[0068] A. Input layer: the input is a grayscale image with a resolution of 256*256. The dimension of the input data is N*224*224*1, where N is the number of images.

[0069] B. 3 convolutional layers, the convolution kernel size is 5*5, the step size is 2, the number of convolution kernels in each convolution layer is (32, 64, 128) respectively, and the activation function is the ReLU function;

[0070] C. Batch normalization layer;

[0071] D. 2 convolutional layers, the convolution kernel size is 3*3, the step size is 2, the number of convolution kernels in each convolution layer is (256, 512) respectively, and the activation function is the ReLU function;

[0072] E, Dropout layer, dropout probability is 0.1;

[0073] F, Flatten layer, which expands the feature map output by the last convolutional layer into a one-dimensional vector;

[0074] G, fully connected layer, outputs the reduced encoding vector with a dimension of 512 and the activation function is Softmax;

[0075] H, fully connected layer, outputs the reconstructed decoding features, the dimension of which is consistent with the output vector dimension of the Flatten layer in the encoder. The activation function is Softmax;

[0076] I. Dropout layer, the dropout probability is 0.1;

[0077] J. 2 deconvolution layers, the deconvolution kernel size is 3*3, the step size is 2, the number of convolution kernels in each convolution layer is (256, 128), and the activation function is the ReLU function;

[0078] K, batch normalization layer;

[0079] L, 3 deconvolution layers, the convolution kernel size is 5*5, the step size is 2, the number of convolution kernels in each convolution layer is (64, 32, 1) respectively, and the activation function is the ReLU function;

[0080] M, output layer, the output is a grayscale image with a resolution of 256*256, and the dimension of the output data is N*256*256*1, where N is the number of images.

[0081] The loss function of each convolutional autoencoder of the independent dimensionality reduction coding unit of the morphological forward view and backward view sketch is the L2 loss of the input image and the output image, and Adam is used for weight update with a learning rate of 0.00005.

[0082] The multi-view sketch fusion and extraction unit is an autoencoder, which is composed of an encoder and a decoder with a mirror structure, including the following connected in sequence:

[0083] a) Input layer: the input data is two one-dimensional vectors of size 1*512, which are obtained by the convolutional encoder corresponding to the forward view and the convolutional encoder corresponding to the backward view respectively;

[0084] b) The concatenation layer concatenates the two one-dimensional vectors of the input layer into a one-dimensional vector of size 1*1024;

[0085] c) Dropout layer, the dropout probability is 0.1;

[0086] d) Fully connected layer, outputs an intermediate vector of size 1*1024, and the activation function is Softmax;

[0087] e), batch normalization layer;

[0088] f), Dropout layer, the dropout probability is 0.1;

[0089] g) Fully connected layer, outputs an intermediate vector of size 1*512, and the activation function is tanh;

[0090] h), fully connected layer, outputs the reconstructed intermediate vector of size 1*1024, and the activation function is softmax;

[0091] i) Dropout layer, the dropout probability is 0.1;

[0092] j) Fully connected layer, outputs the reconstructed vector of size 1*1024, and the activation function is softmax;

[0093] k), output layer, the output data is the reconstructed 1*1024 size vector.

[0094] The autoencoder loss function of the multi-view sketch fusion and extraction unit is the L2 loss between the vectors of the concatenation layer and the output layer. Adam is used for weight update with a learning rate of 0.0001.

[0095] The application of the independent dimension reduction coding unit of the morphological forward view and backward view sketch and the multi-view sketch fusion and extraction unit includes two steps: training and reasoning. In the training step of the independent dimension reduction coding unit of the morphological forward view and backward view sketch, the sketch data set of the morphological forward view and backward view is used for unsupervised training. In the reasoning step, the sketch samples that do not overlap with the sketch data set used for training are used as the input of the convolutional encoder, and the 1*512 size coding vector output by the fully connected layer is used as the result of reasoning. In the training step of the multi-view sketch fusion and extraction unit, the coding vector data set of the morphological forward view and backward view is used for unsupervised training. In the reasoning step, the coding vector samples that do not overlap with the training set are used as the input of the encoder, and the 1*512 size coding vector output by the fully connected layer is used as the result of reasoning and named as the overall morphological coding vector.

[0096] The implicit surface generation unit based on overall morphological coding uses a signed distance function SDF(·) to represent the implicit surface, and requires the implicit surface to be a single continuous surface with watertight properties, which is used to reflect a single, closed morphological entity. The form of the signed distance function is:

[0097] SDF(x)=p

[0098] Where p is the shortest distance from the spatial point x to the implicit surface. When p>0, the spatial point x is outside the implicit surface, that is, the outside of the corresponding morphological entity. When p<0, the spatial point x is inside the implicit surface, that is, the inside of the corresponding morphological entity. When p=0, the spatial point x is on the implicit surface, that is, the surface of the morphological entity. The implicit surface generation unit based on the overall morphological encoding uses a fully connected neural network to approximate the nonlinear directed distance function SDF(·), including the following connected in sequence:

[0099] A) Input layer: the input data is the overall morphological code of size 1*512 and the coordinates of the spatial point x;

[0100] B) The concatenation layer concatenates the overall morphological code of the input layer and the x-coordinate of the spatial point into a one-dimensional vector of size 1*515;

[0101] C) 2 fully connected layers, the output is an intermediate vector of size 1*512, and the activation function is softmax;

[0102] D), Dropout layer, the dropout probability is 0.1;

[0103] E), batch normalization layer;

[0104] F), 3 fully connected layers, the output is an intermediate vector of size 1*512, and the activation function is softmax;

[0105] G), fully connected layer, the output is a scalar, and the activation function is tanh.

[0106] The implicit surface triangular mesh unit uses a marching cube algorithm to achieve visualization of the implicit surface. The spatial sampling resolution of the marching cube algorithm is 256*256*256, and the isosurface division threshold is 0.

[0107] The present invention provides a morphological design method for reconstructing a three-dimensional model based on a sketch, which includes: an independent dimension reduction coding unit for morphological forward and backward perspective sketches, a multi-perspective sketch fusion and extraction unit, an implicit surface generation unit based on overall morphological coding, and an implicit surface triangulation unit.

[0108] A morphological design method for reconstructing a three-dimensional model based on a sketch mainly includes the following steps:

[0109] Step 1: Constructing the forward and backward perspective sketch datasets of the morphology. Obtain all 3D models in the subcategory corresponding to the specific product morphology in the open ShapeNet dataset. For each 3D model sample, use the line drawing rendering tool of the blender platform to render the sketches in the forward and backward perspectives.

[0110] Step 2: Under the independent dimensionality reduction encoding units of the morphological forward view and backward view sketches, convolutional autoencoders are trained for the morphological forward view sketch and backward view sketch respectively. The training method is unsupervised, and the loss function is the L2 distance function between the input and output images of the autoencoder. The Adam optimizer is used with the default learning rate of 0.00005.

[0111] Step 3: Construct the forward and backward view sketch encoding vector datasets of the morphology. Use the trained convolutional autoencoder for inference, and save the 1*512 encoding vectors of the intermediate output of the convolutional autoencoder for the forward and backward view sketches.

[0112] Step 4: Under the multi-view sketch fusion and extraction unit, use the forward view and backward view sketch encoding vector datasets to train the autoencoder. The training method is unsupervised, the loss function is the L2 distance function between the concatenation layer and the output layer vector of the autoencoder, and the Adam optimizer is used with the default learning rate of 0.0001.

[0113] Step 5: Construct the overall morphological encoding vector dataset. Use the trained autoencoder under the multi-view sketch fusion and extraction unit for inference, and save the overall morphological encoding vector of size 1*512 output by the autoencoder.

[0114] Step 6: Construction of the morphological signed distance field sampling data set. For each 3D model sample, after translating the centroid to the origin of the spatial coordinates, 20,000 points are randomly sampled in space to calculate their signed distance function values, and the coordinates of the spatial points and the signed distance function values ​​are saved.

[0115] Step 7: Under the implicit surface generation unit based on overall morphological coding, the overall morphological coding vector dataset and the morphological signed distance field sampling dataset are combined to train a fully connected neural network.

[0116] Step 8: Under the triangular mesh unit of the implicit surface, the marching cube algorithm is used to convert the generated implicit surface into a visual three-dimensional model represented by a triangular patch mesh.

[0117] Furthermore, in step 1, the 3D model collected from the open datasets on the Internet is represented by a triangular mesh and stored in obj format. When rendering the line drawing on the blender platform, the spatial view adopts a perspective mode. When rendering the forward perspective sketch, the blender camera faces the front of the form. When rendering the backward sketch, the blender camera position is mirror-symmetric with the position when rendering the forward sketch relative to the axial plane of the form.

[0118] The training convolutional autoencoder described in step 2 uses the xavier initialization method for the convolution kernel and the fully connected layer weight parameters, and is built using the tensorflow and keras deep learning frameworks. After the training is completed, the variable values ​​of the convolutional autoencoder are fixed.

[0119] The training autoencoder described in step 4 uses the xavier initialization method for the fully connected layer weight parameters and is built using the tensorflow and keras deep learning frameworks.

[0120] The construction of the morphological signed distance field sampling data set described in step 6 first requires filling holes, repairing non-manifold edges, and repairing self-intersections for each 3D model sample, and then manually screening out samples with a single closed surface. The signed distance function value of the spatial sampling point is calculated by the camera ray method, and the rays emitted from the spatial points are simulated and the earliest time when the rays contact the morphological surface is recorded, which is then converted into the signed distance function value of the spatial point.

[0121] The training fully connected neural network described in step 7 has 100 training iterations. In each training generation, the N samples are divided into m batches and input into the fully connected neural network for forward and backward propagation to update the network weights. The spatial coordinates of the 20,000 sampling points of each sample are respectively concatenated with the overall morphological encoding vector of 1*512 size corresponding to the morphological sample and then input into the fully connected neural network. The training loss is the mean absolute error between the directed distance function value output by the network and the directed distance function value obtained by the actual sampling. The weight parameters of the fully connected layer are initialized using the xavier method, the Adam optimizer, the default learning rate of 0.0001, and the tensorflow and keras deep learning frameworks are used for construction.

[0122] The marching cubes algorithm described in step 8 is implemented using the digital image processing package skimage.

[0123] The coordinate values ​​of the mesh nodes of the 3D model samples obtained from ShapeNet in the x / y / z direction are all within the range of [-1, 1]. Each 3D model sample should include mesh node coordinates, triangle vertex information, node normal vectors and triangle normal vector information, ignoring texture and color information. When rendering line drawings on the blender platform, the line thickness value is within the range of [0.4, 0.6], and the distance from the camera position to the origin in the forward and backward viewing angles of each sample is 1.5.

[0124] The deep learning framework tensorflow and keras are used to build a deep learning framework. The Adam optimizer is used to update the weights of the convolutional autoencoder of the independent dimensionality reduction coding unit, the autoencoder of the multi-view sketch fusion and extraction unit, and the fully connected deep network during the training phase.

[0125] When training the convolutional autoencoder, the sketch image used is a single-channel grayscale image with a bit depth of 8 bits and a pixel value range of 0-255. The pixel values ​​are uniformly scaled to [0, 1] before entering the convolutional autoencoder, and the computer vision library opencv is used to complete the reading and scaling of the sketch.

[0126] When constructing the morphological signed distance field sampling data set, the open source 3D model repair tool Meshfix is ​​used to fill holes, repair non-manifold edges and self-intersections of the original 3D model samples. For samples with a single closed surface that are manually screened, the 3D data processing library open3d is used to check their watertightness to avoid the situation where the defects are not manually screened due to small defects.

[0127] When converting the implicit surface into a triangular mesh representation, the mesh nodes and triangular mesh information output by the marching cubes algorithm are saved in the TriangleMesh class object of the open3d library, and exported to a triangular mesh in the obj format using the io unit of the open3d library.

[0128] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0129] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0130] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A morphological design method based on sketch reconstruction of 3D models, It is characterized in that The method comprises the following steps: Step S1: construct a forward-view sketch dataset and a backward-view sketch dataset of the morphology, obtain all 3D models under the subcategory corresponding to the specific product morphology in the open ShapeNet dataset, and use a line drawing rendering tool to render each 3D model sample in the forward and backward perspectives to obtain a sketch; Step S2: Under the independent dimension reduction encoding units of the morphological forward view and backward view sketches, respectively train the convolutional autoencoders for the morphological forward view sketch and the backward view sketch, the training method is unsupervised, and the loss function is the L2 distance function between the input and output images of the autoencoder; Step S3: construct a forward view sketch encoding vector dataset and a backward view sketch encoding vector dataset of the morphology, use the trained convolutional autoencoder for inference, and save the intermediate output encoding vectors of the convolutional autoencoder for the forward view and backward view sketches respectively; Step S4: Under the multi-view sketch fusion and extraction unit, use the forward view and backward view sketch encoding vector datasets to train the autoencoder, the training method is unsupervised, and the loss function is the L2 distance function between the concatenation layer and the output layer vector of the autoencoder; Step S5: construct an overall morphological coding vector data set, use the trained autoencoder under the multi-view sketch fusion and extraction unit for inference, and save the overall morphological coding vector of the intermediate output of the autoencoder; Step S6: construct a morphological signed distance field sampling data set. For each three-dimensional model sample, after translating the centroid to the origin of the spatial coordinates, sample points in the space to calculate their signed distance function values, and save the coordinates of the spatial points and the signed distance function values. Step S7: Under the implicit surface generation unit based on the overall morphological coding, the overall morphological coding vector data set and the morphological signed distance field sampling data set are combined to train a fully connected neural network; Step S8: Under the triangular meshing unit of the implicit surface, a marching cubes algorithm is used to convert the generated implicit surface into a visual three-dimensional model represented by a triangular face mesh.

2. The morphological design method for reconstructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that The step S1 of constructing the forward perspective sketch and backward perspective sketch data sets of the morphology includes: the three-dimensional model collected from the open data set on the Internet is represented by a triangular mesh and stored in the obj format. When rendering the line draft on the blender platform, the spatial view adopts a perspective mode. When rendering the forward perspective sketch, the blender camera faces the front of the morphology. When rendering the backward perspective sketch, the blender camera position is mirror-symmetric with the position when rendering the forward sketch relative to the axial plane of the morphology.

3. The morphological design method for reconstructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that In the training convolutional autoencoder described in step S2, the convolution kernel and the weight parameters of the fully connected layer are initialized using the xavier method, and the tensorflow and keras deep learning frameworks are used to build it. After the training is completed, the variable values ​​of the convolutional autoencoder are fixed; The training autoencoder described in step S4 uses the xavier initialization method for the fully connected layer weight parameters and is built using the tensorflow and keras deep learning frameworks.

4. The morphological design method for reconstructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that The step S6 described in constructing the morphological signed distance field sampling data set first requires filling holes, repairing non-manifold edges, and repairing self-intersections for each three-dimensional model sample, and then manually screening out samples with a single closed surface, calculating the signed distance function value of the spatial sampling point through the camera ray method, simulating the emitted rays from the spatial point and recording the earliest time when the rays contact the morphological surface, and then converting it into the signed distance function value of the spatial point.

5. The morphological design method for reconstructing a three-dimensional model based on a sketch according to claim 1, It is characterized in that The training of the fully connected neural network described in step S7 has a training iteration number of 100. In each training generation, the N samples are divided into m batches and input into the fully connected neural network for forward and reverse conduction to update the network weights. The spatial coordinates of the 20,000 sampling points of each sample are respectively spliced ​​with the overall morphological encoding vector of size 1*512 corresponding to the morphological sample and then input into the fully connected neural network. The training loss is the mean absolute error between the directed distance function value output by the network and the directed distance function value obtained by the actual sampling. The weight parameters of the fully connected layer are initialized using the xavier method, the Adam optimizer, the default learning rate of 0.0001, and the tensorflow and keras deep learning frameworks are used for construction; The marching cube algorithm described in step S8 is implemented using the digital image processing package skimage.

6. A morphological design system based on sketch reconstruction of 3D models, It is characterized in that The system includes the following modules: Module M1: Construct a forward-view sketch dataset and a backward-view sketch dataset of the morphology, obtain all 3D models under the subcategory corresponding to the specific product morphology in the open ShapeNet dataset, and use the line drawing rendering tool to render each 3D model sample in the forward and backward perspectives to obtain a sketch; Module M2: Under the independent dimensionality reduction encoding unit of the morphological forward view and backward view sketches, convolutional autoencoders are trained on the morphological forward view sketch and backward view sketch respectively. The training method is unsupervised, and the loss function is the L2 distance function between the input and output images of the autoencoder; Module M3: Construct the forward view sketch encoding vector dataset and the backward view sketch encoding vector dataset of the morphology, use the trained convolutional autoencoder for inference, and save the intermediate output encoding vectors of the convolutional autoencoder for the forward view and backward view sketches respectively; Module M4: Under the multi-view sketch fusion and extraction unit, the forward view and backward view sketch encoding vector datasets are used to train the autoencoder. The training method is unsupervised, and the loss function is the L2 distance function between the concatenation layer and the output layer vector of the autoencoder. Module M5: Construct the overall morphological coding vector dataset, use the trained autoencoder under the multi-view sketch fusion and extraction unit for inference, and save the overall morphological coding vector of the intermediate output of the autoencoder; Module M6: Construct a morphological signed distance field sampling data set. For each 3D model sample, after translating the centroid to the origin of the spatial coordinates, sample points in the space to calculate their signed distance function values, and save the coordinates of the spatial points and the signed distance function values. Module M7: Under the implicit surface generation unit based on overall morphological coding, a fully connected neural network is trained by combining the overall morphological coding vector dataset and the morphological signed distance field sampling dataset; Module M8: Under the triangular meshing unit of the implicit surface, the marching cube algorithm is used to convert the generated implicit surface into a visual 3D model represented by a triangular mesh.

7. The morphological design system for reconstructing a three-dimensional model based on a sketch according to claim 6, It is characterized in that The module M1 described in constructing the forward perspective sketch and backward perspective sketch data sets of the morphology includes: the three-dimensional model collected from the open data set on the Internet is represented by a triangular mesh and stored in the obj format. When rendering the line draft on the blender platform, the spatial view adopts a perspective method. When rendering the forward perspective sketch, the blender camera faces the front of the morphology. When rendering the backward perspective sketch, the blender camera position is mirror-symmetric with the position when rendering the forward sketch relative to the axial plane of the morphology.

8. The morphology design system for reconstructing a three-dimensional model based on a sketch according to claim 6, It is characterized in that The training convolutional autoencoder described in the module M2 uses the xavier initialization method for the convolution kernel and the fully connected layer weight parameters, and is built using the tensorflow and keras deep learning frameworks. After the training is completed, the variable values ​​of the convolutional autoencoder are fixed; The training autoencoder described in the module M4 uses the xavier initialization method for the fully connected layer weight parameters and is built using the tensorflow and keras deep learning frameworks.

9. The morphology design system for reconstructing a three-dimensional model based on a sketch according to claim 6, It is characterized in that The module M6 described in constructing a morphological signed distance field sampling data set first needs to fill holes, repair non-manifold edges, and repair self-intersections for each three-dimensional model sample, and then manually screen out samples with a single closed surface, calculate the signed distance function value of the spatial sampling point through the camera ray method, simulate the emitted rays from the spatial point and record the earliest time when the rays contact the morphological surface, and then convert it into the signed distance function value of the spatial point.

10. The morphology design system for reconstructing a three-dimensional model based on a sketch according to claim 6, It is characterized in that The training fully connected neural network described in the module M7 has a training iteration number of 100. In each training generation, N samples are divided into m batches and input into the fully connected neural network for forward and reverse conduction to update the network weights. The spatial coordinates of the 20,000 sampling points of each sample are respectively concatenated with the overall morphological encoding vector of size 1*512 corresponding to the morphological sample and then input into the fully connected neural network. The training loss is the mean absolute error between the directed distance function value output by the network and the directed distance function value obtained by the actual sampling. The weight parameters of the fully connected layer are initialized using the xavier method, the Adam optimizer, the default learning rate of 0.0001, and the tensorflow and keras deep learning frameworks are used for construction; The marching cube algorithm described in the module M8 is implemented using the digital image processing package skimage.

Images