Three-dimensional model generation method, device and electronic device based on single hand-drawn sketch

Abstract

The present invention provides a method, device and electronic device for generating a three-dimensional model based on a single hand-drawn sketch, comprising: obtaining a single hand-drawn sketch; inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch; wherein the three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of the input single hand-drawn sketch. The method, device and electronic device provided by the present invention can predict the viewpoint of a hand-drawn sketch provided by a user, so that the subsequently generated three-dimensional model avoids ambiguity based on the predicted viewpoint.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional model generation, and in particular to a three-dimensional model generation method, device and electronic equipment based on a single hand-drawn sketch. Background Art

[0002] The construction of three-dimensional models is the basis of geometric drawing. Sketch-based three-dimensional modeling can assist relevant staff to complete time-consuming and laborious modeling operations. Existing sketch-based three-dimensional modeling methods can be mainly divided into two categories: retrieval-based and construction-based. Among them, the retrieval-based method combines predefined template metamodels into three-dimensional models, or directly retrieves and extracts from a model set; the construction-based method directly maps the input strokes to the three-dimensional model, which can provide a modeling capability with higher degrees of freedom than the retrieval-based method. Both methods have obvious disadvantages: the retrieval-based method can only generate specific three-dimensional models, or three-dimensional models composed of certain basic shapes; and the construction-based method, due to the lack of prior knowledge, usually requires a large amount of work to obtain satisfactory results for users. In addition, with the popularization of touch screens and the development of VR / AR technology, the demand for three-dimensional content creation by non-professional users is increasing, and the above two methods cannot meet the needs of ordinary users for ease of use and customizability.

[0003] In order to meet the needs of ordinary users for 3D content creation, some works have tried to generate 3D models from a single hand-drawn sketch with the help of deep learning technology. Wang et al. proposed "Unsupervised learning of 3D model reconstruction from hand-drawn sketches", which generates a 3D model represented by voxels from hand-drawn sketches by retrieval. Wang et al. proposed "3D shape reconstruction from free-hand sketches", which can generate a 3D model represented by a point cloud from a single hand-drawn sketch. On the one hand, the above two methods do not directly generate a 3D model represented by a triangular mesh, and cannot be directly used for subsequent editing and drawing; on the other hand, they ignore the ambiguity that is common in hand-drawn sketches, and the generated 3D model cannot well represent the user's intention.

[0004] Therefore, how to avoid the situation where the existing three-dimensional model generated based on hand-drawn sketches causes serious ambiguity in the three-dimensional model due to the inability to estimate the user's perspective intention is still a problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] The present invention provides a 3D model generation method, device and electronic device based on a single hand-drawn sketch, so as to solve the problem that the existing 3D model generated based on hand-drawn sketches has serious ambiguity in the 3D model due to the inability to estimate the user's perspective intention. By introducing perspective labels when training the 3D construction model, the trained 3D construction model can predict the accurate perspective of the user's sketch. The accurate perspective can provide perspective information in the 3D construction model so that the perspective of the constructed 3D model is consistent with the perspective of the sketch, thereby avoiding ambiguity of the 3D model relative to the sketch.

[0006] The present invention provides a method for generating a three-dimensional model based on a single hand-drawn sketch, the method comprising:

[0007] Get a single hand-drawn sketch;

[0008] Inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch;

[0009] The three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of an input single hand-drawn sketch.

[0010] According to a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention,

[0011] The network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, and the image encoder is connected to the three-dimensional generation network.

[0012] According to a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention, the network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, and the image encoder is connected to the three-dimensional generation network, specifically including:

[0013] During the training process of the three-dimensional construction model,

[0014] The image encoder is used to encode the input single hand-drawn sketch sample to obtain the shape feature vector and the view feature vector and output them to the 3D generation network and the view prediction network respectively;

[0015] The view prediction network is used to predict and decode the view feature vector to obtain a predicted view, and determine a predicted view error based on the predicted view and a corresponding view label;

[0016] The view encoder is used to encode the input corresponding view label to obtain a new view feature vector and output it to the three-dimensional generation network;

[0017] The 3D generation network generates a predicted 3D model based on the new view feature vector and the shape feature vector;

[0018] A differentiable rendering module is connected to the output end of the three-dimensional construction model, wherein the differentiable rendering module generates a prediction mask based on the predicted three-dimensional model, and determines a predicted three-dimensional model error based on the predicted mask and a standard mask;

[0019] The loss function in the training process is constructed based on the predicted three-dimensional model error and the predicted viewing angle error.

[0020] According to a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention,

[0021] The loss function in the training process is constructed based on the predicted 3D model error and the predicted viewing angle error, and specifically includes:

[0022] The predicted three-dimensional model error is characterized based on an error between the predicted mask and the standard mask, where the standard mask is a mask of the single hand-drawn sketch sample.

[0023] According to a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention,

[0024] The prediction mask includes a mask of the predicted three-dimensional model under the corresponding viewing angle label and a mask of a random predicted three-dimensional model constructed based on a random viewing angle under the random viewing angle.

[0025] According to a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention, the loss function specifically includes:

[0026] the intersection-over-union ratio between the mask of the predicted 3D model at the corresponding view label and the single hand-drawn sketch sample and the standard mask, the intersection-over-union ratio between the mask of the random predicted 3D model constructed based on random view at the random view and the standard mask, and,

[0027] The cross entropy of the corresponding view label and the random view after passing through the shape discriminator.

[0028] According to a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention, the method further comprises:

[0029] Obtaining a specified viewing angle corresponding to the single hand-drawn sketch;

[0030] Accordingly, the step of inputting the single hand-drawn sketch into the three-dimensional construction model and outputting the three-dimensional model corresponding to the single hand-drawn sketch includes:

[0031] An image encoder inputting the single hand-drawn sketch into the three-dimensional constructed model outputs a shape feature vector to the three-dimensional constructed model;

[0032] Input the specified viewing angle into the viewing angle encoder and output the specified viewing angle vector feature to the three-dimensional construction model;

[0033] The three-dimensional construction model generates a three-dimensional model based on the shape feature vector and the specified viewing angle vector feature.

[0034] The present invention also provides a three-dimensional model generation device based on a single hand-drawn sketch, comprising:

[0035] An acquisition unit, used for acquiring a single hand-drawn sketch;

[0036] A construction unit, used for inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch;

[0037] The three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of an input single hand-drawn sketch.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for generating a three-dimensional model based on a single hand-drawn sketch as described above are implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for generating a three-dimensional model based on a single hand-drawn sketch.

[0040] The present invention provides a method, device and electronic device for generating a three-dimensional model based on a single hand-drawn sketch, by obtaining a single hand-drawn sketch; inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch; wherein the three-dimensional construction model is obtained by training based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting the predicted viewpoint of the input single hand-drawn sketch. Since the viewpoint label is introduced when training the three-dimensional construction model, the trained three-dimensional construction model can predict the accurate viewpoint of the user's sketch, and the accurate viewpoint can provide viewpoint information in the three-dimensional construction model so that the viewpoint of the constructed three-dimensional model is consistent with the viewpoint of the sketch, avoiding ambiguity of the three-dimensional model relative to the sketch. Therefore, the method, device and electronic device provided by the embodiment of the present invention can predict the viewpoint of the hand-drawn sketch provided by the user, so that the subsequently generated three-dimensional model avoids ambiguity based on the predicted viewpoint. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of the process of the method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention;

[0043] Figure 2 An example diagram of a single hand-drawn sketch provided by the present invention;

[0044] Figure 3 A schematic diagram of the neural network structure of the three-dimensional construction model provided by the present invention;

[0045] Figure 4 A flowchart of a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention;

[0046] Figure 5 A schematic diagram of the structure of a three-dimensional model generation device based on a single hand-drawn sketch provided by the present invention;

[0047] Figure 6 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The existing 3D models generated based on hand-drawn sketches generally have the problem of serious ambiguity in the 3D models due to the inability to estimate the user's perspective intention. Figure 1-Figure 4 The present invention describes a method for generating a three-dimensional model based on a single hand-drawn sketch. Figure 1 A schematic diagram of the process of generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0050] Step 110, obtaining a single hand-drawn sketch.

[0051] Specifically, the 3D model generation method based on a single hand-drawn sketch provided by the present invention is implemented on a 3D construction model trained based on machine learning, and the execution subject of the 3D model generation method based on a single hand-drawn sketch provided by the present invention is a 3D model generation device based on a single hand-drawn sketch. When using the trained 3D construction model, it is necessary to input the user sketch to be 3D modeled, and only a single sketch is needed, because only a single hand-drawn sketch sample is used when generating a 3D model during the 3D construction model training process, and it is not necessary to train with multiple images from different perspectives as in the 3D model reconstruction method in the prior art. Figure 2 An example diagram of a single hand-drawn sketch provided by the present invention, such as Figure 2 Described above, displayed in sequence are a single hand-drawn sketch of “airplane”, a single hand-drawn sketch of “table”, a single hand-drawn sketch of “car”, a single hand-drawn sketch of “chair” and a single hand-drawn sketch of “display screen”.

[0052] Step 120, inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch;

[0053] The three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of an input single hand-drawn sketch.

[0054] Specifically, a single hand-drawn sketch of the three-dimensional model to be constructed is input into a trained three-dimensional construction model, and a three-dimensional modeling result of the single hand-drawn sketch is output, wherein the three-dimensional construction model is obtained by training based on a large number of single hand-drawn sketches and corresponding perspective labels, and the three-dimensional construction model includes a perspective prediction network for predicting the perspective of the input single hand-drawn sketch, and the predicted perspective is used in the three-dimensional generation network to provide perspective information to make the constructed three-dimensional model more accurate. It should be noted here that the three-dimensional construction model includes a three-dimensional generation network by default, and therefore, the perspective label input in the three-dimensional construction model training process is mainly used to correct the predicted perspective output by the perspective prediction network, adjust the parameters to be optimized in the perspective prediction network, and also provide accurate perspective information for the subsequent training of the three-dimensional generation network, so that the error of the output three-dimensional model of the three-dimensional generation network relative to the standard model is only related to the adjustment of the parameters to be optimized of the three-dimensional generation network, and is not affected by the predicted perspective error. In the sample single hand-drawn sketch and the corresponding perspective label used for 3D construction model training, the sample single hand-drawn sketch is both the input sample of the perspective prediction network and the input sample of the final 3D generation network, and the corresponding perspective label is used as both the standard reference value (ground truth) of the perspective prediction network and the input sample of the 3D generation network.

[0055] The method provided by the present invention obtains a single hand-drawn sketch; inputs the single hand-drawn sketch into a three-dimensional construction model, and outputs a three-dimensional model corresponding to the single hand-drawn sketch; wherein the three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding perspective label, and the three-dimensional construction model includes a perspective prediction network for extracting the predicted perspective of the input single hand-drawn sketch. Since the perspective label is introduced when training the three-dimensional construction model, the trained three-dimensional construction model can predict the accurate perspective of the user's sketch. The accurate perspective can provide perspective information in the three-dimensional construction model so that the perspective of the constructed three-dimensional model is consistent with the perspective of the sketch, avoiding ambiguity of the three-dimensional model relative to the sketch. Therefore, the method provided by the embodiment of the present invention can predict the perspective of the hand-drawn sketch provided by the user, so that the subsequently generated three-dimensional model avoids ambiguity based on the predicted perspective.

[0056] Based on the above embodiment, in the method, the network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, and the image encoder is connected to the three-dimensional generation network.

[0057] Specifically, the neural network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, and the image encoder is connected to the three-dimensional generation network. Figure 3A schematic diagram of the neural network structure of the three-dimensional construction model provided by the present invention, such as Figure 3 As shown, the overall neural network structure of the three-dimensional construction model consists of an image encoder, a view prediction network, a view encoder and a three-dimensional generation network, wherein the image encoder, the prediction network, the view encoder and the three-dimensional generation network are connected in sequence, and the image encoder also directly outputs to the three-dimensional generation network. The above network structure of the three-dimensional construction model is to form a three-dimensional construction algorithm: when using the three-dimensional construction model to perform three-dimensional reconstruction on a single hand-drawn sketch, the image encoder first performs image encoding on the input single hand-drawn sketch, that is, mapping the input single hand-drawn sketch I to the shape feature vector space. and the view feature vector space The resulting shape feature vector z s and the viewing angle feature vector z v , then the shape feature vector z s Directly output to the 3D generation network at the end of the overall network, the view feature vector z v The output is sent to the view prediction network. After the view prediction network makes a prediction, it outputs the predicted view to the view encoder. The predicted view is the view prediction value expressed in Euler angles. Then the view encoder encodes the predicted view to obtain a new view feature vector. The final 3D generation network is based on the input shape feature vector z s And the new perspective feature vector generates a three-dimensional model as the final output value of the entire three-dimensional construction model.

[0058] Based on the above embodiment, in the method, the network structure of the 3D construction model includes an image encoder, the view prediction network, a view encoder and a 3D generation network connected in sequence, and the image encoder is connected to the 3D generation network, specifically including:

[0059] During the training process of the three-dimensional construction model,

[0060] The image encoder is used to encode the input single hand-drawn sketch sample to obtain the shape feature vector and the view feature vector and output them to the 3D generation network and the view prediction network respectively;

[0061] The view prediction network is used to predict and decode the view feature vector to obtain a predicted view, and determine a predicted view error based on the predicted view and a corresponding view label;

[0062] The view encoder is used to encode the input corresponding view label to obtain a new view feature vector and output it to the three-dimensional generation network;

[0063] The 3D generation network generates a predicted 3D model based on the new view feature vector and the shape feature vector;

[0064] A differentiable rendering module is connected to the output end of the three-dimensional construction model, wherein the differentiable rendering module generates a prediction mask based on the predicted three-dimensional model, and determines a predicted three-dimensional model error based on the predicted mask and a standard mask;

[0065] The loss function in the training process is constructed based on the predicted three-dimensional model error and the predicted viewing angle error.

[0066] Specifically, Figure 2 As shown, the use process of the three-dimensional construction model includes the following four steps: a, b, c and d:

[0067] Step a: The user inputs a single hand-drawn sketch, and the image encoder obtains the view feature vector and shape feature vector;

[0068] Step b, the viewing angle feature vector is passed through a viewing angle prediction network to obtain a viewing angle prediction value expressed in Euler angles;

[0069] Step c, the view prediction value in step b is input into the view encoder, and a new view feature vector is output;

[0070] In step d, the new view feature vector generated in step c and the shape feature vector generated in step a are input into a three-dimensional generation network to obtain a three-dimensional model corresponding to the input single hand-drawn sketch.

[0071] The training process of the three-dimensional construction model includes:

[0072] The image encoder encodes the input single hand-drawn sketch sample to obtain the shape feature vector and the view feature vector and outputs them to the 3D generation network and the view prediction network respectively;

[0073] The view prediction network predicts and decodes the view feature vector to obtain a predicted view;

[0074] The view encoder encodes the input corresponding view label to obtain a new view feature vector and outputs it to the three-dimensional generation network;

[0075] The 3D generation network generates a predicted 3D model based on the new view feature vector and the shape feature vector.

[0076] During the above training process, the training of the view prediction network and the 3D generation network overlaps only a little, and most of them are independent. The overlapping part is that the view prediction network and the 3D generation network share the same data source, that is, a single hand-drawn sketch as input. The independence is reflected in the evaluation of the view prediction network using the view label as the standard reference value (ground truth) and the output predicted view angle value to calculate the error, while the 3D generation network inputs the view label instead of the view prediction value output by the view prediction network, so as to avoid the error of the view prediction being further superimposed on the error of the 3D generation network itself, and to train the view prediction function and the 3D reconstruction function separately as much as possible to obtain a more accurate model.

[0077] Optionally, the image encoding module is a convolutional neural network structure consisting of multiple convolutional layers, nonlinear activation layers, pooling layers and normalization layers. The structure inputs a three-channel hand-drawn sketch image in RGB space and outputs a multi-channel feature map. The feature map then passes through two fully connected layers to generate a view-related feature vector and a shape-related feature vector respectively.

[0078] Optionally, the view prediction network is a fully connected neural network structure composed of multiple fully connected layers and nonlinear activation layers. The structure inputs the view feature vector output by the image encoder and outputs the view prediction represented by three angles of roll, pitch and yaw.

[0079] Optionally, the view encoder is a fully connected neural network structure composed of multiple fully connected layers and nonlinear activation layers. The structure inputs the predicted view represented by three angles of roll, pitch and yaw, and outputs a corresponding new view feature vector that is consistent with the space to which the view feature vector output by the image encoder belongs.

[0080] Optionally, the 3D generation network is a fully connected neural network structure consisting of multiple fully connected layers and nonlinear activation layers. The structure assumes a spherical triangular mesh with a fixed topology, and inputs the shape-related feature vector output by the image encoder and the new view feature vector output by the view encoder, and outputs the offset of each vertex of the spherical triangular mesh relative to the original position.

[0081] Optionally, during the training process, in order to calculate the error between the predicted 3D model and the standard model, a differentiable rendering module is connected to the output end of the 3D construction model to improve the fit between the generated 3D model and the input sketch by reducing the difference between the 3D model rendering mask and the input sketch mask. The differentiable rendering module is characterized by approximating the non-differentiable process in the traditional rasterization pipeline so that it can calculate the differential of the pixel value of the rasterized image relative to the vertex position of the 3D model.

[0082] Based on the above embodiment, in the method,

[0083] The loss function in the training process is constructed based on the predicted 3D model error and the predicted viewing angle error, and specifically includes:

[0084] The predicted three-dimensional model error is characterized based on an error between the predicted mask and the standard mask, where the standard mask is a mask of the single hand-drawn sketch sample.

[0085] Specifically, since the function of the 3D construction model is to extract the perspective of a single hand-drawn sketch and then construct the 3D model, the function of the 3D construction model should include extracting accurate sketch perspectives and constructing accurate sketches. Figure 3 Therefore, when the entire 3D construction model is trained together, the overall loss function should contain two parts, one for measuring the accuracy of the view prediction, and the other for measuring the accuracy of the output 3D model. Therefore, the overall loss function includes the error term of the predicted view relative to the view label, and the error term of the predicted 3D model relative to the real 3D model. The error of the predicted 3D model relative to the real 3D model is characterized by their corresponding masks. Because when constructing the training data set, it is a lot of work to build a corresponding 3D model for each single hand-drawn sketch sample, and the amount of calculation for calculating the error between 3D models is even greater. Therefore, the error between 3D models is converted into the error between the masks formed by their projection onto the same view plane, that is, the single hand-drawn sketch sample in the existing training data set can be used as the data source for calculating the standard mask, reducing the number of labels required in the training data set, and the amount of calculation for the error between masks is also much lower than the amount of calculation for the error between 3D models.

[0086] Based on the above embodiment, in this method, the prediction mask includes the mask of the predicted three-dimensional model under the corresponding viewing angle label and the mask of the random predicted three-dimensional model constructed based on the random viewing angle under the random viewing angle. Specifically, the evaluation of the output predicted three-dimensional model in the three-dimensional construction model training process requires a real three-dimensional model as a standard reference value (ground truth). In order to simplify the calculation of the three-dimensional construction error, the present invention converts the model error into the mask error, that is, a differentiable rendering module is used in the three-dimensional construction model training to improve the fit between the generated three-dimensional model and the input sketch by reducing the difference between the three-dimensional model rendering mask and the input sketch mask. The differentiable rendering module is characterized in that it approximates the non-differentiable process in the traditional rasterization pipeline so that it can calculate the differential of the pixel value of the rasterized image relative to the vertex position of the three-dimensional model. It is specifically explained by the following mathematical formula:

[0087] The input hand-drawn sketch is I, the output 3D model is M, and the additional restriction is that the viewing angle of the object represented by the hand-drawn sketch is V. The process of step a is represented by function E, then E maps the input image I to the shape feature vector space and the view feature vector space The obtained shape feature vector and view feature vector are z s and z v . Let the image space be Then the function E is expressed as follows:

[0088]

[0089] The process of step b is represented by function D v , the obtained viewing angle prediction is Then D v The function is expressed as follows:

[0090]

[0091] The process of step c is represented by E v , then the process has the following properties:

[0092] D v (E v (V))=V

[0093]

[0094] The process of step d is expressed as function D, and the space of the output three-dimensional model is recorded as The function of this process is expressed as follows:

[0095]

[0096] In order to achieve the effect of each module in step ad, the present invention adopts a random perspective reinforcement method based on differentiable rendering for training, and ensures the quality of the generated model through a shape discriminator. The differentiable rendering module inputs a three-dimensional model M and a perspective V, and outputs a mask S of the three-dimensional model rendering image under the perspective. During the training process, the intersection-and-union ratio of the generated model rendering image mask and the real model rendering image mask is used as a loss function for optimization. The intersection-and-union ratio loss is expressed as follows:

[0097]

[0098] In order to achieve perspective-dependent generation, the present invention adopts a random perspective enhancement method based on differentiable rendering for training, that is, to optimize the real perspective V and a random perspective V at the same time. r The intersection-over-union loss is:

[0099]

[0100] Among them, λ r is the preset weighting factor.

[0101] Since the final 3D model generated during the use of the 3D construction model is constructed based on the predicted perspective generated by the perspective prediction network in the model, when evaluating the performance of the final 3D generation network, it is not possible to only use the perspective label for training. It is also necessary to consider whether a relatively accurate 3D model can be output when there is a certain gap between the perspective prediction and the true perspective label. Therefore, the error of the predicted 3D model includes both the error between the mask of the predicted 3D model under the corresponding perspective label and the standard mask, and the error between the mask of the random predicted 3D model constructed based on the random perspective and the standard mask under the random perspective. In the formula, D represents the three-dimensional generation function corresponding to the three-dimensional generation network, and P represents the mask function determined by the projection of the three-dimensional model at the corresponding angle.

[0102] Based on the above embodiment, in the method,

[0103] The loss function specifically includes:

[0104] the intersection-over-union ratio between the mask of the predicted 3D model at the corresponding view label and the single hand-drawn sketch sample and the standard mask, the intersection-over-union ratio between the mask of the random predicted 3D model constructed based on random view at the random view and the standard mask, and,

[0105] The cross entropy of the corresponding view label and the random view after passing through the shape discriminator.

[0106] Specifically, only using the random view enhancement method for training will cause the generated 3D model to undergo unexpected deformation. The shape discriminator can ensure the quality of the generated model while maintaining the view-dependent properties. The shape discriminator is denoted as D s , the discriminator is trained by generative adversarial method, and the cross entropy of the corresponding view label and the random view after passing through the shape discriminator is expressed by the following formula:

[0107]

[0108] In addition to the above optimization objectives, the perspective prediction module in step b and the perspective encoding module in step c should be inverse processes to each other, and the prediction result of the perspective prediction module should be close to the true value. The present invention introduces the following formula to describe the perspective prediction error term:

[0109]

[0110] Therefore, the total loss function is Among them, β1, β2 and β3 are all preset weight coefficients.

[0111] The algorithm model in step ad is trained through a random perspective reinforcement process, and the quality of the generated model is ensured by a shape discriminator. The random perspective reinforcement process obtains a real perspective feature vector and a random perspective feature vector by inputting the real value of the perspective and a random perspective into the perspective encoding module of step c respectively, and requires the model generation module in step d to output a three-dimensional model that conforms to the input sketch under the corresponding perspective when the above two perspective feature vectors are input. The shape discriminator is a fully connected neural network structure composed of multiple fully connected layers and nonlinear activation layers. The structure inputs the vertex offset obtained in step d and outputs a value indicating whether the model represented by the vertex offset is real. The structure is adversarially trained through a gradient reversal layer.

[0112] The above method can directly generate a 3D model represented by a triangular mesh from a single hand-drawn sketch, and accepts view information as an additional input to solve the ambiguity problem of hand-drawn sketches under uncertain view. This method maps the image features of the input sketch to the shape space and view space, and ensures that the output 3D model conforms to both the input hand-drawn sketch and the view constraint through a random view reinforcement training process based on differentiable rendering and a shape discriminator. This method is superior to existing methods in terms of the accuracy and controllability of 3D model generation, and can be used as a 3D content creation tool for non-professionals.

[0113] Based on the above embodiment, the method further includes:

[0114] Obtaining a specified viewing angle corresponding to the single hand-drawn sketch;

[0115] Accordingly, the step of inputting the single hand-drawn sketch into the three-dimensional construction model and outputting the three-dimensional model corresponding to the single hand-drawn sketch includes:

[0116] An image encoder inputting the single hand-drawn sketch into the three-dimensional constructed model outputs a shape feature vector to the three-dimensional constructed model;

[0117] Input the specified viewing angle into the viewing angle encoder and output the specified viewing angle vector feature to the three-dimensional construction model;

[0118] The three-dimensional construction model generates a three-dimensional model based on the shape feature vector and the specified viewing angle vector feature.

[0119] Specifically, if the user only provides a single hand-drawn sketch during use, the method predicts the perspective Generate 3D Model If the user provides additional perspective Generate a 3D model Figure 4 A flowchart of a method for generating a three-dimensional model based on a single hand-drawn sketch provided by the present invention, such as Figure 4 As shown, the method comprises the following steps:

[0120] a) The user inputs a single hand-drawn sketch, and the image encoding module obtains the view-related features and shape-related features;

[0121] b) The view-related features are passed through the view prediction module to obtain the view prediction expressed in Euler angles;

[0122] c) The perspective predicted in step b or the perspective given by the user is passed through a perspective encoding module to obtain new perspective-related features;

[0123] d) The view-related features generated in step c and the shape-related features generated in step a are input into a model generation module to obtain a three-dimensional model corresponding to the input sketch.

[0124] Among them, step c can be encoding the predicted perspective or encoding the perspective given by the user to obtain the output of new perspective-related features. Priority is stipulated here. In the case where the user does not input a specified perspective, the perspective prediction module in step b needs to be used for perspective prediction, and the perspective prediction value is used for subsequent three-dimensional model construction. However, when the user inputs a specified perspective, the specified perspective input by the user has a higher priority, and the specified perspective input by the user is used first for subsequent three-dimensional model construction.

[0125] The following describes a three-dimensional model generation device based on a single hand-drawn sketch provided by the present invention. The three-dimensional model generation device based on a single hand-drawn sketch described below and the three-dimensional model generation method based on a single hand-drawn sketch described above can refer to each other.

[0126] Figure 5 The schematic diagram of the structure of the 3D model generation device based on a single hand-drawn sketch provided by the present invention is as follows: Figure 5 As shown, it includes an acquisition unit 510 and a construction unit 520, wherein,

[0127] The acquisition unit 510 is used to acquire a single hand-drawn sketch;

[0128] The construction unit 520 is used to input the single hand-drawn sketch into a three-dimensional construction model and output a three-dimensional model corresponding to the single hand-drawn sketch;

[0129] The three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of an input single hand-drawn sketch.

[0130] The present invention provides a three-dimensional model generation device based on a single hand-drawn sketch, which obtains a single hand-drawn sketch; inputs the single hand-drawn sketch into a three-dimensional construction model, and outputs a three-dimensional model corresponding to the single hand-drawn sketch; wherein the three-dimensional construction model is obtained by training based on a single hand-drawn sketch sample and a corresponding perspective label, and the three-dimensional construction model includes a perspective prediction network for extracting the predicted perspective of the input single hand-drawn sketch. Since the perspective label is introduced when training the three-dimensional construction model, the trained three-dimensional construction model can predict the accurate perspective of the user's sketch. The accurate perspective can provide perspective information in the three-dimensional construction model so that the perspective of the constructed three-dimensional model is consistent with the perspective of the sketch, avoiding ambiguity of the three-dimensional model relative to the sketch. Therefore, the device provided by the embodiment of the present invention can predict the perspective of the hand-drawn sketch provided by the user, so that the subsequently generated three-dimensional model avoids ambiguity based on the predicted perspective.

[0131] On the basis of the above embodiment, in the device,

[0132] The network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, and the image encoder is connected to the three-dimensional generation network.

[0133] On the basis of the above embodiment, in the device,

[0134] The network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, wherein the image encoder is connected to the three-dimensional generation network, and specifically includes:

[0135] During the training process of the three-dimensional construction model,

[0136] The image encoder is used to encode the input single hand-drawn sketch sample to obtain the shape feature vector and the view feature vector and output them to the 3D generation network and the view prediction network respectively;

[0137] The view prediction network is used to predict and decode the view feature vector to obtain a predicted view, and determine a predicted view error based on the predicted view and a corresponding view label;

[0138] The view encoder is used to encode the input corresponding view label to obtain a new view feature vector and output it to the three-dimensional generation network;

[0139] The 3D generation network generates a predicted 3D model based on the new view feature vector and the shape feature vector;

[0140] A differentiable rendering module is connected to the output end of the three-dimensional construction model, wherein the differentiable rendering module generates a prediction mask based on the predicted three-dimensional model, and determines a predicted three-dimensional model error based on the predicted mask and a standard mask;

[0141] The loss function in the training process is constructed based on the predicted three-dimensional model error and the predicted viewing angle error. Based on the above embodiment, in the device,

[0142] The loss function in the training process is constructed based on the predicted 3D model error and the predicted viewing angle error, and specifically includes:

[0143] The predicted three-dimensional model error is characterized based on an error between the predicted mask and the standard mask, where the standard mask is a mask of the single hand-drawn sketch sample.

[0144] On the basis of the above embodiment, in the device,

[0145] The prediction mask includes a mask of the predicted three-dimensional model under the corresponding viewing angle label and a mask of a random predicted three-dimensional model constructed based on a random viewing angle under the random viewing angle.

[0146] On the basis of the above embodiment, in the device,

[0147] The loss function specifically includes:

[0148] the intersection-over-union ratio between the mask of the predicted 3D model at the corresponding view label and the single hand-drawn sketch sample and the standard mask, the intersection-over-union ratio between the mask of the random predicted 3D model constructed based on random view at the random view and the standard mask, and,

[0149] The cross entropy of the corresponding view label and the random view after passing through the shape discriminator.

[0150] Based on the above embodiment, the device further includes a viewing angle specifying unit, which is used to:

[0151] Obtaining a specified viewing angle corresponding to the single hand-drawn sketch;

[0152] Accordingly, the construction unit is specifically used for:

[0153] An image encoder inputting the single hand-drawn sketch into the three-dimensional constructed model outputs a shape feature vector to the three-dimensional constructed model;

[0154] Input the specified viewing angle into the viewing angle encoder and output the specified viewing angle vector feature to the three-dimensional construction model;

[0155] The three-dimensional construction model generates a three-dimensional model based on the shape feature vector and the specified viewing angle vector feature.

[0156] Figure 6 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a method for generating a three-dimensional model based on a single hand-drawn sketch, the method comprising: obtaining a single hand-drawn sketch; inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch; wherein the three-dimensional construction model is obtained by training based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of the input single hand-drawn sketch.

[0157] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0158] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the three-dimensional model generation method based on a single hand-drawn sketch provided by the above-mentioned methods, and the method includes: obtaining a single hand-drawn sketch; inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch; wherein the three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding perspective label, and the three-dimensional construction model includes a perspective prediction network for extracting the predicted perspective of the input single hand-drawn sketch.

[0159] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the three-dimensional model generation method based on a single hand-drawn sketch provided by the above-mentioned methods, the method comprising: obtaining a single hand-drawn sketch; inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch; wherein the three-dimensional construction model is trained based on a single hand-drawn sketch sample and a corresponding perspective label, and the three-dimensional construction model includes a perspective prediction network for extracting the predicted perspective of the input single hand-drawn sketch.

[0160] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a three-dimensional model based on a single hand-drawn sketch, characterized in that: include: Get a single hand-drawn sketch; Inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch; The three-dimensional construction model is obtained by training based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of an input single hand-drawn sketch; The network structure of the 3D construction model includes an image encoder, the view prediction network, a view encoder and a 3D generation network connected in sequence, and the image encoder is connected to the 3D generation network; The network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, wherein the image encoder is connected to the three-dimensional generation network, and specifically includes: During the training process of the three-dimensional construction model, The image encoder is used to encode the input single hand-drawn sketch sample to obtain the shape feature vector and the view feature vector and output them to the 3D generation network and the view prediction network respectively; The view prediction network is used to predict and decode the view feature vector to obtain a predicted view, and determine a predicted view error based on the predicted view and a corresponding view label; The view encoder is used to encode the input corresponding view label to obtain a new view feature vector and output it to the three-dimensional generation network; The 3D generation network generates a predicted 3D model based on the new view feature vector and the shape feature vector; A differentiable rendering module is connected to the output end of the three-dimensional construction model, wherein the differentiable rendering module generates a prediction mask based on the predicted three-dimensional model, and determines a predicted three-dimensional model error based on the predicted mask and a standard mask; The loss function in the training process is constructed based on the predicted three-dimensional model error and the predicted viewing angle error.

2. The method for generating a three-dimensional model based on a single hand-drawn sketch according to claim 1, characterized in that: The loss function in the training process is constructed based on the predicted 3D model error and the predicted viewing angle error, and specifically includes: The predicted three-dimensional model error is characterized based on an error between the predicted mask and the standard mask, where the standard mask is a mask of the single hand-drawn sketch sample.

3. The method for generating a three-dimensional model based on a single hand-drawn sketch according to claim 2, characterized in that: The prediction mask includes a mask of the predicted three-dimensional model under the corresponding viewing angle label and a mask of a random predicted three-dimensional model constructed based on a random viewing angle under the random viewing angle.

4. The method for generating a three-dimensional model based on a single hand-drawn sketch according to claim 3, characterized in that: The loss function specifically includes: the intersection-over-union ratio between the mask of the predicted 3D model at the corresponding view label and the single hand-drawn sketch sample and the standard mask, the intersection-over-union ratio between the mask of the random predicted 3D model constructed based on random view at the random view and the standard mask, and, The cross entropy of the corresponding view label and the random view after passing through the shape discriminator.

5. The method for generating a three-dimensional model based on a single hand-drawn sketch according to any one of claims 1 to 4, characterized in that: Also includes: Obtaining a specified viewing angle corresponding to the single hand-drawn sketch; Accordingly, the step of inputting the single hand-drawn sketch into the three-dimensional construction model and outputting the three-dimensional model corresponding to the single hand-drawn sketch includes: An image encoder inputting the single hand-drawn sketch into the three-dimensional constructed model outputs a shape feature vector to the three-dimensional constructed model; Input the specified viewing angle into the viewing angle encoder and output the specified viewing angle vector feature to the three-dimensional construction model; The three-dimensional construction model generates a three-dimensional model based on the shape feature vector and the specified viewing angle vector feature.

6. A 3D model generation device based on a single hand-drawn sketch, characterized in that: include: An acquisition unit, used for acquiring a single hand-drawn sketch; A construction unit, used for inputting the single hand-drawn sketch into a three-dimensional construction model, and outputting a three-dimensional model corresponding to the single hand-drawn sketch; The three-dimensional construction model is obtained by training based on a single hand-drawn sketch sample and a corresponding viewpoint label, and the three-dimensional construction model includes a viewpoint prediction network for extracting a predicted viewpoint of an input single hand-drawn sketch; The network structure of the 3D construction model includes an image encoder, the view prediction network, a view encoder and a 3D generation network connected in sequence, and the image encoder is connected to the 3D generation network; The network structure of the three-dimensional construction model includes an image encoder, the view prediction network, a view encoder and a three-dimensional generation network connected in sequence, wherein the image encoder is connected to the three-dimensional generation network, and specifically includes: During the training process of the three-dimensional construction model, The image encoder is used to encode the input single hand-drawn sketch sample to obtain the shape feature vector and the view feature vector and output them to the 3D generation network and the view prediction network respectively; The view prediction network is used to predict and decode the view feature vector to obtain a predicted view, and determine a predicted view error based on the predicted view and a corresponding view label; The view encoder is used to encode the input corresponding view label to obtain a new view feature vector and output it to the three-dimensional generation network; The 3D generation network generates a predicted 3D model based on the new view feature vector and the shape feature vector; A differentiable rendering module is connected to the output end of the three-dimensional construction model, wherein the differentiable rendering module generates a prediction mask based on the predicted three-dimensional model, and determines a predicted three-dimensional model error based on the predicted mask and a standard mask; The loss function in the training process is constructed based on the predicted three-dimensional model error and the predicted viewing angle error.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for generating a three-dimensional model based on a single hand-drawn sketch as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a three-dimensional model based on a single hand-drawn sketch as described in any one of claims 1 to 5 are implemented.

Images