Deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation

Abstract

The present invention relates to the field of computer vision technology, and in particular to a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation, comprising: constructing a data set; constructing a mesh generator, the mesh generator being an encoder-decoder architecture; training the mesh generator using the data set; and using the trained mesh generator to perform 3D modeling on incomplete hand-drawn images and noisy hand-drawn images to obtain a target mesh 3D model. The present invention trains the mesh generator using a 2D convolutional shape discriminator, and gradually refines the discrimination ability of the 2D convolutional shape discriminator by gradually increasing the complexity of the 2D convolutional shape discriminator and the resolution of a multi-scale contour map pair, thereby more effectively guiding the mesh generator to generate a high-quality predicted mesh 3D model, thereby improving the quality of the predicted mesh 3D model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation. Background Art

[0002] Today, 3D models have taken a central position in many manufacturing processes and product designs. This trend has created a huge demand for 3D content. Traditionally, the generation of 3D content is achieved manually by designers using computer-aided design (CAD) technology, a process that requires designers to have a high degree of skill and expertise. They must be proficient in the commands and strategies of CAD software and be able to decompose complex shapes into a series of sequential commands. This is not only challenging, but also a labor-intensive and time-consuming process. The inherent limitations of CAD technology highlight the need for alternative methods that are faster and more user-friendly 3D modeling options.

[0003] Among the existing alternative methods, hand-drawing-based 3D modeling has emerged as a promising solution in recent years. As an important tool, hand-drawing provides an intuitive means of expressing ideas, both in professional design and in daily life. Most existing modeling methods either require precise line drawings from multiple views or use a step-by-step workflow with policy knowledge, which is not user-friendly and still time-consuming. Other modeling methods use template primitives or retrieval-based methods, but these methods lack customizability.

[0004] In order to achieve the goal of fast and intuitive 3D modeling, the existing research direction is to use a single hand-drawn drawing as input to generate a complete high-fidelity 3D model. This is a challenging task due to the sparsity and ambiguity of hand-drawn drawings. Hand-drawn drawings have only one view, are sparse and abstract, lack fine boundary information when drawn, and most importantly, lack texture information for depth estimation, which leads to a lot of uncertainty in the model when learning 3D shapes. The abstract boundaries also make it difficult to interpret, because the same set of strokes may have different interpretations in the 3D world, resulting in ambiguity. In industrial applications, the incompleteness of hand-drawn drawings is also a common problem. Such incomplete hand-drawn drawings may result in the problem of unsatisfactory quality of the generated 3D model due to incomplete drawing, missing details or unclear input. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem of low quality of generating 3D models based on incomplete hand-drawn drawings in the prior art.

[0006] To solve the above technical problems, the present invention provides a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation, comprising:

[0007] Input the incomplete hand-drawn picture in the data set into the mesh generator to obtain a predicted mesh 3D model; the data set includes the incomplete hand-drawn picture and its labeled mesh 3D model;

[0008] Input the predicted mesh 3D model and the labeled mesh 3D model into a differentiable renderer to obtain a predicted 3D model contour map and a labeled 3D model contour map;

[0009] Inputting the predicted 3D model contour image and the labeled 3D model contour image into a 2D convolutional shape discriminator, and obtaining evaluation values ​​of the predicted 3D model contour image and the labeled 3D model contour image respectively;

[0010] A multi-scale mIoU loss is constructed based on the predicted 3D model contour map and the labeled 3D model contour map, and a structure-aware GAN loss is constructed based on the evaluation values ​​of the predicted 3D model contour map and the labeled 3D model contour map. Then, a loss function is constructed using the multi-scale mIoU loss, structure-aware GAN loss, flat loss, and Laplace smoothing loss to optimize the parameters of the mesh generator.

[0011] The incomplete hand-drawn image is 3D modeled using the trained mesh generator to obtain the target mesh 3D model.

[0012] Preferably, the grid generator adopts an encoder-decoder architecture, the encoder adopts a ResNet-18 architecture, and the decoder adopts a cascaded upsampling block.

[0013] Preferably, the step of inputting the predicted mesh 3D model and the labeled mesh 3D model into a differentiable renderer to obtain a predicted 3D model contour map and a labeled 3D model contour map comprises:

[0014] Feed the predicted mesh 3D model and the labeled mesh 3D model of the hand drawing into the differentiable renderer and randomly sample n camera poses , get the camera pose distribution , using the camera pose distribution Get the predicted 3D model contour images and the labeled 3D model contour images of n different perspectives.

[0015] Preferably, before the predicted 3D model contour map is input into the 2D convolution shape discriminator, the predicted 3D model contour map is subjected to feature enhancement by a stroke enhancement module.

[0016] Preferably, the construction of the data set includes:

[0017] Download synthetic data containing various mesh 3D models from the ShapeNet database and standardize the mesh 3D models to ensure consistent size and format;

[0018] By rendering the mesh 3D model, a complete hand drawing corresponding to the mesh 3D model is generated, and a part of the complete hand drawing is manually deleted using an image editing tool to generate a preliminary incomplete hand drawing sample;

[0019] Based on the preliminary incomplete hand-drawn picture samples, an image processing library or an image generation model is used to generate incomplete hand-drawn picture samples, and sampling verification is performed to remove incomplete hand-drawn picture samples that do not meet the requirements;

[0020] The incomplete hand-drawn drawing samples and their corresponding mesh 3D models are saved in a standard format to obtain a data set.

[0021] Preferably, the formula of the loss function is:

[0022] ;

[0023] in, represents the first loss function, represents the multi-scale mIoU loss, represents flat loss and Laplace smooth loss, represents the structure-aware GAN loss, Represents the weight parameter of the structure-aware GAN loss.

[0024] Preferably, the construction of the data set further includes:

[0025] Download synthetic data containing various mesh 3D models from the ShapeNet database, and standardize the mesh 3D models to ensure that the size and format are consistent; generate a complete hand-drawn map corresponding to the mesh 3D model by rendering the mesh 3D model;

[0026] Add noise to the complete hand drawing to obtain a noisy hand drawing sample, including:

[0027] Set the noise intensity and the number of noise regions, and randomly select the center point and radius of the local noise for each noise region to generate a local noise region;

[0028] A blur operation or a sharpening operation is randomly performed on each local noise region, and each local noise region is covered on the complete hand-drawn picture using a mask to obtain a noise hand-drawn picture sample;

[0029] The complete hand-drawn images, noisy hand-drawn image samples and their corresponding mesh 3D models are saved in a standard format to obtain a data set.

[0030] Preferably, the step of inputting the hand-drawn image in the data set into a grid generator to obtain a predicted grid 3D model comprises:

[0031] The noisy hand-drawn picture generates a latent view code and a latent shape code in the mesh generator, and the latent view code is input into the viewpoint decoder to obtain the predicted viewpoint information; the mesh generator generates a predicted mesh 3D model based on the predicted viewpoint information and the latent shape code.

[0032] Preferably, the loss function also includes viewpoint prediction loss and denoising loss, and the formula is:

[0033] ;

[0034] in, represents the second loss function, represents the multi-scale mIoU loss, represents flat loss and Laplace smooth loss, represents the structure-aware GAN loss, represents the weight parameter of the structure-aware GAN loss, represents the viewpoint prediction loss, represents the weight parameter of the viewpoint prediction loss, represents the denoising loss, The weight parameter representing the denoising loss;

[0035] The viewpoint prediction loss The formula is:

[0036] ;

[0037] in, Represents the real viewpoint information, Represents the predicted viewpoint information;

[0038] Denoising loss The formula is:

[0039] ;

[0040] in, Indicates a complete hand drawing. Represents the predicted 3D model contour map, and They represent the pixel points of the complete hand-drawn contour map and the predicted 3D model contour map respectively.

[0041] Preferably, constructing a multi-scale mIoU loss based on the predicted 3D model contour map and the labeled 3D model contour map includes:

[0042] The predicted 3D model contour map and the labeled 3D model contour map are downsampled multiple times to obtain multi-scale contour map pairs consisting of predicted 3D model contour maps and labeled 3D model contour maps with different resolutions. The similarity between the predicted 3D model contour map and the labeled 3D model contour map in each multi-scale contour map pair is quantified. The formula is:

[0043] ;

[0044] in, represents the total number of multi-scale contour map pairs, represents the hyperparameter of the i-th multi-scale contour map pair, which is used to balance the weights between the contour map losses of different resolutions. represents the loss of the i-th multi-scale contour map pair;

[0045] ;

[0046] in, represents the predicted 3D model contour map in the i-th multi-scale contour map pair, represents the label 3D model contour map in the i-th multi-scale contour map pair;

[0047] The structure-aware GAN loss is constructed based on the evaluation values ​​of the predicted 3D model contour map and the labeled 3D model contour map, and the formula is:

[0048] ;

[0049] ;

[0050] in, represents the camera posture, represents the camera pose distribution, represents the camera sampling angle, represents the camera sampling angle distribution, Represents the expected value of the camera sampling angle and camera pose; represents a 2D convolutional shape discriminator, represents a differentiable renderer, Represents a label grid 3D model, represents the predicted mesh 3D model, represents the evaluation value of the predicted 3D model contour map output by the 2D convolutional shape discriminator, represents the evaluation value of the labeled 3D model contour map output by the 2D convolutional shape discriminator, represents the logarithmic loss function, Represents the input of the logarithmic loss function.

[0051] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0052] The present invention discloses a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation. For incomplete hand-drawn images, in the process of training a mesh generator, a differentiable renderer is used to generate a predicted 3D model contour map and a label 3D model contour map and input them into a 2D convolutional shape discriminator. The 2D convolutional shape discriminator can effectively distinguish the authenticity and fidelity of the predicted mesh 3D model, thereby distinguishing the quality and authenticity of the predicted mesh 3D model, enhancing the recognition ability of complex shapes, and improving the accuracy and reliability of the predicted mesh 3D model. In addition, the present invention performs multiple downsampling on the predicted 3D model contour map and the label 3D model contour map, respectively, to obtain a multi-scale contour map pair composed of a predicted 3D model contour map and a label 3D model contour map of different resolutions, and then constructs a multi-scale mIoU loss. By gradually increasing the complexity of the 2D convolutional shape discriminator and the resolution of the multi-scale contour map pair, the discrimination ability of the 2D convolutional shape discriminator is gradually refined, thereby more effectively guiding the mesh generator to generate a high-quality predicted mesh 3D model and improving the quality of the predicted mesh 3D model.

[0053] The present invention provides a pen stroke enhancement module before the 2D convolution shape discriminator, which can effectively improve the mesh generator's learning of specific position features to alleviate the challenges brought by the incompleteness of incomplete hand-drawn drawings, thereby improving the accuracy and robustness of the mesh generator in generating a predictive mesh 3D model.

[0054] In addition, the present invention also trains a grid generator on a noisy data set and performs viewpoint prediction on a noisy hand-drawn picture, and adds viewpoint prediction loss and denoising loss during training, thereby improving the accuracy of 3D modeling using noisy hand-drawn pictures in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0056] Figure 1 It is a flowchart of a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation in Example 1;

[0057] Figure 2 This is an example of an incomplete hand-drawn picture sample in the Incompleto3D dataset;

[0058] Figure 3 is a visualization diagram comparing the first embodiment with the prior art, wherein Figure 3 Columns (a) and (e) are incomplete hand-drawn pictures. Figure 3 Columns (b) and (f) in the figure are the label grid 3D models. Figure 3Columns (c) and (g) in the figure are 3D models generated by Sketch2Model, a prior art method. Figure 3 Columns (d) and (h) are the predicted grid 3D models output by the present invention;

[0059] Figure 4 is an example of a noisy hand-drawn picture sample in the noise dataset;

[0060] Figure 5 4 is a flowchart of a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to Embodiment 2;

[0061] Figure 6 is a visualization diagram comparing the second embodiment with the prior art, wherein Figure 6 Column (a) in the figure is a hand-drawn picture of noise. Figure 6 Column (b) in the figure is the 3D model generated by Sketch2model for the complete hand drawing. Figure 6 The (c) column in the figure is the 3D model generated by Sketch2model for the noisy hand drawing. Figure 6 Column (d) is the predicted mesh 3D model output by the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Embodiment 1

[0063] Existing 3D modeling methods can generally be divided into two categories: interactive and end-to-end. Interactive methods involve step-by-step decomposition or specific drawing gestures and annotations, which require expertise and policy knowledge. On the other hand, end-to-end methods that utilize template primitives or retrieval-based techniques can produce satisfactory results but lack customization flexibility. Previous studies have used deep learning techniques for direct 3D model reconstruction, treating it as a single-view Figure 3 However, it is worth noting that hand-drawn modeling and traditional monocular 3D reconstruction show significant differences. Hand-drawn images are sparse, abstract, and lack texture, requiring additional clues to generate high-quality 3D shapes. In addition, incomplete hand-drawn images also pose challenges to the generation of high-quality 3D models due to their incompleteness.

[0064] Therefore, the present invention provides a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation, referring to Figure 1 As shown, the steps include:

[0065] S1. Build a dataset;

[0066] S2. construct a grid generator, wherein the grid generator is an encoder-decoder architecture;

[0067] S3, training the grid generator using the data set;

[0068] S4. Use the trained mesh generator to perform 3D modeling on the incomplete hand-drawn picture to obtain a target mesh 3D model.

[0069] Since it is difficult to find a publicly available dataset that contains both incomplete hand-drawn images and their corresponding 3D models, this embodiment adopts an alternative strategy to generate incomplete hand-drawn images with rich types based on synthetic data using some image processing tools and public image generation models such as Stable Diffusion, and named Incompleto3D. This dataset has been manually sampled and verified to ensure the quality and diversity of the dataset.

[0070] Specifically, the steps of constructing the dataset in S1 include:

[0071] S101, downloading synthetic data including various mesh 3D models from the ShapeNet database, and standardizing the mesh 3D models to ensure that they are consistent in size and format;

[0072] S102, generating a complete hand-drawn picture corresponding to the mesh 3D model by rendering the mesh 3D model, and manually deleting a part of the complete hand-drawn picture using an image editing tool to generate a preliminary incomplete hand-drawn picture sample;

[0073] S103. Based on the preliminary incomplete hand-drawn picture samples, write a Python script and use an image processing library such as OpenCV to automatically generate incomplete hand-drawn picture samples, or generate incomplete hand-drawn picture samples by inputting descriptive text and using an image generation model such as Stable Diffusion, and perform manual sampling verification to ensure the quality and rationality of each incomplete hand-drawn picture, and remove incomplete hand-drawn picture samples that do not meet the requirements; the incomplete hand-drawn picture samples that do not meet the requirements are defined as hand-drawn picture samples whose incomplete areas are larger than a preset area;

[0074] S104, saving the incomplete hand-drawn drawing samples and their corresponding mesh 3D models in a standard format to obtain an Incompleto3D dataset.

[0075] The Incompleto3D dataset contains a variety of incomplete hand-drawn images, aiming to promote technological progress in processing and generating incomplete hand-drawn images. It provides a rich research resource for the fields of computer vision and image processing, and provides important technical support for the research on incomplete hand-drawn images, especially in image restoration and completion, incomplete image recognition, generative adversarial network research, and augmented reality and virtual reality applications. It has important application value. Figure 2 This is an example of an incomplete hand-drawn picture sample in the Incompleto3D dataset.

[0076] Specifically, the specific steps of S3 include:

[0077] S301, input the incomplete hand-drawn picture into a mesh generator to obtain a predicted mesh 3D model.

[0078] The incomplete hand drawing Encoder input to the mesh generator , and obtain the latent shape code The latent shape code Taking into account factors such as semantic categories and concept shapes, the essential characteristics of hand drawings can be captured at a higher level.

[0079] Latent Shape Coding Through the decoder , get the predicted grid 3D model .

[0080] In this embodiment, the decoder uses a cascade upsampling block. The cascade upsampling block gradually infers 3D shape information by calculating the vertex offsets of the template mesh and performing corresponding deformations, gradually improving the spatial resolution, rather than using an MLP-like structure to predict point-by-point positions. The application of the cascade upsampling block improves the shape representation and generates a predicted mesh 3D model. , ensuring that the resulting model is both highly detailed and flawless, making it ready for use in manufacturing software.

[0081] The process of the grid generator can be expressed as:

[0082] ;

[0083] ;

[0084] S302: The hand-drawn predicted grid 3D model and label mesh 3D models Enter the Differentiable Renderer , get the predicted 3D model contour map and the labeled 3D model contour map.

[0085] To supervise the generation process, a differentiable renderer is used To render the generated prediction mesh 3D model and label mesh 3D models , get the predicted 3D model contour map and label 3D model outline The mesh generator is trained in an end-to-end fashion, where supervision is provided by comparing the rendered outlines with the ground truth. The gradients of the differentiable renderer are approximated and used to guide the training process, ensuring that the generated meshes align with the desired outlines. The rendering process can be expressed as:

[0086] ;

[0087] ;

[0088] S303 , input the predicted 3D model contour map and the labeled 3D model contour map into a 2D convolutional shape discriminator, and obtain evaluation values ​​of the predicted 3D model contour map and the labeled 3D model contour map respectively.

[0089] During training, the 2D convolutional shape discriminator combines 3D shapes from real datasets to encourage the mesh generator to produce realistic 3D shapes, which helps maintain efficiency during inference. To discern the quality and realism of the generated shapes, the 2D convolutional shape discriminator predicts a 3D model contour map and label 3D model outline By incorporating two contour maps during the evaluation process, the 2D convolutional shape discriminator becomes able to effectively distinguish the authenticity and fidelity of the generated shapes, achieving a more comprehensive evaluation of the quality and authenticity of the shapes of the generated 3D models.

[0090] In addition, a single contour map is not enough to encapsulate all the information of the 3D mesh, because the result of 3D model generation is a 3D model that can be viewed from different angles. , which is different from the translation task of 2D images. While the contour constraint of the 2D convolutional shape discriminator ensures that the generated model is aligned with the viewpoint of the input hand drawing, it does not inherently ensure that the model produces realistic and consistent results across various viewpoints.

[0091] To overcome this limitation, this embodiment randomly samples n camera poses when generating a contour map using a differentiable renderer. , get the camera pose distribution , using the camera pose distribution It is possible to collect more information about three-dimensional objects and obtain predicted 3D model contour maps and labeled 3D model contour maps from n different perspectives.

[0092] Random pose sampling can force the network to learn to generate reasonable 3D fine structure shapes independent of viewpoints. Multi-view silhouettes contain valuable geometric information related to 3D objects. Silhouette maps obtained from viewpoints with different camera poses can more fully understand the shape and structure of objects, which helps to more accurately and detailed 3D reconstruction. In this embodiment, a differentiable renderer is used to generate 3D models from labeled meshes. Generate label 3D model outline , from the predicted mesh 3D model Generate predicted 3D model contour map , to implement the rendering process.

[0093] The predicted 3D model contour map obtained by the differentiable renderer and label 3D model outline Feeding into a 2D convolutional shape discriminator, the mesh generator is trained to learn the geometric structure of objects across multiple views, which helps ensure that the generated predicted mesh 3D models are not only plausible but also high-fidelity in terms of details.

[0094] Due to the incompleteness of hand-drawn images and the limited information obtained through single-view contour constraints, relying solely on these constraints will limit the complexity of effectively capturing 3D models. Traditional 3D reconstruction relies on rich textures and different features appearing in the image to estimate depth. Incomplete hand-drawn images lack these features and are limited to a single color, which makes it difficult to accurately predict depth in this modeling task. In order to capture the details and true structure of objects, it is necessary to add additional clues to the modeling process. Previous studies have proposed a two-stage post-refinement process in which a rough shape is initially generated and then optimized to align with the contour. This approach allows the initial shape to be enhanced and refined, thereby improving accuracy and realism. However, this approach is not efficient enough for real-time applications. In contrast, the goal of the present invention is to develop a more efficient end-to-end solution that can quickly generate high-fidelity 3D meshes. To this end, this embodiment introduces a pen enhancement module (SEM) before the 2D convolutional shape discriminator to make full use of monochrome information to perform feature enhancement on the predicted 3D model contour map.

[0095] Preferably, before the predicted 3D model contour map is input into the 2D convolution shape discriminator, the predicted 3D model contour map is subjected to feature enhancement by a stroke enhancement module.

[0096] The pen tip enhancement module consists of an inspired position-aware attention unit and a post-processing unit. The position-aware attention unit integrates extensive contextual information into local features to capture spatial correlations. The post-processing unit is used to manipulate features from the position-aware attention module using a series of convolutions, which smoothly merges these enhanced features into the original features before attention in an element-wise manner. The pen tip enhancement module effectively enhances the learning of features at specific locations, especially the learning of boundary information.

[0097] Specifically, the 3D model contour map is predicted as the local feature A, two local features B and C are generated through a convolution layer, and then the local feature D is generated by another convolution layer. After the local feature C is transposed, a matrix multiplication operation is performed with the local feature B, and then the attention map S is generated through the softmax layer. The attention map S effectively improves the utilization rate of the key information represented by the contour. The attention map S is used to generate the output feature by calculating the weighted sum of the local feature A and the features from all positions, which is calculated based on the attention weight determined by the attention map S. The data process of the pen tip enhancement module can be expressed as:

[0098] ;

[0099] in, Represents the predicted 3D model contour map, Indicates the pen stroke enhancement module. Represents the enhanced features output by the pen stroke enhancement module.

[0100] S304: construct a loss function to optimize the parameters of the grid generator.

[0101] In order to train the network effectively, the loss function includes multi-scale mIoU loss, flat loss, Laplace smooth loss and structure-aware GAN loss, and the formula is expressed as:

[0102] ;

[0103] in, represents the loss function, represents the multi-scale mIoU loss, represents the flat loss and Laplace smooth loss, which are used to enhance the realism and visual quality of the predicted mesh 3D model. represents the structure-aware GAN loss, Represents the weight parameter of the structure-aware GAN loss.

[0104] The multi-scale mIoU loss is to downsample the predicted 3D model contour map and the label 3D model contour map multiple times to obtain a multi-scale contour map pair consisting of the predicted 3D model contour map and the label 3D model contour map of different resolutions, and quantify the similarity between the predicted 3D model contour map and the label 3D model contour map in each multi-scale contour map pair. In order to improve efficiency, the resolution of the multi-scale contour map is gradually increased during the training process. The formula is expressed as:

[0105] ;

[0106] in, represents the total number of multi-scale contour map pairs, represents the hyperparameter of the i-th multi-scale contour map pair, which is used to balance the weights between the contour map losses of different resolutions. represents the loss of the i-th multi-scale contour map pair, and the formula is:

[0107] ;

[0108] in, represents the predicted 3D model contour map in the i-th multi-scale contour map pair, Represents the labeled 3D model silhouette in the i-th multi-scale silhouette pair.

[0109] The structure-aware GAN loss adopts the non-saturated GAN loss, and the formula is:

[0110] ;

[0111] ;

[0112] in, represents the camera posture, represents the camera pose distribution, represents the camera sampling angle, represents the camera sampling angle distribution, Represents the expected value of the camera sampling angle and camera pose; represents a 2D convolutional shape discriminator, represents a differentiable renderer, Represents a label grid 3D model, represents the predicted mesh 3D model, represents the evaluation value of the predicted 3D model contour map output by the 2D convolutional shape discriminator, represents the evaluation value of the labeled 3D model contour map output by the 2D convolutional shape discriminator, represents the logarithmic loss function, Represents the input of the logarithmic loss function.

[0113] In summary, the present invention proposes a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation. For incomplete hand-drawn pictures, in the process of training the mesh generator, a plurality of camera poses are randomly extracted when a contour map is generated by a differentiable renderer, and then the predicted 3D model contour maps and the labeled 3D model contour maps of multiple different perspectives are input into a 2D convolutional shape discriminator. Multi-view sampling is used to enable the 2D convolutional shape discriminator to effectively distinguish the authenticity and fidelity of the predicted mesh 3D model, thereby identifying the quality and authenticity of the predicted mesh 3D model, enhancing the recognition ability of complex shapes, and improving the accuracy and reliability of the generated model.

[0114] In addition, the present invention introduces a stroke enhancement module before the 2D convolutional shape discriminator, which can effectively improve the learning of the grid generator on the features of specific positions to alleviate the challenges brought by the incompleteness of hand-drawn drawings, thereby improving the accuracy and robustness of image processing.

[0115] In the process of training the mesh generator, the predicted 3D model contour map and the labeled 3D model contour map are downsampled multiple times respectively to obtain multi-scale contour map pairs consisting of predicted 3D model contour maps and labeled 3D model contour maps of different resolutions, and then the multi-scale mIoU loss is constructed. By gradually increasing the complexity of the 2D convolutional shape discriminator and the resolution of the multi-scale contour map pairs, the discrimination ability of the 2D convolutional shape discriminator is gradually refined, thereby more effectively guiding the mesh generator to generate high-quality predicted mesh 3D models.

[0116] To verify the effect of a deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation proposed by the present invention, this embodiment uses the ResNet-18 architecture as an encoder to extract image features. In order to render the contour, the SoftRas algorithm is used as a renderer. For the standard view, the 3D objects are uniformly oriented with an elevation angle of 0 degrees and an azimuth angle of 0 degrees. In addition, they always maintain a constant distance from the camera. The ground truth viewpoint is used in the rendering process. For each predicted mesh 3D model and label mesh 3D model, n=2 is used for rendering. The contour with the corresponding ground truth is used to calculate the IoU loss. The initial learning rate of the Adam optimizer is 1e-4, multiplied by 0.3 every 800 epochs. The β value is set to 0.9 and 0.999. The total number of training times is set to 2000epochs. The model is trained separately for each class of the dataset. The model is trained and evaluated on four NVIDIA GeForce RTX3090 GPUs.

[0117] The experimental results of the existing method Sketch2model and the present invention on the Incompleto3D dataset are shown in Table 1.

[0118] Table 1. Quality evaluation on the Incompleto3D dataset

[0119]

[0120] As can be seen from Table 1, this example compares the present invention with the existing state-of-the-art techniques through quantitative evaluation, and the results further demonstrate the superiority of this method in generating models with enhanced quality and structural fidelity.

[0121] In addition, this embodiment is also tested under different damage levels. Figure 3 is a visualization diagram comparing the present invention with the prior art, wherein Figure 3 Columns (a) and (e) are incomplete hand-drawn pictures. Figure 3 Columns (b) and (f) in the figure are the label grid 3D models. Figure 3 Columns (c) and (g) in the figure are 3D models generated by Sketch2Model, a prior art method. Figure 3 Columns (d) and (h) in the figure are the predicted mesh 3D models output by the present invention. With the participation of the 2D convolutional shape discriminator and the pen tip enhancement module, the damaged sketches can be successfully processed without obvious performance loss, showing high robustness in real-world industrial applications.

[0122] From the above, it can be seen that existing methods such as sketch2model use 3D shape discriminators, which have high computational complexity and require high time costs for model training and reasoning. In contrast, the present invention uses a 2D convolutional shape discriminator, which greatly reduces the cost of training and reasoning. The 2D convolutional shape discriminator uses a 2D contour map as input. A single contour map lacks structural perception of three-dimensional information. For this reason, the present invention uses a multi-view sampling method to greatly improve the structural perception ability of the model. In response to the challenge of incomplete hand-drawn pictures, the present invention uses a pen tip enhancement module, which is organically combined with a 2D convolutional shape discriminator. By refining the features of the contour map and improving the ability to capture basic stroke information, it can play a complementary role, thereby obtaining more accurate and detailed structural features. Finally, the present invention adopts a progressive generation strategy, which gradually refines the discriminant's discrimination ability by gradually increasing the complexity of the discriminator and the resolution of the input data, thereby more effectively guiding the generator to generate high-quality images.

[0123] Therefore, the present invention has good customizability, novice user-friendliness, fast result reconstruction capability, and incomplete perception capability, and especially shows good reconstruction capability in the case of incomplete hand-drawn images. Embodiment 2

[0124] This embodiment constructs a noise data set and uses the noise data set to train the grid generator.

[0125] The steps of constructing the noise data set include:

[0126] Download synthetic data containing various mesh 3D models from the ShapeNet database, and standardize the mesh 3D models to ensure that the size and format are consistent; generate a complete hand-drawn map corresponding to the mesh 3D model by rendering the mesh 3D model;

[0127] Add noise to the complete hand drawing to obtain a noisy hand drawing sample, including:

[0128] Set the noise intensity and the number of noise regions, and randomly select the center point and radius of the local noise for each noise region to generate a local noise region;

[0129] A blur operation or a sharpening operation is randomly performed on each local noise region, and each local noise region is covered on the complete hand-drawn picture using a mask to obtain a noise hand-drawn picture sample; the blur operation is used to simulate edge softening caused by insufficient line pressure, and the sharpening operation is used to simulate line breakage or excessive depth of the brush stroke caused by excessive line pressure;

[0130] The complete hand-drawn image, noisy hand-drawn image samples and their corresponding mesh 3D models are saved in a standard format to obtain a noisy dataset.

[0131] Figure 4 It is an example of a noisy hand-drawn picture sample in the noise dataset.

[0132] Based on the training of the mesh generator in the first embodiment, this embodiment also predicts the viewpoint information when predicting the mesh 3D model in S301. Figure 5 As shown, the specific steps include:

[0133] The noisy hand-drawn picture is passed through the grid generator encoder to obtain the image feature vector, which is further mapped to the potential viewpoint space and potential shape space to obtain the potential view encoding and potential shape encoding, which are expressed as follows:

[0134] ;

[0135] in, represents the latent view encoding, represents the latent shape encoding, A hand-drawn picture of noise. An encoder representing a mesh generator;

[0136] The potential view code is input into the view decoder to obtain the predicted view information, the formula is:

[0137] ;

[0138] in, represents the predicted viewpoint information, represents a viewpoint decoder;

[0139] The mesh generator generates a predicted mesh 3D model based on the predicted viewpoint information and latent shape encoding, and the formula is:

[0140] ;

[0141] in, represents the predicted mesh 3D model, represents the decoder of the grid generator, Represents the viewpoint encoder.

[0142] Based on the noisy hand-drawn image and the predicted viewpoint information, this embodiment adds viewpoint prediction loss and denoising loss on the basis of the loss function in the first embodiment, and the formula is expressed as follows:

[0143] ;

[0144] in, represents the loss function of this embodiment, represents the multi-scale mIoU loss, represents the flat loss and Laplace smooth loss, which are used to enhance the realism and visual quality of the predicted mesh 3D model. represents the structure-aware GAN loss, represents the weight parameter of the structure-aware GAN loss, represents the viewpoint prediction loss, represents the weight parameter of the viewpoint prediction loss, represents the denoising loss, The weight parameter representing the denoising loss.

[0145] The viewpoint prediction loss The L2 loss is used to constrain the predicted viewpoint information and the real viewpoint information. The formula is:

[0146] ;

[0147] in, Represents the real viewpoint information, Represents the predicted viewpoint information.

[0148] The denoising loss The minimum bidirectional two-dimensional chamfer loss between the contour map of the complete hand drawing and the contour map of the predicted 3D model is:

[0149] ;

[0150] in, Indicates a complete hand drawing. Represents the predicted 3D model contour map, and The pixels of the contour map of the complete hand drawing and the predicted 3D model contour map are respectively represented. The contour map of the complete hand drawing is generated based on the complete hand drawing and the real viewpoint information.

[0151] The experimental results of the existing method Sketch2model and the present invention on the noisy dataset are shown in Table 2.

[0152] Table 2. Quality evaluation on the noisy dataset

[0153]

[0154] As can be seen from Table 2, this example compares the present invention with the existing state-of-the-art techniques on a noisy dataset through quantitative evaluation, and the results further demonstrate the superiority of this method in generating models with enhanced quality and structural fidelity.

[0155] In addition, this embodiment is also tested under different noise conditions. Figure 6 is a visualization diagram comparing this embodiment with the prior art, wherein Figure 6 Column (a) in the figure is a hand-drawn picture of noise. Figure 6 Column (b) in the figure is the 3D model generated by Sketch2model for the complete hand drawing. Figure 6 The (c) column in the figure is the 3D model generated by Sketch2model for the noisy hand drawing. Figure 6 Column (d) in FIG. 1 is the predicted grid 3D model output by the present invention. Figure 6 It can be seen that the present invention can successfully process noisy hand-drawn images without obvious performance loss, showing high robustness in real-world industrial applications.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0160] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation, characterized in that: include: Inputting the hand-drawn picture in the data set into the grid generator to obtain a predicted grid 3D model; the data set includes the hand-drawn picture and its labeled grid 3D model; The predicted mesh 3D model and the labeled mesh 3D model are input into the differentiable renderer to obtain the predicted 3D model contour map and the labeled 3D model contour map, including: Feed the predicted mesh 3D model and the labeled mesh 3D model of the hand drawing into the differentiable renderer and randomly sample n camera poses , get the camera pose distribution , using the camera pose distribution Obtain predicted 3D model contour images and labeled 3D model contour images from n different perspectives; Inputting the predicted 3D model contour image and the labeled 3D model contour image into a 2D convolutional shape discriminator, and obtaining evaluation values ​​of the predicted 3D model contour image and the labeled 3D model contour image respectively; A multi-scale mIoU loss is constructed based on the predicted 3D model contour map and the labeled 3D model contour map, and a structure-aware GAN loss is constructed based on the evaluation values ​​of the predicted 3D model contour map and the labeled 3D model contour map. Then, a loss function is constructed using the multi-scale mIoU loss, structure-aware GAN loss, flat loss, and Laplace smoothing loss to optimize the parameters of the mesh generator. The hand-drawn image is 3D modeled using the trained mesh generator to obtain the target mesh 3D model.

2. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 1, characterized in that: The grid generator adopts an encoder-decoder architecture, the encoder adopts the ResNet-18 architecture, and the decoder adopts a cascaded upsampling block.

3. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 1, characterized in that: Before the predicted 3D model contour map is input into the 2D convolution shape discriminator, the predicted 3D model contour map is subjected to feature enhancement by the pen tip enhancement module.

4. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 1, characterized in that: The construction of the dataset includes: Download synthetic data containing various mesh 3D models from the ShapeNet database and standardize the mesh 3D models to ensure consistent size and format; By rendering the mesh 3D model, a complete hand drawing corresponding to the mesh 3D model is generated, and a part of the complete hand drawing is manually deleted using an image editing tool to generate a preliminary incomplete hand drawing sample; Based on the preliminary incomplete hand-drawn picture samples, an image processing library or an image generation model is used to generate incomplete hand-drawn picture samples, and sampling verification is performed to remove incomplete hand-drawn picture samples that do not meet the requirements; The incomplete hand-drawn drawing samples and their corresponding mesh 3D models are saved in a standard format to obtain a data set.

5. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 4, characterized in that: The formula of the loss function is: ； in, represents the first loss function, represents the multi-scale mIoU loss, represents flat loss and Laplace smooth loss, represents the structure-aware GAN loss, Represents the weight parameter of the structure-aware GAN loss.

6. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 1, characterized in that: The construction of the data set also includes: Download synthetic data containing various mesh 3D models from the ShapeNet database, and standardize the mesh 3D models to ensure that the size and format are consistent; generate a complete hand-drawn map corresponding to the mesh 3D model by rendering the mesh 3D model; Add noise to the complete hand drawing to obtain a noisy hand drawing sample, including: Set the noise intensity and the number of noise regions, and randomly select the center point and radius of the local noise for each noise region to generate a local noise region; A blur operation or a sharpening operation is randomly performed on each local noise region, and each local noise region is covered on the complete hand-drawn picture using a mask to obtain a noise hand-drawn picture sample; The complete hand-drawn images, noisy hand-drawn image samples and their corresponding mesh 3D models are saved in a standard format to obtain a data set.

7. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 6, characterized in that: The method of inputting the hand-drawn image in the data set into the grid generator to obtain the predicted grid 3D model includes: The noisy hand-drawn picture generates a latent view code and a latent shape code in the mesh generator, and the latent view code is input into the viewpoint decoder to obtain the predicted viewpoint information; the mesh generator generates a predicted mesh 3D model based on the predicted viewpoint information and the latent shape code.

8. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 7, characterized in that: The loss function also includes viewpoint prediction loss and denoising loss, and the formula is: ； in, represents the second loss function, represents the multi-scale mIoU loss, represents flat loss and Laplace smooth loss, represents the structure-aware GAN loss, represents the weight parameter of the structure-aware GAN loss, represents the viewpoint prediction loss, represents the weight parameter of the viewpoint prediction loss, represents the denoising loss, The weight parameter representing the denoising loss; The viewpoint prediction loss The formula is: ； in, Represents the real viewpoint information, Represents the predicted viewpoint information; Denoising loss The formula is: ； in, Indicates a complete hand drawing. Represents the predicted 3D model contour map, and They represent the pixel points of the complete hand-drawn contour map and the predicted 3D model contour map respectively.

9. The deep intelligent 3D reconstruction method based on multi-view sampling and progressive generation according to claim 1, characterized in that: The multi-scale mIoU loss is constructed based on the predicted 3D model contour map and the labeled 3D model contour map, including: The predicted 3D model contour map and the labeled 3D model contour map are downsampled multiple times to obtain multi-scale contour map pairs consisting of predicted 3D model contour maps and labeled 3D model contour maps with different resolutions. The similarity between the predicted 3D model contour map and the labeled 3D model contour map in each multi-scale contour map pair is quantified. The formula is: ； in, represents the total number of multi-scale contour map pairs, represents the hyperparameter of the i-th multi-scale contour map pair, which is used to balance the weights between the contour map losses of different resolutions. represents the loss of the i-th multi-scale contour map pair; ； in, represents the predicted 3D model contour map in the i-th multi-scale contour map pair, represents the label 3D model contour map in the i-th multi-scale contour map pair; The structure-aware GAN loss is constructed based on the evaluation values ​​of the predicted 3D model contour map and the labeled 3D model contour map, and the formula is: ； ； in, represents the camera posture, represents the camera pose distribution, represents the camera sampling angle, represents the camera sampling angle distribution, Represents the expected value of the camera sampling angle and camera pose; represents a 2D convolutional shape discriminator, represents a differentiable renderer, Represents a label grid 3D model, represents the predicted mesh 3D model, represents the evaluation value of the predicted 3D model contour map output by the 2D convolutional shape discriminator, represents the evaluation value of the labeled 3D model contour map output by the 2D convolutional shape discriminator, represents the logarithmic loss function, Represents the input of the logarithmic loss function.

Images