Single-view three-dimensional object reconstruction method based on deep learning

Abstract

The invention discloses a single-view three-dimensional object reconstruction method based on deep learning, and relates to the field of computer vision, and the method comprises the steps: S1, constructing an object reconstruction model which comprises an image encoder, a generator G and a discriminator D. The image encoder is used for extracting high-level features from a single-view image, and the discriminator G is used for discriminating the high-level features from the single-view image; the high-level features are converted into potential vectors through a re-parameterization technique; s2, acquiring a training data set; s3, importing the training data set into an object reconstruction model, and performing training optimization on the object reconstruction model to obtain an optimized object reconstruction model; s4, acquiring data to be reconstructed; s5, performing three-dimensional object reconstruction on the to-be-reconstructed data by using the optimized object reconstruction model; an advanced neural network architecture, a generative adversarial network (GAN) and an image encoder (VAE) are adopted, and a spatial attention mechanism is added in the GAN to mine deep features and spatial relationships in a single-view image, so that the precision and robustness of 3D object reconstruction are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a single-view deep learning-based Figure 3 dimensional object reconstruction method. Background Art

[0002] Reconstructing three-dimensional objects from images is one of the main tasks of computer vision, and its purpose is to study how to generate corresponding three-dimensional object structures from a single image or multiple images. Figure 3 3D object reconstruction refers to the process of generating a 3D model of an object from a single 2D image. Unlike 3D object reconstruction, which requires multiple images from different angles, this task only relies on one image. Although multi-view images can provide richer 3D information and more accurate 3D reconstruction, in many practical application scenarios, only a single image can be obtained, and compared with multi-view methods, single-view images are more complex and time-consuming. Figure 3 The reconstruction of 3D objects requires less data processing and lower computing and storage requirements.

[0003] In the field of autonomous driving, the autonomous driving system needs to accurately perceive and understand the surrounding environment to ensure safe driving. Figure 3 3D object reconstruction technology can extract 3D information of an object from a single camera view. By reconstructing the 3D model of the object, the system can more accurately identify and classify surrounding vehicles, pedestrians, obstacles, etc. Accurate 3D object reconstruction can help the system estimate the distance, size and shape of the object, so as to better plan the path and avoid obstacles. Compared with multi-camera or LiDAR systems, single-camera systems have the advantages of low cost and easy installation, and can achieve faster processing speed to meet the real-time requirements of autonomous driving systems. In autonomous driving systems, redundancy and reliability are key factors. Single-view Figure 3 3D object reconstruction can complement LiDAR and multi-view systems to improve the reliability of environmental perception. Figure 3 3D reconstruction can provide basic 3D information to ensure the basic functionality of the system. Figure 3 3D reconstruction can help the autonomous driving system deal with certain visual occlusion problems. By inferring the 3D information of hidden areas in the image, the system can better predict possible obstacles or pedestrians. Figure 3 Maintenance and reconstruction can also provide useful information to improve the safety of driving decisions.

[0004] With the continuous rise of deep learning in recent years, using deep neural networks to reconstruct 3D objects from a single 2D image has become a hot topic. The 3D object reconstruction method based on deep learning is to train the neural network to learn the mapping relationship between 2D images and 3D objects. Deep learning models can better understand and reconstruct complex 3D objects, thus promoting the development of single-view Figure 3 Advances and applications of 3D object reconstruction technology.

[0005] Choy et al. proposed a new architecture called 3D Recursive Reconstruction Neural Network (3D-R2N2), which is based on an extension of the standard LSTM framework and can uniformly handle single and multi-view 3D reconstruction. The network learns the mapping from object images to their underlying 3D shapes from a large amount of synthetic data, and ultimately outputs the reconstruction of the object in the form of a 3D occupancy grid. Although 3D-R2N2 performs well in single-view reconstruction and with high texture sparsity, the method has a strong dependence on a large amount of synthetic data, has specific assumptions about the appearance of objects, and requires high computational resources. In addition, the output accuracy of 3D-R2N2 is limited when dealing with challenges such as large baselines and low textures.

[0006] To solve the problem of long-term memory loss in the above recurrent neural network, Xie et al. proposed a single-view and multi-view neural network called Pix2Vox. Figure 3 dimensional reconstruction framework. It adopts an encoder-decoder structure to generate a coarse 3D volume from each input image. The framework introduces a context-aware fusion module that can adaptively select high-quality reconstructions of each part (such as a table leg) from multiple coarse 3D volumes and fuse them into a final 3D volume. Finally, the fused 3D volume is further refined by a refiner to generate the final output. Due to the large number of parameters and complex network structure of the network, its computational complexity is high. Although Pix2Vox performs well in multi-view situations, it is still limited by the number of views and viewing angles. If the view coverage is insufficient or there is a large gap between the viewing angles, the reconstruction quality may be affected.

[0007] Different from the above methods that unify single view and multi-view in one network framework, Fan et al. proposed a method based on point set generation network for 3D object reconstruction from a single image. Unlike the traditional volume grid or image set representation, this method directly generates point cloud coordinates, converts the input image into a 3D point set through the encoder-predictor network structure, and uses CD (Chamfer Distance) and EMD (Earth Mover's Distance) distances as loss functions to better capture the uncertainty of the shape. This method performs well in maintaining geometric transformation invariance and generating multiple candidate predictions. However, it has limitations in capturing local high-frequency details and has high computational complexity, especially the calculation of EMD may affect the training and inference speed.

[0008] Girdhar et al. proposed a network framework called TL embedding, which combines autoencoders and convolutional networks to learn an embedding space that can generate new 3D objects and make predictions from 2D images. The autoencoder part ensures the generative ability of the embedding representation, while the convolutional network part ensures its predictive ability. The TL embedding network shows strong multi-task processing capabilities and rich 3D object representation capabilities in 3D object reconstruction. The innovative architecture combining autoencoders and convolutional networks provides high-quality generation and prediction effects. However, this method has high computational complexity, strong dependence on a large amount of training data, the risk of overfitting, complex architecture design, and long training time.

[0009] Wu et al. proposed a network framework called 3D-VAE-GAN, which uses volumetric convolutional networks and generative adversarial networks to generate 3D objects from probabilistic space and obtain effective object recognition features under unsupervised learning. The network uses adversarial criteria to capture complex object structures and details, and generates novel objects from probabilistic space without predefined models. However, the training process of generative adversarial networks is prone to mode collapse and insufficient diversity of generated objects. Although the 3D-VAE-GAN framework is able to generate shapes and overall structures, there are still challenges in capturing details. The generated 3D objects may lack fine surface textures, local details, or microstructures, but these details are crucial to making the objects look more realistic and refined. The training of 3D-VAE-GAN is more complex and time-consuming, and there are problems with convergence difficulties and training instability.

[0010] In the field of computer vision and autonomous driving, single-view Figure 3 3D object reconstruction faces many challenges. Traditional methods often rely on multi-view data or manual feature extraction and cannot effectively utilize single-view information, resulting in insufficient reconstruction accuracy and loss of details. In addition, existing deep learning models are not good at processing single-view images. Figure 3When performing D reconstruction tasks, the geometric features and contextual information of the object are often not fully considered, resulting in poor adaptability of the model to different object types. As the complexity and diversity of the input image increase, the reconstruction performance of the model will also decrease significantly. Summary of the invention

[0011] The purpose of the present invention is to design a single-view deep learning-based Figure 3 dimensional object reconstruction method.

[0012] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0013] Single-view based on deep learning Figure 3 dimensional object reconstruction method, including:

[0014] S1. Construct an object reconstruction model, which includes an image encoder, a generator G and a discriminator D. The image encoder is used to extract high-level features from single-view images and convert the high-level features into latent vectors through reparameterization techniques; the generator uses the latent vector to perform preliminary 3D reconstruction of the object; the discriminator evaluates the generated results to ensure that the generated 3D model has high authenticity and detail accuracy; the generator includes n 3D deconvolution layers and activation layers, each 3D deconvolution layer is added with a batch normalization layer, a ReLU layer, a 3D residual block and a 3D spatial attention layer, and the output of the activation layer is the generated result; the discriminator includes n 3D convolution layers and output layers, each 3D deconvolution layer is added with a batch normalization layer, a ReLU layer, a 3D residual block and a 3D spatial attention layer, n is a positive integer, and the 3D spatial attention layer is used to mine deep features and spatial relationships in single-view images;

[0015] S2, obtain training data set;

[0016] S3, importing the training data set into the object reconstruction model, and training and optimizing the object reconstruction model to obtain an optimized object reconstruction model;

[0017] S4, obtaining data to be reconstructed;

[0018] S5. Reconstruct the three-dimensional object using the optimized object reconstruction model.

[0019] The beneficial effects of the present invention are: adopting an advanced neural network architecture, generating a confrontation network GAN and an image encoder VAE, and adding a spatial attention mechanism in the GAN network to mine deep features and spatial relationships in single-view images, thereby improving the accuracy and robustness of 3D object reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The present invention is based on single-view deep learning Figure 3 Flowchart of the method for dimensional object reconstruction;

[0021] Figure 2 The present invention is based on single-view deep learning Figure 3 Network architecture diagram of the 3D object reconstruction method;

[0022] Figure 3 This is a partial image display of the dataset;

[0023] Figure 4 This is a display of images generated by the network of some data sets;

[0024] Figure 5 It is the experimental training data display;

[0025] Figure 6 This is a comparison chart of the experimental results of 3D object generation effect;

[0026] Figure 7 This is a comparison chart of the experimental results of the intersection-and-union ratio indicator for 3D object generation. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0030] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0031] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0032] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0034] like Figure 1 , Figure 2 As shown, single-view deep learning-based Figure 3 dimensional object reconstruction method, including:

[0035] S1. Build an object reconstruction model, which includes an image encoder, a generator G, and a discriminator D. The image encoder is used to extract high-level features from a single-view image and convert the high-level features into latent vectors through reparameterization techniques; the generator uses the latent vector to perform preliminary 3D reconstruction of the object; the discriminator evaluates the generated results to ensure that the generated 3D model has high authenticity and detail accuracy;

[0036] The generator consists of n 3D deconvolution layers and activation layers. Each 3D deconvolution layer adds a batch normalization layer, a ReLU layer, a 3D residual block, and a three-dimensional spatial attention layer. The output of the activation layer is the generation result.

[0037] The discriminator consists of n 3D convolutional layers and output layers. Each 3D deconvolutional layer adds a batch normalization layer, a ReLU layer, a 3D residual block, and a 3D spatial attention layer. n is a positive integer. The 3D spatial attention layer is used to mine deep features and spatial relationships in single-view images.

[0038] The image encoder consists of five convolutional layers and transformation layers from input to output. The kernel size of the convolutional layer is 5×5 and the stride is 2. There are batch normalization layers and ReLU layers in the middle. The transformation layer converts high-level features into a 200-dimensional latent vector z through the heavy parameter technique.

[0039] S2. Obtain training data.

[0040] S3. Import the training data set into the object reconstruction model, use the loss function to train and optimize the object reconstruction model, and obtain the optimized object reconstruction model; the loss function includes adversarial loss, reconstruction loss and KL divergence loss. The reconstruction loss is used to ensure that the generated 3D model is consistent with the input image, the adversarial loss is used to improve the authenticity of the model, and the KL divergence loss is used to make the potential space distribution generated by the image encoder close to the predefined prior distribution. The loss is expressed as:

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] 1 of which , , , They represent generator loss, discriminator loss, reconstruction loss, and KL divergence loss respectively; E represents expected value, D(x) represents the output of the discriminator for input x, and D(G(z)) represents the output of the discriminator D for the data generated by the generator G(z); , is a weight hyperparameter used to balance the contribution of different loss terms; z~p z (z) represents the distribution from potential distribution p z (z) is the sampling z vector; the potential distribution p z (z) is the standard normal distribution; x~p data (x) indicates that the x variable is derived from the probability distribution p of the real data. data (x); z~q(z|x) means that the random variable z is sampled from the conditional probability distribution q(z|x).

[0047] S4. Obtain data to be reconstructed.

[0048] S5. Reconstruct the three-dimensional object using the optimized object reconstruction model.

[0049] This object reconstruction model uses the variational autoencoder generative adversarial network (VAE-GAN) as the basic model architecture, and on this basis, adds a 3D attention mechanism to the GAN network. VAE is used to generate preliminary 3D reconstruction, GAN is used to optimize and adjust the reconstruction results to improve the authenticity and detail richness of the reconstruction, and the 3D attention mechanism is used to mine the deep features and spatial relationships in the single-view image, thereby improving the accuracy and robustness of 3D object reconstruction.

[0050] The image encoder VAE extracts high-level features from the single-view image. These features include key information such as the edge, shape, and texture of the object, providing a basis for subsequent 3D reconstruction. The last layer of the encoder is converted into a latent vector through the reparameterization technique. Using the latent vector, the generator of the GAN network performs preliminary 3D reconstruction of the object. Finally, the discriminator of the GAN evaluates the generated results to ensure that the generated 3D model has high authenticity and detail accuracy. Figure 1 Shown is a single-view Figure 3 Flowchart of the key algorithms for 3D object reconstruction.

[0051] In this experiment, a subset of the ShapeNet dataset was used as the training set and test set. ShapeNet is a dataset widely used for 3D object reconstruction and generation tasks, containing a large number of high-quality 3D models. In the experiment, a separate model is trained for each object category. Taking the chair category as an example, the training set contains 144 different types of chairs, and each chair is taken from 12 different viewpoints. When performing single-view training, only 1 image is selected for each category for training. Figure 3 Shows some image samples of the dataset. Figure 4 The present invention shows a partial single view based on the ShapeNetRendering dataset Figure 3 At the same time, the training set also includes the 3D true point cloud corresponding to each image, which is used to supervise the training of the model.

[0052] In order to prove the effectiveness of the present invention, a comparative experiment was conducted between the present method and the original 3D-VAE-GAN network. The generated 3D object is as follows Figure 6 As shown in the figure, it can be seen that the original 3D-VAE-GAN model has some missing details in the generated effect, such as the missing limbs of the chair, and there is obvious noise in the generated result. The improved 3D-VAE-GAN model (i.e., the object reconstruction model) has significantly improved the generation effect, with richer details, more complete shape and structure of the generated chair, and significantly reduced noise. In addition, the improved model can better capture the characteristics of different categories of the same type, and the diversity of the generated chairs has also been significantly improved.

[0053] This experiment uses Intersection over Union (IOU) and Mean Squared Error (MSE) as the main evaluation indicators of the model. IOU is an important indicator to measure the effect of the generated model, which can comprehensively reflect the generation quality of the model in various categories. The IOU calculation method of this experiment does not detect the bounding box of the generated object, and adopts a direct calculation method. MSE is used to measure the difference between the generated 3D voxel data and the target 3D voxel data. As shown in Table 1, the improved 3D-VAE-GAN model (i.e., the object reconstruction model) has a significant improvement in IOU, from 3e-2 of the original model to 5e-2. This shows that the improved model has a significant improvement in the quality and diversity of the generated 3D objects. The improved 3D-VAE-GAN model (i.e., the object reconstruction model) performs well in mean square error, indicating that the gap between the generated 3D voxel data and the real 3D voxel data is small.

[0054] Through experiments on a subset of the ShapeNet dataset, the object reconstruction model performs well in the single-view 3D object reconstruction task. This is reflected in the following aspects: Detail recovery: The improved model recovers more details in the generated 3D objects and reduces the loss of details. Noise reduction: The 3D objects generated by the improved model have significantly less noise and higher quality. Diversity increase: The improved model captures more diversity in different objects of the same category, and the generated results are richer.

[0055] The above results show that the object reconstruction model has significantly improved the quality and diversity of generated 3D objects, demonstrating its great potential in practical applications.

[0056] Table 1 Comparison of model evaluation indicators

[0057] Model IOU MSE 3D-VAE-GAN 0.0374 0.06777 Improved 3D-VAE-GAN (this object reconstruction model) 0.05 0.05693

[0058] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A single-view 3D object reconstruction method based on deep learning, characterized in that: include: S1. Construct an object reconstruction model, which includes an image encoder, a generator G and a discriminator D. The image encoder is used to extract high-level features from single-view images and convert the high-level features into latent vectors through reparameterization techniques; the generator uses the latent vector to perform preliminary 3D reconstruction of the object; the discriminator evaluates the generated results to ensure that the generated 3D model has high authenticity and detail accuracy; the generator includes n 3D deconvolution layers and activation layers, each 3D deconvolution layer is added with a batch normalization layer, a ReLU layer, a 3D residual block and a 3D spatial attention layer, and the output of the activation layer is the generated result; the discriminator includes n 3D convolution layers and output layers, each 3D deconvolution layer is added with a batch normalization layer, a ReLU layer, a 3D residual block and a 3D spatial attention layer, n is a positive integer, and the 3D spatial attention layer is used to mine deep features and spatial relationships in single-view images; S2, obtain training data set; S3, importing the training data set into the object reconstruction model, and training and optimizing the object reconstruction model to obtain an optimized object reconstruction model; S4, obtaining data to be reconstructed; S5. Reconstruct the three-dimensional object using the optimized object reconstruction model.

2. The single-view 3D object reconstruction method based on deep learning according to claim 1, characterized in that: The image encoder consists of five convolutional layers and a transformation layer from input to output. The transformation layer converts high-level features into latent vectors through the re-parameterization technique.

3. The single-view 3D object reconstruction method based on deep learning according to claim 1, characterized in that: In S3, when using the loss function to train and optimize the object reconstruction model, the loss function includes adversarial loss, reconstruction loss and KL divergence loss. The reconstruction loss is used to ensure that the generated 3D model is consistent with the input image, the adversarial loss is used to improve the authenticity of the model, and the KL divergence loss is used to make the potential space distribution generated by the image encoder close to the predefined prior distribution. The loss is expressed as: ； ； ； ； ； in , , , They represent generator loss, discriminator loss, reconstruction loss, and KL divergence loss respectively; E represents expected value, D(x) represents the output of the discriminator for input x, and D(G(z)) represents the output of the discriminator D for the data generated by the generator G(z); , is a weight hyperparameter used to balance the contribution of different loss terms; z~p z (z) represents the distribution from potential distribution p z (z) is the sampling z vector; the potential distribution p z (z) is the standard normal distribution; x~p data (x) indicates that the x variable is derived from the probability distribution p of the real data. data (x); z~q(z|x) means that the random variable z is sampled from the conditional probability distribution q(z|x).

Images