A three-dimensional object reconstruction method of a multi-modal symmetric adversarial network

Abstract

This invention discloses a 3D object reconstruction method using a multimodal symmetric adversarial network, comprising: S210, using a symmetric learning network to learn the symmetry of the object from the voxel representation of the object's single-view depth image, generating coarse 3D voxels; then using a voxel encoding network and a planar encoding network to learn latent features from the coarse 3D voxels and voxel representations respectively; concatenating the latent features to obtain a latent vector; and finally using a decoding network to generate a target 3D voxel from the latent vector; S220, embedding the generated target 3D voxel and the real 3D voxel into the voxel encoding network, planar encoding network, and point cloud encoding network of a multimodal discriminator, respectively, to help the MSANet model generate more details under the multimodal discriminator loss that integrates voxel, point cloud, and planar losses. According to the technical solution of this invention, more accurate 3D structures can be predicted.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics and 3D reconstruction, and relates to a method for 3D object reconstruction using multimodal symmetric adversarial networks. Background Technology

[0002] Vision-based 3D object reconstruction methods provide fundamental technical support for many advanced multimedia applications, such as intelligent robot grasping, autonomous driving, and virtual reality games. However, traditional 3D reconstruction methods are limited in practical applications due to their reliance on the quality and quantity of input images. Current deep learning-based 3D reconstruction methods have brought new ideas with the rapid development of 3D databases and deep generative models. Today, deep generative models, such as generative adversarial networks, autoencoders, variational autoencoders, diffusion models, and 3D Gaussian sputtering models, are frequently used for 3D object reconstruction from single-view images. However, these deep generative models still struggle to fully learn prior knowledge of the input image, resulting in lower reconstruction accuracy and difficulty in capturing details.

[0003] Recent state-of-the-art deep learning methods utilize 3D representations of point clouds, meshes, voxels, and implicit surfaces. Notably, point clouds lack the connectivity structure of the underlying mesh, making them prone to noise; mesh representations rely on template meshes and therefore cannot handle arbitrary topologies. Voxel representations, with their regular structures, are a simple way to directly generalize pixels to the 3D context and are easily adapted to deep networks. Many studies have also implicitly represented 3D surfaces as continuous decision boundaries for deep neural network classifiers, but these have failed to recover the fine-grained details of objects. Summary of the Invention

[0004] The technical problem to be solved by this invention is: to fully learn the prior knowledge of the object by using a symmetry-aware generator to ensure accurate prediction of missing parts; to fully explore the multimodal representation information of the object's voxels, point clouds and planes by using a multimodal discriminator to provide adversarial feedback on the shape generated by the 3D generative model in order to achieve high-precision 3D reconstruction.

[0005] To address the above technical problems, this invention proposes a method for three-dimensional object reconstruction using multimodal symmetric adversarial networks.

[0006] The technical solution of this invention is as follows:

[0007] A 3D object reconstruction method based on a multimodal symmetric adversarial network (SMARN) is disclosed. The SMARN consists of two parts: 1) a symmetry-aware generator, including a symmetry learning network, a voxel encoding network, a planar encoding network, and a decoding network; and 2) a multimodal discriminator, including a voxel encoding network, a planar encoding network, and a point cloud encoding network. The 3D object reconstruction method includes:

[0008] S210, the symmetry of the object is learned from the voxel representation of the object's single-view depth image using a symmetry learning network to generate a coarse 3D voxel. Then, the latent features are learned from the coarse 3D voxel and the voxel representation using a voxel encoding network and a planar encoding network respectively. The latent features of the two are concatenated to obtain a latent vector. Finally, the target 3D voxel is generated from the latent vector using a decoding network.

[0009] S220 embeds the generated target 3D voxels and the real 3D voxels into the voxel encoding network, plane encoding network and point cloud encoding network in the multimodal discriminator, respectively, to help the MSANet model generate more accurate occluded parts under the multimodal discriminator loss that integrates voxel, point cloud and plane loss.

[0010] A computing device includes: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs a three-dimensional object reconstruction method using a multimodal symmetric adversarial network.

[0011] A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform a three-dimensional object reconstruction method using a multimodal symmetric adversarial network.

[0012] Beneficial effects:

[0013] This invention proposes a novel framework, MSANet, which reconstructs the complete 3D shape of an object from a single depth image. MSANet improves upon traditional GANs by introducing a symmetric learning network, voxels, planes, and a multimodal network of point clouds, and operates using a symmetric-based generator and a multimodal discriminator. These innovations provide usable feature cues for occluded parts and reduce noise. Experimental results demonstrate that MSANet can predict more accurate 3D structures, proving the effectiveness of the innovations and confirming its ability to mine latent features of shapes and learn the mapping between images and 3D shapes. Attached Figure Description

[0014] Figure 1 The MSANet framework used in the three-dimensional object reconstruction method of multimodal symmetric adversarial networks according to an embodiment of the present invention is shown. Detailed Implementation

[0015] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0016] It has been observed that in real-world scenarios, 3D objects often exhibit symmetry about a certain plane. Therefore, to address this issue, this invention proposes a 3D object reconstruction method based on a multimodal symmetric adversarial network (MSANet). This method fully learns the self-symmetric properties of objects and combines multimodal representations of object voxels, point clouds, and projection planes to achieve high-precision 3D reconstruction. MSANet introduces a symmetry-aware generator, which helps to fully learn prior knowledge of the input object, improving reconstruction accuracy. MSANet improves reconstruction quality by designing a discriminator that incorporates voxel, point cloud, and planar adversarial losses, and helps to infer reliable cues from existing parts to accurately reconstruct occluded portions of 3D shapes. Experimental results show that MSANet not only outperforms several state-of-the-art methods on the public ModelNet dataset but also accurately predicts more occluded parts of 3D objects.

[0017] Figure 1 The MSANet framework used in the 3D object reconstruction method of multimodal symmetric adversarial networks according to an embodiment of the present invention is shown below. (Refer to...) Figure 1 Describe the process of this method. For example... Figure 1 As shown, the MSANet model consists of two parts: 1) a symmetry-aware generator, including a symmetry learning network, a voxel encoding network, a planar encoding network, and a decoding network; 2) a multimodal discriminator, including a voxel encoding network, a planar encoding network, and a point cloud encoding network. 3D object reconstruction methods include:

[0018] S210 utilizes a symmetry learning network to learn the symmetry of the object from the voxel representation of the object's single-view depth image, generating coarse 3D voxels. Then, voxel encoding and planar encoding networks are used to learn latent features from the coarse 3D voxels and voxel representations respectively. The latent features of the two networks are concatenated to obtain a latent vector. Finally, a decoding network is used to generate the target 3D voxel from the latent vector. This fully exploits the object's symmetry characteristics and the multimodal fusion features of voxels and planar representations, which helps to improve the reconstruction accuracy of the occluded parts of the object.

[0019] Figure 1The left border shows the framework of the symmetry-sensor generator, which consists of a symmetry learning network, a voxel encoding network, a planar encoding network, and a decoding network. The symmetry learning network consists of five 3D residual blocks and five multilayer perceptrons. Figure 1 The diagram shows five interconnected 3D residual blocks, with the fifth residual block connected to the first through fifth multilayer perceptrons. Each 3D residual block contains three 3D... 3 The convolutional layer has a stride of 1. 3 Each convolutional layer is followed by a ReLU activation function, and each residual block is followed by a max-pooling layer (except for the last layer). The feature channels are {64, 128, 256, 512, 1024}. The feature channels of the multilayer perceptron are {1024, 512, 256, 128, 4}, where the last channel 4 represents the object's symmetry plane parameters. Then, a symmetry learning network is used to extract the voxel representation of the object's single-view depth image. The symmetry plane parameters of the learning object are used to calculate the missing symmetry parts of the voxel representation using the formula for point symmetry about the plane. Then, it is compared with the voxel representation of the object's single-view depth image. Perform logical operations to generate a rough 3D voxel. The logical operation formula is as follows:

[0020] ,

[0021] in, It is a rough 3D voxel.

[0022] The voxel encoding network consists of five 3D residual blocks. The planar encoding network consists of a projection layer, a 2D convolutional layer, a max-pooling layer, and three 2D residual blocks connected in sequence. The 2D convolutional layer has a 7-kernel designation. 2 The step size is 2 2 This is followed by a batch normalization and ReLU activation function, with each 2D residual block containing two 3x3 activation functions. 2 The convolutional layer has a stride of 1. 3 Each convolutional layer is followed by a ReLU activation function. The decoding network consists of four upsampling layers (each followed by a 3D residual block) and one 3D convolutional layer. Each upsampling layer contains a set of 3D... 3 The convolution kernel has a stride of 2. 3 The number of output channels for each upsampling layer is as follows: starting from 2048, the number of output channels for subsequent layers is reduced by half, and finally to 1.

[0023] Loss function of symmetry-aware generator Includes reconstruction losses , generate loss Point cloud loss :

[0024] (1)

[0025] in, , These are the weights of each loss. In one implementation, the reconstruction loss can be assigned a weight. Assign higher weights ,Right now .

[0026] Reconstruction losses An improved cross-entropy loss function is used to address the class label imbalance problem in voxel grids. Compared to the traditional cross-entropy loss function, a penalty parameter is added to further penalize voxels with a value of 1 that are identified as 0. Defined as:

[0027] (2)

[0028] in, It is the target 3D voxel value. These are the voxel values ​​of a real 3D object. It is a penalty parameter, if A larger value indicates a higher probability that more voxels will be occupied, while a smaller value indicates that the probability of most voxels being occupied is less than 0.5. In one embodiment, it is set to 0.85.

[0029] Generation loss An improved version of generative adversarial networks, namely the Wasserstein generative adversarial network, was used for training. Defined as:

[0030] (3)

[0031] in, , and These represent voxel encoding networks, planar encoding networks, and point cloud encoding networks, respectively. It expresses expectation.

[0032] Point cloud loss Loss is calculated only at the sampling points. Defined as:

[0033] (4)

[0034] in, It's weight. , These are the predicted probabilities of sampling points in the real 3D voxel and the target 3D voxel, respectively. Please note that... It can be calculated using the nearest interpolation method with the true voxel.

[0035] S220 embeds the generated target 3D voxels and the real 3D voxels into the voxel encoding network, plane encoding network and point cloud encoding network in the multimodal discriminator, respectively, to help the MSANet model generate more accurate occluded parts under the multimodal discriminator loss that integrates voxel, point cloud and plane loss.

[0036] Design concept of the multimodal discriminator: Traditional GANs often focus on the single-modal representation of objects, resulting in a lot of noise in the reconstructed objects. Therefore, this invention designs a multimodal discriminator that combines the learning of object voxel, planar, and point cloud representations to fully explore the latent prior features of the object, which helps to generate target 3D voxels. Details.

[0037] Figure 1 The right-hand box shows the framework of the multimodal discriminator, which consists of a voxel encoding network, a planar encoding network, and a point cloud encoding network. The voxel encoding network consists of five 3D residual blocks, each followed by a max-pooling layer (except for the last layer), with feature channels {8, 16, 32, 64, 128}. Each 3D residual block contains three 3D features. 3 The convolutional layer has a stride of 1. 3 Each layer is followed by a ReLU activation function. The final output of the voxel-encoded network is R. 8192 3D eigenvectors.

[0038] The planar encoding network consists of sequentially connected projection layers, 2D convolutional layers, max pooling layers, and three 2D residual blocks. The 2D convolutional layers have 7 convolutional kernels. 2 The step size is 2 2 This is followed by a batch normalization and ReLU activation function, with each 2D residual block containing two 3x3 activation functions. 2 The convolutional layer has a stride of 1. 3 Each layer is followed by a ReLU activation function, and the final output of the planar coding network is R. 1024 3D eigenvectors.

[0039] The point cloud encoding network consists of a sampling layer, a feature grid layer, and a multilayer perceptron. The sampling layer randomly selects 6400 sampling points from the generated 3D voxels of the target. The feature grid layer passes the features of the voxels to each sampling point. The multilayer perceptron learns the features of the sampling points to reconstruct precise details of the object. The point cloud encoding network ultimately outputs R. 6400 3D eigenvectors.

[0040] This section optimizes the multimodal discriminator loss by reducing the distribution distance between the true and false reconstructions output by the voxel encoding network, planar encoding network, and point cloud encoding network. , Defined as:

[0041] (5)

[0042] in, It is voxel discrimination loss. It is a planar discriminant loss. Point cloud analysis is used to determine loss.

[0043] (6)

[0044] (7)

[0045] (8)

[0046] In summary, the MSANet proposed in this invention includes a generator loss based on symmetry perception (see formulas (1), (2), (3), (4)) and a multimodal discriminator loss (see formulas (5), (6), (7), (8)).

[0047] This invention uses publicly available ModelNet data for experiments. ModelNet consists of three classes {chair, high stool, toilet}. 20K pairs were trained for each class, and 2K pairs were tested. Each pair represents a voxel from a single-view depth image and the corresponding 3D shape. Each voxel is represented as a binary tensor: 1 indicates the voxel is occupied, and 0 indicates the voxel is not occupied.

[0048] This invention first uses the 3D voxel intersection-over-union (IoU) ratio to evaluate the performance of 3D reconstruction. IoU represents the similarity between the predicted 3D voxels and the actual voxels; a higher IoU value indicates better reconstruction results. The IoU is defined as follows:

[0049] ,

[0050] in, It is the size of a three-dimensional voxel mesh. It is an index of a three-dimensional voxel mesh. and They are the first Predicted and actual values ​​of voxels.

[0051] This invention also uses the average standard cross-entropy loss (CE) between predicted 3D voxels and true 3D voxels to evaluate the reconstruction effect; the lower the CE value, the better the reconstructed 3D shape. CE is defined as:

[0052] ).

[0053] This invention conducted relevant experiments, comparing the proposed MSANet with state-of-the-art reconstruction methods including 3D-CNN, 3D-RecGAN++, 3D-RecAE, Pix2Vox, 3D-VAE-GAN, LALP-Net, DLGAN, and SRPAN. Table 1 shows the comparison and reconstruction results.

[0054] Table 1. Comparison of IoU and CE of reconstruction results for different methods on a single class.

[0055]

[0056] The present invention also provides a computing device, comprising: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the above-described method for reconstructing three-dimensional objects using a multimodal symmetric adversarial network.

[0057] The present invention also provides a readable storage medium storing program instructions, which, when read and executed by a computing device, cause the computing device to perform the above-described method for reconstructing three-dimensional objects using a multimodal symmetric adversarial network.

[0058] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0059] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of explaining or limiting the subject matter of the invention.

Claims

1. A method for reconstructing three-dimensional objects using a multimodal symmetric adversarial network, characterized in that, The multimodal symmetric adversarial network consists of two parts: 1) a symmetry-aware generator, including a symmetry learning network, a voxel encoding network, a planar encoding network, and a decoding network; and 2) a multimodal discriminator, including a voxel encoding network, a planar encoding network, and a point cloud encoding network. The 3D object reconstruction method includes: S210, the symmetry of the object is learned from the voxel representation of the object's single-view depth image using a symmetry learning network to generate a coarse 3D voxel. Then, the latent features are learned from the coarse 3D voxel and the voxel representation using a voxel encoding network and a planar encoding network respectively. The latent features of the two are concatenated to obtain a latent vector. Finally, the target 3D voxel is generated from the latent vector using a decoding network. S220 embeds the generated target 3D voxels and the real 3D voxels into the voxel encoding network, plane encoding network and point cloud encoding network in the multimodal discriminator, respectively, to help the MSANet model generate more accurate occluded parts under the multimodal discriminator loss that integrates voxel, point cloud and plane loss. In S210, the symmetry-aware generator uses a symmetry learning network to learn the symmetry of an object from its voxel representation in a single-view depth image, generating coarse 3D voxels including: Voxel representation of object depth image from a single viewpoint using a symmetric learning network The symmetry plane parameters of the learning object are used to calculate the missing symmetry parts of the voxel representation using the formula for point symmetry about the plane. Then, with voxel representation Perform logical operations to generate a rough 3D voxel. The logical operation formula is as follows: ， in, It is a rough 3D voxel; In the symmetry-aware generator, the symmetry learning network consists of 5 3D residual blocks and 5 multilayer perceptrons. Each 3D residual block contains 3... 3 The convolutional layer has a stride of 1. 3 Each convolutional layer is followed by a ReLU activation function, and each residual block is followed by a max pooling layer, except for the last layer. The feature channels are {64, 128, 256, 512, 1024}, while the feature channels of the multilayer perceptron are {1024, 512, 256, 128, 4}. In the symmetry-aware generator, the voxel encoding network consists of 5 3D residual blocks, and the planar encoding network consists of a sequentially connected projection layer, a 2D convolutional layer, a max-pooling layer, and 3 2D residual blocks. The 2D convolutional layer has 7 convolutional kernels. 2 The step size is 2 2 This is followed by a batch normalization and ReLU activation function, with each 2D residual block containing two 3x3 activation functions. 2 The convolutional layer has a stride of 1. 3 Each convolutional layer is followed by a ReLU activation function.

2. The method for three-dimensional object reconstruction using multimodal symmetric adversarial networks according to claim 1, characterized in that, In the symmetry-aware generator, the decoding network consists of four upsampling layers and one 3D convolutional layer. Each upsampling layer is followed by a 3D residual block, and each upsampling layer contains a set of 3D residual blocks. 3 The convolution kernel has a stride of 2. 3 The number of output channels for each upsampling layer is as follows: starting from 2048, the number of output channels for subsequent layers is reduced by half, and finally to 1.

3. The method for three-dimensional object reconstruction using multimodal symmetric adversarial networks according to claim 1, characterized in that, Loss function of symmetry-aware generator Includes reconstruction losses , generate loss Point cloud loss （1） in, , These are the weights of each loss; Reconstruction losses Using an improved cross-entropy loss function, Defined as: （2） in, It is the target 3D voxel value. These are the voxel values ​​of a real 3D object. It is a penalty parameter; Generation loss An improved version of generative adversarial networks, namely the Wasserstein generative adversarial network, was used for training. Defined as: （3） in, , and These represent voxel encoding networks, planar encoding networks, and point cloud encoding networks, respectively. Expressing expectations; Point cloud loss Loss is calculated only at the sampling points. Defined as: （4） in, It's weight. , These are the predicted probabilities of sampling points in the real 3D voxel and the target 3D voxel, respectively.

4. The method for three-dimensional object reconstruction using multimodal symmetric adversarial networks according to claim 1, characterized in that, In the multimodal discriminator, the voxel encoding network consists of five 3D residual blocks, each followed by a max-pooling layer, except for the last layer. The feature channels are {8, 16, 32, 64, 128}, and each 3D residual block contains three 3D features. 3 The convolutional layer has a stride of 1. 3 Each layer is followed by a ReLU activation function, and the voxel-encoded network ultimately outputs R. 8192 3D eigenvectors; The planar encoding network consists of sequentially connected projection layers, 2D convolutional layers, max pooling layers, and three 2D residual blocks. The 2D convolutional layers have 7 convolutional kernels. 2 The step size is 2 2 This is followed by a batch normalization and ReLU activation function, with each 2D residual block containing two 3x3 activation functions. 2 The convolutional layer has a stride of 1. 3 Each layer is followed by a ReLU activation function, and the final output of the planar coding network is R. 1024 3D eigenvectors; The point cloud encoding network consists of a sampling layer, a feature grid layer, and a multilayer perceptron. The sampling layer randomly selects 6400 sampling points from the generated 3D target voxels. The feature grid layer passes the voxel features to each sampling point. The multilayer perceptron learns the features of the sampling points to reconstruct a more accurate occluded part. The point cloud encoding network ultimately outputs R. 6400 3D eigenvectors.

5. The method for three-dimensional object reconstruction using multimodal symmetric adversarial networks according to claim 4, characterized in that, In the multimodal discriminator, the loss of the multimodal discriminator is optimized by reducing the distribution distance between the true and false reconstructions output by the voxel encoding network, the planar encoding network, and the point cloud encoding network. , Defined as: （5） in, It is voxel discrimination loss. It is a planar discriminant loss. Point cloud analysis is used to determine loss. （6） （7） （8）。 6. A computing device, characterized in that, include: At least one processor and a memory storing program instructions; When the program instructions are read and executed by the processor, the computing device performs the three-dimensional object reconstruction method of the multimodal symmetric adversarial network as described in any one of claims 1-5.

7. A readable storage medium storing program instructions, characterized in that, When the program instructions are read and executed by the computing device, the computing device performs the three-dimensional object reconstruction method of the multimodal symmetric adversarial network as described in any one of claims 1-5.

Images