A semi-supervised single-view 3D object reconstruction method
By using semi-supervised learning and a prototype shape prior module, the generator and discriminator are trained with a small number of labeled samples, which solves the problems of labeling cost and accuracy in 3D shape reconstruction of objects in a single RGB image and achieves efficient 3D shape reconstruction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2022-09-21
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies face problems such as insufficient labeled samples and high labeling costs when reconstructing the 3D shape of an object from a single RGB image. Furthermore, deep learning-based methods struggle to learn strong shape priors when labeled samples are lacking, resulting in poor reconstruction accuracy.
A semi-supervised learning approach is adopted, which combines a prototype shape prior module. The generator and discriminator are trained with a small number of labeled samples. The EMA algorithm is used to update the teacher model parameters, and a pseudo-label quality assessment mechanism is introduced to reduce the need for labeling and improve reconstruction accuracy.
With a small number of labeled samples, it achieves 3D reconstruction accuracy comparable to fully supervised methods, generates more realistic and natural 3D shapes, solves the problems of labeling cost and lack of shape prior, and improves the effect of single-view 3D reconstruction.
Smart Images
Figure CN115578511B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a semi-supervised single-view 3D object reconstruction method. Background Technology
[0002] Reconstructing the 3D shape of an object from a single RGB image has significant practical implications for many computer vision tasks, including 3D printing, virtual reality, and 3D scene understanding. The human visual system stores powerful shape priors, allowing humans to easily infer the 3D shape of an object from a single image. However, this presents a significant challenge for computers, especially when inferring the shape of an object whose back is occluded. Accurately reconstructing the 3D shape of an object from a single RGB image remains a technical challenge.
[0003] Although some methods based on traditional geometry, such as Structure From Motion (SFM)[1] and Simultaneous Localization and Mapping (SLAM)[2], are feasible solutions, they all rely on large-scale data annotation and the solution of camera parameters.
[0004] In recent years, with the widespread research on deep learning, the use of deep neural networks to predict the 3D shape of single-view images has achieved considerable success. Voxel-based research ([3],[4],[5],[6]) models the target of the 3D model as a voxel, which is very suitable for convolutional neural network and Transformer models. This type of method is mainly composed of an encoder-decoder, where the encoder is used to extract the features of the image, and the decoder regresses and predicts the 3D shape of the object based on the depth features extracted from the image. This type of method generally uses supervised cross-entropy loss to train the network. The difference lies in the composition of the network model structure, which has gradually developed from the early LSTM-based method [7] to the later convolutional neural network-based model [8][9], and then to the latest Transformer-based model
[10]
[11] .
[0005] In addition, there are 3D reconstruction methods based on generative adversarial networks (GANs), which train the reconstruction network using a generative adversarial approach. The generator is used to reconstruct the 3D shape of the object, while the discriminator is used to constrain the reconstructed object to appear realistic and natural. There are also mesh reconstruction based on graph neural networks and methods based on depth maps or contour maps. Although these deep learning-based methods have achieved considerable success, they still face the following two challenges: (1) The accuracy of reconstruction comes from a large number of fine-grained 3D shapes labeled, which is time-consuming and costly to obtain the dataset. (2) Inferring the shape of an object from a single image is an ill-posed problem because for a 2D image, there may be multiple reasonable interpretations of the 3D shape. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention aims to propose a semi-supervised single-view 3D object reconstruction method that reconstructs the 3D shape of objects in a given single RGB image. This invention combines semi-supervised learning to alleviate the burden of annotation costs, requiring only half the number of labeled samples to achieve accuracy comparable to fully supervised methods, thus solving the problem of insufficient labeled samples. Furthermore, neural network-based methods struggle to learn strong shape priors when labeled samples are limited. This invention introduces a prototype shape prior module as an intermediate bridge connecting the 2D image and the 3D shape, improving the lack of prior knowledge and inaccurate reconstruction caused by insufficient sample size. Compared to methods requiring external memory networks and depth maps, this invention's prototype shape prior module is lightweight and easily transferable to various neural network structures, effectively improving the accuracy and naturalness of the reconstructed object.
[0007] This invention provides a semi-supervised single-view 3D object reconstruction method, which consists of three stages, as follows:
[0008] Phase 1: Warm-up phase of supervised training with labeled data
[0009] A neural network model consisting of a generator and a first discriminator is trained based on a small number of labeled samples. The generator is used to generate 3D shapes, and the first discriminator is used to judge the quality of the generated 3D shapes; where:
[0010] The generator includes an image feature encoding module, a prototype shape prior module, and a 3D decoding and generation module; image feature encoding
[0011] The module is used to extract multi-scale image features from the input RGB image. The prototype shape prior module is used to calculate cross-attention and extract prior features based on the image features output by the image feature encoding module and the prototype shape features. The 3D decoding and generation module is used to take the image features output by the image feature encoding module and the prior features extracted by the prototype shape prior module as input and output the predicted 3D object voxels.
[0012] The representation of 3D objects is based on voxel representation, which is specifically represented as I(i,j,k)∈{0,1}, where i,j,k represent the 3D coordinates of the point; I(x,y,z)=1 indicates that the point is a voxel that constitutes the object, otherwise it indicates that the point has no voxel;
[0013] The first discriminator is the shape natural discrimination module, which is used to determine whether the distribution of 3D object voxels output by the 3D decoding generation module belongs to the labeled sample or the predicted sample, so as to ensure that the distribution of the generated reconstructed object is consistent with that of the labeled sample.
[0014] The second stage involves generating pseudo-labels from unsupervised data and using consistency loss to train the teacher-student network for collaborative learning.
[0015] The parameters of the neural network model of the generator trained in the first stage are copied to obtain two networks: one as the teacher model and the other as the student model. The teacher model is used to generate pseudo-labels for unlabeled samples, and the student model is responsible for training all labeled and unlabeled samples in the second stage. The labeled samples are trained with the labels as the training target, and the unlabeled samples are trained with the pseudo-labels generated by the teacher model as the training target to update the network parameters. As the student model learns, the parameters of the teacher model are updated and optimized by the EMA algorithm shown in the following formula to obtain the trained teacher model.
[0016] θ t ←αθ t +(1-α)θ s
[0017] Where, θ t These are the parameters of the teacher network, θ s These are the parameters of the student network, where α is the momentum coefficient;
[0018] This invention uses the EMA algorithm to update the teacher model parameters after each step, so that the teacher model can be regarded as an integration of the student model at multiple time steps.
[0019] Phase 3: Testing and Reconstruction Phase
[0020] The Teacher model trained in the second stage is used as the final inference model. The RGB image to be tested is input into the network, and the 3D shape of the object is predicted directly by regression in an end-to-end manner.
[0021] In this invention, in the first stage, the image feature encoder module adopts the ResNet model, and the 3D decoding and generation module adopts a 3D deconvolution layer.
[0022] In this invention, in the first stage, the prototype shape prior module extracts prior features through a multi-head attention model: it uses image features as the query and prototype shape features as the key and value. First, it calculates the attention weights of the image features and prototype shape features, learning the weights of different parts of the prototype shape corresponding to the image features. Then, it multiplies the weights by the prototype shape feature value to extract the features of the different parts of the attention, which serve as the prior features of the current image. The specific method for extracting prior features is as follows:
[0023] Q = W q .Query; K=W k .Key; V = W v .Value
[0024] Prior feature=Multi-head Attention(Q,K,V)
[0025] in, Both are learnable matrix parameters, where C and D are the dimensions of the image features and the prototype features, respectively.
[0026] In this invention, in the first stage, the prototype shape features are extracted from labeled samples using an autoencoder, and then the KMeans clustering method is used to generate the cluster center for each category, which is called the prototype shape. The autoencoder consists of a 3D convolutional layer and a 3D deconvolutional layer.
[0027] In this invention, in the first stage, the shape natural discrimination module is mainly composed of stacked 3D convolutional layers. It uses 3D convolution to extract the features of the generated shape and uses the feature distribution of the label as the target to judge the quality of the generated shape.
[0028] In this invention, in the first stage, the loss function of the neural network model consists of two parts: reconstruction loss L. rec and generating adversarial loss L d Reconstruction loss L rec For BCE loss, generate adversarial loss L. d for:
[0029]
[0030] Where D p and D g For the distribution of predicted 3D and labeled 3D shapes, y p and y g These are the predicted samples and the labeled samples, respectively, and D is the first discriminator;
[0031] The loss function of the final neural network model is:
[0032]
[0033] Where θ f and θ d λ are the parameters of the generator and the first discriminator, respectively. d To generate the weighting coefficients for adversarial loss.
[0034] In this invention, in the second stage, the loss when training the student model with labeled samples is the BCE loss.
[0035] In this invention, in the second stage, the pseudo-labels generated by the teacher model are input into the second discriminator of the adversarial network to generate pseudo-label quality scores as weights, preventing low-quality pseudo-labels from biasing the training of the student model with unlabeled samples. When training the student model with unlabeled samples, the unsupervised loss is as follows:
[0036]
[0037] in, Pseudo-labels generated for the teacher network, y i For the predicted results of the student network, the score i It is the second discriminator pair The quality score is given, where n represents the size of the same batch.
[0038] In this invention, in the second stage, for unlabeled samples, the input to the teacher model is the image after weak data augmentation, while the input to the student model is the image after strong data augmentation.
[0039] The inventors calculated the average IoU (Intersection over Union) between the model-predicted 3D shape and the ground-annotated 3D shape to measure the quantitative evaluation index of the reconstructed shape. This invention was compared with fully supervised methods, and a semi-supervised image recognition method was extended to this task for further comparison, verifying that the method of this invention is superior to the two methods mentioned above. Furthermore, this invention visualizes the reconstructed object and performs visual qualitative evaluations with other methods. The method of this invention is significantly more realistic and natural, closely resembling the annotated samples from the scanned images. In summary, the beneficial effects of this invention are:
[0040] 1. This invention utilizes semi-supervised learning to solve the problems of insufficient labeled samples and high labeling costs in 3D reconstruction tasks. It only requires a small number of labeled samples, and the 3D reconstruction effect of a single image is more accurate. The generated 3D shape is also more realistic and natural. The reconstruction effect can reach or even exceed that of fully supervised methods, which is of great significance for single-view 3D reconstruction tasks.
[0041] 2. To address the problem that the inherent differences between 2D images and 3D shapes cause neural networks to perform poorly in direct regression, this invention introduces a prototype shape prior module based on an attention mechanism. This module can effectively supplement shape prior features and is lightweight, requiring no additional information or storage space. It provides an intermediate bridge for neural networks to infer 3D shapes from 2D image features, reducing the differences between the two modalities.
[0042] 3. To address the issue of pseudo-label quality assessment in semi-supervised learning involving unlabeled samples, this invention proposes a pseudo-label quality scorer based on a generative adversarial network (GAN) discriminator. This filters out samples with low-quality pseudo-labels, preventing bias in model training. Furthermore, the loss function proposed in this invention, compared to traditional supervised loss functions, leads to more stable training and ultimately better object reconstruction. Attached Figure Description
[0043] Figure 1 This is a flowchart of semi-supervised single-view reconstruction.
[0044] Figure 2 This is a schematic diagram of the semi-supervised single-view 3D reconstruction method based on prototype shape prior proposed in this invention. The method consists of two training phases: The first phase trains a 3D shape generator and a shape quality score discriminator using a small number of labeled samples. This phase comprises four key modules: an image feature encoding module; a prototype shape prior module; a 3D decoding and generation module; and a shape natural discrimination module. In this invention, the first three modules are referred to as the generator, and the fourth module as the first discriminator, constructing a generative adversarial loss. In the second phase, the generator portion from the first phase is replicated as a teacher model and a student model, and different target labels and loss functions are used to train the network for unlabeled and labeled samples.
[0045] Figure 3 This is a schematic diagram of the prototype shape prior network structure in this invention. The 2D image is used to extract image features as a query through an image feature encoding module, while different prototype shapes are used to extract prototype shape features as keys and values through a 3D encoder. Finally, the shape prior features of the current image are extracted by calculating cross-attention.
[0046] Figure 4 This is the result image of the present invention on the ShapeNet synthetic dataset. For images with clean, noise-free backgrounds, the method of the present invention generates smoother object surfaces than other methods.
[0047] Figure 5This is the result image of the present invention on the natural dataset Pix3D. For images with complex backgrounds and significant occlusion, the present invention, aided by shape priors, can better infer the true shape of objects, resulting in reconstruction performance far superior to other methods. Detailed Implementation
[0048] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.
[0049] To reduce the workload of image-3D shape annotation, this invention first proposes to train the network model using a small number of labeled samples; that is, the models involved in this invention are all semi-supervised pattern training. Figure 1 In this invention, all 3D shaped objects are represented by voxels as a unified 3D model representation. Specifically, I(i,j,k)∈{0,1}, where i,j,k represent the 3D coordinates of the point; I(x,y,z)=1 indicates that the spatial position of the point is a voxel constituting the object, otherwise it indicates that the spatial position of the point is empty. Voxels are a common representation method for 3D reconstruction and are suitable for the operation and inference of convolutional neural networks.
[0050] The 3D reconstruction system based on prototype shape prior proposed in this invention ( Figure 2 This is a deep learning-based system that mainly includes the following four modules: (1) image feature encoding module; (2) prototype shape prior module; (3) 3D decoding and generation module; and (4) shape natural discrimination module. Given a single RGB image, firstly, the initial image feature extraction module extracts image features; then, the shape prototype shape prior module calculates cross-attention based on the image features and the category prototype prior features to extract prior features; finally, the image features and prior features are concatenated together and used as input to the 3D decoding and generation module to predict the 3D shape of the generated object; in addition, the shape natural discrimination module can judge the generated 3D shape, increasing the stability and naturalness of the generated shape.
[0051] In this invention, the image feature encoding module and the 3D decoding and generation module employ existing single-view 3D reconstruction models based on voxel representation. The image feature encoding module uses the ResNet model, which improves model performance while reducing parameters and computational load. For the 3D decoding and generation module, this invention simultaneously inputs prior features into the module. While the dimensionality of prior features is lower than that of image features, it significantly contributes to improving model accuracy.
[0052] In this invention, the prototype shape prior module ( Figure 3This invention takes image features as input, calculates the weights of image features and prototype shape priors for that category, and then extracts the required feature information for each prototype as the shape prior features corresponding to that image. In the process of weight information and prior extraction, the invention uses image features as the query and the shape features of each prototype as the key and value, extracting prior features through a multi-head attention model.
[0053] In this invention, the shape natural discrimination module's deep neural network is mainly composed of stacked 3D convolutional layers. It uses 3D convolution to extract features of the generated shape and uses the feature distribution of the label as the target to judge the quality of the generated shape.
[0054] In the first stage of this invention, the training of labeled samples involves a loss function comprising two parts: a binary cross-entropy (BCE) loss and a newly added generative adversarial loss. The image feature encoding module, prototype shape prior module, and 3D decoding generation module of the entire network model are used as generators, while the shape natural discrimination module serves as the first discriminator. The network incorporates a generative adversarial loss to enhance training stability and ensure that the distribution of the generated reconstructed objects is consistent with that of the labeled samples. Therefore, the final loss function for the first stage is the sum of the two parts according to their weights.
[0055] The second stage of this invention mainly involves training on unlabeled samples, and the specific steps are as follows:
[0056] (1) First, the generative model trained in the first stage is copied into two networks, namely Teacher model and Student model. Teacher model is used to generate pseudo-labels, while Student model is used to train the model.
[0057] (2) For unlabeled samples, the input of the Teacher model is the image of the sample after weak data augmentation, while the input of the Student model is the image after strong data augmentation.
[0058] (3) The output of the Teacher model is used as pseudo-labels to guide the training of the Student model. In the second stage, the parameters of the discriminator module are fixed and do not participate in the training of the model. In this invention, the pseudo-labels generated by the Teacher model are input into the discriminator module to generate pseudo-label quality scores.
[0059] (4) The pseudo-label quality score is used as a weight term in the unsupervised loss function. For pseudo-labels with higher quality scores, the weight is higher; otherwise, the weight is lower to prevent the model from being biased by low-quality pseudo-labels.
[0060] (5) The unsupervised loss function adopts the relatively smooth L2 loss, and the pseudo-label quality score obtained in the previous step is used as the weight. The supervised loss function is the same as that in the first stage, which is the BCE loss. The loss in this stage is the sum of these two parts according to the weight ratio.
[0061] In the testing phase of this invention, the Teacher model obtained from the second phase of training is used as the final test model. The test image is input into the model, and then the 3D model corresponding to the image can be generated end-to-end to predict the 3D shape of the object.
[0062] The technical solution of the present invention will be further described below with reference to the embodiments.
[0063] Example 1
[0064] The model accepts a static image of a fixed size (224*224) as input to the visual branch. Images of various sizes are scaled and normalized to this size. This invention mentions two different data augmentation methods: weak data augmentation includes center cropping, adding random noise, and adding a random background; strong data augmentation includes random cropping, random noise, random background, random flipping, color flipping, RGB channel transformation, and other methods.
[0065] The extraction of the prototype shape is achieved by training an autoencoder. First, the autoencoder is trained using a small number of labeled samples. The autoencoder is composed of 3D convolutional layers, and the output of the encoder serves as the feature of the 3D shape. Then, features are extracted from all labeled samples using the autoencoder, and the KMeans clustering method is used to generate the cluster center for each category. This invention refers to this cluster center as the prototype shape.
[0066] The design of the prototype shape prior module first uses image features as the query, sets the hidden layer dimension to 2048, and uses a multi-head attention mechanism with 2 heads. The image features and prototype shape features are then subjected to an attention mechanism. Under this cross-attention mechanism, the shape prior of the queried prototype shape features can be embedded based on the image features. This output is used as the shape prior feature of the current image, and together with the image features, it is input into the decoder. This can efficiently and accurately help the network reconstruct a more realistic and reasonable 3D shape.
[0067] The image feature encoding module and the 3D shape decoder are similar in design to [9]. The encoder's skeleton network is ResNet-50, followed by three 2D convolutional layers, a batch normalization layer, and a ReLU activation function layer. Because 3D reconstruction is a dense prediction task, and different stages of ResNet can notice features at different levels, this invention concatenates the outputs of the last three layers of ResNet as the extracted multi-scale image features, and then performs 3D reconstruction of the object based on the multi-scale image features. For a voxel count of 32... 3 The 3D shape at 128 resolution is decoded by a five-layer 3D deconvolution layer, each followed by a batch normalization layer and a ReLU activation function layer. 3 The voxel decoder consists of seven 3D deconvolutional layers, batch normalization layers, and ReLU activation function layers.
[0068] The shape natural discrimination module is a discriminator that determines whether the distribution of 3D voxels is real (labeled samples) or fake (predicted samples). For a voxel resolution of 32... 3 The shape is determined by a discriminator consisting of four 3D convolutional layers, max pooling layers, and ReLU activation function layers, for a resolution of 128. 3 The module, in its shape, consists of five 3D convolutional layers, max pooling layers, and ReLU activation function layers. Finally, it is joined by two linear layers and a sigmoid activation function layer.
[0069] Method Evaluation. For voxel-based 3D reconstruction models, since they have referable 3D voxel shapes, IoU (Intersection over Union) is used as the evaluation index.
[0070]
[0071] in, and p (i,j,k) represents the value of the (i,j,k)th genus, t is a human-defined threshold, and I is a defined conditional function that is 1 if the condition is met and 0 otherwise.
[0072] Figure 4 This is the result image of the present invention on the ShapeNet synthetic dataset. For images with clean, noise-free backgrounds, the method of the present invention generates smoother object surfaces than other methods.
[0073] Figure 5 This is the result image of the present invention on the natural dataset Pix3D. For images with complex backgrounds and significant occlusion, the present invention, aided by shape priors, can better infer the true shape of objects, resulting in reconstruction performance far superior to other methods.
[0074] The specific experimental results of the method of the present invention on the synthetic dataset ShapeNet and the natural dataset Pix3D are shown in Table 1 and Table 2, respectively.
[0075] Table 1
[0076] Method\Annotation Scale 1％ 5％ 10％ 20％ Supervised[8] 41.13 50.32 53.99 58.06 MeanTeacher
[12] 43.36(+2.23) 51.92(+1.60) 55.93(+1.94) 58.88(+0.82) MixMatch
[13] 41.77(+0.64) 51.23(+0.91) 52.62(-1.37) 57.43(-0.63) FixMatch
[14] 42.44(+1.31) 51.89(+1.57) 55.79(+1.80) 59.63(+1.57) The method of the present invention 46.99(+5.86) 55.23(+4.91) 58.98(+4.99) 61.64(+3.58)
[0077] Table 2
[0078]
[0079] According to Tables 1 and 2, the model proposed in this invention can significantly improve the performance of 3D reconstruction models and enhance reconstruction results.
[0080] References
[0081] [1]Schonberger, JL, Frahm, JM: Structure-from-motion revisited. In: CVPR (2016).
[0082] [2] Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., Leonard, JJ: Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Trans. Robotics (2016).
[0083] [3] Wu, J., Zhang, C., Xue, T., Freeman, WT, Tenenbaum, JB: Learning aprobabilistic latent space of object shapes via 3d generative-adversarial modeling. In: NeurIPS (2016).
[0084] [4]Michalkiewicz,M.,Parisot,S.,Tsogkas,S.,Baktashmotlagh,M.,Eriksson,A.,Belilovsky,E.:Few-shot single-view 3-d object reconstruction withcompositional priors.In:ECCV(2020).
[0085] [5]Yang,G.,Cui,Y.,Belongie,S.,Hariharan,B.:Learning single-view 3dreconstruction with limited pose supervision.In:ECCV(2018)4.
[0086] [6]Wu,J.,Zhang,C.,Zhang,X.,Zhang,Z.,Freeman,W.T.,Tenenbaum,J.B.:Learning shape priors for single-view 3d completion and reconstruction.In:ECCV(2018).
[0087] [7]Choy,C.B.,Xu,D.,Gwak,J.,Chen,K.,Savarese,S.:3d-r2n2:A unifiedapproach for single and multi-view 3d object reconstruction.In:ECCV(2016).
[0088] [8]Xie,H.,Yao,H.,Sun,X.,Zhou,S.,Zhang,S.:Pix2vox:Context-aware 3dreconstruction from single and multi-view images.In:ICCV(2019).
[0089] [9]Xie,H.,Yao,H.,Zhang,S.,Zhou,S.,Sun,W.:Pix2vox++:multi-scalecontext aware 3d object reconstruction from single and multiple images.IJCV(2020).
[0090]
[10] N.Kolotouros,G.Pavlakos,and K.Daniilidis.Convolutional meshregression for single-image human shape reconstruction.In CVPR,2019.
[0091]
[11] Wang,D.,Cui,X.,Chen,X.,Zou,Z.,Shi,T.,Salcudean,S.,Wang,Z.J.,Ward,R.:Multi-view 3d reconstruction with transformers.In:ICCV(2021).
[0092]
[12] Tarvainen,A.,Valpola,H.:Mean teachers are better role models:Weight-averaged consistency targets improve semi-supervised deep learningresults.In:NeurIPS(2017).
[0093]
[13] Berthelot,D.,Carlini,N.,Goodfellow,I.,Papernot,N.,Oliver,A.,Raffel,C.A.:Mixmatch:A holistic approach to semi-supervised learning.In:NeurIPS(2019).
[0094]
[14] Sohn,K.,Berthelot,D.,Carlini,N.,Zhang,Z.,Zhang,H.,Raffel,C.A.,Cubuk,E.D.,Kurakin,A.,Li,C.L.:Fixmatch:Simplifying semi-supervised learningwith consistency and confidence.In:NeurIPS(2020)。
Claims
1. A semi-supervised single-view 3D object reconstruction method, characterized in that, It is divided into three stages, as follows: Phase 1: Warm-up phase of supervised training with labeled data A neural network model consisting of a generator and a first discriminator is trained based on a small number of labeled samples. The generator is used to generate 3D shapes, and the first discriminator is used to judge the quality of the generated 3D shapes; where: The generator includes an image feature encoding module, a prototype shape prior module, and a 3D decoding generation module. The image feature encoding module is used to extract multi-scale image features from the input RGB image. The prototype shape prior module is used to calculate cross-attention and extract prior features based on the image features output by the image feature encoding module and the prototype shape features. The 3D decoding generation module is used to generate predicted 3D object voxels by taking the image features output by the image feature encoding module and the prior features extracted by the prototype shape prior module as input. The first discriminator is a shape natural discrimination module, which is used to determine whether the distribution of 3D object voxels output by the 3D decoding generation module belongs to the labeled sample or the predicted sample, so as to ensure that the distribution of the generated reconstructed object sample and the label sample is consistent. The second stage involves generating pseudo-labels from unsupervised data and using consistency loss to train the teacher-student network for collaborative learning. The parameters of the neural network model of the generator trained in the first stage are copied to obtain two networks: one as the teacher model and the other as the student model. The teacher model is used to generate pseudo-labels for unlabeled samples, and the student model is responsible for training all labeled and unlabeled samples in the second stage. The labeled samples are trained in a supervised manner with the labels as the training target, and the unlabeled samples are trained in an unsupervised manner with the pseudo-labels generated by the teacher model as the training target. As the student model learns, the parameters of the teacher model are updated and optimized by the EMA algorithm shown in the following formula to obtain the trained teacher model. i t ←th t +(1-α)θ s Where, θ t These are the parameters of the teacher network, θ s These are the parameters of the student network, where α is the momentum coefficient; Phase 3: Testing and Reconstruction Phase The teacher model trained in the second stage is used as the final inference model. The RGB image to be tested is input into the network, and the 3D shape of the object is predicted directly by regression in an end-to-end manner.
2. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the first stage, the image feature encoding module uses the ResNet model; the 3D decoding and generation module uses 3D deconvolution layers.
3. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the first stage, the prototype shape prior module extracts prior features through a multi-head attention model: it uses image features as the query and prototype shape features as the key and value. First, it calculates the attention weights of the image features and prototype shape features, learning the weights of different parts of the prototype shape corresponding to the image features. Then, it multiplies the weights by the prototype shape feature value to extract the features of the different parts of the attention, which serve as the prior features of the current image. The specific method for extracting prior features is as follows: Q=W q .Query;K=W k .Key;V=W v .Value Prior feature=Multi-head Attention(Q,K,V) in, Both are learnable matrix parameters, where C and D are the dimensions of the image features and the prototype features, respectively.
4. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the first stage, labeled samples are processed by an autoencoder to extract prototype shape features, and then the KMeans clustering method is used to generate cluster centers for each category. The cluster centers are called prototype shapes. The autoencoder consists of 3D convolutional layers and 3D deconvolutional layers.
5. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the first stage, the shape natural discrimination module is mainly composed of stacked 3D convolutional layers. It uses 3D convolution to extract the features of the generated shape and uses the feature distribution of the label as the target to judge the quality of the generated shape.
6. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the first stage, the loss function of the neural network model consists of two parts: reconstruction loss L rec and generating adversarial loss L d Reconstruction loss L rec Using BCE (Binary Cross Entropy) loss, generate adversarial loss L. d for: Where D p and D g For the distribution of predicted 3D and labeled 3D shapes, y p and y g These are the predicted samples and the labeled samples, respectively, and D is the first discriminator; The loss function of the final neural network model is: Where θ f and θ d λ are the parameters of the generator and the first discriminator, respectively. d To generate the weighting coefficients for adversarial loss.
7. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the second stage, the loss used to train the student model with labeled samples is the BCE loss.
8. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the second stage, the pseudo-labels generated by the teacher model are input into the second discriminator of the adversarial network, generating pseudo-label quality scores as weights to prevent low-quality pseudo-labels from biasing the training of the student model with unlabeled samples. When training the student model with unlabeled samples, the unsupervised loss is as follows: in, Pseudo-labels generated for the teacher network, y i For the predicted results of the student network, the score i It is the second discriminator pair The quality score is given, where n represents the size of the same batch.
9. The semi-supervised single-view 3D object reconstruction method according to claim 1, characterized in that, In the second stage, for unlabeled samples, the input to the teacher model is the image with weak data augmentation, while the input to the student model is the image with strong data augmentation.