Single view reconstruction method and device

Abstract

The invention discloses a single-view reconstruction method and device, and relates to the technical field of single-view 3D reconstruction, and the method comprises the following steps: extracting and obtaining image features based on an input image and internal and external parameters of a camera; based on the image features, decoding to obtain point cloud characterization; performing feature enhancement on the point cloud characterization to obtain a densified point cloud feature; based on the densified point cloud features, encoding to obtain a three-plane projection feature token; based on the three-plane projection feature token and the image features, decoding to obtain three-plane features; obtaining 3D Gaussian features based on the point cloud representation and the three-plane features; and based on the 3D Gaussian features, decoding to obtain 3D Gaussian representation. According to the method, the decoder is used for reasoning the point cloud representation and the three-plane features from the single-view image, the 3D Gaussian features are queried from the three-plane features by using the point cloud representation, and finally the 3D Gaussian representation is obtained, so that the texture quality and the rendering speed are remarkably improved while the efficient reconstruction efficiency is kept.

Description

Technical Field

[0001] This application involves single vision Figure 3 The field of D reconstruction technology particularly relates to a single-view reconstruction method and device. Background Art

[0002] Currently, existing implementations mainly focus on using generative models for single-view Figure 3 D reconstruction. These methods include a series of techniques based on score distillation sampling (SDS) and diffusion models, which optimize the parameters of the generative model through neural network training to achieve the mapping from a single image to a 3D model. In these implementations, SDS and diffusion models play an important role in improving the accuracy and details of 3D reconstruction. However, these methods often face some challenges in practical applications. First, they may require a lot of time for shape optimization, resulting in long training and optimization cycles. This may limit their usability and practicality in practical applications.

[0003] In single vision Figure 3 In the field of 3D reconstruction, regression methods are a common strategy for mapping information extracted from a single image to the corresponding 3D geometric representation. In existing implementations, regression methods are usually implemented through neural networks, where the parameters of the network are optimized through a training process to reconstruct 3D objects with the greatest possible accuracy. These methods usually regress from images implicitly (such as neural radiance fields) or explicitly (point clouds, meshes), but no matter which representation is used, it is difficult to strike a balance between rendering speed and quality. For example, although explicit-based representations have fast rendering speeds, they are difficult to learn good generalization. Implicit-based representations are easy to learn good generalization but require more rendering time and video memory consumption.

[0004] Therefore, in order to meet practical needs, a single view reconstruction technology is provided. Summary of the Invention

[0005] In response to the defects in the existing technology, the purpose of this application is to provide a single-view reconstruction method and device, which uses a decoder to infer explicit point cloud representation and implicit three-plane features from a single-view image, and uses the point cloud representation to query 3D Gaussian features from the three-plane features, and finally obtains a 3D Gaussian representation, while maintaining high reconstruction efficiency, significantly improving texture quality and rendering speed.

[0006] In order to achieve the above objectives, the technical solution adopted by this application is: In a first aspect, the present application provides a single-view reconstruction method, the method comprising the following steps: Extracting image features corresponding to the input image based on the input image and corresponding camera internal and external parameters; Based on the image features corresponding to the input image, decoding is performed to obtain a corresponding point cloud representation; Performing feature enhancement on the point cloud representation to obtain a densified point cloud feature; Based on the densified point cloud features, encoding obtains corresponding three-plane projection feature tokens; Based on the three-plane projection feature token and the image feature, decoding to obtain corresponding three-plane features; Based on the point cloud representation and the three-plane features, obtaining corresponding 3D Gaussian features; Based on the 3D Gaussian features, decoding is performed to obtain corresponding 3D Gaussian representations.

[0007] On the basis of the above technical solution, the method of extracting image features corresponding to the input image based on the input image and the corresponding internal and external parameters of the camera includes the following steps: The camera internal and external parameters are mapped to a high-dimensional feature space, and image features with camera perspective perception are obtained based on the input image.

[0008] On the basis of the above technical solution, decoding the image features corresponding to the input image to obtain the corresponding point cloud representation includes the following steps: A point cloud decoder based on the Transformer architecture is used to decode the image features corresponding to the input image and obtain the corresponding point cloud representation.

[0009] Based on the above technical solution, the feature enhancement of the point cloud representation to obtain the densified point cloud features includes the following steps: The point cloud representation is feature enhanced, each point in the point cloud is projected onto the image plane, and the local image features corresponding to each point are obtained by interpolating the corresponding projection position on the image to obtain the densified point cloud features.

[0010] On the basis of the above technical solution, the encoding based on the densified point cloud features to obtain the corresponding three-plane projection feature tokens includes the following steps: Performing projection feature calculation on the densified point cloud features to obtain corresponding feature point clouds; The feature point cloud is feature encoded by PointNet and orthogonally projected onto three planes. The features falling in the same grid are mean pooled to obtain the corresponding three-plane projection feature tokens.

[0011] On the basis of the above technical solution, based on the three-plane projection feature token and the image feature, decoding to obtain the corresponding three-plane feature includes the following steps: A three-plane decoder of the Transformer architecture is used to decode the three-plane projection feature tokens to obtain the corresponding three-plane features under the injection of the image features.

[0012] On the basis of the above technical solution, based on the point cloud representation and the three-plane features, obtaining the corresponding 3D Gaussian features includes the following steps: The point cloud representation is queried for corresponding spatial features and features at the projection of the input image in the three-plane features to obtain corresponding 3D Gaussian features.

[0013] On the basis of the above technical solution, based on the 3D Gaussian feature, decoding to obtain the corresponding 3D Gaussian representation includes the following steps: The MLP-based Gaussian decoder decodes the 3D Gaussian features to obtain corresponding 3D Gaussian representations.

[0014] In a second aspect, the present application provides a single-view reconstruction device, comprising: An image feature extraction module is used to extract image features corresponding to the input image based on the input image and corresponding camera internal and external parameters; An image feature decoding module, configured to decode the image features corresponding to the input image to obtain a corresponding point cloud representation; A point cloud representation enhancement module, which is used to enhance the features of the point cloud representation to obtain densified point cloud features; A point cloud feature encoding module, which is used to encode the densified point cloud features to obtain corresponding three-plane projection feature tokens; A feature token decoding module, configured to decode the three-plane projection feature token and the image feature to obtain the corresponding three-plane feature; A 3D Gaussian feature acquisition module, configured to acquire corresponding 3D Gaussian features based on the point cloud representation and the three-plane features; The 3D Gaussian feature decoding module is used to decode and obtain the corresponding 3D Gaussian representation based on the 3D Gaussian feature.

[0015] On the basis of the above technical solution, the image feature extraction module is further used to map the camera internal and external parameters to a high-dimensional feature space, and obtain image features with camera perspective perception based on the input image.

[0016] Based on the above technical solution, the image feature decoding module is also used to use a point cloud decoder of the Transformer architecture to decode the image features corresponding to the input image and obtain the corresponding point cloud representation.

[0017] Based on the above technical solution, the point cloud representation enhancement module is also used to perform feature enhancement on the point cloud representation, project each point in the point cloud onto the image plane, obtain the local image features corresponding to each point by interpolating the corresponding projection position on the image, and obtain the densified point cloud features.

[0018] On the basis of the above technical solution, the point cloud feature encoding module is further used to perform projection feature calculation on the densified point cloud features to obtain corresponding feature point clouds; The point cloud feature encoding module is also used to encode the feature point cloud through PointNet, and perform orthogonal projection onto three planes, perform mean pooling on the features falling in the same grid, and obtain the corresponding three-plane projection feature tokens.

[0019] Based on the above technical solution, the feature token decoding module is also used to use a three-plane decoder of the Transformer architecture to decode and obtain corresponding three-plane features from the three-plane projection feature tokens under the injection of the image features.

[0020] On the basis of the above technical solution, the 3D Gaussian feature acquisition module is further used to query the corresponding spatial features and features at the projection of the input image in the three-plane features of the point cloud representation to obtain the corresponding 3D Gaussian features.

[0021] On the basis of the above technical solution, the 3D Gaussian feature decoding module is further used to decode the 3D Gaussian feature based on the MLP Gaussian decoder to obtain the corresponding 3D Gaussian representation.

[0022] Compared with the prior art, the advantages of this application are: This application uses the Transformer decoder to infer explicit point cloud representations and implicit three-plane features from single-view images, uses the point cloud representation to query 3D Gaussian features from the three-plane features, and finally obtains a 3D Gaussian representation, which significantly improves texture quality and rendering speed while maintaining high reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is a flowchart of the steps of the single-view reconstruction method according to an embodiment of the present application; Figure 2 This is a principle flow chart of the single-view reconstruction method according to an embodiment of the present application; Figure 3 This is a structural block diagram of a single-view reconstruction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] The embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0027] The embodiments of the present application provide a single-view reconstruction method and device, which utilizes a Transformer decoder to infer explicit point cloud representations and implicit three-plane features from a single-view image, uses the point cloud representation to query 3D Gaussian features from the three-plane features, and ultimately obtains a 3D Gaussian representation, significantly improving texture quality and rendering speed while maintaining high reconstruction efficiency.

[0028] To achieve the above technical effects, the overall idea of ​​this application is as follows: A single-view reconstruction method, the method comprising the following steps: S1. Extracting image features corresponding to the input image based on the input image and the corresponding internal and external parameters of the camera; S2. Based on the image features corresponding to the input image, decode to obtain the corresponding point cloud representation; S3, perform feature enhancement on the point cloud representation to obtain the densified point cloud features; S4. Based on the densified point cloud features, encode and obtain the corresponding three-plane projection feature tokens; S5. Based on the three-plane projection feature token and the image feature, decoding is performed to obtain the corresponding three-plane feature; S6. Based on the point cloud representation and the three-plane features, obtain the corresponding 3D Gaussian features; S7. Based on the 3D Gaussian features, decode to obtain the corresponding 3D Gaussian representation.

[0029] The embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0030] First, see Figures 1 and 2 As shown, an embodiment of the present application provides a single view reconstruction method, which includes the following steps: S1. Extracting image features corresponding to the input image based on the input image and the corresponding internal and external parameters of the camera; S2. Based on the image features corresponding to the input image, decode to obtain the corresponding point cloud representation; S3, perform feature enhancement on the point cloud representation to obtain the densified point cloud features; S4. Based on the densified point cloud features, encode and obtain the corresponding three-plane projection feature tokens; S5. Based on the three-plane projection feature token and the image feature, decoding is performed to obtain the corresponding three-plane feature; S6. Based on the point cloud representation and the three-plane features, obtain the corresponding 3D Gaussian features; S7. Based on the 3D Gaussian features, decode to obtain the corresponding 3D Gaussian representation.

[0031] In the existing technology, 3D reconstruction methods based on 2D diffusion models have the significant drawback of requiring long optimization times. For each object, a time-consuming optimization process is required to distill the diffusion model prior from the new perspective to optimize the 3D representation. Secondly, these methods often show insufficient consistency when generating 3D shapes, especially when considering from a single perspective, resulting in significant differences in the 3D model under different perspectives. In addition, due to the lack of clear 3D supervision, the optimized geometric structure often leads to the generation of multiple facial features, the so-called "Janus problem."

[0032] In addition, in single-view Figure 3 In the field of 3D reconstruction, regression methods are a common strategy for mapping information extracted from a single image to the corresponding 3D geometric representation. However, in existing implementations, regression methods are usually implemented using neural networks, and the network parameters are optimized through a training process to reconstruct 3D objects with the greatest accuracy. These methods usually use implicit methods (such as neural radiation fields) or explicit methods (point clouds, meshes) of image regression, but no matter which representation method is used, it is difficult to strike a balance between rendering speed and quality. Taking explicit representation as an example, although the rendering speed is faster, it is difficult to learn good generalization performance; based on implicit representation, although it is easy to learn good generalization performance, the time required for rendering and the consumption of video memory are greater.

[0033] In the embodiment of the present application, a novel hybrid 3D representation proposed by the present method is inferred directly from a single-view image through a Transformer inference model; The Transformer decoder is used to infer explicit point cloud representations and implicit three-plane features from single-view images. The point cloud representation is used to query 3D Gaussian features from the three-plane features, and finally a 3D Gaussian representation is obtained. While maintaining high reconstruction efficiency, the texture quality and rendering speed are significantly improved.

[0034] It should be noted that the technical solutions of the embodiments of this application are intended to address the efficiency and rendering quality issues of single-view 3D geometry reconstruction. Existing methods often suffer from slow optimization and rendering processes, resulting in long training and optimization cycles. Other methods may struggle to strike a balance between rendering speed and quality when directly regressing explicit 3D attributes or using implicit representations.

[0035] Furthermore, the regression method uses a combination of a point decoder and a three-plane decoder. The point decoder generates an explicit point cloud representation, and then uses these points through the three-plane decoder to query Gaussian features to better preserve the details of the 3D object. This design is a trade-off strategy to find a balance between explicit and implicit representations, aiming to achieve more efficient rendering speed and higher quality rendering results.

[0036] Furthermore, the step of extracting image features corresponding to the input image based on the input image and corresponding camera internal and external parameters includes the following steps: The camera internal and external parameters are mapped to a high-dimensional feature space, and image features with camera perspective perception are obtained based on the input image.

[0037] Furthermore, decoding the image features corresponding to the input image to obtain the corresponding point cloud representation includes the following steps: A point cloud decoder based on the Transformer architecture is used to decode the image features corresponding to the input image and obtain the corresponding point cloud representation.

[0038] Furthermore, the feature enhancement of the point cloud representation to obtain the densified point cloud features includes the following steps: The point cloud representation is feature enhanced, each point in the point cloud is projected onto the image plane, and the local image features corresponding to each point are obtained by interpolating the corresponding projection position on the image to obtain the densified point cloud features.

[0039] Furthermore, encoding to obtain corresponding three-plane projection feature tokens based on the densified point cloud features includes the following steps: Performing projection feature calculation on the densified point cloud features to obtain corresponding feature point clouds; The feature point cloud is feature encoded by PointNet and orthogonally projected onto three planes. The features falling in the same grid are mean pooled to obtain the corresponding three-plane projection feature tokens.

[0040] Furthermore, based on the three-plane projection feature token and the image feature, decoding to obtain the corresponding three-plane feature includes the following steps: A three-plane decoder of the Transformer architecture is used to decode the three-plane projection feature tokens to obtain the corresponding three-plane features under the injection of the image features.

[0041] Furthermore, based on the point cloud representation and the three-plane features, obtaining corresponding 3D Gaussian features includes the following steps: The point cloud representation is queried for corresponding spatial features and features at the projection of the input image in the three-plane features to obtain corresponding 3D Gaussian features.

[0042] Furthermore, based on the 3D Gaussian feature, decoding to obtain a corresponding 3D Gaussian representation includes the following steps: The MLP-based Gaussian decoder decodes the 3D Gaussian features to obtain corresponding 3D Gaussian representations.

[0043] The core of the technical solution of the embodiment of the present application is to propose a novel three-plane-Gaussian three-dimensional representation, from which a 3D Gaussian representation can be decoded for reconstruction and rendering, and a network structure is designed based on the representation to realize the inference of a three-dimensional model based on the three-plane-Gaussian representation from a single image. Taking the image and the internal and external parameters of the camera as input, the image encoder outputs the image features, and then the point cloud decoder and the three-plane decoder output the point cloud representation and the three-plane representation respectively. By querying the corresponding spatial features of the point cloud in the three planes, and decoding the final 3D Gaussian representation through the Gaussian decoder, it can be used for subsequent reconstruction and rendering.

[0044] The main operation process is as follows: The first step is to extract image features. For the input image and its camera intrinsic and extrinsic parameters, the camera intrinsic and extrinsic parameters are first mapped to a high-dimensional feature space through an MLP (Multilayer Perceptron). This feature is then injected into the ViT image feature extraction module to obtain input image features with camera perspective awareness for the subsequent three-plane-Gaussian 3D representation decoding process.

[0045] In the second step, based on the image features extracted in the previous step, a Transformer-based point cloud decoder is first used to decode the point cloud representation from the learnable initialization feature tokens. At the same time, the image features are injected into the decoding process through a cross-attention mechanism to guide the decoded point cloud results to be consistent with the image.

[0046] The third step is to perform feature enhancement on the decoded point cloud. Each point in the point cloud is projected onto the image plane and the corresponding local image features are obtained by interpolating the corresponding projected positions on the image. Due to limited computing resources, it is not possible to use a large number of point cloud feature tokens. Therefore, a point cloud upsampling network is used to densify the decoded point cloud. The local image features corresponding to the point cloud provide better guidance for the upsampling process.

[0047] In the fourth step, the densified point cloud is similarly projected to generate a feature point cloud. This is then encoded using a lightweight PointNet. Finally, the feature point cloud is orthogonally projected onto three planes. Features falling within the same grid are average-pooled to obtain a three-plane projection feature token for the point cloud. This feature token is added to the three-plane initialization feature token to ensure geometric consistency of the three-plane features.

[0048] In the fifth step, similar to the point cloud decoder, a three-plane decoder with the same architecture is used to decode the three-plane features from the three-plane initialized feature tokens under the injection of image features, and obtain higher resolution three-plane features through an image upsampling network.

[0049] In the sixth step, after decoding the point cloud and three-plane features, the three-plane-Gaussian three-dimensional representation proposed by this method can be obtained. The point cloud representation is queried in the three planes for the corresponding spatial features and the features at the projection of the input image to obtain the 3D Gaussian features.

[0050] In the seventh step, a Gaussian decoder based on MLP is used to decode the final 3D Gaussian representation from the Gaussian features. Based on this representation, the Gaussian splash rendering method can be used for tasks such as new perspective rendering.

[0051] It should be noted that the technical details of the technical solution of the embodiment of this application are as follows: First, in the first step of image feature encoding: First, the input image and camera features are encoded, and a state-of-the-art ViT model is used to obtain patch-based feature tokens from the input image. To make the image features aware of the camera's internal and external parameters, the camera's internal and external parameters are feature encoded and extracted for use in controlling the ViT model features. Specifically, the intrinsic parameters K∈R^(3×3) and the external parameters T∈R^(4×4) are flattened into one-dimensional vectors and concatenated. The concatenated camera internal and external parameter features are mapped to a high dimension through an MLP network to obtain their feature vectors. Secondly, an adaptive normalization module is used to control the ViT features through camera features. An MLP predicts the scale and offset based on the camera features after each normalization layer for image feature normalization, thereby achieving the control function of image features.

[0052] Second, image feature decoder and three-plane decoder: The technical solution of the embodiment of the present application designs a Transformer-based network structure, which uses the above-mentioned extracted image features to guide the reconstruction of point cloud representation and three-plane representation. The potential features of the point cloud and the three-plane are respectively encoded into a set of feature tokens, namely the point cloud feature token \{f_i^p | i∈\{1,…,N\}\} and the three-plane feature token \{f_i^t| i∈\{1,…,M\}\}. These feature tokens are initialized by learnable position embedding and sent to the multi-layer Transformer module. Each layer of the Transformer structure includes a self-attention layer, a cross-attention layer and an MLP layer. Through the cross-attention layer, the image features enhanced by the perspective information can interact with the point cloud features and the three-plane features respectively to guide the decoding process of the point cloud representation and the three-plane representation.

[0053] Point cloud decoding maps each feature token f_i^p to each point in the point cloud and decodes the corresponding coordinates from the feature tokens through an MLP network. Because the Transformer network's inherent computational and memory requirements cannot handle a large number of tokens simultaneously, the point cloud decoding process initially outputs only a small number of points to construct a rough geometric shape. Based on this initial sparse point cloud, a point cloud upsampling network is then connected to densify the point cloud for improved reconstruction and rendering.

[0054] The three-plane decoder provides the implicit feature field required to decode the 3D Gaussian attributes. By querying the 3D Gaussian attributes of the corresponding position of the explicit point cloud in the implicit feature field, the final 3D Gaussian representation can be obtained. Each output feature token f_i^t is treated as a small image block in the three planes, and then the M×C-dimensional matrix of all feature tokens is pieced together and reshaped into a 3×H×W×C-dimensional tensor, where H and W are the length and width of each plane in the three-plane representation. Similar to the point cloud decoder, the network structure of the Transformer is also limited by the number of tokens used, so M=3×32×32 is set. A deconvolution layer is further used to achieve upsampling of the feature map to improve the resolution of the three-plane representation, and finally a three-plane representation with a resolution of 3×64×64×C is obtained.

[0055] Third, feature enhancement of point cloud representation: The technical solution of the embodiment of the present application introduces projection features in both point cloud upsampling and 3D Gaussian representation decoding to provide more local and detailed information. The projection feature utilizes the explicit geometry provided by the point cloud, projects the geometry onto the image plane of the input perspective through the projection function π, and obtains the local features of the image at the projection location. The low-level local features can better describe the details of the reconstructed target. In addition, the projection function explicitly establishes the geometric relationship between the three-dimensional representation and the input features, which can provide more detailed local features better and more directly than the cross-attention mechanism that relies entirely on learning. Assuming that the camera pose T of the input perspective, the camera intrinsic parameter K and the reconstructed point cloud P are given, the projection position can be calculated by p=π(TP,K), and the feature f_p of the projection position can be obtained by trilinear interpolation on the feature map. Taking into account the problem of point cloud self-occlusion, the present invention adopts a point cloud rasterization method based on depth sorting. Each pixel only considers the point cloud closest to the camera, and the projection features of the remaining occluded point clouds are considered to be zero vectors. The technical solution of the embodiment of the present application splices the original RGB image, the extracted DINOv2 feature map (upsampled to the resolution of the original image by interpolation), the object mask, and the distance transformation map calculated based on the mask to obtain a complete feature image. The acquired projection features will be applied to the point cloud and 3D Gaussian feature decoding. Since the quality of Gaussian splash rendering is affected by the number of 3D Gaussians, it is not enough to generate a 3D Gaussian representation from a low-resolution (N=2048 points) point cloud. The technical solution of the embodiment of the present application uses a two-step point cloud deconvolution module to densify the point cloud from 2048 points to 16384 points. In order to utilize the local information of the input image to guide the local upsampling process, thereby improving the final reconstruction quality of the point cloud.

[0056] Fourth point, geometric perception encoding: Since the point cloud decoder and the three-plane decoder are independent of each other in the whole model, only using rendering supervision cannot guarantee that the explicit representation and the implicit representation can be well aligned. In order to increase the association between the two representations, especially to make the three-plane features have better geometric perception, the technical solution of the embodiment of the present application encodes the features of the point cloud into a learnable three-plane position embedding. First, the projection features of the step are attached to the point cloud coordinates, that is, each point performs vector splicing of its corresponding projection features and spatial coordinates to obtain a point cloud with feature enhancement. Then, the enhanced feature point cloud is sent to a shallow PointNet network with local pooling for point cloud feature extraction, and is projected onto three axis-aligned planes through orthogonal projection. After the features projected into the same block are subjected to the mean pooling operation, the features are superimposed with the corresponding learnable position embedding, thereby realizing geometric perception encoding to better align the point cloud and the three planes.

[0057] Fifth point, three-plane-Gaussian mixed 3D representation: The technical solution of the embodiments of the present application relates to a new three-dimensional Gaussian representation method called three-plane-Gaussian, which aims to apply Gaussian rasterization technology to general reconstruction. Although Gaussian rasterization has advantages in rendering and speed, it has not been fully explored in general reconstruction. The present invention first considers 3D Gaussian as a special point cloud, and attempts to add additional 3D Gaussian properties on the basis of the existing point cloud model to achieve a simple 3D Gaussian reconstruction task. However, the initial attempt to directly predict the 3D Gaussian failed to generate a satisfactory 3D object from a single image. To solve this problem, the present invention introduces three-plane-Gaussian, which is a hybrid 3D representation that combines the advantages of three-plane and point cloud methods to provide fast rendering, high quality and versatility. The introduction of three-plane-Gaussian fills the shortcomings of previous methods, enabling better three-dimensional reconstruction in various scenarios.

[0058] Hybrid Triplanar-Gaussian representation containing a point cloud As explicit geometry, and a three-plane Encode the implicit feature field, and the final 3D Gaussian representation can be decoded from it. The three-plane is obtained by combining three two-dimensional feature planes Organized orthogonally, a three-dimensional feature field can be expressed compactly and efficiently.

[0059] For any position in space , it can be orthogonally projected onto the three characteristic planes to obtain its various projection positions , and calculate the feature vector at the projection location using trilinear interpolation in the three feature maps according to the projection location. For example, On the characteristic plane, we can get , similarly, the projection features of other feature planes can be obtained. Finally, these three feature vectors are concatenated to obtain the spatial features corresponding to position $p$ , Represents a vector concatenation operation.

[0060] Sixth point, 3D Gaussian decoder: A 3D Gaussian decoder is introduced to process a given position in the point cloud P. After querying the features from the three planes, the MLP is used to decode the properties of the 3D Gaussian generated by the point cloud, including opacity, anisotropic covariance (scale and rotation representation) and spherical harmonic (SH) coefficients. Figure 3 The 3D reconstruction task combines triplanar features with projected image features to fully utilize the local features of the input image. Local projected features are calculated using the projection function P, taking into account self-occlusion in the point cloud and implementing projection through point rasterization. Local features include RGB color, DINOv2 features, masks, and 2D distance transforms of the masked regions. This projection-aware condition improves texture quality across the input viewport.

[0061] For each point in the point cloud According to the previous point, the corresponding spatial features can be queried from the three-plane feature field , and finally through an MLP-based decoder The corresponding 3D Gaussian properties can be decoded from it, including opacity , anisotropic covariance spherical harmonic coefficients Etc. The covariance of each anisotropy can be split and represented as a scale vector and a rotation quaternion . 3D Gaussian decoding is shown as follows: In addition, since the point cloud describes the points on the geometric surface of the reconstructed target, and the 3D Gaussian representation is composed of a Gaussian ellipsoid with scale in three directions, its center theoretically will not appear on the geometric surface. To solve the above problem, the technical solution of the embodiment of the present application additionally predicts a small offset vector for the 3D Gaussian center position. , so its final position is calculated as The invention in this chapter also makes full use of the local features of the input image and enhances the features of the three-plane query by the features of the explicit geometry at the input view projection to obtain a more refined texture reconstruction result.

[0062] In summary, the key technical points of the technical solution of the embodiment of the present application are: (1) Hybrid 3D representation of triplane-Gaussian: A new 3D Gaussian representation method called triplane-Gaussian is introduced, which combines the advantages of triplane and point cloud methods and aims to provide fast rendering, high quality and versatility in general reconstruction.

[0063] (2) 3D Gaussian decoder: A 3D Gaussian decoder for processing point clouds is introduced. The properties of the generated 3D Gaussian are decoded by MLP, including opacity, anisotropic covariance, and spherical harmonic coefficients. By projecting the perception condition, the local features of the input image are fully utilized to improve the texture quality.

[0064] (3) Point cloud and triplane decoder: A set of feature tokens are used to represent the latent space features of different 3D representations of point cloud and triplane. The Transformer architecture is used to guide the two decoders through cross attention to finally decode the representation of point cloud and triplane.

[0065] (4) Geometry-aware coding: The Triplane decoder outputs an implicit feature field from the image and the initial point cloud, from which 3D Gaussian attributes are decoded through position queries. Geometry awareness is introduced, and the same projection features as those used for upsampling the point cloud are used to enhance the point, improving the perception of geometric structure.

[0066] The technical advantages of the above key technical points are: (1) Versatility of Three-Plane-Gaussian Mixture 3D Representation: This embodiment introduces a three-plane-Gaussian mixture 3D representation method that combines Gaussian rasterization technology and is versatile and applicable to various reconstruction scenarios. This hybrid representation provides fast rendering, high quality, and versatility, and has greater flexibility in versatility than traditional methods.

[0067] (2) Improved texture quality of the 3D Gaussian decoder: By introducing the 3D Gaussian decoder, the embodiments of the present application can generate 3D objects with greater texture and accuracy. In particular, by effectively utilizing the projected perception condition, the local features of the input image are utilized, improving the texture quality of the generated 3D objects and achieving better visual effects than the existing technology.

[0068] (3) Comprehensive understanding of point cloud and tri-plane decoders: By using the Transformer architecture for point cloud and tri-plane decoders, the present embodiment has a stronger comprehensive ability to understand and parse different 3D representations. This helps to more comprehensively and accurately restore the features of the original point cloud and tri-plane, and achieves better reconstruction results than a single method.

[0069] (4) Improved geometry-aware coding: Through the geometry-aware coding of the Triplane decoder, the present embodiment improves the decoding capability of implicit feature fields when processing images and initial point clouds. This geometry-aware coding helps to better understand geometric structures, especially when parsing 3D Gaussian properties.

[0070] (5) Handling of self-occlusion: The present embodiment implements point cloud projection through point rasterization, taking into account the self-occlusion problem of point clouds. This processing method helps to more accurately restore the geometric relationships in the scene and is more robust than some existing technologies.

[0071] In summary, the embodiments of the present application have obvious advantages over the best existing technologies in terms of versatility, texture quality, comprehensive understanding and geometric perception.

[0072] Furthermore, in order to adapt to different usage scenarios, the embodiments of the present application may be implemented by using the following technical means to replace the solutions: (1) Traditional point cloud methods: Traditional point cloud reconstruction methods have some effectiveness in reconstructing 3D scenes. These methods may use techniques such as point cloud sampling and interpolation, but compared to deep learning methods, they may perform poorly in handling complex scenes and improving rendering quality.

[0073] (2) Voxel-based methods: Some voxel-based methods represent the scene as a 3D voxel grid and restore the 3D scene by filling the voxels. These methods can provide a certain degree of geometric information, but may have challenges in details and efficiency.

[0074] (3) Other deep learning methods: In addition to the described three-plane-Gaussian representation, there are other deep learning methods for 3D scene reconstruction, such as image-to-point cloud generative models or point cloud processing using convolutional neural networks.

[0075] (4) Ray tracing-based methods: Ray tracing technology can be used for high-quality image synthesis and rendering, and can also be applied to the modeling and rendering of 3D scenes. However, these methods may require more computing resources.

[0076] (5) Combining traditional geometric methods and deep learning methods: Combining traditional geometric methods and deep learning techniques to take advantage of their respective strengths may also be an alternative.

[0077] Second, see Figure 3 As shown, an embodiment of the present application provides a single-view reconstruction device, which includes: An image feature extraction module is used to extract image features corresponding to the input image based on the input image and corresponding camera internal and external parameters; An image feature decoding module, configured to decode the image features corresponding to the input image to obtain a corresponding point cloud representation; A point cloud representation enhancement module, which is used to enhance the features of the point cloud representation to obtain densified point cloud features; A point cloud feature encoding module, which is used to encode the densified point cloud features to obtain corresponding three-plane projection feature tokens; A feature token decoding module, configured to decode the three-plane projection feature token and the image feature to obtain the corresponding three-plane feature; A 3D Gaussian feature acquisition module, configured to acquire corresponding 3D Gaussian features based on the point cloud representation and the three-plane features; The 3D Gaussian feature decoding module is used to decode and obtain the corresponding 3D Gaussian representation based on the 3D Gaussian feature.

[0078] In an embodiment of the present application, a Transformer decoder is used to infer explicit point cloud representations and implicit three-plane features from a single-view image, and the point cloud representation is used to query 3D Gaussian features from the three-plane features, ultimately obtaining a 3D Gaussian representation. This significantly improves texture quality and rendering speed while maintaining high reconstruction efficiency.

[0079] Furthermore, the image feature extraction module is further configured to map the camera internal and external parameters to a high-dimensional feature space, and obtain image features with camera perspective perception based on the input image.

[0080] Furthermore, the image feature decoding module is also used to use a point cloud decoder of the Transformer architecture to decode the image features corresponding to the input image and obtain the corresponding point cloud representation.

[0081] Furthermore, the point cloud representation enhancement module is also used to perform feature enhancement on the point cloud representation, project each point in the point cloud onto the image plane, obtain the local image features corresponding to each point by interpolating the corresponding projection position on the image, and obtain the densified point cloud features.

[0082] Furthermore, the point cloud feature encoding module is further used to perform projection feature calculation on the densified point cloud features to obtain corresponding feature point clouds; The point cloud feature encoding module is also used to encode the feature point cloud through PointNet, and perform orthogonal projection onto three planes, perform mean pooling on the features falling in the same grid, and obtain the corresponding three-plane projection feature tokens.

[0083] Furthermore, the feature token decoding module is also used to use a three-plane decoder of the Transformer architecture to decode and obtain corresponding three-plane features from the three-plane projection feature tokens under the injection of the image features.

[0084] Furthermore, the 3D Gaussian feature acquisition module is further used to query the corresponding spatial features and features at the projection of the input image in the three-plane features of the point cloud representation to obtain the corresponding 3D Gaussian features.

[0085] Furthermore, the 3D Gaussian feature decoding module is also used to decode the 3D Gaussian feature based on an MLP Gaussian decoder to obtain a corresponding 3D Gaussian representation.

[0086] It should be noted that the technical problems, technical solutions and technical effects of the single-view reconstruction device mentioned in the embodiment of the present application are similar in technical principles to those of the single-view reconstruction method mentioned in the first aspect, so they will not be elaborated here.

[0087] In the description of this application, it should be noted that the terms "upper" and "lower" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be internal communication between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0088] It should be noted that, in this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0089] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A single view reconstruction method, characterized in that: The method comprises the following steps: Based on the input image and the corresponding internal and external parameters of the camera, extract the image features corresponding to the input image; Based on the image features corresponding to the input image, decoding obtains the corresponding point cloud representation; Performing feature enhancement on the point cloud representation to obtain a densified point cloud feature; Based on the densified point cloud features, encoding obtains corresponding three-plane projection feature tokens; Based on the three-plane projection feature token and the image feature, decoding obtains the corresponding three-plane feature; Based on the point cloud representation and the three-plane features, obtaining corresponding 3D Gaussian features; Based on the 3D Gaussian features, decoding is performed to obtain corresponding 3D Gaussian representations.

2. The single view reconstruction method according to claim 1, characterized in that: The step of extracting image features corresponding to the input image based on the input image and the corresponding internal and external parameters of the camera comprises the following steps: The camera internal and external parameters are mapped to a high-dimensional feature space, and image features with camera perspective perception are obtained based on the input image.

3. The single view reconstruction method according to claim 1, characterized in that: The decoding to obtain the corresponding point cloud representation based on the image features corresponding to the input image includes the following steps: A point cloud decoder based on the Transformer architecture is used to decode the image features corresponding to the input image to obtain a corresponding point cloud representation.

4. The single view reconstruction method according to claim 1, characterized in that: The step of performing feature enhancement on the point cloud representation to obtain a densified point cloud feature comprises the following steps: The point cloud representation is feature enhanced, each point in the point cloud is projected onto the image plane, and the local image features corresponding to each point are obtained by interpolating the corresponding projection position on the image to obtain the densified point cloud features.

5. The single view reconstruction method according to claim 1, characterized in that: The step of encoding and obtaining corresponding three-plane projection feature tokens based on the densified point cloud features comprises the following steps: Performing projection feature calculation on the densified point cloud features to obtain corresponding feature point clouds; The feature point cloud is feature encoded by PointNet and orthogonally projected onto three planes. The features falling in the same grid are mean pooled to obtain the corresponding three-plane projection feature tokens.

6. The single view reconstruction method according to claim 1, characterized in that: Based on the three-plane projection feature token and the image feature, decoding to obtain the corresponding three-plane feature includes the following steps: Using a three-plane decoder of the Transformer architecture, under the injection of the image features, the corresponding three-plane features are decoded from the three-plane projection feature tokens.

7. The single view reconstruction method according to claim 1, characterized in that: Based on the point cloud representation and the three-plane features, obtaining corresponding 3D Gaussian features includes the following steps: The point cloud representation is queried for corresponding spatial features and features at the projection of the input image in the three-plane features to obtain corresponding 3D Gaussian features.

8. The single view reconstruction method according to claim 1, characterized in that: Based on the 3D Gaussian feature, decoding to obtain the corresponding 3D Gaussian representation includes the following steps: The MLP-based Gaussian decoder decodes the 3D Gaussian features to obtain corresponding 3D Gaussian representations.

9. A single view reconstruction device, characterized in that: The device comprises: An image feature extraction module, which is used to extract image features corresponding to the input image based on the input image and corresponding camera internal and external parameters; An image feature decoding module, which is used to decode and obtain a corresponding point cloud representation based on the image features corresponding to the input image; A point cloud representation enhancement module, which is used to enhance the features of the point cloud representation to obtain densified point cloud features; A point cloud feature encoding module, which is used to encode and obtain corresponding three-plane projection feature tokens based on the densified point cloud features; A feature token decoding module, which is used to decode and obtain corresponding three-plane features based on the three-plane projection feature tokens and the image features; A 3D Gaussian feature acquisition module, which is used to acquire corresponding 3D Gaussian features based on the point cloud representation and the three-plane features; The 3D Gaussian feature decoding module is used to decode and obtain the corresponding 3D Gaussian representation based on the 3D Gaussian feature.

Images