A method of monoscopic reconstruction and rendering

Abstract

This invention discloses a method for single-view reconstruction and rendering, belonging to the field of view reconstruction and rendering. The method includes: firstly, generating a multi-view feature representation with strong geometric consistency from a single input image by introducing an image diffusion module with a cross-attention mechanism; then, using a point cloud reconstruction module that combines self-attention and cross-attention, fusing multi-view information to reconstruct an accurate 3D point cloud; finally, constructing a differentiable 3D Gaussian representation based on this point cloud and rendering it to output a new view image, normal map, and depth map. The system adopts a staged training strategy and innovatively uses a composite loss function including multi-scale bidirectional consistency smoothing loss and feature consistency loss for optimization. This invention effectively solves the problems of low geometric accuracy, inconsistency among multiple views, and lack of detail in single-view reconstruction, and can be widely applied in fields such as virtual reality, digital twins, and digitization of cultural heritage.

Description

Technical Field

[0001] This invention belongs to the field of view reconstruction and rendering, and particularly relates to a method for single-view reconstruction and rendering. Background Technology

[0002] Single-view 3D reconstruction aims to recover the 3D geometry and appearance of an object from a single 2D image, and has significant application value in fields such as virtual reality and digital twins. In recent years, deep learning-based methods have made significant progress in this direction. On the one hand, Neural Radiation Fields (NeRF) and its variants achieve high-quality view synthesis through implicit scene representation, but are sensitive to the number of input viewpoints, have limited generalization ability under single-view conditions, and are slow in reconstruction speed. On the other hand, explicit representation methods such as 3D Gaussian Spraying (3DGS) achieve real-time, high-fidelity new viewpoint generation through differentiable rendering, but they rely on accurate multi-view input as initialization and cannot be directly applied to single-image scenes. Meanwhile, generative models, especially diffusion models, show potential in generating multi-view images from single images, but these methods often focus on image-level realism and lack explicit modeling and strong constraints on the consistency of the underlying 3D geometry, leading to difficulties in aligning the generated multi-view images in 3D space. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a method for single-view reconstruction and rendering, comprising:

[0004] Take a single-view input image, and based on the single-view input image, generate a consistent feature representation across multiple perspectives by using an image diffusion module that integrates image diffusion priors and an attention mechanism;

[0005] Based on the aforementioned multi-view consistent feature representation, a 3D point cloud is reconstructed using a point cloud reconstruction module that includes self-attention and cross-attention calculations.

[0006] Based on the three-dimensional point cloud, a three-dimensional Gaussian representation is constructed, and a rendering output containing at least a new perspective color image is obtained through differentiable Gaussian rendering.

[0007] The training of the image diffusion module and the point cloud reconstruction module adopts a phased training strategy and is optimized using a total loss function that includes reconstruction loss, distance alignment loss, normal alignment loss, multi-scale bidirectional consistency smoothing loss and feature consistency loss.

[0008] Optionally, the step of generating a consistent feature representation across multiple views based on the single-view input image, through an image diffusion module that fuses image diffusion priors and an attention mechanism, specifically includes:

[0009] Encode the single-view input image into initial features;

[0010] The initial features are fused with noise features from the diffusion process and then fed into a transformer with a cross-attention mechanism for processing to enhance the multi-view generation of features. Figure 1 To the point of being compatible;

[0011] The processed features are decomposed into multiple feature maps corresponding to different preset viewpoints using a perceptron.

[0012] Optionally, based on the multi-view consistent feature representation, the reconstruction of the 3D point cloud through a point cloud reconstruction module that includes self-attention and cross-attention calculations specifically includes:

[0013] The multiple feature maps are respectively input into multiple parallel processing branches;

[0014] In each branch, self-attention calculation based on the internal feature map of a single branch is performed alternately, as well as cross-attention calculation based on the query matrix of the current branch feature map and the key-value matrix of the other branch feature maps.

[0015] The feature maps output by each branch after attention calculation are fused pixel by pixel, and a point cloud map representing the three-dimensional spatial coordinates is output by a decoding perceptron.

[0016] Optionally, the construction of the three-dimensional Gaussian representation specifically includes:

[0017] Using the positions of each point in the three-dimensional point cloud as initial centers, a series of three-dimensional Gaussian distributions are initialized.

[0018] Each three-dimensional Gaussian distribution is parameterized by its center position, covariance matrix controlling its shape and orientation, opacity, and view-dependent color attributes.

[0019] Optionally, obtaining the rendering output containing at least a new perspective color image through differentiable Gaussian rendering specifically includes:

[0020] Sort according to the depth from the center of each Gaussian distribution in the three-dimensional Gaussian representation to the viewpoint of the target camera;

[0021] Based on the sorting results, for each screen pixel, the color values ​​of the Gaussian distribution that contribute to it are accumulated through alpha mixing to calculate the new perspective color image; the weight of the alpha mixing is determined based on the opacity and cumulative transmittance of each Gaussian distribution.

[0022] Optionally, the rendering output may further include a normal map and a depth map;

[0023] The normal map is calculated by transforming the normal vectors of each Gaussian distribution to the camera coordinate system and then using the same alpha blending mechanism as that used for rendering color images.

[0024] The depth map is calculated by accumulating the distances of each Gaussian distribution to the camera plane according to the same alpha mixing mechanism.

[0025] Optionally, the multi-scale bidirectional consistent smoothing loss is calculated in the following way:

[0026] At multiple scales of the image pyramid, the forward consistency smoothing loss and the reverse consistency smoothing loss are calculated separately and then weighted and summed.

[0027] The forward consistency smoothing loss is used to constrain the structure of the rendered normal map to be consistent with the structure of the prior normal map predicted by the pre-trained deep network.

[0028] The reverse consistency smoothing loss is used to constrain the structure of the prior normal map to be consistent with the structure of the rendered normal map when the confidence of the prior normal map is low.

[0029] Optionally, the feature consistency loss is calculated in the following manner:

[0030] Extract depth features from the reference image and the reconstructed image;

[0031] At multiple scales, the similarity between the depth features of the reference image and the depth features of the reconstructed image is calculated and weighted according to a confidence mask.

[0032] The deep features are obtained from a pre-trained and frozen deep feature extraction network.

[0033] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0034] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] This invention first introduces cross-view feature fusion and cross-attention mechanisms into the diffusion model, significantly improving the geometric consistency of the generated multi-view features in 3D space and laying a reliable foundation for subsequent reconstruction. Secondly, a dedicated point cloud reconstruction module is designed, explicitly reasoning and fusing multi-view information through alternating self-attention and cross-attention operations, achieving a robust mapping from consistent features to accurate 3D point clouds. Finally, an innovative composite loss function including multi-scale bidirectional smoothing constraints and feature consistency regularization is proposed. Guided by a multi-stage training strategy, the model is jointly optimized from multiple dimensions such as geometric alignment, structural smoothing, and cross-view semantic consistency. Experiments show that this invention can generate 3D models with accurate geometric structure, rich texture details, and high consistency across multiple views with only a single input image. On public datasets, the geometric error (Chamfer Distance) and visual quality metrics (PSNR, SSIM) are significantly better than traditional methods such as neural radiation fields, multi-view stereo matching, and native 3D Gaussian spraying. Attached Figure Description

[0037] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0041] Example 1

[0042] like Figure 1 As shown, this embodiment provides a method for single-view reconstruction and rendering, including:

[0043] Take a single-view input image, and based on the single-view input image, generate a consistent feature representation across multiple perspectives by using an image diffusion module that integrates image diffusion priors and an attention mechanism;

[0044] Based on the aforementioned multi-view consistent feature representation, a 3D point cloud is reconstructed using a point cloud reconstruction module that includes self-attention and cross-attention calculations.

[0045] Based on the three-dimensional point cloud, a three-dimensional Gaussian representation is constructed, and a rendering output containing at least a new perspective color image is obtained through differentiable Gaussian rendering.

[0046] The training of the image diffusion module and the point cloud reconstruction module adopts a phased training strategy and is optimized using a total loss function that includes reconstruction loss, distance alignment loss, normal alignment loss, multi-scale bidirectional consistency smoothing loss and feature consistency loss.

[0047] The entire process is divided into an image diffusion module, a point cloud reconstruction module, an image output module, and a Gaussian reconstruction module.

[0048] Image diffusion module:

[0049] (1) Based on standard Stable Diffusion, the output features of the input image after VAE encoding are concatenated and fused with the noise features of the current diffusion step, and then input into a separate Transformer. However, self-attention calculation is not performed. Instead, the generated K and V matrices are concatenated and fused with the K and V matrices of the noise from the current diffusion step, and finally, Transformer self-attention calculation is performed. The aim is to improve the multi-view capability of the "image output module". Figure 1 To the point of being responsive.

[0050] (2) The function of the multilayer perceptron MLP-1 is to divide the features output after diffusion into 6 feature maps according to 6 multi-views, so that they can be used as input for other modules.

[0051] Image output module:

[0052] (1) The 6 feature maps are merged and input into the VAE decoder, and the output azimuth angles are respectively , , and superimposed The pitch angle is used to generate images from 6 perspectives. Instead of outputting 6 separate views, the images from the 6 perspectives are merged into one image.

[0053] (2) The role of the “Image Output Module” is to pre-train the “Image Diffusion Module”, so that each module in the end-to-end architecture is precise and controllable, enhances interpretability, reduces redundant encoders, and speeds up training.

[0054] Point cloud reconstruction module:

[0055] (1) The six feature maps output by the “Image Diffusion Module” are input into the “Point Cloud Reconstruction Module” to perform self-attention within a single image and cross-attention between multiple images.

[0056] (2) The “point cloud reconstruction module” is composed of multiple Transformers linearly stitched together, with 8 Transformers in each group, for a total of 3 groups. In each group, the first and last 3 Transformers are self-attention, which calculates the attention between patches within a single image. The middle 2 Transformers are cross-attention, which calculates the attention between patches in other images. Specifically, the Q matrix of the current feature map is used to perform cross-attention calculation with the K and V matrices of another feature map.

[0057] (3) There are a total of 6 feature maps, which is equivalent to 6 branches of the "point cloud reconstruction module". Each time the attention is crossed, each branch will generate 5 feature maps, which need to be added and fused pixel by pixel. Similarly, before the multilayer perceptron MLP-2, the feature maps generated by the 6 branches also need to be added and fused pixel by pixel. Finally, the MLP-2 decodes and outputs a point cloud map. Each pixel in the point cloud map corresponds to a three-dimensional coordinate point in 3D space.

[0058] Gaussian Reconstruction Module:

[0059] (1) Construct a point cloud using a point cloud map, and then, according to the standard 3D Gaussian Splatting, represent a scene as a set of 3D Gaussian distributions (a Gaussian ellipsoid is a Gaussian distribution). Each Gaussian distribution consists of the center position of the Gaussian distribution (the initial value is the position of the point cloud center) and a covariance matrix that controls the shape and direction of the Gaussian distribution.

[0060] (2) During rendering, the rasterization-based alpha blending method is used for rendering, that is, through volume rendering, the transmittance, the current Gaussian blending weight and the view-related color value are multiplied and accumulated to obtain the final screen pixel color value.

[0061] ;

[0062] in, Represents screen pixels The final color (or the final color of the pixels in the new view image). Indicates to A set of contributing 3D Gaussian distributions sorted by depth from near to far. In other words, before rendering, all Gaussian distributions that may intersect with camera rays need to be sorted according to the depth of their centers from the camera. It is transmittance, representing the transmittance at the 1st twentieth ... Before a Gaussian distribution, The probability that a ray emitted from a point penetrates all Gaussian distributions in front of it without being blocked is given by the formula: . Indicates the first A Gaussian distribution pair Opacity or blend weight. Indicates the first A Gaussian distribution pair And color values ​​under a specific viewpoint.

[0063] (3) The alpha blending mechanism of color rendering can be extended to the rendering of normals and distances. In this way, normal maps and distance maps can also be calculated.

[0064] ;

[0065] ;

[0066] in, Represents screen pixels A set of contributing 3D Gaussian distributions sorted by depth from near to far. It is transmittance, representing the transmittance at the 1st twentieth ... Before a Gaussian distribution, The probability that a ray emitted from a point penetrates all Gaussian distributions in front of it without being blocked is given by the formula: . Indicates the first A Gaussian distribution pair Opacity or blend weight. Indicates the first A Gaussian-distributed unit normal vector. This represents the rotation matrix from the world coordinate system to the camera coordinate system, because It is defined in the world coordinate system, and needs to be transformed to the camera coordinate system for rendering. yes transpose, Indicates the first [number]th ... The normals of a Gaussian distribution. express The final rendered normal in the camera coordinate system. express The final rendering distance. Indicates the distance from the camera center to the... The perpendicular distance to the local plane defined by a Gaussian distribution. A screen pixel. The normal value is The normal values ​​of all screen pixels form a normal map. One screen pixel The distance value is The distance values ​​of all screen pixels form a distance map. .

[0067] Training strategy:

[0068] (1) Step 1: Pre-train the diffusion module + "image output module, using the loss function At this point, the "point cloud reconstruction module" is not trained.

[0069] (2) Second step: Train the diffusion module and the point cloud reconstruction module, and freeze the parameters of the diffusion module. Only the point cloud reconstruction module is trained, using the loss function. Supports end-to-end fine-tuning.

[0070] (3) After training, the diffusion module + point cloud reconstruction module are used for inference. On the one hand, an end-to-end framework is realized, and on the other hand, the accuracy, controllability and interpretability of each module are enhanced.

[0071] Total loss function

[0072]

[0073] in, It is a reconstruction loss. It is the distance alignment loss. and It is normal alignment loss. It is a smoothing loss. It is feature consistency loss. , , , and It is the weighting coefficient.

[0074] Reconstruction losses Reconstruction loss is the most basic loss function and the cornerstone of subsequent loss functions. It ensures that the image rendered from the predicted 3D Gaussian representation is as close as possible to the true value in both color and geometry.

[0075] ;

[0076] in, and These are weighting coefficients. It is a loss in photometric reconstruction. It is structural similarity loss.

[0077] (1) Photometric reconstruction loss: By calculating the reconstructed image Compared with reference image The mean absolute error in pixel color values ​​directly constrains the accuracy of color and brightness in the output image.

[0078] ;

[0079] in, It is a reconstructed image, that is, an RGB image obtained by rendering a Gaussian distribution under the same camera parameters as the reference image. It is a reference image, also known as a real image. yes The total number of pixels is used for normalization. This represents the L1 norm.

[0080] (2) Structural similarity loss: By maximizing the similarity of two images in terms of structure, brightness and contrast (i.e. maximizing SSIM and minimizing the structural similarity loss function), it helps to preserve the structural information of the image and produce a more natural and clearer result, which is a perfect complement to L1 loss.

[0081] ;

[0082] in, It is to reconstruct the image. The reference image is SSIM, which is the structural similarity formula.

[0083] Distance alignment loss ;

[0084] ;

[0085] in, It is a distance map calculated from a Gaussian distribution. Represents pixel coordinates, Indicates pixel coordinates Distance at location, Indicates the first The distance in pixels, expressed differently, has the same meaning. . Using a pre-trained deep estimation network Predicted (input reference image) ), It will also predict disparity maps, which are used to calculate consistency errors. It is a continuous confidence plot. ,in It is a consistency error. This is a hyperparameter. Used to Inaccurate predictions Filter it out. Is with , A continuous-value mask of the same resolution, where each pixel consists of a number from 0 to 1. The closer the value is to 1, the better the position. High confidence level of predicted values, and vice versa A value close to 0 indicates a low confidence level.

[0086] Normal alignment loss , By using double normal alignment loss, robustness is enhanced and local optima are avoided.

[0087] ;

[0088] ;

[0089] in, It is a continuous confidence plot. This represents the cosine similarity. Represents pixel coordinates, Indicates pixel coordinates The normal at the point, Indicates the first The normal of a pixel, the two are expressed differently, but have the same meaning, that is . From The calculated result is for Calculate the spatial gradient of the image, and then derive the normal map using the following formula: ,in Indicates to Find the spatial gradient of the image Quantity, Indicates to Find the spatial gradient of the image Quantity, This indicates standardized operations. Similarly, From The calculated formula is: .

[0090] Multi-scale bidirectional consistent smoothing loss Bidirectional consistency not only allows the rendering results to learn from prior knowledge (i.e., the model) The predicted results also allow the prior to learn from the rendered results in the reliable region, forming a bidirectional, co-evolutionary smoothing process. Multi-scale smoothing involves calculating the smoothness loss at different scales of the image pyramid to capture consistency from macroscopic structure to microscopic details.

[0091] ;

[0092] in, Indicates the dimensions of the pyramids. This indicates the total number of scales. This indicates the normal map after Gaussian distribution rendering at the th... Downsampling results at different scales. express In the Downsampling results at different scales, and From Calculated. The input image is at the 1st Downsampling results at different scales. It is a scale weight decay factor, which gives higher weight to coarse scales.

[0093] Indicates forward consistency, forces rendering of normals. Structure and prior normal The structures are consistent. Indicates reverse consistency, enforces prior normals Structure and rendering normals The structures are consistent. ,in It is a small constant that prevents division by zero. Represents pixel coordinates, Represents the set of all pixel coordinates. Represents a set The number of all elements. Describing the L2 norm, This represents the Laplace operator. Formula By dividing by the gradient magnitude of the RGB image This allows for an automatic reduction in loss in regions with rich textures and sharp edges (large denominator), thus better preserving details. In flat regions (small denominator), it strengthens smoothing constraints, more effectively suppressing noise.

[0094] and There are two adaptive weights. , , here yes The downsampling map remains the same. At the same scale, the resolution is the same. This means that in regions with high prior confidence, we trust the prior, which in turn means we have more faith in the model. The predicted result, therefore forced Structure and The structures are consistent. Represents a rendered image For quality estimation, PSNR or SSIM can be used as quality assessment metrics. This means that when the prior confidence level is low, or the rendering quality is high, we are more likely to believe the rendering result, thus forcing... Structure and The structures are consistent.

[0095] Feature consistency loss Used to constrain the rendered features to maintain multiview under randomly generated pseudoviews. Figure 1 Consistency means that the geometry and appearance must also be consistent with the training view. This is a very powerful regularization technique that directly addresses the core of the overfitting problem—the model only performs well under the training view.

[0096] ;

[0097] in, yes The downsampling map remains the same. At the same scale, the resolution is the same. This represents the cosine similarity. Represents reference image The feature representation, by The input is fed into a pre-trained and frozen deep feature extraction network to obtain feature maps. . It is to reconstruct the image. (i.e., the feature representation of the image rendered with a Gaussian distribution) will Input to The feature maps obtained in the same pre-trained and frozen deep feature extraction network . Indicating the scale of the pyramids (e.g.) , , wait), These are learnable scale weights, allowing the network to autonomously determine which scales are more important for the geometric consistency of the current scene during training. Multi-scale supervision provides richer gradient signals for training, preventing the model from getting stuck in local optima at a single scale, thereby accelerating convergence and improving the overall quality of the reconstructed surface. Small-scale image patches can capture subtle texture features, making them suitable for supervising local geometric details; large-scale image patches contain more contextual information and structural features, better handling weakly textured regions and improving the robustness of matching.

[0098] The technical effect of this embodiment is as follows:

[0099] Highly modular and end-to-end design: The complex 3D reconstruction task is clearly decomposed into modules such as "image diffusion" → "point cloud reconstruction" → "Gaussian reconstruction". This design not only makes the system structure clear, easy to understand and implement, but also achieves end-to-end generation from a single image to high-quality 3D assets, simplifying the traditional multi-step pipeline. Modularity facilitates phased training and debugging, enhancing the system's maintainability and scalability. The end-to-end framework reduces error accumulation in intermediate stages and promises to achieve better overall performance through joint optimization.

[0100] The training strategy is reasonable: a two-stage strategy is adopted to ensure the stability of the training process. The first stage provides a high-quality, consistent multi-view feature foundation for subsequent geometric reconstruction; the second stage focuses on learning geometric information from these reliable features, avoiding the problem of convergence when training multiple complex modules simultaneously.

[0101] Multiview Figure 1 Consistent Generation Mechanism: This mechanism innovates upon standard Stable Diffusion by guiding the diffusion process through feature stitching and Transformer cross-attention to generate highly consistent multi-view image features. It fundamentally solves the common "multifaceted face" problem in diffusion-based 3D generation, providing a reliable prerequisite for subsequent geometric reconstruction—a core foundation for achieving high-quality 3D reconstruction.

[0102] Point Cloud Reconstruction: The point cloud reconstruction module does not simply process a single image. Instead, it alternates between self-attention (mining the internal structure of a single view) and cross-attention (fusing geometric relationships between multiple views), supplemented by pixel-by-pixel fusion operations with multiple branches. This design can fully mine and utilize multi-view information, explicitly inferring the correspondence between different views, thereby more accurately recovering the 3D geometry of the object, which is superior to methods that rely solely on implicit inference.

[0103] Multi-objective loss function: The total loss function is not a simple photometric reconstruction loss, but a composite system that includes reconstruction, geometric alignment, multi-scale smoothing, and feature consistency. Constraints are imposed on the model from multiple dimensions, including color, geometric structure, smoothness, and cross-view consistency, ensuring high quality in both visual fidelity and geometric accuracy of the final 3D model.

[0104] An innovative multi-scale bidirectional consistent smoothing loss not only allows the rendering results to learn from the deep network prior, but also allows the prior to learn from high-quality rendering results when the prior confidence is low, forming a "co-evolutionary" process that enhances robustness. Calculating the loss at different scales of the pyramid simultaneously captures the consistency of both macroscopic structure and microscopic details. Dynamically adjusting based on confidence and rendering quality, it achieves intelligent smoothing that "believes what should be believed." It effectively suppresses noise while preserving details, significantly improving the visual quality of the reconstructed surface.

[0105] Example 2

[0106] This embodiment provides a method for single-view reconstruction and rendering, including:

[0107] This embodiment prioritizes the use of open-source 3D datasets, such as "Smithsonian 3D" with over 3 million 3D models. Then, for each 3D model, six standard viewpoint images are rendered (azimuth: ±30°, ±60°, ±90°, pitch: ±15°). Each image has a uniform resolution of 512×512 pixels, is saved in PNG format, and includes camera parameter files (intrinsic and extrinsic parameters), depth maps (pre-generated by laser scanning or MVSNet), and normal maps (calculated from the depth maps), thus constructing the most basic experimental dataset.

[0108] Open-source general-purpose datasets, such as the DTU dataset, contain 128 scenes, each with 49 or 69 images. Other examples include the BlendedMVS dataset and the Mip-NeRF360 dataset.

[0109] We obtained compliant, non-open-source datasets from domestic partner museums. Due to their small size, these datasets can be used as part of a test set to verify the model's performance.

[0110] The image data is cleaned, including removing blurry, severely occluded, and unevenly lit samples; Mask R-CNN is used to automatically segment the main body of the cultural relics and remove the background; histogram equalization is performed on the images to enhance texture contrast; and bilateral filtering is applied to the depth map to smooth noise and preserve edges.

[0111] Image data is enhanced by random horizontal flipping (within ±10°), random brightness and contrast adjustment (±15%), random addition of Gaussian noise (σ=0.01), and simulation of material changes such as aging and stains on the surface of cultural relics.

[0112] Hardware configuration: During training, eight NVIDIA A100 80GB servers are used, each equipped with a 64-core CPU and 512GB of RAM. A shared storage system using high-speed NVMe SSDs with a total capacity of 1.5PB is employed to store massive amounts of training data and model checkpoints. High-speed InfiniBand interconnect is used for data synchronization during multi-machine distributed training. During inference, two NVIDIA A100 40GB servers with standard CPU and content configurations are used. NVIDIA Triton Inference Server is used for model service to achieve high concurrency and low latency inference.

[0113] The model structure specifically includes:

[0114] The input consists of a single RGB image of an artifact and the corresponding camera parameters. The output includes a multi-view image (6 images, merged into one output), a point cloud map (512×512×3), Stokes model parameters (such as position, covariance, color, and transparency), a rendered image, a normal map, and a depth map.

[0115] Image diffusion module:

[0116] 1) The image diffusion module is based on the Stable Diffusion v1.5 open-source model. The input image is encoded into a 64x64x4 feature map via VAE.

[0117] 2) The diffusion steps are 50, and the noise regulator is DDPM;

[0118] 3) The Transformer has 12 layers and 16 attention heads;

[0119] 4) The feature fusion method is channel stitching + linear projection;

[0120] 5) The output of MLP-1 is 6 feature maps of 64x64x256.

[0121] Image output module:

[0122] 1) The input to the VAE decoder is a merge of 6 feature maps with a resolution of 384x64x256;

[0123] 2) The output image resolution of the image output module is 512x512x3;

[0124] 3) The viewing angle is set to ±30°, ±60°, ±90° for the azimuth angle and ±15° for the pitch angle;

[0125] 4) The output method is to stitch together 6 viewpoint images and multiply them into a single image of 1536x1024x3;

[0126] Point cloud reconstruction module:

[0127] 1) Number of Transformer groups: 3 groups, each with 8 layers (the first 3 layers are self-attention, the middle 2 layers are cross-attention, and the last 3 layers are self-attention).

[0128] 2) Number of attention heads: 16, Hidden layer dimensions: 512;

[0129] 3) Cross-attention method: Q of each feature map is attenuated by K and V of the other 5 maps;

[0130] 4) Blending method: Pixel-by-pixel addition;

[0131] 5) MLP-2 output: 512×512×3 point cloud map.

[0132] (5) Gaussian Reconstruction Module:

[0133] 1) The initial number of Gaussians is the number of points in the point cloud, which is dynamically increased to 100,000-500,000 during training;

[0134] 2) The covariance matrix is ​​initialized to the identity matrix;

[0135] 3) The color network uses a lightweight MLP, with the input being the viewing direction and the output being RGB;

[0136] 4) Rendering resolution: 512×512;

[0137] Training strategy:

[0138] (1) First stage: Pre-training the "image diffusion module" and "image output module" for 100 epochs, with a batch size of 4. The optimizer used is AdamW, the learning rate is set to 5e-5, and the loss function is... The weighting coefficient and ;

[0139] (2) Second stage: Freeze the "image diffusion module", discard the "image output module", train the point cloud reconstruction module, with a training epoch of 200 epochs, a batch size of 2, using AdamW as the optimizer, setting the learning rate to 1e-4, and using the loss function... ;

[0140] 1) The weighting coefficients are respectively , , , and .

[0141] 2) Hyperparameters in loss , which controls the strength of the impact of consistency error on confidence level; the larger the value, the more sensitive the impact of error.

[0142] 3) For Consistency error It can be done right Predict the disparity map, then calculate the depth map, and... Convert to a depth map, then input it. In the formula. This embodiment uses a simpler method, ,in It is a small constant to prevent division by zero, and can be taken as... .

[0143] 4) The total number of pyramids in The corresponding pyramid scales are the original image, 1 / 2, 1 / 4, and 1 / 8 downsampled. Scale weight decay factor. , , , Small constants that prevent division by zero , . Mid-Laplace operator Use a 3x3 discrete Laplace kernel.

[0144] 5) The pre-trained and frozen deep feature extraction network in the image is a pre-trained ResNet-50 (pre-trained using ImageNet, with frozen weights). The feature layers are taken from the Layer 3 output, and the feature map size is 1 / 16 of the input image. Learnable scale weights are used. The initial value is Updates are allowed during training.

[0145] (3) Third stage: end-to-end fine-tuning, training epochs of 50, batch size of 2, optimizer using AdamW, learning rate set to 5e-6, loss function using .

[0146] (4) After training, inference is performed using the "diffusion module" + "point cloud reconstruction module". On the one hand, this achieves an end-to-end framework, and on the other hand, it enhances the accuracy, controllability and interpretability of each module.

[0147] Experimental data:

[0148] (1) Evaluation metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) (in dB), used to measure the quality of the rendered image. Perceptual Similarity Index (LPIPS), used to measure visual quality. Geometric Error between Point Cloud and Real Scan (ChamferDistance()) in mm, and Geometric Recall (F-Score) at a threshold of 1 mm @ 1 mm.

[0149] (2) Comparison methods: Classical Neural Radiation Field (NeRF), Traditional Multi-View Reconstruction Process (MVSNet+Poisson Reconstruction), Original 3D Gaussian Splatting (3DGS), and this method.

[0150] The experimental results are shown in Table 1;

[0151] Table 1

[0152] method PSNR↑ SSIM↑ LPIPS↓ Chamfer↓ F-Score↑ NeRF 28.5 0.91 0.15 2.1 0.75 MVSNet+Poisson 26.8 0.88 0.18 1.8 0.80 3DGS 29.2 0.93 0.12 1.5 0.85 This method 31.6 0.96 0.08 0.9 0.92

[0153] Therefore, this invention significantly outperforms existing methods in multiple metrics, particularly in geometric accuracy (Chamfer Distance) and visual quality (PSNR, SSIM). The multi-stage training strategy effectively improves model convergence speed and stability. Modular design enhances interpretability and facilitates optimization for different artifact materials. The system possesses good generalization ability and is applicable to various artifact types, including bronze, ceramics, and jade.

[0154] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0155] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0156] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of monoscopic reconstruction and rendering, characterized in that, include: Take a single-view input image, and based on the single-view input image, generate a consistent feature representation across multiple perspectives by using an image diffusion module that integrates image diffusion priors and an attention mechanism; Based on the aforementioned multi-view consistent feature representation, a 3D point cloud is reconstructed using a point cloud reconstruction module that includes self-attention and cross-attention calculations. Based on the three-dimensional point cloud, a three-dimensional Gaussian representation is constructed, and a rendering output containing at least a new perspective color image is obtained through differentiable Gaussian rendering. The training of the image diffusion module and the point cloud reconstruction module adopts a phased training strategy and is optimized using a total loss function that includes reconstruction loss, distance alignment loss, normal alignment loss, multi-scale bidirectional consistency smoothing loss and feature consistency loss. The process of generating a consistent feature representation across multiple views based on the single-view input image, through an image diffusion module that integrates image diffusion prior and an attention mechanism, specifically includes: Encode the single-view input image into initial features; The initial features are fused with the noise features from the diffusion process and then fed into a transformer with a cross-attention mechanism for processing to enhance the multi-view consistency of the generated features. The processed features are decomposed into multiple feature maps corresponding to different preset viewpoints using a perceptron. Based on the aforementioned multi-view consistent feature representation, the reconstruction of a 3D point cloud, through a point cloud reconstruction module that includes self-attention and cross-attention calculations, specifically includes: The multiple feature maps corresponding to different preset viewpoints are respectively input into multiple parallel processing branches; In each branch, self-attention calculation based on the internal feature map of a single branch is performed alternately, as well as cross-attention calculation based on the query matrix of the current branch feature map and the key-value matrix of the other branch feature maps. The feature maps output by each branch after attention calculation are fused pixel by pixel, and a point cloud map representing the three-dimensional spatial coordinates is output by a decoding perceptron.

2. The method of claim 1, wherein, The construction of the three-dimensional Gaussian representation specifically includes: Using the positions of each point in the three-dimensional point cloud as initial centers, a series of three-dimensional Gaussian distributions are initialized. Each three-dimensional Gaussian distribution is parameterized by its center position, covariance matrix controlling its shape and orientation, opacity, and view-dependent color attributes.

3. The method of claim 1, wherein, The rendering output obtained by differentiable Gaussian rendering, which includes at least a new perspective color image, specifically includes: Sort according to the depth from the center of each Gaussian distribution in the three-dimensional Gaussian representation to the viewpoint of the target camera; Based on the sorting results, for each screen pixel, the color values ​​of the Gaussian distribution that contribute to it are accumulated through alpha mixing to calculate the new perspective color image; the weight of the alpha mixing is determined based on the opacity and cumulative transmittance of each Gaussian distribution.

4. The method according to claim 3, characterized in that, The rendering output also includes a normal map and a depth map; The normal map is calculated by transforming the normal vectors of each Gaussian distribution to the camera coordinate system and then using the same alpha blending mechanism as that used for rendering color images. The depth map is calculated by accumulating the distances of each Gaussian distribution to the camera plane according to the same alpha mixing mechanism.

5. The method of claim 1, wherein, The multi-scale bidirectional consistent smoothing loss is calculated in the following way: At multiple scales of the image pyramid, the forward consistency smoothing loss and the reverse consistency smoothing loss are calculated separately and then weighted and summed. The forward consistency smoothing loss is used to constrain the structure of the rendered normal map to be consistent with the structure of the prior normal map predicted by the pre-trained deep network. The reverse consistency smoothing loss is used to constrain the structure of the prior normal map to be consistent with the structure of the rendered normal map when the confidence of the prior normal map is low.

6. The method of claim 1, wherein, The feature consistency loss is calculated in the following way: Extract depth features from the reference image and the reconstructed image; At multiple scales, the similarity between the depth features of the reference image and the depth features of the reconstructed image is calculated and weighted according to a confidence mask. The deep features are obtained from a pre-trained and frozen deep feature extraction network.

7. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images