Self-shadow image reconstruction and rendering method and system

Abstract

The invention discloses a method and a system for reconstructing and rendering a self-shadow image. The method comprises the following steps: S1, acquiring a single input image of a target object containing a self-shadow; s2, based on the input image, generating a plurality of generated images of the target object at different visual angles through an image generation module; s3, acquiring camera parameters and three-dimensional geometric representation of the target object through a three-dimensional reconstruction module based on the generated images of the multiple different visual angles; and S4, rendering and outputting a reconstruction result of the target object according to the three-dimensional geometric representation and the camera parameters. The method effectively overcomes the defects that an existing three-dimensional reconstruction technology depends on multi-view-angle images or accurate camera parameters, and reconstruction distortion is caused due to the fact that strong self-shadow is difficult to process.

Description

Technical Field

[0001] This invention relates to the field of image rendering, and more specifically to a method and system for reconstructing and rendering self-shadowed images. Background Technology

[0002] In the real world, the surfaces of complex objects (such as wrinkled clothing, precision mechanical parts, sculptures, etc.) produce strong self-shadows due to the angle of illumination. While these shadows contain rich geometric clues, traditional 3D reconstruction methods often treat them as noise or interference. Furthermore, in many practical applications, such as e-commerce and virtual try-on (generating 3D models directly from product images for AR / VR try-on or display), cultural heritage digitization (3D reconstruction of single historical artifact photographs in museums), film and game production (rapidly converting concept art or live-action footage into basic 3D assets), and industrial inspection and reverse engineering (rapidly generating 3D data from single-angle photographs of faulty parts or samples for analysis), obtaining accurate camera pose for images is extremely difficult, if not impossible. Therefore, solving the "single image, strong shadows, no pose" triangle problem is crucial for overcoming the application bottlenecks of 3D vision in real-world scenarios and promoting its wider adoption and automation.

[0003] Current mainstream 3D reconstruction methods have significant shortcomings when facing the above challenges:

[0004] (1) Strong dependence on camera pose: Most learning-based methods (such as NeRF and multi-view solid geometry) require precise camera pose as input in advance. This prerequisite severely limits their application in arbitrary network images or scenarios lacking calibration information, resulting in poor scalability.

[0005] (2) Vulnerability to self-shadows: Traditional photometric consistency assumptions or shadow processing algorithms have difficulty distinguishing between texture changes and shadow boundaries, and are very likely to incorrectly reconstruct self-shadows as geometric depressions or protrusions on the object surface, resulting in severe artifacts and distortions in the reconstructed model, especially for complex objects.

[0006] (3) Limitations of data-driven approaches: Models that rely solely on large datasets for training will experience a sharp decline in generalization ability when faced with extreme lighting or complex geometry outside the training set distribution, and will be unable to reliably handle high-intensity self-shadows in a single image.

[0007] To overcome the limitations of existing methods, we propose a novel, end-to-end deep learning framework. Its core idea is to "understand" and "decouple" geometry and shadows from a single image through a generative model and a powerful attention mechanism, thereby achieving high-quality reconstruction without camera pose requirements. Key advantages include:

[0008] (1) No camera pose required: It completely eliminates the dependence on pre-computed camera pose, greatly improving the applicability and scalability of the method, enabling it to process single images from any source.

[0009] (2) Strong robustness to self-shadow: By generating multi-view images and cross-attention mechanism, it can effectively distinguish shadows from real geometry, significantly improving the reconstruction quality and integrity under complex lighting conditions.

[0010] (3) High precision and high consistency: The generated multi-view images have high consistency, ensuring that the final reconstruction results are geometrically reasonable and accurate from all angles.

[0011] (4) End-to-end efficient processing: The entire process is integrated into one framework, and a 3D model can be directly output by inputting a single image, making the process simple and efficient. Summary of the Invention

[0012] To address the technical problems mentioned above, this invention provides a method for reconstructing and rendering self-shadowed images, comprising the following steps:

[0013] S1. Obtain a single input image containing the target object with self-shadowing;

[0014] S2. Based on the input image, generate several generated images of the target object from different perspectives through the image generation module;

[0015] S3. Based on images generated from several different perspectives, the camera parameters and three-dimensional geometric representation of the target object are obtained through the three-dimensional reconstruction module;

[0016] S4. Based on the three-dimensional geometric representation and the camera parameters, render and output the reconstruction result of the target object.

[0017] Preferably, step S2 includes:

[0018] The input image is encoded to obtain conditional features;

[0019] Random noise and conditional features are fused and processed by a diffusion transformer to generate image features from several different perspectives.

[0020] Image features from different perspectives are decoded to obtain several corresponding generated images from different perspectives.

[0021] Preferably, step S3 includes:

[0022] The generated images from different perspectives are segmented into image patches, and feature words are extracted from each image patch;

[0023] Add corresponding camera terms to the feature terms for each viewpoint to form the input term sequence for each viewpoint;

[0024] The input word sequence for each viewpoint is fed into a transformer network containing alternating local attention layers and global attention layers for processing, resulting in output features that include geometric and camera information.

[0025] Camera parameters and 3D geometric representations are extracted from the output features.

[0026] Preferably, the 3D reconstruction module is trained using a knowledge distillation method, including:

[0027] Obtain teacher and student models with identical structures;

[0028] The teacher model was trained using images without self-shading, and the parameters were frozen.

[0029] The student model is trained using an image containing self-shading, and supervised using the output of the teacher model.

[0030] Preferably, the loss function used when training the student model includes reconstruction loss, distillation loss, and consistency loss;

[0031] The reconstruction loss is used to constrain the consistency of pixel color and depth between the rendered image and the real image.

[0032] The distillation loss is used to constrain the consistency of the student model and the teacher model in geometric prediction;

[0033] The consistency loss is used to constrain the consistency between the student model and the reference information in terms of semantics, two-dimensional image, and three-dimensional spatial feature distribution.

[0034] Preferably, the calculation method of the local attention layer in the transformer network is as follows: the input word sequence performs self-attention calculation within the image at the same viewpoint;

[0035] The global attention layer in the transformer network is calculated as follows: the input word sequence is used to perform cross-attention calculation with the query vector of the current perspective and the key vectors and value vectors of other perspectives.

[0036] Preferably, the three-dimensional geometric representation includes: a depth map and a point cloud map; the reconstruction result is rendered using a point cloud-based three-dimensional Gaussian sputtering technique.

[0037] The present invention also provides a system for reconstructing and rendering self-shadow images, the system being used to implement the above method, comprising: an acquisition module, a generation module, an input module, and an output module;

[0038] The acquisition module is used to acquire a single input image of a target object containing self-shadowing;

[0039] The generation module is used to generate several images of the target object from different perspectives based on the input image through the image generation module;

[0040] The input module is used to generate images based on several different viewpoints, and the camera parameters and three-dimensional geometric representation of the target object are obtained through the three-dimensional reconstruction module.

[0041] The output module is used to render and output the reconstruction result of the target object based on the three-dimensional geometric representation and the camera parameters.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] (1) No camera pose required: It completely eliminates the dependence on pre-computed camera pose, greatly improving the applicability and scalability of the method, enabling it to process single images from any source.

[0044] (2) Strong robustness to self-shadow: By generating multi-view images and cross-attention mechanism, it can effectively distinguish shadows from real geometry, significantly improving the reconstruction quality and integrity under complex lighting conditions.

[0045] (3) High precision and high consistency: The generated multi-view images have high consistency, ensuring that the final reconstruction results are geometrically reasonable and accurate from all angles.

[0046] (4) End-to-end efficient processing: The entire process is integrated into one framework, and a 3D model can be directly output by inputting a single image, making the process simple and efficient. Attached Figure Description

[0047] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the model structure of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] This embodiment provides a method for reconstructing and rendering self-shadowed images, the steps of which include:

[0053] S1. Obtain a single input image containing the target object with self-shadowing.

[0054] Input an image with high-intensity self-shadowing.

[0055] S2. Based on the input image, generate several generated images of the target object from different perspectives through the image generation module.

[0056] This embodiment uses a DiT-based image generation module. First, it encodes the image using a Variational Autoencoder (VAE) to reduce the feature dimension input to the model. After passing through a standard Diffusion Transformer (DiT) module and then decoding via the VAE, 12 images from different viewpoints are obtained. The azimuth angles of these 12 images are ±30°, ±60°, and ±90°, while the pitch angles are superimposed at ±30°.

[0057] There are two differences from standard DiT: first, the conditional image uses the input image; second, after VAE encoding, the conditional image undergoes a self-attention generation process. Then, with the self-attention generation of noise To be integrated, the integration method is as follows: and The two matrices are concatenated. and Two matrices concatenated, and Adding elements together helps to ensure geometric consistency among the 12 viewpoint images generated.

[0058] To facilitate network generation, these 12 views are stitched together into a single output image.

[0059] S3. Generate images from multiple different perspectives and obtain the camera parameters and three-dimensional geometric representation of the target object through the three-dimensional reconstruction module.

[0060] This embodiment uses a Transformer-based 3D reconstruction module, the specific structure of which includes:

[0061] Image segmentation: An image generated by the "DiT-based image generation module" is segmented into 12 viewpoint images. It means that among them Then The image is further segmented into patches, which are then input into the pre-trained visual model DINOv2, and the output is the corresponding image words. DINOv2 was used here instead of randomly initialized convolutional layers because DINOv2 provides higher quality and better performance terms, making the training process more stable and less sensitive to hyperparameters (such as the learning rate). Furthermore, from an input perspective, DINOv2 takes 12 images as input, and then combines these 12 inputs into the "Transformer-based 3D reconstruction module" to compute global attention.

[0062] Add camera terms: for each image Image lexicons Add a separate camera terminology Both of these lexical units are learnable, but... The initial value is the encoded output of DINOv2. The initial value is a random value.

[0063] Backbone Network: To address the issue that the standard DiT model calculates self-attention across all words, which can lead to difficulties in simultaneously maintaining local consistency within a single image and global correlation between multiple images when there are many input images, a redesign was implemented.

[0064] The entire network consists of N modules (e.g., N=4), each module is composed of 12 attention points, and local attention and global attention are alternately superimposed.

[0065] Local attention is Self-attention computation is performed on other words within the same image. The goal is to allow the model to focus on understanding the content and structure of each image itself, similar to a powerful single-image feature extractor. This also helps to stabilize activation values.

[0066] Global attention is Cross-attention calculation with lexical units of different images aims to establish connections between different images, match corresponding points, and integrate multi-view information, which is the key to 3D reconstruction.

[0067] Therefore, each image One instance of self-attention (i.e., local attention) and 11 instances of cross-attention (i.e., global attention) were performed. During cross-attention, the Q-value of the current image was calculated by comparing it with the KV-values ​​of other images.

[0068] To enable gradients to propagate directly backward along the identity mapping path, ensuring their stability and effectiveness in deep networks, skip connections are added, and local attention connections and global attention connections are maintained.

[0069] S4. Based on the 3D geometric representation and camera parameters, render and output the reconstruction result of the target object.

[0070] The "Transformer-based 3D reconstruction module" incorporates learnable camera terms. During output, these terms are extracted and further fused and refined through a three-layer self-attention layer. Finally, a linear layer regresses the camera parameter vector, including rotation quaternions, 3D translation vectors, and 2D field of view. It's important to note that each... Each has a corresponding camera terminology Therefore, a total of will be generated The camera parameters for the 9D group.

[0071] The generation of depth maps (i.e., each pixel corresponds to a depth value) and point cloud maps (i.e., each pixel corresponds to a 3D point) is similar. Both involve passing the output feature map through a 3x3 convolutional layer to directly predict the depth map and point cloud map.

[0072] Rendering output: The generated point cloud map is used to reconstruct the point cloud and is used as input to a standard 3D Gaussian Splatting model. The output is a 3D Gaussian distribution, and then the corresponding reconstructed image is rendered according to the generated camera parameters.

[0073] Knowledge distillation: To improve learning efficiency and generation quality, a teacher-student model is used for knowledge distillation.

[0074] The teacher model and the student model have the same network structure, but their inputs are different. The teacher model inputs an image without self-shading, while the student model inputs an image with high self-shading.

[0075] The teacher model needs to be trained first, and then the student model is guided by the teacher model with frozen parameters. This can improve the quality of the generated model and result in a model with smaller network parameters that can be reconstructed and rendered in real time, which is beneficial for deployment on devices with low computing power.

[0076] Only the student model will be rendered; the teacher model will not be rendered. The model structure in this embodiment is as follows: Figure 1 As shown. The loss function of this model is as follows:

[0077]

[0078] in, , and These are weighting coefficients. It is a reconstruction loss. It is a distillation loss. It is a loss of consistency.

[0079] Reconstruction loss: Reconstruction loss is the most basic loss function and the cornerstone of subsequent loss functions. It ensures that the image rendered from the predicted 3D Gaussian representation is as close as possible to the true value in both color and geometry.

[0080]

[0081] in, , and These are weighting coefficients; It is the loss of photometric reconstruction; It is the geometric reconstruction loss; It is structural similarity loss.

[0082] Photometric reconstruction loss: calculated by reconstructing the image Compared with reference image The mean absolute error in pixel color values ​​directly constrains the accuracy of color and brightness in the output image.

[0083]

[0084] in, It is a reconstructed image, that is, an RGB image rendered by the 3D Gaussian distribution generated in this embodiment under the same camera parameters (which are predicted and output by the network of this method) as the reference image (i.e., the image without self-shading). It is a reference image, also known as a real image, which is an image without self-shadowing. yes The total number of pixels is used for normalization. This represents the L1 norm.

[0085] Geometric reconstruction loss: calculated by rendering the depth map Compared to the actual depth map The mean absolute error between the two values ​​provides direct supervision of the position and geometry of the 3D Gaussian points, ensuring that the reconstructed 3D structure is accurate.

[0086]

[0087] in, Represents the reconstructed image Depth map, Represents reference image A true depth map. This represents the total number of valid depth points, typically less than [a certain value]. . This represents the L1 norm.

[0088] Structural similarity loss: By maximizing the similarity of two images in terms of structure, brightness, and contrast (i.e., maximizing SSIM and minimizing the structural similarity loss function), it helps to preserve the structural information of the image, producing visually more natural and edge-sharper results, and is a perfect complement to L1 loss.

[0089]

[0090] in, It is to reconstruct the image. The reference image is SSIM, which is the structural similarity formula.

[0091] Distillation Loss: A teacher model with frozen parameters is used for supervision under normal lighting conditions (i.e., no self-shading) to provide geometric guidance to the student model. Geometric knowledge from the teacher network is transferred to the student network by minimizing the difference between the geometric representations (such as depth maps and 3D point maps) predicted by the student and teacher networks under the same input. This helps the student network learn more accurate and robust geometric information under low-light conditions.

[0092]

[0093] in, and These are hyperparameters. This is the L1 distillation loss. It is an adaptive weighted distillation loss, It is a multi-scale geometric alignment loss. It is a comparative loss.

[0094] L1 distillation loss:

[0095]

[0096] in, This indicates the batch size, which is the number of samples in one training iteration. This indicates the number of views. H and W represent the height and width of the image, respectively. Let i represent the set of two-dimensional positions across all batches and all viewpoints, where i represents the position in the nth viewpoint image within the b-th batch. The pixel index. The superscript Stu represents the student model, and the superscript Tea represents the teacher model. This represents the 3D point coordinate vector predicted by the student model at point i, i.e. . This represents the 3D point coordinate vector predicted by the teacher model at point i, i.e. . This represents the depth information predicted by the student model at point i. This shows the depth information predicted by the teacher model at point i. This represents the L1 norm.

[0097] Adaptive weighting mechanism: Different regions have different geometric importance (such as edges and high curvature regions). These regions should be given higher alignment weights to improve alignment quality. This is suitable for use cases with discontinuous depth and high dynamic range.

[0098]

[0099] Among them, weight Generated from the deep gradient or confidence graph of the teacher model, i.e. , here It is the gradient intensity of the depth map. α represents the confidence level of the teacher model's prediction, and β are hyperparameters.

[0100] Multi-scale geometric alignment loss: L1 loss that only uses point coordinates may not be sensitive to local geometric details. Therefore, multi-scale feature alignment enhances geometric consistency and is suitable for use cases with complex structures and rich edges.

[0101]

[0102] Where L represents the total number of layers involved in loss calculation in the feature extractor of the VGG (Visual Geometry Group Network) model. It is the l-th layer VGG feature extractor. and These are the height, width, and number of channels of the l-th layer feature map, respectively. The input to the VGG model is the output of the student model. and Feature maps, and the output of the teacher model and Feature map.

[0103] Contrastive loss: In addition to aligning positive samples (corresponding points), it can also widen the gap between incorrectly matched point pairs through contrastive learning, thereby enhancing the discriminative power of geometric representation. It is suitable for use cases with many occlusions and weak textures.

[0104]

[0105] Where sim is the cosine similarity. It is the temperature coefficient. and These are negative sample sets of points and depths sampled from different locations or different images. and This represents one of the negative samples.

[0106] Consistency loss:

[0107]

[0108] in, and All of these are weight parameters. It is a semantic consistency loss. It is a 2D reconstruction consistency loss. It is a loss of consistency in 3D reconstruction.

[0109] Semantic consistency loss By comparing reconstructed images (This refers to the RGB image rendered using the same camera parameters as the reference image (i.e., the image without self-shadowing) and the reference image (i.e., the image without self-shadowing) in terms of deep features of the pre-trained VGG network. This ensures that the model improves brightness without changing the semantic content of the scene and maintains structural integrity. This is a perceptual loss, insensitive to changes in lighting but sensitive to changes in object structure and shape.

[0110]

[0111] Here, i and j represent the layer indices of the VGG network, which uses multiple intermediate layers of VGG to capture multi-scale information from local details to global semantics. i The weight coefficient of the i-th layer is a learnable scalar parameter that is normalized by the softmax function, enabling the model to dynamically focus on the semantic level that is most helpful to the current recovery task and improve generalization ability. It is a normalization operation that makes the sum of all weight parameters equal to 1. It is the set of selected VGG layers. H i and W i The height and width of the i-th layer feature map are used to normalize the loss so that it is independent of the feature map size. p represents the spatial index on the feature map, used to traverse each pixel on the feature map. Indicates in reconstructing the image The feature activation vector of the i-th layer VGG model at position p. In the reference image The feature activation vector of the i-th layer VGG model at position p. This represents the L1 norm. It is a channel attention vector to address the importance of different channels in the feature map, allowing the loss function to more intelligently focus on feature channels that are crucial to semantic content, while suppressing unimportant or noisy channels, thereby guiding image reconstruction more accurately. i This represents the number of channels in the feature map of the i-th layer. By analyzing the feature map in spatial dimension The average of its L2 norm is calculated and normalized to obtain the value, i.e. MLP stands for Multilayer Perceptron, and a reference image is used here. Features To calculate attention, we need to obtain a stable importance criterion that is independent of the recovery outcome.

[0112] 2D Reconstruction Consistency Loss Used to measure the absolute difference between the reconstructed image and the reference image at the pixel level. It is fundamental to ensuring image color fidelity and overall brightness accuracy. Through the most direct comparison of pixel values, it provides the model with a strong and stable low-level visual signal, ensuring that the reconstructed image remains consistent with the real normal-light image in terms of global brightness, color, and contrast. It is key to achieving high PSNR (Peak Signal-to-Noise Ratio).

[0113]

[0114] in, It is a normalization factor that sums the differences of all scenes, all views, all pixels, and all channels in the entire batch, and then divides it by the total number of elements to obtain the average pixel-level error. Indicates batch size, Indicates the number of multiple views. C and C represent the image height, width, and number of channels, respectively. Let i represent the set of two-dimensional positions across all batches and all viewpoints, where i represents the position in the nth viewpoint image within the b-th batch. The pixel index of channel c. This represents the reconstructed image of the nth view image in the bth batch, at position... The pixel intensity value on channel c. This represents the reference image for the nth view image in the bth batch, at position... The pixel intensity value on channel c. It is an absolute value operation. It employs adaptive weighting based on exposure levels. This addresses the fact that the reconstruction difficulty and importance differ between self-shadowed and non-self-shadowed regions. Self-shadowed regions have high pixel value uncertainty, and direct, strong constraints may amplify noise or produce color casts. For normally exposed areas, higher weights are assigned to ensure accuracy; for excessively dark or bright areas, weights are appropriately reduced, relying more on advanced semantic losses (such as content loss) to guide the recovery. , here and These are hyperparameters, and can be selected. , This creates a bell-shaped curve with a peak at a medium gray level. In this way, the medium brightness region has the highest weight, while the extremely dark and extremely bright regions have lower weights. This strategy allows the model to focus on recovering medium brightness regions with reliable visual information, avoiding unreliable "hard fitting" in regions with extremely weak (extremely dark) or saturated (extremely bright) signals, thereby improving overall visual quality and suppressing noise.

[0115] 3D Reconstruction Consistency Loss In 3D voxel space, the feature distributions of the recovery branch and the teacher distillation branch are forced to remain consistent. This ensures that the reconstructed 3D scene is geometrically consistent and lighting-invariant, better preserving the recovery quality of complex details and anomalous geometric structures in the 3D scene, making the reconstruction result statistically closer to the real scene. The specific operation process is as follows: First, the 2D image features from multiple viewpoints are back-projected into 3D space based on the estimated depth and camera pose, and assigned to 3D voxels. Second, within each voxel grid, the features of all points falling into that voxel are calculated to obtain the average feature of each voxel. Then, the mean μ and standard deviation of the features are calculated across all voxels in the entire 3D scene. And skewness γ. Finally, by comparing the mean, standard deviation, and skewness of the student branch and the teacher branch, the 3D scene of the reconstructed image mapping is forced to be consistent with the 3D scene of the reference image mapping generated by the teacher model in terms of feature distribution, thereby ensuring similarity in average appearance, detail richness, and diversity.

[0116]

[0117] in, The number of feature scales is represented by 'i', which represents the feature scale index and corresponds to different layers in the feature extraction network (such as FPN), thereby capturing multi-scale geometric information from fine to coarse. These represent the weight coefficients for the i-th feature scale. These weight coefficients are normalized, i.e., they satisfy... This is used to balance the contribution of features at different scales to the total loss. Let represent the mean vector of all voxelized features of the student model at the i-th feature scale, which describes the average appearance of the 3D scene mapped from the reconstructed image in the feature space. This represents the mean vector of all voxelized features of the teacher model at the i-th feature scale, representing the feature distribution center that a geometrically correct 3D scene should have under normal lighting (i.e., no self-shadow). It represents the standard deviation vector of all voxelized features of the student model at the i-th feature scale, describing the degree of feature variation in the 3D scene mapped by the reconstructed image, reflecting the richness of surface texture, geometric details, etc. This represents the standard deviation vector of all voxelized features of the teacher model at the i-th feature scale, representing the feature diversity that an ideal 3D scene should have. This represents the skewness vector of all voxelized features of the student model at the i-th feature scale. This represents the skewness vector of all voxelized features of the teacher model at the i-th feature scale. The Wasserstein distance, as it's called, provides a better measure of the overall difference between two distributions. A schematic diagram of the model structure in this embodiment is shown below. Figure 1 As shown.

[0118] Example 2

[0119] The following will, in conjunction with this embodiment, explain in detail how the present invention solves the technical problems in practical work. In the process of digitizing cultural heritage, many cultural relics, due to limitations in preservation conditions, can only be recorded through a single two-dimensional photograph. These photographs often suffer from strong self-shadowing due to unfavorable lighting conditions, and the camera pose information at the time of shooting is missing. Traditional three-dimensional reconstruction methods cannot recover an accurate and complete three-dimensional model of the cultural relic from a single image with strong shadows, resulting in distorted digitization results and loss of detail, making it difficult to meet the needs of high-precision archiving, virtual display, and academic research.

[0120] This invention proposes an end-to-end deep learning framework that starts with a single image of a cultural relic with strong self-shadowing. Through a generative model and attention mechanism, it decouples shadow and geometric information, achieving high-quality 3D reconstruction without requiring camera pose. The system comprises two core modules: a DiT-based image generation module and a Transformer-based 3D reconstruction module. The former generates 12 shadow-free images from a single input image from different perspectives; the latter reconstructs the depth map, point cloud, and camera pose of the cultural relic from the multi-view images, and finally renders a 3D model using 3D Gaussian Splatting.

[0121] 1. Dataset.

[0122] Open source dataset: Images of the “Sculpture” and “Pottery” categories from the publicly available cultural heritage dataset CO3D-v2.

[0123] Self-built dataset: In cooperation with museums, research institutes and other institutions, we obtained authorization to use high-definition digital cameras to take multi-view images of museum artifacts under fixed lighting conditions. Each artifact was photographed from 36 perspectives (one image every 10° azimuth angle and 0° elevation angle), and more than 200 artifacts were collected in total. The resolution of each image is 9504x6336 (Sony Alpha 7R IV).

[0124] Data cleaning: Remove blurry, overexposed, and underexposed images; adjust image resolution through scaling and cropping; and manually annotate shadow areas using Labelme image annotation tool to generate shadow masks.

[0125] Data augmentation: Brightness (±30%) and contrast (±20%) were randomly adjusted, and Gaussian noise (σ=0.01) was added. Composite shadows were randomly generated using the 3D rendering software Blender to simulate self-shadowing.

[0126] Training Hardware: The training phase utilizes eight NVIDIA A100 80GB servers, each equipped with a 64-core CPU and 512GB of RAM. A shared storage system using high-speed NVMe SSDs, totaling 1.5PB, is employed to store massive amounts of training data and model checkpoints. High-speed InfiniBand interconnects are used for data synchronization during multi-machine distributed training. The inference phase uses two NVIDIA A100 40GB servers with standard CPU and content configurations. NVIDIA Triton Inference Server is used for model service, enabling high concurrency and low latency inference.

[0127] 2. Model parameters.

[0128] Input a single RGB image with strong self-shading, resolution 512x512; output 12 multi-view images (512x512), depth map (512x512), point cloud map (512x512x3) and camera parameters (rotation quaternion, 3D translation vector and 2D field of view).

[0129] The DiT-based image generation module uses a VAE encoder to encode the input image into latent features (256x16x16 dimensions). The DiT model has 12 layers, 16 attention heads, and a latent dimension of 768. The output of this module is a 3072×1024 large image composed of 12 viewpoint images.

[0130] Transformer-based 3D reconstruction module:

[0131] Lexicalization: Image patch features were extracted using DINOv2 (ViT-B / 14) with a lexical dimension of 768.

[0132] Camera lexicon: randomly initialized, dimension 768.

[0133] Attention alternation mechanism: local attention (within a single image) + global attention (between multiple images), with 12 attention heads per module, alternating 4 times (N=4).

[0134] Output Header: Linear layer outputs a 9-dimensional vector of camera parameters (quaternion 4 + translation 3 + field of view 2), 3×3 convolution outputs a single-channel depth map, and 3×3 convolution outputs a 3-channel point cloud map.

[0135] Rendering module: Uses 3D Gaussian Splatting as the renderer, 500,000 Gaussian points, 30,000 iterations.

[0136] 3. Training strategies.

[0137] Knowledge distillation settings:

[0138] Teacher model: Input is a shadowless image, trained until convergence (PSNR > 32 dB).

[0139] Student model: The input is a strongly shaded image, and the teacher model is used to freeze parameters for supervision.

[0140] Distillation loss weighting: and .

[0141] Training process:

[0142] Pre-trained DIT module: Uses the CO3D dataset, trained for 100k steps, batch size=8;

[0143] Jointly trained 3D reconstruction module: fixed DiT parameters, 50k training steps, batch size=4;

[0144] Distillation training: Load the teacher model and train the student model for 30k steps, batch size=4;

[0145] Fine-tuning the rendering module: Use Gaussian Splatting to fine-tune 10k steps.

[0146] Hyperparameters:

[0147] The AdamW optimizer is used with a learning rate of 1e-4 to control the step size of each parameter update. Weight decay is set to 1e-5 to prevent excessive weights and improve generalization ability.

[0148] Dynamically adjust the learning rate using cosine annealing, set... That is, a complete cosine period requires 50,000 iterations.

[0149] Loss weights: and and and .

[0150] 4. Indicator evaluation.

[0151] Geometric accuracy:

[0152] CD (Chamfer Distance): The average distance between the point cloud and the ground truth (unit: mm).

[0153] F-Score@5mm: Point cloud matching rate (threshold 5mm).

[0154] Visual quality:

[0155] PSNR (Peak Signal-to-Noise Ratio): The brightness fidelity between the reconstructed image and the ground truth image;

[0156] SSIM (Structural Similarity): Image structural integrity;

[0157] LPIPS (Perceptual Similarity): Visual Perceptual Consistency.

[0158] Camera attitude accuracy:

[0159] Rotation error: The angular difference (in degrees) between the predicted quaternion and the true value.

[0160] Translation error: Euclidean distance (unit: cm).

[0161] 5. Experimental results.

[0162] (1) Comparison method:

[0163] 1) NeRF + known pose, this is the traditional multi-view method;

[0164] 2) ShadowNeRF, which is a NeRF variant for self-shadowing;

[0165] 3) Single-Image Depth (MiDaS) + ICP registration.

[0166] The experimental results are shown in Table 1.

[0167] Table 1

[0168] .

[0169] While the data available in this invention is somewhat limited compared to traditional multi-view methods (NeRF + known pose), traditional methods cannot handle self-shadowing issues. Compared to the professional shadow processing model ShadowNeRF, the method in this embodiment shows significant improvements in all metrics.

[0170] The results of the ablation experiment are shown in Table 2.

[0171] Table 2

[0172] .

[0173] (4) Conclusion:

[0174] 1) No camera pose required: Achieve high-precision 3D reconstruction with completely unknown shooting parameters.

[0175] 2) Strong shadow robustness: It can effectively distinguish shadows from geometric structures and avoid artifacts.

[0176] 3) High-fidelity rendering: A multi-view method that approximates known poses in terms of metrics such as PSNR and SSIM.

[0177] 4) Deployability: Through knowledge distillation, the number of student model parameters is reduced by 40%, and it can run in real time on mobile devices.

[0178] Example 3

[0179] This embodiment also provides a self-shadow image reconstruction and rendering system, including: an acquisition module, a generation module, an input module, and an output module; the acquisition module is used to acquire a single input image of a target object containing self-shadow; the generation module is used to generate several generated images of the target object from different perspectives based on the input image through an image generation module; the input module is used to obtain the camera parameters and three-dimensional geometric representation of the target object based on the generated images from multiple different perspectives through a three-dimensional reconstruction module; the output module is used to render and output the reconstruction result of the target object according to the three-dimensional geometric representation and camera parameters.

[0180] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for reconstructing and rendering self-shadowed images, characterized by the following steps: include: S1. Obtain a single input image containing the target object with self-shadowing; S2. Based on the input image, generate several generated images of the target object from different perspectives through the image generation module; S3. Based on images generated from several different perspectives, the camera parameters and three-dimensional geometric representation of the target object are obtained through the three-dimensional reconstruction module; S4. Based on the three-dimensional geometric representation and the camera parameters, render and output the reconstruction result of the target object.

2. The method for reconstructing and rendering self-shadowed images according to claim 1, characterized in that, Step S2 includes: The input image is encoded to obtain conditional features; Random noise and conditional features are fused and processed by a diffusion transformer to generate image features from several different perspectives. Image features from different perspectives are decoded to obtain several corresponding generated images from different perspectives.

3. The method for reconstructing and rendering self-shadowed images according to claim 2, characterized in that, Step S3 includes: The generated images from different perspectives are segmented into image patches, and feature words are extracted from each image patch; Add corresponding camera terms to the feature terms for each viewpoint to form the input term sequence for each viewpoint; The input word sequence for each viewpoint is fed into a transformer network containing alternating local attention layers and global attention layers for processing, resulting in output features that include geometric and camera information. Camera parameters and 3D geometric representations are extracted from the output features.

4. The method for reconstructing and rendering self-shadowed images according to claim 3, characterized in that, The 3D reconstruction module is trained using knowledge distillation, including: Obtain teacher and student models with identical structures; The teacher model was trained using images without self-shading, and the parameters were frozen. The student model is trained using an image containing self-shading, and supervised using the output of the teacher model.

5. The method for reconstructing and rendering self-shadowed images according to claim 4, characterized in that, The loss functions used when training the student model include reconstruction loss, distillation loss, and consistency loss; The reconstruction loss is used to constrain the consistency of pixel color and depth between the rendered image and the real image. The distillation loss is used to constrain the consistency of the student model and the teacher model in geometric prediction; The consistency loss is used to constrain the consistency between the student model and the reference information in terms of semantics, two-dimensional image, and three-dimensional spatial feature distribution.

6. The method for reconstructing and rendering self-shadowed images according to claim 3, characterized in that, The local attention layer in the transformer network is calculated as follows: the input word sequence performs self-attention calculation within the image at the same viewpoint; The global attention layer in the transformer network is calculated as follows: the input word sequence is used to perform cross-attention calculation with the query vector of the current perspective and the key vectors and value vectors of other perspectives.

7. The method for reconstructing and rendering self-shadowed images according to claim 1, characterized in that, The three-dimensional geometric representation includes a depth map and a point cloud map; the reconstruction result is rendered using a point cloud-based three-dimensional Gaussian sputtering technique.

8. A system for reconstructing and rendering self-shadowed images, said system being used to implement the method according to any one of claims 1-7, characterized in that, include: Acquisition module, generation module, input module, and output module; The acquisition module is used to acquire a single input image of a target object containing self-shadowing; The generation module is used to generate several images of the target object from different perspectives based on the input image through the image generation module; The input module is used to generate images based on several different viewpoints, and the camera parameters and three-dimensional geometric representation of the target object are obtained through the three-dimensional reconstruction module. The output module is used to render and output the reconstruction result of the target object based on the three-dimensional geometric representation and the camera parameters.

Images