A method for 3D reconstruction of multi-view images

Abstract

This invention belongs to the field of computer vision technology, specifically relating to a multi-view image 3D reconstruction method, comprising the following steps: First, a small number of 2D images are input, and a colored point cloud is generated through depth estimation and a 3D Gaussian model is initialized. Coarse-grained reconstruction is then performed by combining photometric loss and geometric regularization. Second, camera trajectories are planned based on the geometric distribution of the coarse model, and multi-view image sequences are acquired. Next, a diffusion model fusing camera pose and visual features is used to fine-tune and repair the images. Finally, the repaired images are used to iteratively optimize the 3D Gaussian model, outputting high-precision 3D reconstruction results. This method requires only sparse view images to achieve high-quality reconstruction, breaking through the dependence of traditional methods on large amounts of data. Through composite conditional coding and dynamic trajectory optimization, geometric consistency and texture realism are effectively improved, artifacts are avoided, and computational efficiency and reconstruction accuracy are balanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a multi-view image three-dimensional reconstruction method. BACKGROUND

[0002] With the development of computer vision and computer graphics, three-dimensional reconstruction technology shows wide application prospects in virtual reality, augmented reality, film production, industrial detection, cultural heritage protection and other fields. As one of the core technologies, multi-view image three-dimensional reconstruction aims to obtain three-dimensional geometric information of a scene or an object through multiple two-dimensional pictures, and then restore the real structure.

[0003] From the perspective of data dependence, the prior art generally has strong dependence on the number of input images, and usually dozens of view images are needed to ensure the reconstruction accuracy. If only a small number of images less than 4 are used, the reconstructed model structure is rough, details are missing, the fine geometric features of the target cannot be restored, and it is difficult to balance the actual needs of less data input and high precision output. In terms of missing area processing, the inevitable occlusion, light change and view limitation in complex scenes will cause local missing of the reconstructed data, and the traditional algorithm usually relies on simple regularization assumption or interpolation method for missing completion, which cannot fully fuse the texture features and geometric logic of the target, and the completion result often has artifacts or structure rupture, which deviates greatly from the real shape.

[0004] For camera pose error, the existing reconstruction process lacks an effective adaptive adjustment mechanism. Small pose deviations (such as position deviation and angle tilt) of the camera during shooting will directly affect the reconstruction accuracy, and the error cannot be corrected in real time, which limits the subsequent model optimization to the initial data defects and greatly limits the detail expression ability. In terms of computational efficiency, the existing method usually uses full parameter iteration in the model detail optimization and missing completion stage, which is time-consuming and difficult to meet the efficiency requirements of real-time reconstruction scenes such as industrial online detection and large-scale scene reconstruction such as digitalization of ancient buildings.

[0005] In addition, some completion algorithms based on generative models only focus on local repair of single-frame images, and do not fully utilize the spatial correlation and semantic consistency between multiple views, resulting in style jump in the completion result under different views, and unable to form a unified three-dimensional structure logic. At the same time, the existing image acquisition usually relies on manual experience to design the camera path, and cannot adaptively plan the trajectory based on the geometric features of the initial coarse-grained model, which easily causes uneven coverage of the view and insufficient sampling of the key area, further limiting the quality of the subsequent reconstruction data and the model detail restoration ability.

[0006] Based on the defects of the prior art, it is urgent to design a technical scheme capable of combining a small number of multi-view image inputs and realizing efficient and high-precision three-dimensional reconstruction, so as to break through the limitations of the prior art and promote the in-depth application of three-dimensional reconstruction technology in various fields. SUMMARY

[0007] The purpose of the present application is to provide a multi-view image three-dimensional reconstruction method to solve the problems of insufficient accuracy, poor defect area completion effect, no effective adaptive adjustment of camera pose error, and low calculation efficiency of the existing multi-view image three-dimensional reconstruction technology in the background art.

[0008] To solve the above technical problems, the technical scheme adopted by the present application is:

[0009] A multi-view image three-dimensional reconstruction method, comprising the following steps:

[0010] Coarse-grained model reconstruction: receiving multiple two-dimensional images of different views as input, performing coarse-grained model reconstruction based on the two-dimensional images, obtaining image depth information through depth estimation, generating color point cloud by fusing color information and initializing three-dimensional Gaussian model, and optimizing model parameters using a multi-objective function containing luminosity loss and geometric regularization;

[0011] Controllable camera trajectory image acquisition: planning a camera trajectory based on the geometric distribution of the coarse-grained three-dimensional model, and acquiring a multi-view image sequence along the camera trajectory;

[0012] Diffusion model fine-tuning and image repair: fine-tuning and repairing the acquired image sequence using a diffusion model, constructing a composite conditional code by fusing camera pose encoding and input image visual encoding, and guiding the model to complete defect area filling;

[0013] Model modification and reconstruction: iteratively optimizing the three-dimensional Gaussian model based on the repaired image sequence, and outputting a high-precision three-dimensional reconstruction model.

[0014] According to the above technical scheme, in the coarse-grained model reconstruction, the luminosity loss function of the coarse-grained model reconstruction is The specific expression is:

[0015]

[0016] Wherein, represents the absolute error between pixels, is an image semantic similarity index, is a weight hyperparameter; this loss function is used to ensure that the model rendering image is highly consistent with the real image at the pixel level and the semantic level, and to promote the restoration of color and details.

[0017] According to the above technical scheme, in the coarse-grained model reconstruction, the geometric regularization term The specific expression is:

[0018]

[0019] in, Indicates depth map in Gradient of position, The constraint, which defines the unit normal vector at the corresponding location, effectively avoids sharp depth jumps, ensuring a smooth and realistic model surface.

[0020] According to the above technical solution, in the controllable camera trajectory image acquisition, the camera trajectory is a closed uniformly distributed path, and the number of acquisition points is set to 12 to 18 to achieve continuous coverage of the viewpoint.

[0021] According to the above technical solution, during the controllable camera trajectory acquisition process, the trajectory sampling density is dynamically optimized for areas with complex textures or insufficient coverage based on the real-time evaluation results of image data quality.

[0022] According to the above technical solution, the diffusion model adopts the stable video diffusion (SVD) model, and the input image is mapped to the latent space through the variational autoencoder (VAE) for denoising and repair.

[0023] According to the above technical solution, in the fine-tuning of the diffusion model and image inpainting, the composite conditional coding is fused through a multi-head cross-attention mechanism, as shown in the following formula:

[0024]

[0025] in Visual encoding of the input image, Encode the camera pose. This is for fine-tuning deviations in pose.

[0026] Based on the above technical solution, the optimization objective of the diffusion model is to minimize the noise prediction error, as shown in the following formula:

[0027]

[0028] in , For images without defects, for Image with added noise at any time They are respectively The noise figure and standard deviation at any given time. and Controlling the intensity of diffused noise, For time step, The sampling noise is a standard normal distribution.

[0029] According to the technical solution, the global optimization updates the Gaussian kernel parameters through gradient descent, and is specifically as follows:

[0030]

[0031] wherein is a set of Gaussian kernel comprehensive parameters, and are photometric and geometric losses respectively, is the number of view angles, is a geometric regularization weight coefficient, represents a repaired image, represents an image generated by rendering based on parameters represents a geometric loss.

[0032] According to the technical solution, the local optimization is performed on the Gaussian kernel subset in the error set, and is specifically as follows:

[0033]

[0034] wherein is a residual weight for increasing the local optimization strength, is a set of Gaussian kernel indexes with large errors; represents a geometric loss of the i-th sample, and the constraint parameter corresponds to the geometric structure rationality, is a geometric loss term, represents a photo-realism loss of the i-th sample.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] The method in the present application realizes high-precision reconstruction under sparse view angles, and can complete three-dimensional reconstruction of a target only through a small number of initial images, thereby breaking through the dependence on multiple data inputs in the prior art, greatly reducing the data acquisition cost while ensuring fine texture and small structure restoration; the method in the present application also combines a composite condition coding and an SVD diffusion model, fully fuses multi-view semantic features and camera pose information, makes the completion result have geometric consistency and texture authenticity, effectively avoids the artifact problem of traditional interpolation methods, and accurately restores the damaged area.

[0037] ​In this invention, by introducing a pose fine-tuning deviation ΔC and a dynamic trajectory optimization mechanism, minute pose errors during the shooting process can be corrected in real time, which greatly improves the spatial consistency of multi-view images and significantly reduces the impact of initial data defects on reconstruction accuracy. Furthermore, by adopting a phased process of coarse-grained modeling and fine-grained optimization, the basic structure can be quickly constructed in a short time. The fine-grained stage only optimizes key parameters and local areas, and the overall computational efficiency is significantly improved compared with existing full-scale optimization methods, which can meet the requirements of near real-time reconstruction. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating the workflow of multi-view image completion and 3D reconstruction of the present invention;

[0039] Figure 2 This is a schematic diagram of the coarse-grained model reconstruction workflow of the present invention;

[0040] Figure 3 This is a schematic diagram of the diffusion model fine-tuning and image repair process of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figure 1 As shown, to achieve the above objectives, the high-precision 3D reconstruction method based on sparse viewpoint input provided by this invention completes 3D reconstruction through a phased collaborative optimization technical path. The specific technical process is as follows:

[0044] First, the input data acquisition operation is performed, receiving four two-dimensional images from different perspectives as initial input. The images must cover the main geometric contours of the target (such as the front, side, top, and bottom surfaces), and the image resolution must be no less than 960×540 and no more than 1920×1080 to ensure that the basic texture and structural features of the target are included, providing a reliable initial data foundation for subsequent modeling.

[0045] After acquiring the input data, the process moves to the coarse-grained model reconstruction stage (e.g., ...). Figure 2As shown in FIG. 1, the first stage builds a coarse-grained 3D Gaussian model based on the initial input images: first, a depth estimation model based on convolutional neural network (CNN) (such as DPT depth estimation model) is used to predict the depth of each input image, and the pixel-level depth value is obtained to establish the mapping relationship between pixel coordinates and depth; then the RGB color information of each image and the depth information of the corresponding pixels are fused to convert into color point cloud in three-dimensional space, and the point cloud density is not less than 100 points per square centimeter to ensure covering all visible areas of the target; then the color point cloud is mapped to a set of three-dimensional Gaussian kernels, the parameters of each Gaussian kernel include position, weight and color, where the position is determined based on point cloud coordinate clustering, and the weight and color are initialized by the statistical characteristics (such as mean and variance) of the local area of the point cloud; finally, a multi-objective function containing luminosity loss and geometric regularization is used to optimize the three-dimensional Gaussian model for 100-200 iterations, and the multi-objective function expression is:

[0046]

[0047] wherein is the absolute error between the input image and the model rendered image, is the semantic similarity index (value range 0-1, the closer to 1 indicates the stronger semantic consistency), is the luminosity loss weight coefficient (value range 0.6-0.8), is the geometric regularization term, and the expression is:

[0048]

[0049] wherein, is the gradient of the depth map at position, is the unit normal vector at the corresponding position, is the geometric regularization weight coefficient (value range 0.2-0.4), and is minimized by gradient descent algorithm to ensure that the geometric structure of the coarse-grained model is highly consistent with the luminosity and semantic features of the input image.

[0050] After the coarse-grained model is constructed, the controllable camera trajectory planning and image acquisition process is initiated: First, a closed, uniformly distributed trajectory around the target is generated based on the geometric contour of the coarse-grained model (such as the maximum circumscribed sphere radius of the target). The trajectory radius is set to 1.5-2 times the maximum circumscribed sphere radius of the target, and 12 acquisition points are evenly set on the trajectory to ensure that the viewing angle between adjacent acquisition points is 30° to achieve full-view coverage of the target. Then, the camera is controlled to move along the planned trajectory, and one image with the same resolution as the initial input image is captured at each acquisition point to obtain a sequence of 12 consecutive viewing angles. At the same time, the quality of the acquired images (such as texture clarity and key structural integrity) is evaluated in real time. If a certain area (such as the depression of the target or the fine texture area) is found to be undersampled or the image is blurry, the acquisition point density in that area is automatically increased (such as reducing the distance between adjacent acquisition points to 1 / 2 of the original distance) to ensure that the image data quality of the key areas meets the requirements of subsequent processing.

[0051] After the image sequence was acquired, diffusion model fine-tuning and image inpainting were immediately carried out (e.g. Figure 3 As shown), this stage uses a geometrically consistent diffusion model to complete the missing parts of the acquired image sequence: First, a stable video diffusion (SVD) model is selected as the base model, and the acquired image sequence is mapped to the latent space through a variational autoencoder (VAE). The latent space dimension is set to 512×512 to reduce the computational cost of the model; then, the visual encoding of the initial 4 input images ( (semantic feature vectors extracted by CNN) and camera pose encoding ( (6-dimensional vectors containing camera position and angle) are fused, introducing pose fine-tuning bias. (Initially set to 0, adaptively optimized through model training), and employs a multi-head cross-attention mechanism (with 8 heads) to achieve deep feature fusion. The constructed composite conditional encoding satisfies the expression to ensure the effective combination of semantic and geometric features. The composite conditional encoding satisfies the expression shown below:

[0052]

[0053] in Visual encoding of the input image, Encode the camera pose. This is for fine-tuning deviations in pose.

[0054] Then, the diffusion model is trained with the goal of minimizing the noise prediction error, and the objective function expression is optimized as follows:

[0055]

[0056] in , ( For images without defects, For the noisy image at time t, , are the noise coefficient and standard deviation at time t, respectively, which linearly increase with is the time step (range 100-1000 steps), is the noise sampled from the standard normal distribution, is the noise prediction function of the model, and the training round is set to 50-100 rounds to ensure that the model has the ability to complete the missing area based on the composite condition coding; finally, the collected missing image sequence is input into the trained diffusion model, and the latent space vector is denoised through the reverse diffusion process (from steps to steps), and then the VAE decoder outputs the image sequence without missing and consistent view. After image repair, enter the final stage of model repair and reconstruction, based on the repaired image sequence to globally and locally optimize the coarse-grained three-dimensional Gaussian model: in the global optimization link, take the repaired image sequence as the reference to calculate the photometric loss

[0057] and geometric loss between the model rendered image and the reference image, update the parameters of all Gaussian kernels through the gradient descent algorithm , and the optimization objective satisfies the expression:

[0058]

[0059] where is the comprehensive parameter set of Gaussian kernel, and are the photometric and geometric losses, respectively, is the number of views, and the iteration number is set to 200-500 to ensure that the global structure of the model is consistent with the reference image; in the local optimization link, the error area with a residual error greater than 0.1 is located by analyzing the residual error distribution (residual error = |rendered pixel value - reference pixel value|) between the model rendered image and the reference image, the corresponding Gaussian kernel subset of the area is extracted, and the residual error weighted local optimization strategy is used to adjust the subset parameters, and the optimization objective satisfies the expression:

[0060]

[0061] where is the residual error weight, with a value range of 0-1, satisfying ​​= residual / maximum residual, to correct the local detail loss and texture discontinuity problem; finally, the global optimization and the local optimization of the three-dimensional Gaussian model are converted into a triangular mesh model (the mesh resolution is not less than 10000 triangular patches), and a high-precision three-dimensional reconstruction model with an average symmetric surface distance (ASSD) of less than or equal to 0.5 mm and a texture similarity (SSIM) of greater than or equal to 0.9 is output.

[0062] The present application adopts this double optimization mechanism, effectively fills in the details of the missing, corrects the depth deviation and color distortion on the basis of ensuring the global consistency of the model, so as to realize the structure reasonable and visual rich high-quality three-dimensional point cloud reconstruction. This strategy has obvious advantages in overcoming the local blur and artifact problems caused by sparse view angle and depth estimation error, etc., and promotes the reality and stability of the final model under multi-view, thereby breaking through the common blur and folding problems in traditional reconstruction.

[0063] At the same time, the present application adopts an iterative feedback mechanism, uses visual and geometric cues to guide the optimization process, gradually reduces the phenomenon of discontinuity of the model surface, and enhances the smoothness and naturalness of the three-dimensional surface. During the reconstruction process, the system also specially considers the coordination of light and texture between multiple views, so that the generated three-dimensional model can more realistically and naturally present diversified light and shadow effects and detail performance.

[0064] Through the in-depth optimization of this module, the entire three-dimensional reconstruction pipeline finally realizes the transformation from rough structure to fine model, not only improves the completeness of the scene, but also greatly enhances the detail richness and visual coherence of visual restoration, so that the generated three-dimensional Gaussian scene and model have the characteristics of high-quality model suitable for multi-angle free-view browsing and immersive interaction, fully meet the needs of virtual reality, cultural heritage digitization and other practical application requirements.

[0065] Embodiment two

[0066] This embodiment is a further refinement of Embodiment 1. To make the technical solution of the present invention clearer and easier to understand, the technical process of the present invention will be described in detail below with reference to specific implementation cases to verify the feasibility and effectiveness of the technical solution. The hardware environment configuration of this embodiment case is as follows: the processor is an Intel Core i9-13900K to ensure data processing efficiency, the memory configuration is 64GB to meet the needs of large-scale data storage, and the graphics card is an NVIDIA RTX 4090 (24GB VRAM) to support computationally intensive tasks such as depth estimation and diffusion model training; the software environment is built on the Ubuntu 22.04 operating system, the deep learning framework is PyTorch 2.5, the depth estimation model is the pre-trained Depth Anything model to ensure depth prediction accuracy, the diffusion model is based on Stable Diffusion v2.1 with adaptive modifications, and the optimization algorithm is AdamW (learning rate set to 1e-4) to improve the model training convergence speed.

[0067] This implementation case selects a Tang Dynasty terracotta figurine as the reconstruction target. The figurine is about 30cm tall and has fine carvings on its surface. It needs to be digitized in three dimensions from all angles using the method of this invention for long-term archiving and digital display of cultural heritage. Therefore, the accuracy and completeness of the detail restoration of the reconstruction model are required to be high. The specific implementation includes the following steps:

[0068] First, input data was acquired by taking four images of the terracotta figurine from different perspectives using a Canon EOS R6 SLR camera: front (0° directly in front of the figurine), left side (90° to the left of the figurine), right side (90° to the right of the figurine), and back (0° directly behind the figurine). The image resolution was downsampled to 1920×1080 to clearly capture the details of the patterns on the surface of the terracotta figurine, ensuring the quality of the initial input data, while reducing the computational load and improving the inference speed of the model.

[0069] The next step is coarse-grained model reconstruction: four input images are fed into the pre-trained Depth Anything model, which outputs a pixel-level depth map with depth values ​​controlled between 0.5-1.5m (corresponding to the actual shooting distance between the camera and the terracotta figure); the "RGB image + depth map" is converted into a color point cloud using the Open3D library, with approximately 500,000 points and a density of approximately 200 points / square centimeter, ensuring coverage of the entire visible surface of the terracotta figure; the K-Means clustering algorithm (100,000 clusters) is used to cluster the color point cloud, with each cluster center serving as the location of the Gaussian kernel. The weights of the Gaussian kernel Set as the percentage of cluster points, color Set the RGB mean of the clustering region to complete the initialization of the 3D Gaussian model; set = 0.8, = 0.2, the multi-objective function is optimized for 5000 iterations using the AdamW algorithm, and the final coarse-grained three-dimensional Gaussian model is obtained.

[0070] After the coarse-grained model is constructed, the controllable camera trajectory planning and image acquisition are started: first, according to the maximum circumscribed sphere radius of the pottery figurine (about 15 cm), the trajectory radius is set to 30 cm (2 times the circumscribed sphere radius), the trajectory shape is a circle with the same height as the center of the pottery figurine, and 12 acquisition points are uniformly set, with an angle of 30° between adjacent acquisition points; the virtual camera designed in the rendering environment moves along the planned trajectory, and 1 image is taken at each of the 12 acquisition points, and 12 image sequences of the pottery figurine are obtained; through real-time evaluation, it is found that the images in the pottery figurine pattern area (initial sampling is insufficient due to obstruction) have a blur problem, so 3 acquisition points are added in the corresponding trajectory segment (including 3 initial acquisition points) in this area, and finally 15 image sequences are obtained to ensure the sufficiency of sampling in the pattern area.

[0071] Next, the diffusion model fine-tuning and image repair are carried out: the modified Stable Video Diffusionv2.1 model is loaded, and the 15 collected images are encoded into a 32x32-dimensional latent space by the Diffusion encoder, with a latent space vector dimension of 8x32x32; similarly, the Diffusion encoder is used to extract the visual encoding of the initial 4 images (dimension 2048), and the camera pose encoding is a 6-dimensional vector (including x, y, z coordinates and roll, pitch, yaw angles), the number of multi-head cross-attention heads is set to 8, ΔC=0 is initialized, and the composite condition encoding C is constructed; the time step t is set to 500 steps, linearly reduced from 0.99 to 0.1, linearly increased from 0.1 to 0.99, the diffusion model is trained for 80 rounds, and after training, the noise prediction error (MSE) of the model is reduced to below 0.01; the 15 collected images are input into the trained model, and the reverse diffusion process from t=500 steps to t=0 steps is performed, outputting 15 pottery figurine image sequences without defects, and the LPSIS of the repaired images is maintained below 0.10.

[0072] Finally, model repair and reconstruction are carried out: taking the 15 repaired images as reference, setting the iteration number to 300 times, updating the Gaussian kernel parameters by gradient descent algorithm, after global optimization, the ASSD of the model rendering image and the reference image is reduced to 0.3 mm; setting the residual threshold to 0.1, positioning the pottery figurine pattern area (residual concentrated area), extracting the corresponding 20,000 Gaussian kernel subsets, setting = residual / 0.5 (the maximum residual of the region is 0.5), 50 iterations of optimization are performed on the subset, and the ASSD of the final pattern region is reduced to 0.2 mm; the optimized three-dimensional Gaussian model is converted into a triangular mesh model (mesh quantity 150,000) by Open3D and saved in PLY format, which can be directly used for VR browsing (supporting 360° rotation viewing) and high-precision measurement (dimensional error ≤0.1 mm), fully meeting the needs of cultural heritage digital archiving.

[0073] The effect of the embodiment is verified by multi-dimensional indexes: in terms of precision index, the average symmetric surface distance (ASSD) between the reconstructed model and the pottery figurine is 0.25 mm, and the texture similarity (SSIM) is 0.93, which is much higher than the precision requirement of cultural heritage digitization; in terms of efficiency index, the total time from data input to model output is 12 minutes, which is 2.75 times more efficient than the existing reconstruction method based on 100 images (time-consuming 45 minutes); in terms of integrity index, the occluded area and the fine pattern area on the back of the pottery figurine are not damaged, and the model has no "artifacts" in all perspectives, with excellent visual coherence, fully verifying the effectiveness and practicality of the technical solution of the present application.

[0074] It should be noted that, in this text, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual such relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0075] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not used to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, and for those skilled in the art, the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for three-dimensional reconstruction of multi-view images, characterized in that: Comprise the following steps: Coarse-grained model reconstruction: receive multiple two-dimensional images of different perspectives as input, perform coarse-grained model reconstruction based on the two-dimensional images, obtain image depth information through depth estimation, fuse color information to generate a color point cloud and initialize a three-dimensional Gaussian model, and adopt a multi-objective function containing luminosity loss and geometric regularization to optimize model parameters; the luminosity loss function of the coarse-grained model reconstruction The specific expression is: wherein, represents the absolute error between pixels, is an image semantic similarity index, is a weight hyperparameter; the loss function is used to ensure that the model rendering image is highly consistent with the real image at the pixel level and semantic level, and promote the restoration of color and details; Geometric regularization term satisfies a specific expression: wherein, represents the gradient of the depth map at position, is the unit normal vector at the corresponding position; this constraint effectively avoids sharp depth jumps, ensuring a smooth and realistic model surface; Controllable camera trajectory image acquisition: based on the geometric distribution of the coarse-grained three-dimensional model to plan the camera trajectory, the camera trajectory is a closed uniform distribution path, the number of sampling points is set to 12-18, and a multi-view image sequence is collected along the camera trajectory; and based on the real-time evaluation results of image data quality, the trajectory sampling density is dynamically optimized for the region with complex texture or insufficient coverage; Diffusion model fine-tuning and image repair: using a stable video diffusion SVD model to fine-tune and repair the collected image sequence, mapping the input image to the latent space for denoising and repair through a variational autoencoder VAE, and constructing a composite conditional code by fusing camera pose encoding and input image visual encoding to guide the model to complete the filling of the missing area; wherein the composite conditional code is fused through a multi-head cross attention mechanism, as shown in the following formula: wherein is an input image visual encoding, is a camera pose encoding, is a pose fine-tuning bias; Model modification and reconstruction: based on the repaired image sequence, iteratively optimize the three-dimensional Gaussian model, and output a high-precision three-dimensional reconstruction model.

2. The method of claim 1, wherein: During the controllable camera trajectory acquisition process, based on the real-time evaluation results of image data quality, the trajectory sampling density is dynamically optimized for the region with complex texture or insufficient coverage.

3. The method of claim 1, wherein: The diffusion model uses a stable video diffusion SVD model, which maps the input image to the latent space for denoising and repair through a variational autoencoder VAE.

4. The method of claim 1, wherein: The optimization objective of the diffusion model is to minimize the noise prediction error, as shown in the following formula: wherein , is a defect-free image, is a noisy image at time instant, , are noise coefficients and standard deviations at time instant, and control the intensity of the diffusion noise, is a time step, is a standard normal distribution sampled noise.

5. The method of claim 1, wherein: Global optimization updates the Gaussian kernel parameters through gradient descent, as shown in the following formula: wherein is a set of Gaussian kernel synthesis parameters, and are photometric and geometric losses, respectively, is the number of view angles, is a geometric regularization weight coefficient, denotes the inpainted image, denotes the image generated by the parametric rendering, denotes the geometric loss.

6. The method of claim 5, wherein: Local optimization is performed on the Gaussian kernel subset in the error set, as shown in the following formula: wherein is a residual weight, for increasing the local optimization strength, is a set of Gaussian kernel indices; denotes the geometry loss of the i-th sample, and the constraint parameter corresponding to the geometric structure rationality, is a geometry loss term, denotes the photo-realism loss of the i-th sample.

Images