Three-dimensional gaussian splash geometry prior model construction and scene new view rendering method

Abstract

The application provides a three-dimensional Gaussian splash geometry prior model construction and scene new view rendering method, wherein the model construction method comprises the following steps: acquiring a training image and a corresponding camera pose thereof; constructing an initial three-dimensional Gaussian splash model; training the initial three-dimensional Gaussian splash model according to the training image and the corresponding camera pose thereof to obtain a geometry prior model; wherein in the training process, a frequency-guided resolution scheduling strategy is adopted to determine a switching time from a low-resolution training stage to a high-resolution training stage, and when the switching time is reached, the resolution of the training image is switched from low resolution to high resolution. The application realizes a leap-forward improvement in the efficiency of geometry prior construction by introducing a frequency-guided resolution scheduling mechanism.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D graphics technology, specifically to a method for constructing a 3D Gaussian splash geometry prior model and rendering a new perspective of a scene. Background Technology

[0002] New perspective rendering of 3D scenes is an important research direction in computer vision and computer graphics. Its goal is to synthesize high-quality images from unobserved viewpoints, given a limited input image and corresponding camera parameters. This technology has wide applications in autonomous driving, virtual reality, digital twins, and film and television production. However, due to the limited scene coverage of recorded videos and the complexity of driving dynamics, generating high-quality images from new camera perspectives deviating from the original acquisition trajectory remains a challenge.

[0003] In recent years, rendering methods based on 3D explicit or implicit representations have been continuously developing, such as multi-view geometry reconstruction methods, neural radiation fields, and their derivatives. These methods typically rely on dense, multi-view, high-quality training data to ensure consistency between geometric structure and texture appearance. However, in real-world applications, due to limitations in acquisition conditions, viewpoint coverage, and cost factors, training data often exhibits sparse viewpoints and limited coverage, leading to problems such as geometric distortion, texture misalignment, or false details when extrapolating to a large range of unobserved viewpoints. Recent representative solutions attempt to leverage the powerful capabilities of diffusion models to correct rendering artifacts and fill in missing regions. For example, Wang et al. (Wang, Q., Fan, L., Wang, Y., Chen, Y., & Zhang, Z. FreeVS: Generative View Synthesis on Free Driving Trajectory. In TheThirteenth International Conference on Learning Representations.) proposed training a stable video diffusion model to repair projection artifacts under new viewpoints. Fan et al. (Fan, L., Zhang, H., Wang, Q., Li, H., & Zhang, Z. (2025). Freesim: Toward free-viewpoint camerasimulation in driving scenes. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 12004-12014).) achieved high-quality extrapolation rendering by periodically translating the trajectory with small steps and progressively optimizing the 3D Gaussian representation.

[0004] While these diffusion-based methods have achieved some success, they still have drawbacks. Specifically, these methods typically require training video diffusion models on large-scale open-world driving datasets, which is computationally expensive and time-consuming. For example, training FreeSim takes approximately 50 hours.

[0005] Furthermore, due to the complexity of training diffusion models, the quality of the generated data is difficult to guarantee. Low-quality or hallucinatory synthetic images, once used for training, will cause cumulative damage to geometric priors, thereby significantly reducing the stability and credibility of the rendering results.

[0006] A search revealed that patent application number 202510211454.1 discloses a method and system for synthesizing 3D human body images from a sparse perspective, capable of generating arbitrary new perspective images under sparse perspective conditions while ensuring image quality. However, this solution does not address the reliability of training data and the synthesis strategy, nor does it involve a mechanism for quality screening and iterative optimization of the generated data to improve geometric prior stability. Summary of the Invention

[0007] To address one of the shortcomings of existing technologies, the purpose of this application is to provide a method for constructing a three-dimensional Gaussian splash geometry prior model and rendering a new perspective of the scene.

[0008] A first aspect of this application provides a method for constructing a three-dimensional Gaussian splash geometry prior model, comprising: Obtain training images and their corresponding camera poses; Construct an initial 3D Gaussian splash model; The initial 3D Gaussian splash model is trained based on the training images and their corresponding camera poses to obtain a geometric prior model. In the training process, a frequency-guided resolution scheduling strategy is adopted to determine the timing of switching from a low-resolution training phase to a high-resolution training phase, and when the switching timing is reached, the resolution of the training image is switched from low resolution to high resolution.

[0009] Optionally, the frequency-guided resolution scheduling strategy, which determines the timing of switching from a low-resolution training phase to a high-resolution training phase, includes: For a resolution of training image set By calculating the average of its discrete Fourier transform Norm measures the significance of frequency The definition is as follows:

[0010] in, Indicates the first Training Views The frequency domain representation of the training image is given by N, where N represents the number of training views, h and w represent the height and width of the training image, respectively, and i and j represent the row and column indices of the pixels in the training view. The training image set is downsampled, and its frequency significance is calculated. , as the frequency components that can be covered under low-resolution conditions; Calculate the original training image set The proportion of mid-to-high frequency residual information in the total frequency energy :

[0011] Based on the aforementioned ratio, the proportion of optimization steps allocated to the high-resolution training phase is determined as follows: ; Based on the optimized step ratio of the high-resolution training phase, the number of iteration steps in the low-resolution training phase is obtained. :

[0012] in This represents the total number of optimization steps; The number of iteration steps As the time for switching.

[0013] Optionally, during the training process, the number of Gaussian primitives is also controlled synchronously, specifically: Before the switching timing, limit the growth of the number of Gaussian primitives; After the switching timing, the number of Gaussian primitives is allowed to increase gradually.

[0014] A second aspect of this application provides a new perspective rendering method for three-dimensional scenes, including: Based on the real camera pose on the original acquisition trajectory, a new target camera pose is generated according to the preset extrapolation step size; The geometric prior model obtained by any of the three-dimensional Gaussian splash geometric prior model construction methods described herein is used to render the pose of the new target camera, resulting in a degraded image. By combining real reference images, a pre-trained 3D augmentation diffusion model is used to generate repaired images in a zero-shot manner; The restored images are comprehensively scored using no-reference image quality assessment metrics and lateral offset, and high-priority samples are selected to form a synthetic training set. The geometric prior model is fine-tuned using the synthetic training set; The above extrapolation, repair, filtering and fine-tuning steps are executed iteratively until the target view range is reached to obtain the final optimized 3D Gaussian splash model; A new perspective rendering of the 3D scene was performed using the final optimized 3D Gaussian splash model.

[0015] Optionally, a geometric prior model is used to render the pose of the new target camera to obtain a degraded image. Combined with a real reference image, a repaired image is generated in a zero-shot manner using a pre-trained 3D enhancement diffusion model, including: Generate a new target camera pose based on a preset extrapolation step size; The new target camera pose is rendered using a geometric prior model to obtain a degraded image; Obtain a pre-trained 3D augmented diffusion model; The degraded image The image is fed into a pre-trained 3D augmentation diffusion model along with a real reference image as dual inputs to obtain the restored image.

[0016] Optionally, the step of comprehensively scoring the repaired image using a no-reference image quality assessment metric and lateral offset, and selecting high-priority samples to form a synthetic training set, includes: For each restored image, calculate its no-reference image quality assessment index and obtain the lateral offset of its corresponding camera pose relative to the original acquisition trajectory. The comprehensive priority score is obtained by normalizing the no-reference image quality assessment index and then weighting it together with the normalized lateral offset. All candidate repair images are sorted in descending order according to the comprehensive priority score, and samples are selected sequentially until the number of samples in the synthetic training set reaches a preset upper limit, which does not exceed a specified proportion of the number of original real training data.

[0017] Optionally, the comprehensive priority score is: ,in, and These are the weighting coefficients for the mass term and the displacement term, respectively. This represents the BRISQUE score of the k-th restored image. This represents the highest BRISQUE score among all restored images. This indicates the preset target lateral offset. This indicates the lateral offset.

[0018] Optionally, the extrapolation step size is 0.5 meters.

[0019] Optionally, the target viewing angle range has a lateral offset of no more than 3 meters.

[0020] A third aspect of this application provides a new perspective rendering system for three-dimensional scenes, comprising: The extrapolation module generates a new target camera pose based on the real camera pose on the original acquisition trajectory and according to the preset extrapolation step size; The repair module uses the geometric prior model obtained by the three-dimensional Gaussian splash geometric prior model construction method described in any one of the above to render the pose of the new target camera, obtain the degraded image, and combine it with the real reference image to generate the repair image in a zero-shot manner through a pre-trained three-dimensional enhancement diffusion model. The screening module uses the no-reference image quality assessment index and the lateral offset to comprehensively score the repaired image and select high-priority samples to form a synthetic training set. The fine-tuning module synthesizes a training set for fine-tuning the geometric prior model. The training module iteratively executes the above extrapolation, repair, filtering and fine-tuning steps until the target view range is reached, and obtains the final optimized 3D Gaussian splash model; The rendering module uses the final optimized 3D Gaussian splash model to render the 3D scene from a new perspective.

[0021] The three-dimensional Gaussian splash geometric prior model construction method of this application effectively improves the efficiency of geometric prior construction by introducing frequency-guided resolution scheduling.

[0022] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0023] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a method for constructing a three-dimensional Gaussian splash model according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a new perspective rendering method for a 3D scene according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating the framework structure of a new perspective rendering method for a 3D scene according to an exemplary embodiment; Figure 4 This is a framework diagram of a new perspective rendering system for a three-dimensional scene according to an exemplary embodiment; Figure 5 This is a performance comparison of the present application and existing generative novel perspective synthesis methods in a test scenario, as illustrated by an exemplary embodiment. Detailed Implementation

[0024] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0025] Existing technologies typically require training video diffusion models on large-scale open-world driving datasets, which is computationally expensive and time-consuming. To address these issues, this application provides a method for constructing a three-dimensional Gaussian splash geometry prior model.

[0026] Reference Figure 1As shown in one embodiment of this application, a method for constructing a three-dimensional Gaussian splash geometric prior model includes the following steps: S100, acquires training images and their corresponding camera poses.

[0027] Specifically, training images refer to the initial collection of real-world scene multi-view images with sparse viewpoints. Camera pose refers to the position and orientation parameters (i.e., camera extrinsic parameters) of the camera in the three-dimensional world coordinate system at the moment each training image is acquired, used to establish the projection relationship between three-dimensional spatial points and two-dimensional image pixels.

[0028] S200, construct the initial three-dimensional Gaussian splash model.

[0029] Specifically, the initial 3D Gaussian splash model is a set of multiple 3D Gaussian primitives, each containing position, covariance, opacity, and color attributes. This model is initialized using sparse point clouds generated by Structure for Motion Reconstruction (SfM) based on the training images and camera poses obtained in step S100, and serves as the initial state for subsequent frequency-guided optimization training.

[0030] S300 trains the initial 3D Gaussian splash model based on the training images and their corresponding camera poses to obtain the geometric prior model.

[0031] During the training process, a frequency-guided resolution scheduling strategy is adopted to determine the timing of switching from the low-resolution training phase to the high-resolution training phase, and when the switching timing is reached, the resolution of the training images is switched from low resolution to high resolution.

[0032] Specifically, the low-resolution training phase refers to using downsampled low-resolution training images as supervision images during model optimization; the high-resolution training phase refers to using the original high-resolution training images as supervision images during model optimization.

[0033] Specifically, training iteratively updates the Gaussian primitive parameters by minimizing the L1 loss and SSIM loss between the rendered image and the supervised image.

[0034] The above embodiments effectively improve the efficiency of geometric prior construction by introducing a frequency-guided resolution scheduling mechanism. Compared with traditional methods that perform indiscriminate optimization at full resolution, the above embodiments can accurately identify the geometric complexity distribution in the scene and prioritize the allocation of computing resources to key high-frequency details. This architecture, which decouples "coarse model construction" from "fine-grained repair," greatly shortens the convergence time of large-scale autonomous driving scene reconstruction, making real-time or near-real-time scene generation possible.

[0035] To reduce computational overhead and training costs, in some specific embodiments of this application, a frequency-guided resolution scheduling strategy is adopted in step S300 to determine the timing of switching from the low-resolution training phase to the high-resolution training phase, thereby realizing the gradual optimization of its geometric structure from low resolution to high resolution. This may include the following steps: S301, for a resolution of training image set By calculating the average of its discrete Fourier transform Norm is used to measure the significance of its frequency. The definition is as follows:

[0036] in, Indicates the first Training Views The frequency domain representation of the training image is given by N, where N represents the number of training views, h and w represent the height and width of the training image, respectively, and i and j represent the row and column indices of the pixels in the training view. S302, downsample the training image set and calculate its frequency significance. , as the frequency components that can be covered under low-resolution conditions; S303, Calculate the original training image set The proportion of mid-to-high frequency residual information in the total frequency energy :

[0037] The overall frequency energy here refers to the frequency saliency of the original high-resolution training image calculated in S301.

[0038] A larger proportion indicates that the low-resolution image has lost more details, thus requiring an earlier switch to the high-resolution training phase. Conversely, a smaller proportion indicates that the low-resolution image already contains most of the information, allowing for a delayed switch to more fully stabilize its structure at lower resolution, thereby saving computational resources.

[0039] S304, Based on the above proportional relationship, the proportion of optimization steps that should be allocated to high-resolution rendering can be determined as follows: .

[0040] S305, based on the ratio determined in S304, determine the number of iteration steps to switch from low resolution to high resolution. The calculation is as follows:

[0041] in, This represents the total number of optimization steps.

[0042] The strategy described above ensures that high-frequency supervision is introduced only after low-frequency components have been fully fitted, thereby effectively avoiding aliasing and redundant computation, and concentrating more computational resources on the high-fidelity detail fitting stage.

[0043] To further improve overall efficiency, the number of Gaussian primitives needs to be coordinated with the rendering resolution scheduling process. If the number of primitives is not constrained at low resolutions, directly performing densification operations will lead to excessive generation of Gaussian primitives, resulting in unnecessary computational overhead. Therefore, in one specific embodiment of this application, a Gaussian primitive scheduling mechanism is introduced, specifically: Dynamically control the number of active Gaussian primitives The growth process is synchronized with the resolution scheduling strategy.

[0044] For example, the total training steps are set to 10,000 steps, and the switching steps are set to 4,000 steps.

[0045] So, in the low-resolution stage from 0 to 4,000 steps, the densification operation of Gaussian primitives is disabled, and the number of primitives increases only slightly from the initial 10,000 to 12,000; in the high-resolution stage from 4,000 to 10,000 steps, the densification operation is enabled, and the number of primitives gradually increases to 150,000.

[0046] The Gaussian primitive scheduling mechanism in the above embodiment actively suppresses the rapid growth of the number of primitives in the low-resolution stage, and only gradually increases the necessary number of primitives after the resolution is improved, which significantly reduces the computational overhead of the early training.

[0047] Based on the same technical concept, another embodiment of this application provides a new perspective rendering method for three-dimensional scenes, such as... Figure 2 and Figure 3 As shown, it includes the following steps: S1, Based on the real camera pose on the original acquisition trajectory, generate a new target camera pose according to the preset extrapolation step size; S2, using any of the three-dimensional Gaussian splash geometric prior model construction methods to obtain a geometric prior model, renders the pose of the new target camera to obtain a degraded image, and combines it with a real reference image to generate a repaired image in a zero-shot manner through a pre-trained three-dimensional enhancement diffusion model. S3 uses the no-reference image quality assessment index and lateral offset to comprehensively score the repaired image, and selects high-priority samples to form a synthetic training set. S4. A synthetic training set is used to fine-tune the geometric prior model, and the above extrapolation, repair, filtering, and fine-tuning steps are iteratively executed until the target viewpoint range is reached, resulting in the final optimized 3D Gaussian splash model. ; S5 uses the final optimized 3D Gaussian splash model. To render 3D scenes from a new perspective.

[0048] The above embodiments, by introducing a reliability-guided screening mechanism, comprehensively evaluate the BRISQUE quality score and geometric expansion value of the repaired images generated by the diffusion model, retaining only high-confidence samples for model fine-tuning, effectively blocking the contamination of geometric priors by low-quality or hallucinatory synthetic data, avoiding error accumulation, and significantly improving the stability and credibility of new perspective rendering.

[0049] Progressive small-step extrapolation rendering and zero-shot generative inpainting. When the synthetic viewpoint is far from the original acquired trajectory, the geometric reconstruction results relying solely on the original geometric prior model often quickly exhibit severe geometric degradation. In this case, a generative model needs to be introduced for compensation. However, existing state-of-the-art novel viewpoint synthetic generative models are typically trained on static multi-view stereo or surround-view datasets. When directly applied to sparse, long-trajectory autonomous driving data such as WOD (Waymo Open Dataset), they often face significant domain mismatch problems. Adapting these models usually requires costly fine-tuning and easily introduces illusory structures with inconsistent 3D structures. This application efficiently utilizes powerful pre-trained generative models in a zero-shot manner, thereby avoiding the computational burden caused by data domain mismatch and additional training. In some specific embodiments of this application, S2, based on the geometric prior model, i.e., the 3D Gaussian splash model obtained in steps S100-S200 above, the small-step extrapolation pose is rendered to obtain a degraded image. Combined with a real reference image, a repaired image is generated in a zero-shot manner using a pre-trained 3D enhancement diffusion model. This can be achieved through the following steps: S21, the pre-trained 3D augmented diffusion model Difix3D+ is selected for zero-shot inpainting, and this model is denoted as... Zero-sample repair refers to repairing without updating... Given the model parameters, its generative capabilities are directly utilized to repair degraded images. Although It is trained based on view-style data, but its key advantage lies in its ability to support the introduction of real reference images. As additional condition input.

[0050] S22, by employing a progressive small-step extrapolation strategy, utilizes as a geometric prior model The rendered image contains only mild degradation.

[0051] The new target camera pose obtained by S1; S23, It operates under two conditions: on the one hand, it operates under two conditions. Geometric cues are provided as the primary conditional input, while a separate reference image is introduced on the other hand. .

[0052] Specific, individual reference images The reference image is derived from the original training image set obtained in step S100. Specifically, the reference image is the real-world image captured corresponding to the original camera pose used to generate the degraded image. For example, if the new target camera pose is extrapolated from the pose of the k-th frame of the original acquisition trajectory, then the reference image is the real-world image of the k-th frame of the original acquisition trajectory.

[0053] S24, in order to overcome domain differences while ensuring texture realism and geometric consistency, Directly set to real captured images .exist Provide geometric constraints, Providing both texture and realism guidance, It can adaptively transfer its learned 3D priors to autonomous driving trajectory scenarios without any additional training, achieving high-quality image inpainting and completion, performing quality filtering, and obtaining repaired images.

[0054]

[0055] This method transforms the complex problem of large-scale structural extrapolation into a task of repairing local geometric and texture inconsistencies, thereby effectively bridging data domain differences under zero-sample conditions.

[0056] The above embodiments effectively mitigate the geometric collapse problem of 3D Gaussian splashing when it is far from the trajectory by employing a progressive small-step extrapolation strategy. Utilizing the dual-condition guidance interface of the pre-trained diffusion model, the restoration process preserves both the geometric constraints of the original scene (guided by the degraded image) and injects real-world texture distribution (through ground truth reference). This approach ensures that the generated virtual perspective image maintains strict spatial logical consistency and visual coherence even under drastic pose shifts.

[0057] To further suppress content illusion and artifact interference, this application employs a reliability guarantee mechanism to screen high-quality synthetic data and feed it back into iterative optimization. Specifically, the restored image is first scored using the BRISQUE (Blind / Referenceless Image Spatial Quality Evaluator) metric, retaining the highest-scoring foreground view. Furthermore, the size of the synthetic dataset is limited by hyperparameters to ensure it does not exceed the proportion of the original data, thus reducing the risk of geometric damage. Finally, a comprehensive priority score is calculated based on the BRISQUE score and lateral offset, prioritizing data points that maximize geometric expansion and have reliable quality for model fine-tuning.

[0058] In one specific implementation, S3, the Irep is comprehensively scored using the no-reference image quality assessment metric BRISQUE and the lateral offset to select high-priority samples, impose quantity constraints, and form a synthetic training set. This can be achieved through the following steps: S31, Define the dataset.

[0059] In each iteration (corresponding to one pose extrapolation step size) Dynamically construct synthetic training datasets .

[0060] The dataset consists of several sample pairs. Composition, in which This represents the corresponding camera pose. The current candidate data pool consists of data retained from the previous iteration. Compared with the newly generated data in this round Together they form a subset of data, which is then selected through a two-stage screening mechanism to obtain the final subset of data used for training.

[0061] S32, to maximize the geometric expansion effect while satisfying data size constraints, for each candidate sample... Calculate a composite priority score The rating balances image quality with the urgency of spatial extrapolation. Defined as the normalized BRISQUE image quality score and lateral displacement. Weighted sum:

[0062] and These are the weighting coefficients for the mass term and the displacement term, respectively.

[0063] S33, through the All samples according to The samples are sorted, and the highest-scoring samples are selected sequentially until the specified quantity constraint is met, forming the synthetic training dataset used in the current iteration. This strategy can dynamically maintain the dataset structure during multiple iterations, prioritizing the retention of newly generated samples that are of high quality and have significant geometric expansion value.

[0064] The above embodiments construct a closed-loop feedback mechanism based on reliability assessment, and rigorously screen generative repair results through a composite dynamic priority scoring system. Unlike traditional generative models that may produce inaccurate phenomena, this mechanism can automatically identify and eliminate synthetic data with hallucinatory features or 3D inconsistencies. This iterative process of "high-quality data feedback geometric priors" ensures that the model learns only realistic and reliable scene representations.

[0065] In some specific implementations, S4, a synthetic training set is used to fine-tune the geometric prior model, and the above extrapolation, repair, filtering, and fine-tuning steps are iteratively executed until the target view range is reached, obtaining the final optimized model. The model can be developed using the following steps: S41 will select high-priority samples that have passed reliability screening. Feedback to the geometric prior model In this context, it is used to fine-tune and update the parameters of existing Gaussian primitives, thereby gradually improving the expressive power of the geometric model within an expanded viewpoint range.

[0066] Specifically, the fine-tuning update here refers to: taking the current parameter state of the geometric prior model as the initial state, adding high-priority samples to the training set, directly using the original high-resolution image as the supervision signal, calculating the rendering loss and backpropagating, updating the position, color, and opacity parameters of the Gaussian primitives, and obtaining the updated geometric prior model.

[0067] S42, repeat S2 to S4, gradually expanding the extrapolation range of the viewing angle until the preset target distance is reached (e.g., ).

[0068] S43 uses the model that iterates step by step and reaches the target distance as the final optimization. Model.

[0069] In some other embodiments, after all iterations are completed, the newly optimized model is used to perform a new perspective synthesis test, and the reconstruction quality index PSNR, SSIM and generation quality index FID are calculated respectively to comprehensively evaluate the performance of the method of this application.

[0070] Based on the same technical concept, other embodiments of this application, such as Figure 4 As shown, a new perspective rendering system 100 for three-dimensional scenes is also provided, including: Extrapolation module 110 generates a new target camera pose based on the real camera pose on the original acquisition trajectory and according to the preset extrapolation step size; Repair module 120 uses a geometric prior model obtained by any of the three-dimensional Gaussian splash geometric prior model construction methods to render the pose of the new target camera, obtain a degraded image, and combine it with the real reference image Igt to generate a repaired image in a zero-shot manner through a pre-trained three-dimensional enhanced diffusion model. The screening module 130 uses the no-reference image quality assessment index and the lateral offset to comprehensively score the repaired image and select high-priority samples to form a synthetic training set. Fine-tuning training module 140, synthesizing training sets for fine-tuning geometric prior models; Training module 150 iteratively executes the above extrapolation, repair, filtering, and fine-tuning steps until the target field of view is reached, obtaining the final optimized result. Model; Rendering module 160, using final optimization The model is rendered from a new perspective in a 3D scene.

[0071] The specific implementation techniques of each module / unit in the above embodiments of this application can be referred to the steps of the three-dimensional scene new perspective rendering method in the above embodiments, and will not be repeated here.

[0072] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0073] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0074] Experimental Example 1 Ten wide street scene sequences from the Waymo Open Dataset were selected for testing. In each sequence, 81 frames were uniformly sampled as input. The test model (using the aforementioned novel 3D scene rendering system or method to obtain the final image) was used. The performance of the model at lateral offsets of 0m (original acquisition trajectory), 1m, 2m and 3m was used to test the novel perspective synthesis performance of this application.

[0075] Table 1 compares the performance of existing generative new perspective synthesis methods in the test scenario.

[0076]

[0077] The experimental results are shown in Table 1. The results show that the application maintained a certain reconstruction quality on the recorded trajectory 0m.

[0078] like Figure 5 As shown, in more challenging extrapolation tasks (1m to 3m), our method significantly outperforms all baseline methods in terms of FID across all offset distances. Particularly at the 3m offset, our method reduces the FID to 17.62, a reduction of approximately 17.7% compared to the state-of-the-art (SOTA) FreeVS method. This result validates that our method significantly improves the generalization performance of the model on unknown data domains by mitigating the model's dependence on frequency domain shortcuts and reducing overfitting to the training data.

[0079] Experimental Example 2: Selected scenarios were used to test the time consumption during the reconstruction phase. The effect of the introduced frequency-guided scheduling strategy on... Impact of modules. Comparison of standard 3DGS models with the optimized models in this application. The module's time consumption and reconstruction accuracy in single-scene sequence training.

[0080] Table 2 shows the standard 3DGS model and the optimized model of this application. Comparison of time consumption and reconstruction accuracy in single-scene sequence training of modules.

[0081]

[0082] Table 2 shows the results of this application. The module achieves a training time reduction of approximately 66% through dynamic resolution and primitive scheduling, without loss of reconstruction quality (PSNR), demonstrating its efficiency as a geometric prior module.

[0083] Experimental Example 3: The total time taken to complete the entire workflow was calculated: from initial geometric model training and generative model preparation to the completion of full asymptotic extrapolation of 1m, 2m, and 3m offsets, to verify the efficiency of this application in the test scenario.

[0084] Table 3 compares the total time taken to complete the entire workflow.

[0085]

[0086] Table 3 shows that the results verify that this application can complete the extrapolation tasks of each level of a single scene in just 3.1 hours on a single A800 GPU, which is far superior to the existing technology in terms of efficiency and performance.

[0087] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for constructing a three-dimensional Gaussian splash geometric prior model, characterized in that, include: Obtain training images and their corresponding camera poses; Construct an initial 3D Gaussian splash model; The initial 3D Gaussian splash model is trained based on the training images and their corresponding camera poses to obtain a geometric prior model. In the training process, a frequency-guided resolution scheduling strategy is adopted to determine the timing of switching from a low-resolution training phase to a high-resolution training phase, and when the switching timing is reached, the resolution of the training image is switched from low resolution to high resolution.

2. The method for constructing a three-dimensional Gaussian splash geometric prior model according to claim 1, characterized in that, The frequency-guided resolution scheduling strategy determines the timing of switching from the low-resolution training phase to the high-resolution training phase, including: For a resolution of training image set By calculating the average of its discrete Fourier transform Norm measures the significance of frequency The definition is as follows: ； in, Indicates the first Training Views The frequency domain representation of the training image is given by N, where N represents the number of training views, h and w represent the height and width of the training image, respectively, and i and j represent the row and column indices of the pixels in the training view. The training image set is downsampled, and its frequency significance is calculated. , as the frequency components that can be covered under low-resolution conditions; Calculate the original training image set The proportion of mid-to-high frequency residual information in the total frequency energy : ； Based on the aforementioned ratio, the proportion of optimization steps allocated to the high-resolution training phase is determined as follows: ; Based on the optimized step ratio of the high-resolution training phase, the number of iteration steps in the low-resolution training phase is obtained. : ； in This represents the total number of optimization steps; The number of iteration steps As the time for switching.

3. The method for constructing a three-dimensional Gaussian splash geometric prior model according to claim 2, characterized in that, During the training process, the number of Gaussian primitives is also controlled synchronously, specifically: Before the switching timing, limit the growth of the number of Gaussian primitives; After the switching timing, the number of Gaussian primitives is allowed to increase gradually.

4. A new perspective rendering method for 3D scenes, characterized in that, include: Based on the real camera pose on the original acquisition trajectory, a new target camera pose is generated according to the preset extrapolation step size; The geometric prior model obtained by the three-dimensional Gaussian splash geometric prior model construction method according to any one of claims 1-3 is used to render the pose of the new target camera to obtain a degraded image. By combining real reference images, a pre-trained 3D augmentation diffusion model is used to generate repaired images in a zero-shot manner; The restored images are comprehensively scored using no-reference image quality assessment metrics and lateral offset, and high-priority samples are selected to form a synthetic training set. The geometric prior model is fine-tuned using the synthetic training set; The above extrapolation, repair, filtering and fine-tuning steps are executed iteratively until the target view range is reached to obtain the final optimized 3D Gaussian splash model; A new perspective rendering of the 3D scene was performed using the final optimized 3D Gaussian splash model.

5. The three-dimensional scene new perspective rendering method according to claim 4, characterized in that, A geometric prior model is used to render the pose of the new target camera, resulting in a degraded image. Combined with a real reference image, a repaired image is generated in a zero-shot manner using a pre-trained 3D enhancement diffusion model, including: Generate a new target camera pose based on a preset extrapolation step size; The new target camera pose is rendered using a geometric prior model to obtain a degraded image; Obtain a pre-trained 3D augmented diffusion model; The degraded image and the real reference image are used as dual inputs and fed into a pre-trained 3D enhancement diffusion model to obtain the repaired image.

6. The three-dimensional scene new perspective rendering method according to claim 4, characterized in that, The process of comprehensively scoring the repaired image using a no-reference image quality assessment metric and lateral offset, and selecting high-priority samples to form a synthetic training set, includes: For each restored image, calculate its no-reference image quality assessment index and obtain the lateral offset of its corresponding camera pose relative to the original acquisition trajectory. The comprehensive priority score is obtained by normalizing the no-reference image quality assessment index and then weighting it together with the normalized lateral offset. All candidate repair images are sorted in descending order according to the comprehensive priority score, and samples are selected sequentially until the number of samples in the synthetic training set reaches a preset upper limit, which does not exceed a specified proportion of the number of original real training data.

7. The three-dimensional scene new perspective rendering method according to claim 6, characterized in that, The overall priority score is: ,in, and These are the weighting coefficients for the mass term and the displacement term, respectively. This represents the BRISQUE score of the k-th restored image. This represents the highest BRISQUE score among all restored images. This indicates the preset target lateral offset. This indicates the lateral offset.

8. The three-dimensional scene new perspective rendering method according to claim 4, characterized in that, The extrapolation step size is 0.5 meters.

9. The three-dimensional scene new perspective rendering method according to claim 4, characterized in that, The target viewing angle range is defined as having a lateral offset of no more than 3 meters.

10. A new perspective rendering system for three-dimensional scenes, characterized in that, include: The extrapolation module generates a new target camera pose based on the real camera pose on the original acquisition trajectory and according to the preset extrapolation step size; The repair module uses the geometric prior model obtained by the three-dimensional Gaussian splash geometric prior model construction method described in any one of claims 1-3 to render the pose of the new target camera, obtain the degraded image, and combine it with the real reference image to generate the repair image in a zero-shot manner through a pre-trained three-dimensional enhanced diffusion model. The screening module uses the no-reference image quality assessment index and the lateral offset to comprehensively score the repaired image and select high-priority samples to form a synthetic training set. The fine-tuning module synthesizes a training set for fine-tuning the geometric prior model. The training module iteratively executes the above extrapolation, repair, filtering and fine-tuning steps until the target view range is reached, and obtains the final optimized 3D Gaussian splash model; The rendering module uses the final optimized 3D Gaussian splash model to render the 3D scene from a new perspective.

Images