A three-dimensional gaussian sputter rendering and camera self-calibration method based on a general camera model

Abstract

The application discloses a three-dimensional Gaussian sputter rendering and camera self-calibration method based on a general camera model. The method maps a three-dimensional point or direction of a camera coordinate system to a two-dimensional pixel coordinate by defining a learnable camera mapping function, and calculates a mapped Jacobian matrix to transfer a three-dimensional Gaussian covariance into a two-dimensional covariance and convert the two-dimensional covariance into a quadratic form parameter. Gaussian sputter rasterization is performed based on the quadratic form parameter to obtain a rendering image. With a reconstruction error as a target, an effective field of view mask and camera robustness are used to realize joint optimization and online self-calibration of camera parameters and three-dimensional Gaussian parameters. The scheme is suitable for fisheye, panoramic and catadioptric non-pinhole cameras, and can improve the rendering quality and stability of a large field of view under unknown or time-varying distortion conditions.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and computer vision, and in particular to a method for 3D Gaussian sputtering rendering and camera self-calibration based on a universal camera model. Background Technology

[0002] Most existing novel perspective synthesis and 3D reconstruction methods are based on pinhole camera models or rely on predefined fixed-parameter wide-angle / fisheye camera models (such as isometric projection models, Kannala-Brandt models, etc.). These methods typically assume that the camera imaging process can be accurately described by finite-dimensional analytical parameters and remains invariant throughout the imaging process. However, in real-world applications, especially in cases of ultra-wide field-of-view imaging, strong nonlinear distortion, complex lens structures, or manufacturing errors, these assumptions often prove difficult to uphold.

[0003] Specifically, fixed-parameter models have limited expressive power when the field of view is close to or exceeds 180°, making it difficult to simultaneously ensure projection accuracy in both the central and peripheral regions. This can easily lead to significant geometric distortion and reprojection deviations at the edges of the field of view. Furthermore, parameters obtained from offline calibration are difficult to maintain long-term validity when the lens has assembly errors, temperature drift, mechanical aging, or individual differences. For zoom lenses or variable focal length imaging systems, traditional methods typically require calibration at multiple focal lengths or the introduction of complex segmented modeling and interpolation strategies, resulting in complex system implementation and poor robustness. Moreover, for panoramic cameras, catadioptric cameras, or off-center imaging systems, existing parametric models often require customized derivation for specific hardware, lacking versatility and scalability.

[0004] The 3D Gaussian Splatting (3DGS) method, proposed in recent years, has demonstrated excellent efficiency and rendering quality in novel perspective synthesis. It achieves efficient differentiable rendering by projecting a 2D Gaussian ellipse onto the image plane. However, this method is highly sensitive to the camera projection model: the shape, scale, and orientation of the 2D Gaussian footprint are all determined by the Jacobian matrix, which maps from 3D to 2D. If there is a deviation between the camera model and the actual imaging process, the Jacobian matrix will experience systematic errors, leading to Gaussian ellipse degradation, instability, or scale distortion. This manifests as image blurring, edge tearing, and artifact accumulation, especially noticeable in large perspectives and high-distortion regions.

[0005] Therefore, existing technologies urgently need a camera modeling method that does not rely on a fixed analytical form and has stronger expressive power and adaptability, enabling it to be jointly optimized with 3D Gaussian parameters in the rendering and reconstruction closed loop. By introducing a learnable general camera mapping model, adaptive modeling of complex, unknown, or time-varying distortions can be performed while ensuring differentiability. This not only enables online self-calibration of camera parameters but also stabilizes the projection Jacobian of the Gaussian ellipse, improving the geometric consistency and numerical stability of the 2D footprint, thereby significantly enhancing the overall quality and robustness of new perspective synthesis and 3D reconstruction. Summary of the Invention

[0006] This invention proposes a novel field-of-view synthesis and 3D reconstruction method for cameras with unknown or complex distortions. This method combines a learnable general camera model, Jacobi robust covariance transfer, joint self-calibration training, and an effective field-of-view masking mechanism, which can stably generate high-quality images under conditions of high distortion, ultra-large field of view, and time-varying camera parameters. It is applicable to large field-of-view scenes of non-pinhole cameras such as fisheye, panoramic, and catadioptric imaging.

[0007] This invention provides a 3D Gaussian sputtering rendering and self-calibration method based on a universal camera model, comprising the following steps:

[0008] (1) Obtain multi-view images and obtain three-dimensional Gaussian parameters;

[0009] (2) Construct a learnable general camera mapping model Define a learnable camera mapping function to map a 3D point or unit direction in the camera coordinate system to 2D pixel coordinates;

[0010] (3) Two-dimensional footprint covariance propagation based on Jacobian matrix Calculate the Jacobian matrix J of the camera mapping function. θ Based on this, the three-dimensional Gaussian covariance is mapped to a two-dimensional covariance and then converted into a quadratic parameter; Gaussian sputtering rasterization is performed based on the quadratic parameter to generate a rendered image of the current viewpoint;

[0011] (4) In the differentiable rendering closed loop, the camera model parameters and the three-dimensional Gaussian parameters are jointly optimized. Through the joint optimization process, the camera model is self-calibrated online, and the spatial consistency of the three-dimensional Gaussian parameters is improved simultaneously.

[0012] In step (2) above, the camera mapping function is composed of the superposition of the basic mapping provided by the analytical camera model and the neural residual; wherein, the analytical camera model is at least one of the pinhole camera model, the isometric projection model or the Kannala-Brandt model, and the neural residual is generated by a multilayer perceptron containing Fourier feature inputs.

[0013] In step (2) above, for zoom or time-varying intrinsic parameter cameras, low-dimensional conditional variables are introduced as representation parameters of the camera imaging state and are introduced as input into a learnable camera mapping function so that the camera mapping function changes continuously with time or lens state, and time smoothing regularization is applied to the conditional variables; the conditional variables represent time index, focal length number, or focal length parameters read from the camera EXIF ​​information.

[0014] In step (2) above, for cameras that are not centered or contain reflections, the camera mapping function uses three-dimensional points as independent variables and employs unscented transformation to approximate the two-dimensional covariance, or introduces secondary rays in the rendering.

[0015] In steps (1) and (4) above, the three-dimensional Gaussian parameters include position, scale, rotation, color or spherical harmonic coefficients, and opacity; in step (4) above, the camera parameters include analytical model intrinsic parameters and neural residual network weights.

[0016] In step (4) above, the reconstruction error is used as the objective function, and the effective field of view mask and camera robustness regularization are used for joint optimization.

[0017] In step (4) above, the effective field mask is a ring mask, which is generated based on the principal point and radius threshold, and can be superimposed with threshold segmentation and morphological operations; the camera robustness regularization includes spectral norm constraints and determinant barrier function constraints on the Jacobian matrix to suppress elliptical footprint divergence and mapping folding.

[0018] In step (4) above, joint optimization adopts a phased or annealing strategy, including: first fixing the analytical intrinsic parameters, and then unfreezing the focal length, principal point and neural residual parameters in sequence.

[0019] In steps (3) and (4) above, a lookup table of direction to pixel and Jacobian is established on the unit sphere and interpolation is performed during rendering. During training, the gradient is backpropagated to the interpolation weights.

[0020] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0021] Furthermore, the present invention provides a rendering and self-calibration apparatus, including a processor and a memory, wherein the memory stores a program that, when executed on the processor, implements the above-described method.

[0022] Furthermore, the present invention provides an imaging system including an optical imaging component, an image sensor, and a computing module, wherein the computing module is configured to perform the above-described method.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. Achieve online self-calibration under unknown or time-varying distortion conditions. It eliminates the need for complex offline calibration processes, significantly reducing system deployment and maintenance costs, and is suitable for zoom lenses or imaging systems whose parameters change over time.

[0025] 2. Significantly improves rendering quality in scenes with large field of view and high distortion. Under ultra-large field-of-view conditions such as fisheye and panoramic, it effectively reduces reprojection errors and edge artifacts, and improves image clarity, continuity, and objective indicators such as PSNR and SSIM.

[0026] 3. The unified architecture possesses excellent versatility and scalability. The same framework can be naturally extended to off-center cameras, catadioptric cameras, and a variety of unconventional imaging systems without the need to re-derive the imaging model for specific hardware.

[0027] 4. Balancing accuracy and efficiency, suitable for engineering implementation. By employing small-scale neural networks, covariance propagation, and acceleration strategies, this approach improves modeling accuracy while maintaining rendering and training efficiency, making it suitable for deployment in practical 3D reconstruction and novel perspective synthesis systems. Attached Figure Description

[0028] Figure 1 This is a flowchart of the overall data flow of the system of the present invention.

[0029] Figure 2 This is a schematic diagram illustrating the relationship between the camera baseline and the neural residual structure.

[0030] Figure 3 This is a schematic diagram comparing the morphology and numerical stability of the two-dimensional Gaussian elliptical footprint under conditions of no Jacobian regularization and with Jacobian regularization in an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram of the structure and smooth change of the conditional variable z for zoom / time-varying intrinsic parameters.

[0032] Figure 5 This is a schematic diagram illustrating the approximate covariance of non-central camera extension and unscented transformation.

[0033] Figure 6 This is a comparison chart of the rendering effects of the traditional 3D Gaussian sputtering scheme and the present invention.

[0034] Figure 7 This is a comparison chart of the convergence and tracking effects of the camera focal length parameters over time in simulated zoom and time-varying distortion scenarios according to an embodiment of the present invention.

[0035] Figure 8 This is a block diagram of an embodiment of the device / system. Detailed Implementation

[0036] The following will describe in more detail the novel perspective synthesis and 3D reconstruction method for omnidirectional images proposed in this invention, with reference to the accompanying drawings. The embodiments described below are only for explaining the technical solutions of this invention and are not intended to limit the scope of protection of this invention. Equivalent substitutions or modifications made by those skilled in the art to the embodiments without departing from the spirit and substance of this invention should fall within the scope of protection of this invention.

[0037] This invention provides a method for 3D Gaussian Splatting (3DGS) rendering and camera self-calibration based on a universal camera model. The main steps are as follows:

[0038] Step 1: Obtain input data Collect or acquire a collection of multi-view images of the target scene, as well as initial 3D Gaussian parameters (usually initialized by sparse point clouds), as the data basis for subsequent rendering and optimization.

[0039] Step 2: Construct a learnable, general camera mapping model Define a trainable camera mapping function π θ This function is used to map a 3D point or unit direction vector in the camera coordinate system to 2D pixel coordinates. Specifically, the camera mapping function π... θ A hybrid architecture combining analytical and residual methods is employed. This architecture consists of a fixed analytical camera model (including at least one of pinhole, isometric projection, or Kannala-Brandt models) as a baseline, superimposed with a neural residual network composed of multilayer perceptrons (MLPs). By superimposing the neural residuals, nonlinear distortions that the analytical model cannot fit are compensated for.

[0040] The above methods can significantly improve the camera model's ability to represent unknown distortions, non-ideal lens errors, and complex imaging processes.

[0041] Step 3: Two-dimensional footprint covariance propagation based on Jacobian matrix

[0042] To address the issue of the sensitivity of the 2D Gaussian footprint to the projection model during 3D Gaussian sputtering rendering, the Jacobian matrix of the learnable camera mapping function is calculated.

[0043] Specifically, for the camera mapping function π θ Regarding the 3D point X in the camera coordinate system c Taking the partial derivative, we obtain the Jacobian matrix: J θ = ∂π θ / ∂X c ;

[0044] Furthermore, the covariance Σ of the three-dimensional Gaussian matrix is ​​used to... 3D Transmitted to the two-dimensional image plane, we obtain the two-dimensional covariance matrix: Σ 2D = J θ · Σ 3D · J θ ᵀ;

[0045] The two-dimensional covariance matrix is ​​then converted into a quadratic (conic) parametric form for use by the three-dimensional Gaussian sputtering rasterization renderer, thereby ensuring that the shape, scale, and orientation of the two-dimensional Gaussian elliptical footprint have good geometric consistency and numerical stability.

[0046] Step 4: Constructing a joint self-calibration end-to-end training framework. In the differentiable rendering closed loop, the learnable camera model parameters and 3D Gaussian parameters are jointly inversely optimized. Specifically, image reconstruction error is used as the main optimization objective, while introducing the following constraints and regularization terms: a ring mask based on the effective field of view to shield invalid pixels outside the camera imaging area and avoid abnormal gradient interference; robust regularization terms for the Jacobian matrix of the camera mapping function, including spectral norm constraints and determinant barrier functions, to suppress local projection degradation and numerical instability. Through the joint optimization process, online self-calibration of the camera model is achieved, and the spatial consistency of the 3D Gaussian parameters is improved simultaneously.

[0047] To further accelerate the implementation and deployment of optimization strategies and ensure training and rendering efficiency, a narrow-width neural network structure (such as the lightweight MLP configuration described later) can be preferred when constructing the learnable camera mapping function. In the rendering computation, lookup tables (LUTs) and interpolation strategies are combined to accelerate the process, thereby maintaining low computational overhead while ensuring expressive power and meeting the real-time requirements of the actual system.

[0048] The following are specific examples.

[0049] Example 1: A Learnable Camera Modeling Method Based on Analytical Models and Neural Residuals

[0050] like Figure 1 and Figure 2 As shown, this embodiment provides a learnable camera modeling method based on the combination of analytical camera model and neural residual, which is applicable to imaging systems such as pinhole cameras, isometric projection cameras, and Kannala-Brandt (KB) fisheye cameras.

[0051] The method includes the following steps:

[0052] Step 1-1: Construct the analytical camera mapping baseline model

[0053] First, a traditional analytical camera model is selected as the basic mapping function π0 to map the unit direction vector d in the camera coordinate system to two-dimensional pixel coordinates. The analytical camera model can be any one of the pinhole model, isometric projection model, or Kannala-Brandt model.

[0054] Steps 1-2: Introduce neural network residuals to improve modeling capabilities

[0055] To compensate for the insufficient expressive power of analytical models under conditions of large field of view, high distortion, and actual lens errors, a two-dimensional pixel residual function Δπ, composed of a multilayer perceptron (MLP), is introduced on the basis of the analytical mapping function π0. θ Thus, the final learnable camera mapping function is obtained: π θ (d) = π0(d) + Δπ θ , where Δπ θ It is used to compensate for nonlinear distortions that are difficult to characterize using analytical models.

[0056] Preferably, the input of the MLP is a unit direction vector d, and the output is a two-dimensional pixel offset. The network structure can be adjusted according to computing resources and accuracy requirements. To ensure real-time rendering efficiency, the MLP adopts a lightweight structure, with the number of network layers preferably set to 2 to 5, and the number of hidden units preferably set to 32 to 128. At the same time, to improve the fitting ability of high-frequency distortion details, positional encoding is used for the input vector, where the number of Fourier feature bands is preferably set to 4 to 10.

[0057] Steps 1-3: Calculate the Jacobian matrix based on automatic differentiation

[0058] In the differentiable rendering framework, the camera mapping function π is obtained by automatic differentiation. θ Differentiate to obtain its Jacobian matrix J with respect to the 3D point or direction vector in the camera coordinate system. θ This is used for subsequent covariance propagation calculations of the two-dimensional Gaussian footprint.

[0059] Steps 1-4: Introduce Jacobi robust regularization to prevent footprint degradation.

[0060] To avoid divergence, folding, or numerical instability in the two-dimensional Gaussian footprint during projection, the Jacobian matrix J is... θ Apply robust regularization constraints.

[0061] Specifically, one or a combination of the following regular expressions are introduced:

[0062] Regularization terms based on the spectral norm of the Jacobian matrix are used to limit local magnification effects;

[0063] Based on det(J) θ J θ T A lower bound barrier function is used to prevent degradation of the projected mapping. In the overall loss function, the weight coefficient of the Jacobi robust regularization term is preferably set to 10. -4 Up to 10 -2 This is to achieve the best balance between suppressing rendering artifacts and preserving geometric details.

[0064] like Figure 3 As shown, this intuitively demonstrates the key impact of Jacobian matrix robustness regularization on the morphology of two-dimensional projected footprints during the three-dimensional Gaussian covariance propagation process.

[0065] 1. Left image (without regularity):

[0066] This figure illustrates the Jacobian matrix J of the learnable camera mapping function when the camera is at the edge of a large field of view or experiences severe distortion, without any constraints. θ They often become pathological.

[0067] Visual representation: The two-dimensional Gaussian elliptical footprint is stretched to an extreme degree, presenting a slender "needle-like" or "strip-like" shape.

[0068] Technical consequences: This extreme form means that the condition number of the covariance matrix is ​​too large, causing the eigenvalues ​​in one direction to be much larger than those in the other. In rasterization rendering, this can cause severe numerical instability, leading to gradient explosion or vanishing, and ultimately producing high-frequency noise, edge tearing, or "spiky" artifacts in the rendered image.

[0069] 2. Right diagram (with Jacobi regularity):

[0070] The figure illustrates the effect of introducing Jacobi regularization in this invention.

[0071] Visual representation: The two-dimensional Gaussian elliptical footprint maintains a relatively healthy and full elliptical shape, and the ratio of its major and minor axes is limited to a reasonable range.

[0072] Technical principle:

[0073] The spectral norm constraint limits the maximum singular value of the Jacobian matrix, preventing excessive amplification during the projection process; the determinant barrier prevents the Jacobian determinant from approaching zero, avoiding folding or degradation of the projection mapping.

[0074] Technical effect: By forcibly constraining the properties of the Jacobian matrix, the transferred two-dimensional covariance matrix is ​​ensured to have good numerical properties, thereby eliminating rendering artifacts and guaranteeing the convergence stability of the joint optimization process.

[0075] Steps 1-5: Jointly optimize camera parameters and 3D Gaussian parameters

[0076] In the rendering loop, camera parameters and 3D Gaussian parameters are jointly optimized simultaneously. The 3D Gaussian parameters include position, scale, rotation, color or spherical harmonic coefficients, and opacity; the camera parameters include analytical model intrinsic parameters and neural residual network weights. During the joint optimization process, an effective field-of-view mask is used to shield invalid regions, with its inner and outer radii preferably covering 30% to 95% of the long side of the image. Furthermore, to ensure optimization stability, a phased training annealing strategy is employed, with the number of training annealing rounds (e.g., the transition period from fixed intrinsic parameters to complete unfreezing of optimization) preferably set to 500 to 5000 rounds.

[0077] Example 2: Conditional Modeling Method for Zoom or Time-Varying Intrinsic Cameras

[0078] like Figure 4 As shown in Example 1, this example further proposes a conditional camera modeling method applicable to zoom lenses or time-varying intrinsic parameter cameras.

[0079] Specifically, a low-dimensional condition variable z is introduced. t As a parameter representing the camera's imaging state, and introduced as input into the learnable camera mapping function, π is obtained. θ (d; z t ); where the condition variable z t It can represent a time index, focal length number, or focal length parameter read from the camera's EXIF ​​information.

[0080] Preferably, to ensure the continuity of imaging parameters as they change over time or focal length, the condition variable z is... t A temporal smoothing regularization term is introduced to suppress the impact of parameter abrupt changes on rendering stability. The weight coefficient of this residual smoothing regularization term is preferably set to 10. -4 Up to 10 -3 This ensures the continuity and robustness of camera parameters as they change over time or focal length.

[0081] Using the above method, adaptive modeling of camera intrinsic parameters changing with time or focal length can be achieved without recalibration. Specific verification results are detailed in Example 4 below.

[0082] Example 3: Extended Implementation of a Non-Centered Camera

[0083] like Figure 5 As shown, when the imaging system has a significant non-central optical path structure (such as a catadioptric camera or a complex optical system), this embodiment further extends the above method.

[0084] Specifically, instead of using only the unit direction vector as input, the 3D point Xc in the camera coordinate system is directly used as the camera mapping function π. θ The input is used to adapt to non-central imaging models.

[0085] In the process of estimating a two-dimensional Gaussian footprint, one of the following methods can be used:

[0086] The three-dimensional Gaussian distribution is sampled and propagated using the unscented transform (UT) to approximate the two-dimensional covariance.

[0087] Multiple secondary rays are introduced during the rendering stage to numerically approximate the non-central imaging path.

[0088] The above-described extensions significantly enhance the modeling capabilities and applicability of this invention for non-central camera systems. Detailed verification results are provided in Example 5 below.

[0089] Example 4: Algorithm Performance Verification and Comparison

[0090] To verify the technical effects of the methods described in Embodiments 1 to 3 above, a comparative experimental environment was constructed in this embodiment.

[0091] 1. Experimental Setup

[0092] Dataset: A synthetic dataset containing wide field-of-view fisheye images and real-world fisheye camera video sequences were selected.

[0093] Comparison with benchmark:

[0094] Method A (benchmark): Traditional 3D Gaussian sputtering method, using a fixed pinhole model or a simple radial distortion model calibrated offline by COLMAP.

[0095] Method B (the present invention): The "analytical + neural residual" camera model described in Example 1 is adopted, and Jacobi robust regularization and joint optimization are enabled.

[0096] 2. Verification regarding "significantly improved rendering quality"

[0097] Experimental conditions: New perspective synthesis was performed in a fisheye scene with a field of view (FOV) exceeding 180 degrees.

[0098] Experimental results:

[0099] Qualitative analysis: such as Figure 6As shown, Method A (the baseline) exhibits significant blurring and tearing artifacts in the image edge regions (high distortion areas). This is due to the Gaussian projection Jacobian calculation error caused by the fixed model's inability to accurately fit the nonlinear distortion of the edges. In contrast, the image generated by Method B (the present invention) has clear texture and continuous geometric structure in the edge regions, without obvious elliptic degradation.

[0100] Quantitative indicators: such as Figure 6 As shown, image quality metrics are calculated on the test set. Method B improves the peak signal-to-noise ratio (PSNR) by approximately 9.1 dB and the structural similarity (SSIM) by approximately 0.05 compared to Method A. This indicates that the present invention effectively suppresses reprojection errors through Jacobi covariance propagation and robust regularization.

[0101] 3. Verification of "Online Self-calibration under Unknown or Time-Varying Distortions"

[0102] Experimental conditions: Simulate a zoom lens scenario. In the training data, a disturbance is artificially introduced in which the focal length changes continuously with time t. No real camera intrinsic parameter initial values ​​are provided; only rough initial values ​​are provided.

[0103] Experimental procedure:

[0104] Enable the condition variable z as described in Example 2 t And time smoothing regularization.

[0105] During training, observe the convergence curve of the camera parameters.

[0106] Experimental results:

[0107] like Figure 7 As shown, the online self-calibration performance verification results of the method of this invention for camera intrinsic parameters (taking focal length as an example) are presented under unknown and time-varying distortion conditions. In the figure, the horizontal axis represents time t (number of video frames), and the vertical axis represents the focal length value.

[0108] Realistic Value (Black Dashed Line): The black dashed line in the graph represents the simulated real camera focal length variation curve. This curve exhibits a non-linear trend with sinusoidal fluctuations, designed to simulate the continuous change of focal length over time during shooting with a zoom lens, or time-varying distortion caused by temperature and mechanical vibration.

[0109] Initialization value (gray dotted line): The gray dotted line represents the initial coarse intrinsic parameter value provided during the initial training phase. This value is a fixed constant and deviates significantly from the true value (e.g., the deviation exceeds 200 pixels at t=0), simulating real-world conditions when offline calibration is lacking or when only coarse EXIF ​​information is relied upon.

[0110] The method of this invention (solid line):

[0111] The solid line shows the parameter estimation curve obtained by optimization using the method described in this invention (introducing the condition variable z and time smoothing regularization). Experimental results show that:

[0112] Fast convergence phase: In the early stage of training (approximately the first 10-20 frames), the method of this invention can quickly correct erroneous initialization parameters, driving the focal length estimate to converge rapidly toward the true value, demonstrating the model's powerful online self-calibration capability.

[0113] Dynamic tracking phase: In the subsequent process, despite the continuous and non-linear dynamic changes in the true focal length, the solid line still closely follows the black dashed line, proving that the condition variable z t It effectively captured the camera's instantaneous state.

[0114] Stability: The estimated curve remained smooth without any severe high-frequency jitter, which verifies that the time smoothing regularization term effectively suppressed numerical noise during the optimization process, ensuring the robustness of time-varying parameter estimation and thus avoiding image jitter and geometric distortion in the rendered video.

[0115] In summary, Figure 7 This invention demonstrates that it can achieve high-precision parameter adaptation and reconstruction in imaging systems with zoom or parameters that change over time without relying on complex offline calibration procedures.

[0116] Example 5: Rendering and self-calibration experiment verification based on optimized parameter settings

[0117] To verify the technical effect of the present invention under the above-mentioned preferred parameter settings, this embodiment was tested on a typical large field-of-view fisheye dataset.

[0118] 1. Experimental parameter settings

[0119] Based on the methods described in Examples 1 and 2, a specific experimental model is constructed, and the key parameters are configured as follows:

[0120] Camera network structure: It adopts a 3-layer MLP with 64 hidden units and 6 Fourier feature bands;

[0121] Regularization weights: The Jacobi robust regularization weight is set to 0.005, and the residual smoothing regularization weight is set to 0.0005;

[0122] Training strategy: The effective field-of-view mask radius is set to 90% of the long side of the image, the number of training annealing rounds is set to 2000, and the total number of iterations is 30,000.

[0123] 2. Experimental Results and Analysis

[0124] The method of the present invention (Ours) configured as described above is compared with the traditional 3DGS method (Baseline) that only uses a fixed Kannala-Brandt (KB) model:

[0125] Improved rendering quality: such as Figure 6 As shown, in the high distortion region at the edge of the field of view (FOV > 160°), the traditional method...

[0126] The previous method exhibited obvious blurring and elliptical stretching artifacts; in contrast, the image generated by the method of this invention has clear edge texture and continuous geometric structure. Quantitative indicators show that the peak signal-to-noise ratio (PSNR) of the method of this invention is improved by 9.1 dB compared to the benchmark, and the structural similarity (SSIM) is improved by 0.05.

[0127] Self-calibration convergence: With the initial focal length error set at 15%, after 2000 rounds of annealing training, the relative error of the camera's internal participation in the optimized model converges to within 0.5%, and the Jacobian regularization term effectively controls the condition number of the Jacobian matrix within a safe range, without numerical divergence.

[0128] 3. Conclusion

[0129] Experimental results show that, within the above-mentioned preferred parameter range, the present invention can effectively solve the rendering artifact problem of large field-of-view cameras under unknown distortion, achieve high-precision online self-calibration, and achieve the invention objective of improving rendering quality and stability.

[0130] Furthermore, the present invention also provides a 3D Gaussian sputtering rendering and self-calibration device based on a universal camera model. This device includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, runs as several functional modules. These program modules cooperate with each other to jointly implement the methods described in the above embodiments. Figure 8 As shown, the program module executable by the processor includes:

[0131] 1. Input Module: This module is configured to acquire the initial data required for rendering and reconstruction. Specifically, the input module is responsible for receiving the acquired multi-view image data (as a ground truth reference), initial 3D Gaussian parameters (including position, covariance, color, opacity, etc., typically initialized from sparse point clouds), and initial intrinsic parameter estimates of the camera (if any). The input module then passes this data to the camera modeling module and the rendering pipeline.

[0132] 2. Camera Modeling Module: This module is configured to build and manage learnable, generic camera mapping functions π. θSpecifically, this module defines the mapping rules from 3D points or direction vectors in the camera coordinate system to the 2D image plane. This mapping function is typically constructed by superimposing an analytical camera model (such as the pinhole camera model or the Kannala-Brandt model) with neural network residuals (such as a multilayer perceptron (MLP)). This module receives the input 3D data, calculates the projected coordinates based on the current camera parameters, and passes the mapping relationship to the next level module. For zoom or time-varying scenes, this module is also responsible for handling the input of the condition variable z.

[0133] 3. Jacobian Calculation Module: This module is configured to perform the core calculations for covariance transfer. Specifically, based on automatic differentiation or analytical derivation, this module calculates the Jacobian matrix J of the aforementioned camera mapping function with respect to the 3D points in the camera coordinate system. θ Furthermore, this module utilizes the formula Σ 2D = J θ · Σ 3D · J θ The module maps the three-dimensional Gaussian covariance matrix to a two-dimensional covariance matrix and converts it into a quadratic (Conic) parameter for use by the rasterizer. This module ensures the geometric correctness of the Gaussian elliptical footprint under nonlinear distortion.

[0134] 4. Rendering Module: This module is configured to perform rasterization operations based on Gaussian sputtering. It receives quadratic parameters from the Jacobian calculation module, along with the color, depth, and other attributes of the 3D Gaussian. Using Gaussian sputtering technology, it projects the 3D Gaussian onto 2D pixel coordinates to generate a rendered image for the current viewpoint.

[0135] 5. Output Module: This module is configured to output the final result and calculate the error. On one hand, it outputs the generated rendered image; on the other hand, it compares the rendered image with the real multi-view images provided by the input module, and calculates the reconstruction error (Loss) based on pixel-level differences (such as L1 Loss, D-SSIM Loss). This reconstruction error will be passed as a feedback signal to the optimization control module.

[0136] 6. Optimization Control Module: This module is configured to perform a joint backpropagation and self-calibration process. It receives the reconstruction error calculated by the output module, combines it with a pre-defined effective field-of-view mask and camera robustness regularization terms (such as Jacobian spectral norm constraints), and calculates the gradient using the backpropagation algorithm. Subsequently, this module simultaneously updates the camera parameters (including analytical intrinsic parameters and neural residual network weights) and 3D Gaussian parameters in the camera modeling module, achieving online self-calibration and refined scene representation. The updated parameters are used in the next rendering loop until the model converges.

[0137] Through the coordinated operation of the above modules, this device can achieve high-quality large field-of-view synthesis and 3D reconstruction under unknown or time-varying distortion conditions.

[0138] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A 3D Gaussian sputtering rendering and self-calibration method based on a universal camera model, characterized in that, Includes the following steps: (1) Obtain multi-view images and obtain three-dimensional Gaussian parameters; (2) Construct a learnable general camera mapping model Define a learnable camera mapping function π θ This maps a 3D point or unit direction in the camera coordinate system to a 2D pixel coordinate system. (3) Two-dimensional footprint covariance propagation based on Jacobian matrix Calculate the Jacobian matrix J of the camera mapping function. θ Based on this, the three-dimensional Gaussian covariance is mapped to a two-dimensional covariance and then converted into a quadratic parameter. Gaussian sputtering rasterization is then performed based on the quadratic parameter to generate a rendered image of the current viewpoint. (4) In the differentiable rendering closed loop, the camera model parameters and the three-dimensional Gaussian parameters are jointly optimized. Through the joint optimization process, the camera model is self-calibrated online, and the spatial consistency of the three-dimensional Gaussian parameters is improved simultaneously.

2. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 1, characterized in that, In step (2), the camera mapping function is composed of the superposition of the basic mapping provided by the analytical camera model and the neural residual; wherein, the analytical camera model is at least one of the pinhole camera model, the isometric projection model or the Kannala-Brandt model, and the neural residual is generated by a multilayer perceptron containing Fourier feature inputs.

3. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 2, characterized in that, In step (2), for zoom or time-varying intrinsic parameter cameras, low-dimensional conditional variables are introduced as representation parameters of the camera imaging state and are introduced as input into a learnable camera mapping function so that the camera mapping function changes continuously with time or lens state, and time smoothing regularization is applied to the conditional variables; the conditional variables represent time index, focal length number, or focal length parameters read from the camera EXIF ​​information.

4. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 2, characterized in that, In step (2), for cameras that are not centered or contain reflections, the camera mapping function uses three-dimensional points as independent variables and employs unscented transformation to approximate the two-dimensional covariance, or introduces secondary rays in the rendering.

5. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 2, characterized in that, The three-dimensional Gaussian parameters include position, scale, rotation, color or spherical harmonic coefficients, and opacity; the camera parameters include analytical model intrinsic parameters and neural residual network weights.

6. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 1, characterized in that, In step (4), the reconstruction error is used as the objective function, and joint optimization is performed by combining the effective field-of-view mask and the camera robustness regularization.

7. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 6, characterized in that, In step (4), the effective field mask is a ring mask, which is generated based on the principal point and radius threshold, and can be superimposed with threshold segmentation and morphological operations; the camera robustness regularization includes spectral norm constraints and determinant barrier function constraints on the Jacobian matrix to suppress elliptical footprint divergence and mapping folding.

8. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 1, characterized in that, In step (4), the joint optimization adopts a phased or annealing strategy, including: first fixing the analytical intrinsic parameters, and then unfreezing the focal length, principal point and neural residual parameters in sequence.

9. The three-dimensional Gaussian sputtering rendering and self-calibration method according to claim 1, characterized in that, In steps (3) and (4), a lookup table of orientation to pixel and Jacobian is established on the unit sphere and interpolation is performed during rendering. During training, the gradient is backpropagated to the interpolation weights.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1-9.

Images