Robustness enhancement method and system for three-dimensional gaussian splatter appearance based on generative latent optimization

Abstract

The application provides a three-dimensional Gaussian splash appearance robustness enhancement method and system based on generative latent optimization, assigns an independent image appearance latent vector for each image in a training image set for encoding global appearance attributes of the image, assigns an independent Gaussian appearance latent vector for each Gaussian particle in a three-dimensional scene for encoding adjustable appearance features of the particle, generates a color adjustment amount and an opacity adjustment amount for the Gaussian particle based on the image appearance latent vector corresponding to a target image and the Gaussian appearance latent vector of the target Gaussian particle through a transformation network, modifies rendering attributes of the target Gaussian particle according to the color adjustment amount and the opacity adjustment amount, and inputs the adjusted Gaussian particle into a rasterization renderer to generate a target view image. The application enables precise appearance adaptation of microscopic regions such as local shadow boundaries and highlight reflection regions, and eliminates artifacts caused by global parameter overfitting from a mechanism.

Description

Technical Field

[0001] This invention belongs to the field of 3D scene reconstruction and new perspective synthesis technology, and particularly relates to a method and system for enhancing the robustness of 3D Gaussian splash appearance based on generative latent optimization. Background Technology

[0002] 3D Gaussian splashing technology uses explicitly parameterized 3D Gaussian particles to represent the scene and achieves real-time novel perspective synthesis through rasterization rendering. However, in existing methods, the color and opacity parameters of each Gaussian particle are fixed values, making it difficult to adapt to significant appearance differences (such as lighting variations, exposure deviations, and transient occlusions) between real-world images. When training images have such differences, the model forces the Gaussian particle colors to overfit to specific images, resulting in color distortion, structural artifacts, and abnormal transparency in cross-view rendering.

[0003] Current improvements such as SWAG and SpotLessSplats attempt to introduce image-level appearance latent vectors to mitigate appearance differences through global adjustments. However, such methods have a fundamental flaw:

[0004] Modeling granularity is too coarse: Image-level latent vectors cannot distinguish the appearance characteristics of local areas of the scene (such as shadow edges and reflective surfaces), resulting in loss of detail.

[0005] Implicit queries are inefficient: relying on neural networks to implicitly infer appearance from spatial coordinates increases computational overhead and reduces real-time performance;

[0006] Lack of consistency constraints: The absence of an appearance correlation mechanism between adjacent Gaussian particles leads to color abruptness in the rendering results. These shortcomings severely limit the application of 3D reconstruction technology in dynamic scenes such as autonomous driving and virtual reality. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, the present invention aims to provide a method and system for enhancing the robustness of 3D Gaussian splash appearance based on generative latent optimization. This method establishes a dual latent vector fine-grained control architecture and a spatially driven regularization mechanism. By assigning image-level appearance latent vectors to each image and Gaussian-level appearance latent vectors to each Gaussian particle, and designing a lightweight transformation network to generate particle-level adjustment values, it achieves accurate modeling of appearance differences. Simultaneously, by applying similarity constraints based on particle spatial distribution, it ensures rendering consistency. This addresses the problems of traditional technologies, such as coarse granularity in appearance modeling, weak adjustment capabilities, reliance on implicit feature queries, lack of controllability and consistency, and difficulty in accurately representing appearance differences between views while maintaining rendering efficiency.

[0008] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention provides a method for enhancing the robustness of three-dimensional Gaussian splash appearance based on generative latent optimization, comprising:

[0010] Assign an independent image appearance latent vector to each image in the training image set. , used to encode global appearance attributes of an image;

[0011] Assign an independent Gaussian appearance latent vector to each Gaussian particle in the 3D scene. , used to encode adjustable appearance features of particles;

[0012] Based on the latent vector of the image appearance corresponding to the target image and the latent vector of the Gaussian appearance of the target Gaussian particle, a transformation network is used to generate the color adjustment amount and opacity adjustment amount for the Gaussian particle.

[0013] The rendering properties of the target Gaussian particle are modified according to the color adjustment amount and the opacity adjustment amount, and the adjusted Gaussian particle is input into the rasterization renderer to generate the target view image.

[0014] In some embodiments, the transformation network is a lightweight multilayer perceptron, which includes:

[0015] The input building block is configured to input latent vectors of image appearance. With Gaussian appearance latent vector Perform dimension concatenation to form a joint input vector. h in =[ ; ],in Encodes global appearance attributes of the image. Encoding the inherent material properties of Gaussian particles;

[0016] The hierarchical transformation module is configured to transform the joint input vector starting from a hidden layer. h in Perform linear transformations and nonlinear activations to generate implicit feature vectors, which are then passed to the next hidden layer. The current hidden layer performs linear transformations and nonlinear activations on the input implicit feature vectors, outputs an abstract implicit vector, and passes it to the next hidden layer. This process is repeated until the original adjustment vector is output.

[0017] The adjustment amount decoupling module splits the original adjustment amount vector into a color adjustment amount Δc and an opacity adjustment amount Δo. The color adjustment amount Δc acts on the spherical harmonic function coefficient of the Gaussian particle, and the opacity adjustment amount Δo acts on the transparency value of the Gaussian particle, and is constrained to the interval [-1,1] by a set function.

[0018] In some embodiments, the total number of layers of the lightweight multilayer perceptron does not exceed 3, the hidden layer dimension does not exceed 256, and there is no residual connection structure.

[0019] In some embodiments, a regularization step is also included:

[0020] For the Gaussian appearance latent vector Apply spatial smoothing constraints to ensure that the Gaussian appearance latent vectors of spatially adjacent Gaussian particles remain consistent in the latent space.

[0021] In some embodiments, the regularization step includes:

[0022] For each target Gaussian particle, find its K nearest neighbor Gaussian particles based on its three-dimensional spatial position;

[0023] Calculate the similarity distance between the pairwise Gaussian appearance latent vectors of the K nearest neighbor Gaussian particles;

[0024] A regularized loss function is constructed based on the similarity distance to optimize the Gaussian appearance latent vector.

[0025] In some embodiments, the regularization step includes:

[0026] a) Neighborhood construction: For the first i A target Gaussian particle, based on its spatial position μ i Based on the positional relationships with all Gaussian particles, a similarity calculation method is constructed, which uses the 2-norm—Euclidean distance—as shown in the following formula:

[0027]

[0028] The K particles with the smallest distance are selected to form a neighborhood index set. ;

[0029] b) Similarity calculation: For any two Gaussian particles in the neighborhood ( j , l ), calculate its Gaussian appearance latent vector. The weighted cosine similarity loss is calculated using the following formula:

[0030]

[0031] Among them, satisfying and ;

[0032] c) Regularized loss aggregation: Summing the neighborhood losses of target particle i, and constructing a global regularized loss function by traversing all target particles. The calculation formula is as follows:

[0033]

[0034] Where M is a random sample subset of Gaussian particles in the scene.

[0035] In some embodiments, attenuation weight Determined by spatial distance, the calculation formula is as follows:

[0036]

[0037] in The spatial position of the Gaussian particle. This is a configurable attenuation coefficient.

[0038] In some embodiments, a pre-computation optimization mechanism is used when performing attribute adjustment and rendering steps: before rendering begins, the corresponding image appearance latent vector is obtained in advance based on the reference image associated with the target viewpoint and stored in the cache unit.

[0039] The cached image appearance latent vectors and the Gaussian appearance latent vectors of each Gaussian particle in the scene are sequentially assembled into a joint input vector set. Through a single batch forward propagation process, the color and opacity adjustment of all Gaussian particles are calculated synchronously.

[0040] The generated adjustment values ​​are linked to the rendering properties of the corresponding Gaussian particles in real time and immediately sent to the rasterizer to generate the image.

[0041] In some embodiments, dynamic transformation of the global appearance is also included:

[0042] Receive the new reference image identifier corresponding to the target appearance pattern;

[0043] Extract the image appearance latent vectors of new reference images from the pre-built latent vector library and directly overwrite the original vectors in the cache unit;

[0044] Based on the updated image appearance latent vector, the batch forward propagation process is re-executed to generate a set of adjustments under the new appearance pattern;

[0045] After binding the new adjustment to the Gaussian particle property, output a view image with the target appearance through the rasterizer renderer.

[0046] Secondly, the present invention provides a three-dimensional Gaussian splash appearance robustness enhancement system based on generative latent optimization, applying the three-dimensional Gaussian splash appearance robustness enhancement method based on generative latent optimization as described above, including:

[0047] The image-level latent vector assignment module is configured to assign an independent image appearance latent vector to each image in the training image set. , used to encode global appearance attributes of an image;

[0048] The Gaussian-level latent vector assignment module is configured to assign an independent Gaussian appearance latent vector to each Gaussian particle in the 3D scene. , used to encode adjustable appearance features of particles;

[0049] The appearance transformation prediction module is configured to generate color adjustment and opacity adjustment for the Gaussian particle based on the image appearance latent vector corresponding to the target image and the Gaussian appearance latent vector of the target Gaussian particle through a transformation network.

[0050] The attribute adjustment and rendering module is configured to modify the rendering attributes of the target Gaussian particle according to the color adjustment amount and the opacity adjustment amount, and input the adjusted Gaussian particle into the rasterization renderer to generate a target view image.

[0051] Compared with the prior art, the present invention has at least the following beneficial effects:

[0052] This invention employs a triple technical architecture of dual latent vector atomization control, lightweight decoupling transformation, and spatial regularization constraints. It achieves atomized decomposition of appearance control by assigning an independent image-level appearance latent vector to each image and an independent Gaussian-level appearance latent vector to each Gaussian particle. The image latent vector encodes global illumination features, while the Gaussian latent vector represents the inherent material properties of the particles. Both are transformed by a transformation network to generate particle-level adjustment values ​​(Δc, Δo). This architecture overcomes the coarse-grained limitations of existing technologies that rely on image-level latent vector or spatial coordinate queries, enabling precise appearance adaptation for micro-regions such as local shadow boundaries and specular reflection areas, thus fundamentally eliminating artifacts caused by overfitting global parameters.

[0053] A lightweight transform network and pre-computation pipeline are employed, limiting the transform network to a two-layer fully connected structure. The input layer directly concatenates two latent vectors, and the output adjustment is obtained through ReLU activation and linear transformation. This structure avoids redundant parameters and residual connections, reducing computational complexity. During the inference stage, image latent vectors are pre-cached, and Gaussian particle latent vectors are assembled into batch tensors, completing the full-scene adjustment prediction in a single forward propagation.

[0054] To address rendering jumps caused by discontinuities in the appearance of adjacent Gaussian particles, a latent vector similarity optimization based on K-nearest neighbors is proposed. A local neighborhood graph is constructed based on the spatial location of the particles, and a weighted cosine similarity loss is applied to the Gaussian latent vectors within the neighborhood, with the weight decaying exponentially with Euclidean distance. This constraint forces Gaussian particles that are physically close to each other to be continuously distributed in the latent space, effectively suppressing color banding phenomena from outside the domain and improving the visual consistency of complex structures.

[0055] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0056] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0057] Figure 1 This is a simplified flowchart illustrating a method for enhancing the robustness of 3D Gaussian splash appearance based on generative latent optimization in one implementation.

[0058] Figure 2 This is a schematic diagram illustrating the specific process logic of a method for enhancing the robustness of 3D Gaussian splash appearance based on generative latent optimization in one implementation.

[0059] Figure 3 This is a schematic diagram of similarity regularization in a generative latent optimization-based method for enhancing the robustness of 3D Gaussian splash appearance, implemented in one method.

[0060] Figure 4 This is a schematic diagram of the framework of a three-dimensional Gaussian splash appearance robustness enhancement system based on generative latent optimization in one implementation method.

[0061] Figure 5 This is a visualization of the results in one embodiment.

[0062] Figure 6 This is a schematic diagram illustrating the rendering of two corresponding states from a new perspective based on latent vectors provided by a reference image in one embodiment. Detailed Implementation

[0063] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0065] In the description of this invention, when a specific device is described as being located between a first device and a second device, an intermediary device may or may not be present between the specific device and the first or second device. When a specific device is described as being connected to other devices, the specific device may be directly connected to the other devices without an intermediary device, or it may not be directly connected to the other devices but may have an intermediary device.

[0066] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0067] The applicant discovered:

[0068] 3D Gaussian Splatting (3DGS) is a recently proposed method for 3D reconstruction and novel perspective image synthesis. It employs an explicitly parameterized 3D Gaussian model to model the scene and uses rasterization rendering to generate high-quality images. In the original 3DGS method, each 3D Gaussian point is bound to only one fixed color and opacity parameter, which are globally optimized during training to minimize reconstruction error. However, this fixed parameter is difficult to fit simultaneously to multiple images with significant appearance differences, leading to obvious overfitting: the color of the Gaussian point is forcibly adjusted to fit the appearance of a specific image, but this results in distortion in other images, causing rendering artifacts, structural distortion, color jumps, and other problems, severely affecting the fidelity and consistency of novel perspective images. Furthermore, lacking a robust appearance modeling mechanism, the 3DGS model is prone to misinterpreting unpredictable appearance changes (such as strong shadows or short-term occlusion) as geometric features, introducing floating Gaussian points or abnormally transparent areas, reducing the accuracy and visual quality of the 3D structure.

[0069] Therefore, existing reconstruction methods based on 3D Gaussian Splatting suffer from coarse-grained appearance modeling, weak adjustment capabilities, reliance on implicit feature queries, and a lack of controllability and consistency when processing real image sets. As a result, they are unable to accurately express the appearance differences between views while maintaining rendering efficiency.

[0070] In light of this, to achieve an appearance modeling mechanism with each Gaussian granularity, improve adjustment accuracy, and adapt to local lighting and color differences; to move away from the NeRF-style implicit query structure and propose a structured, interpretable, and lightweight color adjustment model; to support independent control of color and opacity, avoiding interference and improving stability; to maintain appearance consistency across different viewpoints, avoiding rendering jumps and artifacts; and to balance editability and real-time performance, providing support for practical scene deployment and style control, the applicant proposes the following:

[0071] Reference Figure 1 In a first aspect, this embodiment proposes a method for enhancing the robustness of three-dimensional Gaussian splash appearance based on generative latent optimization, including:

[0072] Assign an independent image appearance latent vector to each image in the training image set. , used to encode global appearance attributes of an image;

[0073] Assign an independent Gaussian appearance latent vector to each Gaussian particle in the 3D scene. , used to encode adjustable appearance features of particles;

[0074] Based on the latent vector of the image appearance corresponding to the target image and the latent vector of the Gaussian appearance of the target Gaussian particle, a transformation network is used to generate the color adjustment amount and opacity adjustment amount for the Gaussian particle.

[0075] The rendering properties of the target Gaussian particles are modified based on the color and opacity adjustments, and the adjusted Gaussian particles are then input into the rasterizer to generate the target view image.

[0076] It should be noted that by introducing an independent appearance modeling mechanism for each Gaussian particle, fine-grained modeling and adjustment of appearance changes between different images are achieved, thereby enhancing robustness to appearance differences between photos while maintaining the model's real-time rendering capabilities and system controllability. Figure 2 , specifically:

[0077] (1) Give each Gaussian the ability to transform to different appearances based on the photo: Assign an independent image appearance latent vector to each photo and each Gaussian. Gaussian appearance latent vector Image appearance latent vector Gaussian appearance latent vectors are used to encode the appearance information of an image. Used to encode the appearance features of Gaussian;

[0078] (2) The transform network is equivalent to a pre-defined lightweight MLP generator, which can efficiently complete the mapping from input to output;

[0079] (3) Predict the appearance adjustment amount based on the latent vector: Using a lightweight MLP generator, the information of two latent vectors is efficiently combined to predict the color adjustment amount and opacity adjustment amount for the Gaussian particle.

[0080] (4) Gaussian appearance adjustment: Using the predicted adjustment amount, the color and opacity of each Gaussian are adjusted to achieve fine appearance transformation at the granularity of a single Gaussian.

[0081] (5) Rasterized output rendering: The transformed Gaussian properties will be used as the input of the rasterizer to obtain the rendered image after appearance transformation, which is the target view image;

[0082] (6) Achieve appearance transformation by changing the reference image: Based on the above process, as long as the latent vector of the reference image is changed, the appearance of the scene can be transformed and a rendering image with an appearance similar to the reference image can be produced.

[0083] Traditional 3D Gaussian splash models store geometric and appearance attributes in a coupled manner, requiring overall re-optimization when lighting conditions change. However, this embodiment creatively distinguishes between two types of latent variables: image-level latent vectors carry the global lighting atmosphere of the entire image (e.g., cool tones on a cloudy day), while Gaussian-level latent vectors record the inherent material properties of particles (e.g., the reflective properties of marble). By establishing a transformation network, the global appearance vector is used to directionally modulate the local material vector, generating physical adjustments to color and opacity. This allows the renderer to change the overall lighting style of the scene simply by adjusting the image latent vectors, while preserving the original geometric structure. Essentially, it constructs a two-layer decoupling system of "global-local," maintaining physical accuracy while supporting flexible appearance transfer.

[0084] As one implementation method, the transformation network is a lightweight multilayer perceptron, which includes:

[0085] The input building block is configured to input latent vectors of image appearance. With Gaussian appearance latent vector Perform dimension concatenation to form a joint input vector. h in =[ ; ],in Encodes global appearance attributes of the image. Encoding the inherent material properties of Gaussian particles;

[0086] The hierarchical transformation module is configured to transform the joint input vector starting from a hidden layer. h in Perform linear transformations and nonlinear activations to generate implicit feature vectors, which are then passed to the next hidden layer. The current hidden layer performs linear transformations and nonlinear activations on the input implicit feature vectors, outputs an abstract implicit vector, and passes it to the next hidden layer. This process is repeated until the original adjustment vector is output.

[0087] The adjustment decoupling module splits the original adjustment vector into color adjustment Δc and opacity adjustment Δo. The color adjustment Δc acts on the spherical harmonic function coefficients of the Gaussian particle, and the opacity adjustment Δo acts on the transparency value of the Gaussian particle, and is constrained to the interval [-1,1] by a set function.

[0088] Preferably, the lightweight multilayer perceptron has a total number of no more than 3 layers, a hidden layer dimension of no more than 256, and no residual connection structure.

[0089] Preferably, when performing a linear transformation and nonlinear activation on the joint input vector hin, the nonlinear activation is ReLU activation.

[0090] Preferably, when constrained to the interval [-1,1] by a set function, the set function is the Sigmoid function, the function expression is 2*sigmoid(x)-1, and the value range of sigmoid(x) is (0, 1).

[0091] It should be noted that directly modifying the spherical harmonic coefficients and opacity can easily produce non-physical rendering results. Therefore, in this implementation, a lightweight multilayer perceptron is used as the precise control hub: the global image vector and the Gaussian material vector are concatenated, and through no more than three nonlinear transformation layers, with no skip connections between layers to prevent overfitting, the correlation between the two latent variables is gradually deconstructed. The final output layer is explicitly split into a color adjustment amount Δc and an opacity adjustment amount Δo—where Δc acts on the spherical harmonic function basis, indirectly controlling the light reflection behavior; Δo is compressed to a reasonable range [-1,1] through an S-curve function to ensure that the material transparency follows the optical laws. The advantage of this design is that the lightweight network structure ensures real-time computational efficiency while precisely separating the control of the two major physical processes of light reflection and medium absorption through decoupling the output.

[0092] Furthermore, while assigning independent latent vectors to each Gaussian can effectively achieve scene appearance transformations, the latent vectors between adjacent Gaussians are often relatively discrete and lack intrinsic correlation. This discretization problem can cause issues during out-of-domain inference; specifically, when using the latent vectors of an image to predict the appearance adjustments for regions not seen during training, artifacts are easily generated. The main reason is that the latent vectors between adjacent Gaussians overfit the training image with excessively large differences, leading to large differences in adjustments during out-of-domain inference and causing color abrupt changes. In other words, unconstrained material latent vectors can easily disrupt scene continuity. For example, adjacent particles on a wall may learn drastically different reflectivities, resulting in mosaic-like artifacts during rendering.

[0093] Therefore, combining Figure 3In some possible embodiments, the appearance consistency within local regions is enhanced by utilizing the K-NN algorithm to select spatially adjacent Gaussians and imposing similarity constraints on their latent vectors. Specifically, this embodiment also includes a regularization step.

[0094] Gaussian appearance latent vector Apply spatial smoothing constraints to ensure that the Gaussian appearance latent vectors of spatially adjacent Gaussian particles remain consistent in the latent space.

[0095] The strategy employed in this embodiment transforms the spatial continuity prior into a differentiable mathematical constraint, forcing neighboring particles in three-dimensional space to maintain similarity in the abstract latent space of appearance, thereby preserving the physical consistency of the object's surface material. This effectively alleviates the latent vector discrepancy problem between adjacent Gaussians, thus reducing color abrupt changes and improving the model's generalization ability.

[0096] Preferably, the regularization step includes:

[0097] For each target Gaussian particle, find its K nearest neighbor Gaussian particles based on its three-dimensional spatial position;

[0098] Calculate the similarity distance between the pairwise Gaussian appearance latent vectors of the K nearest neighbor Gaussian particles;

[0099] A regularized loss function is constructed based on similarity distance to optimize the Gaussian appearance latent vector.

[0100] In this implementation, the abstract constraint is decomposed into an executable three-step algorithm: neighborhood construction → similarity calculation → loss aggregation. First, statically predefined neighborhoods are rejected because they cannot adapt to changes in geometric structure during training. Therefore, dynamic neighborhood construction is used, calculating the Euclidean distance from the target particle to all particles in real time, dynamically selecting the K nearest neighbors, and calculating based on real-time coordinates. This ensures that the neighborhood is updated synchronously with the optimization of Gaussian particle positions, guaranteeing that the constraint always applies to the latest geometric structure. Then, for any pair of particles within the neighborhood (… j,l ), calculate the cosine similarity loss of its latent vectors, and decay the weights. Bound to spatial distance, the exponential decay form is mathematically equivalent to a kernel function, such that:

[0101] When the particle distance approaches 0, w approaches 1, achieving strong constraint;

[0102] As the particle distance approaches infinity, w approaches 0, and the constraint disappears.

[0103] Because two particles can be considered as the same material unit when they are infinitely close; the constraint is lifted when the distance exceeds the material correlation scale. Then, the neighborhood loss is calculated for particles in the randomly sampled subset M to optimize the Gaussian appearance latent vector.

[0104] Specifically, the regularization steps include:

[0105] a) Neighborhood construction: For the first i A target Gaussian particle, based on its spatial position μ i Based on the positional relationships with all Gaussian particles, a similarity calculation method is constructed, which uses the 2-norm—Euclidean distance—as shown in the following formula:

[0106]

[0107] The K particles with the smallest distance are selected to form a neighborhood index set. ;

[0108] b) Similarity calculation: For any two Gaussian particles in the neighborhood ( j , l ), calculate its Gaussian appearance latent vector. The weighted cosine similarity loss is calculated using the following formula:

[0109]

[0110] Among them, satisfying and ;

[0111] c) Regularized loss aggregation: Summing the neighborhood losses of target particle i, and constructing a global regularized loss function by traversing all target particles. The calculation formula is as follows:

[0112]

[0113] Where M is a random sample subset of Gaussian particles in the scene.

[0114] Preferably, attenuation weight Determined by spatial distance, the calculation formula is as follows:

[0115]

[0116] in The spatial position of the Gaussian particle. This is a configurable attenuation coefficient.

[0117] It should be noted that this scheme introduces a spatial smoothing loss function: First, a spatial neighborhood is established for each Gaussian particle based on the three-dimensional Euclidean distance. Then, the weighted similarity of all particles within the neighborhood to the material vector is calculated. The weights decrease exponentially with increasing particle distance—meaning that the constraint strength of adjacent particles is much greater than that of distant particles. This loss function forces neighboring points in three-dimensional space to form a continuous manifold in the latent space, resulting in a smooth transition of material properties like real objects. This scheme transforms the prior continuity of the physical world into mathematical constraints, eliminating local abrupt noise and enhancing cross-view rendering consistency.

[0118] Alternatively, as one implementation, a set of spatial locations and latent vectors corresponding to N Gaussian vectors for a given scene is provided:

[0119]

[0120] This is based on the spatial location of Gaussian. Using Euclidean distance as the metric, the Gaussian spatial distance between the i-th and j-th elements is defined as follows:

[0121]

[0122] Based on this distance, we can obtain the index set of the k nearest neighbor Gaussians of the i-th Gaussian. Based on this set, the cosine distance between each pair of latent vectors of the k Gaussian vectors is then calculated as the loss value:

[0123]

[0124]

[0125] Where M is a subset of Gaussian vectors extracted from the entire Gaussian set for K-NN. Furthermore, considering that the similarity of latent vectors typically weakens with increasing spatial distance, a distance-decreasing weight is added to apply stronger regularization to vectors that are closer in distance:

[0126]

[0127] in It is a hyperparameter that varies with the scale of the scene to ensure a reasonable regularization strength.

[0128] To minimize loss Optimizing the target pair can ultimately achieve a 3D spatial smoothing filter effect, making the latent vectors between adjacent Gaussian pairs... More similar, thus effectively avoiding artifacts during extra-domain inference.

[0129] As one implementation method, a pre-computation optimization mechanism is adopted when performing attribute adjustment and rendering steps: before rendering begins, the corresponding image appearance latent vector is obtained in advance based on the reference image associated with the target viewpoint and stored in the cache unit.

[0130] The cached image appearance latent vectors and the Gaussian appearance latent vectors of each Gaussian particle in the scene are sequentially assembled into a joint input vector set. Through a single batch forward propagation process, the color and opacity adjustment of all Gaussian particles are calculated synchronously.

[0131] The generated adjustment values ​​are linked to the rendering properties of the corresponding Gaussian particles in real time and immediately sent to the rasterizer to generate the image.

[0132] It's important to note that traditional rendering pipelines perform appearance adjustment calculations particle-by-particle, which can lead to severe performance bottlenecks. When a scene contains millions of Gaussian particles, sequentially executing the forward propagation of the transformation network results in a sharp drop in frame rate, failing to meet real-time interactive requirements. This implementation restructures the computational flow—aggregating particle-level calculations, originally scattered throughout the rendering process, into scene-level batch processing. Essentially, it avoids redundant computations through data flow reconstruction: observing that all particles from the same viewpoint share the same latent image appearance vector, this vector can be reused as a global modulation factor. By simply loading it into the cache before rendering begins, the frequent I / O constraints during rendering are eliminated.

[0133] The pre-computation architecture establishes a three-layer pipeline:

[0134] Global Vector Cache: During the rendering initialization phase, the corresponding latent vector of the image appearance is extracted from the trained latent vector library based on the reference image index associated with the target viewpoint. This vector, as the core parameter controlling the overall tone and lighting of the scene, is preloaded into the GPU shared memory to avoid repeated access to video memory in subsequent rendering loops.

[0135] Tensor parallel computation: All Gaussian particle material latent vectors and cached image vectors are stacked in particle index order to construct a giant input matrix (dimension: number of particles × (image vector dimension + material vector dimension)). The transformation network is invoked in a single batch, utilizing the GPU's thousands of CUDA cores to synchronously compute the full-scene adjustments. This process transforms the O(N) complexity of serial computation into a matrix multiplication problem, fully leveraging the hardware's parallel potential.

[0136] Zero-copy attribute binding: The generated adjustment matrix is ​​directly associated with the rendering attribute memory address of each particle through memory mapping. When a particle is scheduled by the rasterizer, its spherical harmonics and transparency values ​​have already been modified in the preprocessing stage. The rendering process is completely consistent with native 3DGS, without any additional branch judgment overhead.

[0137] As one implementation method, dynamic transformation of the global appearance is also included:

[0138] Receive the new reference image identifier corresponding to the target appearance pattern;

[0139] Extract the image appearance latent vectors of new reference images from the pre-built latent vector library and directly overwrite the original vectors in the cache unit;

[0140] Based on the updated image appearance latent vector, the batch forward propagation process is re-executed to generate a set of adjustments under the new appearance pattern;

[0141] After binding the new adjustment to the Gaussian particle property, output a view image with the target appearance through the rasterizer renderer.

[0142] It should be noted that the principle of dynamic appearance transformation in this embodiment lies in the construction of a closed-loop response link:

[0143] Intent Receiving Layer: The user inputs a target appearance identifier, which the system parses into a predefined reference image ID. This design supports multimodal interactions such as semantic tags and thumbnail clicks, allowing art directors to work without needing to access technical parameters.

[0144] Vector permutation layer: Retrieves the image appearance vector corresponding to the target ID from an offline-built latent vector dictionary. This dictionary is generated during the training phase and stores the appearance encoding of all reference images. The permutation operation directly overwrites the original vector in the cache unit; this process involves only a single memory write and takes microseconds.

[0145] Incremental recalculation layer: Since the material vector of the Gaussian particles remains unchanged, the input matrix only needs to be reconstructed with the new image vector, repeating the batch forward propagation process in the previous embodiment. The key innovation lies in the local update mechanism: only the adjustment amount needs to be recalculated, rather than the parameters of the entire scene, reducing the computational cost to one ten-thousandth of that of the original training.

[0146] Seamless rendering layer: New adjustments are bound to particle properties via atomic operations. The rasterizer automatically renders the next frame with the updated parameters, achieving a smooth visual transition without screen tearing or stuttering.

[0147] Based on the pre-computation optimization mechanism and dynamic appearance transformation in the two implementation methods above, a deep technical coupling is formed: pre-computation provides real-time assurance for dynamic switching, and dynamic switching verifies the extended value of pre-computation. Together, they shape the appearance-programmable rendering pipeline—by separating control parameters (image latent vectors) and rendering parameters (particle properties), 3D scenes are given dynamic skinning capabilities similar to CSS style sheets for the first time. When a user replaces the image vectors, they are essentially manipulating the hyperparameters of the rendering behavior of millions of Gaussian particles, without needing to be aware of the complex underlying calculations.

[0148] Reference Figure 4 Secondly, this embodiment provides a robust enhancement system for the appearance of three-dimensional Gaussian splashes based on generative latent optimization, applying the robust enhancement method for the appearance of three-dimensional Gaussian splashes based on generative latent optimization as described in the above embodiment, including:

[0149] The image-level latent vector assignment module is configured to assign an independent image appearance latent vector to each image in the training image set. , used to encode global appearance attributes of an image;

[0150] The Gaussian-level latent vector assignment module is configured to assign an independent Gaussian appearance latent vector to each Gaussian particle in the 3D scene. , used to encode adjustable appearance features of particles;

[0151] The appearance transformation prediction module is configured to generate color adjustment and opacity adjustment for the Gaussian particle based on the image appearance latent vector corresponding to the target image and the Gaussian appearance latent vector of the target Gaussian particle through a transformation network.

[0152] The attribute adjustment and rendering module is configured to modify the rendering attributes of the target Gaussian particles based on the color adjustment amount and the opacity adjustment amount, and input the adjusted Gaussian particles into the raster renderer to generate the target view image.

[0153] In this enhancement system, a dual latent vector modeling structure is constructed using an image-level latent vector allocation module and a Gaussian-level latent vector allocation module. An image-level appearance latent vector is assigned to each image, and a Gaussian-level latent vector is assigned to each Gaussian particle. The two are combined as the input of a lightweight multilayer perceptron (MLP) to achieve fine-grained modeling based on both image and spatial conditions.

[0154] The appearance transformation prediction module adopts a lightweight appearance adjustment network design, constructing a low-complexity lightweight multilayer perceptron (MLP). The input is the concatenation of the image latent vector and the Gaussian latent vector, and the output is the color adjustment amount ∆c and the opacity adjustment amount ∆o of the Gaussian particle under the current image. ∆c and ∆o are used to adjust the Gaussian color and opacity, respectively, and participate in the final image rendering.

[0155] Preferably, it also includes a regularization implementation module for implementing a spatial neighborhood-based appearance similarity regularization mechanism, using a K-nearest neighbor structure to construct local connections between Gaussians, applying similarity constraints to the appearance transformations (∆c, ∆o) of adjacent Gaussians in the same image, guiding their changes to maintain semantic coherence, and reducing local mutations and high-frequency noise.

[0156] The mechanism proposed by the attribute adjustment and rendering module can be directly integrated into the standard rendering pipeline of 3DGS without modifying the rendering pipeline. Appearance adjustments can be cached, and only forward propagation is required in the inference stage, supporting real-time rendering.

[0157] Furthermore, to better illustrate this point, specific implementation examples are provided below:

[0158] In this embodiment, experiments were conducted on four datasets to verify the effectiveness of the invention. The results are shown in Table 1. This invention is significantly superior to the previous method in all aspects.

[0159] Table 1 Experimental Results

[0160]

[0161] Figure 5 The visualization results of this experiment are shown, and the comparison of various processing methods in the figure demonstrates that the rendering results of this invention have fewer artifacts and the colors are closer to the true values.

[0162] Figure 6 illustrates how the present invention can transform the appearance of a rendered image using the latent vectors of two reference images. Specifically, according to... Figure 6 The latent vector provided in the lower right corner renders the corresponding two states from a new perspective.

[0163] In summary, compared with the prior art, the above embodiments have at least the following technical advantages:

[0164] This invention employs Gaussian-level latent vector design for appearance, breaking through the limitation of existing methods that can only perform appearance adjustments at the image level. It achieves more precise local appearance transformation capabilities, and is particularly suitable for processing appearance differences in detailed areas such as the edges of occluded objects, shadows, and light spots.

[0165] By explicitly assigning latent vectors directly to Gaussian, these latent vectors can undertake most of the learning task. Therefore, using a lightweight MLP can complete the mapping from latent vectors to adjustment values, which greatly improves computational efficiency and also enhances the interpretability and debugging convenience of the model.

[0166] The number of latent vectors also increases with the number of Gaussians. Therefore, the model's ability to model a scene also increases to some extent with the scale or complexity of the scene, avoiding the problem of limited model capacity and inability to achieve high-fidelity reconstruction, similar to NeRF.

[0167] The appearance of each Gaussian is entirely determined by its position, as well as the position of the photograph, in the latent space, decoupling it from the 3D spatial position of the Gaussian. On the one hand, this avoids the strategy of querying latent features from 3D spatial position, as used in previous methods (such as SWAG, NeRF-W, etc.), thus improving efficiency; on the other hand, the latent space has a higher dimension, providing more information and giving it greater capabilities for local appearance transformation; an additional advantage is that it facilitates post-editing: the correct colors can still be calculated after the scene has been translated, rotated, or scaled.

[0168] Instead of directly predicting the final color, the adjustment amount Δc of the color can be predicted using the spherical harmonic function employed by 3DGS to model the color related to the view direction. This reduces the learning burden on the appearance transformation model, and the model can be removed during inference after the adjustment amount has been calculated, thus avoiding additional overhead for rendering.

[0169] The opacity adjustment Δo: (i) Predicting the adjustment rather than directly predicting the final opacity is to avoid the opacity reset mechanism proposed by 3DGS failing, because there is no way to easily reset a certain predicted value of the MLP, and this reset mechanism is the key to avoiding getting trapped in local optima. (ii) Taking a value in [0,1] is to ensure that the appearance transformation model is only responsible for handling temporary parts. Persistent parts are handled by the properties of Gaussian itself.

[0170] This invention constructs spatial adjacency relationships between Gaussians using KNN and applies similarity regularization between appearance changes during training to guide the appearance adjustments of adjacent Gaussians to be more coordinated in terms of semantics and distribution, reduce isolated mutations, and improve the continuity, stability, and noise resistance of appearance modeling.

[0171] The method is lightweight in overall design, with low computational overhead during the training phase and the ability to pre-cache latent vectors of images during the inference phase, maintaining a real-time rendering speed close to that of the original 3DGS, making it suitable for large-scale scene deployment and online applications.

[0172] With this appearance transformation model, during training, it will be responsible for transforming the scene to the appearance corresponding to the real value, thereby avoiding overfitting the real value by adding extra Gaussian through densification and effectively eliminating artifacts.

[0173] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method for enhancing the robustness of 3D Gaussian splash appearance based on generative latent optimization, characterized in that, include: Assign an independent image appearance latent vector to each image in the training image set. , used to encode global appearance attributes of an image; Assign an independent Gaussian appearance latent vector to each Gaussian particle in the 3D scene. , used to encode adjustable appearance features of particles; Based on the latent vector of the image appearance corresponding to the target image and the latent vector of the Gaussian appearance of the target Gaussian particle, a transformation network is used to generate the color adjustment amount and opacity adjustment amount for the Gaussian particle. The rendering properties of the target Gaussian particle are modified according to the color adjustment amount and the opacity adjustment amount, and the adjusted Gaussian particle is input into the rasterization renderer to generate the target view image.

2. The method as described in claim 1, characterized in that, The transformation network is a lightweight multilayer perceptron, which includes: The input building block is configured to input latent vectors of image appearance. With Gaussian appearance latent vector Perform dimension concatenation to form a joint input vector. h in =[ ; ],in Encodes global appearance attributes of the image. Encoding the inherent material properties of Gaussian particles; The hierarchical transformation module is configured to transform the joint input vector starting from a hidden layer. h in Perform linear transformations and nonlinear activations to generate implicit feature vectors, which are then passed to the next hidden layer. The current hidden layer performs linear transformations and nonlinear activations on the input implicit feature vectors, outputs an abstract implicit vector, and passes it to the next hidden layer. This process is repeated until the original adjustment vector is output. The adjustment amount decoupling module splits the original adjustment amount vector into a color adjustment amount Δc and an opacity adjustment amount Δo. The color adjustment amount Δc acts on the spherical harmonic function coefficient of the Gaussian particle, and the opacity adjustment amount Δo acts on the transparency value of the Gaussian particle, and is constrained to the interval [-1,1] by a set function.

3. The method as described in claim 2, characterized in that, The lightweight multilayer perceptron has a total of no more than 3 layers, a hidden layer dimension of no more than 256, and no residual connection structure.

4. The method according to any one of claims 1 to 3, characterized in that, It also includes regularization steps: For the Gaussian appearance latent vector Apply spatial smoothing constraints to ensure that the Gaussian appearance latent vectors of spatially adjacent Gaussian particles remain consistent in the latent space.

5. The method as described in claim 4, characterized in that, The regularization step includes: For each target Gaussian particle, find its K nearest neighbor Gaussian particles based on its three-dimensional spatial position; Calculate the similarity distance between the pairwise Gaussian appearance latent vectors of the K nearest neighbor Gaussian particles; A regularized loss function is constructed based on the similarity distance to optimize the Gaussian appearance latent vector.

6. The method as described in claim 5, characterized in that, The regularization step includes: a) Neighborhood construction: For the first i A target Gaussian particle, based on its spatial position μ i Based on the positional relationships with all Gaussian particles, a similarity calculation method is constructed, which uses the 2-norm—Euclidean distance—as shown in the following formula: The K particles with the smallest distance are selected to form a neighborhood index set. ; b) Similarity calculation: For any two Gaussian particles in the neighborhood ( j , l ), calculate its Gaussian appearance latent vector. The weighted cosine similarity loss is calculated using the following formula: Among them, satisfying and ; The attenuation weight is determined by the spatial distance; c) Regularized loss aggregation: Summing the neighborhood losses of target particle i, and constructing a global regularized loss function by traversing all target particles. The calculation formula is as follows: Where M is a random sample subset of Gaussian particles in the scene.

7. The method as described in claim 6, characterized in that, The attenuation weight The calculation formula is as follows: in , The spatial position of the Gaussian particle. This is a configurable attenuation coefficient.

8. The method as described in claim 1, characterized in that, A pre-computation optimization mechanism is used when performing attribute adjustment and rendering steps: Before rendering begins, the latent vector of the corresponding image appearance is obtained in advance based on the reference image associated with the target viewpoint and stored in the cache unit. The cached image appearance latent vectors and the Gaussian appearance latent vectors of each Gaussian particle in the scene are sequentially assembled into a joint input vector set. Through a single batch forward propagation process, the color and opacity adjustment of all Gaussian particles are calculated synchronously. The generated adjustment values ​​are linked to the rendering properties of the corresponding Gaussian particles in real time and immediately sent to the rasterizer to generate the image.

9. The method as described in claim 8, characterized in that, It also includes dynamic transformations of the global appearance: Receive the new reference image identifier corresponding to the target appearance pattern; Extract the image appearance latent vectors of new reference images from the pre-built latent vector library and directly overwrite the original vectors in the cache unit; Based on the updated image appearance latent vector, the batch forward propagation process is re-executed to generate a set of adjustments under the new appearance pattern; After binding the new adjustment to the Gaussian particle property, output a view image with the target appearance through the rasterizer renderer.

10. A three-dimensional Gaussian splash appearance robustness enhancement system based on generative latent optimization, employing the three-dimensional Gaussian splash appearance robustness enhancement method based on generative latent optimization as described in any one of claims 1 to 9, characterized in that, include: The image-level latent vector assignment module is configured to assign an independent image appearance latent vector to each image in the training image set. , used to encode global appearance attributes of an image; The Gaussian-level latent vector assignment module is configured to assign an independent Gaussian appearance latent vector to each Gaussian particle in the 3D scene. , used to encode adjustable appearance features of particles; The appearance transformation prediction module is configured to generate color adjustment and opacity adjustment for the Gaussian particle based on the image appearance latent vector corresponding to the target image and the Gaussian appearance latent vector of the target Gaussian particle through a transformation network. The attribute adjustment and rendering module is configured to modify the rendering attributes of the target Gaussian particle according to the color adjustment amount and the opacity adjustment amount, and input the adjusted Gaussian particle into the rasterization renderer to generate a target view image.

Images