Three-dimensional Gaussian splash style migration method and system based on context awareness

Abstract

The invention discloses a three-dimensional Gaussian splash style migration method and system based on context awareness. The method comprises the following steps: uniformly modeling geometric features, multi-view convolution features and style feature statistics of three-dimensional Gaussian splash; a view-dependent modulation mechanism is further designed, so that Gaussian colors changing along with the view angle can be generated; and finally, adaptively representing alignment through the appearance context, so that the style features and the rendering features are consistently matched in appearance, thereby realizing three-dimensional scene style migration with high geometric stability, high style fidelity and zero sample capability. According to the method, zero-sample three-dimensional style migration can be realized without re-optimizing the three-dimensional scene model, the problems of multi-view style instability and detail missing are solved, and an efficient and generalized stylization scheme is provided for three-dimensional scene generation, virtual reality and digital art creation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D reconstruction and deep learning, and in particular to a context-aware 3D Gaussian splatter style transfer method and system. Background Technology

[0002] 3D scene style transfer aims to map a given artistic style onto a 3D scene and maintain style consistency across multiple viewpoints. In recent years, with the development of neural rendering, new rendering methods such as 3D Gaussian Splatting (3DGS) have become important technologies for real-time 3D reconstruction and rendering due to their efficiency and differentiability. However, several challenges remain in performing artistic style transfer on this basis.

[0003] Unlike traditional 2D style transfer, 3D style transfer not only needs to present a specific style of texture, color, and brushstroke in a single viewpoint, but also needs to maintain style consistency across viewpoints to avoid flickering or texture drift during view switching. Existing NeRF-based stylization methods can maintain a certain degree of geometric consistency, but they are computationally intensive, difficult to render in real-time, and have limited style control capabilities. While 3DGS-based stylization methods offer high real-time performance, they often employ global or static modulation mechanisms, ignoring spatial contextual relationships (such as cross-view consistency) and appearance contextual relationships (such as local texture dependencies) in the 3D scene. This makes the generated style prone to color inconsistencies, detail breaks, or a lack of artistic expression in multiple viewpoints.

[0004] Furthermore, most existing methods require re-optimizing the 3D scene or retraining the style transfer network when handling new styles, lacking the ability to quickly adapt to new scenes and styles and failing to achieve zero-shot generalization. On the other hand, existing conditional modulation-based methods struggle to simultaneously model the dependencies between geometry, style, and viewpoint, and cannot precisely control viewpoint-related style details. Summary of the Invention

[0005] The technical problem this invention aims to solve is that in 3D Gaussian splash rendering scenes, existing style transfer methods struggle to simultaneously maintain cross-viewpoint appearance consistency, local structural stability, and style expression refinement. Furthermore, introducing a new style typically requires re-optimizing the entire 3D model, resulting in high computational overhead and poor adaptability. Therefore, this invention proposes a context-aware 3D Gaussian splash style transfer method and system from the perspectives of spatial and appearance consistency. By constructing spatial context representations, viewpoint-related appearance modulation, and cross-domain feature alignment, it achieves zero-sample stylization of 3D scenes without the need for retraining.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a context-aware 3D Gaussian splatter style transfer method, specifically, by inputting a pre-trained 3D Gaussian splatter scene and a style reference image together into a viewfinder. Figure One To train the style transfer model, which includes the representation fusion module and the view-dependent Gaussian adaptive module, an additional appearance context adaptive representation alignment module is introduced to train the style transfer model, and the view is fixed during training. Figure One The parameters of the fusion module are optimized only for the parameters of the view-dependent Gaussian adaptive module. After training is complete, the pre-trained 3D Gaussian splatter scene to be style transferred and the style reference image are input into the trained style transfer model to output a multi-view stylized image.

[0008] The view Figure One The representation fusion module is used to acquire convolutional feature maps of the 3D Gaussian splash scene from multiple training perspectives. Simultaneously, it extracts layered convolutional feature maps from the style reference image as style features. Then, each Gaussian point is projected onto the image plane of different training perspectives, and samples are taken from the convolutional feature map of each training perspective to obtain a set of content features corresponding to the spatial location of that Gaussian point. The content feature sets from all training perspectives are fused to obtain cross-viewpoint consistent features. Subsequently, the attribute information of the Gaussian point is fused onto the cross-viewpoint consistent features to obtain mixed features. Channel mean and variance statistics are calculated for the style features, and the calculated statistics are applied to the mixed features using adaptive instance normalization, thereby constructing a spatial context representation that simultaneously includes structural semantics and style statistics.

[0009] The view-dependent Gaussian adaptive module forms a view-related basic representation that includes geometric, directional, and style context based on the position encoding and viewing direction encoding of Gaussian points. It also introduces spatial context representation in the feature linear modulation of the view-related basic representation to generate view-related colors that change with the viewing direction.

[0010] The appearance context adaptive representation alignment module takes style features and rendering features as input. First, it constructs a bidirectional sparse selection matrix using the cosine similarity of the two features. Then, it introduces a Softmax operation with temperature scaling to obtain a context-dependent bidirectional attention matrix, which is then fused with the sparse selection matrix according to weights to form an appearance context-enhanced cross-domain transfer matrix. After obtaining the target feature matrix based on the cross-domain transfer matrix, a total loss containing multiple losses is jointly applied to the style transfer model to maintain the stability and geometric consistency of style transfer. The parameters of the view-dependent Gaussian adaptive module are updated based on minimizing the total loss. The rendering features are extracted from the rendered image, which is generated based on view-related colors using a 3D Gaussian splash renderer.

[0011] Based on the above scheme, each step can be implemented in the following preferred manner.

[0012] As a preferred option in the first aspect mentioned above, in terms of visual Figure One In the spatial context representation fusion module, the construction process is as follows: First, the 3D Gaussian splash scene is rendered from multiple perspectives to obtain rendered images from multiple training perspectives, which are then used as content images. Subsequently, a pre-trained multi-layer perceptual feature network is used to extract the first... The convolutional feature map of the first layer is used as content feature; for the first layer... Gaussian point Project its spatial position onto the first The training viewpoint position, and in the _ training viewpoint position, and in the _ ... Bilinear sampling is performed on the content features of the layer to form the content feature set under this training perspective. Then, the average of the content feature sets under all training perspectives is calculated to obtain the first layer. The Gaussian point at the th... Cross-view consistency features of layers Next, the attribute parameters of each Gaussian point are processed together with the cross-viewpoint consistent features through a pre-trained lightweight multilayer perceptron to obtain mixed features; then, adaptive instance normalization is performed to normalize the style features along the channel dimension. With variance Injected into the hybrid features, thus obtaining the first Style-enhanced content features that integrate spatial context and style a priori layering Ultimately, the spatial context representation of the Gaussian point is composed of multiple levels of style-enhanced content features.

[0013] As a preferred option in the first aspect mentioned above, the obtained style-enhanced content features Represented as:

[0014] ;

[0015] in, This represents the mean of the cross-view consistent feature of all Gaussian points along the channel dimension; This represents the variance of the cross-view consistent feature of all Gaussian points in the channel dimension.

[0016] As a preferred embodiment of the first aspect mentioned above, in the view-dependent Gaussian adaptive module, a multi-resolution hash grid encoder is used to process the first... The spatial location of the nth Gaussian point is encoded to obtain its positional features. A spherical harmonic encoder is then used to encode the nth Gaussian point. The observation directions from each training viewpoint are encoded to obtain directional features; the positional features are then input into a lightweight density multilayer perceptron, which simultaneously outputs the first... The opacity and geometric feature vectors of each Gaussian point are concatenated with the geometric feature vectors and orientation features to obtain the basic representation related to the view. ; Using scaling factor and bias coefficient After performing feature linear modulation on the view-related basic representation, stylized intermediate features are obtained. Finally, the stylized intermediate features are input into a color mapping network, and the output is the first... The view-related colors of Gaussian points; where the viewing direction is a unit vector pointing from the center of the Gaussian point to the center of the camera; the scaling factor and the bias factor are obtained by mapping the spatial context representation through a lightweight multilayer perceptron.

[0017] As a preferred embodiment of the first aspect, in the appearance context adaptive representation alignment module, the view-related color, the attribute parameters of each Gaussian point, and the opacity are first input into the 3D Gaussian splash renderer to generate a rendered image. Then, a multilayer perceptual feature network is used to extract layered convolutional feature maps from the rendered image as rendering features. These rendering features are then flattened and processed... Normalization yields the rendering feature matrix. After flattening out the stylistic features, Normalization yields the style feature matrix. Calculate the cosine similarity matrix between the rendering feature matrix and the style feature matrix, and perform a Top-k operation on each row of the cosine similarity matrix to construct the first sparse selection matrix from the rendering domain to the style domain. and the second sparse selection matrix from the style domain to the rendering domain The first attention matrix of rendering features to style features is calculated by the Softmax operation with temperature scaling, based on rendering features and style features. The second attention matrix of style features to rendering features The first attention matrix and the first sparse selection matrix are weighted and fused to construct a cross-domain transfer matrix from the rendering domain to the style domain. Based on the cross-domain transfer matrix, calculate the projection matrix that maps the style feature matrix to the subspace containing the rendering feature matrix. Subsequently, the transposed projection matrix is ​​multiplied by the rendered feature matrix to obtain the aligned target feature matrix. The total loss is calculated based on the rendering feature matrix, style feature matrix, first sparse selection matrix, second sparse selection matrix, first attention matrix, second attention matrix, cross-domain transfer matrix, and target feature matrix.

[0018] As a preferred embodiment of the first aspect above, the total loss consists of the directional consistency loss. Cyclic consistency loss Cross-domain attention loss Entropy regularization loss The weighted sum is obtained by:

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] ;

[0024] in, This represents the set of feature layers used for feature extraction in a multilayer perceptron feature network. For feature layer index; These are the weight coefficients for directional consistency loss, cyclic consistency loss, cross-domain attention loss, and entropy regularization loss, respectively. Indicates the number of columns in the rendered feature matrix; The column index representing the rendered feature matrix; The first characteristic of the target feature matrix is... Column vector; Represents the rendering feature matrix of the first Column vector; This represents the vector dot product operation; Representing vectors Norm; Represents the identity matrix; Denotes the Frobenius norm of a matrix; Denotes the square of the Frobenius norm of a matrix; This represents the matrix transpose operation; Representing a matrix Norm; These represent the row and column indices of the attention matrix, respectively. Indicates the first attention matrix at the th Line number The element values ​​of the column; The second attention matrix is ​​represented in the th... Line number The element values ​​of the column.

[0025] As a preferred option in the first aspect mentioned above, the cross-domain transfer matrix The calculation method is as follows:

[0026] ;

[0027] ;

[0028] ;

[0029] in, As a balance factor; The operation means retaining the row with the highest similarity in each row of the cosine similarity matrix. Set one value to zero, and the rest to zero. Indicates the preset hyperparameters; This represents the matrix transpose operation; The operation is used to normalize by row; This refers to the temperature parameter.

[0030] As a preferred embodiment of the first aspect mentioned above, the projection matrix The calculation method is as follows:

[0031] ;

[0032] in, It is a diagonal weight matrix; It is a regularization term; This indicates finding the inverse matrix; It is the identity matrix; This represents the matrix transpose operation.

[0033] As a preferred option in the first aspect mentioned above, the target feature matrix The calculation method is as follows:

[0034] ;

[0035] in, This represents the matrix transpose operation.

[0036] In a second aspect, the present invention provides a context-aware 3D Gaussian splatter style transfer system, which includes a memory and a processor;

[0037] The memory is used to store computer programs;

[0038] The processor is configured to, when executing the computer program, implement the context-aware 3D Gaussian splatter style transfer method as described in any of the first aspects above.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] This invention effectively maintains the geometric and appearance consistency of 3D scenes from different perspectives by constructing consistent spatial context features across multiple viewpoints. By introducing temperature-scaling cross-domain attention and directional consistency constraints, this invention can stably transfer style textures without disrupting scene structure during style transfer. Through a view-dependent Gaussian modulation mechanism, this invention can achieve new style transfer without relying on retraining of the 3D scene, significantly reducing computational costs and improving flexibility in practical applications. This invention can be widely applied to 3D content generation, virtual reality rendering, artistic 3D reconstruction, digital content production, multi-view style-consistent rendering, and other 3D appearance editing related fields or application scenarios. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0042] Figure 2 For the purposes of this invention Figure One This is a schematic diagram of the fusion module.

[0043] Figure 3 This is a schematic diagram of the view-dependent Gaussian adaptive module of the present invention;

[0044] Figure 4 This is a schematic diagram of the appearance context adaptive representation alignment module of the present invention;

[0045] Figure 5 This is a schematic diagram of the training and inference process in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram illustrating the test results of the present invention in different styles and scenarios. Detailed Implementation

[0047] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0048] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0049] To address the problems of poor cross-view consistency, insufficient local appearance representation, and the need for re-optimization for new styles in existing 3D stylization methods, this invention proposes a context-aware 3D Gaussian splash style transfer method. This method is primarily used to achieve real-time stylization rendering of images with arbitrary styles without re-optimizing the 3D Gaussian splash scene. Figure One The representation fusion module utilizes multi-view feature sampling and style statistical normalization to construct the spatial context and style prior representation of Gaussian points. Further, this invention inputs geometric encoding, view encoding, and style enhancement features into the view-dependent Gaussian adaptive module, generating view-related colors with style features through conditional modulation, thus achieving modeling of complex appearance details. Simultaneously, combined with the appearance context adaptive representation alignment module, it introduces bidirectional sparse transitivity based on cosine similarity, cross-domain appearance attention with temperature scaling, bidirectional cyclic consistency constraints, directional consistency constraints, and attention entropy regularization to achieve fine feature alignment based on appearance context.

[0050] like Figure 1 As shown, in a preferred implementation of the present invention, the specific method of the above-mentioned context-aware 3D Gaussian splatter style transfer method is as follows: A pre-trained 3D Gaussian splatter scene and a style reference image are jointly input into a view... Figure One To train the style transfer model, which includes the representation fusion module and the view-dependent Gaussian adaptive module, an additional appearance context adaptive representation alignment module is introduced to train the style transfer model, and the view is fixed during training. Figure One The parameters of the fusion module are optimized only for the parameters of the view-dependent Gaussian adaptive module. After training is complete, the pre-trained 3D Gaussian splatter scene to be style transferred and the style reference image are input into the trained style transfer model to output a multi-view stylized image.

[0051] The view Figure One The representation fusion module is used to acquire convolutional feature maps of the 3D Gaussian splash scene from multiple training perspectives. Simultaneously, it extracts layered convolutional feature maps from the style reference image as style features. Then, each Gaussian point is projected onto the image plane of different training perspectives, and samples are taken from the convolutional feature map of each training perspective to obtain a set of content features corresponding to the spatial location of that Gaussian point. The content feature sets from all training perspectives are fused to obtain cross-viewpoint consistent features. Subsequently, the attribute information of the Gaussian point is fused onto the cross-viewpoint consistent features to obtain mixed features. Channel mean and variance statistics are calculated for the style features, and the calculated statistics are applied to the mixed features using adaptive instance normalization, thereby constructing a spatial context representation that simultaneously includes structural semantics and style statistics.

[0052] The view-dependent Gaussian adaptive module forms a view-related basic representation that includes geometric, directional, and style context based on the position encoding and viewing direction encoding of Gaussian points. It also introduces spatial context representation in the feature linear modulation of the view-related basic representation to generate view-related colors that change with the viewing direction.

[0053] The appearance context adaptive representation alignment module takes style features and rendering features as input. First, it constructs a bidirectional sparse selection matrix using the cosine similarity of the two features. Then, it introduces a Softmax operation with temperature scaling to obtain a context-dependent bidirectional attention matrix, which is then fused with the sparse selection matrix according to weights to form an appearance context-enhanced cross-domain transfer matrix. After obtaining the target feature matrix based on the cross-domain transfer matrix, a total loss containing multiple losses is jointly applied to the style transfer model to maintain the stability and geometric consistency of style transfer. The parameters of the view-dependent Gaussian adaptive module are updated based on minimizing the total loss. The rendering features are extracted from the rendered image, which is generated based on view-related colors using a 3D Gaussian splash renderer.

[0054] The following sections will be viewed separately. Figure One The specific processing flow in the representation fusion module, view dependency Gaussian adaptive module, and appearance context adaptive representation alignment module is explained in detail.

[0055] It should be noted that, in the view of this invention Figure One In the spatial context representation fusion module, the construction process is as follows: First, the 3D Gaussian splash scene is rendered from multiple perspectives to obtain rendered images from multiple training perspectives, which are then used as content images. Subsequently, a pre-trained multi-layer perceptual feature network is used to extract the first... The convolutional feature map of the first layer is used as content feature; for the first layer... Gaussian point Project its spatial position onto the first The training viewpoint position, and in the _ training viewpoint position, and in the _ ... Bilinear sampling is performed on the content features of the layer to form the content feature set under this training perspective. Then, the average of the content feature sets under all training perspectives is calculated to obtain the first layer. The Gaussian point at the th... Cross-view consistency features of layers Next, the attribute parameters of each Gaussian point are processed together with the cross-viewpoint consistent features through a pre-trained lightweight multilayer perceptron to obtain mixed features; then, adaptive instance normalization is performed to normalize the style features along the channel dimension. With variance Injected into the hybrid features, thus obtaining the first Style-enhanced content features that integrate spatial context and style a priori layering Ultimately, the spatial context representation of the Gaussian point is composed of multiple levels of style-enhanced content features, which are used to maintain geometric consistency.

[0056] like Figure 2 As shown, in this embodiment, in order to construct a style-sensitive and cross-viewpoint consistent feature representation, the present invention designs a view... Figure One The representation fusion module first projects each Gaussian point onto different training viewpoints, performs bilinear sampling on the corresponding convolutional feature maps, and aggregates their features. This allows a single Gaussian point to obtain a consistent local content representation across viewpoints. Specifically, a pre-trained 3D Gaussian splash scene needs to be prepared, consisting of a series of 3D Gaussian points, including attributes such as spatial location, scale, color, and covariance matrix. Let... This represents a multilayer perceptron feature network (VGG19 network, corresponding to...) used for feature extraction. Figure 2 Feature layer index in multi-layer VGG feature extraction, This includes the ReLU3_1, ReLU3_2, and ReLU3_3 layers of the VGG network in VGG19, to enhance the contextual representation of Gaussian points from both shallow texture and deep semantic perspectives, respectively. This indicates that the three-dimensional Gaussian splash scene is in the 3D Gaussian splash scene at the 3D Ga The first training perspective, the first The training viewpoint position of the projection of the convolutional feature map of the layer (i.e., the content features mentioned above) is denoted as . The number of training views is denoted as Then the first Gaussian point In the The cross-view consistency feature of a layer is represented as:

[0057]

[0058] in, Indicates the Gaussian point index; This represents the training view index.

[0059] Similarly, the style reference image is input into the multilayer perceptual feature network described above to obtain style features. Since features at different levels capture low-level textures and high-level structures respectively, this helps improve subsequent style alignment and appearance consistency.

[0060] Furthermore, this embodiment also compresses and restores cross-view consistent features through a linear bottleneck structure, thereby obtaining stable cross-view consistent features. After obtaining the cross-view consistent features, the present invention further integrates the original attributes of the three-dimensional Gaussian points to enhance the content expressiveness.

[0061] Subsequently, cross-view consistent features First, compare the color of the Gaussian point itself. ,Location A lightweight multilayer perceptron (MLP) is used as a common input, allowing the inherent properties of Gaussian points and viewpoint information to participate in contextual representation, thereby generating hybrid features. This embodiment employs an adaptive instance normalization (AdaIN) operation based on style feature statistics to normalize style features. Mean along the channel dimension With variance By injecting the aforementioned blending features, Gaussian points acquire a texture appearance consistent with the style reference image while maintaining spatial geometric consistency. This fusion operation effectively avoids appearance deviations caused by a single viewpoint, ensuring that each Gaussian point possesses a stable cross-viewpoint representation, ultimately resulting in style-enhanced content features. Represented as:

[0062]

[0063] in, This represents the mean of the cross-view consistent feature of all Gaussian points along the channel dimension; This represents the variance of the cross-view consistent features across all Gaussian points along the channel dimension. Ultimately, the spatial context representation of this Gaussian point is composed of style-enhanced content features from multiple levels. .

[0064] Furthermore, view Figure One This indicates that the fusion module has been optimized during the pre-training phase and maintains parameter freeze during the actual style transfer process to provide a stable spatial context benchmark.

[0065] It should be noted that in the view-dependent Gaussian adaptive module of this invention, a multi-resolution hash grid encoder is used to process the first... The spatial location of the nth Gaussian point is encoded to obtain its positional features. A spherical harmonic encoder is then used to encode the nth Gaussian point. The observation directions from each training viewpoint are encoded to obtain directional features; the positional features are then input into a lightweight density multilayer perceptron, which simultaneously outputs the first... The opacity and geometric feature vectors of each Gaussian point are concatenated with the geometric feature vectors and orientation features to obtain the basic representation related to the view. ; Using scaling factor and bias coefficient After performing feature linear modulation on the view-related basic representation, stylized intermediate features are obtained. Finally, the stylized intermediate features are input into a color mapping network, and the output is the first... The view-related colors of Gaussian points; where the viewing direction is a unit vector pointing from the center of the Gaussian point to the center of the camera; the scaling factor and the bias factor are obtained by mapping the spatial context representation through a lightweight multilayer perceptron.

[0066] In this embodiment, as Figure 3 As shown, the view-dependent Gaussian adaptive module is used to adaptively modulate the color of the 3D Gaussian splash based on the appearance features of the style image, achieving a fine-grained style expression consistent across viewpoints. In this module, the first step is to obtain the... Spatial location of a Gaussian point and the Observation direction from a training perspective And a multi-resolution hash grid encoder is used to analyze the spatial location. Encoding is performed to enhance the perception of details in three-dimensional space, thereby obtaining positional features. The observation direction is the unit vector pointing from the center of the Gaussian point to the center of the camera.

[0067] Meanwhile, in order to model viewpoint-dependent lighting changes (such as highlights and reflections), a spherical harmonic function (SH) encoder is used to measure the viewing direction. Encoding is performed to improve the modeling accuracy of viewpoint-related features, thereby obtaining directional features. Location features The input is fed into a lightweight density multilayer perceptron (Density MLP). Figure 2 (A density network in Gaussian), this network has two output branches. The first branch outputs the opacity of the Gaussian point, denoted as . The first branch is used for subsequent rendering, while the second branch outputs geometric feature vectors. (Right now Figure 2 The intermediate geometric features in the Gaussian point are used to characterize the implicit geometric structure information of the local area.

[0068] Next, the geometric feature vectors With directional features Concatenation is performed to obtain a basic representation of the view that includes information about the geometric structure and view orientation. :

[0069]

[0070] in, This indicates a splicing operation. To inject style information into the appearance generation process, this module also utilizes visual... Figure One The spatial context representation output by the fusion module is indicated. As a conditional input, specifically, the spatial context representation is mapped to channel-wise scaling factors via a lightweight multilayer perceptron. and bias coefficient Subsequently, these two coefficients were used to refine the underlying representation of the view. Perform Feature Linear Modulation (FiLM) to improve the flexibility and stability of style injection and obtain stylized intermediate features. :

[0071]

[0072] in, This indicates element-wise multiplication. This linear modulation operation allows the appearance features of each Gaussian point to adaptively adjust according to its spatial context and target style without altering the underlying geometry.

[0073] Finally, stylize intermediate features The input is fed into a color mapping network (Color MLP, here still using a lightweight multilayer perceptron), and the output is the view-dependent color of the Gaussian point. This allows the rendering results to maintain geometric consistency while presenting a stable, detailed, and consistent stylistic effect across different viewpoints.

[0074] Furthermore, only the view-dependent Gaussian adaptation module needs optimization, while the other modules remain frozen to ensure the stability of the style transfer process and avoid damaging the geometry of the 3D scene. In other words, the multi-resolution hash grid encoder and spherical harmonic encoder within the view-dependent Gaussian adaptation module are not updated, while the density multilayer perceptron and color mapping network parameters need to be updated. The implementation of the density multilayer perceptron is existing technology and will not be elaborated further.

[0075] It should be noted that in the appearance context adaptive representation alignment module of this invention, firstly, the view-related color, the attribute parameters of each Gaussian point, and the opacity are input into a 3D Gaussian splash renderer to generate a rendered image. Then, a multilayer perceptual feature network is used to extract layered convolutional feature maps from the rendered image as rendering features. These rendering features are then flattened and processed... Normalization yields the rendering feature matrix. After flattening out the stylistic features, Normalization yields the style feature matrix. Calculate the cosine similarity matrix between the rendering feature matrix and the style feature matrix, and perform a Top-k operation on each row of the cosine similarity matrix to construct the first sparse selection matrix from the rendering domain to the style domain. and the second sparse selection matrix from the style domain to the rendering domain The first attention matrix of rendering features to style features is calculated by the Softmax operation with temperature scaling, based on rendering features and style features. The second attention matrix of style features to rendering features The first attention matrix and the first sparse selection matrix are weighted and fused to construct a cross-domain transfer matrix from the rendering domain to the style domain. Based on the cross-domain transfer matrix, calculate the projection matrix that maps the style feature matrix to the subspace containing the rendering feature matrix. Subsequently, the transposed projection matrix is ​​multiplied by the rendered feature matrix to obtain the aligned target feature matrix. The total loss is calculated based on the rendering feature matrix, style feature matrix, first sparse selection matrix, second sparse selection matrix, first attention matrix, second attention matrix, cross-domain transfer matrix, and target feature matrix.

[0076] Furthermore, the total loss consists of the directional consistency loss. Cyclic consistency loss Cross-domain attention loss Entropy regularization loss The weighted sum is obtained by:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] in, This represents the set of feature layers used for feature extraction in a multilayer perceptron feature network. For feature layer index; These are the weight coefficients for directional consistency loss, cyclic consistency loss, cross-domain attention loss, and entropy regularization loss, respectively. This indicates the number of columns in the rendering feature matrix (i.e., the total number of spatial pixels in the rendering features). The column index representing the rendered feature matrix; The first characteristic of the target feature matrix is... Column vector; Represents the rendering feature matrix of the first Column vector; This represents the vector dot product operation; Representing vectors Norm; Represents the identity matrix; Denotes the Frobenius norm of a matrix; Denotes the square of the Frobenius norm of a matrix; This represents the matrix transpose operation; Representing a matrix Norm (i.e., the sum of the absolute values ​​of the matrix elements); These represent the row and column indices of the attention matrix, respectively. Indicates the first attention matrix at the th Line number The element values ​​of the column; The second attention matrix is ​​represented in the th... Line number The element values ​​of the column.

[0083] In this embodiment, as Figure 4 As shown, the specific calculation steps in the appearance context adaptive representation alignment module are as follows:

[0084] 1) Let the feature map extracted from the rendered image at the current viewpoint (i.e., the rendering features mentioned above) be... The feature map extracted from the style reference image (i.e., the style features mentioned above) is Flatten the two feature maps and perform... Normalization yields the rendering feature matrix. and style feature matrix .

[0085] 2) To establish robust feature correspondences, this embodiment also calculates the cosine similarity matrix between normalized features and constructs a bidirectional sparse selection matrix based on this matrix, including a first sparse selection matrix from the rendering domain to the style domain. and the second sparse selection matrix from the style domain to the rendering domain. This is used to capture the local appearance relationship between the style domain and the render domain.

[0086]

[0087]

[0088] in, The operation means retaining the row with the highest similarity in each row of the cosine similarity matrix. One value is set to zero, and the rest are set to zero, thereby filtering out noisy matching and establishing compact local appearance associations. This represents the preset hyperparameters.

[0089] 3) To capture more nuanced contextual dependencies, this embodiment introduces a Softmax operation with temperature scaling to calculate a bidirectional attention matrix, including a first attention matrix of rendering features to style features. The second attention matrix of style features to rendering features :

[0090]

[0091]

[0092] in, For temperature parameters; The operation is used to normalize by row, so that each feature point forms a probabilistic distribution in the contralateral domain. Through and Two types of dot products eliminate attention bias caused by differences in semantic scale, improving the robustness of cross-domain matching.

[0093] 4) To combine the stability of sparse matching with the context-awareness of attention, a weighted fusion approach is used to construct a cross-domain transfer matrix from the rendering domain to the style domain. :

[0094]

[0095] in, This is the balance factor. The cross-domain transfer matrix... It retains the skeleton of hard matching while incorporating the details of soft attention, so that style textures can be stably mapped to the corresponding structure of the rendering domain based on local appearance relevance.

[0096] 5) Based on cross-domain transfer matrix Calculate the style feature matrix Mapping to rendering feature matrix Projection matrix of the subspace This embodiment uses regularized least squares solution to solve for the projection matrix to ensure the smoothness of the mapping:

[0097]

[0098] in, It is a diagonal weight matrix; It is a regularization term; This indicates finding the inverse matrix.

[0099] Subsequently, the aligned target feature matrix is ​​obtained through this projection matrix. This feature matrix represents the ideal stylized feature representation while preserving the geometry of the rendered image:

[0100]

[0101] To drive model optimization, this module constructs a total loss consisting of the following four parts. The specific definitions of each item are as follows: Directional consistency loss Used to force the stylized feature directions to remain consistent with the original rendered feature directions, preventing the geometry from being destroyed; Cyclic consistency loss. Used to ensure the sparse selection matrix is ​​invertible between two domains and to constrain feature reconstruction error; cross-domain attention loss. This is used to constrain the product of the bidirectional attention matrices to be close to the identity matrix, improving the reliability of soft matching; entropy regularization loss. To prevent the attention distribution from becoming too sharp or collapsing, and to encourage the network to explore diverse style regions. The calculation methods for the above-mentioned losses have been described previously and will not be repeated here.

[0102] By minimizing the total loss mentioned above, the model can guide the view-dependent Gaussian adaptive module to generate stylized results that conform to both the target style texture and strictly adhere to the 3D geometry.

[0103] Additionally, it should be noted that during the inference phase after the model is trained, [the following is considered]: Figure One The representation fusion module generates a spatial context representation based on the 3D Gaussian splatter scene and style reference image. The view-dependent Gaussian adaptive module generates view-dependent colors that change with the viewing direction based on the spatial context representation. Then, the view-dependent colors are written back to the corresponding Gaussian points and rendered by the 3D Gaussian splatter renderer. This renderer takes the view-dependent colors and the opacity of each Gaussian point as input, while keeping the geometric parameters such as the position, scale, and covariance of the Gaussian points unchanged. It outputs multi-view stylization results in real time to ensure visual consistency and interactivity. Ultimately, it achieves 3D scene stylization that maintains the geometric structure, expresses the style naturally, and is consistent across viewpoints.

[0104] In this invention, the view Figure One The representation fusion module ensures spatial structural consistency, the view-dependent Gaussian adaptive module enables style-based controllable color generation, and the appearance context adaptive representation alignment module handles fine-grained appearance alignment and consistency constraints. These three modules work synergistically to provide a complete 3D Gaussian splatter style transfer method that adapts to new styles without requiring retraining of the 3D scene. During the testing phase, this method does not require re-optimization of the 3D Gaussian splatter scene or feature extraction module; it only needs a new style image as input to generate stylized results through a single forward inference, thus achieving zero-shot style transfer. It should be understood that each module proposed in this invention has high portability and can be applied to other style transfer tasks based on 3D Gaussian splatter representation or other 3D differentiable rendering frameworks.

[0105] To better demonstrate the specific implementation and technical effects of the present invention, the context-aware 3D Gaussian splatter style transfer method shown in the above steps of the preferred implementation is applied to a specific example below.

[0106] Example

[0107] The specific implementation process of the context-aware 3D Gaussian splatter style transfer method used in this embodiment is as described above and will not be repeated here.

[0108] The overall process of the 3D Gaussian splatter style transfer method is as follows: Figure 5 As shown, it can be divided into three stages: data preprocessing, model training, and stylized inference.

[0109] In the data preprocessing stage, this embodiment uses Gaussian point clouds generated from the reconstruction of real 3D scenes as input and renders content images from multiple perspectives for each scene, serving as the source of multi-view content features. Simultaneously, the content images undergo scaling and normalization, and a multilayer perceptron is used to extract feature representations at specified levels for subsequent style statistical injection and appearance alignment. To improve the model's generalization ability, data augmentation methods such as random cropping, color perturbation, or slight geometric transformations can be applied to the content images.

[0110] During the model training phase, pre-generated multi-view content images are first input into the view in batches. Figure One The representation fusion module samples the projection positions of the same Gaussian point in different viewpoints and fuses them across viewpoints to obtain cross-viewpoint consistent features. Simultaneously, it extracts feature statistics from the corresponding layers of the style image to form style-enhanced content features with style priors. These style-enhanced content features from multiple layers constitute the spatial context representation of the Gaussian point. Subsequently, the geometric and directional encodings of the Gaussian point, along with the aforementioned spatial context representation, are input to the view-dependent Gaussian adaptive module, which generates viewpoint-related Gaussian colors through conditional modulation. Afterward, the multi-layer features of the rendered image and the style reference image are input to the appearance context adaptive representation alignment module to complete cross-domain alignment and structural consistency constraints at the appearance level. During training, model parameters are optimized using directional consistency loss, cyclic consistency loss, cross-domain attention loss, and entropy regularization loss until the set number of iterations is reached, completing model training.

[0111] In the image stylization inference stage, the new style image is input into the trained model. There is no need to re-optimize the 3D Gaussian splash parameters. Stylized multi-view rendering results can be generated through only one forward inference.

[0112] In this embodiment, to quantitatively evaluate the effectiveness of the proposed 3D Gaussian splatter style transfer method, the stylization results of the test scene under different methods are objectively measured, and multiple evaluation metrics are introduced. During the testing process, both the reference image and the style image are from the experimental dataset (the scene dataset uses Tanks & Temples and LLFF, and the style dataset uses WikiArt and ARF datasets), and the model does not need to be retrained for the new style. In this embodiment, the test visualization results are as follows: Figure 6 As shown in Table 1, the test data results are as follows:

[0113] Table 1. Test Data Results

[0114]

[0115] Depend on Figure 6 As shown in Table 1, the method of this invention can produce style transfer results with stable appearance and complete structure under different scenes and viewpoints. In multi-view rendering, the image generated by this invention significantly outperforms the comparison method in terms of texture detail and style consistency, with lower cross-view LPIPS and RMSE indices, indicating that the stylized image maintains good appearance consistency across different viewpoints. Simultaneously, the ArtFID score of this invention is significantly reduced, indicating that the generated result is closer to the reference style image in terms of overall artistic style, proving that this invention can significantly improve the performance of 3D scene stylization in terms of appearance consistency, structural preservation, and style realism.

[0116] In summary, this embodiment demonstrates the style transfer effect of the present invention in a real 3D scene. Based on the view... Figure One By incorporating cross-view spatial consistency provided by the fusion module, stylized color expression generated by the view-dependent Gaussian adaptive module, and structural stability maintained by the appearance context adaptive representation alignment module, this invention enables fast, stable, and zero-sample style transfer of 3D scenes, making it applicable to various fields such as virtual reality content generation and digital art creation.

[0117] It should also be noted that the context-aware 3D Gaussian splatter style transfer method in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a context-aware 3D Gaussian splatter style transfer system corresponding to the context-aware 3D Gaussian splatter style transfer method provided in the above embodiments, which includes a memory and a processor;

[0118] The memory is used to store computer programs;

[0119] The processor is configured to implement the context-aware 3D Gaussian splatter style transfer method in the above embodiments when executing the computer program.

[0120] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0121] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0122] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0123] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A context-aware 3D Gaussian splatter style transfer method, characterized in that, The specific approach is as follows: a pre-trained 3D Gaussian splatter scene and a style reference image are input into a style transfer model that includes a view-consistent representation fusion module and a view-dependent Gaussian adaptive module. An additional appearance context adaptive representation alignment module is introduced to train the style transfer model. During training, the parameters of the view-consistent representation fusion module are fixed, and only the parameters of the view-dependent Gaussian adaptive module are optimized. After training is complete, the pre-trained 3D Gaussian splatter scene and the style reference image to be style transferred are input into the trained style transfer model, and a multi-view stylized image is output. The view-consistent representation fusion module is used to obtain convolutional feature maps of the 3D Gaussian splash scene under multiple training views. At the same time, it extracts layered convolutional feature maps from the style reference image as style features. Then, it projects each Gaussian point onto the image plane of different training views and samples the convolutional feature map of each training view to obtain the content feature set corresponding to the spatial position of the Gaussian point. The content feature sets under all training views are fused to obtain cross-view consistent features. Subsequently, the attribute information of Gaussian points is fused into the cross-view consistent features to obtain mixed features. The channel mean and variance statistics are calculated for the style features. The calculated statistics are applied to the mixed features in an adaptive instance normalization manner, thereby constructing a spatial context representation that simultaneously contains structural semantics and style statistics. The view-dependent Gaussian adaptive module forms a view-related basic representation that includes geometric, directional, and style context based on the position encoding and viewing direction encoding of Gaussian points. It also introduces spatial context representation in the feature linear modulation of the view-related basic representation to generate view-related colors that change with the viewing direction. The appearance context adaptive representation alignment module takes style features and rendering features as input, and first constructs a bidirectional sparse selection matrix based on the cosine similarity of the two features. Subsequently, a Softmax operation with temperature scaling is introduced to obtain a context-dependent bidirectional attention matrix, which is then fused with the sparse selection matrix according to weights to form a cross-domain transfer matrix with enhanced appearance context. After obtaining the target feature matrix based on the cross-domain transfer matrix, a total loss containing multiple losses is jointly applied to the style transfer model to maintain the stability and geometric consistency of style transfer. The parameters of the view-dependent Gaussian adaptive module are updated based on minimizing the total loss. The rendering features are extracted from the rendered image, which is generated based on view-dependent colors using a 3D Gaussian splash renderer.

2. The context-aware 3D Gaussian splatter style transfer method as described in claim 1, characterized in that, In the view-consistent representation fusion module, the construction process of the spatial context representation is as follows: First, the 3D Gaussian splash scene is rendered from multiple perspectives to obtain rendered images from multiple training perspectives, which are then used as content images. Subsequently, a pre-trained multilayer perceptual feature network is used to extract the first... The convolutional feature maps of the layers are used as content features; Regarding the first Gaussian point Project its spatial position onto the first The training viewpoint position, and in the _ training viewpoint position, and in the _ ... Bilinear sampling is performed on the content features of the layer to form the content feature set under this training perspective. Then, the average of the content feature sets under all training perspectives is calculated to obtain the first layer. The Gaussian point at the th... Cross-view consistency features of layers Next, the attribute parameters of each Gaussian point are processed together with the cross-view consistent features through a pre-trained lightweight multilayer perceptron to obtain the hybrid features. Then perform adaptive instance normalization to calculate the mean of style features across the channel dimension. With variance Injected into the hybrid features, thus obtaining the first Style-enhanced content features that integrate spatial context and style a priori layering Ultimately, the spatial context representation of the Gaussian point is composed of multiple levels of style-enhanced content features.

3. The context-aware 3D Gaussian splatter style transfer method as described in claim 2, characterized in that, Acquired style-enhanced content features Represented as: ； in, This represents the mean of the cross-view consistent feature of all Gaussian points along the channel dimension; This represents the variance of the cross-view consistent feature of all Gaussian points in the channel dimension.

4. The context-aware 3D Gaussian splatter style transfer method as described in claim 1, characterized in that, In the view-dependent Gaussian adaptive module, a multi-resolution hash grid encoder is used to process the first... The spatial location of the nth Gaussian point is encoded to obtain its positional features. A spherical harmonic encoder is then used to encode the nth Gaussian point. The observation directions from each training viewpoint are encoded to obtain directional features; the positional features are then input into a lightweight density multilayer perceptron, which simultaneously outputs the first... The opacity and geometric feature vectors of each Gaussian point are concatenated with the geometric feature vectors and orientation features to obtain the basic representation related to the view. ; Using scaling factor and bias coefficient After performing feature linear modulation on the view-related basic representation, stylized intermediate features are obtained. Finally, the stylized intermediate features are input into a color mapping network, and the output is the first... The view-related colors of Gaussian points; where the viewing direction is a unit vector pointing from the center of the Gaussian point to the center of the camera; the scaling factor and the bias factor are obtained by mapping the spatial context representation through a lightweight multilayer perceptron.

5. The context-aware 3D Gaussian splatter style transfer method as described in claim 1, characterized in that, In the appearance context adaptive representation alignment module, the view-related color, attribute parameters of each Gaussian point, and opacity are first input into the 3D Gaussian splash renderer to generate a rendered image. A multilayer perceptual feature network is then used to extract layered convolutional feature maps from the rendered image as rendering features. These rendering features are then flattened and processed... Normalization yields the rendering feature matrix. After flattening out the stylistic features, Normalization yields the style feature matrix. ; Calculate the cosine similarity matrix between the rendering feature matrix and the style feature matrix, and perform a Top-k operation on each row of the cosine similarity matrix to construct the first sparse selection matrix from the rendering domain to the style domain. and the second sparse selection matrix from the style domain to the rendering domain The first attention matrix of rendering features to style features is calculated by the Softmax operation with temperature scaling, based on rendering features and style features. The second attention matrix of style features to rendering features The first attention matrix and the first sparse selection matrix are weighted and fused to construct a cross-domain transfer matrix from the rendering domain to the style domain. ; Based on the cross-domain transfer matrix, calculate the projection matrix that maps the style feature matrix to the subspace containing the rendering feature matrix. Subsequently, the transposed projection matrix is ​​multiplied by the rendered feature matrix to obtain the aligned target feature matrix. ; The total loss is calculated based on the rendering feature matrix, style feature matrix, first sparse selection matrix, second sparse selection matrix, first attention matrix, second attention matrix, cross-domain transfer matrix, and target feature matrix.

6. The context-aware 3D Gaussian splatter style transfer method as described in claim 5, characterized in that, The total loss consists of the directional consistency loss. Cyclic consistency loss Cross-domain attention loss Entropy regularization loss The weighted sum is obtained by: ； ； ； ； ； in, This represents the set of feature layers used for feature extraction in a multilayer perceptron feature network. For feature layer index; These are the weight coefficients for directional consistency loss, cyclic consistency loss, cross-domain attention loss, and entropy regularization loss, respectively. Indicates the number of columns in the rendered feature matrix; The column index representing the rendered feature matrix; The first characteristic of the target feature matrix is... Column vector; Represents the rendering feature matrix of the first Column vector; This represents the vector dot product operation; Representing vectors Norm; Represents the identity matrix; Denotes the Frobenius norm of a matrix; Denotes the square of the Frobenius norm of a matrix; This represents the matrix transpose operation; Representing a matrix Norm; These represent the row and column indices of the attention matrix, respectively. Indicates the first attention matrix at the th Line number The element values ​​of the column; The second attention matrix is ​​represented in the th... Line number The element values ​​of the column.

7. The context-aware 3D Gaussian splatter style transfer method as described in claim 5, characterized in that, Cross-domain transfer matrix The calculation method is as follows: ； ； ； in, As a balance factor; The operation means retaining the row with the highest similarity in each row of the cosine similarity matrix. Set one value to zero, and the rest to zero. Indicates the preset hyperparameters; This represents the matrix transpose operation; The operation is used to normalize by row; This refers to the temperature parameter.

8. The context-aware 3D Gaussian splatter style transfer method as described in claim 5, characterized in that, Projection matrix The calculation method is as follows: ； in, It is a diagonal weight matrix; It is a regularization term; This indicates finding the inverse matrix; It is the identity matrix; This represents the matrix transpose operation.

9. The context-aware 3D Gaussian splatter style transfer method according to claim 5, characterized in that, Target feature matrix The calculation method is as follows: ； in, This represents the matrix transpose operation.

10. A context-aware 3D Gaussian splatter style transfer system, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the context-aware 3D Gaussian splatter style transfer method as described in any one of claims 1 to 9.

Images