Construction method of controllable three-dimensional scene stylization framework based on three-dimensional gauss

Abstract

The application discloses a kind of based on three-dimensional Gauss's controllable three-dimensional scene stylization framework construction method, this method proposes a mask matching module, a color conversion module, a style transfer loss function based on image feature alignment, a scene geometry and content protection module and a three-dimensional Gauss scene stylization module.Mask matching module is first obtained the mask matching group of same semantic label;Color conversion module aligns the color distribution of content image, style image and three-dimensional Gauss;Style transfer loss function based on image feature alignment is used to realize scene style transfer;Scene geometry and content protection module will maintain scene geometry information and content information in the process of style transfer;Three-dimensional Gauss scene stylization module optimizes three-dimensional Gauss parameter, realizes scene stylization.The framework constructed by the application not only reaches controllable, fine-grained high, but also reaches high-quality three-dimensional scene stylization.

Description

Technical Field

[0001] This invention relates to the field of 3D scene style transfer in computer vision and computer graphics, specifically to a framework construction method for precise and controllable 3D scene style transfer based on 3D Gaussian. Background Technology

[0002] In recent years, driven by the rapid development of fields such as virtual reality, augmented reality, and video games, the demand for 3D stylization technology has increased significantly. 3D stylization allows artists and developers to convert 2D artworks with unique visual styles into 3D models, thus maintaining stylistic consistency while providing users with an immersive 3D experience. The introduction of Neural Radiation Fields (NeRF) has greatly advanced the synthesis of novel views, facilitating the development of 3D stylization. Subsequently, a large number of NeRF-based 3D style transfer techniques have emerged.

[0003] Subsequently, most optimization-based NeRF stylization methods employ Nearest Neighbor Feature Matching (NNFM) loss. The optimization process is as follows: for each feature vector of the rendered image in the feature map, the nearest feature vector in the feature map of the style image is first identified through nearest neighbor feature matching. Then, the distance between the two feature vectors is minimized. This method achieves high-quality stylization results. However, simply minimizing the distance between adjacent features does not consider the global information of the image. This approach may cause the scene to only adopt local style features of the style image, resulting in the loss of some artistic style during transmission.

[0004] Neural Radiation Fields (NeRF) use a Multilayer Perceptron (MLP) to represent the color and volume density of each point in a scene, making it challenging to separate the scene's appearance from its geometry. Scene stylization affects appearance while potentially compromising geometric information. Furthermore, since NeRF is an implicit representation, performing high-precision, fine-grained scene editing is challenging. Recently, a novel 3D representation called 3D Gaussian Splash (3DGS) has been introduced to address various tasks, including novel view composition. 3DGS achieves higher rendering quality in shorter training times and supports real-time rendering via a fast, differentiable rasterizer. Due to the explicit nature of Gaussians, 3DGS is particularly advantageous for scene editing; however, existing methods generally rely on textual cues for scene editing, which poses challenges for stylization. Abstract art styles are difficult to express verbally, and using text-based descriptions often results in low-quality stylization. Summary of the Invention

[0005] The purpose of this invention is to address the following issues: a) traditional 3D scene stylization lacks the ability to precisely control the stylized region; b) traditional 3D scene style transfer methods often fail to faithfully reproduce the style image; c) solving the problems described in a) and b) requires precise matching of scene regions with style features in the style image, which implicit neural radiation field-based style transfer methods struggle to address. Therefore, this invention proposes a method for constructing a controllable 3D scene style transfer framework based on 3D Gaussians. This framework is used to achieve precise and controllable 3D scene stylization. Unlike previous neural radiation field-based methods, the framework constructed in this invention utilizes explicit 3D Gaussians to achieve controllable scene stylization.

[0006] The correspondence between style content and 3D scene plays a crucial role in controllable scene stylization. Therefore, this invention introduces a mask matching module to achieve precise control. Simultaneously, a color conversion module is included to convert the color distribution of the content image and the 3D Gaussian distribution into the color distribution of the corresponding style image region.

[0007] Existing 3D scene style transfer loss functions are mostly based on nearest-neighbor feature matching, which only transfers the local style of the style image. Therefore, a style transfer loss function based on image feature alignment is designed to ensure that the style transfer result is faithful to the complete style image. Furthermore, to preserve the geometric and content information of the original scene, a scene geometry and content preservation module incorporating multiple loss functions and Gaussian regularization terms is implemented. Finally, a 3D Gaussian scene stylization module is set up to optimize the 3D Gaussian parameters.

[0008] The specific technical solution for achieving the objective of this invention is as follows:

[0009] A method for constructing a controllable 3D scene stylization framework based on 3D Gaussian is characterized by the framework being used to achieve precise and controllable 3D scene stylization. The construction method includes the following specific steps:

[0010] Step 1: Configure the mask matching module

[0011] The user inputs prompts for segmenting the content image and style image into the mask matching module. The module calls the SAM segmentation method to obtain 2D semantic segmentation masks for the content image and style image, and selects the correspondence between scene content and style regions. Then, by back-projecting the semantic labels of the semantic segmentation mask onto a 3D Gaussian surface, the semantic label of each 3D Gaussian surface is determined. The content image, style image and 3D Gaussian surface with the same semantic label are matched together to obtain the mask matching relationship.

[0012] Step 2: Configure the color conversion module

[0013] Based on the mask matching relationship obtained in step one, calculate the linear color transformation parameters of the original scene content image and the color distribution of the three-dimensional Gaussian within the scene to be converted into the style image region that matches them, and convert the content image and the color distribution of the three-dimensional Gaussian into the color distribution of the corresponding style image region.

[0014] Step 3: Set the style transfer loss function based on image feature alignment

[0015] Based on the mask matching relationship obtained in step one, calculate the affinity matrix between the rendered feature map and the style feature map; calculate the alignment matrix to align the rendered feature map to the style feature map using the affinity matrix, and then use the alignment matrix to obtain the aligned feature map; set the loss function using the rendered feature map and the aligned feature map.

[0016] Step 4: Configure scene geometry and content protection modules

[0017] Depth loss is calculated by rendering the initial depth map of the pre-optimized 3D Gaussian and the depth map obtained from the Gaussian rendering during the optimization stage. At the same time, regularization terms are set for the Gaussian scale parameter and opacity parameter to protect the scene geometric information; content loss function and total variational loss are set to protect the scene content information.

[0018] Step 5: Configure the 3D Gaussian scene stylization module

[0019] The 3D Gaussian scene stylization module includes optimizing Gaussian parameters using a reconstruction loss function to obtain a consistent expression; optimizing the 3D Gaussian parameters in the scene using multiple loss functions and regularization terms set in steps three and four, thereby achieving controllable 3D scene stylization based on 3D Gaussians.

[0020] Furthermore, in the mask matching module, the user input content image I... c Style Image I s and two prompt phrases (Text) used to segment the content image and style image respectively. c and Text s The module calls the SAM segmentation method to obtain the semantic mask M of the content image. c and style image semantic mask M s At the same time, the input prompt phrase Text will be displayed. c and Text s Hint words with the same index are treated as a pair of matching hint words and their corresponding mask regions are matched to obtain M. c and M s The mask matching relationship; then, M c The semantic labels are back-projected onto the Gaussian to obtain the corresponding semantic labels of the Gaussian. Specifically, the weight of the j-th semantic label of the i-th Gaussian is first calculated:

[0021]

[0022] in, p represents the weight of the j-th semantic label of the i-th Gaussian, and p is the mask M. c pixels on, α i For the weights of alpha mixing, T i The transmittance is the rate at which the semantic label of the back projection reaches the i-th Gaussian.

[0023] Next, by setting a threshold and comparing it with the calculated semantic label weights, the semantic label corresponding to the Gaussian is determined; for the z-th semantic label, its corresponding mask matching group is obtained. in Let z be the mask region in the style image semantic mask where the semantic label is the z-th semantic label. G is the mask region in the content image semantic mask where the semantic label is the z-th semantic label. z Let be the 3D Gaussian vector in the scene whose semantic label is the z-th semantic label.

[0024] Furthermore, the color conversion module aims to convert the color gamut of the scene representation into the color gamut of the style image; specifically, it makes... Let be the set of all pixel colors in the content image region to be recolored, and let The set of all pixel colors for the style image region; by matching the content color set {p c} and style color set {p s The mean E and covariance Cov of the vector are used to calculate the linear color transformation weights and biases:

[0025] p ct =Ap c +b, c ct =Ac+b,

[0026] stE[p ct ]=E[p s ], Cov[p ct ] = Cov[p s ],

[0027] Where c is the original Gaussian color set, p ct and c ct Let A and b represent the content image color set and Gaussian color set after linear color transformation, respectively. Let A be the weight parameter of the linear color transformation and b be the bias parameter of the linear color transformation. The weight and bias parameters of the linear color transformation are obtained by solving the above formula, and color transformation is performed on the content image and the 3D Gaussian color set in the scene.

[0028] Furthermore, the style transfer loss function based on image feature alignment will render image I... r and style image I s The rendering feature maps F are extracted separately using the same VGG-16 network. r and style feature diagram F s The rendered image is obtained by rendering a 3D Gaussian image within the scene using a fast differentiable rasterizer, as shown in the following formula:

[0029]

[0030] Where C represents the color of the pixel being rendered, c i This indicates the color of the i-th Gaussian region used to render the pixel.

[0031] For the z-th semantic label, based on the mask matching group Ω obtained in step one z Extract feature vector sets from the feature maps corresponding to the rendered image and the style image, respectively. and In order to transform the feature vector set and Alignment, calculate the alignment matrix P z To transform the rendering feature vector set Subsequently, normalized cross-correlation between features is used as a similarity measure to construct an affinity matrix to compensate for the lack of semantic information in image features. The affinity matrix of rendering and style features is represented as A. z The specific calculation formula is as follows:

[0032]

[0033] in A represents z The value at position (i, j); express The i-th eigenvector in express The j-th eigenvector; express With feature vector set Each feature vector Calculate the k-nearest neighbor set obtained by normalizing cross-correlation; express With feature vector set Each feature vector The k-nearest neighbor set is obtained by calculating the normalized cross-correlation; s(x, y) represents the normalized cross-correlation of feature vectors x and y, where x and y are... and The eigenvectors obtained after normalization; using the affinity matrix A zFeature alignment is achieved by more tightly combining highly correlated features within two feature distributions in the feature subspace; the objective function for achieving alignment is expressed as follows:

[0034]

[0035] Where N pair Indicates the number of nearest neighbor pairs. and They represent and The number of eigenvectors in the target function; the alignment matrix P is obtained by solving the objective function. z , thus obtaining the rendering feature vector set The corresponding aligned feature vector set The feature vector set corresponding to z semantic labels By concatenating the pixels according to their original positions, the aligned feature map F can be obtained. rs Let F r (i, j) represents the feature map F r The feature vector at pixel position (i, j), F rs (i, j) represents the feature map F after feature alignment. rs The style loss function for the feature vector at pixel position (i, j) is written as follows:

[0036]

[0037] Where, N F Represents the rendering feature map F r The number of pixels, dist(F) r (i, j), F rs (i, j) is used to calculate the eigenvector F. r (i, j) and eigenvector F rs Cosine similarity between (i, j).

[0038] Furthermore, the scene geometry and content protection module protects scene geometry information through depth constraints and Gaussian parameter regularization; the depth constraint uses a fast differentiable rasterizer to render the depth map D, with the specific formula as follows:

[0039]

[0040] Where, d i Represents the depth value of the i-th Gaussian; depth loss The initial depth map D obtained by calculating using the original Gaussian rendering is then used. init And the rendering depth map D obtained using Gaussian rendering in the current optimization phase rThe depth loss between them is obtained, let N D The depth loss is calculated using the formula for the number of pixels in the depth map:

[0041]

[0042] Gaussian parameter regularization: To prevent overfitting during scene stylization, Gaussian scale and opacity parameters were regularized, setting N... G The number of Gaussians in the scene is given by the following regularization term:

[0043]

[0044] At the same time, by calculating content loss This involves using a perceptual loss between the rendered feature map and the feature map of the original content view to preserve the scene's content information; noise affects image quality and can damage content information, so a total variational loss is introduced. To reduce high-frequency noise in an image while preserving its overall structure, the aforementioned loss function is implemented as follows:

[0045]

[0046] Where N I To render image I r The number of pixels, F c This is the content feature map extracted from the content image using the VGG-16 network.

[0047] Furthermore, the 3D Gaussian scene stylization model uses a reconstruction loss function to optimize the Gaussian parameters to obtain a consistent representation; the reconstruction loss function used for optimization... The formula is as follows:

[0048]

[0049] Where λ is the weight balance coefficient between loss functions; and Representing the rendered image I r and content image I c The average pixel value; representing the rendered image I. r and content image I c The variance of pixel values; Indicates the rendered image I r and content image I c The pixel value covariance; C1 and C2 are constants calculated by stable division;

[0050] The three-dimensional Gaussian parameters in the scene are optimized using multiple loss functions and regularization terms set in steps three and four. The specific calculation formulas are as follows:

[0051]

[0052] Among them, L tot The overall loss function for the 3D Gaussian scene stylization module; prefix coefficient λ s , λ dep , λ sca , λ opa , λ con , λ tv The hyperparameters are preset and their settings are demonstrated through multiple experiments; the overall loss function is used to optimize the 3D Gaussian scene to obtain a stylized 3D scene representation.

[0053] The framework constructed by this invention can achieve precise and controllable stylization of 3D scenes.

[0054] To address this, a mask matching module was designed. Users can control the matching of scene content and style image areas by inputting prompt phrases and adjusting the order of the prompt words in the prompt phrases, thereby achieving controllable and fine-grained scene stylization.

[0055] Furthermore, to improve the quality of scene stylization, a color conversion module is set up. Based on the mask matching group obtained by the mask matching module, a linear color conversion is performed on the content image and the scene Gaussian to align their color gamut with the color gamut of the corresponding style image region, laying the foundation for subsequent stylization.

[0056] Furthermore, to address the issue that existing style transfer loss functions often fail to faithfully represent the complete style image, a style transfer loss function based on image feature alignment is designed. This function aligns the feature distribution of the rendered image with the feature distribution of the style image, thereby obtaining high-quality style transfer results that are faithful to the complete style image.

[0057] Meanwhile, the scene style transfer process involves the loss of scene geometry and content information. To mitigate this issue, a scene geometry and content protection module was implemented. By introducing depth constraints, Gaussian parameter regularization terms, content loss, and total variational loss, the geometric and content information of the scene are protected while ensuring the quality of scene stylization.

[0058] Finally, a 3D Gaussian scene stylization module is configured to optimize the Gaussian parameters. The first part optimizes the 3D Gaussian parameters using a reconstruction loss function to ensure consistency between the 3D Gaussian representation in the content view and the scene. The second part optimizes the 3D Gaussian parameters using the previously configured image feature alignment-based style transfer loss function and multiple loss functions and regularization terms within the scene geometry and content preservation modules, achieving high-quality 3D scene stylization.

[0059] Beneficial effects

[0060] 1) This invention constructs a three-dimensional scene stylization framework based on three-dimensional Gaussian, which can achieve accurate and controllable high-quality scene stylization results in a short time.

[0061] 2) By utilizing the explicit characteristics of 3D Gaussian, a mask matching module and a color conversion module were implemented, laying the foundation for precise and controllable scene stylization.

[0062] 3) In order to make the stylization result faithful to the complete style image, a style transfer loss function based on image feature alignment was designed, which achieved high-quality stylization results.

[0063] 4) By setting up scene geometry and content protection modules, the geometric and content information within the original scene can be protected while stylizing the scene. Attached Figure Description

[0064] Figure 1 This is an architecture diagram of the present invention applied to style transfer in 3D scenes;

[0065] Figure 2 This is a schematic diagram illustrating the calculation of the style transfer loss function based on image feature alignment in this invention;

[0066] Figure 3 This diagram illustrates the controllability and effectiveness of the method of the present invention. Detailed Implementation

[0067] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0068] Example

[0069] See Figure 1 This embodiment includes the following specific steps:

[0070] Step 1. Configure the mask matching module

[0071] To achieve fine-grained and highly controllable 3D scene stylization, it is necessary to match the desired style regions with the scene content. Utilizing explicit Gaussian features, this module aims to obtain a Gaussian mask matching set between the style image region, the content image region, and the scene. For example... Figure 1 As shown, the user input content image I c Style Image I s and two prompt phrases (Text) used to segment the content image and style image respectively. c and Text s The module calls the SAM segmentation method to obtain the semantic mask M of the content image. c and style image semantic mask M s At the same time, the input prompt phrase Text will be displayed. c and Texts Hint words with the same index are treated as a pair of matching hint words and their corresponding mask regions are matched to obtain M. c and M s The mask matching relationship. Then, M... c The semantic labels are back-projected onto the Gaussian surface to obtain the corresponding semantic labels for the Gaussian surface. Specifically, the weight of the j-th semantic label of the i-th Gaussian surface is calculated as follows:

[0072]

[0073] in, p represents the weight of the j-th semantic label of the i-th Gaussian, and p is the mask M. c pixels on, α i For the weights of alpha mixing, T i Let be the transmittance of the back-projected semantic label when it reaches the i-th Gaussian. Next, by setting a certain threshold, the transmittance is compared with the calculated semantic label weights to determine the semantic label corresponding to the Gaussian. For the z-th semantic label, its corresponding mask matching group can be obtained.

[0074] Step 2. Configure the color conversion module

[0075] To enhance the stylization effect, it is necessary to ensure that the color gamut of the stylized scene matches the color gamut of the stylized image. For example... Figure 1 As shown, based on the obtained mask matching groups, color conversion can be performed on the corresponding regions to obtain a content image and scene Gaussian image with color domain alignment, laying a solid foundation for subsequent scene stylization. For each mask matching group, let... The set of all pixel colors for the style image region; by aligning the content color set {p c} and style color set {p s The mean and covariance of the vectors are used to calculate the linear color transformation weights and biases:

[0076] p ct =Ap c +b, c ct =Ac+b,

[0077] stE[p ct ]=E[p s ], Cov[p ct ] = Cov[p s ],

[0078] Where c is the original Gaussian color set, p ct and c ctLet represent the color set of the content image and the Gaussian color set after linear color transformation, respectively. The weights and bias parameters of the linear color transformation can be obtained from the above formula.

[0079] Step 3. Set the style transfer loss function based on image feature alignment

[0080] Existing optimization-based 3D scene style transfer methods mostly employ nearest-neighbor feature matching loss functions to optimize the scene, resulting in style transfer results that are difficult to faithfully reproduce the complete style image. For example... Figure 2 As shown, a loss function based on image feature alignment is used to render image I. r Style Image I s The mask matching group is used as input, where the rendered image is obtained by rendering a 3D Gaussian within the scene using a fast differentiable rasterizer, as shown in the following formula:

[0081]

[0082] Where C represents the color of the pixel being rendered, c i This indicates the color of the i-th Gaussian region used to render the pixel.

[0083] Next, render image I r and style image I s The rendering feature maps F are extracted separately using the same VGG-16 network. r and style feature diagram F s For the z-th mask matching group Ω z Extract feature vector sets from the feature maps corresponding to the rendered image and the style image, respectively. and In order to transform the feature vector set and Alignment, calculate the alignment matrix P z To transform the rendering feature vector set Subsequently, normalized cross-correlation between features is used as a similarity measure to construct an affinity matrix to compensate for the lack of semantic information in image features. Specifically, the affinity matrix of rendering and style features is represented as A. z The specific calculation formula is as follows:

[0084]

[0085] in A represents z The value at position (i, j); express The i-th eigenvector in express The j-th eigenvector; express With feature vector set Each feature vector Calculate the k-nearest neighbor set obtained by normalizing cross-correlation; express With feature vector set Each feature vector The k-nearest neighbor set is obtained by calculating the normalized cross-correlation, with k = 5; s(x, y) represents the normalized cross-correlation of feature vectors x and y, where x and y are... and The eigenvectors obtained after normalization; using the affinity matrix A z Feature alignment is achieved by more tightly combining highly correlated features within two feature distributions in the feature subspace; specifically, the objective function for achieving alignment is expressed as follows:

[0086]

[0087] Where N pair Indicates the number of nearest neighbor pairs. and They represent and The number of corresponding eigenvectors; the alignment matrix P is obtained by solving the objective function. z This allows us to obtain the set of rendering feature vectors. The corresponding aligned feature vector set The feature vector set corresponding to z semantic labels By concatenating the pixels according to their original positions, the aligned feature map F can be obtained. rs Let F r (i, j) represents the feature map F r The feature vector at pixel position (i, j), F rs (i, j) represents the feature map F after feature alignment. rs The style loss function for the feature vector at pixel position (i, j) is written as follows:

[0088]

[0089] Where, N F Represents the rendering feature map F r The number of pixels, dist(F) r (i, j), F rs (i, j) is used to calculate the eigenvector F. r (i, j) and eigenvector F rs Cosine similarity between (i, j). Figure 2 As shown, the rendered feature map F rTo the aligned feature map F rs Style transfer can be achieved through optimization. The alignment process uses an alignment matrix P. z This approach preserves the feature distribution of the original rendered image. Furthermore, by aligning to the entire style feature set, the style transfer result remains faithful to the complete style image.

[0090] Step 4. Configure scene geometry and content protection modules

[0091] During style transfer, scene geometry and content information are affected. For example... Figure 1 As shown, in the process of optimizing the Gaussian parameters of the scene, depth loss, Gaussian parameter regularization term, content loss, and total variational loss are introduced. For the depth loss, a fast differentiable rasterizer can be used to render the depth map D in 3D Gaussian, and the specific formula is as follows:

[0092]

[0093] Where, d i This represents the depth value of the i-th Gaussian. Depth loss. The initial depth map D obtained by calculating using the original Gaussian rendering is then used. init And the rendering depth map D obtained using Gaussian rendering in the current optimization phase r The depth loss between them is obtained, let N D The depth loss is calculated using the formula for the number of pixels in the depth map:

[0094]

[0095] Regarding Gaussian parameter regularization, to prevent overfitting during scene stylization, regularization was applied to the Gaussian scale parameter and opacity parameter, setting N... G The number of Gaussians in the scene is given by the following regularization term:

[0096]

[0097] At the same time, by calculating content loss This involves using a perceptual loss between the rendered feature map and the feature map of the original content view to preserve the scene's content information; noise can affect image quality and damage content information, so a total variational loss is introduced. This reduces high-frequency noise in the image while preserving its overall structure. The loss function mentioned above is implemented as follows:

[0098]

[0099] Where N I To render image I r The number of pixels, Fc This is the content feature map extracted from the content image using the VGG-16 network.

[0100] Step 5. Configure the 3D Gaussian scene stylization module.

[0101] In the first part, because the 3D Gaussian representation in the content view and the scene is inconsistent after the linear color transformation in step 2, a re-optimization is performed to obtain a consistent representation. The reconstruction loss function used for optimization is... The formula is as follows:

[0102]

[0103]

[0104] Where λ is the weight balance coefficient between loss functions; and Representing the rendered image I r and content image I c The average pixel value; representing the rendered image I. r and content image I c The variance of pixel values; Indicates the rendered image I r and content image I c The pixel value covariance; C1 and C2 are constants calculated by stable division.

[0105] The second part optimizes the 3D Gaussian parameters in the scene using the multiple loss functions and regularization terms set in steps three and four. The specific calculation formulas are as follows:

[0106]

[0107] Among them, L tot The overall loss function for the second part of the 3D Gaussian scene stylization module; prefix coefficient λ s ,λde p , λ sca , λ opa , λ con , λ tv The hyperparameters are preset and their settings will be demonstrated through multiple experiments; the overall loss function is used to optimize the 3D Gaussian scene to obtain a stylized 3D scene representation.

[0108] See Figure 3 The method of the present invention can realize scene style transfer of a complete single style map, scene style transfer of a semantically aware single style map, and scene style transfer of a combination of multiple style maps, achieving a controllable and fine-grained high-quality stylization effect.

Claims

1. A method for constructing a controllable 3D scene stylization framework based on 3D Gaussian, characterized in that, The framework is used to achieve precise and controllable 3D scene stylization, and its construction method includes the following specific steps: Step 1: Configure the mask matching module The user inputs a set of prompts for segmenting the content image and style image into the mask matching module. The module then calls the SAM segmentation method to obtain the 2D semantic segmentation mask of the content image and style image, and selects the correspondence between the scene content and style regions. Then, by back-projecting the semantic labels of the semantic segmentation mask onto a 3D Gaussian surface, the semantic label of each 3D Gaussian surface is determined. The content image, style image, and 3D Gaussian surface with the same semantic label are matched together to obtain the mask matching relationship. Step 2: Configure the color conversion module Based on the mask matching relationship obtained in step one, calculate the linear color transformation parameters of the original scene content image and the color distribution of the three-dimensional Gaussian within the scene to be converted into the style image region that matches them, and convert the content image and the color distribution of the three-dimensional Gaussian into the color distribution of the corresponding style image region. Step 3: Set the style transfer loss function based on image feature alignment Based on the mask matching relationship obtained in step one, calculate the affinity matrix between the rendered feature map and the style feature map; The alignment matrix is ​​calculated by aligning the rendered feature map to the style feature map using the affinity matrix, and then the aligned feature map is obtained using the alignment matrix. Set the loss function using the rendered feature map and the aligned feature map; Step 4: Configure scene geometry and content protection modules Depth loss is calculated by rendering the initial depth map of the pre-optimized 3D Gaussian and the depth map obtained from the Gaussian rendering during the optimization stage. At the same time, regularization terms are set for the Gaussian scale parameter and opacity parameter to protect the scene geometric information; content loss function and total variational loss are set to protect the scene content information. Step 5: Configure the 3D Gaussian scene stylization module The 3D Gaussian scene stylization module includes optimizing Gaussian parameters using a reconstruction loss function to obtain a consistent expression; optimizing the 3D Gaussian parameters in the scene using multiple loss functions and regularization terms set in steps three and four, thereby achieving controllable 3D scene stylization based on 3D Gaussians.

2. The construction method according to claim 1, characterized in that, The mask matching module, in this module, user input content image I c Style Image I s and two prompt phrases (Text) used to segment the content image and style image respectively. c and Text s The module calls the SAM segmentation method to obtain the semantic mask M of the content image. c and style image semantic mask M s At the same time, the input prompt phrase Text will be displayed. c and Text s Hint words with the same index are treated as a pair of matching hint words and their corresponding mask regions are matched to obtain M. c and M s Mask matching relationship; Subsequently, M c The semantic labels are back-projected onto the Gaussian to obtain the corresponding semantic labels of the Gaussian. Specifically, the weight of the j-th semantic label of the i-th Gaussian is first calculated: ； ； in, Let represent the weight of the j-th semantic label of the i-th Gaussian, and p be the coordinates of a pixel in the image. Let j be the value of the semantic mask at coordinate p. Let i be the weight of the i-th Gaussian in the alpha mixing on the p-axis. To calculate the transmittance up to the i-th Gaussian point on the p-coordinate; Next, by setting a threshold and comparing it with the calculated semantic label weights, the semantic label corresponding to the Gaussian is determined; for the z-th semantic label, its corresponding mask matching group is obtained. ,in Let z be the mask region in the style image semantic mask where the semantic label is the z-th semantic label. Let z be the mask region in the semantic mask of the content image with the z-th semantic label. Let be the 3D Gaussian vector in the scene whose semantic label is the z-th semantic label.

3. The construction method according to claim 1, characterized in that, The color conversion module aims to convert the color gamut of the scene representation into the color gamut of the style image; specifically, it enables... Let be the set of all pixel colors in the content image region to be recolored, and let The set of colors for all pixels in the style image region; by matching the content color set. and style color set The mean E and covariance Cov are used to calculate the linear color transformation weights and biases: ； Where c is the original Gaussian color set. and Let A and b represent the content image color set and Gaussian color set after linear color transformation, respectively. Let A be the weight parameter of the linear color transformation and b be the bias parameter of the linear color transformation. The weight and bias parameters of the linear color transformation are obtained by solving the above formula, and color transformation is performed on the content image and the 3D Gaussian color set in the scene.

4. The construction method according to claim 1, characterized in that, The style transfer loss function based on image feature alignment will render image I. r and style image I s The rendering feature maps F are extracted separately using the same VGG-16 network. r and style feature diagram F s The rendered image is obtained by rendering a 3D Gaussian image within the scene using a fast differentiable rasterizer, as shown in the following formula: ； in, c represents the color of the rendered image at coordinate p. i This indicates the color of the i-th Gaussian pixel. Let i be the weight of the i-th Gaussian in the alpha mixing on the p-axis. To calculate the transmittance up to the i-th Gaussian point on the p-coordinate; For the z-th semantic label, based on the mask matching group Ω obtained in step one z Extract feature vector sets from the feature maps corresponding to the rendered image and the style image, respectively. and In order to set the feature vectors and Alignment, calculate the alignment matrix To transform the rendering feature vector set Subsequently, normalized cross-correlation between features is used as a similarity measure to construct an affinity matrix to compensate for the lack of semantic information in image features. The affinity matrices of rendering and style features are represented as follows: The specific calculation formula is as follows: ； ； in express The value at position (i,j) in the middle; express The i-th eigenvector in the data. express The j-th eigenvector; express With feature vector set Each feature vector Calculate the k-nearest neighbor set obtained by normalizing cross-correlation; express With feature vector set Each feature vector The k-nearest neighbor set is obtained by calculating the normalized cross-correlation; s(x,y) represents the normalized cross-correlation of feature vectors x and y, where x and y are... and The eigenvectors obtained after normalization; using the affinity matrix Feature alignment is achieved by more tightly combining highly correlated features within two feature distributions in the feature subspace; the objective function for achieving alignment is expressed as follows: ； Where N pair Indicates the number of nearest neighbor pairs. and They represent and The number of eigenvectors in the target function; the alignment matrix is ​​obtained by solving the objective function. , thus obtaining the rendering feature vector set The corresponding aligned feature vector set ; the feature vector set corresponding to z semantic labels By concatenating the pixels according to their original positions, the aligned feature map F can be obtained. rs Let F r (i,j) represents the feature map F r The feature vector at pixel position (i,j), F rs (i,j) represents the feature map F after feature alignment. rs The style loss function for the feature vector at pixel position (i,j) is written as follows: ； ； Where, N F Represents the rendering feature map F r The number of pixels, dist(F) r (i,j),F rs (i,j) is used to calculate the eigenvector F. r (i, j) and eigenvector F rs Cosine similarity between (i,j).

5. The construction method according to claim 1, characterized in that, The scene geometry and content protection module protects scene geometry information through depth constraints and Gaussian parameter regularization. The depth constraint uses a fast differentiable rasterizer to render the depth map D, with the specific formula as follows: ； in, d represents the depth value of the depth map at the p coordinate. i This represents the depth value of the i-th Gaussian. Let i be the weight of the i-th Gaussian in the alpha mixing on the p-axis. To calculate the transmittance up to the i-th Gaussian point on the p-coordinate; depth loss. The initial depth map obtained by calculating using the original Gaussian rendering was then used. And the rendering depth map obtained using Gaussian rendering in the current optimization phase. The depth loss between them is obtained, let N D The depth loss is calculated using the formula for the number of pixels in the depth map: ； Gaussian parameter regularization: To prevent overfitting during scene stylization, the Gaussian scale parameter s and opacity parameter σ are regularized, allowing N... G The number of Gaussians in the scene is given by the following regularization term: ； ； At the same time, by calculating content loss That is, the perceptual loss between the rendered feature map Fr and the feature map of the original content view is used to protect the content information of the scene; noise will affect the image quality and cause the content information to be damaged, so a total variational loss is introduced. To reduce high-frequency noise in an image while preserving its overall structure, the aforementioned loss function is implemented as follows: ； ； Where N I For rendering image I r The number of pixels, F c This is the content feature map extracted from the content image using the VGG-16 network.

6. The construction method according to claim 5, characterized in that, The 3D Gaussian scene stylization module uses a reconstruction loss function to optimize Gaussian parameters to obtain a consistent representation; the reconstruction loss function used for optimization... The formula is as follows: ； ； ； in The weighting balance coefficients between loss functions; and Representing the rendered image I r and content image I c The average pixel value; Representing the rendered image I r and content image I c The variance of pixel values; Indicates the rendered image I r and content image I c The pixel value covariance; C1 and C2 are constants calculated by stable division; The three-dimensional Gaussian parameters in the scene are optimized using multiple loss functions and regularization terms set in steps three and four. The specific calculation formulas are as follows: ； Among them, L tot The overall loss function for the 3D Gaussian scene stylization module; prefix coefficient λ s ,λ dep , λ sca , λ opa ,λ con , λ tv The hyperparameters are preset and their settings are demonstrated through multiple experiments; the overall loss function is used to optimize the 3D Gaussian scene to obtain a stylized 3D scene representation.

Images