3D Gaussian splash model copyright protection method

Abstract

The invention discloses a 3D Gaussian spatter model copyright protection method, which comprises the following steps: acquiring a 3D Gaussian spatter model, performing secondary training on the 3D Gaussian spatter model, embedding watermark information into the 3D Gaussian spatter model, and acquiring the 3D Gaussian spatter model subjected to secondary training; and constructing a message extractor based on a deep neural network, retrieving the image rendered by the 3D Gaussian splash model through the message extractor, and extracting watermark information. According to the method, the watermark information is embedded into the model through secondary training of the 3D Gaussian splash model, and then the watermark information is extracted by using a specific extractor. And meanwhile, relatively high rendering quality is kept. And meanwhile, the model can be effectively prevented from being maliciously stolen.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional modeling, and in particular relates to a 3D Gaussian splash model copyright protection method. Background Art

[0002] 3D Gaussian Splash (3DGS) is an emerging concept that uses a set of Gaussian ellipsoids to model a 3D scene and rasterizes the Gaussian ellipsoids into the image to achieve efficient rendering. Compared with the implicitly represented Neural Radiance Field (NeRF), 3D Gaussian Splash not only significantly improves the rendering speed, but also has better effects in dynamic reconstruction, geometry editing, and physical simulation because it is a display representation. This makes 3D Gaussian Splash have a wider range of application scenarios in reality and has huge potential for 3D content representation in the future field of computer vision.

[0003] Although 3D Gaussian splash has made great progress in compression and regularization, quality improvement, dynamic 3D reconstruction and Gaussian splash 3D editing, there are few studies on copyright protection of 3D Gaussian splash models. In particular, in recent years, more and more large-scene 3D models are applied to real life. Due to the high training cost of large-scene models, once the model is maliciously stolen, the copyright owner may suffer huge losses.

[0004] Therefore, protecting its copyright becomes extremely important.Although the existing digital watermarking technology is widely used in media data protection, it lacks an effective solution for 3D models. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a 3D Gaussian splash model copyright protection method to solve the problems existing in the above prior art.

[0006] To achieve the above object, the present invention provides a 3D Gaussian splash model copyright protection method, comprising the following steps:

[0007] Acquire a 3D Gaussian splash model, and embed watermark information into the 3D Gaussian splash model by performing secondary training on the 3D Gaussian splash model to obtain a secondary trained 3D Gaussian splash model;

[0008] A message extractor is constructed based on a deep neural network, and the image rendered by the 3D Gaussian splash model is retrieved by the message extractor to extract watermark information.

[0009] Preferably, the method for performing secondary training on the 3D Gaussian splash model comprises:

[0010] Performing two-level optimization on the 3D Gaussian splash model;

[0011] Optimize and train the three-dimensional Gaussian model;

[0012] The optimized 3D Gaussian model is trained for noise attack and anti-distortion rendering.

[0013] Preferably, the method for performing two-stage optimization comprises:

[0014] The original 3D Gaussian model is trained, including the standard light loss between the rendered and real pixel values, which is used to adjust the perspective effect of the model;

[0015] The parameters of the three-dimensional Gaussian splash model containing watermark information are trained to implicitly embed the watermark information into the three-dimensional Gaussian splash model.

[0016] Preferably, the method for optimizing the training of the three-dimensional Gaussian model comprises:

[0017] The Gaussian function model is optimized based on the supervision of multi-view images;

[0018] The three-dimensional Gaussian expression is:

[0019]

[0020] in, Mapping a point (x, y, z) in space to a point in spherical coordinates The specified direction is a non-negative radiation value, G is a Gaussian function, μ i is the mean, Σ i is the covariance, c i Represents a color associated with a view.

[0021] Preferably, the loss function expression for optimizing the training of the three-dimensional Gaussian model is:

[0022] L sta =(1-λ)L1+λL D-SSIM

[0023] Where λ is a constant, L1 loss is the absolute value of the difference between the rendered image and the original image pixel by pixel, and L D-SSIM is the multi-view structural similarity loss.

[0024] Preferably, the noise of the noise attack training includes: additive Gaussian noise with a mean of μ and a standard deviation of v; random axis-angle rotation with a parameter of α; random scaling with a parameter of s; and Gaussian blur with a kernel of k.

[0025] Preferably, the message extractor comprises a convolutional layer, a flat layer and a dense layer, wherein the dense layer is activated using a rectified linear unit function, a noise-processed rendered image is input, and watermark information is output.

[0026] Preferably, the protection method further comprises optimizing the extracted watermark information through a classifier.

[0027] Compared with the prior art, the present invention has the following advantages and technical effects:

[0028] The present invention discloses a 3D Gaussian splash model copyright protection method, comprising the following steps: obtaining a 3D Gaussian splash model, embedding watermark information into the 3D Gaussian splash model model by performing secondary training on the 3D Gaussian splash model, and obtaining a secondary trained 3D Gaussian splash model model; constructing a message extractor based on a deep neural network, and retrieving an image rendered by the 3D Gaussian splash model through the message extractor to extract watermark information. The present invention embeds watermark information into the model by performing secondary training on the 3D Gaussian splash model, and then uses a specific extractor to extract the watermark information. At the same time, a high rendering quality is maintained. At the same time, the model can be effectively prevented from being maliciously stolen. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0030] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0031] Figure 2 is a training flow chart of an embodiment of the present invention;

[0032] Figure 3 This is a diagram of the extractor network structure of an embodiment of the present invention;

[0033] Figure 4 Rendering quality diagram under different embedding capacities according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Embodiment 1

[0037] like Figure 1 As shown, this embodiment provides a 3D Gaussian splash model copyright protection method, including the following steps:

[0038] Acquire a 3D Gaussian splash model, and embed watermark information into the 3D Gaussian splash model by performing secondary training on the 3D Gaussian splash model to obtain a secondary trained 3D Gaussian splash model;

[0039] A message extractor is constructed based on a deep neural network, and the image rendered by the 3D Gaussian splash model is retrieved by the message extractor to extract watermark information.

[0040] Furthermore, the method for performing secondary training on the 3D Gaussian splash model includes:

[0041] Performing two-level optimization on the 3D Gaussian splash model;

[0042] Optimize and train the three-dimensional Gaussian model;

[0043] The optimized 3D Gaussian model is trained for noise attack and anti-distortion rendering.

[0044] Further, the two-level optimization is as follows:

[0045] The training process of the three-dimensional Gaussian splash model is redesigned. The training process of this embodiment is divided into two stages:

[0046] The first stage is the training of the original three-dimensional Gaussian model, which involves a standard light loss between the rendered and real pixel values ​​to ensure the perspective effect of the model. After completing the update of the parameter Ψ0 in the first stage, this embodiment uses the second stage to train the parameter Ψ of the three-dimensional Gaussian splash model containing watermark information. This embodiment introduces several methods to achieve robust watermark information recovery with almost no impact on the rendering quality of the image. During the training process, the algorithm first trains a standard Gaussian splash model, and then constrains the training of the CopyR-3DGS model through the image rendered by the standard Gaussian splash model. The training process is as follows: Figure 2 shown.

[0047] Furthermore, the three-dimensional Gaussian model is trained as follows:

[0048] The 3D Gaussian is a display brightness field that flexibly represents the 3D scene through a learnable 3D Gaussian function. These Gaussian functions are optimized under the supervision of multi-view images, which significantly improves the reconstruction efficiency of the model. The display radiation field directly represents the distribution of light in a discrete spatial structure, in which each element stores the radiation information of its respective position in space. The 3D Gaussian is represented as:

[0049] in Mapping a point (x, y, z) in space to a point in spherical coordinates The specified direction, to a non-negative radiation value. G is a Gaussian function with a mean of μ i , the covariance is Σ i , c represents the color associated with the view.

[0050] The loss function of the first stage standard 3DGS model training uses a combination of L1 and D-SSIM terms:

[0051] L sta =(1-λ)L1+λL D-SSIM

[0052] Where λ = 0.2, L1 loss is to compare the difference between the rendered image and the original image pixel by pixel, and then take the absolute value, L D-SSIM is the multi-view structural similarity loss.

[0053] Furthermore, the distortion layer (noise attack training and anti-distortion rendering) is as follows:

[0054] In order to simulate the noise attack scenario in the real world and improve the robustness of the algorithm of this embodiment, a set of microscopic image noise attacks are introduced, including: 1) additive Gaussian noise with a mean of μ and a standard deviation of v; 2) random axis angle rotation with a parameter of α; 3) random scaling with a parameter of s; 4) Gaussian blur with a kernel of k. Due to the differentiable distortion adopted in this embodiment, the network of this embodiment can be trained end-to-end.

[0055] Anti-aliasing rendering should be used during training; it is not part of the 3D Gaussian model. If the model is stolen by a malicious user, using a different rendering strategy can still ensure the robustness of the method of this embodiment.

[0056] Furthermore, the watermark information is restored as follows:

[0057] Let P be the viewing angle when rendering an image from a three-dimensional Gaussian model in this embodiment. This embodiment hopes to recover the watermark information M from the rendered image Ψ(P). Importantly, this embodiment does not want the image rendered by the original three-dimensional Gaussian splash model Ψ0 to be able to recover the watermark information, but requires that the images rendered by Ψ0 and Ψ look the same in visual effect. Therefore, this embodiment optimizes Ψ to minimize the following contrast loss term when training the CopyR-3DGS model:

[0058]

[0059] Where ⊙ is a bit string containing all 0s.

[0060] Extractor Design: To retrieve the message M from the rendered image, this embodiment designs a message extractor:

[0061] E θ :Ψ(P)→M

[0062] The extractor is a deep neural network, where θ is the network parameter to be trained. Specifically, this embodiment is constructed using a series of convolutional layers, flat layers, and dense layers, where the dense layer is activated using a rectified linear unit function, the rendered image after noise processing is input, and the message M is output. The extractor network structure designed in this embodiment is as follows Figure 3 shown.

[0063] This embodiment takes a 800×800×3 noise-processed rendered image as an example. First, the input image is convolved 7 times using a convolution layer with a convolution kernel size of 3×3 to generate a 12×12×128 feature map, and then the feature map is flattened using a flatten layer. Finally, the watermark information is extracted through two dense layers.

[0064] Discriminator design: Since the rendered image has been processed by the noise layer, it may be challenging to accurately restore all the details of M. Therefore, this embodiment trains a classifier To optimize the extraction effect of this embodiment, that is, to classify whether a given 3D Gaussian rendering contains watermark information. The optimization is done with the following cross entropy loss:

[0065]

[0066] This embodiment guides the process of extracting watermark information by the extractor through the prediction of the discriminator, by adding the output of the classifier as input to E θ , so that

[0067] Although the focus of this embodiment is on string information, the framework of this embodiment can be easily extended to other forms, such as using image, text or audio data as watermark information. This embodiment can be achieved by simply changing E θ network structure so that it has corresponding functions.

[0068] In order to prevent the images rendered by CopyR-3DGS from deviating from the fidelity of the visual signals it presents, this embodiment also uses photometric error when training the model:

[0069] L rgb =|θ(P)-θ0(P)|,

[0070] The overall training loss in the second stage can be expressed as:

[0071]

[0072] This embodiment carried out the following experiments:

[0073] The experimental setup is as follows:

[0074] Dataset: This embodiment uses NSVF, LLFF and BlendedMVS datasets for experiments. The NSVF and LLFF datasets are composed of a bunch of real scene data, and the BlendedMVS dataset is composed of synthetic data. This embodiment selects 50 images from each dataset for training. In addition, each scene is rendered at 50 viewing angles to test the effectiveness of the extractor at each viewing angle. In the experiments of this embodiment, this embodiment takes the average of all experimental results.

[0075] Training: This example uses Pytorch to implement the watermark algorithm. This example runs the experiment on an NVIDIA A100 GPU. For all scenes on the NSVF and Mip-NeRF datasets, this example sets λ3=1, λ0=0.01, λ1=0.5, and λ2=0.5. For all scenes on the BlendedMVS dataset, this example sets λ3=1, λ0=0.1, λ1=1, and λ2=1. Other experimental parameters are set to: Learning rate=1×10 -4 , batch_size=150, Adam optimizer is used for optimization, and the stochastic gradient descent algorithm is used to train the model for 5000 epochs.

[0076] Benchmark: According to a large number of literatures, there is no research related to watermarking of 3D Gaussian splash model. Therefore, this example tries four watermark embedding strategies to ensure fair comparison: (1) MBRS+3DGS uses MBRS algorithm to embed watermark information into dataset images before training 3DGS model; (2) DeepStega+3DGS uses 2D watermarking method DeepStega to process images before training NeRF model; (3) HiDDeN+3DGS uses HiDDeN scheme to process images before training NeRF model; (4) CopyR-3DGS.

[0077] Evaluation method: This embodiment evaluates the performance of the method of this embodiment and other methods by following the invisibility, robustness and capacity standards of digital watermarks. For invisibility, this embodiment uses PSNR, SSIM and LPIPS to evaluate the performance to compare the visual quality of the results presented after the watermark information is embedded. For robustness, this embodiment measures the bit accuracy under different distortions to detect whether the extractor of this embodiment can effectively extract the watermark information, and also considers the following information extraction distortions: (1) Gaussian noise, (2) rotation, (3) scaling and (4) Gaussian blur. In terms of watermark capacity, based on the watermark settings of previous researchers for displaying three-dimensional models, this embodiment studies the invisibility and robustness under different message lengths. Since this embodiment conducts experiments at different viewing angles of each scene, the effectiveness of the method of this embodiment can be evaluated.

[0078] The loss weight parameters are chosen as follows:

[0079] Different settings of the loss function weight hyperparameters λ0, λ1, λ2 and λ3 will directly affect the training effect of the watermarked model and the extraction results of the extractor. λ0 is a hyperparameter of the discriminator and generally takes a fixed value. λ1 and λ2 are hyperparameters of the extractor. In order to achieve the simultaneous realization that the watermark information cannot be extracted in the standard 3DGS but can be extracted in CopyR-3DGS, λ1 and λ2 are set to the same value. The impact of the loss parameters on the model is tested by changing the values ​​of λ1 and λ3. This embodiment conducts experiments on the BlendedMVS dataset. In order to balance the extraction ability and rendering quality of the model, λ0 is fixed to 0.1, and λ1 and λ3 are set to (10,1), (1,1), and (1,10) respectively. Comparative experiments are conducted to compare the rendering quality of the watermarked model under different parameter settings, the accuracy of the extracted message, and select the optimal parameter settings. Table 1 shows the PSNR and SSIM values ​​between the model rendered image and the original image when the algorithm embeds 50bit information and the bit accuracy of the extracted message by the extractor.

[0080] Table 1

[0081] <![CDATA[(λ1,λ3)]]> PSNR↑ SSIM↑ Bit accuracy (%)↑ (10,1) 22.53 0.7354 100％ (1,1) 33.21 0.9341 100％ (1,10) 35.42 0.9684 86.21％

[0082] The experimental results show that when the values ​​of λ1 and λ3 are (10,1), the watermark information extracted by the extractor can achieve good bit accuracy, but the rendering quality of the trained watermarked model is not high. When the values ​​of λ1 and λ3 are (1,10), the model can achieve good rendering quality, but will lose some bit accuracy. When the values ​​of λ1 and λ3 are (1,1), the watermarked model can achieve good rendering effect,

[0083] The extractor can also accurately extract the watermark information. Based on the above experimental results, (1,1) is selected as the parameter of the loss function.

[0084] The algorithm capacity is as follows:

[0085] As shown in Table 2, there is a negative correlation between the accuracy of extracting watermark information and the length of the watermark information. This embodiment tests the accuracy of watermark messages with lengths of 50, 100, 150 and 200 bits. It can be seen from the experimental results that as the message length increases, the bit accuracy gradually decreases. However, the algorithm of this embodiment achieves the best bit accuracy in all settings, which proves that the message can be effectively embedded and extracted. Moreover, when the embedded watermark length is not greater than 150 bits, the algorithm of this embodiment can extract the watermark information losslessly. Even if 200 bits of information are embedded, a bit accuracy of 98.31% can be achieved.

[0086] Table 2

[0087]

[0088] At the same time, in order to test the impact of watermark information capacity on the visual quality of model rendering, all methods were trained with embedded information lengths of 50 and 100 bits, because these capacities are more commonly used in information transmission. The rendering quality of the watermarked models trained by all methods is shown in Figure 2. Figure 4 shown.

[0089] It can be seen from the experimental results that the length of the embedded message has no significant effect on the rendering effect of the trained model, and all algorithms can achieve good rendering quality. This may be because after these two-dimensional watermarking methods embed the message into the training set image, the watermark information is smoothed during the Gaussian splash model training process, resulting in the length of the embedded watermark information having no significant effect on the model rendering effect. However, the algorithm in this embodiment does not directly embed the watermark information into the training data set image, but uses adversarial training to embed the watermark information into the model, which effectively realizes the embedding of watermark information.

[0090] The algorithm performance is as follows:

[0091] (1) Reconstruction quality and bit accuracy

[0092] In order to show whether the embedded watermark information will damage the visual quality, the quality of the three-dimensional model generated by the algorithm of this embodiment is visually compared with other methods, and the results are shown in Table 3. It can be seen from the experimental results that each method can achieve good reconstruction quality, but although the MBRS+3DGS, DeepStega+3DGS and HiDDeN+3DGS methods are effective methods in two-dimensional watermarking, their bit accuracy for rendered images is relatively low, which proves that these three methods do not effectively embed information during the 3DGS training process. The view synthesis of 3DGS changes the watermark information embedded by the two-dimensional watermarking method, resulting in the invalidation of the embedded watermark information. The low bit accuracy of these three schemes proves that this is not an effective embedding scheme, but the CopyR-3DGS scheme of this embodiment can achieve a good extraction effect, proving the effectiveness of the method.

[0093] Table 3

[0094]

[0095] (2) Model robustness to two-dimensional distortion

[0096] This embodiment evaluates the robustness of the algorithm of this embodiment by applying several traditional two-dimensional distortions. Specifically, as shown in Table 4, several types of two-dimensional distortions are used, including noise, rotation, scaling, and cropping. This experiment uses different methods to uniformly embed 100 bits of watermark information. It can be seen that the algorithm of this embodiment is quite robust to different two-dimensional distortions and maintains a high accuracy of watermark information extraction.

[0097] Table 4

[0098]

[0099] In addition, in order to evaluate the robustness of CopyR-3DGS, 100 bits of information were embedded through the algorithm of this embodiment, and the Gaussian distribution parameters were pruned using sequential pruning and random pruning methods. Sequential pruning refers to pruning in ascending order of Gaussian opacity, that is, removing Gaussians with lower opacity first. On the other hand, random pruning involves randomly pruning a certain proportion of Gaussian distributions. The quantitative indicators are shown in Tables 5 and 6. Table 5 shows the reconstruction quality and bit accuracy under sequential pruning, and Table 6 shows the reconstruction quality and bit accuracy under random pruning. The experimental results show that sequential pruning has no obvious effect on the rendering quality and watermark extraction of the model, but random pruning will reduce the rendering quality of the model and have no obvious effect on watermark extraction. The proposed method can effectively resist the pruning process.

[0100] Table 5

[0101]

[0102] Table 6

[0103]

[0104] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A 3D Gaussian splash model copyright protection method, characterized in that: The following steps are involved: Acquire a 3D Gaussian splash model, and embed watermark information into the 3D Gaussian splash model by performing secondary training on the 3D Gaussian splash model to obtain a secondary trained 3D Gaussian splash model; A message extractor is constructed based on a deep neural network, and the image rendered by the 3D Gaussian splash model is retrieved by the message extractor to extract watermark information.

2. The 3D Gaussian splash model copyright protection method according to claim 1, characterized in that: The method for performing secondary training on the 3D Gaussian splash model includes: Performing two-level optimization on the 3D Gaussian splash model; Optimize and train the three-dimensional Gaussian model; The optimized 3D Gaussian model is trained for noise attack and anti-distortion rendering.

3. The 3D Gaussian splash model copyright protection method according to claim 2, characterized in that: The method for performing two-stage optimization comprises: The original 3D Gaussian model is trained, including the standard light loss between the rendered and real pixel values, which is used to adjust the perspective effect of the model; The parameters of the three-dimensional Gaussian splash model containing watermark information are trained to implicitly embed the watermark information into the three-dimensional Gaussian splash model.

4. The 3D Gaussian splash model copyright protection method according to claim 2, characterized in that: The method for optimizing the training of the three-dimensional Gaussian model comprises: The Gaussian function model is optimized based on the supervision of multi-view images; The three-dimensional Gaussian expression is: in, Mapping a point (x, y, z) in space to a point in spherical coordinates The specified direction is a non-negative radiation value, G is a Gaussian function, μ i is the mean, Σ i is the covariance, c i Represents a color associated with a view.

5. The 3D Gaussian splash model copyright protection method according to claim 4, characterized in that: The loss function expression for optimizing the training of the three-dimensional Gaussian model is: L sta =(1-λ)L1+λL D-SSIM ; Where λ is a constant, L1 loss is the absolute value of the difference between the rendered image and the original image pixel by pixel, and L D-SSIM is the multi-view structural similarity loss.

6. The 3D Gaussian splash model copyright protection method according to claim 2, characterized in that: The noise of the noise attack training includes: additive Gaussian noise with a mean of μ and a standard deviation of v; random axis-angle rotation with a parameter of α; random scaling with a parameter of s; and Gaussian blur with a kernel of k.

7. The 3D Gaussian splash model copyright protection method according to claim 1, characterized in that: The message extractor comprises a convolutional layer, a flat layer and a dense layer, wherein the dense layer is activated by a rectified linear unit function, a noise-processed rendering image is input, and watermark information is output.

8. The 3D Gaussian splash model copyright protection method according to claim 1, characterized in that: The protection method also includes optimizing the extracted watermark information through a classifier.

Images