Copyright protection method for 3DGS three-dimensional scene and objects in scene

Abstract

The invention discloses a copyright protection method for a 3DGS three-dimensional scene and objects in the scene, and belongs to the technical field of three-dimensional reconstruction. Target objects needing to be protected are automatically extracted from a multi-view image data set through an image sequence segmentation model; combining the original image with the target object image to construct a new data set for watermark network training, and carrying out network training; performing high-quality scene reconstruction on the multi-view image by adopting a 3DGS method; a hidden information loss function is introduced to finely adjust scene color information and a decoder of a watermark network, and copyright watermark embedding of the whole scene and a target object is achieved; the frequency domain consistency loss and the image difference loss are combined and optimized, and the quality of the reconstruction model after the watermark is embedded is ensured. According to the method, copyright protection of a three-dimensional scene level can be realized, fine-grained protection can be carried out on specific objects in the scene, and the method has relatively high practicability and robustness and is suitable for application scenes such as digital asset management and virtual reality content protection.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a copyright protection method for a 3DGS three-dimensional scene and objects within the scene. Background Art

[0002] New perspective synthesis has always been a key task in computer vision, with widespread applications in virtual reality, augmented reality, game development, and digital content creation. Currently, 3DGS, with its outstanding real-time rendering performance and high-precision reconstruction quality, has rapidly become a mainstream technical solution for 3D scene reconstruction tasks.

[0003] Unlike traditional meshes and point clouds, traditional 3D representations are mostly specific to specific objects. However, 3DGS (3D Gaussian Splatting) reconstructions typically focus on the entire scene, with watermarks embedded within the scene itself; objects within the scene do not carry the watermark. Existing 3DGS watermarking strategies struggle to embed watermarks on key objects within the scene. When objects within the scene are segmented and reused, model copyright information cannot be verified. In practical applications, such as virtual reality, digital asset trading, and content reuse, specific objects within the scene often require independent access or reuse. These issues severely limit the applicability and integrity of existing 3DGS copyright protection strategies. While researchers have proposed various innovative scene copyright protection methods, attempting to enhance robustness and concealment through improved watermark embedding strategies, these approaches have generally failed to effectively address the lack of independent watermarks for key objects within the scene. Furthermore, the embedded watermark inevitably compromises reconstruction quality, reducing the authenticity of the reconstructed scene. In addition, the embedded watermark is easily invalidated after undergoing common image processing operations (such as compression, blurring, occlusion, etc.), further weakening its stability and usability in practical applications. Summary of the Invention

[0004] In order to solve the problem that traditional copyright protection solutions only target the overall reconstructed scene but have no protection capabilities for objects in the scene, thus limiting their widespread application; in order to solve the problem that traditional watermark embedding technology often causes a significant decrease in visual quality, affecting the authenticity and visualization of the reconstructed scene; and at the same time, in order to solve the problem that traditional embedded watermarking technology is insufficiently robust and has poor capabilities to cope with common attacks and image perturbations, thus limiting its reliability in practical applications. The present invention provides a copyright protection method for 3DGS three-dimensional scenes and objects within the scenes. The present invention uses the overall scene and target object data sets to train a watermark network; uses 3DGS to perform three-dimensional reconstruction on the scene data set to obtain a reconstructed scene; uses hidden information loss to fine-tune the scene color information and the decoder of the watermark network to achieve copyright watermark embedding of the overall scene and target object; uses frequency domain consistency loss and image difference loss for joint optimization to ensure the quality of the reconstructed model after watermark embedding; and finally achieves a copyright protection method for the reconstructed scene and objects within the scene.

[0005] The technical solution adopted by the copyright protection method of the present invention for 3DGS three-dimensional scenes and objects within the scenes is:

[0006] A copyright protection method for a 3DGS three-dimensional scene and objects in the scene, comprising the following steps:

[0007] S1. Use the image sequence segmentation model to segment the target objects in the scene dataset, obtain target object images from all perspectives, and merge the scene images and target objects to construct a new dataset for the watermark network;

[0008] S2. Use the frequency domain information of the image to train the watermark network, add an upsampling layer in the encoder of the watermark network, and add an adaptive pooling layer in the decoder. This supports watermark network training for data sets of any size and embeds the watermark information into the frequency domain of the image.

[0009] S3. Use the Colmap open source software package to reconstruct dense 3D point cloud data and camera position information from the image dataset and train a Gaussian kernel to represent the real 3D scene.

[0010] S4. Decoding the watermark information of the rendered image and the watermark information of the target object using a decoder, fine-tuning the scene color information and the decoder of the watermark network using hidden information loss, and embedding the copyright watermark of the entire scene and the target object;

[0011] S5. Use discrete wavelet transform to perform frequency domain decomposition on the rendered image, obtain frequency domain information, and use frequency domain consistency loss and image difference loss for joint optimization to ensure the quality of the reconstructed model after embedding the watermark.

[0012] A further improvement of the technical solution of the present invention is that: the step S2 is specifically,

[0013] An upsampling layer is introduced at the end of the watermark message processing branch to ensure that the embedded information maintains the same spatial resolution as the covered image; at the same time, an adaptive average pooling layer is added at the end of the decoder, and the linear projection module is used to restore the decoded watermark result to the original dimension to ensure the consistency and accuracy of information extraction.

[0014] A further improvement of the technical solution of the present invention is that the Gaussian kernel in step S3 includes position μ, covariance matrix Σ, transparency α and spherical harmonic function SH information, the color information is represented by spherical harmonic function, and each Gaussian kernel is defined as,

[0015]

[0016] Where x represents any point in the three-dimensional scene, μ is the center position of the Gaussian, and Σ is the covariance matrix used to represent the rotation and scaling information of the Gaussian in space; its projection covariance matrix Σ' in the two-dimensional image space is calculated by the following formula,

[0017] Σ'=JW T ∑WJ T (1.2)

[0018] Among them, J represents the Jacobian matrix corresponding to the affine approximation of the perspective transformation under the current projection, which is used to capture the local deformation characteristics when mapping from three-dimensional to two-dimensional space; under a specific camera pose, 3DGS can project the 3D Gaussian kernel to the 2D imaging plane, and then superimpose the colors of the N ordered Gaussian kernels mapped to the pixel through the α blending method to calculate the color of the pixel. The calculation formula is,

[0019]

[0020] Among them, c represents the calculated pixel color, c i Represents the color value of each Gaussian kernel, α i Indicates the transparency of each Gaussian kernel for its transparency.

[0021] A further improvement of the technical solution of the present invention is that: the step S4 specifically includes the following steps:

[0022] S4.1. Use the 3DGS rendering pipeline to render the image and add differentiable image attacks on the image and the protected object to enhance the robustness of the watermark.

[0023] S4.2. Fine-tuning the pre-trained 3D Gaussian model using the watermark information loss function Decoder in watermark network This ensures that watermark information can be stably embedded in images rendered from any perspective.

[0024] A further improvement of the technical solution of the present invention is that: the step S4.1 is specifically,

[0025] Render the image using the 3D Gaussian renderer in 3DGS Then, given a rendered image and the mask M of the kth target k , obtain the corresponding target area through element-by-element multiplication operation,

[0026]

[0027] Then, all extracted object regions are combined with the complete scene image to form a mixed input batch.

[0028]

[0029] Among them, K represents the number of objects to be protected in the scene; in order to simulate potential disturbances and improve the robustness of watermark extraction, the above batches are processed with reference to the overall scene watermark embedding process. First pass through the differentiable noise perturbation layer Data enhancement is performed; the perturbed image is then decomposed into multi-scale frequency features by the discrete wavelet transform operator Γ(·) and the watermark decoder Extract the embedded watermark from the frequency representation.

[0030] A further improvement of the technical solution of the present invention is that: the step S4.2 is specifically,

[0031] Input the frequency domain features of the mixed input batch to the decoder D m Used to predict embedded watermark information Finally, the predicted watermark is calculated by the mean square error loss function The difference between the predicted information and the true watermark M, the loss function is defined as follows, which is used to measure the mean square error between the predicted information and the true watermark information,

[0032]

[0033] A further improvement of the technical solution of the present invention is that: the step S5 is specifically,

[0034] The frequency domain annealing regularization strategy based on discrete wavelet transform (DWT) gradually guides the model to transition from low-frequency information to medium- and high-frequency information during the training process. That is, in the early stage of training, low-frequency approximation coefficients are prioritized, and then high-frequency sub-bands are gradually introduced to realize a coarse-to-fine supervision mechanism, thereby improving the balance between rendering quality and watermark embedding. The frequency domain annealing regularization strategy based on discrete wavelet transform is consistent with the frequency domain-based watermark network embedding extraction process, which can reduce the impact of watermark on image perception quality while ensuring watermark robustness and high extraction accuracy.

[0035] A further improvement of the technical solution of the present invention is that the activation of the frequency level is controlled by a time-varying function α(t), where t∈[T0,T] represents the number of training iterations; the frequency band level α(t) activated in the tth iteration is defined as follows,

[0036]

[0037] Among them, T i It is the phase time base, every interval T i Each iteration activates a new wavelet level in turn; for the rendered image The corresponding real image The J-layer DWT wavelet decomposition is expressed as,

[0038]

[0039] in, With L J represents the low-frequency subband of layer J, and Indicates the corresponding three high-frequency subbands b∈LH,HL,HH in the jth layer;

[0040] The frequency domain reconstruction loss function of step-by-step activation is defined as follows:

[0041]

[0042] Due to the adoption of the above technical solution, the technical advancements achieved by the present invention include:

[0043] The present invention can achieve reliable ownership verification even when the target object is extracted and reused separately; the present invention designs a frequency-domain-based watermark embedding network, which can effectively alleviate the distortion problem of image quality caused by spatial domain embedding, and has good cross-dataset generalization ability and is applicable to images of arbitrary resolution; at the same time, the present invention can gradually optimize the low-frequency structure and high-frequency details of the image during the training process, achieving a good balance between rendering fidelity and watermark robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1It is a flow chart of a copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to the present invention;

[0045] Figure 2 It is a digital watermark embedding result of a watermark embedding network based on spatial domain and frequency domain for a copyright protection method of a 3DGS three-dimensional scene and objects in the scene according to the present invention;

[0046] Figure 3 This is a schematic diagram of the network structure of a watermark embedding network for a copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to the present invention;

[0047] Figure 4 It is the watermark embedding and extraction result of the whole scene of the copyright protection method for 3DGS three-dimensional scene and objects in the scene of the present invention;

[0048] Figure 5 It is a watermark embedding extraction result of the whole scene and the objects in the scene according to a copyright protection method for 3DGS three-dimensional scene and objects in the scene of the present invention;

[0049] Figure 6 The present invention is a flow chart of a method for watermark embedding and scene reconstruction quality optimization for a copyright protection method of a 3DGS three-dimensional scene and objects in the scene. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. In the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0051] like Figure 1 As shown, the present invention specifically includes:

[0052] 1. Construction of watermark network dataset:

[0053] This method uses the image sequence segmentation model CUTIE to automatically extract target objects from multi-view image sequences. CUTIE leverages the spatiotemporal consistency and structural information between images to accurately segment target regions that are consistent across frames. After segmentation, CUTIE automatically outputs a masked region of the target object in each frame, which is then combined with the original image to form a dataset sample.

[0054] 2. Watermark network training:

[0055] The present invention uses a deep neural network (DNN) with adversarial training and channel attention mechanism to embed and extract watermark information. In order to solve the image artifacts caused by embedding watermarks in the spatial domain of large-scale images, the present invention transfers the watermark embedding process from the spatial domain to the frequency domain. The embedding results of different domains are as follows: Figure 2 In the embedding stage, the original image I is first decomposed into several frequency sub-bands by discrete wavelet transform (DWT), and then the frequency domain features are combined with the watermark message to be embedded M = {m1,m2,...m n}∈{0,1} N Input the encoder together for joint embedding. Figure 1 The comparison shows the image effects after spatial domain and frequency domain watermark embedding. The results show that spatial domain embedding has obvious artifacts, while frequency domain embedding can significantly reduce such distortion.

[0056] In the model training phase, the constructed training dataset consists of the whole image I={I1,I2,...I n} and its corresponding segmentation target area Composition, where ε i,j Represents image I i The jth target area extracted from the image is obtained. In response to the diverse image resolution requirements, the encoder-decoder structure is improved to better adapt to high-resolution image input. Specifically, an upsampling layer is introduced at the end of the watermark message processing branch to ensure that the embedded information and the covered image maintain the same spatial resolution; at the same time, an adaptive average pooling layer is added at the end of the decoder, and the linear projection module is used to restore the decoded watermark result to the original dimension to ensure the consistency and accuracy of information extraction. The network structure of the watermark network is as follows: Figure 3 shown.

[0057] 3. 3D scene reconstruction:

[0058] In the modeling phase of the present invention, COLMAP is first used to perform structured beam adjustment and multi-view stereo reconstruction on the image dataset of a given scene. After obtaining the camera parameters and the initial 3D structure, 3DGS models the scene using a series of 3D Gaussian distributions. Each Gaussian kernel contains position μ, covariance matrix Σ, transparency α, and spherical harmonics SH information. The color information is represented by spherical harmonics. Each Gaussian can be defined as:

[0059]

[0060] Where x represents an arbitrary point in the 3D scene, μ is the center of the Gaussian, and Σ is the covariance matrix, which represents the rotation and scale information of the Gaussian in space. Given an observation angle π, the rendering process first projects the 3D Gaussian into the 2D image space. Considering the Gaussian's 3D covariance matrix Σ and the camera's observation transformation matrix W, its projection covariance matrix Σ' in the 2D image space is calculated using the following formula:

[0061] ∑'=JW T ∑WJ T (1.2)

[0062] Where J represents the Jacobian matrix corresponding to the affine approximation of the perspective transformation under the current projection, which is used to capture the local deformation characteristics when mapping from 3D to 2D space. After the projection is completed, for each pixel in the image, all Gaussians contributing to that pixel are sorted by depth. The final color value is calculated by accumulation based on transparency, and its expression is as follows:

[0063]

[0064] Among them, c represents the calculated pixel color, c i Represents the color value of each Gaussian kernel, α i Indicates the transparency of each Gaussian kernel for its transparency.

[0065] 4. Digital watermark embedding:

[0066] In the present invention, the pre-trained three-dimensional Gaussian model Decoder in watermark network Fine-tuning is performed to ensure that the watermark information M={m1,m2,...m n}∈{0,1} N For the overall scene watermark embedding, during the training phase, only the decoder in the watermark network is used. From the rendered image To achieve fine-grained copyright protection of key objects in the scene, the corresponding area is extracted from the rendered image using a target-specific mask. Given a rendered image and the mask M of the kth target k , obtain the corresponding target area through element-by-element multiplication operation:

[0067]

[0068] All extracted object regions are then combined with the full scene image to form a mixed input batch:

[0069]

[0070] Where K represents the number of objects to be protected in the scene. In order to improve the ability of the watermark to be robustly extracted after experiencing multiple image attacks, a differentiable noise layer is introduced during the training process. Noise is added to the mixed input batch β to generate a noisy image batch. Subsequently, the noisy image batch is subjected to discrete wavelet transform Γ(·) to obtain the frequency feature. Finally, the frequency domain feature is input to the decoder Used to predict embedded watermark information The watermark loss function is defined as follows, which is used to measure the mean square error between the predicted information and the true watermark information:

[0071]

[0072] in, Indicates the calculation of the MSE loss between the two. The watermark extraction results of the overall scene and the target object are as follows Figure 4 、 Figure 5 As shown in the figure, the bit error rate of watermark extraction of rendered images under different image attacks is shown.

[0073] 5. Reconstruction quality optimization:

[0074] Since watermark embedding inevitably modifies the color information of the reconstructed scene, it has a certain impact on the rendering quality of the reconstructed model. To alleviate this problem, the present invention proposes a frequency-domain annealing regularization strategy based on discrete wavelet transform (DWT), which gradually guides the model to transition from low-frequency information to medium- and high-frequency information during the training process. That is, in the early stages of training, priority is given to low-frequency approximation coefficients, and then high-frequency sub-bands are gradually introduced to implement a coarse-to-fine supervision mechanism, thereby improving the balance between rendering quality and watermark embedding. This strategy is consistent with the frequency-domain-based watermark network embedding and extraction process, and can reduce the impact of the watermark on the perceived quality of the image while ensuring the robustness and high extraction accuracy of the watermark.

[0075] The activation of the frequency level is controlled by a time-varying function α(t), where t∈[T0,T] represents the number of training iterations. The frequency band level α(t) activated in the tth iteration is defined as follows:

[0076]

[0077] Among them, T i It is the phase time base, each interval T i Each iteration activates a new wavelet level. The corresponding real image The J-layer DWT wavelet decomposition is expressed as:

[0078]

[0079] in, With L J represents the low-frequency subband of layer J, and Represents the corresponding three high-frequency sub-bands b∈LH,HL,HH in the jth layer.

[0080] The frequency domain reconstruction loss function of the step-by-step activation is defined as follows:

[0081]

[0082] The overall process of watermark embedding and reconstruction quality optimization is as follows Figure 6 shown.

[0083] In the above-mentioned embodiments, the present invention provides a copyright protection method for 3DGS three-dimensional scenes and objects within the scenes. The present invention implements copyright protection for 3DGS reconstructed scenes and objects within the scenes; designs a digital watermark network that supports data of any size; modifies the embedding domain of the watermark network to reduce obvious artifacts produced in the image after watermark embedding; during the watermark embedding process, objects in the data set are segmented and watermarks are embedded in the target objects to achieve protection of the entire scene and the target objects; proposes a frequency domain-based annealing loss to gradually improve the frequency domain information of the reconstructed scene, thereby ensuring the quality of the scene reconstruction after watermark embedding; and finally implements a copyright protection method for 3DGS three-dimensional scenes and objects within the scenes, thereby providing copyright protection for the reconstructed scene and the target objects within the scene.

[0084] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the concept and scope of the present invention. Any modifications and improvements made to the technical solution of the present invention by a person of ordinary skill in the art without departing from the design concept of the present invention shall fall within the scope of protection of the present invention. The technical content for which protection is sought in the present invention is fully set forth in the claims.

Claims

1. A copyright protection method for a 3DGS three-dimensional scene and objects within the scene, characterized by: The following steps are included: S1. Use the image sequence segmentation model to segment the target objects in the scene dataset, obtain target object images from all perspectives, and merge the scene images and target objects to construct a new dataset for the watermark network; S2. Use the frequency domain information of the image to train the watermark network, add an upsampling layer in the encoder of the watermark network, and add an adaptive pooling layer in the decoder. This supports watermark network training for data sets of any size and embeds the watermark information into the frequency domain of the image. S3. Use the Colmap open source software package to reconstruct dense 3D point cloud data and camera position information from the image dataset and train a Gaussian kernel to represent the real 3D scene. S4. Decoding the watermark information of the rendered image and the watermark information of the target object using a decoder, fine-tuning the scene color information and the decoder of the watermark network using hidden information loss, and embedding the copyright watermark of the entire scene and the target object; S5. Use discrete wavelet transform to perform frequency domain decomposition on the rendered image, obtain frequency domain information, and use frequency domain consistency loss and image difference loss for joint optimization to ensure the quality of the reconstructed model after embedding the watermark.

2. The copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to claim 1, characterized in that: The step S2 is specifically as follows: An upsampling layer is introduced at the end of the watermark message processing branch to ensure that the embedded information maintains the same spatial resolution as the covered image; at the same time, an adaptive average pooling layer is added at the end of the decoder, and the linear projection module is used to restore the decoded watermark result to the original dimension to ensure the consistency and accuracy of information extraction.

3. The copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to claim 1, characterized in that: In step S3, the Gaussian kernel includes position μ, covariance matrix ∑, transparency α and spherical harmonic function SH information. The color information is represented by spherical harmonic function. Each Gaussian kernel is defined as: Where x represents any point in the three-dimensional scene, μ is the center position of the Gaussian, and Σ is the covariance matrix used to represent the rotation and scaling information of the Gaussian in space; its projection covariance matrix Σ' in the two-dimensional image space is calculated by the following formula, In=JW T ŚWJ T (1.2) Among them, J represents the Jacobian matrix corresponding to the affine approximation of the perspective transformation under the current projection, which is used to capture the local deformation characteristics when mapping from three-dimensional to two-dimensional space; under a specific camera pose, 3DGS can project the 3D Gaussian kernel to the 2D imaging plane, and then superimpose the colors of the N ordered Gaussian kernels mapped to the pixel through the α blending method to calculate the color of the pixel. The calculation formula is, Among them, c represents the calculated pixel color, c i Represents the color value of each Gaussian kernel, α i Indicates the transparency of each Gaussian kernel for its transparency.

4. The copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to claim 1, characterized in that: The step S4 specifically includes the following steps: S4.

1. Use the 3DGS rendering pipeline to render the image and add differentiable image attacks on the image and the protected object to enhance the robustness of the watermark. S4.

2. Fine-tuning the pre-trained 3D Gaussian model using the watermark information loss function Decoder in watermark network This ensures that watermark information can be stably embedded in images rendered from any perspective.

5. The copyright protection method for 3DGS three-dimensional scenes and objects within the scenes according to claim 4, characterized in that: The step S4.1 is specifically as follows: Render the image using the 3D Gaussian renderer in 3DGS Then, given a rendered image and the mask M of the kth target k , obtain the corresponding target area through element-by-element multiplication operation, Then, all extracted object regions are combined with the complete scene image to form a mixed input batch. Among them, K represents the number of objects to be protected in the scene; in order to simulate potential disturbances and improve the robustness of watermark extraction, the above batches are processed with reference to the overall scene watermark embedding process. First pass through the differentiable noise perturbation layer Data enhancement is performed; the perturbed image is then decomposed into multi-scale frequency features by the discrete wavelet transform operator Γ(·) and the watermark decoder Extract the embedded watermark from the frequency representation.

6. The copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to claim 4, characterized in that: The step S4.2 is specifically as follows: Input the frequency domain features of the mixed input batch to the decoder Used to predict embedded watermark information Finally, the predicted watermark is calculated by the mean square error loss function The difference between the predicted information and the true watermark M, the loss function is defined as follows, which is used to measure the mean square error between the predicted information and the true watermark information, 7. The copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to claim 1, characterized in that: The step S5 is specifically as follows: The frequency domain annealing regularization strategy based on discrete wavelet transform (DWT) gradually guides the model to transition from low-frequency information to medium- and high-frequency information during the training process. That is, in the early stage of training, low-frequency approximation coefficients are prioritized, and then high-frequency sub-bands are gradually introduced to realize a coarse-to-fine supervision mechanism, thereby improving the balance between rendering quality and watermark embedding. The frequency domain annealing regularization strategy based on discrete wavelet transform is consistent with the frequency domain-based watermark network embedding extraction process, which can reduce the impact of watermark on image perception quality while ensuring watermark robustness and high extraction accuracy.

8. The copyright protection method for a 3DGS three-dimensional scene and objects within the scene according to claim 7, characterized in that: The activation of the frequency level is controlled by a time-varying function α(t), where t∈[T0,T] represents the number of training iterations; the frequency band level α(t) activated in the tth iteration is defined as follows, Among them, T i It is the phase time base, each interval T i Each iteration activates a new wavelet level in turn; for the rendered image The corresponding real image The J-layer DWT wavelet decomposition is expressed as, in, With L J represents the low-frequency subband of layer J, and Indicates the corresponding three high-frequency subbands b∈LH,HL,HH in the jth layer; The frequency domain reconstruction loss function of step-by-step activation is defined as follows:

Images