Robust watermarking method for three-dimensional deformable grid model

Abstract

The invention provides a robust watermarking method for a three-dimensional deformable grid model. The robust watermarking method comprises the following steps: calculating a grid saliency map of a three-dimensional grid model by utilizing a saliency map calculation module; encoding the watermark information into a watermark vector through a full connection layer by using the watermark information encoding network, and splicing the watermark vector with the original three-dimensional model to form a fused matrix vector; encoding a matrix vector fused by the watermark information and the three-dimensional grid model by using a watermark embedding network, and guiding a watermark embedding process through a grid saliency map; obtaining a three-dimensional grid model with a watermark signal; utilizing an attack network to simulate various attacks on the three-dimensional grid model embedded with the watermark; and extracting watermark information from the three-dimensional grid model under various attacks by using a watermark extraction network. According to the method, through significance mapping of the three-dimensional grid, the watermark embedding strength can be adjusted, the invisibility of the watermark is remarkably improved, and meanwhile various attacks can be resisted. According to the method, relatively high watermark robustness and relatively small watermark embedding distortion are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of robust watermarking, and in particular to a robust watermarking method for three-dimensional deformable mesh models. Background Art

[0002] With the development of computer graphics and advanced intelligent manufacturing, three-dimensional deformable models are widely used in industrial production, medicine, the film industry, the video game industry, computer-aided design, and other fields. A large number of three-dimensional models are exchanged and circulated on the Internet, which poses a huge challenge to the copyright protection of three-dimensional models. As an effective copyright protection measure, digital watermarking technology has been widely used, which can add copyright information to three-dimensional models in an invisible way. To effectively protect three-dimensional models, a watermarking method with both robustness and invisibility is achieved by using graph convolutional networks.

[0003] Three-dimensional deformable models are a special form of mesh with a shared template, that is, a fixed topology. This allows them to match different instances through deformation, such as human faces, bodies, hands, and animal movements. However, creating a three-dimensional deformable model dataset is not only time-consuming and costly, but also extremely easy to copy and spread in the digital age, which directly threatens the rights and interests of digital product owners. Therefore, it is particularly important to protect the copyright of three-dimensional deformable models.

[0004] For three-dimensional deformable robust watermarking, existing three-dimensional mesh watermarking algorithms based on deep learning usually use the same strategy to penalize the displacements of different vertices in the three-dimensional mesh when embedding watermarks, thus limiting their performance in terms of invisibility. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides a robust watermarking method for three-dimensional deformable mesh models. The present invention adopts different strategies for the displacements of vertices with different saliencies in the mesh, thereby achieving high robustness and high invisibility.

[0006] The technical solution of the present invention is: a robust watermarking method for three-dimensional deformable mesh models, including the following steps:

[0007] S1), construct a saliency map calculation module, and use the saliency map calculation module to calculate the mesh saliency map of the three-dimensional mesh model;

[0008] S2), use the watermark information encoding network to encode the watermark information into a watermark vector through a fully connected layer,

[0009] and splice it with the original three-dimensional model to form a fused matrix vector;

[0010] S3), encode the matrix vectors obtained by fusing the watermark information and the 3D mesh model using the watermark embedding network, then calculate the saliency map for the vertices of the 3D mesh model, and fuse the watermark information with the vertices with low mesh saliency; calculate the mesh saliency map for the entire 3D mesh model in each iteration process in turn, and then complete the process of guiding watermark embedding; obtain the 3D mesh model with the watermark signal;

[0011] S4), use the attack network to simulate various attacks on the 3D mesh model with the embedded watermark;

[0012] S5), use the watermark extraction network to extract the watermark information from the 3D mesh models under various attacks.

[0013] Preferably, in step S1), construct a saliency map calculation module, and use the saliency map calculation module to calculate the mesh saliency map, which specifically includes the following steps:

[0014] S11), calculate the curvature K of each vertex of the 3D mesh model h , and determine the salient region by identifying the region that is significantly different from the surrounding environment;

[0015]

[0016] where A i is the area of the Voronoi region of vertex v i ; N i is the set of neighboring vertices of vertex v i ; v j is a neighboring vertex of v i ; w ij is the weight of the edge between vertex v i , v j ; n i is the normal vector of vertex v i ;

[0017] S12), use the search bounding box length to define the benchmark for different scale levels of the 3D mesh, and calculate the Gaussian weighted average G(K h (v), σ i ) of the average curvature at different scales;

[0018]

[0019] where σ i = {2∈, 3∈, 4∈,... | ∈ = L×0.003}, L is the border length of the 3D mesh model; N(v, σ i ) = {x|||x - v|| < σ i} represents the set of points whose Euclidean distance to vertex v is within σ iA set of points within; x represents the vertex element in the set;

[0020] S13) After normalizing and non-linearly mapping the Gaussian weighted average value, the saliency regions at different scales are accumulated to obtain the saliency map distribution of the three-dimensional mesh model.

[0021] Preferably, in step S1), the saliency loss function l of the saliency map calculation module saliency is:

[0022]

[0023] where, represents the saliency value of the i-th vertex; N represents the total number of vertices; V enc,i represents the i-th vertex after embedding the watermark, V in,i represents the i-th input vertex.

[0024] Preferably, in step S2), the watermark information encoding network uses a multi-layer perceptron and the Tanh activation function to map the watermark information into a coding vector.

[0025] W out = Tanh(W in , encoder)

[0026] where, encoder represents the fully connected layer, W in is the input watermark, and W out is the output watermark vector.

[0027] Preferably, in step S2), the watermark information is encoded into a watermark vector through the fully connected layer by using the watermark encoding network, which specifically includes the following steps:

[0028] S21) Input the watermark information into the fully connected layer network and output potential coding watermark vectors of different dimensions as training data;

[0029] S22) Iteratively train and adjust the parameters of the watermark embedding network so that the generated watermark coding vector can better represent the watermark information in the latent space.

[0030] Preferably, in step S3), the watermark information and the model are encoded by using the watermark embedding network to obtain a three-dimensional model with a watermark signal, which specifically includes the following steps:

[0031] S31) The watermark embedding network performs a downsampling operation on the fused matrix vector, fuses the mesh model and the watermark feature, and obtains a robust representation of the watermark information;

[0032] S32) Upsample the fused three-dimensional grid model and watermark features, and restore the three-dimensional grid model layer by layer; after upsampling, output the three-dimensional model with watermark information through a fully connected layer.

[0033] Preferably, in step S3), the above-mentioned upsampling and downsampling perform feature fusion through a graph attention mechanism.

[0034] Preferably, in step S3), the watermark embedding loss l of the watermark embedding network enc is:

[0035]

[0036] where N represents the total number of vertices, V enc represents the vertex after embedding the watermark, and V in represents the input vertex.

[0037] Preferably, in step S4), use the attack network to simulate various attacks on the three-dimensional model with the embedded watermark, specifically:

[0038] S41) Design a fully connected network as the parameter of the attack network;

[0039] S42) Use the three-dimensional grid model with the watermark as the input, and use the perturbation acting on the three-dimensional grid model as a simulated attack, and add it to the three-dimensional grid model with the watermark to simulate the attack in the real scenario.

[0040] Preferably, in step S5), the watermark extraction network uses a multi-level structure to generate an intermediate representation of the grid, and processes the output result through a multi-layer perceptron. The watermark extraction accuracy is used as the loss function of the watermark extraction network.

[0041] Preferably, in step S5), use the watermark extraction network to extract the watermark information from the three-dimensional grid model under various attacks, specifically:

[0042] S51) Design a multi-layer fully connected layer as the watermark extraction network;

[0043] S52) Use the three-dimensional grid model of various attacks as the input of the watermark extraction network, and extract the watermark features through the watermark extraction network;

[0044] S53) Input the watermark feature information into the activation layer to obtain the final watermark vector.

[0045] Preferably, in step S5), the watermark extraction loss l of the watermark extraction network dec is:

[0046]

[0047] Among them, λ att represents the attack loss; W att represents the watermark extracted after passing through the attack network; W out represents the watermark information output by the decoder; W in is the input watermark

[0048] Preferably, the overall loss function L is defined as:

[0049] L = λ enc l enc + λ saliency l saliency + λ dec l dec

[0050] Among them, λ enc represents the embedding loss hyperparameter; l enc represents the watermark embedding loss; λ saliency represents the saliency loss hyperparameter; l saliency represents the saliency loss function; λ dec represents the decoding loss hyperparameter;

[0051] l dec is the watermark extraction loss.

[0052] The beneficial effects of the present invention are as follows:

[0053] 1. By introducing the saliency mapping of the three-dimensional mesh, the present invention can adjust the intensity of watermark embedding, significantly improving the invisibility of the watermark and at the same time being able to resist various attacks;

[0054] 2. The loss function designed by the present invention can effectively measure the overall impact of the deformation of different regions of the three-dimensional mesh on human visual perception and can also be used as an evaluation index for the degree of visual saliency modification;

[0055] 3. The present invention achieves high watermark robustness and small watermark embedding distortion, and can realize the copyright protection of three-dimensional deformable models. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic flow chart of the robust watermark method of the present invention;

[0057] Figure 2 is a downsampling flow chart of the three-dimensional model watermark embedding network of the present invention;

[0058] Figure 3 is a schematic diagram of the watermark information encoding network of the present invention;

[0059] Figure 4This is the saliency distribution map of the 3D mesh model of the present invention;

[0060] Figure 5 This is the effect diagram of the 3D mesh model of the present invention when it is under different attacks. Detailed implementation manners

[0061] The following further describes the detailed implementation manners of the present invention with reference to the accompanying drawings:

[0062] As Figure 1 shown, this embodiment provides a robust watermarking method for a 3D deformable mesh model, including the following steps:

[0063] S1), construct a saliency map calculation module, and use the saliency map calculation module to calculate the mesh saliency map of the 3D mesh model; specifically including the following steps:

[0064] S11), calculate the curvature K of each vertex of the 3D mesh model h , and determine the saliency region by identifying the region that is significantly different from the surrounding environment;

[0065]

[0066] wherein, A i is the area of the Voronoi region of vertex v i ; N i is the set of neighboring vertices of vertex v i ; v j is a neighboring vertex of v i ; w ij is the weight of the edge between vertex v i , v j ; n i is the normal vector of vertex v i ;

[0067] S12), use the search bounding box length to define the benchmark of different scale levels of the 3D mesh, and calculate the Gaussian weighted average G(K h (v), σ i ) of the average curvature at different scales;

[0068]

[0069] wherein, σ i = {2∈, 3∈, 4∈,... | ∈ = L×0.003}, L is the border length of the 3D mesh model; N(v, σ i ) = {x|||x - v|| < σ i} represents the set of points whose Euclidean distance to vertex v is within σ i ; x represents the vertex element in the set;

[0070] S13), After normalizing and non-linearly mapping the Gaussian weighted average value, the saliency regions at different scales are accumulated to obtain the saliency map distribution of the three-dimensional mesh model, as Figure 4 shown.

[0071] In this embodiment, the saliency loss function l of the saliency map calculation module saliency is:

[0072]

[0073] where, represents the saliency value of the i-th vertex; N represents the total number of vertices; V enc,i represents the i-th vertex after embedding the watermark, V in,i represents the i-th input vertex.

[0074] S2), Use the watermark information encoding network to encode the watermark information into a watermark vector through a fully connected layer, and splice it with the original three-dimensional model to form a fused matrix vector;

[0075] The watermark information encoding network of this embodiment is as Figure 3 shown; The watermark information encoding network is used to map the input binary watermark information into a coded vector; In order to achieve efficient coding and accelerated convergence, the watermark information encoding network adopts a multi-layer perceptron and uses the Tanh activation function to map the watermark information into a coded vector, aligning the watermark information with the three-dimensional mesh model data so that it can be trained with the three-dimensional model in the same dimension.

[0076] W out = Tanh(W in , encoder)

[0077] where, encoder represents the fully connected layer, W in is the input watermark, and W out is the output watermark vector.

[0078] Encoding the watermark information into a watermark vector by using the watermark encoding network specifically includes the following steps:

[0079] S21), Input the watermark information into the fully connected layer network and output potential coded watermark vectors of different dimensions as training data;

[0080] S22), Iteratively train and adjust the parameters of the watermark embedding network so that the generated watermark coded vector can better represent the watermark information in the latent space.

[0081] S3). Encode the matrix vector obtained by fusing the watermark information and the 3D mesh model using the watermark embedding network, then calculate the saliency map for the vertices of the 3D mesh model, and fuse the watermark information with the vertices with low mesh saliency; calculate the mesh saliency map for the entire 3D mesh model in each iteration in turn, and then complete the process of guiding watermark embedding; obtain the 3D mesh model with the watermark signal; in this embodiment, the input includes the latent encoding and the vertices of the original mesh, where the vertex features only include spatial coordinates, and a five-level hierarchical structure with a sampling factor of 4 is adopted; specifically, it includes the following steps:

[0082] S31). The watermark embedding network performs downsampling on the fused matrix vector, fuses the mesh model and the watermark features, and obtains a robust representation of the watermark information; the process of downsampling is as Figure 2 shown;

[0083] S32). Perform upsampling on the fused 3D mesh model and the watermark features, and gradually recover the 3D mesh model; after upsampling, output the 3D mesh model with the watermark information through a fully connected layer.

[0084] In this embodiment, the above upsampling and downsampling perform feature fusion through the graph attention mechanism.

[0085] In this embodiment, the watermark embedding loss l enc of the watermark embedding network is:

[0086]

[0087] where N represents the total number of vertices, V enc represents the vertices after embedding the watermark, and V in represents the input vertices.

[0088] S4). Use the attack network to simulate various attacks on the 3D mesh model with the embedded watermark; specifically:

[0089] S41). Design a fully connected network as the parameters of the attack network;

[0090] S42). Use the 3D mesh model with the watermark as the input, and use the perturbation acting on the 3D mesh model as the simulated attack, and add it to the 3D mesh model with the watermark to simulate the attack in the real scenario. As Figure 5 shown. The attacks in this embodiment include affine transformation, Gaussian noise, cropping, smoothing, and simplification attacks, etc., to enable more robust watermark extraction for the subsequent extraction network.

[0091] S5). Use the watermark extraction network to extract the watermark information from the 3D mesh models under various attacks, specifically:

[0092] S51), designing a multi-layer fully-connected layer as the watermark extraction network;

[0093] S52), using 3D mesh models of various attacks as the input of the watermark extraction network, and extracting watermark features through the watermark extraction network;

[0094] S53), inputting the watermark feature information into the activation layer to obtain the final watermark vector.

[0095] The described watermark extraction network uses a multi-level structure to generate an intermediate representation of the mesh and processes the output result through a multi-layer perceptron. The watermark extraction accuracy is used as the loss function of the watermark extraction network. The watermark extraction loss l dec is:

[0096]

[0097] where λ att represents the attack loss; W att represents the watermark extracted after passing through the attack network; W out represents the watermark information output by the decoder; W in is the input watermark

[0098] Finally, the overall loss function L is defined as:

[0099] L = λ emc l enc + λ saliency l saliency + λ dec l dec

[0100] where λ enc represents the embedding loss hyperparameter; l enc represents the watermark embedding loss; λ salienct represents the saliency loss hyperparameter; l saliency represents the saliency loss function; λ dec represents the decoding loss hyperparameter;

[0101] l dec is the watermark extraction loss.

[0102] The above embodiments and descriptions in the specification only illustrate the principles and the best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed.

Claims

1. A robust watermarking method for three-dimensional deformable mesh models, characterized in that, It includes the following steps: S1), construct a saliency map calculation module, and use the saliency map calculation module to calculate the grid saliency map of the three-dimensional mesh model; S2), use the watermark information encoding network to encode the watermark information into a watermark vector through a fully connected layer, and splice it with the original three-dimensional model to form a fused matrix vector; S3), use the watermark embedding network to encode the fused matrix vector of the watermark information and the three-dimensional mesh model, then calculate the saliency map for the vertices of the three-dimensional mesh model, and fuse the watermark information with the vertices with small grid saliency; calculate the grid saliency map for the entire three-dimensional mesh model in each iteration process in turn, and then complete the process of guiding watermark embedding; Obtain a three-dimensional mesh model with a watermark signal; S4), use the attack network to simulate various attacks on the three-dimensional mesh model embedded with the watermark; S5), use the watermark extraction network to extract the watermark information from the three-dimensional mesh models under various attacks.

2. The robust watermarking method for a three-dimensional deformable mesh model according to claim 1, characterized in that: In step S1), construct a saliency map calculation module, and use the saliency map calculation module to calculate the grid saliency map, which specifically includes the following steps: S11), calculating the curvature K of each vertex of the three-dimensional mesh model h , and determining the significant region by identifying the region that is significantly different from the surrounding environment; S12), using the search bounding box length as a benchmark for defining different scale levels of the three-dimensional grid, calculate the Gaussian weighted average G(K h (v), σ i ) S13), after normalizing and non-linearly mapping the Gaussian weighted average value, accumulate the saliency regions at different scales to obtain the saliency map distribution of the three-dimensional mesh model.

3. A robust watermarking method for three-dimensional deformable mesh models according to claim 2, characterized in that: In step S11), the curvature K of each vertex h is calculated by the following expression: Among them, A i is the area of the Voronoi region of vertex v i ; N i is the set of neighboring vertices of vertex v i ; v j is a neighboring vertex of v i ; w ij is the weight of the edge between vertices v i , v j ; n i is the normal vector of vertex v i ; In step S12), the Gaussian weighted average value G(K h (v), σ i ) is calculated by the following formula: Among them, σ i ={2∈, 3∈, 4∈, …|∈ = L×0.003}, where L is the border length of the three-dimensional grid model; N(v, σ i )={x|||x - v|| < σ i} represents the set of points whose Euclidean distance to vertex v is within σ i ; x represents the vertex element in the set.

4. A robust watermarking method for a three-dimensional deformable mesh model according to claim 1, characterized in that: In step S2), the watermark information encoding network uses a multi-layer perceptron and the Tanh activation function to map the watermark information into an encoded vector. W out = Tanh(W in , encoder) Among them, encoder represents a fully connected layer, and W in is the input watermark, and W out is the output watermark vector.

5. A robust watermarking method for a three-dimensional deformable mesh model according to claim 4, characterized in that: In step S2), use the watermark encoding network to encode the watermark information into a watermark vector through a fully connected layer, which specifically includes the following steps: S21), input the watermark information into the fully connected layer network, and output potential encoded watermark vectors of different dimensions as training data; S22), iteratively train and adjust the parameters of the watermark embedding network to make the generated watermark encoding vector better represent the watermark information in the latent space.

6. A robust watermarking method for a three-dimensional deformable mesh model according to claim 1, characterized in that: In step S3), use the watermark embedding network to encode the watermark information and the model to obtain a three-dimensional model with a watermark signal, which specifically includes the following steps: S31), the watermark embedding network performs a downsampling operation on the fused matrix vector, fuses the mesh model and the watermark features, and obtains a robust representation of the watermark information; S32), perform an upsampling operation on the fused three-dimensional mesh model and the watermark features, and layer by layer restore the three-dimensional mesh model; after upsampling, output a three-dimensional model with watermark information through a fully connected layer.

7. A robust watermarking method for three-dimensional deformable mesh models according to claim 1, characterized in that: In step S4), use the attack network to simulate various attacks on the three-dimensional model embedded with the watermark, specifically: S41), design a fully connected network as the parameters of the attack network; S42), use the three-dimensional mesh model with the watermark as the input, and use the perturbation acting on the three-dimensional mesh model as a simulated attack, and add it to the three-dimensional mesh model with the watermark to simulate the attack in the real scenario.

8. A robust watermarking method for three-dimensional deformable mesh models according to claim 1, characterized in that: In step S5), use the watermark extraction network to extract the watermark information from the three-dimensional mesh models under various attacks, specifically: S51), design a multi-layer fully connected layer as the watermark extraction network; S52), use the three-dimensional mesh models of various attacks as the input of the watermark extraction network, and extract the watermark features through the watermark extraction network; S53), input the watermark feature information into the activation layer to obtain the final watermark vector.

9. A robust watermarking method for a three-dimensional deformable mesh model according to claim 1, characterized in that: The overall loss function L of the method is defined as: L = λ enc l enc + λ saliency l saliency + λ dec l dec Among them, λ enc represents the embedding loss hyperparameter; λ saliency represents the saliency loss hyperparameter; λ dec represents the decoding loss hyperparameter; l enc represents the watermark embedding loss; l saliency represents the saliency loss function; l dec is the loss of watermark extraction.

10. A robust watermarking method for a three-dimensional deformable mesh model according to claim 9, characterized in that: The significance loss function l of the described significant graph calculation module saliency is as follows: Among them, represents the significance value of the i-th vertex; N represents the total number of vertices; V enc,i represents the i-th vertex after embedding the watermark, and V in,i represents the i-th input vertex;; The watermark embedding loss l of the watermark embedding network described above enc is as follows: where N represents the total number of vertices, and V enc represents the vertices after embedding the watermark, and V in represents the input vertices; The watermark extraction loss l of the described watermark extraction network dec is as follows: Among them, λ att represents the attack loss; W att represents the watermark extracted after passing through the attack network; W out represents the watermark information output by the decoder; W in is the input watermark.

Images