Methods, systems, and applications of watermark robustness enhancement based on antagonistic immunity model.

Abstract

This invention discloses a robustness enhancement method for model watermarking based on adversarial immunity, primarily addressing the problem that existing model watermarking methods are unable to resist various watermark removal attacks. The solution involves: using a public dataset suitable for image classification tasks and a convolutional neural network as the original data and target model, respectively; training the target model with the original data; using a generative model as the watermark generation model and training it with the original data; inputting the original data into the trained watermark generation model for forward propagation to output watermark data; concatenating the original data and watermark data and inputting the concatenated data into the trained target model for forward propagation to obtain the model watermark; adding minor perturbations to the original data to generate adversarial examples, concatenating these examples with the original data, and inputting them into the model watermark for robustness training, resulting in an enhanced robust model watermark. This invention effectively enhances the generalization ability of model watermarking under different data distributions, improves model robustness, and can be applied in many fields such as information security, financial services, and healthcare.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, and in particular relates to a method, system and application for enhancing the robustness of model watermarking, which can be used in many fields such as information security, financial services, healthcare and legal services. Background Technology

[0002] With the widespread adoption and accelerated commercialization of artificial intelligence models, model watermarking technology has emerged as an important research direction in the field of deep learning. Its core function is to provide a means for confirming ownership of models.

[0003] Currently, model watermarking technology can be broadly categorized into two types based on whether model involvement is required for watermark extraction: black-box watermarking and white-box watermarking. Within black-box model watermarking schemes, based on the watermark embedding and verification principles, they can be further divided into data-independent model watermarking, statistical characteristic-based model watermarking, and backdoor model watermarking. Backdoor model watermarking is an anti-tampering technique that combines backdoor attacks with watermarking technology. It is typically used to protect the intellectual property rights of deep learning models or to embed certain information without affecting model performance. Its core idea is to add a hidden backdoor to the model and use this backdoor to embed watermark information. This method not only cleverly hides the watermark information but also effectively proves ownership of intellectual property rights through the model's behavior.

[0004] Patent document CN202411448920.X discloses a "robust watermarking method for screen-captured images based on attention mechanism and contrastive learning." The method first uses an encoder to generate an encoded image containing watermark information. Second, the encoded image and the carrier image are input into a discriminator, which outputs a predicted value. Then, the encoded image is distorted. Next, the distorted encoded image is input into a decoder to extract the watermark information hidden within it. Finally, based on the discriminator's prediction and a joint loss function, the encoder, discriminator, and decoder are trained. The trained encoder and decoder form a screen-capture watermarking model, which is used to encode and decode screen-captured images. This method ensures the invisibility of the encoded image by optimizing the watermark encoding process. However, its attention-based watermark generation method is susceptible to interference and misleading by attackers. Attackers can add specific interference information to the image, causing the attention mechanism to focus on incorrect areas, thus ignoring important parts of the watermark information and making the watermark easily removable.

[0005] Patent document CN202411011804.1 discloses "A Robust Generative Model Watermarking Processing Method and System Resistant to Attacks". It first obtains a dataset based on the target model to be watermarked, where the dataset consists of input-output data pairs of the target model. Second, the dataset is divided into two parts: a first dataset and a second dataset. The first dataset is used for the initial watermarking training stage, and the second dataset is used for the adversarial training stage of the watermarking model. Then, in the initial watermarking training stage, the first dataset is fed into a watermark embedding network for labeling, and then simulated attacks and watermark extraction are performed on the labeled images. Finally, in the adversarial training stage of the watermarking model, the second dataset is used to train two alternative models, and then the watermark extraction network is fine-tuned to extract the watermark from the output of the watermarked alternative models. While this method can improve the invisibility of the model watermark, it suffers from robustness loss in multiple rounds of processing. After multiple rounds of processing, the integrity and extractability of the watermark will be affected. Furthermore, this method only simulates and defends against known common attack methods and lacks sufficient defense against watermark removal attacks that attackers have carefully designed to target their watermark generation and extraction mechanisms.

[0006] Patent document CN202310525668.7 discloses "A method for constructing a reversible robust watermark embedding and extraction model resistant to image attacks," which mainly includes a first Haar wavelet transform, a channel encoding module, a channel decoding module, a reversible module, an embedding and separation module, a first image attack module, and a second image attack module. The channel encoding module includes a channel decoder, an upsampler, an expander, and a second Haar wavelet transform connected in sequence; the channel decoding module includes a second separator, a first inverse Haar wavelet transform, a third separator, and a channel decoder connected in sequence; the reversible module includes multiple feature aggregation layers, which are connected in sequence and then connected to a convolutional neural network; the embedding and separation model includes a first separator, a second inverse Haar wavelet transform, and a third Haar wavelet transform; the first image attack module includes a first attack layer, and the second image attack module includes a second attack layer and a third attack layer. This method improves the accuracy of watermark information extraction by having multiple modules work together. However, to ensure the watermark's imperceptibility—that is, to not affect the visual quality and usability of the original image—the embedding strength of the watermark is limited. If the embedding strength is too low, the model will face challenges in maintaining the integrity and detectability of the watermark when facing some routine image processing operations or carefully planned adversarial attacks, which in turn makes the watermark easy to remove. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing model watermarking technologies by proposing a robust enhancement method for model watermarking based on adversarial immunity, so as to improve the extraction effect of model watermarks and ensure that they can effectively resist various watermark removal attacks.

[0008] The technical solutions for achieving the objectives of this invention include a method and system for enhancing the robustness of model watermarking, and the use of model watermarking.

[0009] 1. A method for enhancing the robustness of model watermarks based on adversarial immunity, characterized by comprising:

[0010] (1) Select an existing convolutional neural network model suitable for image classification tasks as the initial target model, and use a public dataset suitable for image classification tasks as the original data to train the target model to obtain the trained target model.

[0011] (2) Use the existing variational autoencoder (VAE) including encoder and decoder as the initial watermark generation model, and train the initial watermark generation model using the original data to obtain the trained watermark generation model.

[0012] (3) Input the original data into the trained watermark generation model to generate watermark data x. wm ;

[0013] (4) After concatenating the watermark data with the original data, input it into the trained target model for forward propagation to complete the embedding of the watermark in the model;

[0014] (5) Add tiny perturbations that are difficult to detect with the naked eye to the original data to generate adversarial examples;

[0015] (6) Input the original data into the model watermark after watermark embedding for forward propagation, and then input the adversarial sample and the original data into the model watermark for robust training to obtain a robust model watermark.

[0016] 2. A robust model watermarking system, characterized in that it comprises:

[0017] The dataset preprocessing module is used to perform horizontal flipping, pixel padding, random cropping, and normalization operations on the public dataset suitable for image classification tasks in sequence, and the preprocessed dataset is used as the original data.

[0018] The target model training module is used to input the raw data into the initial target model for forward propagation to obtain the trained target model;

[0019] The watermark generation model training module is used to input the original data into the initial watermark generation model for forward propagation to obtain the trained watermark generation model.

[0020] The watermark data generation module is used to input the original data into the trained watermark generation model. The encoder of the watermark generation model performs feature transformation and information hiding on the original data, and then the decoder optimizes and extracts the intermediate data generated during the encoding process to obtain the generated watermark data.

[0021] The watermark data embedding module is used to concatenate the generated watermark data with the original data and then input it into the trained target model for forward propagation to obtain the model watermark.

[0022] The robust training module for model watermarking is used to add minute perturbations that are difficult to detect with the naked eye to obtain adversarial examples to the original data. The original data and adversarial examples are then concatenated and input into the model watermarking for forward propagation, thereby achieving robust training of the model watermarking and obtaining a model watermark with enhanced robustness.

[0023] 3. A method for enhancing the robustness of model watermarking, characterized by comprising:

[0024] S1) Users select a matching watermark extraction algorithm based on the specific algorithm used by the developer when embedding the watermark in the model.

[0025] S2) The user loads the model watermark into the set extraction environment and runs the extraction algorithm to obtain the watermark information. During this process, the stability and security of the environment must be ensured to prevent external interference from affecting the accuracy of the extraction results.

[0026] S3) Verify whether the key features such as the pre-set copyright owner's specific identifier and copyright statement in the extracted watermark information can be correctly identified:

[0027] If all key features in the watermark can be correctly identified, it indicates that the watermark is intact, and step S5 is executed.

[0028] Otherwise, it indicates that the watermark is missing or damaged, and an attempt is made to repair the watermark information by proceeding to step S4.

[0029] S4) Use the algorithm employed when embedding the watermark in the model to repair the watermark information, and verify whether the key features in the repaired watermark can be correctly identified:

[0030] If correctly identified, it indicates successful watermark repair; proceed to step S5.

[0031] Otherwise, it indicates that the copyright of the model is suspected of being misappropriated, and a further investigation is needed based on the dissemination channels of the model;

[0032] S5) Compare the key features of the extracted watermark information with the key features of the original watermark information to determine whether the model's copyright has been misappropriated:

[0033] If the two are completely identical, then the copyright ownership of the model is determined to be correct;

[0034] Otherwise, if the copyright of the model is deemed to have been misappropriated, appropriate copyright protection measures should be initiated to pursue liability for infringement.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] Firstly, this invention uses a variational autoencoder (VAE) as the initial watermark generation model. During the training process of the watermark generation model, the original data is input into the encoder of the watermark generation model. The perceptual loss between the encoder output data and the original data is used as the key constraint for training the watermark generation model. Compared with existing watermark generation model training methods, this invention significantly improves the quality of the watermark data generated by the watermark generation model, making the watermark data generated by this invention almost visually indistinguishable from the original data. This greatly improves the concealment of the watermark data when embedded in the model, effectively avoiding the risk of the model being identified or tampered with due to the easy detection of the watermark. This provides more reliable technical support for model copyright protection and watermark applications.

[0037] Secondly, this invention performs robust training on the trained model watermark. Specifically, it adds minute perturbations that are difficult to detect with the naked eye to generate adversarial examples to the original data. These adversarial examples are then concatenated with the original data to form adversarial heterogeneous data, which is then input into the model watermark for forward propagation. During this process, a penalty mechanism is set to constrain the model watermark's behavior of predicting the original data as watermark data. At the same time, the robustness training of the model watermark is constrained by the loss function of adversarial heterogeneous data. Therefore, compared with existing model watermarks, the model watermark trained by this invention has stronger robustness and generalization ability, and can effectively resist various current watermark removal attacks. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the implementation of the robustness enhancement method for model watermarking based on antagonistic immunity in this invention.

[0039] Figure 2 This is a block diagram of the robustness enhancement system for model watermarking based on antagonistic immunity according to an embodiment of the present invention;

[0040] Figure 3 This is a diagram illustrating the method of using watermarks in the model according to the present invention. Detailed Implementation

[0041] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings:

[0042] Example 1: The present invention provides a robust watermark enhancement method for models based on antagonistic immunity.

[0043] To address the current situation where existing model watermarks cannot effectively resist various watermark removal attacks, this invention enhances the resistance of the model watermark to various interferences and attacks through an additional robust training process, and enhances the generalization ability of the model watermark under different data distributions, resulting in a highly robust model watermark.

[0044] Reference Figure 1 The implementation steps of this example include the following:

[0045] Step 1: Pre-training dataset.

[0046] 1.1) Select existing public datasets suitable for image classification tasks, such as CIFAR10, CIFAR100, MNIST, ImageNet, SVHN, etc., as the original data;

[0047] This example uses the CIFAR10 dataset as the original data. The images in this dataset are divided into 10 different categories, and each category contains 6,000 images.

[0048] 1.2) Perform preprocessing operations on the original data in sequence, including horizontal flipping, pixel filling, random cropping, and normalization. Then, divide the preprocessed original data into training and test sets in a 5:1 ratio.

[0049] Step 2: Pre-train the target model.

[0050] 2.1) Select existing convolutional neural network models suitable for image classification tasks, such as ResNet18, ResNet50, VGG, AlexNet, etc., as the initial target model;

[0051] This example uses a ResNet18 convolutional neural network as the initial target model, which consists of 18 convolutional layers and 1 fully connected layer.

[0052] 2.2) Input the training set data into the initial target model in batches for forward propagation, and use cross-entropy loss to measure the difference between the probability distribution predicted by the model and the label distribution of the original data;

[0053] 2.3) Select the stochastic gradient descent optimizer as the optimizer for the target model, calculate the gradient of the cross-entropy loss with respect to the target model parameters, and then update the target model parameters according to the opposite direction of the gradient.

[0054] 2.4) Use the test set data to evaluate the target model and calculate the model accuracy on the test set respectively;

[0055] 2.5) Repeat steps 2.2) to 2.4) to train the target model for 100 iterations, select target models with an accuracy greater than 85%, sort these target models in descending order of their accuracy, and save the highest-accurate target model locally.

[0056] Step 3: Pre-train to generate a watermark model.

[0057] 3.1) Select the existing variational autoencoder (VAE) as the initial watermark generation model. Its structure consists of two parts: an encoder and a decoder. The encoder is used to map the input data to the latent space, learn the latent feature representation of the input data and calculate the latent variables. The decoder reconstructs the original data based on the latent variables and transforms the representation in the latent space back to the space where the input data is located.

[0058] 3.2) Input the original data x into the encoder of the initial watermark generation model, and output the mean μ and log(σ) of the latent space. 2 The mean μ represents the central location of the data distribution in the latent space, reflecting the average characteristics of the original data x in the latent space. The logarithm and variance log(σ) 2 This measures the degree of dispersion of data in the latent space; taking the logarithm is for the stability and convenience of numerical computation.

[0059] 3.3) Through the mean μ and log(σ) of the latent space 2 The distribution of data in the latent space is determined by the following formula:

[0060]

[0061] Among them, f μ and These are two different output branches of the encoder neural network;

[0062] 3.4) Sample random variables ∈ from the latent space, and combine the mean μ and log-variance log(σ) of the encoder output. 2 ), calculate the latent variable z:

[0063]

[0064] Where ⊙ represents element-wise multiplication;

[0065] 3.5) Input the sampled latent variable z into the decoder of the initial watermark generation model, and output reconstructed data with the same dimension as the original data x.

[0066]

[0067] Where Decoder represents the decoder;

[0068] 3.6) Calculate the original data x and the reconstructed data. Reconstruction loss L MSE In this step, either the mean squared error loss or the mean absolute error loss can be selected as the calculation method for the reconstruction loss L. MSE The standards are implemented as follows:

[0069] 3.6.1) Calculate the reconstruction loss L using the mean squared error loss. MSE Iterate through the original data x and reconstruct the data. For each corresponding pixel, calculate the original data x and the reconstructed data. The average of the squared differences between corresponding elements is used as

[0070] Reconstruction loss L MSE :

[0071]

[0072] Where, x i It is the value of the i-th pixel in the original data x. It is the value of the i-th pixel in the generated image, and N is the total number of pixels in the image;

[0073] 3.6.2) Calculate the reconstruction loss L using the mean absolute error loss. MSE Iterate through the original data x and reconstruct the data. For each corresponding pixel, calculate the original data x and the reconstructed data. The average of the absolute errors of corresponding elements is used as the reconstruction loss L. MSE :

[0074]

[0075] Where, x i It is the value of the i-th pixel in the original data x. It is the value of the i-th pixel in the generated image, and N is the total number of pixels in the image;

[0076] 3.7) Existing pre-trained convolutional neural networks suitable for image classification tasks, such as VGG, MobileNet, ResNet18, and ResNet50, are selected to measure the difference between the original data x and the reconstructed data. Differences at the level of perception;

[0077] This example uses ResNet50 convolutional neural network as the pre-trained convolutional neural network, which consists of convolutional layers, pooling layers, multiple residual block groups and fully connected layers.

[0078] 3.8) Combine the original data x and the reconstructed data The images are fed into a pre-trained ResNet50 convolutional neural network, and their image feature representations φ at layer i are extracted. i (x) and The perceptual loss L is obtained by calculating the Euclidean distance between the two feature representations. Per :

[0079]

[0080] in, It is the Euclidean distance, representing the difference between two feature images;

[0081] 3.9) Based on the random variable ∈, mean μ, and logarithm / variance log(σ) in 3.3), 2 Given the latent variable z, calculate the difference between the posterior distribution q(z|x) and the prior distribution p(z) of the latent variable z as the KL divergence loss L. KL :

[0082]

[0083] Wherein, the posterior distribution q(z|x) satisfies the normal distribution N(μ,σ) 2 The prior distribution p(z) satisfies the standard normal distribution N(0,1);

[0084] 3.10) The reconstruction loss L calculated above MSE Perceived loss L Per and KL divergence loss L KL The sum is used as the total loss of the watermark generation model. The weights of the KL divergence loss are dynamically adjusted using linear annealing. Specifically, the annealing factor β(t) gradually increases as training progresses, eventually reaching 1 in the later stages of training. This allows the KL divergence term to play a more significant role in the later stages of training.

[0085]

[0086] Where λ1 and λ2 are respectively L MSE and L Per The weight hyperparameters are adjusted according to training needs, where t is the current training step and T is the total number of training steps.

[0087] 3.11) Calculate the total loss using a stochastic gradient descent optimizer. The gradient of the watermark generation model parameters is used to update the model parameters in the opposite direction of the gradient, thus obtaining the watermark generation model for the current training round.

[0088] 3.12) Repeat steps 3.2) to 3.11) to iterate and update the model multiple times until the loss occurs. The model converges, resulting in the trained watermark generation model.

[0089] Step 4: Generate watermark data.

[0090] 4.1) Input the original data x into the encoder of the trained watermark generation model, and output the mean μ' and log-variance log(σ') of the latent space. 2 ):

[0091]

[0092] Among them, f μ and These are two different output branches of the encoder neural network;

[0093] 4.2) Sample random variables ∈' from the distribution of the latent space, and use the random variables ∈', the mean μ', and the log-variance log(σ') 2 Calculate the latent variable z':

[0094]

[0095] Where ⊙ represents element-wise multiplication;

[0096] 4.3) Input the latent variable z' into the decoder of the watermark generation model after training, and output the watermark data x. wm :

[0097] x wm =Decoder(')

[0098] Where Decoder represents the decoder;

[0099] Step 5: Embed the model watermark.

[0100] 5.1) Combine the original data x with the watermark data x wm The watermark-enhanced data is obtained by stitching the images together along the same dimension at a 1:1 ratio.

[0101]

[0102] 5.2) Represent the label of the original data x as y, and the watermark data x wm The label is represented as y wm Then, the labels y of the original data and the labels y of the watermark data are... wm The data tags are stitched together along the same dimension at a 1:1 ratio to obtain the stitched watermark-enhanced data tags.

[0103]

[0104] 5.3) Enhance watermark data The input is fed into the trained target model for forward propagation, and the probability distribution predicted by the model is calculated using the cross-entropy loss function. Enhanced data tags with watermarks The difference between them is used as the watermark loss L wn :

[0105]

[0106] Among them, L ce Represents the cross-entropy loss function;

[0107] 5.4) Calculate the watermark loss L using a stochastic gradient descent optimizer. wm Regarding the gradient of the model parameters, the parameters of the trained target model are updated according to the opposite direction of the gradient to complete the watermark embedding training of the target model in the current round.

[0108] 5.5) Repeat steps 5.3) to 5.4) to iterate and update the model multiple times until the watermark loss L wm Convergence enables the embedding of watermarks into the model.

[0109] Step six: Generate adversarial examples.

[0110] In this step, adversarial examples can be generated using either Optimized Projective Gradient Descent (PGD) or Gradient-Based Fast Signed Gradient (FGSM), with the implementation methods as follows:

[0111] 6.1) Adversarial examples are generated using the gradient-based Fast Gradient Signed Method (FGSM), as implemented below:

[0112] 6.1.1) Input the original data x into the model watermark to obtain the prediction result f(x), and calculate the loss function L of the original data x based on the difference between the prediction result f(x) and the original data label y. x :

[0113] L x =L ce (f(x),y)

[0114] Among them, L ce Represents the cross-entropy loss function;

[0115] 6.1.2) Calculate the loss function L x Gradient of the original data x

[0116]

[0117] Where c is the number of categories, f i (x) is the i-th component of the model output f(x);

[0118] 6.1.3) Based on the gradient of the obtained original data x Using symbolic functions To determine the direction of its perturbation to the original data x;

[0119] Where sign(·) represents the direction of the perturbation obtained by the sign function. Taking the input data k as an example, its representation is as follows:

[0120]

[0121] If the output of the sign function is 1, the perturbation direction is positive; if the output of the sign function is -1, the perturbation direction is negative; if the output of the sign function is 0, no perturbation is added to the original data x.

[0122] 6.1.4) Randomly select a value between 0.01 and 0.1 as the perturbation size ω, which is used to determine the degree of difference between the adversarial example and the original data x;

[0123] 6.1.5) Based on the determined disturbance direction The adversarial example x is generated by calculating the perturbation magnitude ω. Adv :

[0124]

[0125] 6.2) Adversarial examples are generated using the optimized projective gradient descent (PGD) method, as implemented below:

[0126] 6.2.1) Initialize a perturbation δ with the same shape as the original data x. t =0, t=0,1,2,...,T, where T is the maximum step size for iterative updates;

[0127] 6.2.2) This disturbance δ t Superimposed on the original data x, we obtain the current adversarial sample x+δ t The adversarial sample x+δ t The prediction result f(x+δ) is obtained by inputting it into the model watermark. t According to the prediction result f(x+δ) t The adversarial example x+δ is calculated by comparing the difference between the original data label y and the original data label y. t loss function L δ :

[0128] L δ =L ce(f(x+δ t ),y)

[0129] Among them, L ce Represents the cross-entropy loss function;

[0130] 6.2.3) Calculate the loss function L δ Regarding the disturbance δ t gradient

[0131]

[0132] Where c is the number of categories, f i (x+δ t ) is the model output f(x+δ) t The i-th component of );

[0133] 6.2.4) Along the gradient Direction update of the perturbation δ at step t+1 t+1 :

[0134]

[0135] Where, δ t α is the perturbation at step t, α is the perturbation step size at each step, and sign(·) indicates the direction of the perturbation obtained by the sign function;

[0136] 6.2.5) Update the perturbation δ t+1 Projected onto L centered at zero with radius ∈ ∞ Within the norm sphere, the current constrained perturbation δ' is obtained. t+1 :

[0137] δ′ t+1 =clip(δ t+1 ,-∈,∈)

[0138] The clip function will δ t+1 Each element is restricted to the range [-∈, ∈];

[0139] 6.2.6) Repeat steps 6.2.2) to 6.2.5) until the set maximum number of iterations T is reached, to obtain the final result.

[0140] Constraint perturbation δ' T Then the disturbance δ' T Add to the original sample x to generate the final adversarial sample x Adv :

[0141] x Adv =x+δ′ T .

[0142] Step 7: Robustness training of the model watermark.

[0143] Although the watermark obtained from the above steps has completely embedded the watermark information, it lacks the ability to learn the distribution of different data features and exhibits poor robustness under current adversarial attacks with strong attack capabilities. To improve its ability to resist various watermark removal attacks, the model watermark is subjected to robust training to enhance its generalization ability and model performance in different data and environments, thereby obtaining a more robust model watermark. The implementation includes the following:

[0144] 7.1) Input the original data x into the model watermark for forward propagation, and calculate the training loss L of the original data x based on the difference between the probability distribution f(x) predicted by the model and the label y of the original data. bn :

[0145] L bn =L ce (f(x),y)

[0146] Among them, L ce Represents the cross-entropy loss function;

[0147] 7.2) During the training process in 7.1), if the original data x is misidentified by the model as watermarked data x... wm In such cases, the penalty for such raw data x can be increased to reduce the model's misclassification of it as watermarked data x. wm To increase the probability of watermarking and thus enhance the robustness of the model, the following penalty rule is specifically formulated:

[0148] Define a penalty mask m, which stipulates that if the label y of the original data x is not equal to the watermark data x, then... wm The tag y wm And the predicted label of the original data x in the model equals watermark data x wm The tag y wm At that time, the original data x and its corresponding label y that satisfy the specification are marked by the penalty mask m, and the original data and label marked by the penalty mask m are defined as the penalty data x respectively. p and penalty data label y p :

[0149] x p =x[m],y p =y[m]

[0150] in,

[0151] 7.3) Compare the original data x with the adversarial sample x AdvBy stitching the data together in a 1:1 ratio along the same dimension, we obtain the stitched adversarial heterogeneous data.

[0152]

[0153] 7.4) Copy the label y of the original data to obtain y', and concatenate it with the label y of the original data in a 1:1 ratio along the same dimension to obtain the concatenated adversarial heterogeneous data label.

[0154]

[0155] 7.5) The spliced ​​adversarial heterogeneous data The input is fed into the model watermark after watermark embedding for forward propagation, based on the probability distribution predicted by the model. heterogeneous data tags Difference calculation to combat heterogeneous data Training loss L robusst :

[0156]

[0157] Among them, L ce Represents the cross-entropy loss function;

[0158] 7.6) Based on the probability distribution f(x) predicted by the model from the penalty data. p ) and penalty data x p The tag y p The difference between them is used to calculate the loss L for penalized training. penalty :

[0159] L penalty =L ce (f(x p ),y p )

[0160] Among them, L ce Represents the cross-entropy loss function;

[0161] 7.7) The training loss L of the original data x above bn Combating heterogeneous data Training loss L robust And the loss of L due to punishment training penalty The sum is used as the total loss L of the model watermark. retotal ;

[0162] To avoid the model watermarking getting stuck in local optima during the early stages of training due to a complex loss function, thus failing to balance complexity and performance, a linear growth approach is used to dynamically adjust L.robust and L penalty The weights of these two losses are gradually increased with each training epoch, allowing the model to focus on learning the feature distribution of the original data x in the early stages of training, while focusing more on adversarial heterogeneous data in the later stages. Training and penalty training are used to improve the overall performance and generalization ability of the model:

[0163] L retotal =L bn +w t L robust +u t L penalty

[0164] Among them, w init For L robust The initial weights, w final For L robust The final weight,

[0165]

[0166] u init For L penalty The initial weights, u final For L penalty The final weight,

[0167]

[0168] t represents the current training round, and T represents the total number of training rounds.

[0169] Example 2: Robustness of Model Watermarking System Based on Adversarial Immunity

[0170] Reference Figure 2 This example includes a dataset preprocessing module 1, a target model training module 2, a watermark generation model training module 3, a watermark data generation module 4, a watermark data embedding module 5, and a model watermark robustness training module 6. Its working principle is as follows:

[0171] The dataset preprocessing module 1 is used to perform horizontal flipping, pixel filling, random cropping and normalization operations on the raw data suitable for image classification tasks in sequence, and divide the preprocessed raw data into training set and test set. The preprocessed raw data is then output to the target model training module 2, the watermark generation model training module 3, the watermark data generation module 4, the watermark data embedding module 5 and the model watermark robust training module 6 respectively.

[0172] The target model training module 2 selects an existing convolutional neural network model suitable for image classification tasks as the initial target model, performs forward propagation on the initial target model using the training set in the original data, updates the model parameters, and evaluates it through the test set to obtain the trained target model output to the watermark data embedding module 5.

[0173] The watermark generation model training module 3 uses a variational autoencoder including an encoder and a decoder as the initial watermark generation model. It uses the raw data input by the dataset preprocessing module 1 to perform forward propagation on the initial watermark generation model, iterates and updates the model parameters multiple times until the loss function converges, and obtains the trained watermark generation model output to the watermark data generation module 4.

[0174] The watermark data generation module 4 uses the encoder in the trained watermark generation model to perform feature transformation and information hiding on the original data input by the dataset preprocessing module 1, and then uses its decoder to optimize and extract the intermediate data generated during the encoding process, and outputs the generated watermark data to the watermark data embedding module 5.

[0175] The watermark data embedding module 5 concatenates the watermark data with the original data input by the dataset preprocessing module 1 and then performs forward propagation on the target model input by the target model training module 2, iterating and updating the model parameters multiple times until the loss function converges, and then outputs the model watermark to the model watermark robust training module 6.

[0176] The robust training module 6 for model watermarking adds subtle perturbations that are difficult to detect with the naked eye to the original data input by the dataset preprocessing module 1 to obtain adversarial examples. Then, the original data and adversarial examples are concatenated and forward propagated to the model watermark. The model parameters are iteratively updated multiple times until the loss function converges, thereby achieving robust training of the model watermark and obtaining a model watermark with enhanced robustness.

[0177] Example 3: How to use model watermarks

[0178] In real-world scenarios involving open-source model sharing, to prevent unauthorized copying, dissemination, and use of models, model watermarking can be used to strengthen copyright protection. Typically, watermark information containing key copyright features, such as the copyright owner's specific identifier and copyright statement, is pre-embedded into the model. Once suspected infringement is detected, users can select a matching watermark extraction algorithm based on the watermark embedding method used by the developer to extract the watermark information in a stable and secure environment. The key features of the extracted watermark information are then checked to ensure they are complete and consistent with the original watermark. If the verification results show any anomalies, legally valid evidence can be provided to the copyright owner to protect their rights, thereby effectively safeguarding intellectual property rights.

[0179] Reference Figure 3 The implementation steps of this example include the following:

[0180] The first step is to select a watermark extraction method: users select a matching watermark extraction algorithm based on the specific algorithm used by the developer when embedding the watermark in the model.

[0181] In this step, assuming the developer uses a discrete cosine transform algorithm to embed the copyright watermark into the model's parameters, the user needs to select a watermark extraction algorithm based on the discrete cosine transform algorithm to effectively extract the hidden watermark information in the model.

[0182] The second step is to extract the watermark information: The user loads the model watermark into the set extraction environment and runs the extraction algorithm to obtain the watermark information. During this process, it is necessary to ensure the stability and security of the environment. It is advisable to load the model watermark in a specially built secure server environment that has undergone strict network security protection and system stability testing to prevent external interference from affecting the accuracy of the extraction results.

[0183] The third step is to verify the integrity of the watermark information: This involves verifying whether key features such as the pre-defined copyright owner's identifier and copyright statement in the extracted watermark information can be correctly identified.

[0184] If all key features in the watermark can be correctly identified, it indicates that the watermark is intact, and proceed to step five.

[0185] Otherwise, it indicates that the watermark is missing or damaged. Attempt to repair the watermark information and proceed to step four.

[0186] The fourth step is to repair the watermark information: using the algorithm employed when embedding the watermark in the model, the watermark information is repaired, and the key features in the repaired watermark are verified to be correctly identified.

[0187] If correctly identified, the watermark has been successfully repaired; proceed to step five.

[0188] Otherwise, it indicates that the copyright of the model is suspected of being misappropriated, and it is necessary to investigate the distribution channels of the model on major open source platforms, resource websites, etc., and check whether there is any unauthorized distribution and modification.

[0189] Step 5: Compare watermark information: Compare the key features of the extracted watermark information with the key features of the original watermark information to determine whether the model's copyright has been misappropriated.

[0190] If the two are completely identical, then the copyright ownership of the model is determined to be correct;

[0191] Otherwise, if the copyright of the model is deemed to have been misappropriated, appropriate copyright protection measures should be initiated to pursue liability for infringement.

[0192] Application examples:

[0193] Based on the above steps, taking the copyright protection of game models developed by game companies as an example, the usage of model watermarks is explained as follows:

[0194] S1) Select watermark extraction method:

[0195] When developing game models, game companies use a watermark embedding algorithm based on neural network parameter perturbation and a verification method based on parity check coding to embed the company's copyright logo, copyright statement and other information into the parameters of the game model in a specific way. When a game company suspects that a similar game on the market has copyright infringement, it needs to select a matching watermark extraction algorithm based on the algorithm used to embed the watermark.

[0196] S2) Extract watermark information:

[0197] The game company's technicians loaded the watermark of the suspected infringing game model into a specially built, secure, and stable extraction environment. This environment underwent rigorous network security protection and system stability testing to prevent external interference from affecting the accuracy of the extraction results. Then, a selected watermark extraction algorithm was run to obtain the watermark information from the game model; S3) Verify the integrity of the watermark information:

[0198] Technicians examine the extracted watermark information to check whether key features such as the pre-set identifiers of the game company's copyright owner, such as the company's logo's numerical code, copyright statement, copyright year, and the name of the copyright-owning company, can be correctly identified.

[0199] S4) Repair watermark information:

[0200] Since the game company initially used a watermark embedding algorithm based on neural network parameter perturbation, the technicians recalculated the position of the missing or damaged watermark information in the neural network parameters according to the principle of the algorithm, and re-embedded the correct watermark information into the corresponding parameter position.

[0201] S5) Inspect the repaired watermark information:

[0202] Technicians used the parity check code method used by the developer when embedding the watermark to verify the repaired watermark information. All the key features of the watermark information could be correctly identified, indicating that the watermark repair was successful.

[0203] S6) Compare watermark information:

[0204] Technicians compared the key features of the extracted watermark information with the key features of the original watermark information one by one:

[0205] If the extracted watermark information shows that the company logo's numerical code matches the original watermark information, but the company name in the copyright statement has been altered, it is determined that the copyright of the game model has been misappropriated. In this case, corresponding copyright protection measures will be immediately initiated, and an infringement warning letter will be sent to the infringing party, demanding that they cease the infringing behavior and bear the corresponding legal responsibility. If the infringing party refuses to cooperate, the game company will protect its copyright rights through legal channels.

[0206] The above descriptions are merely a few specific examples of the present invention and do not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of the claims and protection of the present invention.

[0207] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A method for enhancing the robustness of model watermarks based on adversarial immunity, characterized in that, include: (1) Select an existing convolutional neural network model suitable for image classification tasks as the initial target model, and use a public dataset suitable for image classification tasks as the original data to train the target model to obtain the trained target model. (2) The existing variational autoencoder, which includes an encoder and a decoder. As the initial watermark generation model, the original data is used to train the initial watermark generation model to obtain the trained watermark generation model. (3) Input the original data into the trained watermark generation model to generate watermark data. ; (4) After concatenating the watermark data with the original data, input it into the trained target model for forward propagation to complete the embedding of the model watermark; (5) Add subtle perturbations that are difficult to detect with the naked eye to the original data to generate adversarial examples; (6) Input the original data into the model watermark after watermark embedding for forward propagation, and then input the adversarial sample and the original data into the model watermark for robustness training to obtain a robust model watermark.

2. The method according to claim 1, characterized in that: Step (1) uses the original data to train the initial target model, which is implemented as follows: 1a) Perform horizontal flipping, pixel filling, random cropping and normalization operations on the original data in sequence, and then divide the preprocessed original data into training set and test set in a 5:1 ratio. 1b) Input the training set data into the initial target model in batches for forward propagation. Use cross-entropy loss to measure the difference between the probability distribution predicted by the model and the label distribution of the original data. Use the stochastic gradient descent optimizer to calculate the gradient of the cross-entropy loss with respect to the model parameters, and then update the model parameters according to the opposite direction of the gradient. 1c) Evaluate the model using data from the test set, and calculate the model accuracy and loss on the test set; 1d) Repeat 1b) and 1c) to train the model for 100 iterations, select target models with an accuracy greater than 85%, sort these target models in descending order of their accuracy, and save the highest-accurate target model locally.

3. The method according to claim 1, characterized in that: Step (2) trains the initial watermark generation model using the original data, which includes the following: 2a) The original data The input is fed into the encoder of the initial watermark generation model, and the output is the mean of the latent space. and logarithmic variance : ， ， in, and These are two different output branches of the encoder neural network; 2b) Sampling random variables from the distribution of the latent space Using random variables mean and logarithmic variance Calculate latent variables : ， in, This indicates element-wise multiplication; 2c) The latent variables obtained from sampling The input is fed into the decoder that initially generates the watermark model, and the output is the same as the original data. Reconstructed data with the same dimensions : ， in, Indicates decoder; 2d) Define the loss function for the initial watermark generation model. : ， in, For raw data With reconstructing data Reconstruction loss, For raw data With reconstructing data Perceptual loss, for Divergence loss, For the current training step Annealing factor; yes Weight hyperparameters, yes The weight hyperparameters; 2e) Calculate the above loss using the optimizer. The gradient of the watermark generation model parameters is used to update the model parameters in the opposite direction of the gradient, thus obtaining the watermark generation model for the current training round. 2f) Repeat steps 2a) to 2e) to iterate and update the model multiple times until loss occurs. The model converges, resulting in the trained watermark generation model.

4. The method according to claim 3, characterized in that: In step 2d) Perceived loss , They are represented as follows: ， ， ， in, For raw data The 1 pixel, To reconstruct data The 1 pixel, , For raw data In pre-trained networks The Middle Image feature representation of the layer, To reconstruct data In pre-trained networks The Middle Image feature representation of layers, pre-trained network It is calculating perceived loss The existing convolutional neural network used at that time.

5. The method according to claim 1, characterized in that: The generation of watermark data in step (3) is implemented as follows: 3a) The original data The input is fed into the encoder of the trained watermark generation model, and the output is the mean of the latent space. and logarithmic variance : ， ， in, and These are two different output branches of the encoder neural network; 3b) Sampling random variables from the distribution of the latent space ', using random variables ', mean 'and log-variance' Calculate latent variables : ， in, This indicates element-wise multiplication; 3c) Latent variables The data is input into the decoder of the trained watermark model and outputs the watermark data. : ， in, This indicates the decoder.

6. The method according to claim 1, characterized in that: In step (4), the watermark data is concatenated with the original data and then input into the trained target model for forward propagation to complete the embedding of the watermark in the model. The implementation includes the following: 4a) The original data With watermark data The watermark-enhanced data is obtained by stitching the images together along the same dimension at a 1:1 ratio. : ； 4b) The original data The label is represented as watermark data The label is represented as Then label the original data and watermark data tags The data tags are stitched together along the same dimension at a 1:1 ratio to obtain the stitched watermark-enhanced data tags. : ； 4c) Enhance watermark data The input is fed into the trained target model for forward propagation, and the probability distribution predicted by the model is calculated using the cross-entropy loss function. Enhanced data tags with watermarks The difference between them is used as the watermark loss. : ， in, Represents the cross-entropy loss function; 4d) Calculate watermark loss using the optimizer Regarding the gradient of the model parameters, the parameters of the trained target model are updated according to the opposite direction of the gradient to complete the watermark embedding training of the target model in the current round. 4e) Repeat steps 4c) through 4d) to iterate and update the model multiple times until the watermark is lost. Convergence enables the embedding of watermarks into the model.

7. The method according to claim 1, characterized in that: Step (5) adds minute perturbations that are difficult to detect with the naked eye to the original data to generate adversarial examples. The implementation includes the following: 5a) Initialize a data set with the original data Perturbations of the same shape =0, t=0,1,2,..., , For the maximum of iterative updates Step length; 5b) This disturbance Overlay onto the original data The above yields the current adversarial examples. + The adversarial sample + The prediction result is obtained by inputting it into the model watermark. According to the prediction results With raw data labels Difference calculation of adversarial examples + loss function : ， in, Represents the cross-entropy loss function; 5c) Calculate the loss function Regarding the disturbance gradient : ， Where c is the number of categories, It is the model output result The One component; 5d) Along the gradient Direction update +1 step perturbation : ， in, It is the first The disturbance of the step, It is the perturbation step size at each step. ( The sign function is indicated by the direction of the perturbation. 5e) Update the perturbation Projected onto zero-centered radius Within the norm sphere, the current constrained perturbation is obtained. : ， in, The function will Each element is limited to [ Within the range; 5f) Repeat steps 5b) to 5e) until the set maximum number of iterations is reached. This yields the final constrained perturbation. Then the disturbance Add to original sample Generate the final adversarial example : 。 8. The method according to claim 1, characterized in that: Step (6) involves inputting the original data into the watermarking model after watermark embedding for forward propagation, and then concatenating the adversarial examples with the original data and inputting them into the watermarking model for robust training. The implementation includes the following: 6a) The original data The input is fed into the model watermark after watermark embedding for forward propagation, based on the probability distribution predicted by the model. Labels of raw data Calculate the difference between the original data training loss : ， in, Represents the cross-entropy loss function. This represents the predicted output of the original data in the model; 6b) Set penalty rules: Set a penalty mask The original data is specified. tags Not equal to watermark data tags And the original data Predicted labels in the model Equal to watermark data tags At that time, through the penalty mask Mark the original data that meets the requirements. and its corresponding tags And will be masked with penalties The raw data and labels of the markers are defined as penalty data. and penalty data labels : ， ， in, , 6c) Transfer the raw data Adversarial Examples By stitching the data together in a 1:1 ratio along the same dimension, we obtain the stitched adversarial heterogeneous data. : ； 6d) Label the raw data Get a copy ', and compare it with the labels of the original data. By splicing the data in the same dimension at a 1:1 ratio, we obtain the spliced ​​adversarial heterogeneous data labels. : ； 6e) The spliced ​​adversarial heterogeneous data The input is used for robust training of the watermark model after watermark embedding. 6e1) Let the total loss function of the model watermark be... for: ， in, It represents the adversarial heterogeneous data predicted by the model watermark. probability distribution Combating heterogeneous data tags The differences between them express The weights; The loss from training is used to penalize the model's watermark predictions; it represents the penalty data. probability distribution With penalty data tags The differences between them express The weights; 6f) Calculate the above loss using the optimizer. Regarding the gradient of the model watermark parameters, the trained model watermark parameters are updated according to the opposite direction of the gradient to enhance the robustness of the model watermark in the current round. 6g) Repeat steps 6a) to 6f) to iterate and update the model multiple times until the loss is reached. Convergence completes robust training of the model watermark.

9. A robust model watermarking system, characterized in that, include: The dataset preprocessing module is used to perform horizontal flipping, pixel padding, random cropping, and normalization operations on the public dataset suitable for image classification tasks in sequence, and the preprocessed dataset is used as the original data. The target model training module is used to input the raw data into the initial target model for forward propagation to obtain the trained target model; The watermark generation model training module is used to input the original data into the initial watermark generation model for forward propagation to obtain the trained watermark generation model. The watermark data generation module is used to input the original data into the trained watermark generation model. The encoder of the watermark generation model performs feature transformation and information hiding on the original data, and then the decoder optimizes and extracts the intermediate data generated during the encoding process to obtain the generated watermark data. The watermark data embedding module is used to concatenate the generated watermark data with the original data and then input it into the trained target model for forward propagation to obtain the model watermark. The robust training module for model watermarking is used to add minute perturbations that are difficult to detect with the naked eye to obtain adversarial examples to the original data. The original data and adversarial examples are then concatenated and input into the model watermarking for forward propagation, thereby achieving robust training of the model watermarking and obtaining a model watermark with enhanced robustness.

10. A method of using the model watermarking system of claim 9, characterized in that, include: S1) Users select a matching watermark extraction algorithm based on the specific algorithm used by the developer when embedding the watermark in the model. S2) The user loads the model watermark into the set extraction environment and runs the extraction algorithm to obtain the watermark information. During this process, the stability and security of the environment must be ensured to prevent external interference from affecting the accuracy of the extraction results. S3) Verify whether the pre-defined copyright owner identifier and key features of the copyright statement in the extracted watermark information can be correctly identified: If all key features in the watermark can be correctly identified, it indicates that the watermark is intact, and step S5 is executed. Otherwise, it indicates that the watermark is missing or damaged, and an attempt is made to repair the watermark information by proceeding to step S4. S4) Use the algorithm employed when embedding the watermark in the model to repair the watermark information, and verify whether the key features in the repaired watermark can be correctly identified: If correctly identified, it indicates successful watermark repair; proceed to step S5. Otherwise, it indicates that the copyright of the model is suspected of being misappropriated, and a further investigation is needed based on the dissemination channels of the model; S5) Compare the key features of the extracted watermark information with the key features of the original watermark information to determine whether the model's copyright has been misappropriated: If the two are completely identical, then the copyright ownership of the model is determined to be correct; Otherwise, if the copyright of the model is deemed to have been misappropriated, appropriate copyright protection measures should be initiated to pursue liability for infringement.

Images