A method for copyright protection of adversarial generation networks
By embedding the training task of watermark image generation in the adversarial generation network model, the problem that the existing technology is difficult to protect the copyright of the adversarial generation neural network model is solved, and the robustness and concealment of copyright protection is enhanced without affecting the model performance.
Patent Information
- Application Number
- CN202210110918.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-29
- Publication Date
- 2025-03-28
- Estimated Expiration
- 2042-01-29
AI Technical Summary
The prior art is difficult to effectively protect the copyright of adversarial generative neural network models, especially in terms of robustness and concealment, and is difficult to apply to adversarial generative neural network models in different input and output sample forms.
A condition-based adversarial generation network copyright protection method is proposed. By adding watermark images to the model to generate training tasks, the watermark is embedded in the model training process by using the characteristics of the adversarial generation network, ensuring that the watermark is effectively embedded under the premise that the model performance is not affected.
It realizes copyright protection of adversarial generative models without changing the network structure and affecting the performance of the model, enhancing the robustness and concealment of the watermark, and being able to remotely verify model ownership and track the source of leakers.
Smart Images
Figure CN114493972B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence security technology, and specifically relates to an adversarial generation network copyright protection method. Background Art
[0002] Recently, neural networks have achieved breakthrough success in the field of artificial intelligence and have attracted the attention of many researchers. With the popularization of artificial intelligence, some technology companies, such as Google, Xiaomi and other companies, have launched artificial intelligence online services with advanced neural network models, providing consumers with smart products and high-quality services. Such excellent smart services have brought great convenience to people's lives. As artificial intelligence gradually promotes the development of various fields of social economy, how to ensure the healthy and safe popularization and use of artificial intelligence has become a problem worthy of attention. It is worth noting that as a key component in a product or service, commercial neural network models are not easy to train and establish. Because such models require a large amount of training data sets and expensive computing resources. At the same time, neural network models with commercial value are vulnerable to theft by attackers, and the intellectual property rights of model owners are vulnerable to infringement. Therefore, how to protect the intellectual property rights of neural network models has become an urgent problem to be solved.
[0003] Digital watermarks were first used to embed digital signals in data that need to be protected without affecting the availability of the original data. Watermark technology can be used for copyright protection and data tracking. With the increasing popularity of neural network models, the concept of watermarks has been extended to the field of artificial intelligence models. Watermarks are embedded in models that need to be protected and used to protect neural network models. When the owner of a model finds a model that is suspected of being stolen, he or she can respond to copyright disputes and protect his or her rights by activating the watermark hidden in the model. An embedded effective model watermark needs to meet several basic requirements: 1) Fidelity: The diversity and clarity of the images generated by the model will not be reduced by the embedded watermark; 2) Robustness: When subjected to watermark attacks, such as fine-tuning and parameter pruning, the model watermark will not be deleted; 3) Security: The attacker will not detect the watermark; 4) Integrity: The extracted watermark has a small false alarm rate. Model watermarks that meet the above requirements can be put into use.
[0004] At present, there are still many problems to be solved in the field of neural network watermarking, such as: 1) There are only a few watermarking algorithms for generative neural network models, and their robustness needs to be improved; 2) Some neural network watermarking algorithms are easily detected by attackers, and their concealment needs to be improved; 3) In the model watermarking algorithm, it is hoped that not only the copyright of the model can be proved, but also that the watermark can carry a certain amount of information to improve the capacity of the model watermarking algorithm. Therefore, the research on neural network model watermarking algorithms still needs more detailed, in-depth and targeted research, which will have a deeper practical significance and far-reaching impact on the further development of artificial intelligence security research, promoting the healthy and safe practical use of neural networks, and combating illegal and criminal acts that use stolen models for profit. ,
[0005] In current research, most model watermarking algorithm research focuses on neural network models with classification and prediction functions, but such model watermarking algorithms are difficult to apply to adversarial generative neural network models with different input and output sample forms. However, the commercial value of adversarial generative models also needs to be protected.
[0006] Current neural network watermarking technologies are mainly divided into white-box watermarking and black-box watermarking. Among them, white-box watermarking embeds watermarks through model parameters, and it is proposed to use parameter regularizers to embed watermarks in model parameters during neural network training. Although many white-box watermarking methods can effectively embed watermarks and resist watermark attacks, they require the model owner to access the structure and weights of the target model during the verification process, which greatly limits its application in practice. In order to solve this problem, a black-box watermarking method is proposed, which uses backdoor technology to protect the ownership of the model. Through model training or fine-tuning, abstract samples are put into the model training set as backdoor sets, which not only guarantees the original performance of the model, but also can perform ownership authentication. Black-box watermarking can embed watermarks into the backdoor set of the protected network, and detect watermarks by predicting the output of the model. This model watermark can complete verification by remotely accessing the model. However, most of the proposed black-box watermarking algorithms are for classification neural network models, and cannot be applied to adversarial generative neural network models due to the different sample forms of input and output. Summary of the invention
[0007] In view of the shortcomings of the prior art, the present invention proposes a conditional adversarial generative network copyright protection method. According to the redundant characteristics of neural network parameters, a training task of watermark image generation is added to the model, thereby achieving the purpose of model property rights protection and preventing attackers from stealing the model for improper benefits.
[0008] The condition-based adversarial generation network copyright protection method specifically includes the following steps:
[0009] Step 1: Collect N pictures containing the specified object as watermark images;
[0010] The designated object is pictures of the same object in different scenes; for example, if the designated object is a person, pictures of the same person in different scenes are required.
[0011] Step 2: Use the specified key character string through the key generation function to obtain a watermark label of a specific length, and use the watermark label as the watermark label of the watermark image collected in step 1; combine the watermark image and the watermark label as a trigger set.
[0012] The key generation function includes brotli compression algorithm compression.
[0013] Step 3: Put the trigger set obtained in step 2 and the training task image into the adversarial generative network for training to obtain a watermarked network model, as follows:
[0014] Step 3.1, package the trigger set and training task images into tfrecords files for use as training sets;
[0015] The training task images are the original data sets of the adversarial generative network, which are used to train the adversarial generative network.
[0016] Step 3.2, initialize the parameters of the generator and discriminator networks;
[0017] Step 3.3, the generator obtains a generated image based on the random vector z and the label, and uses the generated image, the training task image and the label as inputs of the discriminator; the label includes the watermark label of the trigger set and the original label of the training task image.
[0018] The discriminator returns the quality evaluation result of the generated image based on the calculation as feedback for adjusting the generator parameters. The generator and discriminator train the model according to the adversarial network formula (1).
[0019]
[0020] Among them, x is the real data belonging to P r Distribution, P r represents the image from the training set, z is the noise data belonging to P g Distribution, P g is a random vector with standard normal distribution, E(.) represents the mathematical expectation, G(z) is the generated image obtained by the generator, and D(x) represents the quality evaluation result predicted by the discriminator. α represents the influence parameter of the watermark, L wm Indicates the loss of watermark.
[0021] Step 3.4: Repeat step 3.3. After n rounds, the trained generator and discriminator of watermarked images are obtained. The generator is deployed in the cloud and provided through API.
[0022] Step 4: The model owner holds the watermark label. When a dispute arises over the copyright of a network model, the model owner inputs a specific watermark label into the generator, and the generator returns the corresponding watermark image, which is compared with the provided training watermark image to complete copyright verification.
[0023] 1. The present invention has the following beneficial effects:
[0024] A watermark for protecting adversarial generative models is proposed, which can protect the copyright of the model without changing the structure of the network itself and without affecting the performance of the model, enriching the research on black-box model watermarks.
[0025] The watermark also embeds a certain amount of watermark information through tag embedding, which can remotely verify the model ownership and trace the source of the leaker.
[0026] The watermark embedding method is based on the neural network model characteristics of finding local minima based on model parameter redundancy and training loss, trains additional watermarks in the model, and can be applied to a variety of adversarial generation models, especially in ProGAN and StyleGAN2, with strong algorithm adaptability. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is a flow chart of the model watermark embedding and verification method in the present invention. DETAILED DESCRIPTION
[0028] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0029] The present invention discloses a copyright protection method for adversarial generative networks. The present application embeds a black box model watermark in an adversarial generative network. Since the loss function adjustment is to find a local minimum of the loss during the training of the neural network model, the model can find a new local minimum of the loss after the watermark is embedded, which will not affect the performance of the model itself. Based on this feature, the present application puts the pre-set trigger set watermark and the model task training data into the neural network model for training, and embeds the verification watermark without affecting the performance of the model itself. Among them, the discriminator can learn the ability to discriminate the watermark and feed it back to the generator, and the generator can generate a watermark image according to a specific label. The method of the present application can embed a watermark in the neural network model to achieve the purpose of protecting copyright. Compared with the existing technology, this method can protect the copyright of a new type of neural network model-the generative adversarial network model, and plays a significant role in verifying whether the neural network model has been stolen and protecting the commercial value of the neural network model technology.
[0030] like Figure 1 As shown, a method for protecting copyrights of adversarial generation networks specifically includes the following steps:
[0031] Step 1: Collect N pictures containing the specified object as watermark images.
[0032] In the present invention, 100 images of slime characters were collected, and data enhancement was performed by methods such as rotation, left-right flipping, noise addition, and replication, and finally 3000 color watermark images with a size of 32×32 were obtained.
[0033] Step 2: Set a short string text as the key string and express it in binary form. The specified key string is passed through the key generation function to obtain a watermark label of a specific length. All watermark images have a unified watermark label, and the watermark image and watermark label constitute a trigger set.
[0034] Furthermore, the key generation function includes MD5.
[0035] Furthermore, the specific key string can also be compressed into a watermark label of a specific length containing only {0,1} using the brotli compression algorithm launched by Google that combines a variant of the LZ77 algorithm, Huffman coding, and second-order text modeling, and used as the watermark label of the watermark image collected in step 1.
[0036] Step 3: Put the trigger set obtained in step 2 and the training task image into the adversarial generative network for training to obtain a network model with a watermark. The present invention uses the ProGAN model for experiments, as follows:
[0037] Step 3.1. During the training process, the model not only needs to learn the original training task, but also needs to learn how to generate the watermark image. Due to the large amount of data, a more efficient way to process data I / O is important. In the experiment, the model is trained using the tensorflow framework, and the binary stream tfrecords file is used to store the file, which is conducive to fast data reading and storage. Therefore, the trigger set and training task images are packaged into tfrecords files for use as training sets.
[0038] Step 3.2, initialize the generator parameters θ and the discriminator parameters ω;
[0039] Step 3.3, random vector z is sample data with standard normal distribution, which is used to input into the generator to generate images. The network structure of the generator is progressive, and the generated image is obtained according to the random vector z and the true label. The task of the discriminator is to judge the quality of the generated image.
[0040] During the training process, the generated image, training task image and label are used as the input of the discriminator. The discriminator returns the quality evaluation of the generated image based on the calculation as feedback for adjusting the generator parameters. The generator and discriminator adjust the parameters according to the adversarial network formula (1):
[0041]
[0042] Among them, x is the real data belonging to P r Distribution, P r represents the image from the training set, z is the noise data belonging to P g Distribution, P g is a random vector with standard normal distribution, E(.) represents the mathematical expectation, G(z) is the generated image obtained by the generator, and D(x) represents the quality evaluation result predicted by the discriminator. α represents the influence parameter of the watermark, L wm Indicates the loss of watermark.
[0043] In each round, the parameters of the generator and the discriminator are constantly adjusted. The generator expects the quality of the generated images to be good enough so that the output images have both good quality and can meet the corresponding labels. In the process of adjusting the parameters of the generator, it is necessary to make the discriminator have a good evaluation of the generated images and minimize the distance difference from the original images. The model loss calculation here adopts the Wasserstein distance calculation method proposed in WGAN plus the loss of ACGAN with category judgment. The discriminator is required to be able to distinguish the generated fake images and classify them. The discriminator is equivalent to standing on the opposite side of the generator during the training process. Its task is to minimize the loss of distinguishing the real data while maximizing the loss of distinguishing the generated images.
[0044] Step 3.4, set an epoch, and repeat step 3.3. In this example, we set epoch = 10000, and use the Adam optimizer to adjust the learning rate to speed up the convergence. After the training, the generator and discriminator with watermarks are obtained. The Fréchet Inception Distance evaluation index is used to judge the performance of the generated model. This index can show the distance between the Inception feature vectors of the real image and the generated image in the same domain, and can evaluate the quality and diversity of the generated image. The lower the index, the better the performance.
[0045] When providing AI services in actual applications, the generated model can be deployed in the cloud and provided through API. The model owner holds the watermark label, and the model user can obtain the corresponding service by calling the API.
[0046] Step 4: The model owner holds the watermark label. When a dispute arises over the copyright of a network model, the model owner inputs a specific watermark label into the generator, which returns the corresponding watermark image and compares it with the provided training watermark image to complete copyright verification. In practice, the similarity between the output watermark image and the pre-provided verification image is evaluated based on image quality evaluation indicators. We combine structural similarity index, peak signal-to-noise ratio, and cosine similarity to objectively evaluate the similarity and quality of image watermarks.
Claims
1. A method for protecting copyrights through adversarial generation network, characterized in that The steps include: Step 1: Collect N pictures containing the specified object as watermark images; Step 2: Use the specified key string through a key generation function to obtain a watermark label of a specific length, and use the watermark label as the watermark label of the watermark image collected in step 1; combine the watermark image and the watermark label as a trigger set; Step 3: Put the trigger set obtained in step 2 and the training task image into the adversarial generative network for training to obtain a network model with a watermark; Step 4: The network model owner holds the watermark label. When a dispute arises over the copyright of the network model, the model owner inputs the specific watermark label into the generator, and the generator returns the corresponding watermark image, which is compared with the provided training watermark image to complete the copyright verification. Step 3 is implemented as follows: Step 3.1, package the trigger set and training task images into tfrecords files for use as training sets; Step 3.2, initialize the parameters of the generator and discriminator networks; Step 3.3, the generator obtains a generated image based on the random vector z and the label, and uses the generated image, the training task image and the label as the input of the discriminator; the discriminator returns the quality evaluation result of the generated image based on the calculation as feedback for the generator parameter adjustment; The generator and discriminator train the model according to the adversarial network formula (1); Among them, x is the real data belonging to P r Distribution, P r represents the image from the training set, z is the noise data belonging to P g Distribution, P g is a random vector with standard normal distribution, E(.) represents the mathematical expectation, G(z) is the generated image obtained by the generator, D(x) represents the quality assessment result predicted by the discriminator; α represents the influence parameter of the watermark, L wm Indicates the loss of watermark; Step 3.4: Repeat step 3.
3. After n rounds, the trained generator and discriminator of watermarked images are obtained. The generator is deployed in the cloud and provides services through API.
2. The adversarial generative network copyright protection method according to claim 1, characterized in that The labels described in step 3.3 include the watermark labels of the trigger set and the original labels of the training task images.