Generative model-oriented image data set copyright verification method based on spatial watermark

Abstract

The invention provides a generative model-oriented image data set copyright verification method based on a spatial domain watermark. The method includes a protected data set generation and verification module. The protected data generation module is used for selecting samples with a fixed proportion from the original data set, carrying out watermark adding operation on the selected samples, and forming a protected data set by unselected samples in the original data set and a watermark sample set; the module for verifying whether the suspicious model uses the protected data set comprises the following steps: carrying out watermark adding operation on all samples to generate a watermark data set; performing enhancement operation on the watermark data set to generate an enhanced data set; training a watermark classifier by using the watermark data set, the enhanced data set and the original data set; enabling the suspicious generation model to randomly generate a sample; judging whether the generated sample has a watermark or not by using a classifier; and verifying by using a hypothesis testing mode to verify whether the suspicious generation model uses the protected data set or not.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a data set protection method for generating models. Background Art

[0002] In recent years, deep learning technology has developed rapidly. For deep neural networks, the importance of data sets is self-evident. By using high-quality data sets for training, neural networks can usually show excellent and promising performance. However, in order to build such a data set, it usually takes a lot of time and resources, so the data set is a valuable asset. For the owner of the data set, it is important to detect whether a generated model is trained using a published data set. Such detection can help protect the security of the data set. By verifying whether the generated model is trained using a published data set, the owner of the data set can ensure that their assets are properly protected. Summary of the invention

[0003] The present invention aims to provide an effective data set ownership protection method based on spatial domain watermark and oriented to generative model.

[0004] The data set protection method for generating models based on spatial watermarks described in the present invention includes the following two modules: generating a protected data set and verifying whether a suspicious model is trained using the protected data set:

[0005] Generate protected dataset module: select a certain number of samples S from the original dataset D, add a preset watermark t to the selected samples, and form a watermark sample set S w , the unselected samples D in the original data set u And watermark sample set S w Composed of protected dataset D p ;

[0006] Verify whether the suspicious model is trained using a protected dataset: generate a watermark dataset; generate an enhanced dataset; train a classifier to determine whether a sample carries a watermark signal; and perform a hypothesis test to verify whether the model is trained using a protected dataset.

[0007] In a preferred embodiment, a protected data set module is generated, a certain number of samples S are selected from the original data set D, and a preset watermark t is added to the selected samples to form a watermark sample set S w , the unselected samples D in the original data set u And watermark sample set S w Composed of protected dataset D p , the specific way to add watermark is: Assuming that the sample in the original data set is f(x,y) and the watermark sample is t, the formula for adding a watermark to the sample f(x,y) in the data set is as follows, where μ represents the transparency of the watermark, p represents the horizontal coordinate distance from the upper left corner vertex of the watermark image to the upper left corner vertex of the original data image in the sample set, q represents the vertical coordinate distance from the upper left corner vertex of the watermark image to the upper left corner vertex of the original data image in the sample set, and w and h represent the width and height of the watermark image respectively.

[0008] In a preferred embodiment, verifying whether the suspicious model uses the protected data set for training the module to generate the watermark data set is specifically as follows: For all samples in the original data set D, add a preset watermark t to the selected samples to generate a watermark data set D w ; Add watermark to the sample. The method of adding watermark to the sample is the same as the method of adding watermark to the protected data generation module.

[0009] In a preferred embodiment, verifying whether the suspicious model uses the protected data set for training the module to generate an enhanced data set is specifically as follows: Use the watermark dataset to train the VAE model, and select the samples with obvious watermarks generated by the model as the enhanced dataset D z ;

[0010] In a preferred embodiment, verifying whether the suspicious model uses the protected data set to train the module to train the classifier to determine whether the sample carries a watermark signal is specifically as follows: Set the labels of all samples in the original dataset D to y n = 0, the watermark data set D w The labels of all samples in are set to y s = 1, and the enhanced dataset D z The labels of all samples in are set to 1, and the classifier is trained using the above dataset.

[0011] In a preferred embodiment, the module for verifying whether the suspicious model is trained using the protected data set assumes that the verification model is trained using the protected data set, specifically: Construct the null hypothesis H 0 : No suspicious model was detected to have been trained using the released dataset; Set the alternative hypothesis H 1 : Suspicious models were detected to be trained using protected datasets; Set β as the probability that the classifier mistakenly predicts the sample label without watermark as watermarked, and set σ as the probability that the classifier predicts the sample prediction label generated by the suspicious model as watermarked; given the certainty thresholds τ = 0.05, γ = 0.05, if it satisfies: Then we can reject the null hypothesis H at the significance level γ. 0 , indicating that the suspicious model is trained using the protected dataset, where N is the number of samples generated by the suspicious model in step 5, t 1-γ is the (1-γ)th quantile of a distribution with (N-1) degrees of freedom. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram showing an application scenario in which the present invention can be applied; DETAILED DESCRIPTION

[0013] The technical solution provided in this specification is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only the parts related to the relevant inventions are shown in the accompanying drawings. It should be noted that, in the absence of conflict, the embodiments of this specification and the features in the embodiments can be combined with each other.

[0014] As mentioned earlier, infringers may use datasets without authorization, which will bring hidden dangers and losses to the owner of the dataset. Encryption is a classic data protection strategy. With this method, the dataset is converted into a format that can only be interpreted by authorized users who hold a specific key. Users with the key can easily decrypt the dataset to obtain the original content, while unauthorized users without the key cannot decode and access the data. Although this ensures the security of the data, it also limits the use of the dataset accordingly.

[0015] To this end, the present invention provides a dataset protection method for a generated model, thereby realizing dataset protection for a generated model. Taking the dataset to be protected as an open source dataset for an image generation task as an example, Figure 1 FIG. 1 is a schematic diagram showing an application scenario in which the embodiments of this specification can be applied. Figure 1As shown, a part of the data is selected from the original data set to form a selected sample set, and a watermark is added to the selected sample set to form a watermarked sample set. The unselected samples in the original data set and the watermarked data set form a protected data set. Watermarks are added to all samples in the original data set to generate a watermarked data set, and the VAE model is trained using the watermarked data set to generate samples. Samples with obvious watermark signals are selected to form an enhanced data set. A classification model is trained using the original data set, the watermarked data set, and the enhanced data set to enable it to correctly classify samples with watermark signals. In the application scenario, the owner of the data set deploys the protected data set online for training the generation model. When it is suspected that an image generation model is trained using the protected data set, the suspicious model is generated with a certain number of samples, and the classifier is used to detect the probability of watermarks in the samples, and the hypothesis test is used to verify. The judgment basis is as follows: If the suspicious image generation model is trained using the protected data set, the suspicious image generation model will learn the relevant watermark knowledge in the protected data set, and some samples in its generated sample set may be watermarked.

[0016] See also Figure 1 , Figure 1 The figure shows a flow chart of a data set protection method for a generated model based on a spatial domain watermark. Figure 1 As shown, the data set protection method for the generated model based on spatial domain watermarking may include the following steps:

[0017] Step 1: Select some samples from the original data set to form a selected sample set, embed a watermark at a preset position for each data in the selected sample set to form a watermark sample set, and combine the watermark sample set and the unselected data in the original data set to form a protected data set.

[0018] In step 1, as for the size of the selected sample set, a preset number of samples can be selected from the original data set to form a sample data set; or a preset proportion of samples can be selected from the original data set to form a selected sample set. As for the method of selecting samples, it can be randomly selected from the entire original data set; or some samples can be selected to form a sample data set according to the characteristics of the data set to be protected. For example, if the data set to be protected is the Celeba data set, sample data can be selected according to the facial attributes to form a sample data set. Obviously, the smaller the proportion of the selected sample set to the original sample set, the more hidden the watermark is, and the less likely it is for the user of the data set to find it. However, the less likely the generative model trained using the data set will learn the watermark knowledge in the published data set, the lower the probability that the randomly generated image of the generative model can detect the watermark, which may affect the judgment result. Therefore, it is very important to select samples with an appropriate sample ratio. It is recommended to use a sample ratio between 5% and 20%.

[0019] In step 1, the original data in the selected sample set is an image, and the preset watermark is a watermark image. The preset position is the distance between the upper left corner vertex of the watermark image and the upper left corner vertex of the original image sample in the selected data set. The watermark embedding method can be to add the pixel values ​​of the corresponding positions of the selected sample and the preset watermark according to the ratio μ. Assume that each image in the selected sample set is represented as: f selected (x,y)x∈[0,W),y∈[0,H) The watermark image is represented as: t(x,y)x∈[0,w),y∈[0,h) Where W and H represent the width and height of the samples in the original dataset, respectively; w and h represent the width and height of the watermarked image, respectively. Assume that the preset position is (p, q), where p represents the horizontal coordinate distance between the upper left corner vertex of the watermark image and the upper left corner vertex of the original data image in the sample set, and q represents the vertical coordinate distance between the upper left corner vertex of the watermark image and the upper left corner vertex of the original data image in the sample set. Use the preset adding weight factor μ, μ∈[0,1], μ represents the transparency of the watermark added at the preset watermark adding position of the sample in the selected image. For each sample x in the selected sample set selected Embed the preset watermark image t according to the preset position, the formula is as follows:

[0020] Step 2: Add watermarks to all samples in the original dataset D to generate watermark dataset D w ;

[0021] The operation of adding a watermark in step 2 is the same as step 1.

[0022] Step 3: Use the watermark dataset to train the VAE model, let it generate samples, and select samples with obvious watermark signals to form an enhanced dataset.

[0023] In step 3, the hidden layer dimension of the VAE model is 128, the training batch size is 64, the model learning rate is 0.005, the decay rate is set to 0.95, and the KL divergence is set to 0.00025.

[0024] Step 4: Use the original data set, watermarked data set, and enhanced data set to train a classification model so that it can correctly classify samples with watermarked signals.

[0025] In step 4, the label of the sample in the original dataset is set to y n = 0, the labels of the samples in the watermark dataset and the enhanced dataset are set to y s =1.

[0026] In step 4, the classifier is a convolutional neural network C. The pre-trained ResNet-18 can be used, and the number of output nodes of the fully connected layer of the model is changed to 2. The stochastic gradient descent SGD optimizer is used, and its learning rate is set to e -5 , momentum is set to 0.9, and weight decay is set to 5e -4 , and its loss function is defined as follows: in, Represents the value of the loss function. The goal of classifier training is to minimize this loss value. N is the total number of samples in the training set. C(x) is the predicted probability of the classifier output when the input is a sample in the original data set. C(W(x)) is the predicted probability of the classifier output when the input is a watermark sample and an enhanced sample.

[0027] In step 4, it is necessary to record the probability β that the classifier classifies the non-watermarked sample as the watermarked sample.

[0028] Step 5: Let the suspicious generation model randomly generate N samples.

[0029] In this embodiment, the suspicious model is ordered to generate N samples, where N is related to the proportion of the watermark sample set to the original data set. If the proportion of the watermark sample set to the original data set is also small, due to the randomness of the suspicious generated samples, even if the suspicious generation model uses the published data set for training, it is possible that all the image samples in the generated sample set do not have watermarks, so N needs to be larger at this time.

[0030] Step 6: Use the classifier to determine whether the generated sample has a watermark.

[0031] In step 6, it is necessary to record the proportion σ of the generated samples that the classifier judges as watermarked samples.

[0032] Step 7: Use hypothesis testing to verify whether the suspicious generative model is trained with the published dataset.

[0033] In step 7, construct the null hypothesis H 0 : No suspicious model was detected to have been trained using the released dataset; Set the alternative hypothesis H 1 : Suspicious models were detected to be trained using protected datasets; Set β to the probability that the classifier recorded in step 4 will mistakenly predict the sample label of the watermark as 1, and set σ to the probability that the classifier predicts the sample prediction label generated by the suspicious model as 1; given certainty thresholds τ = 0.05, γ = 0.05, if it satisfies: Then we can reject the null hypothesis H at the significance level γ. 0 , indicating that the suspicious model is trained using the protected dataset, where N is the number of samples generated by the suspicious model in step 5, t 1-γ is the (1-γ)th quantile of a distribution with (N-1) degrees of freedom.

[0034] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A copyright verification method for image datasets based on spatial watermarking for generative models, characterized by The following steps are involved: Step 1: Select some samples from the original data set to form a selected sample set, embed a watermark at a preset position for each data in the selected sample set to form a watermark sample set, and combine the watermark sample set and the unselected data in the original data set to form a protected data set; In step 1, the original data is an image, and the preset watermark is a watermark image; the preset position is the distance between the upper left corner vertex of the watermark image and the upper left corner vertex of the original image sample in the selected data set, and the watermark embedding method is to add the pixel values ​​of the corresponding positions of the selected sample and the preset watermark according to the ratio μ; Assume that each image in the selected sample set is represented as: f selected (x,y)x∈[0,W),y∈[0,H) The watermark image is represented as: t(x,y)x∈[0,w),y∈[0,h) Where W and H represent the width and height of the samples in the original dataset, respectively; w and h represent the width and height of the watermark image, respectively; The preset position is (p, q), where p represents the horizontal coordinate distance between the upper left corner vertex of the watermark image and the upper left corner vertex of the original data image in the sample set, and q represents the vertical coordinate distance between the upper left corner vertex of the watermark image and the upper left corner vertex of the original data image in the sample set; the preset adding weight factor μ is used, μ∈[0,1], μ represents the transparency of the watermark added at the preset watermark adding position of the sample in the selected image; for each sample x in the selected sample set selected Embed the preset watermark image t according to the preset position, the formula is as follows: Step 2: Add watermarks to all samples in the original dataset D to generate watermark dataset D w ; Step 3: Use the watermark dataset to train the VAE model to generate samples and form an enhanced dataset; Step 4: Use the original data set, watermark data set, and enhanced data set to train a classification model so that it can correctly classify samples with watermark signals; Set the label of the sample in the original dataset to y n = 0, the labels of the samples in the watermark dataset and the enhanced dataset are set to y s =1; The classifier is a convolutional neural network C, using ResNet-18, and the number of output nodes of the fully connected layer of the model is changed to 2. The stochastic gradient descent SGD optimizer is used, and the learning rate is set to e -5 , momentum is set to 0.9, and weight decay is set to 5e -4 , and its loss function is defined as follows: in, represents the value of the loss function. The goal of classifier training is to minimize this loss value. N is the total number of samples in the training set. C(x) is the predicted probability of the classifier outputting samples in the original data set. C(W(x)) is the predicted probability of the classifier outputting watermarked samples and enhanced samples. Record the probability β that the classifier classifies a non-watermarked sample as a watermarked sample; Step 5: Let the suspicious generation model randomly generate N samples; Step 6: Use the classifier to determine whether the generated sample has a watermark; Record the proportion σ of the generated samples that the classifier judges as watermarked samples; Step 7: Use hypothesis testing to verify whether the suspicious generative model is trained with the published dataset. Construct the null hypothesis H0: No suspicious model was detected using the released data set for training; Set the alternative hypothesis H1: The suspicious model is detected to be trained using a protected dataset; Set β to the probability that the classifier recorded in step 4 will mistakenly predict the sample label of the watermark as 1, and set σ to the probability that the classifier predicts the sample prediction label generated by the suspicious model as 1; given certainty thresholds τ = 0.05, γ = 0.05, if it satisfies: Then the null hypothesis H0 is rejected at the significance level γ, indicating that the suspicious model is trained using the protected data set, where N is the number of samples generated by the suspicious model in step 5, and t 1-γ is the (1-γ)th quantile of a distribution with (N-1) degrees of freedom.

Images